Photonic matrix multiplication lights up photonic accelerator and beyond — part 3/4

Cleaned-text code points [28564, 43852). Bibliography omitted here; source.md includes it.

MPLC, as a supersized MVM method, is an inborn alternative to the ONN. In 2018, Lin et al. presented an all-optical diffractive deep neural network (D 2 NN) architecture to perform machine learning 46 . The schematic diagram is shown in Fig. 8a . Five phase-only transmission masks were used to classify images of handwritten digits and fashion products at the speed of light. Then, a modified D 2 NN based on class-specific differential detection was designed to improve the inference accuracy 47 . The information processing capacity of MPLC was recently discussed in detail by Kulce et al. 121 , proving that the dimensionality of the all-optical solution space is linearly proportional to the number of phase planes. While it may be difficult to train the D 2 NN due to the existence of vanishing gradients, it has been suggested to address this issue by directly connecting the input and output using a learnable light shortcut, which offers a direct path for gradient backpropagation in training 122 . The MPLC-D 2 NN can be applied not only in image identification but also in optical logic operations 54 , OAM multiplexing and demultiplexing 50 , optical linear perceptrons 56 and Ising machines 52 . As shown in Fig. 8b , the optical logic functions were performed by a two-layer D 2 NN, and different logic operations were output from different ports after the training 54 . The incident wave was physically encoded at the input layer, and then the compound metasurfaces (hidden layer) scattered the encoded light into one of two small designated areas at the output layer, which provided information on the output logic states. On this foundation, multiple logic gates can be further cascaded to enable more complex or customer-defined functionalities. This universal design strategy holds potential in several applications, such as cryptographically secured wireless communication, real-time object recognition in surveillance systems, and intelligent wave shaping inside biological tissues. Figure 8c presents the coupling and separation of OAM modes with the D 2 NN. Here, four plane masks with pixels of 256×256 were designed to couple and separate four OAM modes. The optical machine learning decryptor in Fig. 8d was realized with single-layer holographic perceptrons, which were trained to complete optical inference missions 56 . This decryptor could perform optical inference for single or whole classes of keys through symmetric and asymmetric decryption. The decryptors could be nanoprinted on complementary metal-oxide–semiconductor (CMOS) chips by galvo-dithered two-photon nanolithography (GD-TPN) with axial nanostepping of 10 nm. The high resolution achieved by GD-TPN allowed achieving a small feature size for the holographic perceptrons at near-infrared telecommunication wavelengths and a neuron density of >500 million neurons per square centimeter. MPLC was also applied in a spatial-photonic Ising machine. The principle of a photonic Ising machine with spatial light modulation is depicted in Fig. 8e 51 , 123 . The spins were encoded into binary optical phases of 0 and π at separated spatial points by an SLM. Intensity modulation was used to set the spin interaction via another SLM. Recurrent feedback from the far-field camera allowed evolution of the phase configuration toward the Ising ground state. It developed a novel hardware with an optics-enabled parallel architecture for large-scale optimizations. A photonic scheme for combinatorial optimization analogous to adiabatic quantum algorithms and classical annealing methods was further studied 52 . More recently, Ruan et al. experimentally evaluated the phase diagram of a high-dimensional spin-glass equilibrium system with 100 fully connected spins under gauge transformation 124 and synchronously proposed implementing an antiferromagnetic model through optoelectronic correlation computation with 40000 spins for the number-partitioning problem 125 . The nonlinear activation functions for D 2 NN were also proved using laser-cooled atoms with electromagnetically induced transparency 126 . To seek a more general and reconfigurable MPLC-based ONN, an optoelectronic fused computing framework based on optical diffraction was proposed, which supports several kinds of neural networks and maintains a high model complexity with millions of neurons 58 . The principle diagram of the basic diffractive processing unit (DPU) is presented in Fig. 9a, b . A digital micromirror device (DMD) and an SLM were assembled to implement the input nodes, and a CMOS sensor was used to implement the optoelectronic neurons. It consists of large-scale diffractive neurons and weighted optical interconnections, enabling the processing of large-scale visual signals, such as images and videos. Three types of ONNs were configured, including the D 2 NN in Fig. 9c , the diffractive network in network (D-NIN-1) in Fig. 9d , and the diffractive recurrent neural network (D-RNN) in Fig. 9e .

a Classification of images of handwritten digits and fashion products 46 . b Optical logic operations 54 . c OAM multiplexing and demultiplexing 50 . d Optical linear perceptrons 56 . e Photonic Ising machine 51 , 123 . a Reprinted by permission from AAAS 46 . b Reproduced from ref. 54 with permission of Springer Nature: Light: Science & Applications. c Reprinted from ref. 50 with the permission of IEEE Publishing. d Reproduced from ref. 56 with permission of Springer Nature: Light: Science & Applications. e Reproduced from ref. 123 . with permission of De Gruyter Publishing

a DPU. b Programmable optoelectronic devices to implement the DPU. c – e Three different types of neural network architectures were constructed, including the D 2 NN, D-NIN-1, and D-RNN. a – e Reproduced from ref. 58 with permission of Springer Nature: Nature Photonics

### MZI-MVMs

Different from MPLC-MVM, the main advantage of MZI-MVM is the potential small size, allowing miniaturized ONN chips. In 2017, Shen et al. proposed a new architecture for a fully optical feedforward neural network, as shown in Fig. 10a 3 . The device, containing 56 programmable MZIs, demonstrated its utility for vowel recognition. It improved the computational speed and power efficiency over advanced electronics for conventional deep learning tasks. Thereafter, an optical convolutional neural network was further proposed. As shown in Fig. 10b , the optical delay lines were implemented with microrings, and the MVM was implemented efficiently in photonic circuits by an MZI mesh 71 . However, the training of these networks was quite difficult and should be followed. Hughes et al. introduced a highly efficient method for in situ training of an ONN. Figure 10c presents a schematic illustration of the proposed method, which uses adjoint variable methods to derive the photonic analog of the backpropagation algorithm 127 . The genetic algorithm was also demonstrated as an efficient method to on-chip train the ONNs 128 . A similar mesh could be expanded to implement a complex-valued neural network 77 . As shown in Fig. 11a , the complex-valued ONN could encode information in both phase and magnitude with MZIs (marked in red). The reference light used for coherent detection was introduced by the MZI in green. The complex-valued weight matrix was implemented with the MZIs in blue. Then, on-chip coherent detection was implemented by the remaining black MZIs. The input preparation, weight multiplication and coherent detection were all integrated onto a single chip, which offered significantly enhanced computational speed and energy efficiency.

a Optical feedforward neural network 3 . b Optical convolutional neural network 71 . c In situ training of an ONN 127 . a Reproduced from ref. 3 with permission of Springer Nature: Nature Photonics. b Reproduced from ref. 71 with the permission of the authors. c Reprinted with permission from ref. 127 © The Optical Society

a Complex-valued ONN with MZI mesh 77 . b , c Photonic recurrent Ising machines with MZI mesh 74 . b The principle of Ising machines and c the energy evolution as a function of time. a Reprinted from ref. 77 with permission from Springer Nature: Nature Communications. b , c Reprinted with permission from ref. 74 © The Optical Society

In addition to neural networks, efforts have also been made to unleash the potential of these photonic architectures by developing algorithms that optimally exploit photonic fundamental advantages. In 2020, Roques-Carmes and Shen et al proposed the photonic recurrent Ising sampler (PRIS) 75 , a heuristic method tailored for parallel architectures allowing fast and efficient sampling from distributions of arbitrary Ising problems. They later experimentally demonstrated the PRIS by combining electronics and silicon-on-insulator photonics 74 . Figure 11b presents the algorithm iteration of the PRIS. The spin state vector was encoded in the amplitudes of coherent optical signals at the input. The transmission matrix of the MZI mesh was dependent on the problem-specific Ising coupling matrix. The output of the matrix multiplication is noisy with Gaussian perturbation. After several algorithm steps, the energy shown in Fig. 11c could approach the ground state, and then the results of optimization for a specific Ising problem were obtained.

### WDM-MVMs

In 2014, Tait and his colleagues proposed, for the first time, using MRR arrays as a matrix computation method primitive for photonic neural networks 82 . This work, for the first time, introduced a scalable neural network architecture called “broadcast-and-weight” based on the WDM concept. In this architecture, as shown in Fig. 12a , neural network weights can be continuously tuned to achieve both positive and negative weights analogous to neural weights. In the same work 82 , Tait et al. also first introduced a network design allowing scalable and cascadable ONNs by employing wavelength reuse, followed by an experimental demonstration in 2017 84 , concurrently with other silicon photonic neuromorphic architectures 3 . This network architecture can be applied to construct both feedforward and recurrent neural networks. Microring weighting banks were also employed for optical CNNs 91 , 92 , 93 . In CNNs, as shown in Fig. 12b , the input images are divided into small patches, and these patches are converted into small matrices for MVM operations. In 2019, an all-optical spiking neural network based on phase-change materials (PCMs) was experimentally demonstrated 89 . As shown in Fig. 12c , the input vectors were loaded on beams with different wavelengths and weighted by PCMs. Moreover, the nonlinear activation function was implemented in optics by changing the resonant wavelengths of the microring when the summed power altered the state of PCMs. Figure 12d shows a photonic tensor core for neural networks using PCMs as the reconfiguration elements 129 . The input matrix was modulated by high-speed modulators, and the kernel matrix was loaded using photonic memory based on PCMs. The weighted inputs were then incoherently summed using a photodetector.

a Optical broadcast-and-weight network showing parallels with the neural network mode 82 , 84 . b Optical convolutional neural network 92 . c All-optical spiking neurosynaptic networks 89 . d Photonic dot product engine for machine learning 129 . a Reprinted from ref. 82 with the permission of IEEE Publishing. b Reprinted from ref. 92 with the permission of IEEE Publishing. c Reprinted from ref. 89 with permission from Springer Nature: Nature. d Reproduced from ref. 129 with the permission of AIP Publishing

Recently, a convolutional photonic processor with extremely high computing throughputs was demonstrated by exploring different dimensions of light. Feldmann et al. demonstrated a highly parallel convolutional processer using an integrated photonic tensor core, achieving 10 12 multiply-accumulate operations per second 90 . A conceptual illustration of the photonic architecture is shown in Fig. 13a . Highly parallel MVMs were performed by using multiple groups of wavelengths generated from a soliton-based optical frequency comb. PCMs were applied as nonvolatile actuators, and thus, convolutional processing can be performed with extremely low power. Another photonic convolutional accelerator realized highly parallel computing by utilizing wavelength-and-time interleaving, as shown in Fig. 13b , which achieved up to 10 trillion operations per second 16 . The input data vector was encoded as the intensity of light with an electro-optical Mach–Zehnder modulator (EOM), and then the wavelength-dependent delay achieved by a single-mode fiber (SMF) was used to reshape the signals at different wavelengths. The convolutional operation was performed at the speed of light by summing the powers at presupposed wavelengths after spectral shaping. These works suggest that photonics is coming of age and in some cases can begin to outperform electronic computation.

a Parallel convolutional processer chip 90 . b Photonic convolutional accelerator 16 . a Reprinted from ref. 90 with permission from Springer Nature: Nature. b Reprinted from ref. 16 with permission from Springer Nature: Nature

Regarding neural networks, all three MVM methods can be used in the linear part of neural networks to achieve photonic acceleration. In contrast, MPLC-based ONNs have the most powerful computing ability and can solve classification problems with all-optical methods, but the refresh rate of spatial planes is limited. MZI-based ONNs are reconfigurable for different situations, but their scale is limited, and electronics-aided learning is needed for complex tasks. To date, WDM-based ONNs have a larger scale than MZI-based ONNs, while they are incoherent computations, where differential detection is often carried out in tandem. Table 2 summarizes the performance comparison of state-of-the-art photonic AI accelerators with electronic hardware. In general, photonic computing has obvious advantages in terms of signal rate, latency, power consumption and computing density, and its accuracy is generally lower than that of electrical computing.

## Discussion

### Scalability and cascadability of ONNs

There exists a huge gap between the number of weights of ANN in electrical and optical MVMs, for example, the weight parameters of ResNet-50, a popular and widely used deep learning network architecture presented by Microsoft in 2016, have already reached 25 million 130 . To alleviate the issue, one direct and effective solution is to manufacture larger-scale photonic integrated circuit (PIC) chips, and indeed, Lightmatter Inc. has released the world-record 64×64 sized MZI mesh integrated chip ‘Mars’ in 2020, which is capable of performing 4096 MAC operations each time when a new set of input vectors is fed in, and the computing capacity is estimated to be 8 TOPS 95 . Similar to integrated circuits, the PIC chips provide the potential to achieve larger scale and higher integration density as the manufacture technologies improve. Furthermore, optical devices promise massive parallelism by employing WDM and mode division multiplexing (MDM) 17 , 90 , these parallel operations can be performed in a single physical optical processing core 90 .