Photonic matrix multiplication lights up photonic accelerator and beyond — part 2/4

Cleaned-text code points [15819, 28564). Bibliography omitted here; source.md includes it.

Benefiting from the large-scale computing capability of spatial planes, MPLC can achieve very powerful matrix functions 44 . For example, Joel Carpenter et al. realized the classification of 210 Hermite–Gaussian modes or Laguerre-Gaussian modes using only 7 phase planes with a pixel size of \(274\times 274\) 49 . A schematic diagram of the Laguerre-Gaussian mode sorter is shown in Fig. 3a . First, Gaussian beams from different positions were injected into the device and converted to different orthogonal Hermite–Gaussian modes by MPLC based on the wavefront matching method 96 . Then, a cylindrical lens pair was used to convert the Hermite–Gaussian mode into the Laguerre-Gaussian mode. The realized super-multimode multiplexer and demultiplexer are of great significance in multimode optical communications. As shown in Fig. 3b , this powerful mode sorter was further used to create time-reversed waves, where all classical linear physical dimensions of light were simultaneously controlled independently 53 . This device can independently address the amplitude, phase, spatial mode, polarization and spectral/temporal degrees of freedom simultaneously through the programming of the SLM. Ninety spatial/polarization modes controlled over 4.4 THz at a resolution of ~15 GHz were demonstrated, covering a total of ~26,000 spatiospectral modes. A reprogrammable metahologram was further designed for optical encryption, as shown in Fig. 3c 55 . The encrypted information was divided into two matrices using two phase planes, and the enciphered message emerged only when the two planes matched.

a Laguerre-Gaussian mode sorter 49 . b Arbitrary vector spatiotemporal field generation 53 . c Optical encryption 55 . a Reprinted from ref. 49 with permission from Springer Nature: Nature Communications. b Reprinted from ref. 53 with permission from Springer Nature: Nature Communications. c Reprinted from ref. 55 with permission from Springer Nature: Nature Communications

Some other applications have also been demonstrated. The MPLC technique was a helpful tool for optimal transverse distance estimation, as shown in Fig. 4a 97 . The measurements were performed in two dimensions far beyond the Rayleigh limit over a large dynamic range. Some theoretical studies were performed. For example, a scalable nonmode selective Hermite–Gaussian mode multiplexer was proposed, as shown in Fig. 4b , where 256 Hermite–Gaussian modes were designed using only seven phase masks 98 . In Fig. 4c , Li et al. implemented the linear polarization mode and Hermite–Gaussian mode demultiplexing hybrids with similar methods 99 , 100 . Each input mode was converted to four fundamental modes with a 90-degree phase difference located at nonoverlapping positions. Local light was uniformly mapped to the fundamental modes with the same phase, which exactly overlapped with output spots from the input modes. The complex amplitudes of the input modes could be retrieved from the interference light intensities. Furthermore, an ultrabroadband polarization-insensitive optical hybrid using MPLC was experimentally verified 57 . As shown in Fig. 4d , 14 phase masks and a gold mirror were employed to carry out the optical hybrid, and a measurement bandwidth of 390 nm was obtained.

a Transverse distance estimation 97 . b Nonmode selective Hermite–Gaussian mode multiplexer 98 . c Mode demultiplexing hybrid 100 . d Ultrabroadband polarization-insensitive optical hybrid 57 . a Reprinted with permission from ref. 97 © The Optical Society. b Reproduced from ref. 98 with the permission of Chinese Laser Press. c Reproduced from ref. 100 with the permission of Chinese Laser Press. d Reprinted from ref. 57 with the permission of IEEE Publishing

Integrated MPLC-MVM was also successfully verified. In 2017, Tang et al. first theoretically proved a novel integrated reconfigurable unitary MPLC-MVM using multimode interference couplers 59 . The schematic diagram is presented in Fig. 5a . The transmission matrix was decomposed into a series of programmable unitary diagonal matrices and fixed unitary diffractive matrices. In theory, an arbitrary unitary transmission matrix can be configured by tuning the unitary diagonal matrices, provided that enough phase planes are assigned. In 2018, the integrated MPLC-MVM was experimentally verified for reconfigurable all-optical on-chip MIMO three-mode demultiplexing 60 . Figure 5 b shows the details of the three-channel MIMO demultiplexing chip. Furthermore, Saygin et al. built a more universal architecture for integrated MPLC-MVM in 2020 61 . In addition, a ten-port unitary optical processor has been experimentally demonstrated 62 . Figure 5c presents the device operating principle, where the fixed unitary diffractive matrices are implemented using multiport directional couplers. This processer offers a new flexible and robust architecture for large-scale MVMs.

a Schematic diagram 59 . b Three-channel MIMO demultiplexing chip 60 . c Ten-port unitary optical processor 62 . a Reprinted from ref. 59 with the permission of IEEE Publishing. b Reprinted with permission from ref. 60 © The Optical Society. c Reproduced from ref. 62 with permission of ACS publications

### MZI-MVMs

The MZI-MVM, as an integrated photonic matrix computation method, is quite suitable in on-chip optical signal processing 32 , 70 . Based on the orthogonal matrix transformation, it is competent to manipulate the spatial orthogonal modes. Figure 6a shows a reconfigurable add-drop multiplexer for spatial modes sampled by the grating array 66 . It could extract a specified spatial mode from a light beam, leaving the other modes undisturbed. It also allows a new signal to be reloaded on that mode. Similarly, as Fig. 6b shows, an MZI mesh based on the orthogonal matrix transformation was used as a 4 × 4-port universal linear circuit, enabling self-adaptation to implement the desired functions 68 . The same structure shown in Fig. 6c could further automatically undo strong mixing between modes as a mode descrambler 69 . The theoretical analysis for the initialization procedure, training and optical multiple-input multiple-output equalizers was discussed in detail in refs. 72 , 101 , 102 . More generally, the MZI-based orthogonal matrix mesh was theoretically proved to have the ability to analyze and generate multiple modes using self-configuring methods 76 . The concept and architecture are presented in Fig. 6d , where an example of a square grating coupler array is illuminated by the input light. While these self-configuring methods require many built-in optical power monitors, they bring additional loss and rapidly increase the number of monitors with the extension of the network, making both the electronic layout and iterative algorithm quite complex. In 2020, Zhou et al. proposed and experimentally demonstrated a common self-configuring method without any information from the inner structure 103 , 104 . Figure 6e shows an example of the iteration process, where a switching matrix was self-configured from a random state. The training was finished using the numerical gradient algorithm inspired by deep learning 3 , which is practicable for a general “black box” system. A similar idea was applied for an all-in-one photonic polarization processor chip 105 , 106 . Other MZI meshes were also reported for multipurpose silicon photonics signal processors, such as a hexagon mesh 107 and a square mesh 108 .

a Add-drop mode multiplexer 66 . b 4 × 4-port universal linear circuit 68 . c Optical mode descrambler 69 . d Analysis and generation of multimode optical fields 76 . e Self-configured example 103 . a Reprinted with permission from ref. 66 © The Optical Society. b Reprinted with permission from ref. 68 © The Optical Society. c Reproduced from ref. 69 with permission of Springer Nature: Light: Science & Applications. (d) Reprinted with permission from ref. 76 © The Optical Society. e Reprinted from ref. 103 with the permission of ACS Publishing

### WDM-MVMs

The WDM-MVM can be directly executed without any algorithms, benefiting from the one-to-one mapping relation between wavelengths and matrix elements. This correlation makes the WDM methods practicable for wave shaping combined with frequency–time mapping 109 , 110 . As shown in Fig. 7a, b , a 1 × 8 MRR array was fabricated for on-chip programmable pulse shaping. The spectral shape and width could be tuned by changing the resonant wavelengths of the MRRs. The square-shape transfer function is demonstrated and presented in Fig. 7b . Other shapes, such as an isosceles triangle and a sawtooth triangle, were also verified. Furthermore, the MRR array can be used for MVM, provided that a sum operation on multiple wavelengths is performed, called “microring weight banks”, as shown in Fig. 7c 83 . A balanced photodetector (PD) yielded the sum and difference of weighted signals. The reconfigurability and scalability of the channel count of the MRR weight banks were experimentally demonstrated in ref. 111 with a comprehensive theoretical analysis 112 . Different methods of controlling large-scale MRRs for matrix computation were proposed and demonstrated in refs. 85 , 113 , 114 . Afterwards, the microring weight bank was applied for various signal processing methods, such as fiber nonlinearity compensation 115 and photonic PCA 87 . PCA aims to extract the principal components (PCs) solely based on the statistical information of the weighted addition output. Figure 7d presents an experimental example of the obtained two-channel waveforms of both the 1 st and 2 nd PCs, evidencing the effectiveness of photonic PCA. The weight bank was further used for photonic ICA to identify the underlying sources that form the basis of the observed data 86 . As shown in Fig. 7e , photonic ICA retrieved the corresponding independent components (ICs) from the received mixture waveforms. By combining the photonic PCA and ICA together, a two-step procedure for a complete photonic BSS pipeline was achieved 88 . The BSS is a powerful technique for achieving signal decomposition with minimal knowledge on either the source characteristics or the mixing process. Figure 7f gives an example of ICs retrieved from mixed radio-frequency waveforms with the BSS technique 88 .

a , b On-chip programmable pulse processor employing a cascaded MZI-MRR structure 110 . c Microring weight banks 83 . d Photonic principal component analysis 87 . e Photonic independent component analysis 86 . f Radio-frequency blind source separation 88 . a , b Reproduced from ref. 110 with permission of Springer Nature: Nature Photonics. c Reprinted with permission from ref. 83 © The Optical Society. d Reprinted with permission from ref. 87 © The Optical Society. e Reprinted with permission from ref. 86 © The Optical Society. f Reprinted with permission from ref. 88 © The Optical Society.

In comparison, coherent MVMs are usually applied in multimode signal processing. The MPLC method can manage massive modes benefiting from the ability of large-scale matrix computation. The main limits are that it is bulky and difficult to refresh with a fast response. The MZI method is easy to integrate, and the functions of the MZI mesh can be autoconfigured since the phase shifters can work faster. However, the scale of matrix computation is limited, and this method can work only for a few modes. Compared with the MZI method, the WDM-MVM method has a more compact footprint, and it is much easier to configure the transmission matrix and apply WDM-MVM for programmable pulse shaping, photonic PCA, ICA and BSS.

## MVMs for optical neural networks

AI technology has been widely used in various electronics industries, such as for deep-learning-based speech recognition and image processing. MVM, as the basic building block of ANNs, occupies most of the computing tasks, such as over 80% for GoogleNet and OverFeat models 116 . Improving the MVM performance is one of the most effective means for ANN acceleration. Compared with electrical computing, optical computing is poor at data storage and flow control, and the low efficiency of optical nonlinearities limits the applications in nonlinear computation 117 , such as activation functions. While it has significant advantages on massively parallel computing through multiplexing strategies of wavelength, mode and polarization 17 , 90 , extremely high data modulation speeds up to 100 GHz 118 , 119 . Hence, photonic networks are quite good at MVM. The combination of optical computing and AI is expected to realize intelligent photonic processors and photonic accelerators 120 . In recent years, AI technology has also seen rapid developments in the field of optics.

### MPLC-MVMs