Photonic matrix multiplication lights up photonic accelerator and beyond — part 1/4

Cleaned-text code points [0, 15819). Bibliography omitted here; source.md includes it.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

View all journals

Saved research

Content Explore content

About the journal

Publish with us

RSS feed

Review Article

Open access

Published: 03 February 2022

# Photonic matrix multiplication lights up photonic accelerator and beyond

Hailong Zhou 1 ,

Jianji Dong ORCID: orcid.org/0000-0002-1852-8650 1 ,

Junwei Cheng 1 ,

Wenchan Dong 1 ,

Chaoran Huang 2 ,

Yichen Shen ORCID: orcid.org/0000-0003-3800-2143 3 ,

Qiming Zhang 4 , 5 ,

Min Gu ORCID: orcid.org/0000-0003-4078-253X 4 , 5 ,

Chao Qian 6 ,

Hongsheng Chen ORCID: orcid.org/0000-0002-5735-9781 6 ,

Zhichao Ruan ORCID: orcid.org/0000-0001-8311-6970 7 &

…

Xinliang Zhang 1

Light: Science & Applications volume 11 , Article number: 30 ( 2022 ) Cite this article

73k Accesses

604 Citations

28 Altmetric

Metrics details

## Abstract

Matrix computation, as a fundamental building block of information processing in science and technology, contributes most of the computational overheads in modern signal processing and artificial intelligence algorithms. Photonic accelerators are designed to accelerate specific categories of computing in the optical domain, especially matrix multiplication, to address the growing demand for computing resources and capacity. Photonic matrix multiplication has much potential to expand the domain of telecommunication, and artificial intelligence benefiting from its superior performance. Recent research in photonic matrix multiplication has flourished and may provide opportunities to develop applications that are unachievable at present by conventional electronic processors. In this review, we first introduce the methods of photonic matrix multiplication, mainly including the plane light conversion method, Mach–Zehnder interferometer method and wavelength division multiplexing method. We also summarize the developmental milestones of photonic matrix multiplication and the related applications. Then, we review their detailed advances in applications to optical signal processing and artificial neural networks in recent years. Finally, we comment on the challenges and perspectives of photonic matrix multiplication and photonic acceleration.

### Similar content being viewed by others

### Direct tensor processing with coherent light

### The potential of multidimensional photonic computing

### An integrated large-scale photonic accelerator with ultralow latency

### Explore related subjects

Integrated optics

Optoelectronic devices and components

Photonic devices

## Introduction

Over the past few years, there has been an ever-growing demand for artificial intelligence and fifth-generation communications globally, resulting in very large computing power and memory requirements. The slowing down or even failure of Moore’s law makes it increasingly difficult to improve their performance and energy efficiency by relying on advanced semiconductor technology 1 , 2 . Moreover, the clock frequency of traditional electrical processing methods is generally limited to several GHz 3 , which can no longer meet the demands of super-high-speed and low-latency mass data processing. Matrix computation is one of the most widely used and indispensable tools of information processing in science and engineering 4 , 5 . Most signal processing, such as the discrete Fourier transform and convolution operation, can be attributed to matrix computations. On the other hand, since the concept of artificial intelligence (AI) was put forward in 1956 for the first time 6 , artificial neural networks (ANNs) have been rapidly developed and widely used in various fields 7 . Due to the continuous substantial increase in information capacity, general electronic processors seem to be incapable of executing high-complexity AI tasks in the foreseeable future 1 . To solve this challenge, chips oriented to AI applications have emerged, such as neural network processing units (NPUs) 8 . At present, AI chips have been widely used in almost every type of big data processing in areas such as search, news, e-commerce, cloud computing, and inverse design of functional devices 9 , 10 , 11 , 12 , 13 . Typically, neural network algorithms represented by deep learning, such as forward neural networks (FNNs), convolutional neural networks (CNNs) and spiking neural networks (SNNs), are characterized by many training parameters, especially in heavy matrix computations 14 .

Traditionally, matrix computation is completed by an electrical digital signal processor, and its speed and power consumption are greatly limited by the nature of the electronic devices themselves. Therefore, traditional electrical methods are hard to simultaneously achieve high-capacity and low-latency matrix information processing limited by the Moore’s law 1 , 2 . However, for some applications, such as ultrafast neural networks 15 , large bandwidth and low latency are simultaneously required; thus, a new medium for matrix computations and interconnects is urgently needed for the implementation of high-performance and energy-efficient matrix computations. Optical devices can have a superlarge bandwidth and low power consumption 16 . And light has an ultrahigh frequency up to 100 THz and multiple degrees of freedom in their quantum state 17 , 18 , making optical computing one of the most competitive candidates for high-capacity and low-latency matrix information processing in the “More than Moore” era 1 . For example, a Fourier transform was performed at the speed of light with a lens 19 . Motivated by its very high prospect, photonic matrix multiplication has been developed rapidly in recent years and has been widely applied in photonic acceleration for optical signal processing 20 , 21 , 22 , AI and optical neural networks (ONNs) 15 , 23 , 24 . A lot of review works on photonic acceleration have been made, these works mainly focused on integrated photonic neuromorphic systems 1 , 15 , 23 , 24 , 25 , 26 , 27 , 28 , nanophotonics and machine learning blend 29 , 30 , reservoir computing 31 , programmable nanophotonics 21 , 22 , 32 . As a fundamental and important part of photonic acceleration, photonic matrix multiplication computation for photonic acceleration has not been systematically reviewed. Here, we review the advances of photonic acceleration from the perspective of photonic matrix multiplication. We first discuss the methods and developmental milestones of photonic matrix multiplications and then review the progress in cutting-edge fields of optical signal processing and optical neural networks. Finally, a perspective for photonic matrix multiplications is discussed.

## Matrix-vector multiplication

The methods for photonic matrix-vector multiplications (MVMs) mainly fall into three categories: the plane light conversion (PLC) method, Mach–Zehnder interferometer (MZI) method and wavelength division multiplexing (WDM) method. The detailed mechanism of these MVMs can be found in ref. 33 , which offers an easy-to-read overview of principle and development of photonic matrix computation. The first kind of optical MVM (PLC-MVM) is implemented by the diffraction of light in free space. Figure 1a shows a typical MVM configuration 34 , 35 . First, the incident vector of X distributed along the x direction can be expanded and replicated along the y direction through a cylindrical lens or other optical elements. Then, the spatial diffraction plane is used to adjust each element independently, and its transmission matrix is W . Finally, the x -direction beams are combined and summed in a similar way, and the final output vector of Y along the y direction is the product of the matrix of W and the vector X , that is, Y =WX . The second MVM mainly consists of an MZI network (i.e., MZI-MVM). Figure 1b shows the configuration diagram, which is based mainly on rotation submatrix decomposition and singular value decomposition 36 . The calibration of the transmission matrix is more difficult since every matrix element is affected by multiple dependent parameters. The third MVM (i.e., WDM-MVM) is an incoherent matrix computation method based on the WDM technology. Figure 1c shows a typical diagram based on microring resonators (MRRs). The input vector of X is loaded on beams with different wavelengths, which pass through the microrings with one-one adjustment of the transmission coefficients of W . Then, the total output power vector is given by Y=WX .

a PLC-MVM. b MZI-MVM. c WDM-MVM

Photonic matrix multiplication has come a long way and developed rapidly in recent years. Figure 2 summarizes the development history and milestones of photonic matrix computation. In the preliminary stage, only some fixed matrix computations were implemented using optical methods such as the Fourier transform 19 . Thereafter, the initially programmable MVM was demonstrated with spatial optical elements based on single PLC (SPLC) 34 . For example, a fully parallel, high-speed incoherent optical method was employed to utilize the discrete vector multiplier at a high speed 37 , while the update of the matrix at high frame rates was restrained with current spatial light modulators (SLMs). Matrix multiplications involving optical array modulators, such as electrooptic modulations, direct driven LED arrays, and acousto-optic Bragg cells, were accomplished with faster frame rates 34 , 38 , 39 . A photorefractive crystal 40 , 41 , 42 and nonlinear material 43 could be optionally applied to implement MVMs. In the SPLC-MVM method, only one dimension is used for the input/output vectors, and the scale ( \(\propto N\) ) of vectors is still limited. A more powerful PLC-MVM for unitary spatial mode manipulation was proved with multiplane light conversion (MPLC) 44 , 45 , in which the input/output vectors are distributed in the whole two-dimensional plane, and the scale is proportional to \({N}^{2}\) . Afterwards, the MPLC technique was widely used in various fields, such as for all-optical machine learning 46 , 47 , 48 , the Laguerre-Gaussian or orbital angular momentum (OAM) mode sorter 49 , 50 , the photonic Ising machine 51 , 52 , time-reversed optical waves 53 , optical logic operations 54 , optical encryption and perceptrons 55 , 56 , optical hybrid 57 and neuromorphic optoelectronic computing 58 . Although MPLC can achieve ultralarge-scale MVMs, the devices are bulky, and the reprogramming speed for weight encoding is still limited. A mini-sized and universal MVM is more practical, especially in integrated photonic applications. In 2017, Tang et al. first proposed a novel integrated reconfigurable unitary optical mode converter using multimode interference couplers, which shared a similar principle with MPLC 59 . Then, it was used for all-optical on-chip multi-input-multi-output (MIMO) mode demultiplexing 60 . In 2020, the integrated MPLC technique was further analyzed by Saygin et al. as a novel matrix decomposition method based on multichannel blocks 61 and then was experimentally proven on a silicon photonic chip 62 .

a MPLC-MVM. b MZI-MVM. c WDM-MVM

In 1994, Reck et al. proposed a recursive algorithm that could factorize any \(N\times N\) unitary matrix into a sequence of two-dimensional matrix transformations, which paved the way for future photonic integrated computation based MZIs 36 . Then, Miller et al. suggested that the MZI network could be self-configured to define functions assisted by transparent detectors 63 , 64 , 65 , 66 . The MZI mesh was then applied in an add-drop multiplexer for spatial modes 66 , universal linear optical components 65 , automatic MIMO 64 and universal beam couplers 63 . In 2016, Clements et al. proposed a brand-new universal matrix framework based on an alternative assemblage of MZIs and phase shifters, which is superior to that proposed by Reck et al. Only half the optical depth of the Reck design is required, and the optical loss is significantly reduced 67 . Ribeiro et al. experimentally demonstrated a 4 × 4-port universal optical linear circuit chip with the MZI mesh on integration platforms 68 . Thereafter, the applications of MZI-MVMs were further extended to ONNs 3 , light descramblers 69 , modular linear optical circuits 70 , optical CNNs 71 , equalizers 72 , digital-to-analog conversion (DAC) 73 , Ising machines 74 , 75 , mode analysis 76 and complex ONNs 77 .

Generally, the footprint of the MZI reaches over 10,000 μm 2 per interferometer unit, which remains a bottleneck to further improve the computing density of the MZI mesh. The WDM-MVM based on microring arrays was proposed by Xu et al., who used compact microrings with a diameter of only a few microns 78 , 79 . This approach encodes information on different optical wavelengths rather than spatial modes. Compared to other physical dimensions, the wavelength dimension has the most abundant orthogonal channels in optics, up to hundreds of channels 80 , 81 . Silicon MRR arrays for matrix operations were first conceptualized by Xu and Soref in 2011 78 . They were then demonstrated by Yang et al. using a 4 × 4 silicon microring modulator array but with binary values of 0 and 1 only 79 . In 2014, Tait and his colleagues proposed using MRR arrays as a matrix computation method primitive for photonic neural networks 82 and achieved continuous matrix values from -1 to 1 by continuously tuning the MRRs. The WDM-MVM was further used for photonic weight banks 83 , 84 , 85 , 86 , principal component analysis (PCA) 87 , independent component analysis (ICA) 86 , blind source separation (BSS) 88 , TeraMAC neuromorphic photonic processor 18 , the optical SNN 89 , TeraMAC photonic tensor core 90 , optical CNN 91 , 92 , 93 , and photonic convolutional accelerator for the ONN 16 , 94 .

Table 1 summarizes the performance comparison of different photonic matrix multiplication methods. In general, the PLC-MVM method is coherent and can operate in the whole complex field. Its scale is very large, input vector sizes of 357 for SPLC-MVM 48 and 490000 ( N = 700) for MPLC-MVM 58 were reported, easily up to 10 3 for SPLC-MVM and 10 6 for MPLC-MVM with SLMs 58 . However, the device size is quite large, and hence, the integrated counterpart was pursued 59 , 60 , 61 . The MZI-MVM method is also coherent, but its scale is far smaller than that of the PLC-MVM method ( N = 64 was reported by Lightmatter 95 ). The main advantage is that it can be integrated into a chip. The WDM-MVM method is more compact. The scale is restricted by the number of wavelengths and can be ~10 2 with soliton crystal microcombs 16 , provided all the wavelengths are used for a single MVM. A balanced photodetector summing weighted signals allows for positive and negative weights 82 . WDM-MVM is incoherent and can be used for real-valued matrices. For these methods, the assigned transmission matrices for SPLC-MVM and WDM-MVM can be directly written in, while some algorithms are needed to load the transmission matrices for the MPLC-MVM and MZI-MVM methods. All these MVM methods have been widely applied in various fields. In the following, we review the detailed applications of MVMs in optical signal processing and photonic AI.

## MVMs for optical signal processing

The photonic matrix multiplication network itself can be used as a general linear photonic loop for photonic signal processing 32 . In recent years, MVM has been developed as a powerful tool for a variety of photonic signal processing methods.

### MPLC-MVMs