TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings — part 6/6

Cleaned-text code points [62098, 76246). Bibliography omitted here; source.md includes it.

TPU
v4:
An
Optically
Reconfigurable
Supercomputer
for
ML
with
Hardware
Support
for
Embeddings
Industrial
Product
ISCA
‘23,
June
17–21,
2023,
Orlando,
FL,
USA
[3]
Barroso,
L.A.,
Hölzle,
U.
and
Ranganathan,
P.,
2018.
The
datacenter
as
a
computer:
Designing
warehouse-scale
machines
,
Third
Edition.
Synthesis
Lectures
on
Computer
Architecture,
13(3),
pp.i-189.
[4]
Bloomberg
October
26,
2015.
Google
turning
its
lucrative
web
search
over
to
AI
machines
.
[5]
Bouknight,
W.J.,
Denenberg,
S.A.,
McIntyre,
D.E.,
Randall,
J.M.,
Sameh,
A.H.
and
Slotnick,
D.L.,
1972.
The
ILLIAC
IV
system
.
Proceedings
of
the
IEEE
,
60(4),
pp.
369-388.
[6]
Brown,
T.,
Mann,
B.,
Ryder,
N.,
Subbiah,
M.,
Kaplan,
J.D.,
Dhariwal,
P.,
Neelakantan,
A.,
Shyam,
P.,
Sastry,
G.,
Askell,
A.
and
Agarwal,
S.,
2020.
Language
models
are
few-shot
learners
.
Advances
in
Neural
Information
Processing
Systems
33
(NeurIPS
2020).
[7]
Camara,
J.M.,
Moreto,
M.,
Vallejo,
E.,
Beivide,
R.,
Miguel-Alonso,
J.,
Martínez,
C.
and
Navaridas,
J.,
2010.
Twisted
torus
topologies
for
enhanced
interconnection
networks
.
IEEE
Transactions
on
Parallel
and
Distributed
Systems
,
21(12),
pp.
1765-1778.
[8]
Camarero,
C.,
Martinez,
C.
and
Beivide,
R.,
2014.
Lattice
graphs
for
high-scale
interconnection
topologies
.
IEEE
Transactions
on
Parallel
and
Distributed
Systems
,
26(9),
pp.
2506-2519.
[9] 
Chowdhery,
A.,
Narang,
S.,
Devlin,
J.,
Bosma,
M.,
Mishra,
G.,
Roberts,
A.,
Barham,
P.,
Chung,
H.W.,
Sutton,
C.,
Gehrmann,
S.
and
Schuh,
P.,
2022.
PaLM:
Scaling
language
modeling
with
pathway
s.
arXiv
preprint
arXiv:2204.02311.
[10] 
Colamco,
2022.
Mellanox
QM8790
-
Quantum
HDR
Switch,
https://www.colamco.com/product/mellanox-infiniband-switch-mq
m8790-hs2f-1503410?utm_source=froogle&utm_medium=referral.
[11]
Covington,
P.,
Adams,
J.
and
Sargin,
E.,
2016.
Deep
neural
networks
for
Youtube
recommendations
.
In
Proceedings
of
the
10th
ACM
conference
on
recommender
systems,
pp.
191-198.
[12] 
Dally,
W.J.
and
Towles,
B.P.,
2004.
Principles
and
practices
of
interconnection
networks
.
Elsevier.
[13]
Deepmind,
Nov
18,
2019,
Advanced
machine
learning
helps
Play
Store
users
discover
personalised
apps
.
[14]
Devlin,
J.,
Chang,
M.W.,
Lee,
K.
and
Toutanova,
K.,
2018.
BERT:
Pre-training
of
deep
bidirectional
transformers
for
language
understanding
.
arXiv
preprint
arXiv:1810.04805.
[15]
Dungworth,
M.,
Harrell,
J.,
Levine,
M.,
Nelson,
S.,
Oberlin,
S.
and
Reinhardt,
S.P.,
2011.
CRAY
T3E
.
[16] 
Fan,
X.,
Weber,
W.D.
and
Barroso,
L.A.,
2007.
Power
provisioning
for
a
warehouse-sized
computer.
In
2007
ACM/IEEE
34th
International
Symposium
on
Computer
Architecture
(ISCA),
pp.13-23.
[17]
Farrington,
N.
.,
G.
Porter,
S.
Radhakrishnan,
H.
H.
Bazzaz,
V.
Subramanya,
Y.
Fainman,
G.
Papen,
and
A.
Vahdat.
Helios:
A
hybrid
electrical/optical
switch
architecture
for
modular
data
centers
.
In
Proceedings
of
ACM
SIGCOMM,
Aug.
2010.
[18] 
Ghaemmaghami,
B.,
Ozdal,
M.,
Komuravelli,
R.,
Korchev,
D.,
Mudigere,
D.,
Nair,
K.
and
Naumov,
M.,
2022.
Learning
to
Collide:
Recommendation
System
Model
Compression
with
Learned
Hash
Functions
.
arXiv
preprint
arXiv:2203.15837.
[19] 
Google,
Tracking
our
carbon
free
energy
progress,
https://sustainability.google/progress/energy/.
[20] 
Google,
Google
Data
Center
Efficiency
,
https://www.google.com/about/datacenters/efficiency/.
[21] 
Graphcore,
2022.
IPU-POD64
Reference
Design
Datasheet,
docs.graphcore.ai/projects/
ipu-pod64-datasheet/en/latest/
[22] 
Horowitz,
M.,
2014.
Computing's
energy
problem
(and
what
we
can
do
about
it
).
In
2014
IEEE
International
Solid-State
Circuits
Conference
Digest
of
Technical
Papers
(ISSCC),
pp.
10-14.
[23]
Insight,
2022.
Mellanox
Quantum
QM8790
-
switch
-
40
ports,
https://www.insight.com/en_US/shop/
product/MQM8790-HS2F/Mellanox/MQM8790-HS2F/MellanoxQua
ntumQM8790-swit/
.
[24] 
International
Energy
Agency,
Global
Energy
&
CO2
Status
Report
2019,
Report
Extract
Emissions
.
[25]
Jouppi,
N.P.,
Young,
C.,
Patil,
N.,
Patterson,
D.,
Agrawal,
G.,
Bajwa,
R.,
Bates,
S.,
Bhatia,
S.,
Boden,
N.,
Borchers,
A.,
Boyle,
R.,
Cantin,
P.,
Chao,
C.,
Clark,
C.,
Coriell,
J.,
Daley,
M.,
Dau,
M.,
Dean,
J.,
Gelb,
B.,
Ghaemmaghami,
T.,
Gottipati,
R.,
Gulland,
W.,
Hagmann,
R.,
Ho,
C.R.,
Hogberg,
D.,
Hu,
J.,
Hundt,
R.,
Hurt,
D.,
Ibarz,
J.,
Jaffey,
A.,
Jaworski,
A.,
Kaplan,
A.,
Khaitan,
H.,
Killebrew,
D.,
Koch,
A.,
Kumar,
N.,
Lacy,
S.,
Laudon,
J.,
Law,
J.,
Le,
D.,
Leary,
C.,
Liu,
Z.,
Lucke,
K.,
Lundin,
A.,
MacKean,
G.,
Maggiore,
A.,
Mahony,
M.,
Miller,
K.,
Nagarajan,
R.,
Narayanaswami,
R.,
Ni,
R.,
Nix,
K.,
Norrie,
T.,
Omernick,
M.,
Penukonda,
N.,
Phelps,
A.,
Ross,
J.,
Ross,
M.,
Salek,
A.,
Samadiani,
E.,
Severn,
C.,
Sizikov,
G.,
Snelham,
M.,
Souter,
J.,
Steinberg,
,
D.,
Swing,
A.,
Tan,
M.,
Thorson,
G.,
Tian,
B.,
Toma,
H.,
Tuttle,
E.,
Vijay
Vasudevan,
Walter,
R.,
Wang,
W.,
Wilcox,
E.,
and
Yoon,
D.H.
2017.
In-datacenter
performance
analysis
of
a
tensor
processing
unit.
In
ACM/IEEE
44th
Annual
International
Symposium
on
Computer
Architecture
(ISCA),
pp.
1-12.
[26]
Jouppi,
N.P.,
Yoon,
D.H.,
Kurian,
G.,
Li,
S.,
Patil,
N.,
Laudon,
J.,
Young,
C.
and
Patterson,
D.,
2020.
A
domain-specific
supercomputer
for
training
deep
neural
networks
.
Communications
of
the
ACM
,
63(7),
pp.
67-78.
[27]
Jouppi,
N.P.,
Yoon,
D.H.,
Ashcraft,
M.,
Gottscho,
M.,
Jablin,
T.B.,
Kurian,
G.,
Laudon,
J.,
Li,
S.,
Ma,
P.,
Ma,
X.,
Norrie,
T.,
Patil,
N.,
Prasad,
S.,
Young,
C.,
Zhou,
Z.,
and
Patterson,
D.
2021.
Ten
lessons
from
three
generations
shaped
Google’s
TPUv4i.
In
2021
ACM/IEEE
48th
Annual
International
Symposium
on
Computer
Architecture
(ISCA),
pp.
1-14.
[28]
Kamil,
S.,
Pinar,
A.,
Gunter,
D.,
Lijewski,
M.,
Oliker,
L.
and
Shalf,
J.,
2007,
May.
Reconfigurable
hybrid
interconnection
for
static
and
dynamic
scientific
applications
.
In
Proceedings
of
the
4th
Int'l
conference
on
computing
frontiers,
pp.
183-194.
[29]
Kamil,
S.,
Oliker,
L.,
Pinar,
A.
and
Shalf,
J.,
2009.
Communication
requirements
and
interconnect
optimization
for
high-end
scientific
applications
.
IEEE
Transactions
on
Parallel
and
Distributed
Systems
,
21(2),
pp.
188-202.
[30] 
Karandikar,
S.,
Mao,
H.,
Kim,
D.,
Biancolin,
D.,
Amid,
A.,
Lee,
D.,
Pemberton,
N.,
Amaro,
E.,
Schmidt,
C.,
Chopra,
A.
and
Huang,
Q.,
2018,
June.
FireSim:
FPGA-accelerated
cycle-exact
scale-out
system
simulation
in
the
public
cloud
.
In
2018
ACM/IEEE
45th
Annual
International
Symposium
on
Computer
Architecture
(ISCA),
pp.
29-42.
[31]
Khani,
M.,
Ghobadi,
M.,
Alizadeh,
M.,
Zhu,
Z.,
Glick,
M.,
Bergman,
K.,
Vahdat,
A.,
Klenk,
B.
and
Ebrahimi,
E.,
2021,
August.
SiP-ML:
high-bandwidth
optical
network
interconnects
for
machine
learning
training
.
In
Proceedings
of
the
2021
ACM
SIGCOMM
2021
Conference
(pp.
657-675).
[32]
Li,
S.,
Tan,
M.,
Pang,
R.,
Li,
A.,
Cheng,
L.,
Le,
Q.V.
and
Jouppi,
N.P.,
2021.
Searching
for
Fast
Model
Families
on
Datacenter
Accelerators
.
In
Proceedings
of
the
IEEE/CVF
Conference
on
Computer
Vision
and
Pattern
Recognition,
pp.
8085-8095.
[33]
Li,
S.,
Andersen,
G.,
Chen,
T.,
Cheng,
L.,
Grady,
J.,
Huang,
D.,
Le,
Q.,
Li,
A.,
Li,
X.,
Li,
Y.,
Liang,
C.,
Lu,
Y.,
Ni,
Y.,
Pang,
F.,
Ranganathan
,
P.,
Tan,
M.,
Wicke,
M.,
Wu,
G.,
Zhu,
S.,
and
Jouppi,
N.,
2023.
Hyperscale
Hardware
Optimized
Neural
Architecture
Search
,
In
Proceedings
of
the
28th
International
conference
on
Architectural
Support
for
Programming
Languages
and
Operating
Systems
(ASPLOS).
[34] 
Masanet,
E.,
Shehabi,
A.,
Lei,
N.,
Smith,
S.
and
Koomey,
J.,
2020.
Recalibrating
global
data
center
energy-use
estimates
.
Science
,
367(6481),
pp.984-986.
13

ISCA
‘23,
June
17–21,
2023,
Orlando,
FL,
USA
Jouppi,
et
al.
[35] 
Minkenberg,
C.,
Rodriguez,
G.,
Prisacari,
B.,
Schares,
L.,
Heidelberger,
P.,
Chen,
D.
and
Stunkel,
C.,
2016,
March.
Performance
benefits
of
optical
circuit
switches
for
large-scale
dragonfly
network
s.
In
2016
Optical
Fiber
Communications
Conference
and
Exhibition
(OFC)
(pp.
1-3).
IEEE.
[36]
MLCommons,
V
2.0
Results,
June
29,
2022,
https://mlcommons.org/en/training-normal-20
/.
[37] 
Mudigere,
D.,
Hao,
Y.,
Huang,
J.,
Jia,
Z.,
Tulloch,
A.,
Sridharan,
S.,
Liu,
X.,
Ozdal,
M.,
Nie,
J.,
Park,
J.
and
Luo,
L.,
2022,
June.
Software-hardware
co-design
for
fast
and
scalable
training
of
deep
learning
recommendation
models
.
In
Proceedings
of
the
49th
Annual
International
Symposium
on
Computer
Architecture
(pp.
993-1011).
[38]
Nayak,
P.,
2021.
MUM:
A
new
AI
milestone
for
understanding
information
.
Google,
May,
18.
[39]
Norrie,
T.,
Patil,
N.,
Yoon,
D.H.,
Kurian,
G.,
Li,
S.,
Laudon,
J.,
Young,
C.,
Jouppi,
N.
and
Patterson,
D.,
2021.
The
design
process
for
Google's
training
chips:
TPUv2
and
TPUv3
.
IEEE
Micro
,
41(2),
pp.
56-63.
[40] 
Nvidia,
2020.
Nvidia
A100
Tensor
Core
GPU
Architecture
.
[41] 
Nvidia,
2021.
Nvidia
DGX
SuperPOD:
Scalable
Infrastructure
for
AI
Leadership
Reference
Architecture
.
[42]
Patterson,
D.,
Gonzalez,
J.,
Le,
Q.,
Liang,
C.,
Munguia,
L.M.,
Rothchild,
D.,
So,
D.,
Texier,
M.
and
Dean,
J.,
2022.
The
carbon
footprint
of
machine
learning
training
will
plateau
then
shrink
,
IEEE
Computer
,
55(7),
pp.
18-28.
[43]
Poutievski,
L.,
Mashayekhi,
O.,
Ong,
J.,
Singh,
A.,
Tariq,
M.,
Wang,
R.,
Zhang,
J.,
Beauregard,
V.,
Conner,
P.,
Gribble,
S.
and
Kapoor,
R.,
2022.
Jupiter
evolving:
transforming
Google's
datacenter
network
via
optical
circuit
switches
and
software-defined
networking
.
In
Proceedings
of
the
ACM
SIGCOMM
2022
Conference,
pp.
66-85.
[44] 
Raghavendra,
R.,
Ranganathan,
P.,
Talwar,
V.,
Wang,
Z.
and
Zhu,
X.,
2008,
March.
No
"power"
struggles:
coordinated
multi-level
power
management
for
the
data
center
.
In
Proceedings
of
the
13th
International
Conference
on
architectural
support
for
programming
languages
and
operating
systems
(ASPLOS),
pp.
48-59.
[45] 
Schwartz,
R.,
Dodge,
J.,
Smith,
N.A.
and
Etzioni,
O.,
2020.
Green
AI
.
Communications
of
the
ACM
,
63(12),
pp.
54-63.
[46]
Scott,
S.L.,
1996.
Synchronization
and
communication
in
the
T3E
multiprocessor
.
In
Proceedings
of
the
7th
International
conference
on
Architectural
Support
for
Programming
Languages
and
Operating
Systems
(ASPLOS),
pp.
26-36.
[47]
Sequin,
C.H.,
1981.
Doubly
twisted
torus
networks
for
VLSI
processor
arrays
.
In
Proceedings
of
the
8th
International
annual
Symposium
on
Computer
Architecture
(ISCA),
pp.
471-480.
[48] 
Sethi,
G.,
Bhattacharya,
P.,
Choudhary,
D.,
Wu,
C.-J.,
and
Kozyrakis,
C.,
2022.
FlexShard:
Flexible
Sharding
for
Industry-Scale
Sequence
Recommendation
Models
,
arxiv
preprint
arXiv:2301.02959.
[49] 
Shalf,
J.,
Kamil,
S.,
Oliker,
L.
and
Skinner,
D.,
2005,
November.
Analyzing
ultra-scale
application
communication
requirements
for
a
reconfigurable
hybrid
interconnect.
In
SC'05:
Proceedings
of
the
2005
ACM/IEEE
Conference
on
Supercomputing
(pp.
17-17).
[50]
Shaw,
D.E.,
Deneroff,
M.M.,
Dror,
R.O.,
Kuskin,
J.S.,
Larson,
R.H.,
Salmon,
J.K.,
Young,
C.,
Batson,
B.,
Bowers,
K.J.,
Chao,
J.C.
and
Eastwood,
M.P.,
2008.
Anton,
a
special-purpose
machine
for
molecular
dynamics
simulation
.
Communications
of
the
ACM
,
51(7),
pp.
91-99.
[51]
Singla,
A.,
Singh,
K.
Ramachandran,
L.
Xu,
and
Y.
Zhang.
Proteus:
A
topology
malleable
data
center
network
.
In
Proceedings
of
ACM
Hotnets,
Oct.
2010.
[52]
Taylor,
P.,
2022.
Data
center
average
annual
power
usage
effectiveness
(PUE)
worldwide
2007-2022
,
Nov
22.
[53] 
Teh,
M.
Y.,
S.
Zhao,
P.
Cao,
and
K.
Bergman.
Couder:
Robust
topology
engineering
for
optical
circuit
switched
data
center
networks
.
arXiv
preprint
arXiv:2010.00090,
Sept.
2020.
[54] 
Thoppilan,
R.,
De
Freitas,
D.,
Hall,
J.,
Shazeer,
N.,
Kulshreshtha,
A.,
Cheng,
H.T.,
Jin,
A.,
Bos,
T.,
Baker,
L.,
Du,
Y.
and
Li,
Y.,
2022.
LaMDA:
Language
models
for
dialog
applications
.
arXiv
preprint
arXiv:2201.08239.
[55] 
Urata,
R.,
Liu,
H.,
Yasumura,
K.,
Mao,
E.,
Berger,
J.,
Zhou,
X.,
Lam,
C.,
Bannon,
R.,
Hutchinson,
D.,
Nelson,
D.
and
Poutievski,
L.,
2022.
Mission
Apollo:
Landing
Optical
Circuit
Switching
at
Datacenter
Scale
.
arXiv
preprint
arXiv:2208.10041.
[56] 
US
Energy
Information
Agency,
2022.
Oklahoma
State
Profile
and
Energy
Estimates,
https://www.eia.gov/state/
?sid=OK#:~:text=In%202021%2C%20wind%20supplied%2041,ele
ctricity%20net%20generation%20from%20wind.
[57] 
Vahdat,
A.,
H.
Liu,
X.
Zhao,
and
C.
Johnson,
2011.
The
emerging
optical
data
center
.
In
Proceedings
of
the
Optical
Fiber
Communications
Conference.
[58] 
Vaswani,
A.,
Shazeer,
N.,
Parmar,
N.,
Uszkoreit,
J.,
Jones,
L.,
Gomez,
A.N.,
Kaiser,
Ł.
and
Polosukhin,
I.,
2017.
Attention
is
all
you
need
.
Advances
in
neural
information
processing
systems,
30
(NeurIPS
2017).
[59] 
Wang,
S.,
Wei,
J.,
Sabne,
A.,
David,
A.,
Llbeyi,
B.,
Hechtman,
B.,
Chen,
D.,
Murthy,
K.
S.,
Maggioni,
M.,
Zhang,
Q.,
Kumar,
S.,
Guo,
T.,
Xu,
Y.,
and
Zhou,
Z.,
2023.
Overlap
Communication
with
Dependent
Computation
via
Decomposition
in
Large
Deep
Learning
Models,
In
Proceedings
of
the
28th
International
conference
on
Architectural
Support
for
Programming
Languages
and
Operating
Systems
(ASPLOS).
[
60
] 
Wang,
W.,
Khazraee,
M.,
Zhong,
Z.,
Ghobadi,
M.,
Jia,
Z.,
Mudigere,
D.,
Zhang,
Y.
and
Kewitsch,
A.,
2022.
Topoopt:
Co-optimizing
network
topology
and
parallelization
strategy
for
distributed
training
jobs
.
arXiv
preprint
arXiv:2202.00433.
[61] 
Williams,
S.,
Waterman,
A.
and
Patterson,
D.,
2009.
Roofline:
an
insightful
visual
performance
model
for
multicore
architectures
.
Communications
of
the
ACM
,
52(4),
pp.65-76.
[62] 
Wu,
Q.,
Deng,
Q.,
Ganesh,
L.,
Hsu,
C.H.,
Jin,
Y.,
Kumar,
S.,
Li,
B.,
Meza,
J.
and
Song,
Y.J.,
2016.
Dynamo:
Facebook’s
Data
Center-Wide
Power
Management
System
.
2016
ACM/
IEEE
43rd
Annual
International
Symposium
on
Computer
Architecture
(ISCA),
Seoul,
(2016),
pp.
469–480.
[63] 
Xu,
Y.,
Lee,
H.,
Chen,
D.,
Hechtman,
B.,
Huang,
Y.,
Joshi,
R.,
Krikun,
M.,
Lepikhin,
D.,
Ly,
A.,
Maggioni,
M.
and
Pang,
R.,
2021.
GSPMD:
general
and
scalable
parallelization
for
ML
computation
graphs
.
arXiv
preprint
arXiv:2105.04663.
[64] 
Zhao,
M.,
Agarwal,
N.,
Basant,
A.,
Gedik,
B.,
Pan,
S.,
Ozdal,
M.,
Komuravelli,
R.,
Pan,
J.,
Bao,
T.,
Lu,
H.
and
Narayanan,
S.,
2022,
June.
Understanding
data
storage
and
ingestion
for
large-scale
deep
recommendation
model
training:
Industrial
product
.
In
Proceedings
of
the
49th
Annual
International
Symposium
on
Computer
Architecture
(ISCA),
pp.
1042-1057.
14