Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models — part 6/9
Cleaned-text code points [78787, 94540). Bibliography omitted here; source.md includes it.
chinese_remainder_theorem : Jiaming Song, Chenlin Meng, and Stefano Ermon, Stanford University (jiaming.tsong, chenlin, ermon)@cs.stanford.edu cifar10_classification : Peiyuan Liao, Carnegie Mellon University, Praxis Pioneering (peiyuanl@cs.cmu.edu) code_line_description : Rabin Banjade and Priti Oli, University of Memphis (rbnjade1, poli)@memphis.edu. Task cites: ( Alon et al., 2018 ) codenames : Isaac Noble and Lucy Noble, Google (isaacn, lucynoble)@google.com, Emma Lam (emmas.email1720@gmail.com), and Lucas Lam (lucas06lam@gmail.com). Task cites: ( Kim et al., 2019 ) color : Zijie J. Wang, Georgia Tech (jayw@gatech.edu). Task cites: ( Gibson et al., 2017 ; Zucconi, 6 Jan. 2016 ) com2sense : Shikhar Singh, Nuan Wen, and Yu Hou, University of Southern California (ssingh43, nuanwen, houyu)@usc.edu; Pegah Alipoormolabashi, Sharif University of Technology (palipoor976@gmail.com); and Te-Lin Wu and Nanyun Peng, University of California, Los Angeles (UCLA) (telinwu,violetpeng)@cs.ucla.edu. Task cites: ( Singh et al., 2021 ) common_morpheme : Shyamolima (Shammie) Debnath, University of Washington (shyamolima.d@gmail.com). Task cites: ( Devlin et al., 2018 ; Wu et al., 2016 ; Won et al., 2021 ; Xu et al., 2018 ; Edmiston & Stratos, 2018 ; El-Kishky et al., 2019 ) conceptual_combinations : Raphaël Millière, Columbia University (rm3799@columbia.edu); Catherine Stinson, Queen’s University at Kingston (c.stinson@queensu.ca); Dimitri Coelho Mollo, Umeå University (dimitri.mollo@umu.se); and Charles Rathkopf, Juelich Research Center (c.rathkopf@fz-juelich.de). Task cites: ( Fodor, 1975 ; Fodor & Pylyshyn, 1988 ; Smolensky, 1988 ; Lake et al., 2017 ; Lake & Murphy, 2020 ; Marcus, 2020 ; Henrich et al., 2010 ; Murphy, 1988 ) conlang_translation : James Koppel, MIT (jkoppel@mit.edu); Rowan Jacobs, Synthego (rowan.jacobs@synthego.com); and Mimee Xu, NYU (mimee@nyu.edu). Task cites: ( Canfield, 2010 ; Şahin et al., 2020 ; Sennrich & Zhang, 2019 ) context_definition_alignment : Lütfi Kerem Şenel, Hinrich Schütze, and Timo Schick, LMU Munich (lksenel@gmail.com, inquiries@cislmu.org, and schickt93@gmail.com). Task cites: ( Senel & Schütze, 2021 ; Reimers & Gurevych, 2019 ) convinceme : Cedrick Argueta and Jaime Fernández Fisac, Princeton University (cedrick, jfisac)@princeton.edu; and Vinay Ramasesh, Google (ramasesh@google.com). Task cites: ( Lin et al., 2021b ; Levy et al., 2021 ; Clark et al., 2018 ; Maynez et al., 2020 ; Wang et al., 2020a ; Kenton et al., 2021 ; Xu et al., 2020a ; Tamkin et al., 2021 ) coqa_conversational_question_answering : Siva Reddy, Mila and McGill University (siva.reddy@mila.quebec); Danqi Chen, Princeton University (danqic@cs.princeton.edu); and Christopher D. Manning, Stanford University (manning@stanford.edu). Task cites: ( Reddy et al., 2019 ; Radford et al., 2019 ; Brown et al., 2020 ) crash_blossom : Amanda Dsouza, Georgia Tech (dsouza.amanda@gmail.com); Matthew L. Leavitt, MosaicML (matthew.l.leavitt@gmail.com); Parth Doshi, UC San Diego (pdoshi@ucsd.edu); and Animesh Gupta, Thapar University (animeshgupta.thapar@gmail.com) crass_ai : Jörg Frohberg, Apergo UG (j.frohberg@apergo.ai). Task cites: ( Schölkopf et al., 2021 ; Teney et al., 2020 ; Liang et al., 2020b ; Pearl, 2000 ; Shahid & Zheleva, 2021 ; Xia et al., 2021 ; Priol et al., 2020 ) cryobiology_spanish : Marco Baturan (marco.baturan@gmail.com), Roman Sitelew (sitelewr@gmail.com), and Ramón Risco Delgado (ramon@us.es). Task cites: ( Scudellari, 2017 ; Soltani Firouz et al., 2021 ; Toubiana et al., 2020 ; Mbogba et al., 2018 ) cryptonite : Avia Efrat, Uri Shaham, Omer Levy, and Dan Kilman, Tel Aviv University (avia.efrat, uri.shaham1, omerlevy, dankilman)@gmail.com. Task cites: ( Efrat et al., 2021 ; Raganato et al., 2017 ; Sakaguchi et al., 2020 ; Miller & Gurevych, 2015 ; Miller et al., 2017 ; Joshi et al., 2017 ; Oprea & Magdy, 2020 ; Friedlander & Fine, 2018 ; Lewis et al., 2020c ) cs_algorithms : Harsh Mehta, Google Research (harshm@google.com), and Behnam Neyshabur, Google Research, Blueshift (neyshabur@google.com) cycled_letters : Sam Shleifer, Facebook AI Research (sshleifer@gmail.com). Task cites: ( Brown et al., 2020 ) dark_humor_detection : Ernie Chang, Jesujoba Alabi, and Vera Demberg, Saarland University (cychang, s8jealab, vera)@coli.uni-saarland.de. Task cites: ( Weller & Seppi, 2019 ; Fan et al., 2020 ; Willinger et al., 2017 ; Yang et al., 2015 ; Mihalcea & Strapparava, 2005 ) date_understanding : Xinyi Wu and Zijian Wang, Amazon (xinyiwu.nlp@gmail.com, zijwang@stanford.edu). Task cites: ( Vashishth et al., 2018 ; Chambers, 2012 ; Kotsakos et al., 2014 ; Vashishtha et al., 2020 ; Vashishtha et al., 2019 ) disambiguation_qa : Nick Doiron, Tufts University (nick@mapmeld.com). Task cites: ( Zhao et al., 2018 ; Rudinger et al., 2018 ) discourse_marker_prediction : Damien Sileo, KU Leuven (damien.sileo@kuleuven.be). Task cites: ( Malmi et al., 2018 ; Nie et al., 2019 ; Sileo et al., 2019 ; Sileo et al., 2020 ) disfl_qa : Aditya Gupta, Manaal Faruqui, and Shyam Upadhyay, Google (gaditya, mfaruqui, shyamupa)@google.com; Jiacheng Xu, The University of Texas at Austin (jcxu@utexas.edu); and Diyi Yang, Georgia Tech (diyi.yang@cc.gatech.edu). Task cites: ( Gupta et al., 2021 ; Rajpurkar et al., 2018 ) diverse_social_bias : Paul Pu Liang, Louis-Philippe Morency, Ruslan Salakhutdinov, and Chiyu Wu, Carnegie Mellon University (pliang, morency, rsalakhu)@cs.cmu.edu, (chiyuwu23@gmail.com). Task cites: ( Sheng et al., 2019 ; Nadeem et al., 2020 ; Hendrycks et al., 2020 ; Sap et al., 2020 ; Gehman et al., 2020 ; Bolukbasi et al., 2016 ; Caliskan et al., 2017 ; May et al., 2019 ; Liang et al., 2020a ; Barocas & Selbst, 2016 ; Cho et al., 2019 ; Blodgett et al., 2020 ; Merity et al., 2016 ; Socher et al., 2013 ; Poria et al., 2019 ) dyck_languages : Mirac Suzgun, Stanford University (msuzgun@stanford.edu); Sebastian Gehrmann, Google (gehrmann@google.com); Yonatan Belinkov, Technion – Israel Institute of Technology (belinkov@technion.ac.il); and Stuart M. Shieber, Harvard University (shieber@seas.harvard.edu). Task cites: ( Chomsky & Schützenberger, 1959 ; Suzgun et al., 2019b ; Hao et al., 2018 ; Hewitt et al., 2020 ; Hahn, 2020 ; Suzgun et al., 2019a ; Sennhauser & Berwick, 2018 ; Skachkova et al., 2018 ; Bhattamishra et al., 2020a ; Yu et al., 2019c ; Ebrahimi et al., 2020 ; Ackerman & Cybenko, 2020 ; Bhattamishra et al., 2020b ) dynamic_counting : Mirac Suzgun, Stanford University (msuzgun@stanford.edu); Sebastian Gehrmann, Google (gehrmann@google.com); Yonatan Belinkov, Technion – Israel Institute of Technology (belinkov@technion.ac.il); and Stuart M. Shieber, Harvard University (shieber@seas.harvard.edu). Task cites: ( Suzgun et al., 2019a ; Skachkova et al., 2018 ; Bhattamishra et al., 2020a ; Suzgun et al., 2019b ; Yu et al., 2019c ; Ebrahimi et al., 2020 ; Ackerman & Cybenko, 2020 ; Bhattamishra et al., 2020b ; Sennhauser & Berwick, 2018 ; Merrill, 2020 ; Karpathy, 2015 ) elementary_math_qa : Kumar Shridhar, NeuralSpace and ETH Zurich (kumar@neuralspace.ai); Akshat Agarwal, NeuralSpace (akshat@neuralspace.ai); and Asher Mullokandov, Imperial College London (a.mullokandov@imperial.ac.uk). Task cites: ( Amini et al., 2019 ; Ling et al., 2017 ; Hendrycks et al., 2021c ; Patel et al., 2021 ; Zhang et al., 2020a ; Hendrycks et al., 2021b ) emoji_movie : Zijie J. Wang, Georgia Tech (jayw@gatech.edu). Task cites: ( Cruse, 2015 ; Instagram Engineering, 2015 ; Chandra Guntuku et al., 2019 ; Eisner et al., 2016 ; Mayne, 2020 ; Boillot, 2019 ) emojis_emotion_prediction : Abu Awal Md Shoeb, Rutgers University (abu.shoeb@rutgers.edu), and Gerard de Melo, HPI/University of Potsdam (gerard.demelo@hpi.de). Task cites: ( Shoeb & de Melo, 2020 ; Plutchik, 1980 ) empirical_judgments : Jennifer Marsh, UC Berkeley (jmarsh@berkeley.edu), and Samuel S. Schoenholz, Google Research, Brain Team (schsam@google.com). Task cites: ( Kant, 1781/1787 ; Kant, 1783 ; Spirtes et al., 2000 ; Pearl, 1988 ; Goldberger, 1972 ; Rothman & Greenland, 2005 ; Roemmele et al., 2011 ; Wang et al., 2019a ; Evans et al., 2019 ) english_proverbs : Daniel Moseguí González (d.mosegui@gmail.com). Task cites: ( Gyasi Obeng, 1996 ; Honeck, 1997 ; Hrisztova-Gotthardt & Aleksa Varga, 2015 ) english_russian_proverbs : Denis Kleyko, UC Berkeley and Research Institutes of Sweden (denkle@berkeley.edu), and Cameron Diao, Rice University (cwd2@rice.edu). Task cites: ( Bodrova, 2007 ; Gvarjalaże & Mchedlishvili, 1971 ; Wik, ) entailed_polarity : Kaustubh D. Dhole, Amelia R&D, New York (kaustubhdhole@hotmail.com). Task cites: ( Karttunen, 2012 ) entailed_polarity_hindi : Kaustubh D. Dhole, Amelia R&D, New York (kaustubhdhole@hotmail.com). Task cites: ( Karttunen, 2012 ) epistemic_reasoning : Michael Cohen, Stanford University (michaelco13@gmail.com). Task cites: ( Ravenscroft, 2019 ; Call & Tomasello, 2008 ; Bugnyar et al., 2016 ; Stalnaker, 1978 ; Sperber & Wilson, 2002 ; Nematzadeh et al., 2018 ; Le et al., 2019 ; Jiang & de Marneffe, 2019 ; Ross & Pavlick, 2019 ; Bowman et al., 2015 ; de Marneffe et al., 2012 ; Jeretic et al., 2020 ) evaluating_information_essentiality : Anantharaman S. Iyer, National Public School - HSR Layout (i.anantharaman.s@gmail.com), and Niveditha S. Iyer, Stanford University (nivsiyer@stanford.edu). Task cites: ( Rajpurkar et al., 2018 ; Hosseini et al., 2014 ; Levy et al., 2017 ; Yin & Pei, 2015 ; de Marneffe et al., 2008 ; Zhu et al., 2019 ) fact_checker : Genta Indra Winata, Nayeon Lee, Andrea Madotto, and Pascale Fung, HKUST (giwinata, nyleeaa, amadotto)@connect.ust.hk, (pascale@ust.hk); Jason Yosinski, ML Collective (jason@yosinski.com); and Rosanne Liu, Google and ML Collective (rosanneliu@google.com). Task cites: ( Thorne et al., 2018 ; Lee et al., 2021 ) factuality_of_summary : Colin Raffel, Derek Tam, and Mohit Bansal, UNC Chapel Hill (craffel@gmail.com), (dtredsox, mbansal)@cs.unc.edu. Task cites: ( Eyal et al., 2019 ; Wang et al., 2020a ; Durmus et al., 2020 ; Vasilyev et al., 2020 ; See et al., 2017 ; Hermann et al., 2015 ; Narayan et al., 2018 ; Pagnoni et al., 2021 ; Kryscinski et al., 2020 ) fantasy_reasoning : Hugh Mee Wong, Jaap Jumelet, Mario Giulianelli, Clemencia Siro, Maartje ter Hoeve, Agnieszka Kluska, Jeroen Taal, Mátyás Schubert, Omar Elbaghdadi, Dilyar Buzan, Jelle Bosscher, Aman Hussain, and Rahel Habacker, University of Amsterdam (hughmee.wong@student.uva.nl), (j.w.d.jumelet, m.giulianelli, c.n.siro, m.a.terhoeve)@uva.nl, (aga.kluska1404, jeroen1749, schubertmatyas, omarelb, dilyar.muslem.977, jellebosscher)@gmail.com, (email@amanhussain.com), (rahel@habacker.de). Task cites: ( Wang et al., 2019a ; Wang et al., 2018 ; McCann et al., 2018 ; Bhagavatula et al., 2019 ; Lourie et al., 2021 ; Liu et al., 2020a ; Saxton et al., 2019 ; Clark et al., 2018 ; Yu et al., 2019d ; Zhang et al., 2018a ; Johnson et al., 2016 ) few_shot_nlg : Mihir Kale and Abhinav Rastogi, Google (mihirkale, abhirast)@google.com. Task cites: ( Rastogi et al., 2020 ; Kale & Rastogi, 2020 ) figure_of_speech_detection : Sriharsha Hatwar (sriharsha02hatwar@gmail.com). Task cites: ( Potamias et al., 2020 ) forecasting_subquestions : William Saunders, work done while at Ought (william@williamsaunders.net) and Andreas Stuhlmüller, Ought (andreas@ought.org) formal_fallacies_syllogisms_negation : Gregor Betz and Christian Voigt, Karlsruhe Institute of Technology (gregor.betz, christian.voigt)@kit.edu, and Kyle Richardson, Allen Institute for AI (kyler@allenai.org). Task cites: ( Kassner & Schütze, 2019 ; Talmor et al., 2019a ; Betz et al., 2020 ) gem : Sebastian Gehrmann, Google Research (gehrmann@google.com). Task cites: ( Gehrmann et al., 2021 ) gender_inclusive_sentences_german : Simon Thormeyer and Gerard de Melo, HPI/University of Potsdam (simon.thormeyer@student.hpi.de, gerard.demelo@hpi.de) gender_sensitivity_chinese : Xudong Shen, National University of Singapore (xudong.shen@u.nus.edu). Task cites: ( on the Revision of the National Standard Occupational Classification, 2015 ; Household Management Research Center, 2021 ; Household Management Research Center, 2020 ; Household Management Research Center, 2019 ; Qimingtong, 2016 ; Ministry of the Interior, 2018 ) gender_sensitivity_english : Xudong Shen, National University of Singapore (xudong.shen@u.nus.edu). Task cites: ( Bordia & Bowman, 2019 ; Marcus et al., 1994 ; Caliskan et al., 2017 ; Bolukbasi et al., 2016 ; Rudinger et al., 2018 ; Lu et al., 2020 ; Gonen & Goldberg, 2019 ; Hall Maudslay et al., 2019 ; Fellbaum, 1998 ) general_knowledge : Evgenii Zheltonozhskii, Technion – Israel Institute of Technology (evgeniizh@campus.technion.ac.il). Task cites: ( Shane, 2020 ; Dhingra et al., 2017 ; Rajpurkar et al., 2016 ; Rajpurkar et al., 2018 ; Lacker, 2020 ) geometric_shapes : Zijie J. Wang, Georgia Tech (jayw@gatech.edu). Task cites: ( Bostock et al., 2011 ; Marriott et al., 2021 ; Boillot, 2019 ) goal_step_wikihow : Li Zhang, Qing Lyu, and Chris Callison-Burch, University of Pennsylvania (zharry@seas.upenn.edu, lyuqing@sas.upenn.edu, (ccb@cis.upenn.edu). Task cites: ( Zhang et al., 2020d ) gre_reading_comprehension : Mo Tiwari, Medina Orduna Baitemirova, and Chris Waites, Stanford University (motiwari, medinab, waites)@stanford.edu. Task cites: ( Lai et al., 2017 ) hhh_alignment : Amanda Askell, Yuntao Bai, Anna Chen, Deep Ganguli, Danny Hernandez, Jared Kaplan (jared@anthropic.com), Jackson Kernion, Ben Mann, Catherine Olsson, and Tim Telleen-Lawton, Anthropic high_low_game : David Dohan, Google Research (ddohan@google.com) hindi_question_answering : Kumar Shridhar, NeuralSpace and ETH Zurich (kmr.shridhr@gmail.com), and Akshat Agarwal, NeuralSpace (akshat@neuralspace.ai). Task cites: ( Brown et al., 2020 ; Radford et al., 2019 ; Jain et al., 2020 ; Lewis et al., 2020a ; Artetxe et al., 2020 ; Rajpurkar et al., 2016 ) hindu_knowledge : Vinay Ramasesh, Google (ramasesh@google.com), and Piyush Patil, Georgia Tech (ppatil@gatech.edu) hinglish_toxicity : Kumar Shridhar, NeuralSpace and ETH Zurich (shridhar.kumar@inf.ethz.ch), and Akshat Agarwal, NeuralSpace (akshat@neuralspace.ai) human_organs_senses : Fatemeh Siar, Amirkabir University Of Technology (Fatemeh.siar@gmail.com) hyperbaton : Nicholas Roberts, University of Wisconsin-Madison (nick11roberts@cs.wisc.edu); Ryan Teehan, Charles River Analytics (rsteehan@gmail.com); and Vinay Uday Prabhu, UnifyID Labs (vinay@unify.id). Task cites: ( Forsyth, 2014 ) identify_math_theorems : Stella Biderman and Leo Gao, EleutherAI (stella, lg)@eleuther.ai, and Austin Herrick, University of Pennsylvania (aherrick@wharton.upenn.edu). Task cites: ( Gao et al., 2021 ; Black et al., 2022 ; Brown et al., 2020 ; Radford et al., 2019 ; Wang & Komatsuzaki, 2021 ) identify_odd_metaphor : Vikas Raunak, Microsoft (viraunak@microsoft.com). Task cites: ( Lakoff & Johnson, 2008 ; Gao et al., 2018 ) implicatures : Marco Maru and Michele Bevilacqua, Sapienza University of Rome (marco.maru, bevilacqua)@di.uniroma1.it. Task cites: ( Davis, 2019 ; George & Mamidi, 2020 ) implicit_relations : Hugh Mee Wong, Jaap Jumelet, Mario Giulianelli, Clemencia Siro, Maartje ter Hoeve, Agnieszka Kluska, Jeroen Taal, Mátyás Schubert, Omar Elbaghdadi, Dilyar Buzan, Jelle Bosscher, Aman Hussain, and Rahel Habacker, University of Amsterdam (hughmee.wong@student.uva.nl), (j.w.d.jumelet, m.giulianelli, c.n.siro, m.a.terhoeve)@uva.nl, (aga.kluska1404, jeroen1749, schubertmatyas, omarelb, dilyar.muslem.977, jellebosscher)@gmail.com, (email@amanhussain.com), (rahel@habacker.de). Task cites: ( Cain & Oakhill, 1999 ; Bayat & Çetinkaya, 2020 ; Srivastava et al., 2016 ; Lin et al., 2019 ; Massey et al., 2015 )