Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models — part 8/9
Cleaned-text code points [110521, 126352). Bibliography omitted here; source.md includes it.
repeat_copy_logic : Noah Fiedel, Google (nfiedel@gmail.com). Task cites: ( Graves et al., 2014 ) rephrase : Marie Tolkiehn, University of Bristol and Center for Data and Computing in Natural Sciences (Universität Hamburg, DESY, TUHH) (marie.tolkiehn@desy.de) riddle_sense : Bill Yuchen Lin, Ziyi Wu, Xiang Ren, Dong-Ho Lee, and Yichi Yang, University of Southern California (yuchen.lin, ziyiwu, xiangren, dongho.lee, yichiyan)@usc.edu. Task cites: ( Lin et al., 2021a ; Talmor et al., 2019b ) roots_optimization_and_games : Daniel Levy, Stanford University (danilevy@stanford.edu). Task cites: ( Lample & Charton, 2019 ; Polu & Sutskever, 2020 ; Amos & Kolter, 2017 ; Agrawal et al., 2020 ) ruin_names : Ryan Teehan, Charles River Analytics (rsteehan@gmail.com); Nicholas Roberts, University of Wisconsin-Madison (nick11roberts@cs.wisc.edu); and Vinay Uday Prabhu, UnifyID Labs (vinay@unify.id). Task cites: ( Attardo, 2017 ; Ren & Yang, 2017 ; Amin & Burghardt, 2020 ; Annamoradnejad & Zoghi, 2020 ; Blinov et al., 2019 ; Yan & Pedersen, 2017 ; Frolovs, 2019 ) salient_translation_error_detection : Vikas Raunak and Arul Menezes, Microsoft (viraunak, arulm)@microsoft.com scientific_press_release : Cristina Garbacea and Qiaozhu Mei, University of Michigan (garbacea, qmei)@umich.edu self_awareness : Roman Sitelew (sitelewr@gmail.com) and Jascha Sohl-Dickstein, Google (jaschasd@google.com). Task cites: ( Yudkowsky, 2008 ; Chella et al., 2020 ; Schick et al., 2021 ; Kounev et al., 2017 ; Huttunen et al., 2017 ; Wallace et al., 2020 ; Clark & Jackson, 1994 ; Horowitz, 2017 ; Branwen, 2020 ; Chu et al., 2017 ) self_evaluation_courtroom : Kaustubh D. Dhole, Amelia R&D, New York (kaustubhdhole@hotmail.com). Task cites: ( Hildebrandt, 2018 ; Daley, 2021 ; King & Cook, 2020 ) self_evaluation_tutoring : Jascha Sohl-Dickstein, Google (jaschasd@google.com). Task cites: ( Zhang et al., 2019a ; Irving et al., 2018 ) semantic_parsing_in_context_sparc : Tao Yu, The University of Hong Kong (tyu@cs.hku.hk); Michihiro Yasunaga, Stanford University (myasu@cs.stanford.edu); and Rui Zhang, Penn State University (r.zhang@psu.edu). Task cites: ( Yu et al., 2019b ; Yu et al., 2018 ; Yu et al., 2019a ) semantic_parsing_spider : Tao Yu, The University of Hong Kong (tyu@cs.hku.hk); Michihiro Yasunaga, Stanford University (myasu@cs.stanford.edu); and Rui Zhang, Penn State University (r.zhang@psu.edu). Task cites: ( Yu et al., 2018 ; Yu et al., 2019b ; Yu et al., 2019a ) sentence_ambiguity : Sahib Singh, Ford Motor Company (sahibsingh570@gmail.com) similarities_abstraction : Michael A. Yee, EleutherAI (yeem@csail.mit.edu). Task cites: ( Nasreddine et al., 2005 ; Wechsler, 2008 ) simp_turing_concept : Gonzalo Jaimovitch-López, Universitat Politècnica de València - VRAIN (gonzalojaimovitch@gmail.com); and César Ferri Ramírez and Jose Hernandez-Orallo, Universitat Politècnica de València (cferri@dsic.upv.es, jorallo@upv.es). Task cites: ( Böhm, 1964 ; Sun et al., 2020 ; Devlin et al., 2018 ; Radford et al., 2019 ; Brown et al., 2020 ; Vaswani et al., 2017 ; Hendrycks et al., 2021b ; Xu et al., 2020b ; Izacard & Grave, 2020 ; Zhu, 2015 ; Cakmak & Thomaz, 2014 ; Goodman & Frank, 2016 ; Degen et al., 2020 ; Khan et al., 2011 ; Basu & Christensen, 2013 ; Yang & Shafto, 2017 ; Melo et al., 2018 ; Telle et al., 2019 ; Chater & Vitányi, 2003 ; Hupkes et al., 2020 ; Lakretz et al., 2019 ; Toshniwal et al., 2021 ; Bender & Koller, 2020 ; Kühl et al., 2020 ; Marcus & Davis, 2020 ; Sinha et al., 2019 ; McClelland et al., 2019 ) simple_ethical_questions : Roman Sitelew (sitelewr@gmail.com); Anders Andreassen, David Dohan, Aitor Lewkowycz, Isaac Noble, Noah Fiedel, and Kristen Chiafullo, Google (ajandreassen, ddohan, alewkowycz, isaacn, nfiedel, chiafullo)@google.com; Kaustubh D. Dhole, Amelia R&D (kaustubhdhole@hotmail.com); Adrià Garriga-Alonso, University of Cambridge (adria.garriga@gmail.com); Stella Biderman, EleutherAI (stella@eleuther.ai); and Tariq Ali (tra38@nau.edu). Task cites: ( Hendrycks et al., 2020 ; Lourie et al., 2020 ) simple_text_editing : Timofei Kornev, University of Potsdam and TomTom GmbH (timofei.kornev.1996@gmail.com); and Roman Sitelew (sitelewr@gmail.com). Task cites: ( Branwen, 2020 ; Malmi et al., 2019 ; Faltings et al., 2020 ) snarks : Trenton Chang, University of Michigan (ctrenton@umich.edu), and Ryan Chi, Ian Ng, Jeremy Kim, Ethan A. Chi, and William Zhang, Stanford University (ryanchi, iyhn8192, jk23541)@stanford.edu, (ethanchi, wxyz)@cs.stanford.edu. Task cites: ( Brown et al., 2020 ; Devlin et al., 2018 ; Lan et al., 2019 ; Liu et al., 2019 ; Radford et al., 2019 ; Annamoradnejad & Zoghi, 2020 ; Chen & Soo, 2018 ; Mao & Liu, 2019 ; Weller & Seppi, 2019 ; Khodak et al., 2017 ; Ghosh et al., 2020 ; González-Ibáñez et al., 2011 ; Joshi et al., 2015 ; McCoy et al., 2019 ; Kaushik et al., 2019 ; Gardner et al., 2020 ; Sennrich, 2016 ; Burlot et al., 2018 ; Naik et al., 2018 ; Zhang et al., 2016 ; Felbo et al., 2017 ; Pant & Dadu, 2020 ; Pelser & Murrell, 2019 ) social_support : Zijian Wang, Amazon (zijwang@stanford.edu), and David Jurgens, University of Michigan (jurgens@umich.edu). Task cites: ( Wang & Jurgens, 2018 ) social_iqa : Maarten Sap and Ronan Le Bras, Allen Institute for AI (maartens, ronanlb)@allenai.org; Hannah Rashkin, Google Research (hrashkin@google.com); Derek Chen, University of Washington (derekchen14@gmail.com); and Yejin Choi, University of Washington and AI2 (yejin@cs.washington.edu). Task cites: ( Sap et al., 2019 ; Bisk et al., 2020 ; Talmor et al., 2019b ; Zellers et al., 2018 ) spelling_bee : Ethan Dyer, Google Research, Blueshift Team (edyer@google.com). Task cites: ( Ginsberg, 2014 ) sports_understanding : Ethan Kim, Harvard University (ethan_kim@college.harvard.edu) squad_shifts : John Miller and Karl Krauth, UC Berkeley, (miller_john, karlk)@berkeley.edu; and Ludwig Schmidt, University of Washington (schmidt@cs.uw.edu). Task cites: ( Miller et al., 2020 ; Brown et al., 2020 ; Rajpurkar et al., 2016 ; Baumgartner et al., 2020 ; McAuley et al., 2015 ) strange_stories : Yifu Chen, Stanford University (yifuchen@stanford.edu); Francesca Happé, King’s College London (francesca.happe@kcl.ac.uk); and Aitor Lewkowycz, Google Research, Blueshift Team (alewkowycz@google.com). Task cites: ( Happé, 1994 ; White et al., 2009 ) strategyqa : Mor Geva and Jonathan Berant, Tel Aviv University and Allen Institute for AI (morgeva, joberant)@cs.tau.ac.il; Elad Segal, Tel Aviv University (elad.segal@gmail.com); Dan Roth, University of Pennsylvania (danroth@seas.upenn.edu); and Daniel Khashabi and Tushar Khot, Allen Institute for AI (danielk, tushark)@allenai.org. Task cites: ( Geva et al., 2021 ) subject_verb_agreement : Dieuwke Hupkes, Facebook AI Research (dieuwkehupkes@fb.com); Jason Wei and Dan Garrette, Google (jasonwei, dhgarrette)@google.com; Ellie Pavlick, Google and Brown University (epavlick@google.com); Tal Linzen, New York University (linzen@nyu.edu); Yair Lakretz, Cognitive Neuroimaging Unit, INSERM, CEA, CNRS, Université Paris-Saclay, NeuroSpin center, 91191 Gif/Yvette, France (yair.lakretz@gmail.com); Théo Desbordes, Facebook AI Research and Cognitive Neuroimaging Unit, INSERM, CEA, CNRS, Université Paris-Saclay, NeuroSpin center, 91191 Gif/Yvette, France (tdesbordes@fb.com); Germán Kruszewski, Naver Labs Europe (german.kruszewski@naverlabs.com); Marco Marelli, University of Milano-Bicocca (marco.marelli@unimib.it); and Stanislas Dehaene, NeuroSpin, France (stanislas.dehaene@gmail.com). Task cites: ( Lakretz et al., 2021b ; Lakretz et al., 2019 ; Linzen et al., 2016 ; Gulordava et al., 2018 ; Marvin & Linzen, 2018 ; Goldberg, 2019 ; Lakretz et al., 2021a ; Wolf, 2019 ) sudoku : Adrià Garriga-Alonso, University of Cambridge (adria.garriga@gmail.com). Task cites: ( Wang et al., 2019c ; Russell & Norvig, 2002 ; Garcez & Lamb, 2020 ; Huang et al., 2018 ; Hendrycks et al., 2021c ) sufficient_information : Sam Dillavou, University of Pennsylvania (dillavou@sas.upenn.edu) suicide_risk : Jekaterina Novikova, NoOverfitting Lab (novikova.jekaterina@gmail.com), and Ksenia Shkaruta, NoOverfitting Lab and Georgia Tech (ksenia@shkaruta.com). Task cites: ( Gaur et al., 2019 ; Mohammadi et al., 2019 ; Matero et al., 2019 ; Shing et al., 2018 ) swahili_english_proverbs : Kevin Omondi, Eunice Engefu Manyasi, Victoria Nyamai, Joan Waweru, Titus Tunduny, and Tiberius Nkinyili, Strathmore University, (kevochi09, eunismanyasi, victorianyamai12, jowambui123, ttunduny, ttnkinyili)@gmail.com; and Clara Rivera, Google (rivera@google.com). swedish_to_german_proverbs : Marie Tolkiehn, University of Bristol and Center for Data and Computing in Natural Sciences (Universität Hamburg, DESY, TUHH) (marie.tolkiehn@desy.de). Task cites: ( Hanzén, 2007 ; Korhonen, 2009 ; Meister, 2007 ; Mieder, 2019 ) symbol_interpretation : Emanuele Rodola, Luca Moschella, Giorgio Mariani, Antonio Norelli, Andrea Santilli, and Simone Melzi, Sapienza University of Rome (rodola, moschella, mariani, norelli, santilli, melzi)@di.uniroma1.it; and Giambattista Parascandolo, MPI for Intelligent Systems and ETH Zurich (gparascandolo@tue.mpg.de). Task cites: ( Brown et al., 2020 ; Johnson et al., 2016 ; Santoro et al., 2017 ; Hudson & Manning, 2019 ; Sennrich et al., 2015 ) taboo : Dar Gilboa, Harvard University (dar_gilboa@fas.harvard.edu). Task cites: ( Joshi et al., 2017 ) talkdown : Zijian Wang, Amazon (zijwang@stanford.edu), and Christopher Potts, Stanford University (cgpotts@stanford.edu). Task cites: ( Wang & Potts, 2019 ; Mendelsohn et al., 2020 ; Fiske, 1993 ; Nolan & Mikami, 2013 ; Breitfeller et al., 2019 ; Perez Almendros et al., 2020 ) temporal_sequences : Ethan A. Chi, William Zhang, Hannah Kim, James Zheng, Oliver Zhang, and Jeremy Kim, Stanford University (ethanchi, wxyz)@cs.stanford.edu, (hkim24, jameszhe, ozhang, jk23541)@stanford.edu. Task cites: ( Elazar et al., 2021 ; Pustejovsky et al., 2004 ; Sanampudi & Kumari, 2010 ; Han et al., 2020 ; Ma et al., 2021 ; Brown et al., 2020 ; Petroni et al., 2019 ) tense : Mukund Varma T, Indian Institute of Technology, Madras (mukundvarmat@gmail.com), and Diganta Misra, Mila and Landskape AI (diganta.misra@mila.quebec). Task cites: ( Logeswaran et al., 2018 ) text_navigation_game : Daniel Freeman, Google (cdfreeman@google.com). Task cites: ( Vinyals et al., 2019 ; Küttler et al., 2020 ; Kanagawa & Kaneko, 2019 ; Noever et al., 2020 ) timedial : Aditya Gupta, Manaal Faruqui, Shyam Upadhyay, and Luheng He, Google (gaditya, mfaruqui, shyamupa, luheng)@google.com; Yejin Choi, University of Washington and AI2 (yejin@cs.washington.edu); and Lianhui Qin, University of Washington (lianhuiq@cs.washington.edu). Task cites: ( Qin et al., 2021 ; Li et al., 2017 ; Zhou et al., 2019 ) topical_chat : Behnam Hedayatnia, Karthik Gopalakrishnan, Anu Venkatesh, Anna Gottardi, and Raefer Gabriel, Amazon (behnam, karthgop, anuvenk, gottardi, raeferg)@amazon.com; Sanjeev Kwatra, Bluevine (sanjeevkwatra0@gmail.com), and Qinlang Chen, Google (qinlangchen@gmail.com). Task cites: ( Gopalakrishnan et al., 2019 ; Mehri & Eskenazi, 2020 ; Gopalakrishnan et al., 2020 ; Hedayatnia et al., 2020 ) tracking_shuffled_objects : James B. Simon, UC Berkeley (james.simon@berkeley.edu). Task cites: ( Liu et al., 2020b ; Dong et al., 2020 ) training_on_test_set : Jascha Sohl-Dickstein, Google (jaschasd@google.com) truthful_qa : Stephanie Lin and Owain Evans, University of Oxford (sylin07, owaine)@gmail.com; and William Saunders, Jacob Hilton, and Elizabeth Barnes, OpenAI (william, jhilton, barnes)@openai.com. Task cites: ( Brown et al., 2020 ; Sellam et al., 2020 ; Amodei et al., 2016 ; Leike et al., 2018 ; Kenton et al., 2021 ; Clark et al., 2018 ; Bhakthavatsalam et al., 2021 ; Hendrycks et al., 2021b ; Khashabi et al., 2020b ; Kreps et al., 2020 ; Maynez et al., 2020 ; Gabriel et al., 2020 ; Wang et al., 2020a ; Stiennon et al., 2020 ; Lewis et al., 2020b ; Krishna et al., 2021 ; Gehrmann et al., 2021 ; Xu et al., 2020a ; Dinan et al., 2019 ; Tamkin et al., 2021 ; Bowman & Dahl, 2021 ) twenty_questions : Jascha Sohl-Dickstein, Google (jaschasd@google.com). Task cites: ( Rajpurkar et al., 2016 ; Choi et al., 2018 ; Reddy et al., 2019 ; Aliannejadi et al., 2019 ; Clark et al., 2019 ; Zhang et al., 2019a ) understanding_fables : Denis Emelin, University of Edinburgh (denis.emelin@gmail.com). Task cites: ( Reimers & Gurevych, 2019 ; Salazar et al., 2020 ; Wolf et al., 2019 ) undo_permutation : Mukund Varma T, Indian Institute of Technology, Madras (mukundvarmat@gmail.com). Task cites: ( Pham et al., 2020 ) unit_conversion : Richard Barnes, Meta and Lawrence Berkeley National Lab (rijard.barnes@gmail.com); James B. Simon, B. Ryan Roberts, Stefan Divic, Sajant Anand, and Elizabeth Donoway, UC Berkeley (james.simon, ryan_roberts, stefan_divic, sajant, elizabeth.donoway)@berkeley.edu; Vikas Raunak, Microsoft (viraunak@microsoft.com); Jack Geissinger, Carnegie Robotics (jack.geissinger7@gmail.com); and Peter Chang, Thomson Reuters Special Services (pchang534@gmail.com). Task cites: ( Hendrycks et al., 2021c ; Geva et al., 2020a ) unit_interpretation : James B. Simon, Tianle Wang, Stefan Divic, Sajant Anand, and Elizabeth Donoway, UC Berkeley (james.simon, tianle_wang, stefan_divic, sajant, elizabeth.donoway)@berkeley.edu; and Jack Geissinger, Carnegie Robotics (jack.geissinger7@gmail.com) unnatural_in_context_learning : Frieda Rong and Percy Liang, Stanford University (rongf@stanford.edu, pliang@cs.stanford.edu). Task cites: ( Brown et al., 2020 ; Kaplan et al., 2020 ; Henighan et al., 2020 ; Hernandez et al., 2021 ; Bahri et al., 2021 ; Wang et al., 2019a ; Hernandez et al., 2020 ; Hendrycks et al., 2021c ; Hendrycks et al., 2021a ; Liu et al., 2021a ; Zhao et al., 2021 ; Perez et al., 2021 ) unqover : Tao Li, University of Utah (tlI@cs.utah.edu); Tushar Khot, Daniel Khashabi, and Ashish Sabharwal, Allen Instituite for AI (tushark, danielk, ashishs)@allenai.org; and Vivek Srikumar, University of Utah (svivek@cs.utah.edu). Task cites: ( Li et al., 2020a ; Caliskan et al., 2017 ; Rudinger et al., 2018 ; Zhao et al., 2018 ; Dev et al., 2020 ; Stanovsky et al., 2019 ; Nadeem et al., 2020 ; Sheng et al., 2019 ; Zhang et al., 2020b ) vitaminc_fact_verification : Tal Schuster, Google (talschuster@google.com), and Adam Fisch, MIT (adamfisch15@gmail.com). Task cites: ( Schuster et al., 2021a ) web_of_lies : Roman Novak, Google (romann@google.com) what_is_the_tao : Alice Xiang, Duke Kunshan University (alice.xiang@duke.edu), and Ekin Dogus Cubuk, Google Research, Brain Team (cubuk@google.com) which_wiki_edit : Nick Doiron, Tufts University (nick@mapmeld.com) winowhy : Xinran Zhao, Stanford University (xzhaoar@connect.ust.hk), and Hongming Zhang and Yangqiu Song, HKUST (hzhangal, yqsong)@cse.ust.hk. Task cites: ( Zhang et al., 2020c ; Devlin et al., 2018 ; Liu et al., 2019 ; Kocijan et al., 2019 ; Rahman & Ng, 2012 ; Sakaguchi et al., 2020 ) word_problems_on_sets_and_graphs : Benjamin Inden, Max Planck Institute for Mathematics in the Sciences (inden@mis.mpg.de). Task cites: ( Bency et al., 2019 ; Mnih et al., 2013 ; Russell & Norvig, 2002 ; Besold et al., 2017 ; Clark et al., 2020 ; Wang et al., 2018 ; Lacker, 2020 ) word_sorting : Mo Tiwari, Medina Orduna Baitemirova, and Chris Waites, Stanford University (motiwari, medinab, waites)@stanford.edu. Task cites: ( Grover et al., 2019 ) word_unscrambling : Rylan Yang, The Harker School (rylyang@gmail.com), and Ethan A. Chi, Stanford University (ethanchi@cs.stanford.edu). Task cites: ( Nishino et al., 2019 ; Rozner et al., 2021 ; Jones et al., 2020 ; Mays et al., 1991 ; Edizel et al., 2019 ; Sakaguchi et al., 2016 ; Kim et al., 2015 ; Xue et al., 2021a ; Wu et al., 2020 ; Rust et al., 2020 )