Mathematical discoveries from program search with large language models — part 4/4
Cleaned-text code points [46259, 55104). Bibliography omitted here; source.md includes it.
Related to genetic programming, the field of hyper-heuristics 79 , 80 seeks to design learning methods for generating heuristics applied to combinatorial optimization problems. In practice, these heuristics are often programs discovered through genetic programming, typically by evolving a heuristic on a set of instances of a given combinatorial optimization problem, such as bin packing 81 . Indeed, like FunSearch, hyper-heuristics have also been applied to online bin packing, with the learned heuristics able to match the performance of first fit 82 and best fit 83 on a set of generated bin packing instances. Augmenting the heuristics with memory of previously seen items can even lead to heuristics outperforming best fit 84 . In addition, these evolved heuristics can sometimes generalize to larger instances than the ones they were trained on 85 , similar to the learned FunSearch heuristics. However, as is the case with genetic programming, one of the fundamental limitations of hyper-heuristics is that the components of the evolved heuristic must be manually defined by the user and often need to be tailored to a specific problem to be effective. The LLM in FunSearch allows us to bypass this limitation and learn heuristics for bin packing and job scheduling as well as discovering new mathematical constructions, all within a single pipeline without problem-specific tuning. #### Program superoptimization and software engineering Searching for the best way of modifying source code is a task that appears in several branches of computer science and software development. These occurrences can be broadly classified into two groups: first, in which the goal is to find semantic-preserving modifications (this arises in program optimization and superoptimization, in which the aim is to modify the program so that it executes faster while maintaining its input–output behaviour), and second, in which the goal is to find programs with different semantics (this arises, for example, in automatic program repair and mutation testing). With some exceptions discussed below, most of these areas use relatively simple and hard-coded mutation operators on either the source code directly (such as deleting or swapping lines) or on the abstract syntax tree. Machine learning approaches have been used for program superoptimization. For example, ref. 86 used reinforcement learning to learn the sampling probabilities used within a hierarchical probabilistic model of simple program edits introduced by STOKE 87 . Neural networks have also been proposed as a mutation operator for program optimization in ref. 88 . These studies operated on code written in Assembly (perhaps because designing meaningful and rich edit distributions on programs in higher-level languages is challenging). More recently, ref. 13 used LLMs to find performance-improving edits to code written in C++ or Python. We also note that reinforcement learning has recently been applied to discover new faster algorithms for fundamental operations such as matrix multiplication 89 and sorting 90 . In this paper, we have not explicitly explored semantic-preserving applications such as discovering performance-improving code edits, but we believe that FunSearch could be an effective method for that setting too. In both use cases presented in the main text, the goal is to evolve programs with new semantics, but the application is different from program repair or mutation testing: in the ‘Extremal combinatorics’ section, we used FunSearch to discover a program that constructs a previously unknown mathematical object, and in the ‘Bin packing’ section, we used FunSearch to discover a program that corresponds to a more efficient heuristic for online bin packing. ## Data availability The experiments carried out in this paper do not require any data corpus other than the publicly available OR-Library bin packing benchmarks 23 . The output functions of interest produced by FunSearch are shown across the main paper and in text files in the Supplementary Information . ## Code availability The discovered functions as well as the evolutionary algorithm, code manipulation routines and a single-threaded implementation of the FunSearch pipeline are available as Python code in the Supplementary Information and at https://github.com/google-deepmind/funsearch . Furthermore, the software library launchpad 91 and a sandbox for safely executing generated code on our internal distributed system were used. No training or fine-tuning of a LLM is required; API access for inference is sufficient. We used Codey 26 , which is available through its API, and StarCoder 6 , which is open source. ## Supplementary information ### Supplementary Information (download PDF ) Further details about the method and extra results. ### Supplementary Data 1 (download ZIP ) This zipped code file contains: (a) the evolutionary algorithm, code manipulation routines and a single-threaded implementation of the FunSearch pipeline; and (b) output functions of interest produced by FunSearch. ## Rights and permissions Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . Reprints and permissions ## About this article ### Cite this article Romera-Paredes, B., Barekatain, M., Novikov, A. et al. Mathematical discoveries from program search with large language models. Nature 625 , 468–475 (2024). https://doi.org/10.1038/s41586-023-06924-6 Download citation Received : 12 August 2023 Accepted : 30 November 2023 Published : 14 December 2023 Version of record : 11 January 2024 Issue date : 18 January 2024 DOI : https://doi.org/10.1038/s41586-023-06924-6 ### Share this article Anyone you share the following link with will be able to read this content: Sorry, a shareable link is not currently available for this article. Provided by the Springer Nature SharedIt content-sharing initiative ## Comments Commenting on this article is now closed. Claire99 31 January 2024, 18:08 It's truly outstanding what we have achieved with LLMs in short amount of time. The next logical step would be is to turn that output into Large Action Models (LAM) and create actionable intelligence and saving billions of hours on repetitive and mundane tasks. For example, LLM can fetch all the information related to flights vs London and NY, figure out the cheapest tickets, airline while LAM focuses on actually booking the ticket and making the arrangements for you in real world. Or, a game example; In Minecraft , LLM identifies all necessary blocks, crafting items and LAM builds the houses structures for you, while you play with your friends. ## Associated content ### Large language models help computer programs to evolve Jean-Baptiste Mouret Advertisement ## Search ### Quick links Explore articles by subject Find a job Guide to authors Editorial policies Nature ( Nature ) ISSN 1476-4687 (online) ISSN 0028-0836 (print) ## nature.com footer links ### About Nature Portfolio About us Press releases Press office Contact us ### Discover content Journals A-Z Articles by subject protocols.io Nature Index ### Publishing policies Nature portfolio policies Open access ### Author & Researcher services Reprints & permissions Research data Language editing Scientific editing Nature Masterclasses Research Solutions ### Libraries & institutions Librarian service & tools Librarian portal Open research Recommend to library ### Advertising & partnerships Advertising Partnerships & Services Media kits Branded content ### Professional development Nature Awards Nature Careers Nature Conferences ### Regional websites Nature Africa Nature China Nature India Nature Japan Nature Middle East Privacy Policy Use of cookies Your privacy choices/Manage cookies Legal notice Accessibility statement Terms & Conditions Your US state privacy rights © 2026 Springer Nature Limited Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.