StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models

arxiv:2403.07714

Original source · Paper JSON

Retrieval, embeddings, benchmarks · 2024-03-12

Cites (0) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2

Contents

• StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models
• Abstract
• 1 Introduction
• 2 Stability Analysis on ToolBench
• 2.1 Stability of Performance
• 2.2 Stability of Evaluation
• 2.3 Stability of API Status
• 3 StableToolBench
• 3.1 Virtual API Server
• Caching System.
• API Simulator.
• API Calling Rules.
• 3.2 Stable Evaluation System
• Judging Solvable Tasks.
• Metrics.
• 4 Experiment
• 4.1 Performance
• 4.2 Stability of Virtual API Server
• 4.3 Turing Test of API Simulator
• 4.4 Diversity of API Simulator
• 4.5 Effectiveness of Caching System
• 4.6 Human Evaluation of Evaluator
• 5 Related Work
• Tool Learning Benchmarks.
• Tool Inference Methods.
• Evaluation in Tool Learning.
• 6 Conclusion
• Acknowledgement
• Limitations
• Appendix A Comparison of Reported and Reproduced Performance
• Appendix B Statistics of API change information
• Appendix C Stability Test Scores with Virtual API Systems
• Appendix D Call Error Identification and Cache Filtering Rule
• Appendix E Configurations of API Diversity Analysis
• Appendix F Prompts of API simulation
• Appendix G Prompt to Filter Solvable Task
• Appendix H Prompt Used to Make API Calls
• Instructions for reporting errors