StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models — part 1/2

Cleaned-text code points [0, 15532). Bibliography omitted here; source.md includes it.

# StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models

###### Abstract

LargeLanguageModels ( LLMs ) havewitnessedremarkableadvancementsinrecentyears , promptingtheexplorationoftoollearning , whichintegratesLLMswithexternaltoolstoaddressdiversereal - worldchallenges . AssessingthecapabilityofLLMstoutilisetoolsnecessitateslarge - scaleandstablebenchmarks . However , previousworksreliedoneitherhand - craftedonlinetoolswithlimitedscale , orlarge - scalerealonlineAPIssufferingfrominstabilityofAPIstatus . Toaddressthisproblem , weintroduceStableToolBench , abenchmarkevolvingfromToolBench , proposingavirtualAPIserverandstableevaluationsystem . ThevirtualAPIservercontainsacachingsystemandAPIsimulatorswhicharecomplementarytoalleviatethechangeinAPIstatus . Meanwhile , thestableevaluationsystemdesignssolvablepassandwinratesusingGPT -4 astheautomaticevaluatortoeliminatetherandomnessduringevaluation . ExperimentalresultsdemonstratethestabilityofStableToolBench , andfurtherdiscusstheeffectivenessofAPIsimulators , thecachingsystem , andtheevaluatorsystem .

## 1 Introduction

With the rapid developments of Large Language Models (LLMs; Brown et al., 2020 ; Gemini Team, 2023 ; OpenAI, 2023 ; Touvron et al., 2023 ), tool learning which leverage LLMs to schedule a variety of external tools has attracted enormous attention Nakano et al. (2022) ; Yao et al. (2022b) ; Lu et al. (2023) . Previous studies ( Hao et al., 2023 ; Hsieh et al., 2023 ; Schick et al., 2023 ; Tang et al., 2023 ) aim to augment LLMs with tools to enhance performance on conventional natural language processing (NLP) downstream tasks, while recent work ( Qin et al., 2023a ; Yao et al., 2022a ; Cai et al., 2024 ) primarily focus on solving real-world scenarios that require the use of tools. In general, tool learning complements the capabilities of vanilla LLMs and bridges the gap to real-world applications.

To assess the capability of LLMs to use tools, a series of tool learning benchmarks have been introduced. Several pioneering studies have heavily relied on human-crafted offline tools Yang et al. (2023b) ; Xu et al. (2023) or hand-selected online tools Li et al. (2023b) ; Li et al. (2023a) ; Chen et al. (2023b) . While these tools are high-quality, their scale remains relatively small, thereby limiting their ability to accurately reflect real-world scenarios. To address this limitation, subsequent studies ( Tang et al., 2023 ; Ye et al., 2024 ; Qin et al., 2023b ) have advocated for leveraging extensive collections of online tools that span across various domains. Owing to the increased scale, the automatic evaluation of tool learning has moved closer to real-world scenarios. However, concerns have been raised regarding the stability of these online tools, which has implications for the reproducibility and comparability of benchmark performance over time. For instance, the well-recognised ToolBench Qin et al. (2023c) has shown performance discrepancies that cannot be reproduced months after its release, as analysed in . This is even more important when faced with a complex environment, where APIs and tools keep changing while the evaluation should maintain its consistency across time.

Existing large-scale benchmarks may struggle to provide stable evaluations for various reasons. We propose several hypotheses for this issue. Firstly, the complexity of tasks involving tool usage makes it challenging for the common automatic evaluator, gpt-3.5 , to function effectively as a discriminator. As discussed in , the evaluator cannot reliably determine whether a task is solvable or unsolvable, leading to variability in model performance due to this capability limitation. Secondly, the stability of API status for a significant portion of online tools (55.6% in ToolBench) is inconsistent. Users may be required to authorise the use of these tools or APIs, and tools provided by developers may be accessible during the initial construction of the benchmark but become unavailable later. This fluctuation further undermines the reliability and reproducibility of model performance assessments over time. This situation results in a problem where the constructed queries in the benchmarks may no longer be completed with their originally referenced tools. Consequently, it is crucial to strike a balance between enhancing the stability of these benchmarks and maintaining their diversity and scope.

To address these issues, we propose a new benchmark named StableToolBench, which incorporates a virtual API system and a stable evaluation system. We first build a virtual API system to replace the real one. As a start, we build a caching system to store the outputs of API calls. This approach ensures the stability and reproducibility of API behaviours. Given the limited number of benchmark questions, our caching system can cover a significant number of API call scenarios. However, relying solely on a cache is insufficient because many APIs remain unavailable. To resolve this problem, we use large language models (LLMs) to simulate the behaviours of these APIs. Specifically, we feed the documentation and few-shot real API calls if available in the cache to LLMs and ask LLMs to mock the behaviour of the APIs given a request. As a result, users can always get responses from APIs in an indistinguishable way as long as the LLMs are accessible. On the whole, our system first tries to find a hit in the cache. Unless there is a cache miss and a real API call is not received, the simulated server will be used.

We then improve the evaluation system to make it more stable. We design two metrics (i.e., SoPR and SoWR) after judging solvable tasks and replace all the automatic evaluators with GPT-4 to mitigate the randomness and indistinguishability during evaluation. Experiments demonstrate that our virtual API system, when combined with the improved evaluation system, can provide stable evaluation against API modifications. Furthermore, our system exhibits significant reliability in terms of realism, diversity, and documentation following accuracy.

The main contributions of our work are summarised as follows: • A tool-learning benchmark featured a large number of cached stable simulated APIs, well-balancing stability and reality of the APIs and much more stable evaluation metrics.

• Extensive experiments show that our benchmark provides much more stable model performance, robust to various types of API failures.

• Besides enhanced stability, our virtual API system exhibits reality, diversity and reliability comparable to that of the real API system.

## 2 Stability Analysis on ToolBench

In this section, we initiate a comprehensive analysis to reveal the stability of established tool benchmarks, using Toolbench ( Qin et al., 2023b ) as a case study. We examine the stability of ToolBench by investigating three key dimensions: performance, evaluation, and API status.

### 2.1 Stability of Performance

Benchmarks are designed to consistently evaluate the performance of various models over time. To test this consistency, we reproduce the model performances and record any variations. Our study employs Chain-of-Thought (CoT; Wei et al., 2023 ) and Depth First Search (DFS) strategies, leveraging ChatGPT and ToolLLaMA for comparative analysis. We adhere strictly to the configurations detailed in ToolBench, utilising the ChatGPT version gpt-3.5-turbo-0613 and ToolLLaMA-v2 . As depicted in , we compare the original Pass Rates for the I1-Instruction group reported by ToolBench with the Pass Rates we reproduced for four conventional methods.

Our findings indicate a notable decline in the performance of all methods over time, which raises concerns about the stability of ToolBench as a benchmark.

### 2.2 Stability of Evaluation

In ToolBench, there are two types of metrics, including Pass Rate (PR) and Win Rate (WR). PR is calculated based on using gpt-3.5-turbo-16k to determine if a task is solvable and whether the generated answer can solve the corresponding task. details the computation process of PR. Specifically, a solvable task results in a pass if the answer is solved, a failure if unsolved, and is randomly determined if unsure. For tasks deemed unsure, a pass is assigned only if the answer is solved; otherwise, a random outcome is chosen. If a task is unsolvable, the result defaults to a pass regardless of the answer status. Moreover, WR is derived from the comparative PR of paired candidates. A candidate’s WR increases by one each time it passes while the other fails. In all other situations, WR relies on gpt-3.5-turbo-16k to automatically evaluate the paired candidates.

To assess the stability of evaluation, we perform both CoT and DFS using gpt-3.5-turbo-0613 on the I1-Instruction group dataset. These analyses are conducted using the provided tools and repeat over three iterations each. The resulting PR and WR are presented in , with detailed task and answer items. Despite PRs of DFS being generally higher than CoT, the contribution of the task is larger than the answer. However, it is worth noting that the tasks are the same in both CoT and DFS, where their results are expected to be consistent. On the contrary, the discrimination of answers between CoT and DFS is weak, where a considerable proportion are unsolved. Moreover, WR does not reflect the same trend as PR, where the second run of WR in DFS (48.0) is even lower than the first run of CoT. Therefore, all the phenomena reflect that gpt-3.5-turbo-16k can not assume the role of the automatic evaluator in tool learning, which will be discussed in .

### 2.3 Stability of API Status

We investigate the change of API status in ToolBench. In detail, we scan the original APIs downloaded from ToolBench, and use gpt-4-turbo to automatically write calls via the prompts as shown in . According to the keyword in API feedback, we classify these APIs into three categories: success, not availability, and not authorisation. The API status and the detailed errors of not availability are presented in .

As can be seen, only 44.4% of API calls are successful, while other API calls are mostly not available with various errors and some are not authorised.

Furthermore, to validate the impact of API call failures on the stability of model performance, we manually make some success tools down by returning a special failure call. Specifically, we randomly sample a proportion of success tools containing success APIs found in . At testing time, when sampled tools are called, a special response will be thrown: {‘‘error’’: ‘‘’’, ‘‘response’’: ‘‘This API did not return any useful information...’’} to simulate the API call failures. We conduct baseline models with different proportions (i.e., 0%, 10%, 20%, and 50%) of sampled APIs on the I1 Instruction set. Due to the issues in evaluation, we use our stable evaluation system proposed in as the same as our main experiments. For each experiment, we evaluate three times and report the average scores as shown in . It can be seen that the performance degrades a lot when the proportion of successful APIs is down, thus the impact of API status on stability is considerable.

## 3 StableToolBench

Considering that stability is a crucial feature of benchmarking, in this paper, we specifically design a virtual API server and stable evaluation system to improve the stability based on ToolBench, and propose a new benchmark, named StableToolBench.

### 3.1 Virtual API Server

With real APIs, many of the failures encountered when reproducing its experiments are caused by expired APIs, network issues, or failed authentication. To address this problem, we specifically propose a virtual API server with two components as illustrated in , including a caching system and API simulator. Moreover, we design API calling rules to combine these two components to ensure the virtual API server is stable.

#### Caching System.

We first implement a caching system that stores responses of all API callings to ensure consistency. The caching system uses keys composed of their category, tool, API name, and arguments. As a start, we populate the initial cache using the API call history from the training data and the reproduced test data released in ToolBench. To ensure the quality of cached APIs, only valid records following the rule in will be saved. It is worth noting that we will also reserve some APIs with exceptions to keep the reality. In this way, most API responses will be readily available, allowing the benchmark focus on probing the tool usage ability of designed methods with minimal impact on tool availability. Furthermore, the API call in new experiments will also be continuously updated in the cache to ensure scalability. The statistics of cache are shown in .

Additionally, as an extra benefit, this approach reduces the latency introduced by interacting with real APIs, and also saves the costs for the API simulator discussed below.

#### API Simulator.

Due to the limited coverage of the caching system, we propose to use LLMs to simulate API responses that are not in the cache and unavailable. Specifically, we ask gpt-4-turbo to simulate the API behaviour based on the original documentation in ToolBench. The API documentation includes the descriptions of the functions and their corresponding parameters. To mitigate the difference between simulated and real APIs, we use real API calls in the caching system as few-shot examples ( Brown et al., 2020 ) for the LLM to better mock the behaviours. We keep the maximum number of examples at five. When less than five examples exist in the cache, all of them will be used. Detailed prompts can be found in .

#### API Calling Rules.

Based on the caching system and the API simulator, we create API calling rules to ensure the stability of the virtual API server. When a call request ( e , args ) (\mathrm{e},\mathrm{args}) , where e \mathrm{e} is the API endpoint and args \mathrm{args} is the arguments for that endpoint, is received, the system will first search the caching system for ( e , args ) (\mathrm{e},\mathrm{args}) pair. If a cache hit exists, the cached response will be directly returned. When there is a cache miss, then the system will try to call the real API for a response to maintain the reality of the whole system. If the real API calling is successful, the response will be returned. However, when the caching system does not contain and the real API is not available, the system will finally call the simulated API. The final response whether from the real API or the simulated API will be saved to update in the caching system.

### 3.2 Stable Evaluation System

In this section, we propose a two-phase evaluation process, including judging solvable tasks and evaluating with two metrics, as shown in . Moreover, we replace all the automatic evaluators with gpt-4-turbo .

#### Judging Solvable Tasks.

Considering that both unsolvable and unsure tasks would introduce enormous randomness while the results of tasks fluctuate wildly, we try to obtain a fixed collection of solvable tasks to eliminate the problems. To achieve this, we use three state-of-the-art LLMs, i.e., gpt-4-turbo , gemini-pro , and claude-2 , to determine whether a task is solvable or unsolvable. The prompt is shown in . The task will be judged as solvable when more than two models evaluate it as solvable. The statistics of solvable tasks across all test datasets in ToolBench are shown in .

#### Metrics.