StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models — part 2/2

Cleaned-text code points [15532, 31461). Bibliography omitted here; source.md includes it.

We then report Solvable Pass Rate (SoPR) and Solvable Win Rate (SoWR) based on our obtained solvable tasks. Due to the limitation of gpt-3.5-turbo-16k in tool learning, we uniformly adopt gpt-4-turbo as the automatic evaluator. SoPR is in essence PR with all tasks solvable and only assesses the answers using the same prompt in ToolBench. The evaluator assigns outcomes of answers categorised as Solved, Unsolved, or Unsure, which respectively contribute scores of 1, 0.5, and 0 to the overall SoPR calculation. As for SoWR, when one is solved and the other is unsolved, the solved one wins. Under other circumstances, gpt-4-turbo will be used to make a win-lose decision.

## 4 Experiment

### 4.1 Performance

Following ToolBench, we run gpt-3.5-0613 , gpt-4-0613 , ToolLLaMA-v2 with CoT and DFS, replenishing with latest models gpt-3.5-1106 and gpt-4-turbo . The results of SoPR and SoWR are shown in and . Generally, GPT-4 series models outperform GPT-3.5 models, while ToolLLaMA performs worst with the same inference algorithm. Also, DFS significantly outperforms CoT whichever LLMs are used. These phenomena are consistent with ToolBench. Furthermore, probably thanks to the improved function calling capabilities, newer GPT models performed better.

### 4.2 Stability of Virtual API Server

Following the same setups as in , we manually make the same success tools not available during the running time. In our design, when a call is on an unavailable tool, it will be directed to the simulated API immediately. Compared to , the results as shown in are much more stable with our virtual API server. Even when 50% of APIs are not available, changes in performance are still not significant, which is explainable within the range of variance.

Considering we use gpt-4-turbo as the backbone of the API simulator which may change even with the same version number, we ablate different versions and different temperatures of gpt-4-turbo . The results are shown in . Under different settings of the backbone LLMs, the performance change is still acceptable within the variance of LLM evaluation, indicating the robustness of our API simulators.

### 4.3 Turing Test of API Simulator

To test the effectiveness of API simulators, we design a “Turing Test” Turing (2009) between the real APIs and the simulated ones. Note that we believe it is not required for API simulators to exactly output the same answers as those of real APIs, where rationality is more important. For example, when a query asks about the weather today, the API simulator does not need to retrieve the “real” temperature. Instead, the API simulator needs to generate a reasonable temperature number.

To do the test, we first sample 70 available real APIs and their corresponding simulated APIs. Given the API callings and their real and simulated response pairs, we ask three human annotators to determine which response more closely resembles an actual API response overall, based on the given descriptions of the API functions. We ask human annotators to evaluate along three dimensions: Overall, Format Accuracy and Answer Relevance. The annotator first need to answer which response is overall more like a real response. When assessing format accuracy, annotators must determine which response more accurately adheres to the format specifications outlined in the documentation. In evaluating answer relevance, they are tasked with identifying which response more effectively fulfills the instruction in accordance with the documentation’s guidelines. The results are shown in . Surprisingly, human annotators cannot distinguish simulated and real APIs very well, where the simulated APIs are judged to act more like real situations. Moreover, the proportion of tie is much larger, indicating that simulated APIs can work very similarly to real APIs.

In addition to the Turing Test mentioned above, we assess the quality of LLM simulations by evaluating the adherence of simulated outputs to their corresponding documentation. To conduct this evaluation, we randomly sample 50 simulated outputs along with their documentation from the Turing Test dataset. A human evaluator is then tasked with determining whether the LLM simulations reasonably follow the provided documentation. The results indicate that 90% of the simulations are deemed reasonable, 6% are considered unreasonable, and 4% are uncertain. These findings suggest that the LLM is highly capable of generating simulated responses that adhere closely to the provided documentation.

### 4.4 Diversity of API Simulator

With the LLM simulation, API simulators will not exactly feedback the same as real APIs. Hence, a natural concern is whether the simulated APIs will degrade in diversity in API functionalities. To study the problem, firstly, we explore the distribution of real and simulated API responses. We first use all 246 APIs in the Tool API category from the successful APIs mentioned in . Then, we use the same call arguments to call these real and simulated APIs. All the responses are encoded using S-BERT Reimers and Gurevych (2019) and their corresponding embeddings are visualised by UMAP McInnes et al. (2018) . Detailed configuration is shown in . The result is shown in . As can be seen from the figure, real and simulated APIs occupied similar embedding space, indicating that the diversity of simulated APIs is similar to the real APIs.

Secondly, we try to explore the behaviour when a simulated API is given several calls with different input arguments. We sample 60 APIs from all successful APIs and make 5 different calls to each API, using the same prompt as in . We then count the number of APIs that give exactly the same responses in any 2 of the 5 calls. Results show that only 2 of 60 APIs contain such responses, which supports the sufficient diversity of our simulated APIs.

### 4.5 Effectiveness of Caching System

To show the effectiveness of our caching system in maintaining the stability of the virtual API server, we run several methods and record the cache hit rates. In detail, we run four methods used in ToolBench, gpt-3.5-turbo-0613 and gpt-4-0613 with CoT and DFS. Reproduction data in ToolBench of these methods has been used in the cache. Results are recorded in , which shows that rerunning these in-domain methods has a very high cache hit rate. This means that most of the call responses are fixed and instability from the API system are much smaller. Nevertheless, models and methods may change over time, and therefore, we further run gpt-3.5-turbo-1106 and gpt-4-turbo-preview with CoT and DFS. As can be seen in the table, although the cache hit rates are smaller with these out-of-domain methods, the scores are still high enough to mitigate the instability significantly.

### 4.6 Human Evaluation of Evaluator

Considering that GPT-3.5 is limited to evaluating the performance in tool learning, we replace the automatic evaluators with stronger LLMs. In this section, we manually assess the correctness of different automatic evaluators. Here, we sample 100 task-solvable questions, 50 answer-solving questions in the PR / SoPR evaluation, and 50 comparison questions in the WR / SoWR evaluation from the experiments running.

We then collect all the corresponding answers of different LLMs during previous evaluations. These questions are further manually labeled by three human annotators to obtain the ground truth. With the ground truth, we calculate the accuracy scores of these models and show the results in . It can be seen that our used LLMs (i.e., Claude 2, Gemini, and GPT-4) are much better than GPT-3.5 in ToolBench to determine the solvability of tasks, where the Gemini and GPT-4 outperform by a large margin. In both the evaluation of answers and comparison, GPT-4 significantly outperforms GPT-3.5, especially in the comparison to compute WR. It is worth noting that all the accuracies of GPT-3.5 are lower than 70%, indicating that GPT-3.5 cannot assess the performance in tool learning.

## 5 Related Work

#### Tool Learning Benchmarks.

Recent studies have shed light on the burgeoning capabilities of LLMs in understanding and mastering tools ( Li et al., 2023a ; Patil et al., 2023 ; Yang et al., 2023b ; Song et al., 2023 ; Tang et al., 2023 ; Ye et al., 2024 ; Xu et al., 2023 ) . Gaining access to external tools endows LLMs with real-time factual knowledge ( Yang et al., 2023a ) , multimodal functionalities ( Gupta and Kembhavi, 2023 ) , and specialised skills in vertical domains ( Jin et al., 2024 ) . However, few work has been done to explore the stability of the tool environment in specific benchmarks and how it affects the LLMs’ performance in tool-augmented tasks.

#### Tool Inference Methods.

Recent literature has begun to explore various methodologies for integrating tool functionalities within LLMs. Notably, the robust in-context learning prowess of LLMs, as demonstrated in Brown et al. (2020) , has facilitated the augmentation of LLMs with external tools via in-context tool descriptions and demonstrations ( Hsieh et al., 2023 ; Ruan et al., 2023 ; Mialon et al., 2023 ) . An alternative approach involves the explicit training of LLMs ( Patil et al., 2023 ; Tang et al., 2023 ; Chen et al., 2023a ; Qin et al., 2023c ; Huang et al., 2023 ) using datasets enriched with tool interactions, thereby familiarising models with the nuances of tool usage.

#### Evaluation in Tool Learning.

Evaluating the performance of LLMs in tool-augmented tasks presents unique challenges and opportunities. Numerous works have been developed for the assessment of tool utilisation, primarily emphasising response comparison ( Zhuang et al., 2023 ) , tool call accuracy ( Patil et al., 2023 ) , or a synthesis of these aspects ( Li et al., 2023a ) . Distinguishing itself, Qin et al. (2023c) introduces an innovative methodology by integrating a large language model (LLM) as a judge to evaluate the comprehensive solution path. Subsequent research ( Wang et al., 2023 ) focuses on the multi-turn interaction capabilities of LLMs with both tools and user feedback. In a departure from the aforementioned approaches, Chen et al. (2023b) presents itself as the inaugural benchmark specifically tailored for the fine-grained assessment of tool utilisation capabilities. However, there exists a gap in the literature concerning the exploration of evaluation stability when evaluating the tool usage capabilities of LLMs.

## 6 Conclusion

In this paper, we propose StableToolBench, a benchmark developed to enhance the stability of ToolBench. Our analysis identified instability issues in the evaluation processes of ToolBench and API status, causing variability in model performance assessments. To address this, we implement a caching system for consistent data availability. We also replace the real API server with an LLM-simulated virtual server for reliable API behaviour simulation. Experiments show that StableToolBench significantly improves the stability of model performance evaluations, with the simulated APIs offering realism and the caching system contributing greatly to the enhanced stability of the benchmark.

## Acknowledgement

This work is supported by the National Natural Science Foundation of China (No. 62276152, 61925601). We also extend our gratitude to Jingwen Wu and Yao Li for their assistance with human evaluation and additional suggestions.

## Limitations

In this work, we propose StableToolBench, a new tool learning benchmark with increased stability but non-declining reality. However, our work faces the following limitations. Firstly, we used GPT-4 as the automatic evaluator in the evaluation process and as the backbone server, which increase the cost of using our benchmark. Secondly, the GPT-4 backboned server demonstrate strong performance in simulating API behaviours. Nevertheless, the backbone LLM may experience significant upgrades, which may affect the performance. Therefore, we believe that the ultimate solution is to solve the problem with a trained open-source LLM. However, current open-source LLMs are not performant enough to simulate API behaviours well. As a result, closed-source LLMs are the only options. We believe that when open-source LLMs are strong enough to be well suited for this task, In the future, we may turn to open-source LLMs when they are strong enough to be well suited for this task. Thirdly, although the cache hit rates are high with our explored methods, new methods will be developed in the future. Whether this cache will still be effective is unsure. In this regard, we aim to continuously update the cache in the future in a slow pace for both balanced stability and effectiveness.

## Appendix A Comparison of Reported and Reproduced Performance

Detailed comparison scores of reported and reproduced performance are shown in .

## Appendix B Statistics of API change information

Detailed statistics of API change categories and information are shown in and .

## Appendix C Stability Test Scores with Virtual API Systems

Detailed scores of stability tests of various models are shown in . Note that in addition to GPT 3.5 Turbo 0613 and GPT 4 0613, we report the performance of newer versions, namely GPT 3.5 Turbo 1106 and GPT 4 Turbo Preview.

## Appendix D Call Error Identification and Cache Filtering Rule

We identify call errors and filter out invalid call to RapidAPI based on keyword occurences. In detail, we identify the following error: • Not Connected Error: when error information contains HTTP or the response infomation contains HTTP error, connection, rate limit, time(d) out ;

• Not Found Error: when the error information or response contains not found, not available, API doesn’t exists, Service Not Found, internal error or 404 error message;

• Parameter Change: when the error information or response contains parameter, parse, is not defined ;

• Parsing Error: when the error information starts with Function executing from ;

• Not Authorised: when the error information or response contains authoriz(s), unauthoriz(s), blocked user, unsubscribe, credential, disabled for your subscription, ACCESS_DENIED or 401, 403 error message;

• Other Errors: messages with non-empty error messages;

• Success: Other calls.

We consider all types of errors when identifying errors. However, when filtering the cache, we do not conside the“Other Errors”.

## Appendix E Configurations of API Diversity Analysis

The configurations of diversity analysis are as follows: • Embedding model: all-mpnet-base-v2 ;

• UMAP metric (distance metric): correlation;

• Num of neighbours: 15;

• Min distance: 0.5.

## Appendix F Prompts of API simulation

The prompt used to simulate API behaviours is shown in .

## Appendix G Prompt to Filter Solvable Task

The prompt used to filter solvable tasks is shown in .

## Appendix H Prompt Used to Make API Calls

The prompt used to construct API calls to scan availables is shown in .

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .