Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question? — part 4/4

Cleaned-text code points [47390, 49037). Bibliography omitted here; source.md includes it.

Several recent works investigate the reliability of using MCQA to evaluate LLMs. However, the overwhelming majority of these experiments focus solely on prompt sensitivity and LLM robustness Zheng et al. (2024) ; Pezeshkpour and Hruschka (2023) ; Wang et al. (2024) ; Alzahrani et al. (2024) , collectively pointing out that LLM performance on MCQA can substantially change if the in-context examples, choice order, and symbols used to denote options, among other properties, are altered. Conversely, we focus on the issue of artifact exploitation , allowing us to holistically assess MCQA tasks and datasets, rather than the different ways LLMs can be implemented to perform MCQA.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .