Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?

arxiv:2402.12483

Original source · Paper JSON

Retrieval, embeddings, benchmarks · 2024-02-19

Cites (0) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?
• Abstract
• 1 Introduction
• 2 Experimental Setup
• 2.1 MCQA Task Definition
• 2.2 Models
• 2.3 Datasets
• 2.4 Evaluation Protocol
• 3 Performing MCQA With No Question
• 3.1 Prompt Design
• 3.2 Results
• 4 Hypothesis 1: Memorization
• 4.1 Prompt Design
• 4.2 Results
• 5 Hypothesis 2: Choice Dynamics
• 5.1 Prompt Design
• 5.2 Converting Individual Scores
• 5.3 Results
• 6 Hypothesis 3: Question Inference
• 6.1 Prompt Design
• 6.2 Quantitative Results
• 6.3 Qualitative Analysis
• 7 Related Work
• 7.1 Dataset Artifacts
• 7.2 MCQA Decision Making
• 8 Conclusion
• 9 Limitations
• 10 Ethical Considerations
• 11 Acknowledgements
• Appendix A Experimental Setup
• A.1 Dataset Details
• A.2 LLM Inference
• A.3 Prompt Creation
• A.4 Prompt Box Example
• A.5 Output Validity Statistics
• A.6 Metrics
• Appendix B Detailed Results
• B.1 Scaling Laws
• B.2 Decoding Strategies
• B.3 Partial-input Models vs Prompting
• B.4 Falcon Individual Evaluation
• B.5 One-Step Question Inference
• B.6 Qualitative Analysis
• B.6.1 Annotation Procedure Details
• B.6.2 Additional Results
• Appendix C Related Work on MCQA Evaluations
• Instructions for reporting errors