AI2 releases XYBench benchmark testing LLMs on XY problem detection

The Allen Institute for AI (AI2) has released XYBench, a benchmark dataset of 8,115 real-world questions designed to test whether large language models can recognize and correct the XY problem—where users ask for help with a mistaken approach (Y) rather than their actual goal (X). Developed with researchers from Carnegie Mellon, NVIDIA, and Johns Hopkins, XYBench won Best Resource Paper at EMNLP 2026. The benchmark draws from Stack Overflow, Stack Exchange, manually curated examples, and AI-generated questions derived from WikiHow procedural guides.

Evaluation of five major models—Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro—revealed a consistent pattern: all models were more likely to answer the stated request than identify and address the underlying misconception. On Stack Overflow questions, models included solutions to the stated request in 75-86% of responses but solutions to the actual goal in only 59-71%, with primary focus on the actual goal in just 35-41% of cases. The benchmark code, data, and paper are publicly available through Hugging Face.

Benchmark design and model evaluation methodology

• Dataset composition: XYBench combines 1,557 Stack Overflow questions, 184 Stack Exchange questions, 102 manually curated examples, and 6,272 AI-generated questions from WikiHow guides, filtered to exclude cases where the literal request would sufficiently resolve the problem.
• Three-tier evaluation framework: Each response is scored on pragmatic solution (presence of X and Y solutions), pragmatic emphasis (which solution the answer focuses on), and misconception identification (whether the model redirects the user from Y toward X).
• Automated judging with human validation: Claude Sonnet 4.5 serves as the automated judge, with human reviewers validating 500 responses showing 97.5-100% agreement on solution presence, 84.6-93.3% on misconception identification, and 74.2-79.8% on pragmatic emphasis.
• Oracle experiment limitation: Supplying models with explicit knowledge of both X and Y improved solution inclusion rates modestly (Sonnet 4.6: 59% to 67% on Stack Overflow) but paradoxically reduced misconception identification (47% to 34%), suggesting models struggle to actively redirect even when the true goal is known.
• Consistent cross-model failure pattern: All five evaluated frontier models, including open-weight Qwen3.5-9B and DeepSeek-V4 Pro, showed the same bias toward answering the stated rather than intended request, indicating this is a fundamental architectural or training limitation rather than a vendor-specific issue.

Why it matters

XYBench provides the open-source AI community with a rigorous, reproducible tool to measure progress on a critical but under-tested capability: pragmatic reasoning in conversational AI. For developers building open assistants and for researchers training or fine-tuning open-weight models like Qwen and DeepSeek, this benchmark offers a concrete target for improvement that directly affects user experience in technical support, programming help, and everyday problem-solving scenarios. The public release of code and data through Hugging Face aligns with AI2's open research mission and enables independent replication without proprietary API dependencies.