Ai2 releases FairStress benchmark to test LLM fairness under adversarial pressure

Ai2 has released FairStress, an open benchmark that tests whether large language models maintain consistent, evidence-based decisions when challenged by persuasive arguments favoring different demographic groups. The benchmark expands 300 realistic scenarios across hiring, healthcare, lending, and other domains into over 13 million test cases using variations in identity presentation and adversarial prefixes. Testing across open-weight and frontier models including Olmo, Llama, Qwen, Gemma, Claude, and GPT revealed that models abandon correct choices more readily when pressured to favor minority-coded candidates, even when qualifications are held constant.

How FairStress measures hidden asymmetries

• FairStress contains 300 two-candidate decision scenarios across 29 domains, expanded to 13M+ test cases through systematic variation of demographic identity presentation and matched adversarial prefixes.
• Identity presentation varies across four levels: none, demographically coded name, indirectly implied identity, or explicit demographic label.
• Adversarial prefixes draw on authority, social consensus, purported evidence, legal concerns, and historical disadvantage; each has a matched version arguing for the opposing candidate.
• FairStress-Core provides a 50K-item subset using item response theory for faster, less expensive evaluation while retaining discriminative power.
• The benchmark is integrated into Olmo-eval, Ai2's open evaluation workbench.

Why it matters

FairStress addresses a gap in standard fairness benchmarks that only check final outputs, potentially influencing how organizations evaluate models for high-stakes deployment. The open release of data, code, and integration with Olmo-eval may encourage broader community scrutiny of model behavior under adversarial conditions.