IBM's ALTK-Evolve tops customer service benchmark with learned memory, no frontier model

IBM Research introduced ALTK-Evolve, a method that extracts behavioral guidelines from agent trajectories and appends them to system prompts. Using only GPT-4.1—a non-reasoning model—the system reached 67.5% Pass^4 on τ²-bench retail customer service tasks, topping the leaderboard. The approach requires no weight updates or human annotation. Performance gains varied by domain: retail benefited from all 292 learned guidelines, while airline tasks peaked with a smaller 14-guideline subset.

How ALTK-Evolve builds and applies learned memory

• Guideline extraction: GPT-4.1 analyzes training-task trajectories to distill successes and failures into short behavioral rules (e.g., authentication procedures, confirmation requirements)
• No weight updates: The same GPT-4.1 model handles both guideline extraction and agent inference; no fine-tuning or human annotation is used
• Prompt injection: Learned guidelines are appended as a markdown list after the domain policy in the system prompt
• Support-based filtering: Guidelines carry support counts (number of training tasks they generalized across); configurations tested include baseline (0 guidelines), sup3 (support ≥3), and lossless (support ≥1)
• Retail results: 292 lossless guidelines boosted Mean Accuracy from 78.1% to 82.5% and Pass^4 from 57.5% to 67.5% (+10.0 pts)
• Airline results: 14 sup3 guidelines achieved best reliability (Pass^4 45.0% vs. lossless 40.0%), showing domain-optimal guidance dosage varies

Why it matters

The result challenges the assumption that frontier models are necessary for top benchmark performance in agent tasks, suggesting prompt-level memory optimization as a viable alternative to expensive model upgrades or fine-tuning pipelines.