Microsoft open-sources Scope, a Kubernetes-based evaluation platform for AI coding agents

Microsoft has open-sourced Scope, a platform built on Kubernetes for evaluating AI coding agent behavior against real products. The tool emerged from internal work by Microsoft Developer Relations that began over a year ago. It is designed to support a continuous evaluation loop: give an agent a task, inspect its interactions, improve the product or guidance, and re-evaluate. The Azure Cosmos DB team used Scope to compare bare models with its Agent Kit across eight end-to-end scenarios in five languages, leading to 20 Agent Kit issues and 27 documentation issues being resolved.

How Scope structures agent evaluation

• Scope runs on Kubernetes and is available under the MIT License
• Evaluations are designed to be repeatable experiments rather than one-off demos or single CI tests
• The platform addresses variability from models, agent runtimes, skills, plugins, MCP servers, and custom instructions
• Three evaluation focus areas: documentation guidance, SDK discovery and correct adoption, and tooling/task completion
• Azure Cosmos DB case study: compared bare models vs. Agent Kit across 8 end-to-end scenarios in 5 languages
• Specific measurable outcome: after one Agent Kit update, hierarchical partition-key adoption went from 47% to 100%, OFFSET/LIMIT usage dropped from 47% to 0%

Why it matters

Scope addresses a gap in how open-source projects and product teams evaluate their software as AI agents become a new class of user. For SDK and API maintainers, it offers a structured way to discover where agents fail and whether documentation or code changes fix those failures. The Cosmos DB example suggests the tool can produce concrete, measurable improvements in agent behavior.