Microsoft open-sources Scope, a self-hosted tool for testing AI coding agents

Microsoft has open-sourced Scope, a self-hosted testing framework for AI coding agents. The tool runs identical tasks through different agents—such as GitHub Copilot and Claude Code—and scores their output using criteria defined as a dependency graph. An LLM-based Judge service evaluates results, with full traceability through logs, workspace snapshots, and tool calls. There is no hosted version; users must deploy it locally with Docker, Node 22, and other dependencies, or on Kubernetes via Helm. Contributing requires accepting the Microsoft Contributor License Agreement.

How Scope evaluates agents

• Task-driven evaluation: Users define a task, optional starting codebase, and agent configuration; Scope drives agents through the workflow
• Dependency-graph criteria: Success conditions chain together—one check can gate another, such as verifying a GET /health endpoint returns JSON after building a Node.js server
• LLM-based Judge: An automated scoring service evaluates runs against criteria, though Microsoft warns it can err and important findings need human review
• Full traceability: Each run preserves live logs, generated files, workspace snapshots, and tool calls for debugging failed criteria
• Versioned profiles: Agent, model, skills, MCP servers, and extensions are saved as profiles for controlled A/B comparison
• Repeatability: Built-in repetition helps distinguish flaky one-off failures from systemic problems
• Deployment options: Local Docker Compose setup or Kubernetes via Helm chart; requires GitHub Copilot entitlement for default worker and model provider credentials for Judge

Why it matters

Scope gives SDK maintainers, documentation teams, and CLI developers a reproducible way to measure whether AI agents can actually use their tools correctly—shifting from anecdotal reports to structured evaluation. However, the self-hosted requirement and Copilot dependency limit immediate accessibility for individual developers.