# Microsoft open-sources Scope, a self-hosted tool for testing AI coding agents

> **Key takeaway:** Microsoft released Scope under the MIT license to let developers evaluate and compare how different AI coding agents perform on the same tasks.

**Published:** 2026-10-20T08:13:49+00:00 
**Source:** It's FOSS Community News 
**Category:** distro-news 
**Canonical URL:** https://fosswire.org/news/microsoft-open-sources-scope-a-self-hosted-tool-for-testing-.html 

## Summary
Microsoft has open-sourced Scope, a self-hosted testing framework for AI coding agents. The tool runs identical tasks through different agents—such as GitHub Copilot and Claude Code—and scores their output using criteria defined as a dependency graph. An LLM-based Judge service evaluates results, with full traceability through logs, workspace snapshots, and tool calls. There is no hosted version; users must deploy it locally with Docker, Node 22, and other dependencies, or on Kubernetes via Helm. Contributing requires accepting the Microsoft Contributor License Agreement.

## How Scope evaluates agents
- **Task-driven evaluation**: Users define a task, optional starting codebase, and agent configuration; Scope drives agents through the workflow
- **Dependency-graph criteria**: Success conditions chain together—one check can gate another, such as verifying a GET /health endpoint returns JSON after building a Node.js server
- **LLM-based Judge**: An automated scoring service evaluates runs against criteria, though Microsoft warns it can err and important findings need human review
- **Full traceability**: Each run preserves live logs, generated files, workspace snapshots, and tool calls for debugging failed criteria
- **Versioned profiles**: Agent, model, skills, MCP servers, and extensions are saved as profiles for controlled A/B comparison
- **Repeatability**: Built-in repetition helps distinguish flaky one-off failures from systemic problems
- **Deployment options**: Local Docker Compose setup or Kubernetes via Helm chart; requires GitHub Copilot entitlement for default worker and model provider credentials for Judge

## Why it matters
Scope gives SDK maintainers, documentation teams, and CLI developers a reproducible way to measure whether AI agents can actually use their tools correctly—shifting from anecdotal reports to structured evaluation. However, the self-hosted requirement and Copilot dependency limit immediate accessibility for individual developers.