NVIDIA open-sources NeMo agent optimization tools for trace analysis and evaluation authoring

NVIDIA has published open-source tooling under its NeMo framework to systematically diagnose and fix failures in AI agent deployments. The release centers on two components: NeMo Compass, which analyzes stored agent traces to identify recurring problems and prioritize fixes, and NeMo Eval Author, which generates custom evaluations in Harbor or NVIDIA NeMo Gym formats using coding agent skills. These tools are demonstrated through the NemoClaw blueprint, which runs a Hermes enterprise assistant inside an NVIDIA OpenShell sandbox and completes a full optimization cycle from trace analysis through harness modification to regression testing. The tutorial requires Ubuntu 24.04 with Docker, OpenShell 0.1.2, Harbor 0.22.0, and an NVIDIA API key, with optional integration of Codex or Claude Code for evaluation authoring.

The technical approach treats the agent harness—not just the model—as the primary optimization surface, examining prompts, tool descriptions, routing rules, memory, and control flow. In a Dassault Systèmes engineering example cited by NVIDIA, this loop eliminated 142 malformed tool calls and improved overall evaluation scores by 28% while reducing tool calls per run by 35%. The tools support Model Context Protocol (MCP) connections to enterprise systems including chat, email, files, and tickets.

NeMo agent optimization architecture and open-source components

• NeMo Compass signal processing: Analyzes JSONL trace corpora against behavioral definition files (ETHOS.md) using multiple signals including unusual behavior detection, tool usage patterns, and divergence from intended agent behavior, outputting structured backlogs with linked trace IDs.
• NeMo Eval Author evaluation generation: Provides coding agent skills and workflows to create Harbor 0.22.0 or NVIDIA NeMo Gym format evaluations, including verifiers that check specific criteria such as unauthorized tool invocations without human confirmation.
• Hermes Agent harness as optimization surface: The tutorial modifies system prompts including Hermes SOUL.md and tool execution rules rather than retraining models, with the complete harness code and configuration available through the NemoClaw blueprint repository.
• OpenShell sandbox integration: Runs the optimization cycle in an isolated environment using OpenShell 0.1.2 with MCP connectors to simulated enterprise services, enabling safe iteration on agent behavior before production deployment.
• Evaluation regression testing: The workflow reruns both development and held-out task evaluations after harness changes to verify improvements and detect regressions, supporting continuous integration for agent reliability.

Why it matters

These tools address a critical gap in agentic AI operations by moving beyond generic benchmarks to domain-specific, trace-driven improvement loops that independent developers can replicate. For the Linux and open-source AI community, the Harbor format support and OpenShell sandbox integration provide vendor-neutral evaluation infrastructure, though full independence requires scrutiny of whether NeMo Relay and the NVIDIA API key create runtime dependencies on NVIDIA's cloud services. The Dassault Systèmes engineering validation demonstrates practical enterprise applicability, while the GitHub-hosted tutorial and blueprint code enable community extension to alternative agent frameworks beyond Hermes.