Microsoft opened Scope for testing agent experience. Benchmark the product, not just the bot

THE SIGNAL IN ONE SENTENCE

A coding agent can fail because the model misunderstood the job. It can also fail because your documentation buried the command, your extension hid the state, your MCP server returned an awkward payload or your product demanded a permission nobody could discover. Those are different failures. Most leaderboards flatten them into one red cell. Microsoft's open-source Scope project is an attempt to keep the product in the experiment. The research-preview platform runs the same coding task across different agents, models, tools and product surfaces. A team defines what success means, saves the setup, watches the run and inspects the evidence behind the result. Scope is less interested in crowning a permanent smartest bot than in asking a more practical question: what happened when this particular agent met this particular product under this particular set of conditions? That shift sounds modest. It changes what a product team can fix. Suppose two agents are asked to add a health endpoint to the same service. One finds the correct documentation, installs the right package and passes the tests. The other edits the wrong file, loops on a permission error and eventually produces code that looks plausible but does not run. A normal scorecard says one passed and one failed. An agent-experience benchmark asks which setup each agent used, what model and version were attached, which skills and MCP servers were available, what the agent tried first, where it got stuck, what feedback it received and whether a product change altered the trajectory. Scope gives that investigation a reusable structure. First comes the task. Teams can submit a coding request against a versioned starting codebase. The task should describe the work, while evaluation criteria describe the observable result. That separation matters. If the prompt quietly contains the answer, the benchmark is testing obedience to a tutorial. If the criteria are vague, the final score becomes an argument about taste. Microsoft's documentation recommends atomic, observable and outcome-focused criteria. A check such as 'GET /health returns HTTP 200 with JSON' can be inspected. 'The code is high quality' leaves too much room for the judge to improvise. Scope arranges those checks inside ordered gates such as Requirements, Build and Test. Criteria within a gate can form a directed acyclic graph. A later check runs only when its prerequisites pass. If there is no package file, for example, the system can skip the checks that depend on that package instead of spraying misleading failures across the report. This creates a useful failure map. A run can stop at requirements because the agent misunderstood the task. It can stop at build because the product exposed the wrong dependency path. It can stop at test because the implementation missed an edge case. Those are not interchangeable red lights, and product teams should not repair them with the same tool. The second important object is the profile. A Scope profile bundles the worker type, model, agent version, MCP servers, skills and supported extensions. Profiles are versioned, and past runs keep the exact profile version they used. That means a comparison can pin the surrounding machinery instead of referring vaguely to an agent name that changes underneath the benchmark. This is a quiet improvement over the sentence 'we tested Claude' or 'we tried Copilot.' Which model? Which agent release? Which system instructions? Which tools? Which skills? Which permissions? Which operating environment? Which product documentation? A coding agent is a stack, not a mascot. Scope currently documents support for GitHub Copilot, Claude Code and a VS Code Copilot driver configuration. Its public demonstration lists OpenAI Codex and OpenCode as planned and Cursor as not integrated. That is a practical limit, not a footnote. A platform cannot support universal claims about agent experience while its adapter shelf remains incomplete. The third object is the evidence trail. Scope streams logs while a worker runs and can preserve generated files, workspace snapshots, criteria results, captured activity, reports and insights. Its API surface also includes downloadable run archives, tool-call records, trajectories, session recordings and snapshots. That evidence can turn a bad score into a buildable fix. Maybe an agent never discovered the CLI's help command. Maybe the documentation example used a stale flag. Maybe the MCP server exposed twelve similarly named tools and the model chose the wrong one. Maybe a permission prompt arrived after the agent had already committed to an impossible route. Maybe the agent completed the task but the automated judge looked for the wrong artifact. Each case suggests a different product change. The most useful experiment is often not Agent A versus Agent B. It is Product Version A versus Product Version B. Keep the task, codebase, agent profile and criteria fixed. Change one product surface. Rewrite the installation page. Rename the ambiguous tool. Add an example. Improve an error message. Package a skill. Then run enough repetitions to see whether the trajectory changes. That is closer to ordinary product research than a horse race. It also respects a fact that gets lost in model discourse: agents are users now, but people still own the product. An agent-friendly interface should not become a private back door around human comprehension. The command that helps a model should also help a developer understand what will happen. A machine-readable error should still be legible to the person on call. A skill that guides an agent through deployment should make its permissions, assumptions and rollback path visible. The goal is not to optimize the product for fluent automation at any cost. It is to reduce avoidable confusion while keeping human control intact. Scope is open source under the MIT license, and its documentation describes local development and Kubernetes self-hosting. The production architecture uses familiar infrastructure: a database for configuration and results, queues for dispatch, Redis for live events and blob storage for larger artifacts. Self-hosting is useful for organizations that cannot casually send code, logs or recordings to somebody else's benchmark service. It is not an automatic privacy certificate. The artifacts Scope can preserve are exactly the artifacts that may contain secrets, proprietary code, internal paths, customer data or authentication material. A serious deployment needs data classification, secret scrubbing, retention limits, access controls and deletion procedures before the first enthusiastic benchmark run. The judge deserves equal skepticism. Scope asks an automated judge to evaluate each criterion against the agent's output. That can scale review and produce consistent reports. It can also make mistakes. Microsoft's own documentation says results describe the tested tasks and configurations, not a universal agent ranking, and that important conclusions need human review. That warning should sit beside every chart. A judge may miss a subtle implementation bug. It may reward a superficial artifact that matches the criterion language. It may misread a skipped dependency as evidence about the agent. If the same model family helps write the criteria, performs the task and grades the result, correlated blind spots can travel through the entire pipeline. The practical answer is a human audit lane. Sample passing and failing runs. Re-execute the product. Inspect the changed files. Compare the judge's explanation with the actual criteria. Track reversals after human review. Keep disputed outcomes visible instead of quietly replacing them with the preferred label. Teams should also measure more than task success. Scope's public example highlights quality, tokens and duration. A production scorecard can add retries, tool calls, permission prompts, human interventions, destructive attempts, secret exposure, test quality and repair cost. A faster agent that burns twice the tokens and needs a senior engineer to untangle the diff may not be the better product experience. The benchmark should include the boring tasks too. Agents are often tested on clean greenfield builds because success is easy to recognize. Real products contain partial migrations, old configuration, contradictory docs, flaky tests and permissions inherited from three reorganizations. A useful suite needs installation, upgrade, rollback, debugging, accessibility, security and recovery tasks, not only the happy path. It also needs adversarial variations. Change an identifier. Remove network access. Rotate a credential. Feed the product an empty input. Start from a slightly older version. Put two plausible commands in the docs and see which one wins. The purpose is not to trick the agent for sport. It is to find where the product communicates by accident. Scope is still a research preview. Its repository, documentation and integration matrix will change. That makes it a promising instrument, not a settled standard. The deeper idea is durable. When agents use software, evaluate the relationship between the agent and the software. Preserve the setup. Define success before the run. Keep the trace. Change one thing at a time. Review the judge. Publish the limits. The bot may be the most charismatic part of the experiment. The product is where the team can usually do something on Monday morning.