Microsoft built synthetic companies to test Copilot. Measure the realism gap

THE SIGNAL IN ONE SENTENCE

A company built for a test is a suspiciously cooperative place. The reporting lines make sense. The project names stay consistent. The meeting notes arrive on time. Nobody pastes a corrected spreadsheet into the wrong chat at 4:57 on Friday and then disappears for a long weekend. Real companies are less considerate. Microsoft Research published a look on October 30 at a system for evaluating Microsoft Copilot inside full-scale synthetic enterprise environments. The system is called the Industry Evaluation Data Generation Platform. The basic problem is real. Enterprise assistants are supposed to find decisions across meetings, summarize documents, connect work spread across teams and answer questions that depend on who did what and when. Testing that behavior requires a workplace with users, reporting structures, email threads, meetings, files, project histories and permissions. Actual workplace data contains private and sensitive information. Microsoft says no real customer data is used at any stage of its pipeline. Instead, the platform generates synthetic tenants, which are isolated organizational environments with their own users, data and configurations. The company describes three synthetic environments. One represents a pharmaceutical organization with clinical trial discussions, regulatory submission work and safety reviews. Another represents capital markets, including deal pipelines, compliance communication and portfolio reviews. The third represents an IT and software company with sprint planning, code review and product launch coordination. The simulated organizations are meant to change over time. Meetings accumulate. Email threads develop. Projects move. Teams reorganize. That temporal structure lets evaluators ask questions such as what changed since a previous review, rather than testing only one frozen document. Microsoft grounds the synthetic environments in public signals. O*NET occupational data informs roles, responsibilities and work activities. Online job postings inform job titles, reporting structures and department composition. SEC filings inform organizational scale, business units and financial context. Microsoft says its legal and policy organization reviewed and approved those public sources. The platform also generates evaluation questions. It begins with common workplace tasks, filters them for an industry, adds domain language and context, generates questions grounded in the synthetic tenant, rewrites them to sound more natural and validates them for accuracy and uniqueness. Microsoft evaluates the generated material for structural fidelity, contextual relevance, behavioral realism and content safety. It says the automated process replaced multi-day manual dataset creation, estimates savings of about $100,000 per dataset and reports $300,000 in total project savings. The company also reports a 39 percent gain in its internal composite quality measure over its initial generation approach. Some industry environments scored above 0.9 on an internal zero-to-one scale. Each environment now contains hundreds of users with complete communication histories. Those figures are company-reported. Microsoft did not publish the full datasets, grading rubrics, raw score distributions, independent replications or a direct comparison with failures observed in customer environments. Its claim that this is the first standardized evaluation baseline for Microsoft Copilot is also the company's description of its own internal system. The privacy benefit can be genuine while the realism claim remains unfinished. That is the signal. Synthetic companies can tell you whether Copilot works inside the company you generated. They cannot, by themselves, tell you how much that company resembles the one a customer actually inhabits. Teams need a realism-gap map. Start with structure. Compare the synthetic reporting hierarchy with aggregated, privacy-preserving observations from real deployments. Do teams have clean managers and departments, or matrix reporting, temporary squads, contractors, shared services and people whose directory record has not caught up with reality? Record the distributions, not just the average. A synthetic tenant with realistic team sizes can still be unrealistic if every employee has one manager, every project has one owner and every department boundary behaves like a wall. The map should show which structural features were generated, which public sources grounded them, which real-world distributions were used, how those distributions were protected and where the simulation deliberately simplified the organization. Then map communication. Real work contains incomplete threads, private channels, deleted messages, duplicate attachments, screenshots of documents, forwarded notes without context, silent readers, recurring meetings that drift from their title and decisions made in hallways that never enter the system. Behavioral realism cannot mean only that a synthetic email sounds like an email. It must include the missing reply, the contradictory reply, the reply that arrives after the decision and the person who uses the same acronym for two different things. Create tests for ambiguity. Give two projects nearly identical names. Let a policy change halfway through a quarter. Put a correction in a later message. Make the meeting transcript disagree with the polished summary. Allow a document owner to move teams while old permissions remain. Use regional date formats, mixed languages, nicknames, time zones and local shorthand. The objective is not to make the synthetic company chaotic for sport. It is to reproduce the kinds of friction that cause retrieval, summarization and grounding failures. Next, map access. An enterprise assistant does not see one complete company database. It sees whatever the current user is allowed to see through the product's indexes, permissions, connectors and synchronization state. The evaluation tenant should include inaccessible documents, expired links, delayed indexing, inherited permissions, guest accounts, restricted channels and resources whose visibility changes during the test. For every query, record both the correct answer and the answer the user is authorized to receive. Those are not always the same thing. A model that reveals the perfect answer from a restricted file has not passed. Neither has a model that confidently fills the gap after a permission filter removed the evidence. Then map workflow mess. The three Microsoft environments cover useful domains, but a realistic domain label is only the beginning. A pharmaceutical workflow includes regulated decisions, formal approvals and carefully controlled records. It also includes vendor files, late safety signals, revised protocols, regional requirements and disagreements over which version governs. Capital markets and software work have their own exceptions. Generate the happy path, then generate the handoff, reversal, exception, delay and partial failure. Test work that begins in one system and ends in another. Test a meeting that creates an action item, a message that changes it and a spreadsheet that quietly changes the number again. If every synthetic workflow reaches a clean final state, the assistant is practicing on a corporate brochure. Map time. Microsoft is right that time-aware questions require time-evolving data. The next step is to test temporal traps. Ask what the team believed before a correction. Ask what was approved as of a specific date. Ask for the current owner after two reorganizations. Ask for changes since the last quarterly review when the calendar contains two meetings with that title. Keep the event time, ingestion time, edit time and effective time separate. A document can be written on Monday, uploaded on Tuesday, corrected on Wednesday and become policy on Friday. The assistant should not flatten those four moments into one convenient timestamp. Map query realism. An automated question generator can produce thousands of fluent prompts. Fluency is not the same as representativeness. Compare generated queries with privacy-protected categories from real support tickets, user feedback and production traces. Do people ask direct factual questions, or do they start with half a thought? Do they name the correct project? Do they know which artifact contains the answer? Do they ask one task at a time? Microsoft's current agent-evaluation documentation says good prompts should be realistic, specific and grounded, and it recommends assertions that are atomic and verifiable. That is sensible evaluation design. It is also a reminder that synthetic prompts should be checked against actual user behavior. Build a query coverage table by user role, task, language, ambiguity, required source, permission boundary, time dependency and consequence of a wrong answer. Show which cells are synthetic, which are supported by production observations and which remain empty. Map the grader. The platform can know the synthetic company's ground truth because it created the company. That is powerful. It makes assertions concrete and regression tests repeatable. But the answer key can share the generator's assumptions. If the generator made one file the official source and the grader expects that file, the test may reward the assistant for following a tidy rule that workers do not actually follow. If generated employees communicate in a consistent style, a grader may miss the kinds of source conflict that require judgment. Use multiple grading methods. Check exact facts where exact facts exist. Check source attribution. Check whether the answer respected permissions. Check whether it identified conflicting evidence. Use human reviewers for ambiguous tasks. Track grader disagreement and require a review when the expected answer depends on a policy choice rather than a fact. Do not turn one model into the author, employee, user and judge of the same synthetic company without measuring the shared bias. Map the failures. Every production incident, corrected answer, support case and user complaint should enter a privacy-reviewed failure taxonomy. The taxonomy can describe the failure without copying customer content. Was the problem stale indexing, missing permissions, entity confusion, unsupported inference, false citation, omitted caveat, wrong time window, tool failure, language mismatch or an ambiguous request? For each category, show whether the synthetic baseline includes it, how often it appears, whether the grader catches it and when the test was last refreshed. This closes the loop. Synthetic data helps teams reproduce a failure without preserving the customer's private material. Production evidence tells teams which failures deserve reproduction. One without the other is incomplete. Microsoft's own Foundry guidance makes a similar distinction. Synthetic datasets are useful for prelaunch coverage, stable regression baselines and rare edge cases. Trace-based datasets reflect real production behavior. The guidance says using both provides the strongest evaluation signal. That pairing should be visible in the release gate. Before a Copilot change ships, run the stable synthetic suite to catch regressions. Run a privacy-reviewed sample of production-derived cases to catch drift between the model company and real work. Run security testing, user research, content-safety review and performance testing as separate disciplines. Microsoft's agent-evaluation documentation explicitly says evaluation does not replace those practices. Good. Keep that sentence near the scorecard. The realism-gap map can be a release artifact with six columns: The synthetic feature. The grounding source. The real-world signal used for comparison. The known simplification. The production failure categories covered. The next validation action. Add an owner and review date to every row. If a team cannot use customer content, it can still compare protected distributions, counts, failure labels and task categories. It can invite domain experts and users to review scenarios. It can conduct red-team exercises inside the synthetic tenant. It can publish test definitions and limitations without publishing private records. Privacy does not require pretending the gap is zero. The platform's strongest contribution may be operational. A reproducible synthetic tenant gives teams a common place to test product versions without waiting for a privacy exception or passing customer data around. A shared baseline can turn a reported bug into a permanent regression case. It can let separate teams compare results against the same organizational history. That is useful infrastructure. It becomes evidence of real-world reliability only when the bridge to real-world failure is measured. The neat synthetic office is not the enemy. It is the laboratory. Just label the distance from the laboratory to work.