Google built a benchmark for visual office work. Keep the artifact in the grade

THE SIGNAL IN ONE SENTENCE

A polished final answer can hide an awful lot of unfinished work. The number may be right while the spreadsheet formula is broken. The slide may contain the right facts while the labels overlap. The report may cite a source that never supports its claim. The architectural markup may look convincing until somebody notices that the opening on the elevation does not match the floor plan. Google DeepMind and Artificial Analysis launched the Multimodal Knowledge Agent benchmark, or MKA, on November 10. It is designed to test agent work across visually complex professional files instead of reducing the job to a text question and a text answer. The private evaluation contains 200 tasks across 10 industry verticals and 83 occupations. Google says domain experts created tasks for fields including construction, engineering, finance, healthcare, law and government, logistics and mapping, science, software and data, education and business. Each task begins with a realistic prompt and a bundle of files. Those inputs can include architectural drawings, engineering schematics, clinical scans, tables, charts and other visually dense material. The requested deliverable is also real: a spreadsheet, presentation, report, text file or marked-up image. Google says a seasoned professional would normally spend three to eight hours on each task. The average expert-written rubric contains roughly 70 yes-or-no checks. Google describes the overall rubric weighting as 75 percent factual accuracy and 25 percent visual design. Artificial Analysis runs every tested model inside the same open-source Stirrup agent harness. Its benchmark page says the agent receives a Linux sandbox, code execution, web search, page fetching and an image tool where supported, with a limit of 250 turns. The deliverable does not get graded by reading the agent's victory speech. An agentic judge opens the submitted files, renders their pages, inspects spreadsheet formulas and checks the artifact against the task rubric. Artificial Analysis says it uses GPT-6 Luna at medium effort as the judge. Google reports that domain experts agreed with the automated grader 96 percent of the time when they graded model outputs themselves. The 200 scored tasks remain private to reduce leakage into training data or search results. A separate set of 10 public tasks is available on Hugging Face so people can inspect the format and try the harness. The headline score is Climb Weighted Satisfaction Rate. Artificial Analysis divides rubric criteria into harder Climb checks and baseline Regression checks, then calculates the weighted share each model satisfies. On the launch leaderboard, Artificial Analysis reports Claude Fable 5.2 at 57 percent Climb WSR, Gemini 4 Carbon at 54 percent and Claude Opus 5.5 at 53 percent. Google highlights that Gemini 4 Carbon ranked second in quality while completing tasks at about one-sixth the average cost of the top-scoring model. Those are launch results, not a law of nature. Prices, model versions and harness behavior can change. Google helped create the benchmark and is using the result to market Gemini 4 Carbon. Artificial Analysis says it conducts the evaluations independently, but its agentic judge is still a model, and the private test set cannot be fully audited by the public. That is the verified reporting. The signal is bigger than the leaderboard. MKA treats the finished file as part of the answer. That sounds obvious until you look at how many AI evaluations stop at extracted text, a multiple-choice response or a judge reading the final paragraph. Real knowledge work ends in objects other people must open, edit, present, sign, calculate with and sometimes defend. Keep the artifact in the grade. Start with factual fidelity, but do not stop there. Every consequential claim should trace to a source inside the input bundle or a permitted outside source. A report needs citations attached to the specific statement they support. A presentation needs labels that remain legible when projected. A marked-up drawing needs callouts placed on the correct feature, not merely near it. For spreadsheets, inspect formulas as executable logic. Check whether ranges include the correct rows, whether units match, whether dates align and whether copied formulas shift correctly. Flag hard-coded values where a formula was expected. Recalculate the workbook in more than one compatible engine when the file will travel across tools. The visual grade should measure function, not decoration. A beautiful slide that hides the exception is worse than an ugly one that makes the decision clear. A chart needs readable axes, honest scales, distinguishable colors and a source. A report needs headings, page breaks, captions and tables that survive export. Accessibility belongs inside this score, including reading order, contrast, alternative text and keyboard usability where the format supports it. Editability matters too. A professional artifact is rarely the end of the conversation. A spreadsheet should keep formulas, named ranges and data types rather than flattening everything into pixels. A presentation should use editable shapes instead of one giant image. A report should preserve headings and tables. A marked-up drawing should keep layers or annotations that a reviewer can isolate. Grade the handoff. Can the next person tell what changed, what still needs judgment and where the evidence lives? Can they revise one assumption without rebuilding the whole file? Can they compare the model's output with the expert example without proprietary software or missing fonts? Then count the hidden cleanup. An agent may submit a file that earns a decent rubric score while leaving a human to repair formulas, resize charts, resolve citations, fix pagination, replace inaccessible colors or recreate an export. That labor is part of the result. Record minutes to first usable draft, minutes to verified delivery and minutes of expert correction. The shortest generation time can still produce the longest day. Keep two grades when the job demands it. One grade should measure the artifact exactly as submitted. The other should measure the artifact after a qualified person performs a bounded review. The distance between those scores is the repair gap. That gap tells a team more about deployability than a single average. Preserve failure examples, not only the total score. If a model consistently loses footnotes, breaks formulas or invents labels, publish those patterns by file type and occupation. A model that works well on business presentations may still fail construction drawings or clinical images. Artificial Analysis already reports results by vertical and output type. The next useful layer is a library of recurring failure modes with sanitized artifacts people can inspect. Test the judge too. The 96 percent agreement figure is encouraging, but agreement is not the same as complete validity. Report which criteria produced disagreement, whether the judge missed subtle formula errors, how often layout rendering changed the result and whether some occupations were harder to grade consistently. Use at least three kinds of checks. Deterministic checks should validate formulas, file structure, required fields and other machine-testable constraints. A model judge can inspect semantic and visual criteria at scale. Domain experts should review high-consequence work, disagreement cases and a statistically useful sample of ordinary runs. Do not let the model judge see the benchmark owner's preferred narrative. Blind model names and provider metadata where practical. Randomize presentation order. Regrade a sample with a second judge and with people. Publish confidence intervals and criterion-level agreement instead of one comforting percentage. The private set creates a necessary tension. Keeping tasks secret reduces contamination and search leakage. It also limits outside scrutiny. The public set should therefore do more than offer ten nice demos. It should include the schemas, rubric structure, reference rendering process, file validators, scoring code and representative failures needed to challenge the method. Rotate private tasks on a published schedule. Retire a portion into the public set. Record which models may have seen related material during training. Use canary files or synthetic identifiers to detect leakage. When an evaluation company reruns a model, preserve the exact model version, API settings, harness commit, tool versions, system instructions, turn limit, runtime, cost and every submitted file. Cost needs a receipt as well. Artificial Analysis says its cost metric includes calls to the model under test and excludes judge calls. That is a reasonable comparison for inference prices, but it is not the full cost of a usable workflow. Add tool compute, file conversion, retries, failed runs, judge calls, expert review and repair labor. Show both model-only cost and end-to-end delivery cost. Measure reliability across repeated runs. One beautiful artifact can be luck wearing a blazer. Run the same task several times, vary file order and introduce harmless formatting changes. Track whether the model returns the required file, whether scores swing and whether the failure is graceful enough for a person to recover. Security belongs in the artifact grade when agents browse, execute code and open files. Test malicious instructions hidden in documents, unsafe macros, external links, embedded objects and files that imitate trusted templates. The best artifact is not useful if creating it leaks data or executes something the reviewer never approved. The benchmark is a meaningful step because it moves evaluation closer to the place where professional work becomes consequential. It asks whether an agent can look at the messy evidence and return an object that survives inspection. Now keep pushing the grade toward the whole job. Open the file. Recalculate it. Render it. Edit it. Trace every claim. Count the cleanup. Test the judge. Price the repair. Let a domain expert say where the machine still fumbles. The chat response can say complete. The artifact gets the final word.