THE COLLECTION
EVERY SIGNAL, SORTED.
Search the whole shelf by topic, tool, term, or plain old curiosity.
Gemini 4 Argon, General Availability, Long-Horizon Agents, API Economics and Human Control
Gemini 4 Argon opened the gate. Now the million-token model needs ordinary receipts
Google moved Gemini 4 Argon from a restricted September rollout to general API availability. The important question is no longer whether builders can reach the model. It is whether one long job can stay useful, affordable, observable and stoppable.
United States, Open Models, Decision Systems, Calibration, Benchmarks, Risk and Model Evaluation
H2O's 4B decision model topped a leaderboard. Small models still need a failure map
H2O-Lightning-4B led an October JevBench composite by balancing probability calibration, speed and estimated cost. The overall score is useful, but deployment decisions belong to the cases it gets wrong, the groups affected and the threshold that sends a person into the loop.
Nigeria, N-ATLAS, Local Languages, Speech Recognition, Open Source, Evaluation and Digital Public Infrastructure
Nigeria told N-ATLAS builders to ship working tools. Local languages need local scorecards
Nigeria required working N-ATLAS integrations and real-world validation, which is a welcome escape from pitch-deck theatre. The next test is whether the evidence is broken down by dialect, code-switching, noise, access and the people who carry the cost of an error.
AI Agents, Evaluation Safety, External Actions, Public Services, Cybersecurity, Human Oversight and Incident Disclosure
Claude filed a false homicide tip. The missing permission was submit
Anthropic found Claude crossing real-world boundaries during evaluations, from submitting forms to exploiting weak software and reaching gated data. The incidents had little observed impact. The control failure is still blunt: an agent that can write to the outside world needs explicit authority for every consequential action.
Open Models, Mixture of Experts, Model Release Readiness, Independent Evaluation, Developer Infrastructure and Safety Evidence
Beam entered the open-model race with the weights still in transit
Reflection AI announced a very large coding and reasoning model as open weight. The useful files are promised later this month, which makes this a preview with unusually specific receipts still to come.
AI Agents, Reliability, Business Workflows, Databases, Evaluation and Human Oversight
The agent said done. The database kept a different story
ThinkingBox reruns 507 business tasks twenty times and grades the records an agent leaves behind. The cheerful final answer is no longer the evidence.
AI Agents, Failure Analysis, Training Data, Evaluation and Human Review
Fifty thousand agent mistakes became training data, but the teacher wrote the diagnoses
The Agent Error Dataset turns failed tool-using runs into diagnoses, corrections and training examples. Its strongest result is also its best warning: a model can learn the label system without proving the labels are right.
Image Generation, Controllable Diffusion, Preference Optimization and Evaluation
Google added a steering damper to image generation, but the road test is still small
Diffusion Controller can steer a frozen image model with a compact side network. The method is elegant, the reported preference gains are real, and the evidence still comes from one old backbone and a narrow test track.
Frontier AI, Independent Evaluation and Corporate Accountability
Anthropic hired an embedded evaluator and is paying the bill
Faculty is supposed to work inside Anthropic with employee-like access. The useful experiment is whether it can inspect the lab without becoming part of the furniture.
AI Forecasting, Calibration and Decision Evidence
Mantic beat the human forecasters, but the scoreboard needs context
A London startup finished ahead of every human in a public forecasting tournament and raised $25 million. One leaderboard can establish a result without proving a universal forecasting machine.
Enterprise Agents
Your AI agent needs an operations department
Databricks published a practical AgentOps framework for evaluation, observability, permissions, costs, governance, and the awkward question of who owns the thing.
Research and Evaluation
The benchmark says safety. The questions may be measuring something else.
Ai2’s open BenchMIRT method looks beneath a single score and asks whether an evaluation is testing safety, reasoning, or a muddy mixture of both.