Test What Happens When the Model Stops Answering
On September 3 I cut FDAgent's model picker to six, and a test ran each model through the product. One provider's outage came back looking like a finished answer. The bug was in my code, and the fix was to give a model error its own exit.

On September 3 I cut FDAgent's model picker to six models. The same day, running each of them through the product found a bug: when a model stopped responding partway through a run, FDAgent could present its last sentence as a finished answer.
A model picker looks like a feature list. It's also a list of failure modes. Every model on it can be slow, wrong, rate-limited or down on the day a user picks it, and each of those failures has to look right in the product. Most model testing checks the answers. This bug lived in what the product does when there isn't one.
Launch posts are hypotheses
Between August 26 and September 2, three more candidates arrived. On August 26 Z.ai released GLM-5.3-Flash and called it "a highly competitive default choice for a broad range of workloads." On September 2 Google shipped Gemini 3.8 Flash and Meta shipped Muse Spark 1.3. Every launch post makes the case to be your default.
FDAgent lets you "ask about drugs, devices, trials, inspections, and enforcement in plain English, then open the record behind every claim." So its eval measures facts. It runs the production agent, with the same runtime and tools users get, on regulatory questions, and scores what share of the expected facts show up in each answer.
The September 3 change made GLM-5.3-Flash the default after it passed the CI smoke set and beat the old default on it. That's a small sample, and a weekly run on the production runtime is there to tell me if the call was wrong. The same change cut the picker to six: GLM-5.3-Flash, two variants of GPT-5.6 Luna, xAI's Grok 4.6, Gemini 3.8 Flash and Muse Spark 1.3. All six passed a live call through the gateway before it shipped, and any model I retire falls back to the default, so a user's saved choice can't break their session.
A live call proves a model answers. It doesn't prove the product does the right thing when the model stops answering, and that's the test that found the bug.
The model that answered without answering
Then a six-model smoke test ran each of them through the app itself, one case per model. Five returned every expected fact. Grok 4.6 scored zero, three runs in a row.
The logs said why. Every Grok message ended on an error stop: xAI's API was returning 503, and the model never answered. A direct call through the gateway returned 503 too. That part was upstream availability, not my configuration. A model that passed its live check that morning was down within hours.
The bug was what FDAgent did with it. The run showed as completed, and its "answer" was the model's opening sentence, a line announcing what it was about to research.
The cause was one missing check. The runner that drives the agent recorded each message as it ended but never looked at why it ended. It exited cleanly, so the route took the last thing the model said as the final answer and showed no notice.
A model error gets its own exit
The fix gives a model error its own exit path. The runner now records the last stop reason and error message. If the agent ends on an error, it writes the provider's message out and exits with a dedicated code. The route's existing salvage path takes it from there. If there's a usable partial answer, the run shows as partial, with a notice that the model stopped responding before the research finished and that the user can try again or pick another model. If there isn't, the run fails with the provider's message. It is never shown as completed.
It shipped the same day, with a new test that decodes the runner script FDAgent actually ships and checks the stop-reason handling, on top of an existing suite of more than 1,800 tests.
Test the failure path, not just the models
That bug wasn't Grok's. Any model on the list could hit a provider outage, and any of them could have produced the same fake answer, so the fix lives in the runner, not in any model's configuration.
It also took luck to find. Only one model happened to be down that day, and a product run across the list happened to include it. The new runner test removes the luck for this failure: it checks the stop-reason handling on every build, whether or not a provider is down.
The eval's score told me a run had gone wrong. It took the logs to show that the product had called a failure an answer, and only a test aimed at the failure path keeps that from coming back.
