Don't Ship a Number Your Model Didn't Earn
At the GPT-6 Astra hackathon I built Approval Radar in a day. Its per-drug approval probability is always empty, because FDA's public data can't support one, and the held-out tests show what it can support.

Approval Radar, which I built at the GPT-6 Astra hackathon, has a field for a per-drug approval probability. It's always empty.
OpenAI released GPT-6 Astra on September 3. On September 10 I spent the day at the GPT-6 Astra Hackathon at OpenAI's New York office, where everything had to be built that day and submitted as a public repository.
I picked a domain I already know from FDAgent: FDA drug approvals.
Approval Radar
Approval Radar follows 33 real drug review episodes across 29 companies, with 98 curated primary sources. It starts from public records: ClinicalTrials.gov, Drugs@FDA, drug labeling and PubMed. For each drug, Astra reads trial results and FDA documents and turns the evidence into an approval outlook, a decision brief and specific diligence questions.
In one run, Astra was challenged with a reassuring FDA inspection, noticed that it covered clinical-research practices rather than manufacturing, and kept its outlook on the drug's manufacturing concern unchanged.
The number it doesn't show
The obvious feature for an approval radar is a per-drug probability. Drug X, 73% likely.
There isn't one, and the reason is the data. FDA publishes its review outcomes as aggregate tables, so repeated applications and common sponsors can't be separated by drug. The archive of complete response letters, where FDA explains why it didn't approve an application in a given cycle, is selectively disclosed. A per-drug model built on that record would look more confident than it is.
That alone rules out a per-drug number. What I could test was what the public data supports at the cohort level, on a split built so it couldn't flatter the model. Training came from FDA's FY2019 report, covering applications received in fiscal 2013 to 2017. The held-out years, 2021 to 2023, were labelled from the FY2025 report, and the training source was checked to predate the first test cohort. Source PDFs are pinned by hash, so a changed source fails verification until someone reviews it.
Two questions got two different answers:
- Do review classes give useful reference rates? As history, yes. Across three held-out years and two review classes, six annual first-cycle rates in all, class means missed by 6.9 percentage points on average, against 11.6 for a single pooled rate. Priority and standard reviews approve at very different rates, so that's a sanity check on six observations, not a forecasting model.
- Can class predict whether FDA acts on an application by its goal date? No. On a separate held-out set of 603 actions, the class model's Brier score was 0.0390 and the pooled rate's was 0.0383. The simpler model did slightly better.
The second test isn't the approval question, since an on-time action can be a rejection, and with nearly every action landing on time, a single base rate is hard to beat. It was the one per-application question the public numbers could grade, and class information didn't help there either. The reason there's no per-drug number is the structure of the data, not this score.
The README is just as strict about what it can't claim. Because the tables are aggregates, it says outright that a validation claim separated by drug "would be false."
Unknowns stay unknown
The rest of the build keeps the same line between what it found and what it didn't. The evidence map separates retrieved records, searches that came back empty, searches that failed and sources never checked. And when one drug's evidence packet turned out thin, its favorable label was withheld after review instead of being quietly edited.
The field stays empty
So the per-drug probability field is null. The outlook Astra gives is a qualitative judgment, labelled as one, next to the evidence, the open questions and cohort reference rates labelled as history, not prediction. The README lists what a real drug-level model would need: a complete cohort of application cycles, timestamped feature snapshots, reliable records of unsuccessful and pending outcomes, and calibration measured going forward. Until a model has those, the field stays null.
