Verification of published specifications
A model's context window is a single integer. It ought to be the easiest fact in this industry to get right, and it is one of the most reliably wrong. I check them one model at a time against the lab's own documentation, and I attach the source and the date to every figure.
Why this goes wrong
Model specifications are scattered across a dozen documentation sites in a dozen formats, several of them rendered by JavaScript and unreadable to anything simple. So catalogues are built from each other rather than from the labs, and a figure that is wrong at the top of that chain arrives downstream looking like consensus.
Every row comes from something the lab published about that exact model: its own documentation page, or for open weights the config file shipped with the model and the model card beside it. Not a sibling, not a family page, not another catalogue. Each row links to what it was read from.
One writes K and M and means decimal, one writes exact powers of two, one follows neither rule and is right anyway, and four never print an exact number anywhere this table could reach.
A value verified for a family and then asserted about one of its members is the error that is hardest to find later, because the wrong number now agrees with its neighbours.
Of the seventeen labs in this table, ten publish no maximum output figure anywhere this table has read them, which for most of them means the model index and every model page. That is not an omission to be filled in. It means any catalogue carrying a maximum output for those models is carrying a number that has no first-party source behind it, and no amount of field validation will ever say so, because the field is populated, well formed, and plausible. Checking that a value is present is easy. Checking that it came from somewhere is the work.
Field note ·
Two catalogues both carry a record called
openai/gpt-chat-latest. One publishes a context window of 400,000 tokens. The
other publishes 1,050,000. Neither of them is wrong.
chat-latestAn id ending in -latest is defined by recency, not by identity. It promises
you the newest model in a family and says nothing about which model that currently is. Two
sellers deciding that independently are under no obligation to keep deciding the same way.
Fifteen of these ids are carried by both catalogues. Twelve agree exactly. Three differ, and two of the three differ for a separate reason that has nothing to do with naming. So exactly one has drifted. The twelve are the part to be careful about: that agreement is a coincidence holding today, not a contract, and nothing announces the day it stops.
Move a workload between these two providers on the assumption that a model id is portable and you get a different model. Going one way your prompts are suddenly 2.6 times too long for the window and start failing. Going the other way they simply cost more than you budgeted. No field is malformed, no validator fires, and nobody involved has made a mistake.
Both catalogue records and both OpenAI pages were read again on 11 September 2026 before this was published. The catalogues are not named here, because neither of them is the problem.
Field note ·
Fifteen model ids left one catalogue’s published list over the last fourteen days. One of them came back. Of the fourteen still absent, every single one returns HTTP 200 today. Six return a record with an empty route list — the id is real, the response is well formed, and there is nothing in it to call.
It has to be asked, or the six above mean nothing. This card used to answer it from a
sample of twenty. On every id on the list
was read instead — all 456 — and
24 of them return an empty route list.
They account for completely: 18 are
alias ids of the …-latest kind, which is all
18 of the aliases on the list and is simply how
an alias is published; 6 are the
catalogue’s own router products, which pick a route per request and so have no single
endpoint to name. Ordinary third-party model ids:
0 of
432. For an ordinary id an
empty route list is not unusual, it is unheard of. For an alias or a router it is not a
signal at all.
A catalogue usually publishes more than one name for one model — a short id, a dated snapshot, a free-tier or batch variant. They need not answer alike. Five of these ids named a model that was being served, at full health, under the catalogue’s own canonical name for it: fourteen, nine, thirteen, thirteen and three routes. What had gone was one variant, not the model. Three others answer as a different model altogether, and say so in their own id field.
The check a careful team builds for itself is does my model id still resolve. One request, no dataset, and it looks like a health probe. On all fourteen of these it is green. Resolving and being served are different questions, and only one of them has an error code.
It is a description of one catalogue on one date, not a prediction about any id on it. A model served by one supplier is not in trouble; plenty of them are first-party routes from the lab that made the model. What the number says is how often there is nothing behind the one route — and that when a supplier stops, for whatever ordinary reason, there is no second route for the traffic to fall to.
A withdrawal is a change to what is offered, not to what answers, so it surfaces as an absence — and nothing in an API response is shaped like an absence. You find it by holding today’s list against an older one, which means having kept the older one. And having kept enough of them: the id that came back had been missing from twenty-four separate readings across eight days before it reappeared. A two-day window would have called it withdrawn. A two-week one would not.
A call that used to work returns a 200 with no completion in it. Whether that surfaces as an exception, an empty string, or a retry loop depends entirely on how your client handles a response it was not expecting — and that is not a decision anybody made on purpose.
Thirty-six readings of one catalogue’s published list between 5 and 18 September 2026; every departed id re-read on 18 September. Each run checks that an invented id returns 404 and that a listed id returns routes, and classifies nothing if either control fails. Two of the fourteen were first absent in today’s reading, so their absence is a single observation — which is the point of the paragraph above rather than an exception to it. Two figures in this note are later than 18 September, and both come from the same census — a reading of the route list of every one of the 456 listed ids on 23 September 2026, dated in its own card. The empty-route-list card was re-answered from it rather than from the sample of twenty it used to quote, and the 246 single-supplier ids are counted from it. Both carry the date of the census rather than the date of this note. The catalogue is not named, because nothing here is a mistake on its part: withdrawing a model is ordinary, publishing several names for one model is ordinary, and this is what both look like from outside.
Free and public
What each lab publishes for a model’s context window and maximum output tokens. Dated, with the page each figure was read from. Nothing is inferred, nothing is reconciled between labs, and where a lab publishes nothing the field is empty rather than guessed. It is free, it ranks nobody, and it is available as JSON.
| Model | Lab | Context | Max output | As published |
|---|---|---|---|---|
| Loading the reference… | ||||
There is a second table for a second kind of number: when each platform says a model stops working. 665 rows across eight platforms, read from the same kind of first-party page. A retirement date is the one field whose staleness breaks a caller’s code rather than just misinforming them, and it belongs to a model on a platform: the same weights are retired in one place months before another stops serving them, and neither publisher is wrong.
How the work is done
Each of these exists because I got it wrong once and wrote down what it cost. They are the whole method, and there is nothing else behind them.
A figure counts when it comes from the organisation that made the model. Another catalogue agreeing is not corroboration; it is usually the same number twice. Several unrelated copies agreeing against a first-party page is evidence about the copies.
Checking a family and asserting a member is the failure that survives review, because the fix makes the record consistent with its siblings and therefore invisible. Image variants in particular carry limits nothing like their text siblings.
The raw fetch is stored at the moment the finding is written, not afterwards. When a publisher acts on a report, the state that was measured stops existing, so a report backed only by a count has a shelf life, and the better the work goes the shorter it is.
Records checked and found correct are said to be correct, in writing, by name. A report that only ever contains hits gives you no way to calibrate the ones that are hits, and a finding I cannot stand behind is withheld even when it would read better.
Nothing is published about who got what wrong. No scorecards, no rankings, no naming. A public league table of other people’s errors would be worth less than it costs, and it would make every subsequent report an accusation rather than a favour.
The record so far, in full
That last number is on the page on purpose. Most organisations will not act on an unsolicited correction, and a page that lists only the wins is not worth reading. What the record does say clearly is that the ones who act decide quickly: no report has ever been picked up late. So the cost of finding out is one email and about a day.
Working together
Audits and ongoing verification are no longer offered. The project stopped on 8 Oct 2026. The reference and lifecycle pages stay up as they stood on that date and are not maintained.