On this page

In March 1951, America's intelligence analysts told the government that a Soviet attack on Yugoslavia that year "should be considered a serious possibility". A few days later, a State Department official asked one of the authors a simple question: what odds did "serious possibility" mean?
The author, Sherman Kent, went back to the board that had agreed the sentence and asked each member. The answers ran from 20 to 80 against an attack, to 80 to 20 for one. The same three words, signed off by the same people, meant almost opposite things.
Studies in Intelligence · Vol. 8, No. 4 · Fall 1964
Words of Estimative Probability
Sherman Kent
“an attack on Yugoslavia in 1951 should be considered a serious possibility.”
01What Kent proposed
Kent spent the next decade arguing that an estimate is only useful if the reader takes from it the odds the writer meant. In 1964 he published a short essay with a table that tied each everyday word to a band of odds:

Not everyone liked it. Some analysts felt numbers claimed a precision nobody had; Kent called them the "poets", and himself one of the "mathematicians". But the idea outlived the argument. Today the US intelligence agencies write to a standard list, set in a directive from 2015: "very likely" means 80 to 95 per cent, "unlikely" 20 to 45, "roughly even chance" 45 to 55. A reader can hold the writer to it.
02The same problem in a business
Company reports are full of Kent's problem. "Sales look soft in the South." "There may be an issue with collections." "We're likely to hit the target." Each sentence sounds informed, and each lets the writer mean whatever turns out to be true. Two people read the same line and walk away with different pictures, and nobody notices until the quarter closes.
The fix is the one Kent found: every word that sounds like confidence has to stand for a number, and the number has to come from somewhere the reader can check.
03How Othor AI words its confidence
Every finding Othor AI reports carries one of a few fixed words, and each word is decided by code from a test, never chosen by the AI that writes the sentence around it:
- Measured means the claim is a computed fact: a value, a share, a gap. Nothing about cause is being inferred.
- Assessed · High means a statistical test found it, the chance of it being noise is below 1 in 1,000, and the effect is large enough to matter. Assessed · Medium means it passed the night's test for chance across every number tested. A low-confidence result is never put in a Morning Brief.
- Not determinable means the fact is measured but its cause can't be told from the data Othor AI reads. It says so, lists the possible reasons, marks the ones ruled out, and says what would settle the rest.
Above the list, Sweep says how much of the night could be chance: "at most 1 in 100 could be chance", across every test it ran.

04And when it looks ahead
Forecasts get the same treatment. Each one shows a likely range, the band the outcome should fall in four times out of five, and, where there is a goal, the chance of reaching it as a number. Its confidence word comes from its own record:

| Word | What it means |
|---|---|
| Reliable | At least three in four past periods landed inside the range, and the typical miss was 10% or less |
| Rough | Fewer than three in five landed inside, or the typical miss was over 25% |
| Fair | Everything in between |
So "Reliable" isn't a mood. It is a count of how often this forecast, for this number, has been right before, and you can open it and see each period it got right or wrong.
Kent would have recognised the goal, if not the machinery: a word in a report is a promise about odds. The least a reader is owed is to know which odds, and where they came from.