SABINE HOSSENFELDER: Experts have underestimated AI capability for years, across mathematics, cybersecurity, and other fields

The Gist

Sabine Hossenfelder says this is not new. Experts in 2022 thought AI would win Olympiad gold in 2030, and it did so in 2025; similar early arrivals happened in cybersecurity and biology tests. Her 2023 example, AI proving top-journal theorems, has not clearly been met yet, and the forecasters' own review warns its checks tend to catch underestimates first. This packet steelmans the argument Sabine Hossenfelder makes in this video, in its best light, for analysis; it is not an endorsement of her conclusions.

Conclusion

Underestimation of AI capability is a recurring pattern in forecasts made since 2022, across different expert groups and fields, and is not explained by any single surprise such as the release of ChatGPT; it is a systematic tendency that users of such forecasts should correct for.

Premises

1. In October 2023, 2,778 AI researchers put even odds on AI routinely and autonomously proving theorems publishable in top mathematics journals about 22 years later.
2. In 2022, the median expert in a forecasting tournament expected AI to reach gold-medal level at the International Mathematical Olympiad in 2030, and AI reached that level in July 2025.
3. Forecasts of AI's autonomous cybersecurity ability have been overtaken: in 2024 language-model agents exploited known vulnerabilities from their descriptions, and a benchmark threshold experts expected in 2028 was reached in February 2026.
4. A biosecurity benchmark that experts expected AI to match in 2030 was likely matched in April 2025.
5. The misses span forecasts made before and after ChatGPT's release and by different expert groups, which points to a systematic tendency rather than one surprise.

Assumptions

• The 2023 milestone required AI to routinely and autonomously prove theorems publishable in top mathematics journals, including choosing the theorems; a flood of low-value AI-written preprints does not meet that bar, and the strongest candidate, the Navier-Stokes claim, is still under review.
• No specific 2023 expert forecast that AI could not autonomously find and exploit vulnerabilities 'for years' was located; 2024 research showed language-model agents exploiting 87 percent of known, already-disclosed vulnerabilities when given their descriptions but only 7 percent without them, and Google's first AI-found exploitable bug in late 2024 was described as highly experimental, so the 2024 mark was partly reached rather than fully.
• The forecasting group's own review warns that interim accuracy checks are biased toward catching underestimates, because a forecast can be shown too low as soon as reality passes it but shown too high only after its deadline, and most of its AI questions are unresolved.

Analysis

Argument type: Inductive.

Potential Fallacies

• Hasty generalization / unrepresentative sample (Premises 2-5 and Conclusion) — A small, selected set of resolved benchmark cases is generalized to a 'systematic tendency' across fields and expert groups. No sampling method is given, and forecasts that were accurate or too optimistic (e.g., self-driving timelines, radiology replacement, agent hype) are not counted. The stated interim-check bias (A3) makes the sample tilt toward underestimates even more.
• Selection / resolution-asymmetry bias (Premises 2-4 with Assumption 3) — Forecasts that are too low can be shown wrong as soon as reality passes them, while forecasts that are too high can be shown wrong only after their deadlines. The set of visible misses is therefore skewed toward underestimates. The argument acknowledges this but does not estimate how much of the pattern survives the correction.
• Treating an unresolved forecast as evidence of a miss (Premise 1) — P1 accurately reports a 22-year forecast whose milestone, per A1, has not been met. It records what experts said, but it cannot yet be scored as an underestimate. Placing it alongside resolved cases borrows their weight without equivalent evidence.
• Counting correlated evidence as independent (Premise 5) — Misses in math, cyber, and bio may share one driver, such as the 2024-25 shift to reasoning models and reinforcement learning on verifiable tasks, and the forecasting communities overlap. Cross-domain consistency is therefore compatible with one hard-to-predict technical shift rather than a stable forecaster bias, and P5 does not address this rival explanation. The 'single surprise' it rules out (ChatGPT) is a weaker alternative than the common-cause one.
• Benchmark-to-capability equivocation (Premises 2-4, title and Conclusion) — The evidence consists of narrow benchmark thresholds (competition problems, multiple-choice troubleshooting, exploit benchmarks), while the title and conclusion speak of AI 'capability' generally. The cyber result (87% with descriptions vs 7% without) and the hedged 'likely matched' in P4 show how far benchmark success can sit from real-world competence.
• Partial support presented as full support / weak baseline (Premise 3) — For cybersecurity, A2 concedes that no specific contrary expert forecast was located and the 2024 mark was only partly reached. Only the 2028 to February 2026 benchmark comparison is a clean forecast-versus-outcome pair, so 'overtaken' and the title's 'for years' are stronger than the documented evidence.
• Is-ought gap / unspecified correction (Conclusion) — 'Users should correct for it' requires knowing the direction, size, and domain-specificity of the bias and that it will persist. None is estimated. A bias seen in a fast-scaling period may shrink or reverse as forecasters update, so a blanket correction risks overcorrection.

Counterarguments

These challenges are not ranked.

• Conclusion (High impact) — The cases are a non-random sample from a single era of rapid progress. Experts have also overestimated AI progress (autonomous driving by ~2020, radiologists obsolete within five years, repeated 'year of the agent' predictions, slipping AGI dates). Without a denominator of all forecasts, the data cannot distinguish a systematic low bias from a few salient misses in verifiable, benchmarkable domains.
• Premises 2-4 (High impact) — The authors of the cited review concede that interim checks catch underestimates earlier than overestimates, and most AI questions are unresolved. Resolved misses are therefore over-represented by construction, and the argument gives no estimate of how much of the pattern survives this correction.
• Premise 5 (High impact) — A single shared driver, the late-2024 shift to reasoning models and reinforcement learning on verifiable-reward tasks, explains correlated misses in math, code-based cyber tasks, and exam-style expert tests. This supports a narrower claim (experts missed one technical shift) rather than a general forecaster bias.
• Premises 2-4 (Medium impact) — Benchmark milestones are proxies. Labs optimize for famous targets like the IMO, contamination and saturation are possible, and a multiple-choice virology test or described-vulnerability exploitation is far from open-ended lab work or novel exploit discovery. Early benchmark hits may reflect targeting rather than underestimated general capability.
• Premise 1 (Medium impact) — The 2023 milestone has not been met and may take many years; the forecast may prove accurate. It cannot count toward the pattern until resolved, which reduces the number of clean cases to about three.
• Conclusion (Medium impact) — 'Correct for it' is unspecified in direction, magnitude, and domain, and it presumes the bias persists. If forecasters update after repeated misses, or progress plateaus, the advice could produce overcorrection, hype, and misallocated resources. Applied recursively, the correction should erode its own premise.
• Premise 3 (Low impact) — No specific expert forecast that AI could not exploit vulnerabilities 'for years' was found, and the 2024 result was partial, so the 'overtaken' framing for that portion lacks a documented baseline.

Suggested Improvements

• Representative sampling — Report the full set of resolved and pending AI forecasts from the cited tournament and surveys, with signed error for each, including forecasts that were accurate or too optimistic. A claim of systematic bias needs a denominator; selected examples can only show that some forecasts were beaten.
• Correcting for resolution asymmetry — Score only forecasts whose deadlines have passed, or model the censoring explicitly, and show whether underestimation exceeds what the interim-check bias alone would generate. This directly answers A3 and tests whether the pattern survives the acknowledged measurement bias.
• Pre- versus post-ChatGPT analysis — Show separately that forecasts made before and after November 2022 each err in the same direction, using resolved post-ChatGPT items rather than the unresolved 2023 survey item. P5 asserts the span but currently rests on a thin 'after' side.
• Rival common-cause explanation — Address the reasoning-model/RL-on-verifiable-tasks hypothesis explicitly and explain what evidence distinguishes a forecaster bias from a shared technical shock. This is the strongest alternative to P5 and determines whether the bias is likely to persist.
• Scope and wording of the claim — Narrow the conclusion to benchmarkable, verifiable-task domains during 2022-2026, treat P1 as a pending forecast, and soften 'for years' in the title for cybersecurity. A calibrated claim matches the author's own caveats (A1-A3) and is much harder to rebut.
• Specifying the correction — Give a direction, approximate size, and domain-specific application of the proposed correction, or recommend wider uncertainty intervals and scenario planning instead of a one-way shift. Makes the advice actionable and guards against overcorrection into hype.
• Benchmark validity — State what each benchmark does and does not show about real-world capability, and name which expert groups made each forecast. Prevents readers from equating benchmark success with general competence and 'experts' with a single bloc.

Scenario Tests

• A systematic registry study of all resolved AI-capability forecasts from the cited sources finds a clear majority too slow, even when only past-deadline questions are counted. (Supports) — The systematic-tendency claim would be well supported, and a calibrated correction factor could be estimated.
• The same study finds a roughly symmetric error distribution once overestimates and deadline-resolved questions are included, with underestimation concentrated in verifiable-reward domains. (Challenges) — The conclusion would shrink to a domain-specific claim; a general 'correct for it' rule would be unsupported.
• By around 2030 AI still has not routinely and autonomously produced top-journal-publishable theorems of its own choosing, and agents plateau on real-world cybersecurity tasks. (Challenges) — The 2023 forecast would look accurate, and the underestimation pattern would look like an artifact of a particular scaling era; upward correction would have been harmful.
• AI systems begin routinely producing independently verified, self-chosen, top-journal-quality theorems well before the 2045-ish median. (Supports) — P1 would become a genuine, clean miss from the post-ChatGPT period, strengthening P5's claim that the pattern is not a pre-ChatGPT artifact.
• Experts and forecasters who learned from the 2022-2025 misses now shorten their timelines, and new forecasts resolve roughly on target. (Neutral) — The bias may have been a transitional learning effect rather than a stable tendency; the correction advice would become obsolete as forecasters recalibrate.
• Evidence emerges of benchmark contamination or lab-specific tuning for the IMO, virology, or cyber benchmarks, with real-world competence lagging. (Challenges) — Several premises weaken at once; the lesson shifts from 'experts underestimate' to 'benchmarks overstate capability'.

Coherence & Relevance

The argument is coherent as a cumulative inductive case, and its candor in A1-A3 is a real strength. It supports a narrower claim: several prominent forecasts on benchmarkable tasks were overtaken early. The jump from there to a systematic, cross-field, correctable bias is not closed. The sample is selected, the evaluation method is acknowledged to favor underestimates, one headline premise is unresolved, one is partial, and correlated causes are unaddressed. The conclusion is stated more firmly than the argument's own caveats permit.

• In October 2023, 2,778 AI researchers put even odds on AI routinely and autonomously proving theorems publishable in top mathematics journals about 22 years later. (Weak) — Provides a baseline forecast, not an observed miss; the milestone is unmet and the horizon unelapsed. Its relevance depends on future resolution and on its role as the main post-ChatGPT data point for P5.
• In 2022, the median expert in a forecasting tournament expected AI to reach gold-medal level at the International Mathematical Olympiad in 2030, and AI reached that level in July 2025. (Strong) — Directly relevant as a resolved underestimate, but it is one case, predates ChatGPT, and may reflect a domain-specific optimization surprise rather than a general bias.
• Forecasts of AI's autonomous cybersecurity ability have been overtaken: in 2024 language-model agents exploited known vulnerabilities from their descriptions, and a benchmark threshold experts expected in 2028 was reached in February 2026. (Moderate) — Only the 2028-to-2026 benchmark comparison is a clean forecast-versus-outcome pair; the 2024 component lacks a located baseline forecast and was partly reached (A2).
• A biosecurity benchmark that experts expected AI to match in 2030 was likely matched in April 2025. (Moderate) — Relevant as a resolved directional miss, but hedged ('likely') and dependent on how well the benchmark tracks real virologist-level capability.
• The misses span forecasts made before and after ChatGPT's release and by different expert groups, which points to a systematic tendency rather than one surprise. (Moderate) — Bridges the cases to the conclusion but near-restates it. It lacks independence checks, a denominator, separate pre/post-ChatGPT resolved evidence, and engagement with the common-cause (reasoning-model) explanation.

View this argument on LogicFirst.ai