CAL NEWPORT: Scaling language models is producing jagged gains in structured domains rather than a rising tide of general capability, so the bet on scale alone is unlikely to pay off

The Gist

Newport argues AI isn't getting smarter at everything at once. It's getting much better where there's lots of clean data with right answers, like coding and math, while other areas lag. Measured progress keeps speeding up and AI reached gold-medal level at a top math contest, but those gains are also in structured areas, which is partly his point. This packet steelmans the arguments made by Cal Newport and Ed Elson in their best light for analysis and is not an endorsement of their conclusions.

Conclusion

The evidence points to jagged, domain-specific gains from language models, strongest where data are structured and answers checkable, rather than a general rise in capability, so the frontier labs' bet that scaling these models alone will produce general intelligence is unlikely to pay off.

Premises

1. The frontier labs' strategy assumed that scaling language models would yield general intelligence that other software could simply call on.
2. Capability gains are better pictured as uneven progress along particular paths than as a rising water level that lifts performance on all tasks together.
3. A field experiment with 758 consultants found that GPT-4 improved speed and quality on tasks inside its capability frontier but made users 19 percentage points less likely to reach the correct answer on a task just outside it.
4. Recent gains are concentrated in domains with abundant structured data and checkable answers, such as programming, mathematics, and cybersecurity, and research finds that reinforcement training on such tasks transfers poorly to unstructured domains and can degrade some general abilities.
5. A widely predicted 2025 wave of agents doing general knowledge work did not arrive at the scale forecast.
6. A high-profile 2026 mathematics result came shortly after human researchers had made progress on a closely related problem and used very large amounts of compute, which shows depth in a narrow domain rather than general ability.
7. A leading researcher who helped build the scaling paradigm said in 2025 that the field is moving from an age of scaling to an age of research.

Assumptions

• The argument concerns the generality of gains, not whether models are improving.
• Video and image generation have also advanced quickly, which fits the picture of particular paths proving navigable rather than contradicting it.
• Public estimates of the compute for OpenAI's 2026 Navier-Stokes result range from about $6.5 million to $15 million at list prices rather than $22 million, and the result's formal verification is disputed and not yet accepted.
• Measured task horizons for frontier models have doubled roughly every seven months over the long run and faster since 2024, though mainly on software tasks, so progress is accelerating even where its generality is disputed.
• Language-model systems achieved gold-medal standard at the 2025 International Mathematical Olympiad working in natural language, which shows strong reasoning gains within structured domains.

Analysis

Argument type: Inductive.

Potential Fallacies

• Non sequitur (missing bridging premise) (Inference from P2–P6 to the conclusion) — The premises describe how gains are distributed now. The conclusion forecasts a ceiling on scaling. Nothing establishes that present jaggedness will persist at greater scale or that unevenness implies insufficiency. Jagged gains across many verifiable domains could still sum to broad or economically transformative capability.
• Unfalsifiable framing / false dichotomy (P2, A2, A5 and the conclusion) — 'Rising tide' and 'jagged paths' are presented as exclusive and are not operationalized. Rapid advances in video, images or olympiad math are absorbed as 'paths proving navigable', while shortfalls count as non-generality. Capability can be broad on average and uneven in detail, and human expertise is itself jagged. Without a stated threshold for 'general', almost any outcome fits.
• Straw man / outdated target (P1 and the conclusion) — The labs' strategy is portrayed as 'scale alone yields general intelligence'. Since about 2024 labs have relied on reinforcement learning, reasoning models, tools and agent scaffolding. P4's evidence concerns this blended approach, so refuting pure scaling does not show the labs' actual bet is likely to fail.
• Hasty generalization from limited or dated evidence (P3, P6) — One field experiment on GPT-4 (2023) with a single deliberately constructed outside-frontier task, and one disputed mathematics result, are used to support claims about the trajectory of the whole paradigm. The consultant study also measures human-AI interaction and over-reliance as much as model generality.
• Appeal to authority / over-reading testimony (P7) — 'Age of research' signals a shift in emphasis. It is compatible with scaling continuing to matter alongside new ideas, and the speaker has a stake in a research-first narrative. It does not show that scaling has stopped working.
• Inferring capability limits from a missed forecast (P5) — A missed executive forecast reflects timing optimism, incentives, and deployment friction (reliability, integration, liability) as well as capability. 'At the scale forecast' is unquantified, and coding agents did spread widely, so the shortfall depends on how the forecast is read.
• Begging the question / disputed example used as clean exemplar (P6 with A3) — P6 builds its interpretation ('depth in a narrow domain rather than general ability') into the premise. A3 says the result's verification is disputed, so it cannot yet cleanly illustrate anything. Building on prior progress and using heavy compute is also typical of human breakthroughs, so these features do not discriminate between narrow and general ability.
• Selection / measurement confound (P4) — Gains are visible where benchmarks and verifiable answers exist and where labs concentrate effort and reward environments. This confounds an inherent limit of scale with allocation choices and measurability.

Counterarguments

These challenges are not ranked.

• Conclusion (High impact) — Jaggedness may be a transitional feature. Each new domain brought into verifiable-reward training widens the frontier, and verifier expansion, synthetic environments, rubric-based rewards and tool composition keep extending what counts as 'checkable'. Jagged coverage could converge on general coverage, which is the labs' claim.
• Conclusion (High impact) — Payoff does not require generality. Programming, mathematics and security are high-value domains, and automating software and AI research could itself accelerate progress. The labs' bet can pay off commercially even if no general intelligence emerges, so the move from 'jagged' to 'unlikely to pay off' conflates technical and economic senses of payoff.
• Premise 1 (High impact) — Labs do not bet on scale alone. They combine pretraining, reinforcement learning, reasoning models, tools and scaffolding. The evidence cited in P4 describes that blended approach, so the argument attacks a partly outdated or simplified position.
• Premise 2 (High impact) — Human intelligence is also jagged. If uneven profiles disqualify a system from being general, no system would qualify, so the criterion is either trivial or needs a stated threshold. The binary also hides a mixed picture in which a broad baseline rises with uneven peaks.
• Premise 3 (Medium impact) — The study used GPT-4 on one constructed outside-frontier task and measures user over-reliance and workflow effects as much as model capability. Later models may have moved the frontier, so it cannot show that the pattern persists with scale.
• Premise 4 (Medium impact) — Concentration of gains partly reflects where benchmarks exist and where labs invest. Unstructured-domain progress may be real but hard to measure, and newer recipes (rubric or model-graded rewards) may improve transfer. Natural-language IMO gold hints at reasoning that extends beyond formally checkable settings.
• Premise 5 (Medium impact) — Technology diffusion has long lags, and agent adoption depends on reliability, integration and liability. A one-year timing miss is weak evidence about capability, and widespread agent deployment may simply arrive later.
• Premise 6 (Medium impact) — Most human breakthroughs also build on neighboring progress and use large resources. If the result is verified, it may count as evidence of real capability. If it is not, it cannot support the thesis either way.
• Premise 7 (Medium impact) — 'Age of research' most plausibly means that pretraining scale needs to be supplemented with new ideas, not that scaling has failed. This reading is compatible with the labs' current hybrid strategy.
• Conclusion (Medium impact) — The accepted assumptions (accelerating task horizons, IMO gold, fast video and image progress) match the trend of repeated 'scaling has hit a wall' predictions that did not hold. The argument concedes strong evidence for the opposing view while offering no pre-specified falsifier of its own.

Suggested Improvements

• Define 'general intelligence' and falsifiers — State what pattern of results would count as general capability, such as consistent performance on unstructured, open-ended tasks without task-specific training. Specify what evidence would count against the thesis. This would make the jagged-versus-rising-tide contrast testable and stop success from being absorbed as 'another path proving navigable'.
• Quantify jaggedness — Report dispersion of improvement rates and cross-domain correlation of gains over a broad, pre-specified task battery across model generations, for example the share of variance explained by a common factor. A measurement would replace the metaphor and show whether gains are uneven, correlated, or both.
• Bridge from description to prediction — Add an explicit premise and evidence that checkable-domain gains do not bootstrap into other domains, for example longitudinal transfer measurements across successive model generations and controlled comparisons of scale versus reward-environment investment. The step from 'uneven now' to 'unlikely to pay off' is the argument's main gap.
• Target the labs' actual strategy — Reframe the thesis around scaling plus verifiable-reward training and scaffolding, and cite lab statements on their current approach, not the 2022–24 pretraining narrative. This avoids the straw-man objection and makes the P4 evidence bear directly on the target.
• Separate technical generality from economic payoff — State whether 'pay off' means reaching general intelligence or earning returns on investment, and address that narrow gains in coding and security may be commercially valuable. The audience will read 'pay off' financially, and the evidence addresses only capability architecture.
• Update and broaden the evidence — Replicate the outside-frontier result with current models and multiple tasks, and add adoption data separating capability limits from deployment friction. Treat the 2026 math result as unconfirmed until verified, and present the P5 shortfall alongside the broad uptake of coding agents. Current, multi-case evidence would reduce reliance on a dated study, a disputed anecdote and a missed forecast.
• Address safety and labor implications of jagged capability — Note that strong narrow capability in programming and cybersecurity is consequential in its own right. This prevents the thesis from being read as reassurance and makes it more balanced.

Scenario Tests

• Reinforcement training on code and math begins to show clear positive transfer to open-ended, unverifiable tasks (planning, writing, judgment) in successive model generations. (Challenges) — The 'poor transfer' premise would be obsolete and the pattern would look like a rising tide with uneven slope. The thesis would need to be abandoned or heavily qualified.
• Labs expand verifiers, simulators and learned graders so that many more domains become checkable, and gains follow. (Challenges) — Jagged coverage converges on general coverage. The structured/unstructured boundary proves movable and jaggedness becomes a transitional stage, not a ceiling.
• The Navier-Stokes result is formally verified, or similar open-problem results appear across several domains. (Challenges) — P6 would turn from an illustration of narrowness into counterevidence, or at least into a contested example. The thesis currently rests on a contingent fact.
• Time-horizon measurements remain steep on software but stay flat on non-software, unstructured knowledge tasks over several more model generations. (Supports) — This would show persistent concentration in checkable domains, strengthening the claim that gains are domain-specific and not a general rise.
• General-purpose office and computer-use agents remain unreliable through 2026–27 while coding agents keep improving. (Supports) — The pattern would match the jagged picture and show that the 2025 shortfall reflected a structural limit and not just delayed adoption.
• Widespread agent deployment arrives in 2026–27 after reliability and integration problems are resolved. (Challenges) — P5 would be exposed as a mistimed observation, and the shortfall would be read as a diffusion lag.
• Revenue and valuations of the labs keep growing on coding and enterprise automation without any general intelligence. (Neutral) — The descriptive claim of jaggedness holds, but the 'unlikely to pay off' conclusion is undermined unless 'pay off' is confined to generality. This shows why the two senses of payoff need to be separated.

Coherence & Relevance

The argument is coherent in structure: several lines of evidence converge on a diagnosis of uneven, checkability-driven gains, which is then used to forecast the limits of scaling. The diagnosis is moderately well supported, chiefly by the transfer research and the consultant study, and the explicit assumptions concede rapid progress and so keep the claim confined to generality. The forecast is the weak point. It depends on an unstated premise that present jaggedness is structural, on an undefined notion of 'general intelligence', and on a portrayal of the labs' strategy as 'scale alone'. Several supporting premises (the metaphor, the missed forecast, the disputed math result, the researcher's remark) are interpretive or weakly diagnostic, so the conclusion is better read as a plausible, appropriately hedged skeptical case than as an established result.

• The frontier labs' strategy assumed that scaling language models would yield general intelligence that other software could simply call on. (Moderate) — Sets up the target but gives no evidence about outcomes. The labs' actual strategy is more heterogeneous than 'scale alone', so refuting the simplified version may not bear on their real bet.
• Capability gains are better pictured as uneven progress along particular paths than as a rising water level that lifts performance on all tasks together. (Moderate) — Restates the descriptive half of the conclusion and offers no independent support. It does not say what degree of unevenness distinguishes jagged from broad rising gains.
• A field experiment with 758 consultants found that GPT-4 improved speed and quality on tasks inside its capability frontier but made users 19 percentage points less likely to reach the correct answer on a task just outside it. (Moderate) — Shows an uneven, hard-to-detect frontier but concerns one older model, one constructed task and user behavior. It does not address whether scaling broadens the frontier.
• Recent gains are concentrated in domains with abundant structured data and checkable answers, such as programming, mathematics, and cybersecurity, and research finds that reinforcement training on such tasks transfers poorly to unstructured domains and can degrade some general abilities. (Strong) — Directly supports the diagnosis and supplies a mechanism. It concerns post-training method and not pretraining scale, and does not show that poor transfer is permanent.
• A widely predicted 2025 wave of agents doing general knowledge work did not arrive at the scale forecast. (Weak) — Links to the thesis only through the assumption that the shortfall reflects a lack of generality. Deployment friction, forecaster incentives and timing explain it equally well.
• A high-profile 2026 mathematics result came shortly after human researchers had made progress on a closely related problem and used very large amounts of compute, which shows depth in a narrow domain rather than general ability. (Weak) — The inference from context to narrowness is asserted, not shown, and the result's verification is disputed. It is a single case whose features do not discriminate between narrow and general ability.
• A leading researcher who helped build the scaling paradigm said in 2025 that the field is moving from an age of scaling to an age of research. (Moderate) — Signals insider doubt about pure pretraining scale but is opinion and compatible with scaling continuing alongside research. It does not establish that the bet will fail.

View this argument on LogicFirst.ai