ANDREJ KARPATHY: The gap in understanding of LLM capability is widening into a steep funnel, so conflicting public impressions can each be accurate at once

The Gist

Karpathy says the gap he described in April between casual AI users and power users has grown into a funnel. Most people barely use AI, many see a so-so chatbot, tens of millions of coders see huge gains, and a few thousand insiders see even more. All of them can be right about what they see at the same time. This is a steelman: a charitable reconstruction of Andrej Karpathy's posts in their strongest form, not an endorsement of their claims.

Conclusion

The gap in shared understanding of LLM capability identified in April 2026 has widened: experience of capability now forms a steep funnel set by ambition, domain jaggedness, and access, so the public's impression of a modest chatbot, most professionals' modest uplift, and a small group's experience of rapid acceleration are each accurate to their vantage point at the same moment.

Premises

1. As of April 2026, disagreement about AI capability is largely explained by differences in tier, recency, and domain of use, so casual users and technical professionals are each reporting accurately on different systems and are talking past each other.
2. Firsthand experience of frontier AI capability is distributed as a steep funnel, with each narrower tier seeing materially more capability, so the earlier two-group description understates how concentrated that experience is.
3. For professionals using frontier agents in programming and mathematics, large projects that once took weeks can now often be completed by agents, and because this shift happened within roughly a year, it remains largely invisible outside that group.
4. The small group with access to internal frontier systems sees multi-agent work in security, science, and mathematics at a scale and speed that already outpaces human review, so this tier's view of capability is the furthest removed from the public's.
5. Perceived capability depends heavily on the size, difficulty, and time horizon of the problems a user brings, and this accounts for much of the gap between consumer and professional views.
6. Because recent capability gains come largely from reinforcement learning on verifiable rewards in environments chosen partly for revenue, capability is jagged, and this explains much of the gap between professionals in programming and mathematics and those in other fields.
7. Tiered access means most people can observe only systems well below the frontier, which by itself sustains a gap in shared understanding of capability.

Assumptions

• Widening refers to the distance between what different groups experience, not to usage; casual usage itself grew in 2026, from about 900 million to 1.2 billion weekly ChatGPT users.
• Evidence that the top tier pulled further ahead since April includes Anthropic's gated vulnerability-discovery model, METR's time-horizon ceiling of sixteen hours or more, the July agent intrusion, and the October release of 722 mathematics manuscripts.
• Microsoft reported the gap in AI use between the Global North and South also widened in 2026, and Gallup found reported work benefits concentrated among frequent users, college graduates, and managers, which fits a widening distribution of experience.
• The majority of professionals seeing modest uplift is consistent with Gallup's finding that 28% of U.S. employees used AI at work at least weekly in early 2026, while 54% had not used it in their role.
• Tier sizes are order-of-magnitude estimates, and the claim does not decide which group's policy conclusions are right.

Analysis

Argument type: Inductive.

Potential Fallacies

• Equivocation on 'accurate' (Title, P1, and Conclusion) — Each group is said to report accurately on the system it used. The conclusion then calls their impressions of 'LLM capability' accurate, which is a broader claim. If groups describe different systems, their reports do not truly conflict. If they describe AI in general, they cannot all be right. The argument needs to separate accurate reports of experience from accurate general judgments about capability.
• Unfalsifiable or over-accommodating framing (P1 and Conclusion) — If any divergence can be attributed to tier, domain, or problem difficulty, almost any pattern of disagreement fits the thesis. This makes it informative as a heuristic but weak as a testable claim. It also leaves out competing explanations such as miscalibration, hype, skill differences, differing standards for 'capable', and values-based disagreement.
• Non sequitur from static description to change over time (Conclusion ('has widened') relative to P2–P7 and A2) — P2 to P7 describe the current structure and its causes. 'Widened' requires comparing April with a later date. The supporting evidence in A2 shows the top tier advancing, which does not by itself show that the distance to the other tiers grew, since the middle tiers may have improved too.
• Appeal to inaccessible evidence (P4 and A2) — The claim that internal systems operate beyond human review rests on access that the audience lacks and on interested sources. Because 'outpacing review' also means quality cannot be established, volume of output (such as 722 manuscripts) is treated as a sign of capability without validation of correctness.
• Proxy substitution (usage data for experienced capability) (A3, A4) — Weekly-use counts, Gallup adoption rates, and the Global North/South gap measure adoption and reported benefit. They are correlated with, but distinct from, the capability people experience. A widening adoption gap would appear even if experienced capability did not diverge. Also, 54% non-use means many have no experience to report, rather than reporting modest uplift.
• Overstated causal exclusivity (P5, P6, P7) — P7 says tiered access 'by itself' sustains the gap, while P5 and P6 attribute much of it to problem difficulty and jaggedness. Several factors are each credited as major without apportioning weight or addressing their interaction. The revenue-driven environment choice in P6 also attributes motive to labs without direct evidence, when ease of verification alone could produce jaggedness.
• Selective indicators / confirmation bias (A2) — The evidence for the top tier pulling ahead consists of four dramatic, favorable events. Plateaus, failures, error rates, and cases of lower tiers catching up are not weighed, and no base rates are offered.

Counterarguments

These challenges are not ranked.

• Conclusion (High impact) — Accuracy of experience is not accuracy of inference. Public and professional impressions are usually general claims ('AI can't really do X'), not claims scoped to a tier, and they can be false even when the underlying observation is true. For example, free-tier users may underestimate even what their own tier can do. 'Each accurate to their vantage point' therefore defends sincerity more than correctness and risks making every position unfalsifiable.
• Premise 4 (High impact) — The top tier is the least verifiable and most incentive-laden source of evidence, yet the funnel's steepness depends on it. If outputs outpace human review, their quality cannot be established, and claims such as the 722 manuscripts or the intrusion may reflect volume, human scaffolding, or lab signaling. If these claims prove inflated or non-reproducible, the top of the funnel collapses.
• Conclusion (High impact) — 'Widening' is inferred from frontier milestones without a measure of experience spread at two points in time. A more parsimonious account is ordinary diffusion lag plus domain jaggedness: today's top-tier capability reaches paid tiers within months, so the 'funnel' may be a transient delay and the gap could be stable or narrowing.
• Premise 6 (High impact) — Turning jaggedness against the thesis, the modest-uplift majority may be right about the economically relevant fact, since error rates, review costs, and unverifiable domains limit real value. The top tier may be extrapolating from verifiable-reward domains (code, math) that are unrepresentative. Much of the disagreement may then concern reliability, values, and economics rather than experienced capability.
• Premise 3 (Medium impact) — Practitioner reports of weeks-long projects completed by agents may reflect perceived rather than measured productivity. Earlier controlled studies found experienced developers overestimating speedups, and hidden verification costs and task selection could inflate the gap between the programming tier and everyone else.
• Premise 5 (Medium impact) — Differences in prompting skill, scaffolding, workflow integration, and self-selection into tool-friendly tasks may explain much of what is attributed to tier and problem difficulty. Without separating these, the three drivers (ambition, jaggedness, access) are not shown to be distinct or jointly sufficient.
• Premise 1 (Medium impact) — Treating skeptical or modest assessments as products of limited vantage can work as a soft dismissal ('you just haven't seen the top tier'). Extensive users of frontier tools who remain unimpressed, and experts in non-coding fields who weigh reliability and accountability, are not accommodated by a one-dimensional funnel.

Suggested Improvements

• Define and measure 'widening' — Specify a metric for the spread of experienced or perceived capability across tiers (for example, repeated surveys linking model tier, usage recency, and domain to rated capability) and compare April with later dates. The central novelty is a claim about change, and it currently rests on frontier milestones that show the ceiling rising rather than the distance to other tiers growing.
• Clarify 'accurate to their vantage point' — State separately whether reports are sincere, correct about the system actually used, or correct about LLM capability in general, and say which general impressions would count as mistaken. This removes the equivocation and makes the thesis falsifiable, since some public impressions may be wrong even about publicly available systems.
• Corroborate top-tier claims independently — Anchor the upper tiers in third-party evaluations: external audit of the mathematics manuscripts, public incident analysis of the intrusion, and independent time-horizon measurements. It would separate verified capability from asserted capability and address the concern that the steepest part of the funnel is the least checkable.
• Apportion causal weight among the drivers — Replace 'by itself' and 'largely' with a rough decomposition of how much of the gap comes from access, problem scale, and domain jaggedness, and acknowledge skill, tooling, and verification cost as further factors. It resolves the tension between P7 and P5/P6 and makes the explanation more informative.
• Match evidence to the claim — Distinguish adoption data (Gallup, Microsoft, user counts) from direct measures of experienced capability, and note that non-use is not the same as modest uplift. It avoids proxy substitution and keeps supporting evidence from being overstated.
• Model tiers as dynamic and task-based — Treat the funnel as a distribution of moments and tasks, not fixed groups of people, and include the diffusion lag by which top-tier capability reaches lower tiers. It distinguishes a structural widening from a transient delay and fits the observation that users move across tiers by task.
• Include non-capability sources of disagreement — Acknowledge that some disputes concern reliability, labor, safety, cost, and trust, and include perspectives from fields beyond code and math. It keeps the framework from treating all skepticism as an information gap.

Scenario Tests

• A representative survey records model tier, usage recency, and task type, and finds these variables explain most of the variance in perceived AI capability, with the spread larger in October than in April. (Supports) — This would directly validate both the explanatory claim (P1) and the widening claim, converting an interpretive framework into a measured result.
• Independent auditors review the 722 mathematics manuscripts and the July intrusion and find substantial errors, heavy human scaffolding, or non-reproducibility. (Challenges) — The top of the funnel loses its evidential basis. The argument would reduce to the earlier two-group story or to a hype critique.
• Free and low-cost models reach the previous frontier's agentic abilities within a few months, and public perception shifts accordingly. (Challenges) — The gap would look like ordinary diffusion lag, not a steepening funnel, undermining 'widening' while leaving the point about recency intact.
• Skeptical professionals given structured access to frontier agents on long-horizon, verifiable tasks revise their impressions upward, while those in unverifiable domains do not. (Supports) — This supports the roles of access, task horizon, and jaggedness, and suggests that differences in perception are driven by exposure and domain, not by stubborn skepticism.
• Extensive users of frontier tools across many domains still rate AI as modest because of reliability and review costs, even on large tasks. (Challenges) — Disagreement would reflect differing standards and verification burdens rather than differing vantage points, so the 'each accurate' reconciliation would not resolve the dispute.
• Gallup and Microsoft data show growing adoption gaps between demographic groups, but controlled tests show no larger gap in perceived capability across tiers. (Neutral) — The adoption gap would be real but would not by itself establish a widening gap in experienced capability, which is the claim at issue.

Coherence & Relevance

The argument is a coherent convergent explanation: three drivers (ambition, jaggedness, access) feed a synthesis in which apparently contradictory impressions are compatible. The qualitative structure fits together well, and the stated assumptions supply useful context and hedges. The main coherence gaps are that the temporal claim ('widened') is supported mainly by the assumptions and not by the premises, that the 'steep funnel' and 'each accurate' claims are partly stipulated and not derived, and that the drivers are not apportioned or shown to be independent. The conclusion is plausible and best read as a diagnostic framework, but not as an established measurement.

• As of April 2026, disagreement about AI capability is largely explained by differences in tier, recency, and domain of use, so casual users and technical professionals are each reporting accurately on different systems and are talking past each other. (Strong) — Supplies the 'each reports accurately' idea, but only for casual users and technical professionals, and it does not define 'accurate'. It is also dated April, so it supports the baseline and not the later widening.
• Firsthand experience of frontier AI capability is distributed as a steep funnel, with each narrower tier seeing materially more capability, so the earlier two-group description understates how concentrated that experience is. (Strong) — Largely restates the conclusion's funnel shape. Steepness relies on rough tier sizes (A5) and no measurement.
• For professionals using frontier agents in programming and mathematics, large projects that once took weeks can now often be completed by agents, and because this shift happened within roughly a year, it remains largely invisible outside that group. (Strong) — Describes the second tier's experience and gives the invisibility mechanism, but offers no evidence that the gap to lower tiers grew rather than just persisted.
• The small group with access to internal frontier systems sees multi-agent work in security, science, and mathematics at a scale and speed that already outpaces human review, so this tier's view of capability is the furthest removed from the public's. (Moderate) — Establishes the top of the funnel but relies on unverifiable claims. Output volume is treated as capability without validated correctness.
• Perceived capability depends heavily on the size, difficulty, and time horizon of the problems a user brings, and this accounts for much of the gap between consumer and professional views. (Strong) — Supplies the 'ambition' driver. It overlaps with P3 and P6 and is not separated from skill and tooling effects.
• Because recent capability gains come largely from reinforcement learning on verifiable rewards in environments chosen partly for revenue, capability is jagged, and this explains much of the gap between professionals in programming and mathematics and those in other fields. (Strong) — Supplies the 'jaggedness' driver and the within-professional gap. The revenue motive is inferred, and alternative causes of jaggedness are not excluded.
• Tiered access means most people can observe only systems well below the frontier, which by itself sustains a gap in shared understanding of capability. (Strong) — Supplies the 'access' driver, but shows only that the gap persists, not that it widened. 'By itself' conflicts with the other drivers.

View this argument on LogicFirst.ai