GAVIN BAKER: Many AI systems reflecting diverse values are safer for people than control of the technology by a handful of individuals

The Gist

Gavin Baker says no small group should steer AI for everyone. Lots of different AIs are more likely to reflect different people's values and keep power spread out. This is a steelman: Gavin Baker's thread is reconstructed in its strongest fair form, not endorsed.

Conclusion

Fragmentation at the model layer is the best outcome for ordinary people because it spreads decisions across many AI systems and developers instead of concentrating them in a few individuals.

Premises

1. No group of five to twenty individuals should be trusted to make the best decisions about this technology's direction.
2. Many AI systems are more likely to encompass the diversity of human values across the world.
3. People should be able to use AI that shares their own values.
4. History shows the dangers of centralizing powerful technology, and distribution and diffusion better preserve safety, freedom and human agency.

Assumptions

• The claim is a values and governance claim, separate from the infrastructure economics.
• Critics argue many independent models make safety controls harder to enforce; the claim weighs that against concentration risk.

Analysis

Argument type: Inductive.

Potential Fallacies

• False dichotomy (P1 to Conclusion) — The argument sets 'a handful of individuals' against 'many AI systems' as if these were the only options. Intermediate arrangements are left out: a few labs under democratic oversight, regulated pluralism, open standards with shared safety floors, or tiered access by capability. Showing that the worst form of concentration is bad does not show that fragmentation is best.
• Hasty generalization / selective historical analogy (P4) — 'History shows' names no cases. The record is mixed. Diffusion helped with printing, the internet and encryption, but controlled access is widely credited with improving safety for nuclear materials, pathogens and aviation or drug regulation. Whether AI resembles the first group or the second is the central question, and the premise assumes the answer.
• Conflating number of models with diversity of values (P2) — More developers do not automatically yield more value diversity. Models trained on overlapping data, with similar alignment methods, commercial incentives and distillation from one another, can converge on a near-monoculture. The inference needs evidence that the models actually diverge.
• Unsupported comparative inference / superlative overreach (Conclusion (with A2)) — The conclusion claims the 'best' outcome and uses 'safer' in the title. That requires comparing fragmentation with every alternative and weighing its costs against concentration risk. Naming the safety-enforcement trade-off is not the same as showing which side outweighs the other, and the premises do not bridge the gap from 'better than unaccountable concentration' to 'best overall'.
• Equivocation on 'safer' and on 'values' (Title / Conclusion / P2-P3) — Safety from concentrated power (political, autonomy-related) is treated as interchangeable with safety from misuse or catastrophic accident. Fragmentation may improve the first while worsening the second. Likewise, matching AI to users' preferences is treated as a safety benefit although it is a different notion.
• Asymmetric skepticism (P1) — P1 distrusts any small group, yet extends implicit trust to the many developers, users and market processes that fragmentation relies on. Each of those developers is also a small group making consequential decisions, and applied consistently the same skepticism would also undercut the regulators and standards bodies a fragmented ecosystem would need.
• Unqualified principle without limiting test (P2-P3) — P3 and the value-diversity framing contain no boundary for harmful values. A right to AI that 'shares one's values' would on its face cover extremist, abusive or rights-violating values, and any boundary would require someone to decide, which reintroduces the problem P1 was meant to avoid.

Counterarguments

These challenges are not ranked.

• Conclusion (High impact) — For sufficiently capable systems, one weakly safeguarded model can cause catastrophic, irreversible harm (bioweapons assistance, cyber-offense). Safety is set by the least careful widely accessible developer, not the average one, so fragmentation multiplies failure points and removes enforceable chokepoints. Released weights cannot be recalled or patched.
• Conclusion (High impact) — Many competing developers face greater pressure to cut safety testing, and coordination on safety standards is easier among a few accountable actors than among hundreds. Concentration is a governance problem that can be addressed through democratic oversight, antitrust and transparency, while proliferation harms cannot be undone.
• Premise 1 (High impact) — The real alternative to private control by a few individuals is not necessarily fragmentation but accountable governance: regulated frontier labs, public oversight, audits and liability, with pluralism delivered at the fine-tuning and application layer. Distrust of small unaccountable groups supports these options at least as well as it supports fragmentation.
• Premise 4 (High impact) — Nuclear materials, pathogens, gain-of-function research and pharmaceuticals are cases where concentrated, licensed control is credited with improved safety. If distribution always beat concentration, these technologies should be widely diffused, which is absurd, so P4 needs a threshold based on harm severity and reversibility.
• Premise 2 (Medium impact) — Current frontier models show measurable convergence in tone and values because of overlapping data, similar alignment methods and distillation. Model count is therefore a poor proxy for value diversity, and market-driven diversity tracks purchasing power, which under-represents poorer or marginalized populations.
• Premise 3 (Medium impact) — Value-matched AI can entrench echo chambers, polarization and harmful value systems, and can be used to harm third parties or minorities within communities. Authoritarian states could also cite 'AI reflecting our values' to justify state-controlled national models, producing local concentration instead of pluralism.
• Conclusion (Medium impact) — Fragmentation at the model layer may be nominal if compute, cloud, chips and distribution platforms stay concentrated. Model-layer plurality would then sit on a few chokepoints and give false comfort that power is distributed to ordinary people.

Suggested Improvements

• Weigh the safety trade-off explicitly — Carry out the comparison A2 announces. Give criteria for when concentration risk outweighs misuse and enforcement risk, and discuss tail-risk and irreversibility explicitly. The central objection is acknowledged but not answered. A real weighing would turn an asserted trade-off into a demonstrated one.
• Add a capability-tiered limiting principle — Specify that plurality applies above a floor of enforceable safety and rights constraints, and distinguish lower-risk capabilities, where diffusion is clearly beneficial, from the most dangerous ones, where controlled access may be warranted. This preserves the pluralist core while avoiding the reductio that the premises justify unrestricted proliferation or harmful value-matching.
• Consider and rule out alternatives — Compare fragmentation directly with regulated oligopoly, public or multilateral oversight, open standards with shared safety baselines, and polycentric governance, and show why it does better. A superlative claim like 'best outcome' must beat every alternative, not only the worst form of concentration.
• Ground the historical premise — Name specific cases on both sides (for example printing, encryption and open-source software versus nuclear, biological and aviation regimes) and explain which features of AI make it more like the former. This turns a vague 'history shows' into a testable analogy and anticipates the strongest counterexamples.
• Support the link from model count to value diversity — Provide evidence or mechanisms (open weights, regional models, independent training data, interoperability and low switching costs) by which many models produce real value diversity and genuine user choice. Without this, P2 and P3 do not connect developer plurality to benefits for ordinary people.
• Align the claim with its evidence — Either narrow the conclusion to 'fragmentation is a valuable hedge against concentrated, unaccountable control' or add premises showing it reduces misuse and catastrophic risk. Distinguish 'safer' from 'best' and define the senses of safety in play. The premises mainly support freedom and pluralism, so the safety and superlative claims go beyond the evidence.

Scenario Tests

• AI capabilities reach a level where a single widely available model can give meaningful uplift toward mass-casualty weapons. (Challenges) — Proliferation risk then dominates concentration risk, and safety depends on the least careful actor. The argument's balance flips unless it adds a capability threshold or controlled access for the most dangerous tier.
• A few frontier labs gain decisive power over norms, content policies and access, and use them in ways that reflect narrow values or serve incumbent interests. (Supports) — This is the concentration risk P1 and P4 describe. Plural, competing and open alternatives would provide checks, exit options and mutual auditing that a monopoly would not.
• Many new models appear, but all rely on the same few chip, cloud and distribution providers and are largely distilled from the same leading models. (Challenges) — Apparent plurality masks common-mode dependencies and a value monoculture. P2 fails, and the benefits the argument promises do not materialize.
• Regional and community-specific models are built (for example for minority languages and non-Western value systems) and are easy for users to adopt, customize and switch among. (Supports) — This is the best case for P2 and P3: underserved populations gain real choice, and the weaknesses of value-monoculture defaults are mitigated.
• Models tuned to extremist or abusive value systems proliferate, and a serious incident prompts a political backlash and heavy-handed centralized regulation. (Challenges) — Unconstrained fragmentation can provoke the concentration it aims to prevent, which shows that a floor of accountable, democratically legitimate safeguards is needed.
• A hybrid regime emerges: capability-tiered oversight and independent audits for frontier systems, with open and pluralistic models at lower tiers and standardized interoperability. (Neutral) — This captures much of the pluralist benefit and much of the safety benefit, and it shows that the choice is not binary. It is compatible with the argument's concern but not with its superlative conclusion.

Coherence & Relevance

The argument is coherent as a statement of a pluralist, anti-concentration worldview and engages one real objection, but its inferential structure is thin. The four premises mostly restate a single intuition (distributed power is better than concentrated power) rather than offering independent lines of support. They support the modest claim that concentration of unaccountable power is a serious risk and that pluralism has real benefits. They do not support the stronger claims that fragmentation is safer overall or that it is the best outcome, because the safety trade-off named in A2 is never weighed, intermediate governance options are never considered, and 'safer' and 'best' are used as if they meant the same thing.

• No group of five to twenty individuals should be trusted to make the best decisions about this technology's direction. (Moderate) — It establishes only that concentrated, unaccountable decision-making is risky. It does not show that fragmentation avoids equivalent risks or is better than accountable institutional alternatives, and its logic could apply to every small decision-making group.
• Many AI systems are more likely to encompass the diversity of human values across the world. (Moderate) — The step from model count to value diversity is unsupported, and diversity is treated as benign without distinguishing harmful values. It also bears on pluralism and not directly on safety.
• People should be able to use AI that shares their own values. (Weak) — It supports user-level customization, which a few providers could offer, and it has no limiting principle. It does not connect to the system-level claim about fragmentation or to safety.
• History shows the dangers of centralizing powerful technology, and distribution and diffusion better preserve safety, freedom and human agency. (Strong) — This is the premise most directly tied to the conclusion, but the analogy is unspecified and selective, and it does not address dual-use technologies where diffusion raised risk.

View this argument on LogicFirst.ai