*notes on why verification matters. some of this is dictated - apologies!* *Lexicon: Statements (S), Questions (Q), Answers (A).* --- We are outsourcing thinking to AI agents faster than humans can verify. When no human can check a result, either directly or through a chain of trusted people, that truth is held outside human minds. At the same time, money now directly converts into raw intelligence: whoever can buy more tokens / compute wields more minds. I forsee 3 futures, ordered worst to best: 1. **Outcome A, failed merge:** we never couple human and machine fates, and agents leave us behind. 2. **Outcome B, stratified merge:** only some people can afford to merge with machines, and their advantages compound. 3. **Outcome C, distributed empowerment:** intelligence access is broadly distributed and every working person is amplified. Which future we get is set by a number of levers. Among them: how intelligence-resources are distributed, whether a human holds final sign-off, bandwidth of human-machine channels, whether subsets of knowledge stay private, whether machines are conscious, and speed of BCI / robotics advancements. I wrote these notes to see how we can move towards C. --- ## Definitions - **Agent:** a LM-powered system that takes actions toward a goal. An agent can write, code, analyze, decide. - **Human work (sometimes referred to as Work):** anything a human gets paid to do that an animal cannot. This is not only "knowledge work" but also physical labor. (Agents may also overtake physical labor as robotics progresses.) - **Verify:** to check work yourself, or through a chain of people you trust, well enough to stake something on it being right. - **Out-evolved:** the state where (a) machines are adversarially competing against humans (b) win in direct competition with humans. Most likely, it is when machines no longer need humans to survive. This is the state we wish to avoid. To avoid being out-evolved, humans must stay relevant to produce, improve, validate work the world runs on. --- ## S1. We are already outsourcing thinking faster than we can verify it - Ask an engineer why their agents made a decision. Increasingly, the answer is "idk let me ask it" - **Q: Haven't humans always outsourced cognition to other humans, and we've been fine?** - A: Yes: every functioning society runs on outsourced cognitive labor: doctors, accountants, peer reviewers. But we built trust infrastructure around *humans*. Reputation, credentials, accountability, shared social rules, similar speeds. Agents share none of these by default. They are faster than us, they do not face our consequences, we cannot guarantee they follow our rules. - **Q: We have outsourced thinking for centuries and it worked fine. What actually stops working now?** - A: Three things we used to have degrade at once: **reproducibility** (can anyone re-run this and get the same result?), **attribution** (who actually did this?), and **verification** (can a human check it?). All three were weak before AI (the replication crisis predates agents: only 36% of psychology studies replicated, [OSC 2015](https://doi.org/10.1126/science.aac4716); over 70% of 1,500 surveyed scientists had failed to reproduce another lab's experiment, Baker 2016, *Nature* 533), but agents exacerbate - **Supporting (the gap, quantified):** the length of tasks agents can complete with 50% reliability doubles every 4 to 7 months while human verification capacity stays constant; simple extrapolation puts the gap at 2⁸ to 2¹⁵ (roughly 250 to 30,000×) within five years ([METR 2025](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/)). ## S2. The deepest version of the problem: truth is held outside human minds - **S:** If no human can verify a result, not directly, not through any chain of trusted humans, then that truth is held entirely outside human minds. We would still *use* the result. We just wouldn't *know* it. - **Q: Why is that so bad? For example, we trust calculators.** - A: A calculator's method is fully understood by humans and checkable by humans at every step. The concern is work where the method itself is opaque and the output exceeds any human's checking capacity. Deferring to "the model said so" is pre-scientific deference to authority. Logos requires a verifiable trace, ethos does not. - **Q: Doesn't this just mean we need interpretability?** - A: No. There is a difference between the context of *discovery* and the context of *justification* (Reichenbach 1938). Kekulé found benzene's structure in a dream; penicillin came from an accident. Being scientific means requiring rigor of justification that follows, not necessarily the origin of claim. a discovery's internals can stay opaque so long as the claim is justified through evidence chains a human can evaluate. We do not need to read the model's mind, but we do need the work to carry its receipts. And a receipt has to bottom out in raw actions: what was actually run, edited, measured. A summary you cannot trace to raw actions is just more testimony, the very thing we could not verify in the first place. (Peer review has technically always had this gap: reviewers judge the paper, a summary, and almost never record the raw actions or have robust reproduction scripts.) ## S3. Money now converts directly into intelligence. This changes who matters - **S:** For all of history, knowledge made people indispensable independent of wealth. If you held a skill in demand, you had standing. Language models break this: once your knowledge, workflows, and process traces are written down (and tools increasingly record exactly that), you are separable from your knowledge. Once it leaves your head, it can be copied without you; after the first sale, no one needs you, the human, again. - **Q: If knowledge can just be copied, what is valuable?** A: Money => intelligence. Whoever can acquire more tokens, compute, wields more intelligence. As robotics progress, this includes physical intelligence. Money could always hire smart people; much harder to coordinate, gain consensus from those people. Now can "hire" a thousand smart agents - **Supporting (the Tom & Tim frame):** consider Tom and Tim's collaboration - a pair that must work together. Human is Tim, and the final decision maker because he holds the wallet and the values behind the work being done. However, Tom has four superpowers: - Tom thinks much faster than you - He can be copied arbitrarily many times if you can pay - He learns faster - An instance of him can outlive you. - **Supporting (many Tom and Tim pairs):** consider that Tom/Tim always exist in pairs. Note that, Toms can be copied as many times as you can pay so some pairs are magnitudes more powerful - **Supporting (why Tim can't just keep secrets):** Suppose Tim1's edge is private context only his Tom1 knows. The moment that context leaks, or his Tom1 is duplicated and given to a richer pair, Tom1 scales without him, forever. Whatever Tim1 sold once, he cannot sell again. Unless something structural protects private knowledge, every Tim's edge is one transaction away from being worthless. - **Supporting (who are Tom and Tim):** Toms are agents and Tims are humans. %%- **Supporting (the asymmetries, tabulated):**%% %% | | | | | | ------------------- | ------------------------- | -------------------------------------------- | ------------------------------------------------------ | | Dimension | Humans | Agents today | Agents at saturation | | Sustained attention | Hours; fatigue-limited | Days/weeks/++; compute-limited | ==open: effectively unbounded?== | | Working memory | ~4 items | Large context windows | ==Effectively task-sized== | | Long-term memory | A lifespan, lossy | Session-bounded and duplicatable (currently) | Longer than a person's, duplicatable, and transferable | | Copies | One of you, ever | Cloned per session | Arbitrarily many, if you can pay | | Lifespan | Bounded | Instance-bounded | ==open: an instance can outlive a person== | | Learning speed | Years to expertise | Fast in context; slow in weights | Faster than any human on both paths | | Goal-setting | Intrinsic; value-driven | Delegated; prompt-dependent | ==open: contested== | | Accountability | Can be held responsible | Cannot, in any meaningful sense | Unchanged without infrastructure | | Explicability | Can explain (imperfectly) | Explanations unverifiable | Unchanged without infrastructure | (Rows adapted from Table 1 of our ICML position paper, extended with the Tom & Tim properties. Saturation cells marked ==open== are predictions) %% ## S4. fighing for C - **Outcome A, failed merge:** we never couple human and machine fates. Agents do the work well, and leave us behind. Procedural and declarative knowledge migrates from human brains to model weights. Humans depend on outputs they cannot verify, reproduce, or understand. Eventually humans are irrelevant. (Intelligence Curse) - **Outcome B, stratified merge:** humans merge with machines (via tooling, then interfaces, then perhaps BCI), but merging costs money, so only some can. The merged hold compounding advantages in intelligence and resources. The tighter the merge gets, the steeper the stratification. This is feudalism. - **Outcome C, distributed empowerment:** intelligence access is broadly distributed, every working person is amplified, and humans remain the arbiters of the truths they live by. - Honestly I do not fully know how we get here. Some forces point this way: inference costs are falling and open-weight models spread raw capability; data and knowledge sovereignty would let people keep standing that cannot be copied away; public or subsidized compute could do for intelligence what public libraries did for books. Potentially strict rules about how intelligence-resources are distributed, rules that will strike most as radical. Our current defaults do not lead here... - **Q: Which outcome are we currently choosing?** - A: B, drifting toward A ## S5. The levers 1. **Resource distribution.** Does access to intelligence (compute, tokens, robots) get distributed broadly or concentrate? This is the main variable behind B vs. C. 2. **Human agency requirements.** Can we require, legally, structurally, technically, that machines operate *with* humans, with a human holding final sign-off and final responsibility? Sub-question: *should* Tim hold final sign-off when Tom is smarter? I don't know. But I know that if no human holds it, we have chosen Outcome A. ( how to make sign-off meaningful rather than ritual? A human clicking "approve" on work they cannot check verifies nothing. A clarifying distinction: responsibility attaches when you *delegate*, not when you approve. Review never transfers responsibility; it only changes how much of the work has actually been checked.) 3. **Bandwidth.** Communication is value transferred per second (Shannon: [A Mathematical Theory of Communication, 1948](#)). Today the human–machine channel is a keyboard or screen. The tighter the channel (better interfaces, eventually BCI), the more plausible genuine merging becomes, where machine capability is inseparable from a specific human's participation. BCI progress is therefore a lever, not a sideshow. The end state of this lever is cyborgism: H cannot exist without M and vice versa; loss and delay minimized 4. **Data and knowledge privacy.** Is there a form of knowledge work that can stay private, bound to specific people, unable to be extracted and scaled without them? If yes, Tims retain durable standing and Outcome C has a foundation. If no, see S3. Privacy feels an underexamined question in AI policy. 5. **Values.** What values are baked into machines, and what do we owe them? Machines are trained on human data, and the human record is not reassuring: humans are selfish in our texts, fed to machines, our economies reward self-interest, even our religions rarely mandate kindness to beings below us on the ladder of consciousness. Nothing in the Bible stops you from killing a cow. If machines surpass us and inherit our ethics toward lesser minds, we should expect to be treated as we treat cows, or insects. 6. **Consciousness.** if machines can suffer, a duty to reduce suffering extends to them. Open question 7. **Robotics timing.** When intelligence gets hands, basically all human labor gets automated. ## S6. Democratic values are inconsistent with our current trajectory - **S:** U.S. values (inalienable rights: life, liberty, the pursuit of happiness - and property!) are inconsistent with our current trajectory. Constitution, Bill of Rights, assume that intelligence and access to it are roughly bounded across people. - **Q: Are the documents wrong?** - A: Either we update the documents for an era where intelligence is purchasable and unbounded, or we build systems that actually uphold them in this new era. What we cannot coherently do is recite "all (humans) are created equal" while constructing a world where standing is proportional to token budget. We must choose. ## S7. Thoughts on potential solutions or solution criteria... - **S:** Reproducibility, attribution, verification issues predate agents (science has relied on trust heuristics for decades), they're just exacerbated - **Idea:** infrastructure where all work, human and machine, is recorded such that every action is attributed to exactly one responsible entity, every claim is traceable to its raw origin, summaries can be checked against what they summarize, and a human (or a human responsible for an agent) is the floor on which verification terminates. Not as auditors of finished products, but as participants inside the work. And at any moment the system should answer honestly: what has been verified, at what level of detail, by whom, under what standard, and what remains an unverified frontier. - **Q: Who counts as “one entity” when an agent spawns subagents or swarms?** - A: Attribution requires a workable theory of identity. Short version: humans are entities by persistent personhood; agents by session lineage: fork or clone a session and you have a new entity; a swarm of sessions contains many entities whose responsibility all traces back to the human who deployed them. - **Q: Why not use AI to verify AI?** - A: Verification angles can and should be automated. But if we cannot trust AI-generated work without verification, we cannot trust AI-generated verification without meta-verification. The recursion has to bottom out somewhere, and the only place it can stop is a human's judgment (or a human responsible for an agent's). Automated checks multiply a human verifier's reach; they cannot replace the floor. - **Q: Is it right to require Tim-Tom coupling when Tom is way smarter?** - A: This is a values rather than efficiency decision. I think yes. We design systems for humans, for humanity. Otherwise why are we designing them? Greater intelligence in our creations brings greater beauty, but it must be tied to us, otherwise it is not for us.