The Phenomenology of Getting AGI-Pilled

Large language models are forcing questions about mind and meaning that philosophy long deferred. We need to take another look.

First, a couple of updates.

At AEIdeas, I recently wrote about “The Political Backlash to Data Centers,” where I pulled together all of the polling on data centers and the state bills that would impose data center moratoria. In mid February, I submitted testimony for New Hampshire’s HB 1589, which would impose social media interoperability. It’s a topic I’ve written about a lot over the years and this testimony summarizes that research.

I’ve also been thinking about “The Coming Fight to Define the Agentic Web.” As AI agents start to act and shop on our behalf, major platforms are deciding whether to treat those agents as customers or intruders. This fight will shape commerce infrastructure for years to come.

Finally, in late January, I wrote “Jagged Intelligence, Jagged Adoption,” riffing on AI pioneer Andrej Karpathy’s notion of jagged intelligence to argue that business level adoption of AI will be jagged as well. To better understand this jaggedness, I put together a table that compiled major enterprise surveys on AI adoption and ROI. Across studies, only 5-15% of organizations report significant, measurable ROI from AI initiatives.

---

On the AEI panel last week titled “Moral Questions in the Age of AI,” Duke Law Professor Nita Farahany relayed the moment she realized large language models (LLMs) can produce outputs that rival a human. It was a moment people are increasingly experiencing, where the categories we use to understand ourselves are destabilized. When a machine produces an essay indistinguishable from one written by a thoughtful person, we are first taken aback by its capabilities. But then we are left speechless that we cannot articulate what the difference between us and them is.

She got AGI-pilled.

While the panel covered a lot of ground, it was a missed opportunity to discuss bigger ontological, metaphysical, and epistemological questions that AEI can uniquely cover. The arrival of systems that can write and reason forces a much needed confrontation with our settled categories. LLMs have deepened our understanding of the way things are, the nature of communication, what is distinctively human, and what separates mind from machine. But by not tackling these issues head on, the panelists ended up dancing around fundamental philosophical questions.

What does it mean that matrix multiplication employed in transformers, trained on neural nets, boosted by reinforcement learning, and all the rest can create something that closely approximates human-level writing and reasoning? And if a machine can do the things we considered distinctively human, what is distinctively human?

Gideon Lewis-Kraus’ widely shared New Yorker piece, “What Is Claude? Anthropic Doesn’t Know, Either,” is about one stream of this question, which can be understood tracking the mechanistic interpretability work at Anthropic. Lewis-Kraus writes:

Language is, or rather was, our special thing. It separated us from the beasts. We weren’t prepared for the arrival of talking machines. Ellie Pavlick, a computer scientist at Brown, has drawn up a taxonomy of our most common responses. There are the “fanboys,” who man the hype wires. They believe that large language models are intelligent, maybe even conscious, and prophesy that, before long, they will become superintelligent. The venture capitalist Marc Andreessen has described A.I. as “our alchemy, our Philosopher’s Stone—we are literally making sand think.” The fanboys’ deflationary counterparts are the “curmudgeons,” who claim that there’s no there there, and that only a blockhead would mistake a parlor trick for the soul of the new machine. In the recent book “The AI Con,” the linguist Emily Bender and the sociologist Alex Hanna belittle L.L.M.s as “mathy maths,” “stochastic parrots,” and “a racist pile of linear algebra.”

And then he later says:

We don’t know if it makes sense to call them intelligent, or if it will ever make sense to call them conscious. But she’s also making a more profound point. The existence of talking machines—entities that can do many of the things that only we have ever been able to do—throws a lot of other things into question. We refer to our own minds as if they weren’t also black boxes. We use the word “intelligence” as if we have a clear idea of what it means. It turns out that we don’t know that, either.

Those early lines got me: Language is, or rather was, our special thing. It separated us from the beasts. For decades, evidence from animal cognition has chipped away at the idea that language cleanly separates us from the rest of the living world. Sperm whales are the strongest case.

Recently, 19th Century whaler logbooks from the North Pacific, one of the last whale breeding grounds to be hunted, were digitized and researchers found that the ships’ strike rate fell by 58% in less than two and half years after the whalers first arrived in the region. The dramatic decline suggests that there was some kind of information sharing among whale groups. Whalers have long talked about codas but it wasn’t until the 1950s that science finally recognized sperm whales communicate through click patterns that we now know vary by clan and can be carried over great distances through the SOFAR channel.

The whale evidence matters because it collapses the clean version of the language argument. If cultural transmission, combinatorial structure, and long-distance communication exist outside human language, then language was never the bedrock we assigned it. The question was never whether we could talk. It was whether talking, by itself, explained anything about mind, meaning, or moral standing. We just avoided the uncomfortable conclusion because no other species was making us face it directly.

But machines have accelerated that erosion. They do not merely communicate, they are able to participate in discourse. They are drafting contracts, writing poetry, summarizing case law, and simulating empathy. We defined intelligence operationally for decades, then built systems aimed toward those operations and were shocked that they worked. The shock is that our definitions were always more mechanistic than we admitted.

Indeed, is there any reason that the most useful and widely adopted form of AI is the chatbot?

Dialogue is the foundation of philosophical thinking. Famously, it was Protagoras’ dialogical way of arguing that sparked Socrates to search for a firm footing of truth in the Socratic dialogues. The dialogic form turns out to be generative for machines for the same reason Protagoras thought it was generative for humans. Thinking, at its most productive, is an argument with itself.

But if gradients can generate essays that pass for thoughtful reflection, then perhaps some portion of what we call reasoning is statistical pattern completion at scale. That does not mean human thought reduces to linear algebra. It does mean that at least part of what we experience as insight may emerge from processes that are less mysterious than we suppose.

Some have retreated to the safety of taste. But in a must-read essay, Will Manidis eviscerates that refuge: “No one proposed it. No one even had to. It was a clean and easy answer to the question everyone in technology has been asking: what are humans for once the models get good enough?”

Manidis is right in the broad strokes: Taste as we know it arrived when the patron left the room. In the Middle Ages into early Modernity, creative production was a negotiation between patron and maker, oriented toward something transcendent. The Sistine Chapel happened because Julius II and Michelangelo fought. Work emerged from friction, not selection. But when modern production markets arose in the early 20th century, capital and labor got unbundled. The collector replaced the patron. Now art is evaluated by the finished work or the process. Tom Wolfe documented this in The Painted Word, while Pierre Bourdieu produced the academic tome.

Manidis is right that AI again flips this relationship. We should see AI as giving us the ability to make something transcendent. But scarcity was never just an economic necessity, it was also a condition of production. A book took years to write because the author had to develop something worth saying. The difficulty was not incidental to the value. It was constitutive of it. What happens to discourse when the friction of writing and artistry is gone?

Matthew Crawford, author of Shop Class As Soulcraft, was on the AEI panel as well. He should have been asked how AI alters the thesis of his book. The mechanic’s knowledge of an engine is not a set of propositions about engines. It is a practiced relationship with a kind of object, accumulated through friction and failure and learning. A model trained on every repair manual might be able to perfectly tighten a nut but it still cannot express the feeling when a bolt is about to strip.

Indeed, exploring this topic would help partly answer the question Mike Bird of The Economist asked,

What’s the best understanding of why AI-generated writing is still so janky (e.g. better than many students, or non-professional writers/academics, but usually not nearly as good as professional writers)?

AI produced writing is good but it still lacks the voice of an author or the vision of an auteur. You can try to prompt it to create output in a certain style like Joan Didion or Tom Wolfe but it still lacks their cadence. And it lacks that cadence because the gallop of writing is a record of how a particular mind moves.

These machines have also reopened old questions: What does it mean that different AI architectures, trained on different data for different tasks, are independently arriving at similar internal representations of the world?

Vision models and language models, built differently and trained differently, appear to be converging on the same underlying structure of concepts like shape, color, and spatial relations. It’s called the Platonic Representation Hypothesis. Models were never told to build these structures. They emerged independently from exposure to data. If true, it could mean models are latching onto something real about the structure of reality. If true, it touches almost every major fault line in modern philosophy from debates over rationalism to empiricism to structuralism.

The rationalist reads this as evidence that certain structures are prior to experience, that the mind latches onto form because form is really there. The empiricist has a harder time. If two systems trained on entirely different sensory inputs converge on the same representations, is it still true that structure is purely a product of experience? The structuralist, meanwhile, finds it vindicating. Reality has an architecture, and any sufficiently powerful learning system will find it. None of these positions is formally settled but the evidence is now empirical, which fundamentally changes the terms of the debate.

Indeed, if thinking is more mechanistic than we assumed, one question becomes clearer: What is it that cannot be captured by representation and prediction alone?

For a panel with at least one avowed Catholic, I was surprised that there was no mention that LLMs might point to a seemingly radical position in this materialist world, the existence of a spirit, of a soul. LLMs’ ability to reason suggests that what is distinctively human has to lie elsewhere. The soul, in the classical Christian account, was never just about the ability to produce language. It was long understood to be the seat of will, intentionality, moral responsibility, and the capacity for the good, endowed by the Creator. If machines can simulate thought, perhaps what distinguishes us is this first-person perspective ordered toward truth and the good. Humans aren’t systems that model the world. They are individuals that stand in moral relation to it.

The panel was also a chance to probe that uncomfortable trend: Why does everyone bring up the Terminator scenario? What is it about the nature of humans that our first creation is going to kill us?

If you spend any time in the AI debates, you will encounter people who are convinced that AI will kill us all. Our history is littered with such worries, about nuclear weapons, industrial pollution, and engineered pathogens. But there is something deeper about the apocalyptic tone with AI. We do not merely worry that AI will malfunction. We worry that it will supersede us, make us obsolete, or turn us into paperclips. We project onto it the very tendencies we know lurk in ourselves. When humans encounter something weaker, we exploit it. When we encounter something stronger, we fear domination. The AI discourse reads at times like a mirror held up to ourselves, a retelling of the Tower of Babel in Genesis 11, or Prometheus, or Frankenstein. The fear of AI killing us is a dramatized version of a more basic truth that we have yet to grasp, that we are creatures capable of building beyond our wisdom. The Fall of Man in Genesis 3 has been reborn for the AI age.

While my day to day work is in AI policy, the AI debate is mostly conducted at the wrong altitude. We argue about capabilities, safety benchmarks, and economic disruption, which are all important, and yet still, the deeper questions go unasked. What are we if AI can approximate reason? What is thought if gradients can reproduce it? What is left of human distinctiveness once we strip away every faculty a machine can replicate? These are not rhetorical provocations. They are the questions that the AEI panel should have been asking. These are the questions we should be asking.

Until next time,

🚀 Will