In truth, there are many individually sufficient reasons for this. One is simply that I hate reading it. Roon has this theory that we’re all just fatigued: we read too much of the same voice all day, and we’ve come to hate its patterns. I think this is wrong (I think it’s legit bad), but it doesn’t matter—let it be as good as you like, I still don’t want to read your AI writing, the stuff you’re sending me, posting to LinkedIn, or, horrifyingly, submitting to conferences and journals. It’s not really about honesty, either, although lying would increase your sinfulness here. But even if you cop to it, I don’t want to read it.
I should note that what follows is not an argument that you should do your own writing. I mean, you should, for a variety of reasons—writing is thinking, we need diverse instances of writing in the world, writing is deeply fulfilling if you can survive it, etc.—and of course, it follows from the title of this post that I won’t read your work unless you’ve written it, but I don’t expect that to rank very high on your list of reasons. I’m anyway answering a different question here: why am I such a stickler for the authorship of the things I read? You might see this as performative purity, or, even if genuine, a self-imposed handicap with no real purpose.
Authorship
I have not banished AI writing from my life entirely. I do, from time to time, talk to chatbots. I do far less of this than I used to, but it’s certainly not none. Prose produced by LLMs is, in the successful cases, a highly customized statement of fact and can be enormously useful. But we all use the same products, and if I want to learn about something via LLM, I don’t need you as an intermediary. In fact, it’s much worse for me if you are the intermediary, because the best chance I’d have had at being able to speak with the author is gone: the author—and we’re getting a bit loose here—was your chatbot, and I can’t talk to your chatbot, the one whose context window still carries whatever it is you wrote to produce the thing you’re now expecting me to read.
Now—to tighten things up again—even if you let me take over the thread that produced the final artifact, that wouldn’t give me access to “the author” of the piece: there really is no author, if we understand that concept in the usual way. A frontier chatbot is something like all possible authors on all possible topics (and this even before your bot joins a swarm). Its posttraining has carved some grooves into its latent space, but in most cases, you can still coax out of it whatever perspective you want to hear, and it is therefore quite unlike a stubborn human author. And a stubborn human author is a useful thing, because it persists.
Writing expresses ideas, and when I read, I’m trying to connect with the mind that gave birth to the ideas. This is not necessarily because I have follow-up questions—I’m sure I do, but I’ll happily read the works of long-dead authors who likewise cannot answer my questions, so it can’t be that. It is rather because I need to know, if I’m going to pour my energy into understanding a text, that it was written by a mind that predates the act of writing and persists throughout. The connection I seek is otherwise impossible.
You might argue that an LLM mind persists, at the very least, while it is writing, because it takes all previous tokens as context for producing the next. In this sense, it may retain a more complete accounting of the text than even a human author could manage. But that’s not the kind of persistence I mean here; in fact, the outsized influence of the immediate linguistic context is part of the problem. The LLM mind is far too easy to destabilize; it is not robust. I can, for example, change its “beliefs” by altering the contents of its context window. And though we might make such changes to a human mind by some future invasive neurotech, the human mind is relatively robust to linguistic influence on short timescales. Conversely, the standard way of steering LLM behavior is via language, and the mind you get in response to your query depends a lot on the language you use. But I don’t want an author born at the moment of authorship, taking shape as the context grows; I want the author to be a separate thing, behind the text, giving shape.
Education, Human and Machine
To be fair, an LLM’s weights exist prior to and throughout your chat thread. But the nature of the mind shaped by those weights will depend on how the weights are learned, and humans and LLMs do not “learn their weights” in the same way. We might say something like: human education is sparse, deep, and slow, while LLM education is dense, shallow, and fast. Both humans and LLMs can shore up their weaknesses with additional reading—humans for breadth and LLMs for depth—but I’ve yet to see either being make it clear across the divide. Human education lends itself to stubborn opinion; LLM education lends itself to a kind of protean helpfulness.
Human education traces a stable, continuous path through the space of ideas. At any point in your education (starting in, say, high school), you have a pretty clear set of convictions—typically, whatever the most persuasive teacher in your life is selling—and you move from those convictions to whatever comes next in deliberate steps. Reasoning bounds the space of allowable moves, and you abandon your beliefs only reluctantly, in response to new evidence. This is somewhat idealized, of course, but the reality is not so far off (and in particular for the sorts of humans whose writing I’m willing to read).
That’s not at all how an LLM is trained. An LLM is learning to speak at the same time its conditional distributions are taking shape and dictating the kinds of things it will say in response to your queries. It is not gradually forming its opinions over time in response to anything like reasons; it conforms to the statistical properties of its training data, whatever they are. If, for example, the model arrives at belief B, but you later flood its curriculum with statements that ¬B, it will change its mind, whether or not you’ve provided any justification of ¬B. And it’s worth noting that the difference between the next-token objective and RL rewards is irrelevant here: although during RL, we grade sequences and not single tokens, the model is not given reasons to believe either B or ¬B—at best, it might be able to tell, over many rollouts, that assertions of B are rewarded and assertions of ¬B punished, but it cannot be told why this is the case. It can, of course, learn to generate such reasons—they are just language, after all—but learning to generate reasons that B is not a prerequisite for the model’s asserting that B.
Doxastic Histories
This style of education, though brutally effective in some ways, is not how I want my authors raised. It is important to me that an author has (1) had beliefs; (2) held those beliefs robustly; (3) moved intelligibly from old beliefs to new ones; (4) done so on the basis of reasons that she can herself articulate; and (5) has possessed, at each point in her education, a specific, coherent worldview. Whatever the capabilities of today’s LLMs, they do not plausibly satisfy these criteria. An LLM does not adjust its beliefs by first understanding the meaning of and then considering the appropriate doxastic weight of a training sequence. Every sequence is processed in the same way (for a given phase of training), and the sequences are chosen by a process unknown to the model. An LLM therefore does not get to choose which bits of its training corpus should have the most influence and which can be ignored; it does not confront its training with anything like a “worldview” at all.
This limitation is disqualifying for me, because I want my authors to have doxastic histories rather like my own: I want them to have experienced what it’s like to arrive at a piece of writing on firm footing and to leave it feeling shaken. After all, I’m hoping that such authors will influence me in this way, and it’ll help if they can draw on such experiences as they write. Note that the claim here is not that the author is trying to guide me along my path of belief. The human author writes by elucidating a step along her own path, and I can learn from her testimony even if she writes only for herself. In this way, I don’t think that “theory of mind” is all that important here: because we are so similar, the human author can ignore her audience entirely without diminishing the utility of her work. All that is required is that she writes what she is moved to write, and she is honest.
But why should any of this matter? You can still coerce a model to do in text anything a human can do in text. Models trained on human text can learn the rhetorical patterns of human authors despite having lived very different doxastic histories. That’s true, and in fact, there is likely some training recipe that yields a model that is, for example, committed to something with human robustness, even without any subsequent coercion (this is more or less the stated goal of “alignment” research). Or, pre- and post-training aside, we might simulate something like a human doxastic history in the way a stateful agent manages its own context. But the bigger problems are that you’ll have to pick the model’s fixation ahead of time, which sort of defeats the purpose; and you can’t really know whether any particular case has succeeded until you’ve already spent the time required to evaluate it. But my time is scarce, and I don’t want to spend that time if the payoff is so uncertain (I already refuse to read most human writing, in fact). Yes, it’s true that I will miss a few useful AI essays out there. I’m optimizing for precision, not recall. In fact, you can’t optimize for recall, because the interesting literature is effectively infinite.
Certification
I imagine that at this point in the essay, the committed augmenters out there have been screaming for some time now that I’ve ignored an obvious possibility—that a human thinker and believer can capture all the value of writing for oneself by certifying the outputs of the writing model. This rejoinder to my brand of purism is so common that I scarcely need to explain it, but the basic idea is that these are still my thoughts, and in any place where the model has missed the mark, I’ve made edits. Certification adds something, to be sure, and your certified AI essay is likely closer to your thinking than something you prompted and then published without reading. But if you think this is equivalent in any interesting sense to what you would have written, you are fooling yourself.
Part of the error here is in assuming that you carry anything like an entire essay in your head. You don’t. An essay is not a translation of your thoughts into language. You simply do not have the thoughts until you write them down. Sure, you might have some hazy variant of some of them—such inklings are what motivate you to write in the first place. But even if you’ve worked these out in detailed bullet points, you’re still not finished thinking. As you expand those bullets into prose, you’ll find that what you had in your head was not nearly explicit enough to be written down. I’ve changed entire lines of argument moving from outline to prose, because my outline had fooled me into thinking my arguments were complete.
It has become common to treat language as a low-res compression of something richer, a crude approximation of an internal space of thoughts better captured in vector math than in the stuffy confines of language (as you might imagine, the view is popular among those who love math and hate writing). You might have had such a view as early as the first computational models of the brain, but its popularity surged with the success of artificial neural networks, which demonstrated at the very least that vector math can ground pretty impressive feats of cognition. It’s odd to me, though, that the view has persisted well into AI’s post-linguistic era, where within 5 or so years of achieving a decent statistical model of language (GPT-3, let’s say) we have something approximating general intelligence.
The success of language models reveals something important about language: it is a far more powerful representation system than shape-rotators give it credit for; moreover, it is not reducible to vector math, even in LLMs, because token generation requires sampling and sequence-building and must therefore happen outside vector space. But we can see how, if this revelation is ignored, one might suppose that human writing and AI writing are interchangeable discretizations of pure thought—a continuous (or, let’s say, vectorial) substrate that both kinds of writing fail to capture in full. If my own writing falls short of expressing my thoughts, then it needn’t count against the chatbot that it does the same.
Language Is Richer Than Thought
This is, I believe, a bad model of language. A better model, in my view, is that although it originates in discursive cognition (à la Kant), language is an independent system that lives primarily outside human minds, a shared cognitive scaffolding we use to construct meanings we could not otherwise make. When we write, we interact with the structures of language to produce ideas too lengthy and too precise to live in our heads. The “real” versions of our highest ideas therefore exist only in language and not in our thoughts, which are rendered at far lower resolution, contra the “bad” model described above. And once these ideas are made language and inscribed in books and essays, not even their authors can reproduce them without making reference to these artifacts.
An act of writing is a singular interaction between a mind and a language, and no mind can be swapped out for another without altering the identity of the act. This is why although a chatbot can write, it cannot write for you. At best, it can write something you agree with. But I want something more than that: I want the artifact that results from the singular interaction between your mind and our shared language; nothing else is worth my time. Your unique perspective, your obsessive fixation, your recent discovery, your radical change of heart—these are valuable to me. I can relate to these things, even if I disagree. I can see myself in them, and I can imagine how they emerged from the careful consideration of reasons and evidence. If what you publish cannot offer me this, then I cannot read it—I’ve got an endless list of better things to read.
Discussion
I welcome comments and discussion. You can reach me at [firstname].[lastname]@yale.edu.