Some time ago, frontier models would stumble when asked how many letter r’s there are in the English word strawberry. The answer “two” caused such astonishment that the strawberry became a meme. The best models now solve it, and can even spell the word out to make sure, but the fact remains that they do not read as we do. When we see or imagine strawberry, we can treat it as s-t-r-a-w-b-e-r-r-y, point at each letter and count. Before the model does anything of the kind, however, another mechanism has already turned the string into pieces that may correspond to a whole word, a mark of punctuation, an accidental syllable or a fragment that means nothing at all to us. That difference does not by itself explain all its counting errors, but it opens a window onto the strange way machines receive language.
They are called tokens, a term that comes from the Old English tācen, sign or mark, kin to the German Zeichen, signal. There is another old word for a mark of recognition, the Greek symbolon, famous for being applied to an object broken in two that allowed each holder to recognise whoever had the complementary half. A host and his guest could each keep one part of a piece and pass it on to children and grandchildren who, generations later and on a journey, could revive the pact of hospitality by recognising each other when the two parts fitted together. Symballein means precisely to “throw (ballein) together (syn)” the parts. It resembles the token and also differs from it, because the symbolon was worth its complementary half inasmuch as together they restored a lost totality. The token of a model, by contrast, has no missing part to complete it. It is a piece of a computational segmentation, able to combine, depending on the context, with any other that is statistically probable. The fragments lack totality, because totality would presuppose symbol, sense and reference to the world, all of them wholly foreign to the model. The symbolon is a class of token that reveals two logics of operation.
A token is, then, a discrete mark that can stand in for something else. In ordinary English it is a counter, like the old subway token or the casino chip, a disc whose value depends on the system in which it circulates. In the early twentieth century, Charles Sanders Peirce distinguished between type and token, so that the word “the” is a type, an abstract form, and each “the” printed on this page is a token, a concrete instance of that form. In computing, a token is the discrete unit of a sequence transformed so that it can be operated on. The tokens of language models need not coincide with the units we recognise in language. Neither letters, nor words, nor morphemes either, and much less units of sense. Nor are they minimal pieces, because they could be divided further still. They are the pieces a tokeniser included in the vocabulary in order to represent text.
This idea that the world is made of pieces that recombine has been with us for a long time, and it carries the ancient air of Leucippus and Democritus, who in the fifth century BC imagined that everything that exists is composed of atoms, literally that which cannot be cut, moving in the void. The Roman poet Lucretius, in De rerum natura (first century BC), found for that intuition an excellent metaphor mounted as a play on words. In Latin, elementa meant both the letters of the alphabet and the elements of matter. Just as a few letters combined form different words, a few types of atoms rearranged compose our entire universe. The alphabet became the model of the cosmos. Twenty-one centuries later, a language model pushes the idea to its limit by building everything it says out of a closed inventory of recombinable pieces, its tokens.
The idea of continuity is reassuring, but alphabetic writing always keeps the level of the letter visible, whereas the model’s counters do not. Let us recall that the word as a unit separated by spaces, which is what we see when we read, is a late invention. The Greeks and the Romans wrote in scriptio continua, without separation between words, so that reading meant unravelling that continuous thread by segmenting the flow. Spaces between words appeared with the Irish and later Anglo-Saxon monks of the early Middle Ages, who learned Latin as a foreign language and needed to bring out the lexical boundaries in order to understand better. The model, very much in its own way, invents another geography of the text through another operation of segmentation.
It makes sense. If the model worked with whole words, it would need a gigantic vocabulary. “House”, “houses”, “housed” and “housing” would count as independent units, as would every verb conjugation, every proper name, every technical term and every newly coined word. Worse still, any word not in the vocabulary would be a problem. The opposite alternative would be to cut the text into individual characters. That solves the vocabulary problem, and a few hundred or a few thousand signs are enough to write practically anything, but with far longer sequences. The model would have to reconstruct over and over the combinations of letters that constantly appear together, driving up the cost of computation and wasting capacity on learning regularities that are far too elementary. Subword tokens are the middle ground, allowing any word to be represented with a relatively small vocabulary while keeping frequent sequences together, like a tightrope walker between the size of the vocabulary and the length of the text. How did we get there?
In 1949, an Italian Jesuit, Roberto Busa, set out to index every word in the complete works of Thomas Aquinas, some eleven million words of Latin. To that end he secured the support of Thomas J. Watson, the legendary president of IBM, in an undertaking that seemed out of all proportion for the machines of the day. Busa would later recount that, knowing Watson had a negative report in front of him, he walked into the meeting with a small sign he had found in reception bearing the IBM motto, “the difficult we do at once; the impossible takes a little longer”, and brandished it at the right moment. Watson had an impossible commission and Busa could wait. He accepted, and the Index Thomisticus took more than thirty years. It ran on millions of punched cards and is the founding act of computational linguistics and the digital humanities. The old distinction between type and token, abstract form and concrete occurrence, came naturally, since counting tokens against types gave a measure of the richness of a vocabulary. For decades tokenising was bound up with the word, until language models cut smaller pieces.
One of the main techniques is called byte-pair encoding. A programmer, Philip Gage, published it in 1994 in a magazine for developers, The C Users Journal, as a method of data compression. The idea was to go through a file, find the pair of adjacent symbols that repeated most often, replace its occurrences with a new symbol, record the equivalence and repeat the procedure. Each round invented an abbreviation for a frequent combination and the file kept shrinking. Two decades later, in Edinburgh, one of the world capitals of machine translation, a different problem emerged. As three researchers building translators with neural networks were struggling with the rare or unknown words that jammed their systems, they realised that a version of Gage’s old compression trick served to build a vocabulary of subwords. If instead of compressing we keep the abbreviations as they appear, we obtain a catalogue of word fragments that covers them all, because even the rare ones are assembled out of known pieces.
The program with which they implemented it, subword-nmt, ended up in the hands of half the world. The procedure begins with single letters and goes on fusing the most frequent pairs into ever larger counters, guided by frequency alone. The result is a vocabulary in which common words are a single counter and rare ones break into pieces that follow no logic of sense, only of statistics. “Dog” may be a whole token; an infrequent or technical word splinters into three or four fragments that mean nothing on their own. A counter is equivalent, on average, to about three quarters of a word in English, and considerably less in languages of non-Latin script. Since they are fragmented more, they are dearer to process, creating a silent inequality.
The leap to many of the large models came with the variants of BPE applied to bytes. For GPT-2, in 2019, Alec Radford and his team swapped letters for raw bytes, so that with a base vocabulary of a mere 256 pieces the model can write anything, in any language or alphabet, without ever running into a word it does not know, and then the most frequent byte sequences are combined into larger tokens, avoiding the cost of processing everything byte by byte. That tokeniser was enormously influential. Cheap and complete at the same time is an offer hard to refuse, and that is why, with all its defects, the variants of subword tokenisation remain in use.
Let us go back to the strawberry. When the model reads strawberry it sees two or three tokens, something like “str”, “aw”, “berry”. Asking it to count the r’s is like asking someone who reads at a glance how many strokes of ink there are. Spelling ceases to be a simple task, and those who mock the clumsiness of the model that answers “two” miss the most interesting part. For a time it seemed that the whole problem lay in the model counting badly because tokenisation hid the characters from it. Recent research, however, suggests that the operation of counting itself and the properties of the architecture are also involved. Tokenisation changes the grid through which text enters the system, but it does not by itself determine what the system will be able to reconstruct afterwards.
The problem becomes more serious when numbers and their capricious slicing appear. “380” may enter as a single token and “381” be split into “38” and “1”, so that instead of a clean row of digits the model sees an irregular mosaic. It therefore does not add or multiply digit by digit. It has to learn numerical regularities from representations whose boundaries obey criteria that were not designed for arithmetic. Tokenisation is, for that reason, one of the factors that can make symbolic calculation harder, though it is not enough on its own to explain the arithmetic errors of models. Some modern tokenisers introduce special rules for figures precisely in order to make this part of the landscape more regular.
Phantoms are not lacking. In 2023, a study came upon “haunted tokens”, the most celebrated of them called SolidGoldMagikarp, which was the name of a user on a Reddit forum and had stayed in the tokeniser’s vocabulary while barely appearing in the training data. When the model was asked to repeat that counter, the system raved, insulted, changed the subject or claimed to be alive, as though one had trodden on a loose paving stone of the language. They were signs the model had in its alphabet and had never learned to read.
We have already seen that this is not the only way of doing it. The field itself distrusts it. There are models that work directly with letters or with raw bytes, with no vocabulary at all, such as CANINE or ByT5. They have the advantage of escaping strawberry blindness, because they spell and count without effort, at the computational cost already mentioned. That is why CANINE, for example, compresses the string of characters before the expensive layers. The frontier line seeks the best of both worlds. The Byte Latent Transformer, which Meta presented at the end of 2024, uses no fixed counters. It groups bytes into “patches” whose size adjusts by itself, longer where the text is predictable and shorter where it grows dense, according to the model’s own uncertainty about what is coming, so that it spends computation where it is really needed. With this strategy it matched subword models at large scale and has already begun to appear in the open libraries. For all that, the frequency token still reigns in production models, though the signs that it is a stage keep accumulating. The strawberry, the inequality that different tokenisations introduce between languages, and strange phenomena such as the so-called haunted tokens have no single cause, yet all of them are reminders that the way of cutting text is not an innocent decision. Perhaps in a few years we shall look at today’s token as we now look at scriptio continua.
Ferdinand de Saussure defined the sign as the union of a signifier, the image of the word, and a signified, the concept, bound together by an arbitrary convention. The token is in a certain sense lower still. On entering the model it becomes a learned numerical representation and, layer after layer, its relations with other representations make it possible to build syntactic, semantic and conceptual regularities that were not contained in the initial cut. The model begins by working with divisions of the text that no speaker would choose and ends by writing, quoting, arguing and sometimes even moving us. The r’s of strawberry are the mark of a format of reading that is not ours. Understanding what the machine can and cannot do begins with knowing that it looks with other eyes.