The shifts between white and silver in the homepage headline are real token boundaries. Nothing is hard-coded: when the page loads, it reads its own text, trains a small tokeniser on it, and encodes the headline with the result. The headline itself is left out of training, because otherwise the tokeniser would simply memorise it.
The algorithm is byte-pair encoding, the same family used by GPT-style language models. It is small enough to describe in three steps:
- Turn the text into UTF-8 bytes. Those 256 possible values are the starting vocabulary.
- Find the adjacent pair of tokens that appears most often, and give it a new token id.
- Replace every occurrence of that pair, then go back to step 2.
Training stops after 256 merges, or sooner if no pair appears more than once. The loop is short:
while (merges.length < maxMerges) {
const counts = new Map<number, number>();
for (let i = 0; i < ids.length - 1; i++) {
const key = ids[i] * 65536 + ids[i + 1];
counts.set(key, (counts.get(key) ?? 0) + 1);
}
// Pick the most frequent pair, then replace it everywhere.
}
The rows of letters above the headline are the corpus itself, or as much of it as fits on your screen. Each merge joins letters into a token, closes the gap between them and flashes them white, so the page slowly reads itself into fragments of words. The headline uses the same clock and the same vocabulary, shown in white and silver.
Moving the slider takes you to any point in those merges. At zero, every byte is its own token. As merges accumulate, the fragments that are common on the page, such as ing and he , close into single tokens.
The corpus here is under two kilobytes, so this vocabulary is a toy. It knows this page and very little else. Type your own sentence into the box and you can see where it runs out: unfamiliar words stay broken into small pieces, while words from the page collapse quickly.
Non-English text shows the same thing more sharply. A character outside ASCII takes several bytes, and a token that holds only part of one is drawn as hexadecimal, because those bytes don’t yet make up a whole character.
snapstr is my attempt at the serious version of this: a byte-level BPE tokeniser with a target of at least 100 MB/s on one thread. The C++ tokeniser isn’t written yet, and so far the repo holds the Python trainer and the oracle that proves it. This page has no such target. It retrains from scratch on every visit, which is a slow way to run a tokeniser and a good way to watch one work.