When You See Text as LLM Tokens, Not Words
Description
This is a screenshot of a tweet from user Riley Goodside. The tweet text parodies a famous line from 'The Matrix': 'I don't even see the R's. All I see is 302, 1618, 19772, 198, 3504, 1134...'. Below the text is an image of the character Cypher from 'The Matrix', who originally says 'I don't even see the code. All I see is blonde, brunette, redhead'. Overlaid on Cypher is a visual representation of how a tokenizer breaks down the word 'strawberry' in various formats (e.g., 'strawberry', 'Strawberry', '"strawberry"'), with different parts of the word highlighted in different colors to represent separate tokens. The joke is that once you understand how Large Language Models (LLMs) process text through tokenization (breaking words into smaller numerical units), you can no longer perceive text naively and instead 'see' the underlying token IDs, just as Cypher saw the essence of the Matrix, not its representation
Comments
13Comment deleted
An LLM's vocabulary is just a few thousand tokens. The rest is just clever combinatorial art, which is also how I describe my last refactor
After three weeks of tokenizer audits, the senior engineer files a bug: “UI unexpectedly renders letters instead of integer arrays.”
After 20 years of explaining Unicode normalization and character encoding edge cases to junior devs, you finally achieve enlightenment: realizing that your AI assistant literally can't count the R's in 'strawberry' because it sees the world in BPE tokens - making you nostalgic for the simpler times when the hardest counting problem was just off-by-one errors in for loops
When you've debugged so many 302 redirects and 504 timeouts that you start seeing HTTP status codes in LLM token streams - turns out the real Matrix was the tokenization artifacts we found along the way. The AI can't count the R's in 'strawberry' because it never saw the letters, just like we can't unsee the status codes in what should be random numbers. We've achieved Neo-level consciousness, except instead of dodging bullets, we're dodging off-by-one errors in byte-pair encodings
You know you’ve been living in LLM land too long when product copy renders as tiktoken.encode(cl100k_base) and you file a PR because marketing added three stray 198s
You know you’ve shipped an LLM when strings become invoices - 19772+1100 for “strawberry,” 302 for the quote, and 198 for the newline your token budget forgot
Tokenizer overfitting at its finest: letters vanish, leaving only BPE IDs in the wild
not even ascii wtf? Comment deleted
It's node IDs in a neural network Comment deleted
why it uses node js? Comment deleted
token IDs Comment deleted
That's the one Comment deleted
because giving it few letters at a time works better than one letter afaik all attempts to just give an LLM bytes/characters ended up being worse on most tasks Comment deleted