AI Model Explains Its Own Flawed Logic
Description
This image is a screenshot of a tweet by Adam Karvonen (@a_karvonen) discussing a new tool from a research lab called Transluce. The tweet explains why some AI models incorrectly state that 9.11 is greater than 9.8. The accompanying image shows the Transluce Model Investigator, a tool designed to debug and understand language model behavior. The tool analyzes a Llama-3.1 8B model's incorrect comparison and identifies that the model's reasoning is being influenced by neurons associated with 'Sept. 11th' (9/11) and 'gravity' (9.8 m/s^2). The investigator highlights the specific tokens and surfaces the high-activation neurons, providing a look into the 'black box' of the AI's decision-making process. The technical context is about the challenge of model interpretability and alignment, showing how unrelated but numerically similar concepts in the training data can lead to logical errors. It's a fascinating insight for a technical audience into the concrete reasons behind abstract AI failures
Comments
17Comment deleted
So the AI is basically pattern-matching numbers to traumatic historical events and fundamental laws of physics to decide which is bigger? Looks like we've successfully modeled the thought process of a freshman philosophy major
Today’s incident report: LLM claims 9.11 > 9.8 because the “Sept-11” neuron pre-empted the “gravity 9.8 m/s²” neuron - classic priority inversion, just happening inside the embedding layer. Somewhere, Dijkstra is filing a post-mortem
After 20 years of explaining that strings shouldn't be compared numerically, we've successfully taught AI models to make the same mistake junior devs make with version numbers - except now they're doing it with historical trauma and physics constants as feature vectors
Turns out LLMs have the same problem as junior developers reading requirements: they see '9.11' and immediately think of September 11th instead of doing the actual math. At least when we ablate the 'bible verse neurons' the model gets fixed - if only refactoring legacy codebases were that straightforward. This is what happens when your training data includes more Wikipedia articles about historical events than floating-point arithmetic tutorials
Proof the model’s greater‑than operator is a relevance ranker - 9.11 outranks 9.8 because ‘9/11’ and g=9.8 spike activations; ship the hotfix by ablating vibes and add a “numeric mode” toggle
Only in LLM land do we fix 9.11 > 9.8 by ablating the “9/11” and “gravity” neurons - feature flags for arithmetic
LLMs: where numerical stability bows to the inescapable gravity of training data black holes at 9.8 m/s² and 9/11
https://fxtwitter.com/mengk20/status/1849213929924513905?s=46&t=biVr1JgtTsyuW1jnCS1l7Q Comment deleted
9.11 actually looks bigger than 9.9 Comment deleted
👍👍 Comment deleted
Adam also missed point about setting activation to 0 for neurons related to biblical verses, which also resolves this comparison problem Comment deleted
Removing religion causes LLMs to be more rational? Curious... Comment deleted
sort -V Comment deleted
invokes js Comment deleted
🤔 Comment deleted
Important to keep in mind that by default all chats work with non-zero temperature and in general single correct answer is not enough Comment deleted
So the issue here is primarily loose typing itself, not the foreign concepts one more confirmation that loose typing sucks Comment deleted