Skip to content
DevMeme
5788 of 7590
AI ML Post #6344 · source on Telegram

AI Model Explains Its Own Flawed Logic

Description

This image is a screenshot of a tweet by Adam Karvonen (@a_karvonen) discussing a new tool from a research lab called Transluce. The tweet explains why some AI models incorrectly state that 9.11 is greater than 9.8. The accompanying image shows the Transluce Model Investigator, a tool designed to debug and understand language model behavior. The tool analyzes a Llama-3.1 8B model's incorrect comparison and identifies that the model's reasoning is being influenced by neurons associated with 'Sept. 11th' (9/11) and 'gravity' (9.8 m/s^2). The investigator highlights the specific tokens and surfaces the high-activation neurons, providing a look into the 'black box' of the AI's decision-making process. The technical context is about the challenge of model interpretability and alignment, showing how unrelated but numerically similar concepts in the training data can lead to logical errors. It's a fascinating insight for a technical audience into the concrete reasons behind abstract AI failures

Comments

17
Anonymous ★ Top Pick So the AI is basically pattern-matching numbers to traumatic historical events and fundamental laws of physics to decide which is bigger? Looks like we've successfully modeled the thought process of a freshman philosophy major
  1. Anonymous ★ Top Pick

    So the AI is basically pattern-matching numbers to traumatic historical events and fundamental laws of physics to decide which is bigger? Looks like we've successfully modeled the thought process of a freshman philosophy major

  2. Anonymous

    Today’s incident report: LLM claims 9.11 > 9.8 because the “Sept-11” neuron pre-empted the “gravity 9.8 m/s²” neuron - classic priority inversion, just happening inside the embedding layer. Somewhere, Dijkstra is filing a post-mortem

  3. Anonymous

    After 20 years of explaining that strings shouldn't be compared numerically, we've successfully taught AI models to make the same mistake junior devs make with version numbers - except now they're doing it with historical trauma and physics constants as feature vectors

  4. Anonymous

    Turns out LLMs have the same problem as junior developers reading requirements: they see '9.11' and immediately think of September 11th instead of doing the actual math. At least when we ablate the 'bible verse neurons' the model gets fixed - if only refactoring legacy codebases were that straightforward. This is what happens when your training data includes more Wikipedia articles about historical events than floating-point arithmetic tutorials

  5. Anonymous

    Proof the model’s greater‑than operator is a relevance ranker - 9.11 outranks 9.8 because ‘9/11’ and g=9.8 spike activations; ship the hotfix by ablating vibes and add a “numeric mode” toggle

  6. Anonymous

    Only in LLM land do we fix 9.11 > 9.8 by ablating the “9/11” and “gravity” neurons - feature flags for arithmetic

  7. Anonymous

    LLMs: where numerical stability bows to the inescapable gravity of training data black holes at 9.8 m/s² and 9/11

  8. dev_meme 1y

    https://fxtwitter.com/mengk20/status/1849213929924513905?s=46&t=biVr1JgtTsyuW1jnCS1l7Q

    1. @turleo 1y

      9.11 actually looks bigger than 9.9

      1. @ageek 1y

        👍👍

  9. dev_meme 1y

    Adam also missed point about setting activation to 0 for neurons related to biblical verses, which also resolves this comparison problem

  10. @Algoinde 1y

    Removing religion causes LLMs to be more rational? Curious...

  11. @SamsonovAnton 1y

    sort -V

    1. @colllapse 1y

      invokes js

  12. @Algoinde 1y

    🤔

    1. dev_meme 1y

      Important to keep in mind that by default all chats work with non-zero temperature and in general single correct answer is not enough

  13. @Algoinde 1y

    So the issue here is primarily loose typing itself, not the foreign concepts one more confirmation that loose typing sucks

Use J and K for navigation