ArXiv Paper: Language Symmetry Shapes LLM Representation Geometry

Description

A screenshot of the first page of an arXiv preprint (arXiv:2602.15029v2 [cs.LG], 25 Feb 2026) titled 'Symmetry in language statistics shapes the geometry of model representations' by Dhruva Karkada, Daniel J. Korchinski, Andres Nava, Matthieu Wyart, and Yasaman Bahri (affiliations include UC Berkeley, EPFL, Johns Hopkins, Google DeepMind). The two-column page shows the abstract and introduction, arguing that LLM internal representations form striking geometric structures - months organized into circles, historical years on smooth one-dimensional manifolds, latitudes/longitudes decodable with linear probes - because symmetry in pairwise co-occurrence statistics governs the manifold geometry of word embeddings. Bullet points cite findings on circular representations for cyclical concepts (days of week, months), 'rippled' 1D manifolds for continuous sequences, and linear decoding of spatiotemporal coordinates via Fourier components. Interpretability research catnip: formal confirmation that the model literally learned that time is a circle

Common Tags

Llm MachineLearning AIResearch DeepLearning NLP NaturalLanguageProcessing NeuralNetworks ExplainableAI ComputationalLinguistics

Comments

Anonymous ★ Top Pick Billions of parameters and gradient descent later, the model independently rediscovered the clock face - peer review pending on whether it also learned the calendar has a Q4

Anonymous ★ Top Pick

Billions of parameters and gradient descent later, the model independently rediscovered the clock face - peer review pending on whether it also learned the calendar has a Q4
Anonymous

The stochastic parrot didn't understand the calendar; it just accidentally learned Fourier analysis.
@saniel42 5mo

ok
@radiroma 5mo

Thanks for the dose of horrors beyond our comprehension before sleep
@tumut 5mo

the caption but unironically
@sysoevyarik 5mo

year 2026, ai bro just discovered how embeddings work
dev_meme 5mo

so what?
@tema3210 5mo

Same as in our brain, just access mechanism differs. So what? I cannot torture a non human being?
@Algoinde 5mo

Wasn't roughly this known since like simple GPTs and tokenized word banks? You could find a dimension in weights of words which effectively encodes "gender", and then take a vector in that dimension from "brother" to "sister", then apply the same vector to "father" and get "mother"?
1. アレックス 5mo
  
  This is the latent space. Theoretically embedding just allow you to map words to numbers and plot distances between them. This is how the words are used in internal reasoning. LLM’s have learned from training data, for example, that the year is a circle, even though that doesn’t make any sense on a surface level.
  
  @Algoinde 5mo
  
  need to actually read le paper this dropped today too https://arxiv.org/abs/2603.05122
2. dev_meme 5mo
  
  Not to this extent
  
  @tema3210 5mo
  
  What's so special this time?
  
  @PatiHox 5mo
  
  😳
  
  @PatiHox 5mo
  
  Oh, hi there!
@Ihor3056 5mo

You expect me to read all this shi?
@offensive_otter 5mo

That’s just a fucking statistics.
1. dev_meme 5mo
  
  First you say "that's just a fucking tweet", now this, how do we even make you believe things? You gonna listen to whispers of sirens in a forest? (I'm not an FBI agent, trust me on this)
  
  @offensive_otter 5mo
  
  Believe in what exactly? That there’s some kind of divine or sentient entity behind these statistics?
@offensive_otter 5mo

If our month names weren’t cyclical (if after December there’s wasn’t January but rather some shit like Elevenary, then Twelvery, and so on), then I bet the months names would’ve formed a linear manifold, same as years.
1. @azizhakberdiev 5mo
  
  that's like saying earth orbit isn't circular, but a spiral and it's never crossing same point twice. It's true, but is that what matters?
@deadgnom32 5mo

but stochastic processes usually form patterns. what's so surprising?
@mmikita 5mo

so, LLMs being trained to be stochastic parrots of human data and mirroring ways that humans imagine continuity becomes a proof of them memorizing with understanding… how, exactly?
Егор 5mo

Nothing sensational in it. Still a stochastic parrot.
@azizhakberdiev 5mo

For more enlightment I suggest actually trying to read how even months have originated. People always knew cycle of seasons, however to track them more accurately they couldn't rely on just days because it would cause deviations. Agricultural societies started using lunar observations to track the months, and calculated that 12 months roughly fit into a whole solar year (notice how solar years are still a superior measure)

Use J and K for navigation

ArXiv Paper: Language Symmetry Shapes LLM Representation Geometry

Description

Categories

Common Tags

Comments

More like this

Usage Reset Joy, Then the Five-Hour Cap Returns

Stuffing Kimi K3 2.8T Into a 3090

Frontier-Lab Agent Intrusion, Five Days Unseen