Nicholas Andresen’s recent post annoys me somewhat because (like almost everyone else) he writes as if “AI” were some kind of “conscious” “agent” that does things with purpose (it starts “In September 2025, researchers released the internal monologue of OpenAI’s GPT-o3 as it decided to lie about scientific data. Here’s what it was thinking…”), but never mind, it’s got a lot of interesting material. He starts with the discovery that “a model’s own output could serve as scratch paper” (this is called “Chain-of-Thought”), and continues:
But something strange is happening to Chain-of-Thought reasoning.
Remember that screenshot we started with? “Glean disclaim disclaim.” “Synergy customizing illusions.” Online, people have started calling this kind of thing Thinkish. There’s a whole emerging vocabulary: “watchers” apparently means human overseers, “fudge” means sabotage, “cunning” means circumventing constraints. Some of the other words – ”overshadows,” “illusions” – seem to mean different things in different contexts, and some combinations resist interpretation entirely.
Weirdly, Thinkish reminds me of home.
I grew up near Gibraltar – a tiny British territory dangling off the southern tip of Spain – where a Spanish-English blend called “Llanito” is spoken. Here’s an example:
“Llévate el brolly que it’s raining cats and dogs.”
To a Llanito speaker, this is completely normal: “take the umbrella, it’s pouring”. To anyone else, it might take a minute to parse – there’s a Spanish verb, borrowed British slang, and an idiom that makes no literal sense in any language.
And Llanito feels great to speak. You’re never stuck – you can always grab the best-fitting piece from each language and move on. It’s just less bueno for everyone else.
That’s Thinkish. The model seems to grab whichever token fits best, optimizing for its own convenience. The problem is that we’re the outsiders now.
He goes on to Alfred the Great’s translation of The Consolation of Philosophy and how languages change, and returns to AI:
Remember: Chain-of-Thought isn’t communication. It’s computational scratch paper that happens to be in English because that’s what the training data was. Whether a human can follow the reasoning has no effect on whether it produces correct answers.
If anything, human readability is a handicap – English is full of redundancy and ambiguity that waste tokens. The evolutionary pressure is toward whatever representation works best for the model, and human-readable English is nowhere near optimal.
That’s pressure toward drift – and unlike with human languages, there’s little pushing back. Old English became unrecognizable despite maximum selection pressure for mutual comprehensibility, though it took a thousand years. Remove that selection pressure and… what? Gradient descent can be very, very fast. […]
So researchers asked: what if we skipped Chain-of-Thought?
What if models could think in their native format – vectors morphing into vectors, activations flowing through layers, all in some high-dimensional space that was never meant for human consumption – and only emit English when they’re done? Projects like Coconut and Huginn are exploring exactly this. Researchers call it “Neuralese.”
Neuralese isn’t like Thinkish. Thinkish is compressed and strange, but it’s still language that you could, in principle, parse with enough effort. Neuralese isn’t a language at all. The reasoning happens in continuous internal latent states, in thinking-that-is-never-words. You can’t read Neuralese. You can only watch what goes in and what comes out.
There’s much more there, and I confess he lost me somewhat as he dove deeper, but there’s plenty of food for thought, and I expect others will get more out of it than I do. Thanks, Trevor!
Llanito reminds me of Diego Marani’s invented language Europanto, which he used in a regular newspaper column and to write a detective novel. It involves snatching the first word from any of the major European languages that comes to mind. As so:
Inspector Cabillot ist el echte europaico fonkzionario wie lutte contra der ingiustice y der mal, por der ideal van una Europa unita y democratica in eine world de pax where se sprache eine sola lingua, der Europanto.
Erat una fria morning de Octubre und eine low fog noyabat las benches des park. Algunos laborantes maghrebinos collectabant der litter singing melancolic tunes. Aan el 200th floor des Euro Tower el Chef Inspector General del Service des Bizarre Dingen, Mr. What, frapped sur the tabula y said:”Dit is keine blague! Appel rapid Cabillot!”.
Given that his writing always focuses on language and translation, it’s unexpected to find that he’s been mentioned here only once that I can see, in 2012.
Ignoring the AI focus, and drifting to Llanito, I recalled a four year old in the outskirts of Sntander, Cantábrica, España, around 1970. Her father, a newly minted PhD from a family of montañés scientists was, of course, a native Spanish speaker. His wife, the child’s mother, was an English woman. Their household was tri-lingual: español, English, and the childrens’ economical blending of the previous two.
E.g., Mamá, quiero más pan con butter.
Butter was shorter and easier for her to say than mantequilla.
PS- is there a term for people who anthropomorphize computer programs?
Other than doofus?
.
Don’t anthropomorphize computers. They hate it.
(I didn’t make that up.)
is there a term for people who anthropomorphize computer programs?
I’m afraid people in general anthropomorphize everything they possibly can.