Nicholas Andresen’s recent post annoys me somewhat because (like almost everyone else) he writes as if “AI” were some kind of “conscious” “agent” that does things with purpose (it starts “In September 2025, researchers released the internal monologue of OpenAI’s GPT-o3 as it decided to lie about scientific data. Here’s what it was thinking…”), but never mind, it’s got a lot of interesting material. He starts with the discovery that “a model’s own output could serve as scratch paper” (this is called “Chain-of-Thought”), and continues:
But something strange is happening to Chain-of-Thought reasoning.
Remember that screenshot we started with? “Glean disclaim disclaim.” “Synergy customizing illusions.” Online, people have started calling this kind of thing Thinkish. There’s a whole emerging vocabulary: “watchers” apparently means human overseers, “fudge” means sabotage, “cunning” means circumventing constraints. Some of the other words – ”overshadows,” “illusions” – seem to mean different things in different contexts, and some combinations resist interpretation entirely.
Weirdly, Thinkish reminds me of home.
I grew up near Gibraltar – a tiny British territory dangling off the southern tip of Spain – where a Spanish-English blend called “Llanito” is spoken. Here’s an example:
“Llévate el brolly que it’s raining cats and dogs.”
To a Llanito speaker, this is completely normal: “take the umbrella, it’s pouring”. To anyone else, it might take a minute to parse – there’s a Spanish verb, borrowed British slang, and an idiom that makes no literal sense in any language.
And Llanito feels great to speak. You’re never stuck – you can always grab the best-fitting piece from each language and move on. It’s just less bueno for everyone else.
That’s Thinkish. The model seems to grab whichever token fits best, optimizing for its own convenience. The problem is that we’re the outsiders now.
He goes on to Alfred the Great’s translation of The Consolation of Philosophy and how languages change, and returns to AI:
Remember: Chain-of-Thought isn’t communication. It’s computational scratch paper that happens to be in English because that’s what the training data was. Whether a human can follow the reasoning has no effect on whether it produces correct answers.
If anything, human readability is a handicap – English is full of redundancy and ambiguity that waste tokens. The evolutionary pressure is toward whatever representation works best for the model, and human-readable English is nowhere near optimal.
That’s pressure toward drift – and unlike with human languages, there’s little pushing back. Old English became unrecognizable despite maximum selection pressure for mutual comprehensibility, though it took a thousand years. Remove that selection pressure and… what? Gradient descent can be very, very fast. […]
So researchers asked: what if we skipped Chain-of-Thought?
What if models could think in their native format – vectors morphing into vectors, activations flowing through layers, all in some high-dimensional space that was never meant for human consumption – and only emit English when they’re done? Projects like Coconut and Huginn are exploring exactly this. Researchers call it “Neuralese.”
Neuralese isn’t like Thinkish. Thinkish is compressed and strange, but it’s still language that you could, in principle, parse with enough effort. Neuralese isn’t a language at all. The reasoning happens in continuous internal latent states, in thinking-that-is-never-words. You can’t read Neuralese. You can only watch what goes in and what comes out.
There’s much more there, and I confess he lost me somewhat as he dove deeper, but there’s plenty of food for thought, and I expect others will get more out of it than I do. Thanks, Trevor!
Llanito reminds me of Diego Marani’s invented language Europanto, which he used in a regular newspaper column and to write a detective novel. It involves snatching the first word from any of the major European languages that comes to mind. As so:
Inspector Cabillot ist el echte europaico fonkzionario wie lutte contra der ingiustice y der mal, por der ideal van una Europa unita y democratica in eine world de pax where se sprache eine sola lingua, der Europanto.
Erat una fria morning de Octubre und eine low fog noyabat las benches des park. Algunos laborantes maghrebinos collectabant der litter singing melancolic tunes. Aan el 200th floor des Euro Tower el Chef Inspector General del Service des Bizarre Dingen, Mr. What, frapped sur the tabula y said:”Dit is keine blague! Appel rapid Cabillot!”.
Given that his writing always focuses on language and translation, it’s unexpected to find that he’s been mentioned here only once that I can see, in 2012.
Ignoring the AI focus, and drifting to Llanito, I recalled a four year old in the outskirts of Sntander, Cantábrica, España, around 1970. Her father, a newly minted PhD from a family of montañés scientists was, of course, a native Spanish speaker. His wife, the child’s mother, was an English woman. Their household was tri-lingual: español, English, and the childrens’ economical blending of the previous two.
E.g., Mamá, quiero más pan con butter.
Butter was shorter and easier for her to say than mantequilla.
PS- is there a term for people who anthropomorphize computer programs?
Other than doofus?
.
Don’t anthropomorphize computers. They hate it.
(I didn’t make that up.)
is there a term for people who anthropomorphize computer programs?
I’m afraid people in general anthropomorphize everything they possibly can.
Don’t anthropomorphize computers. They hate it.
And watch what you say about them.
(I didn’t make that up.)
Reminds me of Sidney Morgenbesser’s celebrated (though, I suspect, mythical) question to B F Skinner, the celebrated unChomskyan: “So you think we shouldn’t anthropomorphise* people?”
* Morgenbesser presumably actually said “anthropomorphize”, but I have transcribed his words into Proper English for clarity.
Skinner was the frying pan, out from which people jumped onto Chomsky.
This presumes you’re already competent in at least one other language. There’s no evidence LLMs are competent in the cognition humans acquire alongside* acquiring language.
* Trying to be neutral as to whether thought precedes words.
I found Andresen not just annoying, but mostly bollox — to use a handy Llanito term.
Don’t anthropomorphize computers. They hate it.
A favorite of mine. I once used that formulation to suggest to a Wikipedian that they not refer to the bot they’d created as “she”, and they replied, “I know. That’s why I do it!”
Around here, speaking la mitad in English and half en español is called Spanglish in English and mocho ‘mutilated’ in Spanish, and speakers can use whichever word comes to their mind first.
This is an account of a paper in a journal published by the Royal Society of Chemistry, no less. It goes far, far beyond suspiciously-looking-like-AI. More here.
I won’t quote any of it. It might kill me.
“ As the world of scientific discovery continues to evolve, these insights serve as beacons guiding researchers toward the development of innovative therapeutic interventions and novel treatments, poised to revolutionize our approach to complex health challenges. These recent studies exemplify the transformative impact of computational methodologies in advancing drug discovery. With a keen focus on drug–target interactions, antiviral strategies, cancer therapies, and diabetic complications, these investigations collectively contribute to a deeper understanding of intricate biological processes. As scientific research continues to evolve, these studies underscore the role of computational tools as vital companions to experimental approaches, paving the way for a more efficient and effective drug discovery landscape that holds the promise of improving global health outcomes.“
The above was published by an entity headed (HAH!) by a fellow who shows little interest in science. It appears to be the result of numerous chatbots engaged in unprotected copulation, a veritable orgy of algorithms.
Thanks @Y (not)
I’m currently in Taiwan. Every business with aspirations to seem cosmopolitan has trite phrases like “unprecedented possibilities” scattered through their signage and marketing material (rather often mis-spelled or using incompatible parts of speech). I presume it just doesn’t matter because nobody’s reading the English anyway.
I’m feeling Andresen’s anthropomorphising is increasingly pernicious: Chain-of-thought appears as a term of art with no evidence thought is involved; manipulating tokens of CofTs might seem superficially to resemble what humans self-describe as ‘reasoning’, again I see no ghost inside the machine.
Previous thoughts regarding Skinner and Chomsky.
This crap would grate in the context of an ad for a used-car lot
If I were buying a used car, I’d prefer the possibilities to be precedented, thank you very much.
Just happened to come across this:
https://worldschoolbooks.com/moore-language-an-in-depth-guide/
Every single example is wrong, in the sense of not actually being in Mooré at all (or in any actual language at all, as far as I can see.)
It falsely claims that Mooré has noun-class based grammatical agreement (it actually has no gender distinctions of any kind, not even “natural” gender), but that is really the least of the problems.
It’s plainly generated in its entirety by a LLM. It was on the first page of a search (via DuckDuckGo) for “Mooré grammar.”
Presumably it generates a parallel piece of grievously mangled plagiarism for a search on “X grammar”, where X is pretty much anything it can find on Wikipedia.
(Unwise to click any of the links offering to download the app, I would think.)
Ah, for the good old days, when searching for “Mooré dictionary” (or ditto any language) would give you a couple of pages of links to dictionaries and triply-plagiarized dictionaries defining “Mooré” (or saying that they have no definition).
That spread of topics is almost as wide as those of the spam “conferences” I get “invited” to. The spam “journals” I get “invited” to have broader topics still…
Preaching to the choir here but, as almost everyone has said, generative AI can’t think.
Whatever this shorthand “chain-of-thought” stuff is, to the AI it’s just a sorting step of words that mean nothing to it, a rearranging like in a con man’s shell game with no ball underneath the shells, a programmed intermediate step it blindly takes as it blindly cobbles together some more verbose slop to trick readers.
it’s maybe sort of comparable to the decision trees a chess computer produces – IF the rules of chess and chess notation were arbitrary and changed with every game.
AG: … IF the rules of chess and chess notation were arbitrary and changed with every game.
George Alec Effinger wrote a science fiction story about this, except I think things changed every single ply. It was collected in his book Idle Pleasures, an anthology entirely of sports stories. The chess story probably wasn’t the best, and I don’t even recall it’s title. (There are a couple stories in the table on contents that I just can’t place.) However, some of the writing was quite entertaining. The story illustrated on the cover, “From Downtown at the Buzzer,” about playing basketball against hivemind aliens, was particularly good. The hockey story, “Breakaway,” where the game is played on a miles-across patch of a frozen moon like Calisto, was also good, if rather grim.
Andresen appears to be deeply credulous about LLMs, judging by the few posts in the archive. For example:
There’s some transhumanist nonsense in one of the others. I think he’s swallowed the whole package. I see all of his posts are also on LessWrong, the Yudkowsky thing.
Calvinball?
I see all of his posts are also on LessWrong, the Yudkowsky thing.
Good lord. Sorry I posted him!
The chess story is apparently “Heartstop,” first published in a Marvel Comics anthology mag. Maybe I will reread it.
I, for one, think it was an interesting article. “Chain-of -thought” is not a bad analogy and people on this blog know all too well that everyone always speaks Llanito, just grabbing the first word that comes to mind (most people who speak more than one language are more careful mind-calibrators not allowing a word from some other language to pop up first). When someone says that computers “play” chess nobody interjects that computers couldn’t possibly have internal state exactly like when humans play. Outer appearance is good enough to use the same word.
Perhaps us dumb bags-of-blood are approaching these bots all wrong: pose the q in Llanito, or Thinkish.