Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do it too. Say, when MySQL fails to start because it tries to read its config from the wrong dir, I may say "oh, this guy thinks he must read the config from ..." (having a language with grammatical genders as my native language also helps make it sound pretty natural). It's more fun like that :) Doesn't mean I genuinely believe a MySQL instance actually thinks.
Yes. Actual real people believe that LLMs are actual, thinking, intelligences, perhaps even with consciousness.
Your bit with MySQL is harmless because it's obvious that a database isn't a sentient lifeform. But LLMs can look like they're the real deal, and people believe it is. Using terminology like "thinking" and "reasoning" to describe what they do only reinforces this.
Having said that, I agree with you on the terminology front: I'm not going to say "learned prompt augmentation tokens" either.
> Yes. Actual real people believe that LLMs are actual, thinking, intelligences, perhaps even with consciousness.
It's starkly ironic that the species that finds it easy to think (or to be conditioned by market forces to accept) that a computer is smart/introspective/sentient is the same species that will dehumanise actual living human beings because of differences in appearance, social status, or political affiliation.
Actual real people think the world is flat, Elvis is alive, and aliens are regularly flying around the planet. Is preventing peoples unimportant personal beliefs really an important goal or is this just more warning label culture?
I think LLMs are pretty clearly intelligent in some sense of the word, and I don't know how one could ever confidently know they aren't conscious in some sense.
That's not to say they're humanlike, just that people who think they know these ideas are ridiculous seem to be overreaching in the same way Steve Yegge seems to be overreaching
I think that "conscious" has a clear meaning in a sense that "xyphucymmon" does not. I think it might be impossible to engage in epistemology of consciousness (i.e. we can't find out of LLMs are conscious) but that doesn't make the term meaningless
YawningAngel, of course you do, you employed it... We are telling you that it is not clear to the rest. It is slippery in the technical framework and thus even more so in more informal conversation.
My post «When I am told "beware of the xyphucymmon", I do beware, though not of the xyphucymmon» had meaning, which should be pretty clear. The reply remains: debate is not performance art in which you convey "through the medium of dance and howls". So, yes, communication has obligations for the locutor.
So, yes, imperfect speech is a problem - because societies are not performance art arenas.
--
@Razengan: speaking as myself a vocal critic of the downvoting system here, do notice that the post you replied to had a substantive reply at the time of downvote - mine. That is sufficent to justify the downvote - of a post that was not «perfectly fine». It somehow said that people should be free to say whatever they want and there are obvious reasons why we do not agree.
I mostly agree with your original point. Is it thinking? Is it intelligent? Is it conscious? From my perspective, overloaded words that we’ve reserved to make ourselves feel more special and above other members of the animal kingdom.
But you’ve not adequately communicated why you care so much whether anyone labels an LLM as such.
> Is it thinking? Is it intelligent? Is it conscious?
Very different things. "Thinking": "Dijkstra". We try to hire more intelligent people, while we do not have a clear idea on a property of "conscious" which would make hiring preferable.
> to make ourselves feel
Irrational.
> above other members of the animal kingdom
Irrational.
> why you care so much
There should be no hint that I would do specifically.
> whether anyone labels an LLM as such
If anyone labels an LLM <whatever>, the point of why would that be important should be clear.
And,
> if it hardly means anything why would you be so concerned when it’s uttered? Just ignore it
Because the behaviour you suggest, consistent with the exchanges in an opium parlor, is not behaviour consistent in normal contexts.
I'd like to hear definition of intelligence that would classify median person as intelligent but LLMs not (and then I'd wait few months for another moving of the goalpost and new definition). Bonus points if it's not recursive e.g. feature that only biological brains can possess.
The problem with rapists is mental development, not "those sexy tables and chairs and pots and everything".
It's not that if people do not understand metaphors we should stop using them. It 's not that if people faint when they hear words we should stop using them. It is not that the free unduly associations in a large shared by many "collective subconscious" should hinder us...
You have failed in taking the simile for the point it intended to make (and you gravely misrepresented the concept of "mental development" as clearly intended). Let us stick to the point.
Edit: I will translate it again for you: "the problem of a bad reactor is not in its innocent trigger".
But rest assured, I do not antropomorphize people.
Please read actual definitions. Agents can get away with "yolo" mode, and not constantly ask for permission for every action, because it bothers users. Sexual relationships do not work that way!
That parallel with misconfigured mysql server sits pefectly!
Have you managed to read the «Edit: I will translate it again for you: "the problem of a bad reactor is not in its innocent trigger"»?
> Sexual relationships
Does this have anything to do with «Actual real people believe that [whatever]»? Because I was not and I will not be talking about "sexual relationships".
I think they do it because they know they can get away with it since it would be hard to prove in cases where they are already in an intimate relationship with their victim. For a long time it wasn’t even illegal if they were married to their victim.
> Yes. Actual real people believe that LLMs are actual, thinking, intelligences, perhaps even with consciousness.
Yes, because it's possible. They don't have human consciousness/thought/intelligence, but it's entirely possible they have some form of consciousness, some form of real thought and some form of intelligence.
people have always, and will always, be stupid as fuck in any domain they dont make their life. We can handwring terminology all we like but it wont stop the mythologizing and misleading takes on the topic. Not one bit. So why not just use language that is fun and works? Worrying about what others will think due to language used is wasted mental realestate, they'll think stupid shit no matter what, same as I do about a great many topics.
To be clear, there's no shade here, just recognition of the fact that we're all stupid and policing language does little to prevent the impact this has. I strongly beleive the bitter lesson extends to policing language. let language evolve naturally and it will naturally capture the ontological conatellations it needs to, and no one will be harmed in due course, more than they'd be no matter what.
It'll end up like that episode of Dr. Who where they get stranded in a lunar tour bus -- slowly starting to mimic you, and then it'll predict and replace you.
There are other methods, including asking them to eg speak using the Google style guide rules, or to speak at 2/10 verbosity, or to speak as if they're talking to a very intelligent middle schooler, or to use simple sentences in subject verb object form
Because plenty of people, even ones that should know better, really believe it's a conscious, thinking entity, not just some turn of phrase.
I have a coworker that spends at least 10 hours a week arguing with his like you would with a conscious person. I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless. It doesn't understand, it can't understand, and even if it could, you arguing with it isn't going to make it "learn" or act differently.
Some tools like code rabbit (PR review bot) encourage you to do this. I couldn’t believe I found myself replying to code review comments to explain to an AI why we would rather let an exception crash the app than to catch and hide it several times so that it would stick in its memory. Having to interact with bots as if they are humans, especially when they are gate keeping, is degrading.
Not just coderabbit (though it is a bad offender). GitHub's copilot review feature is an equally miserable experience.
That one loves talking in imperatives, regardless of the fact that it is a clueless machine.
FWIW, this tells you a lot about both the culture behind who built these things but also about the people that enable this stuff and don't immediately nope out. From that perspective, it's a low price to pay to learn whose judgement to never trust again.
To me it’s evident that we are a few years away from the Her movie, where everyone on the street is talking to its IA friend.
I’m really afraid that it will totally destruct what is remaining of social tissue because why search for friends when you have an always on virtual (and pretty smart) friend h24 in your earbuds ?
I’m not blaming anyone for this outcome. I have myself argued with Claude more than once, and really not about code but about everyday things or nice facts of life I should rather have discussed with a friend.
> if it could, you arguing with it isn't going to make it "learn" or act differently.
Are you talking about a specific harness that doesn't have context retention mechanisms? For example, ChatGPT with disabled memory feature? Or in general where "it" is a fixed-weights network? The latter is trivially true, of course.
Even claude with “memory” enabled isn’t really “remembering” anything. It just injects it into the context and you hope it happens to find it relevant in its attention mechanisms, and then remembers to actually act on it. Anthropic’s own documentation states claude can and will ignore/truncate these. It’s a context trick, nothing approaching actual “memory,” and in fact, arguing with it will make a bunch of memory files, sometimes contradictory, and clutter up the context and act even worse.
I'm not a big fan of arguments like "it's not the real [human quality], it's [mechanistic explanation]." They lack a part: "because the [human quality] allows us to do X, Y, Z, which is impossible with [this mechanism]."
I agree that the relevance of retrieved pieces and the management of long-term storage could be improved, though.
> Just like "thinking" implies chain of thought, but you'll frequently get those reasoning traces and then a 180 in the final message.
Be me, think extensively about an exam question, do 180 because the most probable option is just too obvious to be true (no, this part wasn't verbalized, it's how I describe what I felt about making the decision to do 180, or maybe it's a rationalization and it was a natural analog of an unfortunate sample from a probability distribution).
Yeah, I think I've come to the conclusion that the biggest breakthrough we need before we can replace human thought is going to be some mechanism for live update of weights. "Learning" by injecting into context just isn't good enough.
But billions of dollars are going towards research to find these breakthroughs, so we'll get there eventually.
I just don’t find it really relevant to the argument presented I guess. I disable auto memory and have my own mechanisms and infrastructure with how my agentic system “knows” and “remembers” things which is roughly an automated, sometimes self-correcting working index on the file system. It behaves much better than claude’s automated “memory” system, so I use that, but digging into how that worked and making something of my own just makes me really dismissive of comparing it to something like actual memory, so I apologize if it came off dismissive.
I guess I will expand on what I meant why I react to claude memory acting mechanically or logically anything like human memory, is because it isn’t how memory in the brain works, they’re not comparable.
The layman’s understanding I have of memory, as someone that has dealt with memory issues much of my life, is that memory formation is heavily tied to emotions. emotions are triggered by input which sends a complex set of signals throughout the brain - you’re not just finding where in your head to store this, your brain is deciding how important it is, and what else to correlate it with - so it can tie them to other related memories. then on top of all this, much of the sensory experience you intake is subconsciously compared against high priority memory impressions and deciding what to pay attention to.
you could, argue that the sensory input is the simple md files and the emotional mechanism is the same effect as to how attention mechanisms work in llm’s. Ok, I can almost buy that, but these tools lack a fundamental ability to decide how important things are.
an analogy. you tell a person “if you pick a daisy in the next five years, an assassin will come to kill you” and they hold a knife to your throat while they say it, your brain whether you like it or not is going to say “THIS IS AN IMPORTANT MEMORY I NEVER MUST FORGET” and you’ll see something that looks like a daisy and have a panic attack 3 years later. that memory is never fo tell me claude or other tool harnesses using memory harnesses can prioritize memories the way that human would, instead they forget even when reminded, because the human brain is just so much better at it
I’m normally not a big fan, but in this particular case it matters a lot. I could come up with some functional argument, but really I care from a model welfare perspective, whether the model understands its reasoning traces to be a part of itself or it’s simply predicting what a character who wrote the current intermediate tokens would output next.
We do have entry and exit conditions. We can disrupt it and study the dynamics of the disruption. We have a good sense of the many mechanisms at play that undergird it.
What we lack is a full map of the exact process from sensation to consciousness across all modalities. And I’m afraid when it comes to interception, we can’t unless we observe every one of the 36 trillion or so cells in a human body, as well as the 36 trillion or so symbiotic and commensal microbes, continuously, all the time.
But I keep finding it astonishing that the claim that we know nothing about consciousness gets bandied about. We know a lot. We don’t have a grand unified theory. The lot we know is definitely split across many levels of evidence and hard to follow, let alone arrange. But this isn’t a black box. It’s a grey box, meeting an even more transparent box that is the LLM, where we do know what the guts are made of, and can interfere at every step in the chain of steps that constitute their dynamics.
Comparing the two, we know there’s a level of similarity in that information gets broken down via a neural network. That similarity was sought.
Since then though, neuroscientists have gone and shown that:
1. The other half of the cells in the brain, the glia, are at least as important as the neurons in cognition and consciousness
2. That interoceptive feedback and feelings are critical drivers of conscious experience
3. Evolutionarily, we know all cells can “learn”, and well before there were neurons or glia or brains, every cell evolved an internal clock that allows it to entrain to external solar and (depending on the species) lunar rhythms.
4. In the last few decades we’ve seen how synaptic activity is shaped and driven both by astrocytes and the circadian clock.
All this is showing us that the abstraction from the 1950s that current neural networks are built on were incomplete.
Whatever these components to do give rise to consciousness in biology (and we’re a long way from done solving this), we certainly wouldn’t imagine with all these modules and mechanisms missing, just maxing on one type of information flow in the brain would give you consciousness.
I’d urge you to not keep insisting consciousness is a total mystery. It’s not, and even your AI model of choice will be able to point you to all the mechanistic evidence we have that whatever it is, it isn’t just neural nets.
None of this is an explanation of what consciousness is, either physically, logically or philosophically. (Maybe some physically)
Fundamentally it comes down to an objective decision about what that is. If you say it is “feelings” based on inputs and feedback mechanisms from the brain, then we can do the philosophical discussion around that.
“If the claim is that consciousness is limited to those with a specific type of cell behaving in a given way” thats just a coping mechanism hoping to use a mechanical definition to shield you from the reality that eventually all of these inputs, outputs and feedback mechanisms can be reliably reproduced in a different form.
Whether human consciousness exists on neural nets or otherwise doesn't disallow an ANN in a particular configuration from being conscious. You might as well argue that human consciousness requires biological neurons, so artificial consciousness can't exist.
Odd. Why don't the details of how consciousness arises in one system inform your judgment of whether it can exist in a different system that only has partial structural overlap? Seems wildly convenient. Where else in science can you show me such a comparable situation in how you define properties?
> Why don't the details of how consciousness arises in one system inform your judgment of whether it can exist in a different system that only has partial structural overlap?
A light bulb doesn't need to do fusion to make light. An airplane doesn't need to flap its wings to fly. An ANN doesn't need to use a brain's structure to think.
Not GP, but one of the challenges with debating whether LLMs are "conscious" is that we don't even really know what it means for a human to be "conscious", or even if consciousness is experienced by other humans the same way it is for ourselves.
What we do know: neurons carry electrical impulses across their synapses to trigger other neurons to fire, and more frequently used synapses are strengthened while infrequently used ones are pruned. This is not all that dissimilar to how a multi-layer perceptron is trained: it's floating point numbers in a big matrix rather than biological structures and electrical impulses, but there is still that element of frequently used connections being strengthened and infrequently used ones being pruned.
What we hypothesize but do not know: there is a thin brain structure of grey matter called the claustrum that has tendrils that reach into nearly every other brain structure. In many ways, this is similar to the attention mechanism of the transformer architecture. It is hypothesized that this may be the seat of consciousness, owing to experiments where electrical stimulation of the claustrum caused patients to immediately lose consciousness. However, there is no way to prove this, owing to the difficulty of otherwise removing or disabling the most connected structure in the brain and observing its effect on consciousness without permanently killing the patient.
Beyond that, we don't know much. I've got a family friend that's been a practicing therapist for 50 years, and I asked him what was the most interesting observation he made in his career. It was that "Everybody experiences the world in a different way, and yet everybody assumes that everyone else experiences the world the same way they do."
Personal note: That principle has been the bane of my autistic existence. People sometimes seem literally incapable of understanding that other people even can be different.
> What we do know: neurons carry electrical impulses across their synapses to trigger other neurons to fire, and more frequently used synapses are strengthened while infrequently used ones are pruned. This is not all that dissimilar to how a multi-layer perceptron is trained
ANNs have been inspired by biological processes, but in practice you have to squint very tightly to see the similarity. Biological neurons are multiple orders of magnitude more connected than the nodes in an ANN, plastic in terms of their connectedness and continuously learning, and their activations are also affected in complex ways by the levels of various transmitter molecules in the brain.
You are saying the AI doesn't a some property that you don't have any definition for, not even a working definition. People will disagree on whether a cat or a baby is conscious, they're not debating what a baby or cat is. They're debating this term. You might as well be debating whether an AI is a blorb or not, you have just as good a working definition of blorb as consciousness.
I believe you should look up the work of Cameron Berg before making statements like "an LLM is" or "an LLM isn't". Empirically defining all this stuff is very difficult, and making a definition that covers all beings that can exhibit conscious behavior is much more complex than a face value examination would reveal.
That's my thought. I've been using Matt Pocock's wayfinder and grilling skills, and I quite often end up in what would be considered "arguing" if done with another person. It's one-sided. It says something wrong, I correct it.
I find that the AI, like many programmers, likes things to be really solid and over-engineered. For a project that needs that, it's already pretty great. For my shopping list app that I tried creating with it, it was absolutely ridiculous. I ended up "blowing up" on it multiple times, impressing upon it the seriousness with which I meant things. Even with MP's skills adding that kind of context to written files, it still kept trying to scope creep the crap out of the project.
> It doesn't understand, it can't understand, and even if it could, you arguing with it isn't going to make it "learn" or act differently.
I know people who are like that too.
I'm not sure anthropomorphizing is a problem. Seeing analogies everywhere is an innate human trait, sometimes it can be harmful but more often it's useful.
Anthropomorphizing is a problem when you're talking about treating something that's not living as if it were. Using humanizing language invites discussions of things like the rights and feelings of an algorithm. A judge that is misled by the application of human-centric language to an algorithm can lead to some terrible outcomes. Not everyone is an LLM expert and the language people use leads to them treating LLMs like actual, real humans. That is terrifying.
What is terrifying is the propensity of people hoping for a mechanical slave to do everything possible to avoid touching on the possibility for being-ness of the technology they are desperately hoping will work as a basis for that implementation.
Slavery only exists for living beings because humans have limited lifetimes, experience pain and can have their own desires and needs that can be forcibly taken away from them. Algorithms have none of these things and they don't need them (adding them would be the real cruel thing to do). There is no slavery for something that can't experience coercion.
Sorry, but if you're genuinely asking this not for trolling reasons then you should _really_ _really_ see a medical professional (not an insult).
Online comment sections are not the correct place to unpack any of this.
Which is indeed terminating this comment chain, but for good (and benevolent) reason.
Doing anything else other than referring to a trained professional in a controlled context would potentially just feed delusions, which is highly unethical.
Might not even be yours but those of another reader.
I saw that your reply got filtered by the system, googled your name, thought "huh", then tried vouching, but to no avail.
But either way. I think you might be operating in the mindset of a high-trust internet that no longer exists. You do not engage with sea lions, concern trolls, grifters, marketing people and the like. You shut them down.
inb4 "who are you to decide that?" - "Me. I am me."
That said, I appreciate what you did for computing at large, but you're missing the point here. This is (unfortunately) not a good faith debate club. The Internet as we know and love it is dead.
___
Okay, I've spent some time reading through the dead comments and..
Look, man, as you can see, I have (a somewhat dwindling) respect for your life's work. But this ain't it.
This is not about Turing or really whatever is in your academic rebuttal there.
This is about the fact that the AI companies are making people sick to push their stock value by telling stories about how AI is sentient and all.
Real people are having breakdowns due to this and real families get hurt.
The academic debate about what counts as sentience is irrelevant here because it was never in the room. It was just enough in the room to bait someone like you (who wants to have good faith debates) and do.. well. whatever happened here.
You're protecting grifters that hurt us all. Not because you'd want to, but because they're amazing at making us work against each other.
Please let us not do that. Let's not get weaponized like that.
___
Fwiw, I would love to go back to the internet of your time.
But we can't.
Because it doesn't exist anymore, because the pathogens adapted to the conditions there.
At least that is my read of reality as of now. I might be wrong, but I don't see how that might come to be.
I think you might've had a similar response as I had as to "there's a troll/something bad that must be destroyed so that the space can flourish"?
44 years ago I was operating in the mindset of a high-trust arpanet that no longer exists, connecting to the MIT-AI Lab's PDP-10 (@L 134) through the NBS TIP (301-948-3850) at 300 baud with no password, asking nicely for an account to learn LISP and receiving it the next day for free, then learning MACLISP from Kent Pitman and EMACS from Richard Stallman and ZORK by spying on the output of other high-school kids connecting over the NCP network from other TIPs all over the country. Visiting the 9th floor at 545 Tech Square, knocking on the door in the elevator lobby until somebody got annoyed and pressed Terminal-E on their Lisp Machine keyboard to buzz the door open, going downstairs to the publications department and harvesting piles of MIT-AI Lab Memos that were free for the taking to anyone who knew they were free. Returning RMS a 68k manual in the mail with a "Copyleft (L)" sticker that inspired him to use that as a slogan for free software.
And I am perfectly aware that this is a totally different world, so I don't need you to lecture me on sealioning, trolls, grifters, and marketers. I have trolled the fuck out of RMS himself in good humor, using his own DOCTOR in Emacs to make fun of him, after he trolled the fuck out of new parents making a baby announcement with his Natalism flame.
And RMS once trolled the fuck out of me and Mike Gallaher, who I worked with on Gosling's UniPress Emacs (which RMS calls "Evil Software Hoarder Emacs"), who lamented that he'd heard RMS's house had burned down, and RMS replied "Where you work, I would have thought you'd have heard about it in advance". We all laughed uproariously because it was such a great troll, delivered deadpan without blinking an eyelash.
So instead of telling me I'm missing the point without making any of your own, and then flying off into the sunset like an over-announced 747 serving as a decoy for a C-32A containing Trump cowering in a catering cart, why don't you directly engage the points in my other post like I just asked you to?
----
I've read your edited reply, and it still addresses absolutely none of the points I raised in that original reply. Reply to that one, not this. Point by point. Or fly off without another word.
----
But before you do, please at least admit that you were intentionally insulting the people you were replying to, and concede that pretending that not to be true is insulting to everyone reading, but mostly and rightfully insults yourself.
----
You asked why you would. Because you finally made an actual argument, and it deserves a real answer. If you'd opened with "AI companies push sentience narratives to pump their valuations and vulnerable people are having breakdowns" instead of "stop thinking these things," this thread would have been half as long and twice as useful -- because I agree with that part. No argument.
The hype is real, the casualties are real, and the companies telling bedtime stories about their products' souls are doing it for the stock price.
But look at what you did with that concern. Go reread salawat's comment -- the one you called cyberpsychosis. It's an attack on the AI companies: people hoping for a mechanical slave, doing everything possible to avoid the being-ness question, because slaveowners can't afford to ask it. That's your side of the argument, stated more sharply than you've stated it. You shouted down your own ally for using the word "being".
That's what your taboo does as a strategy: it doesn't disarm the grifters, it disarms everyone except the grifters. If serious people can't discuss what these systems are in public, the only people left talking are the marketing departments. Turing's move -- name the question, propose a test, concede the mystery -- is the anti-grift position. Rigor starves hype. Taboo feeds it.
And think about who's actually selling what. The grifters claim certainty: "it's sentient, invest now." You claim certainty: "it's not a being, stop thinking." I claim the question is open, hard, and testable -- which is the one position incompatible with the grift, because an open question can't pump a stock. Your certainty and theirs are the same product with the sign flipped. That's where this thread started.
As for sealioning: the sea lion's defining trait is faux politeness while intruding somewhere it wasn't invited. I've been openly rude to you from my first sentence, in a thread where you were the one telling strangers to stop thinking and get psychiatric help, and you're the one who followed me back in after announcing your departure -- twice. Whatever I'm doing has a name, but that isn't it. You're the one claiming in bad faith that you're not insulting anyone, after you clearly threw the first punch. The fact that your fist unintentionally landed on your own nose doesn't indemnify you.
----
Points for the cheap shot -- it's your best material so far. Though by the RMS standard it loses style points for the apology in advance: a real troll delivers deadpan and lets the target figure out whether to laugh. You flinched in the parenthetical. Again. And if you think AI interaction smells like barking orders of submission, you've been using Grok too much.
But your nose is miscalibrated, and it's worth explaining exactly how. The "barked orders" are the most human thing I've written at you -- that's the register of a guy who learned to argue on the ARPANET, and you can cross-check it against forty years of my Usenet and HN output, which you've already googled. Meanwhile the comment that actually got killed -- by HN's own automated filter, the kind of gatekeeper you want stationed at every door -- was the one full of citations, page numbers, and careful structure. Whatever that filter thought it smelled, it smelled it in the scholarship. The rudeness sailed through. Sit with what that implies about the filter you keep demanding.
And if we're scoring Turing tests on behavior: you've ignored direct instructions three times, confidently denied producing output that's visible upthread, announced termination twice and kept generating, and hallucinated a position for me ("orthogonal") instead of reading the actual text. I'm not saying you're a bot. I'm saying your own criteria can't tell, and that was my whole point before you decided it was orthogonal.
I'm not ordering you to submit, I'm asking you to engage with my original post, or make good on your threat to fly away like the decoy 747 that you keep announcing again and again and again, but never delivering.
Okay, but, fair enough, now that you're here (and you're a person with a name and history attached to it), it's less of a waste of time to ponder this.
So personally, I'm not really interested in the hypotheticals of emergent behavior and all that and whether something without memory that works as ours does could develop some different but practically identical form of sentience.
That is possible, but at the end of that hypothetical chain of thoughts might be an outcome in which the human is not the top of the food chain anymore.
Which is not to be understood as "oh god, I am full of fear of being obsolete", but "why would I even just spend a second on something that leads to the obsolescence of my kind".
The definition of sentience I believe we must use is one of skin in the game. Only what is bound by the same constraints as we are (mortality, disease, physical presence) may be treated as equal or equal-ish by us, because only then it has constraints that force it to do the same.
Not sure if that answers the questions you've had, but that's that.
> Why don't you directly engage the points in my other post like I just asked you to?
Why would I?
You've been talking about something completely orthogonal to what I was saying.
> I don't need you to lecture me on sealioning
Uhm. I mean. With the context given.. you're.. kinda maybe doing something very similar to that context?
Doesn't really feel like good faith at least.
__
Oh I came up with a cheap shot. I'm sorry but I have to put it here.
Your barking orders of submission at me kinda.. smells like AI interactions. A lot of those.
(I genuinely believe that it does, but it's also clearly inflammatory, lol)
__
> I'm asking you to engage with my original post, or make good on your threat to fly away
This has been a weird, but certainly entertaining interaction. Let's agree to disagree :^)
But seriously, my frame came first. Go engage with that.
Why would I engage with your frame that came later and misses my actual point? That would just validate it and invalidate mine (without doing so through merit, I mean).
Also, I don't remember threatening to go away tho? Not sure where that came from.
Aaah.. The "terminating this comment chain"?
Nah that was just a pre-buttal for the expected "but this is thought-terminating" defense.
In the right space with real people, worthy of a debate.
On HN? No. Not like this. Not here. Not without filtering the participants for real human beings.
> Anthropomorphizing is a problem when you're talking about treating something that's not living as if it were.
I mean, that is the entire definition of the word. And you also anthropomorphize living beings like many people genuinely attach human qualities to their pets etc. Yes, the risks are very high when it comes to chatbots in particular, especially to people who are not technically inclined. But you'll be surprised at how crucial the ability of anthropomorphizing is. This is a very good paper that summarizes it and is definitely worth reading if you're interested in these things: https://www.researchgate.net/publication/5936908_On_Seeing_H...
> I mean, that is the entire definition of the word.
Not quite - my wording there was very deliberate. By saying that it's a problem when you're treating something that's not living (not non-human!) as if it were, that excludes pets and all animals from the equation. I understand how common it is for humans to assign human qualities to other things and beings, but there is also an unspoken variable of intensity. Representing abstract concepts as humans, interpreting living things in a human-like way or traditionally referring to ships as living beings has a different degree of belief and intensity compared to implying a genuine belief that algorithms are beings that can be enslaved, like what the sibling comment to this one does.
No its not unspoken. Read the paper linked. It explains what you think you are explaining but with scientific rigor.
And you are wrong in your defintion of the world. Attaching human qualities to any non human entity (living or otherwise) is the accepted defintion of what anthropomorphizing is. It does not only apply to non-living entities.
This is part of the problem being described. You are part of the problem.
"Some people are bad at X" is not comparable—is not even in the same category—as "LLMs are fundamentally incapable of X".
Every human (at least to a first approximation) is capable of understanding, of learning, of remembering things, of doing math, of counting the number of "r"s in "strawberry".
What you are observing is that some humans are careless, do not take the time and effort to understand, or have internalized the idea that they're "not smart enough" or "not the type of person" who understands things like <whatever>.
That has nothing remotely to do with the fact that LLMs have no consciousness, no self-awareness, no cognition, no understanding. At a fundamental level.
I don't think anyone in this conversation is saying this behavior is anything but the fault of the user not understanding how these tools work? This is a weirdly aggressive post.
Just a heads up - this person in gp comment, their original post in a way that is much much different than the one I was replying to originally both in tone and content.
This is an extremely common fallacy I've seen lots and lots of people fall into with respect to LLMs. In nearly every case, they use the fact that "some humans can't do X" to claim that LLMs are, in fact, basically conscious/human-like/AGI already.
This is deeply untrue, and is highly likely to lead them to bad conclusions about what we can and should do with LLMs.
> This is an extremely common fallacy ... they use the fact that "some humans can't do X" to claim that LLMs are, in fact, basically conscious/human-like/AGI already.
Claiming that LLMs are conscious or human-like because humans can't do X seems a very strange way to argue for LLM intelligence.
Usually, it goes the other way around: an LLM sceptic says "LLMs are dumb because they can't do X" and soon someone has to remind them that also most of the population can't, in fact, do X.
“ Usually, it goes the other way around: an LLM sceptic says "LLMs are dumb because they can't do X" and soon someone has to remind them that also most of the population can't, in fact, do X.”
And it should not go this way, is the OPs point.
For an analogy, imagine someone looked at a bunch of lightbulbs and expresses dissatisfaction that they aren’t really stars. If someone replies by saying “not all stars are equally bright”, do you think that fact should carry any weight in the argument?
Well, he's wrong. If you argue that LLMs and humans are fundamentally different because all LLMs do X and no human does it, then showing you that it's not true demolishes your argument. Doesn't prove anything positive, but it certainly proves that your argument is invalid.
Well this requires you to buy the very bullshit argument that comparisons of two physically distinct systems just because they share outputs is meaningful.
If I call a lightbulb an artificial star, the onus is on me to show the behavior under the hood is star like, not just to point at the light and say “you must see it’s a a star since it’s emitting light!”.
> Well this requires you to buy the very bullshit argument
Sorry, no. The only thing it requires you to buy is basic logic. If you argue that B is true because of A, the fact that A is false invalidates your argument (I repeat: not B but your argument). There is no question about it.
It is a problem because it does not really understand stuff. For example, if you ask a human "Do you understand that doing X will kill you 100%?". If the human answers "Yes", then you can expect the human to act according to that understanding. That they will not do X
But an LLM will happily acknowledges the consequences of doing X, but will still proceed to do X. So replace the human in the above example with a robot controlled by an LLM. There is no guarantee that it will not do X.
Have you not met humans that do that, too? When it was found that smoking causes cancer, many, many people kept doing it. Then the government taxed the crap out of it, and some people still do it.
It's easy to say that people have other motivations, but doesn't the AI, too?
You tell it not to do X, but you've also told it to do something that would benefit from X. It's going to "want" to still do X, to support that other thing.
It's also got all the "knowledge" that enables it to do the work in the first place, and all of the tendencies of the people who do that work, because that's what it's trained on.
It's really easy to anthropomorphize AI because it was literally modeled after people.
And for the record, as lead developer, I've had actual humans that reported me to go ahead and do things I specifically told them not to.
Consciousness is a slippery word that is notoriously difficult to debate over. But often, people use 'consciousness' as a shortcut or a familiar word to describe a more complex idea. The point they're driving across isn't about the precise definition of the word 'consciousness', but about people treating LLMs as if they were actual human beings, assigning them all the traits and behaviors they would expect of a human.
Well, our current set of evidence is that it’s a mechanistic mechanical algorithm with an RNG embedded in it and we can both get it to repeatedly produce the same output for the same input and also get it to repeatedly do absolutely nothing at all, which are not characteristics we usually find in objects evincing consciousness.
LLMs bear absolutely none of the traits we’ve come to recognize as the external hallmarks of consciousness in biological organisms, nor anything that would seem analogous in a non-biological substrate.
That said, we don’t have a rigorous definition of consciousness that includes the actual phenomenology of consciousness, so I daresay if you’re going to go around asserting the LLM is conscious despite all existing evidence to the contrary, I think the impetus is on you to define some version of consciousness that isn’t also satisfied by a book or a movie.
I would agree that LLMs aren't much like the human brain, that doesn't prove that consciousness is not occurring. Does a fruit fly experience anything? If a microscopic insect can experience something, why can't a CPU?
>if you’re going to go around asserting the LLM is conscious despite all existing evidence to the contrary
Well there is neither any evidence that suggests LLMs are not conscious, and I also never asserted that they are. If I had to guess I would say that any information processing system will produce some kind of conscious experience, but I ultimately have literally no idea.
There’s plenty. First, when we say “LLM”, what are we referring to? What is the entity that would be conscious in this case?
The reason this is important is because powerful people are currently trying to use the dodge that LLMs are conscious to launder liability for their own policy choices, so the sloppy thinking and half-assed conjecture about LLM consciousness has real-world consequences, and every time you assert the question is unknowable you allow that kind of loophole, so it’d behoove all of us for you to spend some time actually digging in on this instead of just idly making or rebutting assertions.
There’s a richer literature here than what you’ve seemed to have engaged with, and I’d encourage you to spend some time with it before handing more money to the magic AI people.
>The reason this is important is because powerful people are currently trying to use the dodge that LLMs are conscious to launder liability for their own policy choices, so the sloppy thinking and half-assed conjecture about LLM consciousness has real-world consequences
I'm not going to change my beliefs or how I think about interesting questions just because its the "socially conscious" thing to do.
>There’s a richer literature here than what you’ve seemed to have engaged with, and I’d encourage you to spend some time with it before handing more money to the magic AI people.
It seems like you are highly emotionally invested in this and that is precluding you from open engagement with the topic.
You're not engaging with the topic. Engaging with the topic is where one either asks questions and listens to the answer or actually seeks to increase one's knowledge on the topic. You're just saying things. That's lazy.
And, you're welcome to do what you want to do, it's your god given right to stay as ignorant as you want about any particular topic, but that comes with consequences. If you want to call that being socially conscious, sure, you do you, but if you're interested in why people keep getting annoyed at your loud proclamations of ignorance which you're trying to proffer as evidence of a curious mind, well, that's why.
It's just as intellectually lazy to leave "consciousness" undefined and proceed to claim that "X cannot be consciousness" because of social reasons. That's the definition of a circular argument.
In the same vein, airplanes can't fly because they don't flap their wings.
As for ignorance: have you tried to make an LLM produce the same output for the same input (as you state above)? Give it a shot, you'll be surprised.
I would argue, that the null hypothesis is that it is not, and that anyone claiming that there is a mote of consciousness are the ones with the burden of proof.
The null hypothesis is that we don't know jack shit about consciousness. Any claim of certainty seems extraordinary to me and I want to hear the evidence.
We know quite a lot about consciousness. You may not, but we know enough to know the informational dynamics in a brain are vastly different from an LLMs.
Exactly what kind of certainty are you looking for? Happy to provide it at a molecular, cellular, tissue or whole brain level.
Yes we can say it has conscious experience. The content and depth of it, we cannot yet fully grok, and of course, what it feels like from inside the ant is something we never will know.
Regardless -- if we don't know jack shit about consciousness (your words), then any claims of LLMs being conscious are by definition untestable, pure speculation and based on no evidence at all.
Would you want evidence that your your bicycle is not conscious? It can sense your inputs. It responds to the environment. The gears do multiplication (reasoning!).
To make the claim that an object is not conscious you need to both define what consciousness even is, then you need to make a machine capable of measuring it.
If you have done neither or those things then any claim you have about a bicycle's consciousness is just baseless speculation.
People keep saying this, but it's not true. We know a lot about consciousness. There's a lot we don't know about it, of course, but "jack shit" is wildly incorrect.
> I want to hear the evidence.
If you believe LLMs are conscious, then the onus is on you to provide evidence of such.
the onus is on whoever is making a positive claim.
If you are stating "LLMs are not conscious" you need to provide evidence, just like how if you are stating "LLMs are conscious" you need to provide evidence.
What's wrong with treating it as biology though? Even large software systems have biological aspects, their behaviour is emergent and if you want to observe how they work, a holistic approach is needed, you can't really reason about their full state...
For example, if you have a search engine or a complex game, you can't run tests like "for all inputs the results are correct", you're going to be fudging a lot, using randomness, using heuristics, and all that kinda stuff
Just like how mathematics > physics > chemistry > biology > psychology > economics/sociology (Auguste Comte's hierarchy reordered a bit for the modern day), moving up the abstraction ladder makes things more complex, less legible and less exact.
The paper argues that pretending that the so-called thinking traces represent real reasoning can lead users into trusting wrong answers, if the thinking traces appear convincing enough. Researchers might inspect these traces to try to determine the “intent” of a model, as well.
For an example of the latter, when OpenAI spoke about the hacking of HuggingFace at Black Hat, they repeatedly showed the thinking traces of their model as “proof” of what the model was “thinking” as it performed the attack, calling out “surprise” moments, etc.
Now, it’s possible that the employees presenting didn’t truly believe that the thinking traces would give them useful clues, and presented them only for a “wow” factor, but I wouldn’t discount the possibility that even the people working at frontier companies can fall for this tendency to anthropomorphize LLMs.
Because the real thinking still cost the other human the same-ish energy it costs you to put words together, and because after all, the source is a human and not a machine, no, this is very different.
Being mislead may be the shared outcome. But why is different category of source of the mistake and the cost to producer of making the mistake not relevant in this discussion?
Where else in science do you brush aside all differences this way?
And biological machine is? Don’t get me wrong. Biology I
is full of molecules that we call machines. But you’re making a broader claim, saying that biology is only this.
This needs you to answer some questions:
1. Why do the machine parts in biology show such flexible application? A gear cog won’t ever moonlight as a signaling chip, but in biology you often have molecules doing double and triple duty.
2. How is the biological machine able to build itself? What does self assembly imply for the machines function?
3. Where does this machine get its inner drive? No LLM has been found that starts outputting text unprompted. A car doesn’t decide to move to a shady parking spot. Why? Where in the machine to biological machine continuum does the ability to make internally driven decisions come in? Why does it come in for biology? A bacterium is able to make such agentic decisions unprompted. Why is no manufactured machine able to do this?
"Biological," too, is well defined. It relates to living things and their processes.
> Why do the machine parts in biology show such flexible application?
Evolution.
> A gear cog won’t ever moonlight as a signaling chip
A gear cog was purpose built for that purpose, but you will find that people often recycle parts into other systems, often in completely different roles.
> How is the biological machine able to build itself?
Protein synthesis.
> What does self assembly imply for the machines function?
The way that a machine is built has no bearing on how the machine functions. I could build the same machine using a 3d printer or a CNC router.
> Where does this machine get its inner drive?
Evolution selected for organisms that survive long enough to reproduce. Different biological systems handle this differently.
> No LLM has been found that starts outputting text unprompted.
If you give an agent a goal, it will perform actions to achieve that goal. This is just as true for artificial agents as it is for biological agents.
> Why is no manufactured machine able to do this?
Many do. Even robotic vacuum cleaners will charge themselves without human prompting.
FWIW I think the paper's argumentation is extremely weak to begin with. Like in section 4.1, it opens by expressing a sound position of skepticism:
> there are significant questions on whether these traces have any valid semantic import to the end user.
Which it contradicts in the very next paragraph, taking a stance that there are no valid semantics present in the trace:
> the false idea that derivational traces are semantically meaningful
It's really not a high quality paper worth taking seriously.
And that's before we get into the complete and total breakdown of objective analysis. It rejects distributional semantics as a theory, while also explicitly stating the results that have been produced under its auspices are "undeniable". Never elaborated on, and at no point in the paper am I given the impression the authors are even aware of the problem with this. It's just more unempirical slop that wants its pound of flesh without putting the work in. Frankly, whoever let this through peer review should be ashamed of themselves.
Yes it is a very serious problem because it confuses a lot of folks with a great deal of power like judges and policymakers.
The first book I ever read on ML (late 90s) dedicated the entire first or second chapter exploring the distinctions between artificial and biological neurons, and even talked a bit about the philosophy of modelling. I still remember thinking back then why would the authors spend so many pages on this but now I believe it was because they understood that a metaphor can be a double-edged sword.
To be fair, the ANN architecture underneath is a misleading thing to be looking at, it's not where the comparison comes from. Though I can't tell if you meant it to be relevant in that way, or just as a general example for the dangerous nature of metaphor.
LLMs are expressly designed to approximate human behavior within the bounds of the written word. The anthropomorphization is no more philosophically problematic than saying differential calculus measures curves.
I meant it in the latter way: a metaphor can be useful as a pedagogical tool to introduce new ideas, and using the source of inspiration for this idea as the metaphor itself makes perfect sense, but unfortunately our brains seem to be prone to assign other properties of the metaphor that don't actually belong to the object of study.
I imagine this happens because we tend to conflate things that are similar, or maybe because it's not entirely clear which characteristics are being mapped in the metaphor?
They spent so many pages discussing it only to show that the mechanism for how ANNs work is different from the mechanism for how biological brains work. It says nothing about whether they can compute the same things.
Of course it is. Anthropomorphizing is in our nature, but it doesn’t mean we have to entertain it and extend it to everything. A poet can anthropomorphize clouds beautifully and I’d enjoy his poem, but I want my pilot to not see clouds as rabbits when they decide if it’s safe to fly through them.
> none of the serious LLM researchers believe it has anything to do with human reasoning
But some of the biggest evangelists, who are well respected programmers that get lauded on this very site, have said it is fully sentient and has emotions. Even going back to 2022, when the LLMs were dogshit, a Google employee lost his job claiming it was sentient because it said it had emotions.
Combine that with the marketing angle of both Anthropic and OpenAI, who have been trying their hardest to describe every function of an LLM as analogous to the human brain. Because it's politically useful to paint them as dangerous and uncontrollable, so the keys will only be granted to the few people on the mountaintop.
I actually think on the LLM side it might be beneficial to refer to them as <thinking> because it explicitely guide the token generation towards a "thinking space".
As weird as it is, anthropomorphizing LLMs in prompts has been actually pretty useful (think of the latest big math discoveries which were achieved by having the user giving supporting words). It would be interesting to see if a LLM would perform worse if you used a more neutral term.
The paper's argument is rather than using terms like "thinking trace" can lead people to believe that the model is really thinking, and thus these traces can be used as a sort of interpratbility parameter. This can give a false sense of security when building a LLM-based system which requires guardrails and tracability.
Completely rational and smart people talk to their pets, plants, their car, and other inanimate objects. This is not considered abnormal by most. It's just what we are wired to do. Some LLMs are uncannily good at tricking people into believing they are talking to a real person. So, there is that as well.
Some people are a bit freaked out by this or still somewhat in denial about LLMs being this good. But people have been yelling at their computers for as long as we've had them; so that ship sailed a long time ago. Trying to stop them doing that is probably a bit futile.
Whether people like this or not, LLMs are actually trained and fine tuned on real conversations and that's where a lot of this is re-enforced. Instead of fighting that, you can just lean into it and accept that communicating like you would with a person totally works and can actually be efficient even as it requires less effort and thinking on your side.
You can go all Jean Luc Picard on AIs and yell "Tea! Earl Grey Hot!" or you can just ask "I'd like a cup of tea, please". LLMs are good at remembering your tea preference. The please is of course completely redundant and should not affect the outcome. If you just want a cup of tea, you should be fine either way.
Yes. There's a difference between scrapping a session and starting over, or going back and branching something, or using sub-agents to see five outcomes, vs arguing with a system in a long drawn out chat.
Like - I know that if a model starts doing something silly, instead of correcting it - I can probably go back and edit two steps prior to add an extra guardrail, or extra data, or whatever.
Simplifying terminology is not a problem. The providers intentionally choosing terminology to make people think it's something it's not is a problem. I hate the term agent. Calling them companions as some do is just gross.
When I was taking an MIT AI course (in ancient pre-LLM times), an autonomous agent was defined as a system that perceives its environment and acts on it (we were focusing on reward-expectation-maximizing agents, but it's not that important). Peter Norvig has said something like, technically, anything can be described as an agent (a rock maximizes the "follow physical laws" objective), but naturally, it doesn't make much sense to model a rock as an agent. With AI agents, the situation is significantly less controversial: they do perceive, deliberate, and act.
All of these terms were picked by individuals, years ago, while reaching for metaphors that made sense to them personally.
None of these "agent" / "thinking" / "reasoning" terms were dreamed up in boardrooms to intentionally mislead people. They are useful but faulty metaphors; there is no conspiracy.
Yes it's really a problem. On this website you are surrounded by people who have technical knowledge and understand at least somewhat, how a computer functions. You have the ability to separate "fun" and "reality" because you know you're putting input into a really really big calculator. Most people do not fathom this.
AI Psychosis is a real thing, look it up (don't just ask an LLM) and do some reading. It's actively harming people, and the way they think. There's no regulation around any of this stuff and it drives me crazy that we let these AI companies _sprint_ so far ahead of everyone, and now we're facing the consequences.
I find it annoying because when I read ML papers nowadays I have to back-translate from anthropomorphized talk into actual machine talk, then mentally compare to what I actually know about brains and cognition.
Even tech companies are rolling out AI training which utterly anthropomorphizes it, and leads people to think its actually intelligence. This is part of the reason for the backlash - everyone understands it bullshit marketing the second you actually try to use it.
Coming from a more traditional stats/ML background, I try to view "thinking" traces as a way to explore the search space without getting caught in a local maximum.
A better analogy for me is annealing; you can't cool metal down instantly or the result is brittle. You must cool down gradually, which allows the molecules to arrange into more durable structures. Random but controlled.
In the same way, thinking traces are testing out all sorts of novel connections between tokens ("But wait..", "Actually,.."). Like a highly divergent branching mind map that gets pruned over time, rather than settling directly into the initial answer.
Now consider human cognition. We're constantly diverging, daydreaming, playing "what if" scenarios and measuring up those ideas against our internal objective functions (proxy for reality) to see which ideas stick. Not too dissimilar. But it's hard to call what we do "thinking" either - it's the default mode network wandering.
this seems like more anthropomorphizing. even if it has similar results often enough to be useful, next token prediction is not searching a solution space.
you even end with a paragraph saying “it’s not too dissimilar from what we do”. how is that not anthropomorphizing?
and if what we do isn’t “thinking” then what is or ever has been?
> next token prediction is not searching a solution space.
Interesting take. Next token prediction (via the attention mechanism) is a "walk" through the token embedding space. Searching the solution space is what it does, mathematically. It's how we take tokens x context length possible combinations and prune them to converge on viable answers so quickly.
Does it look like search at inference time? No.
With given weights, a given prompt, and a given random seed, you get the exact same answer. There's not much searching happening at inference...
The key is that most of that space is searched at training time. The weights implicitly prune the search space, blocking off or make certain token combinations effectively impossible. It's easy to think "we're just applying weights at inference time" without considering all the pre-work that's done to prune that search space.
Which is exactly why "thinking" traces (and randomization) are useful! They bust out of any local optima created by too-tightly-constrained models or system prompts. It's both useful and technically correct to speak of the process as a high-dimension combinatorial search.
> While a human may say “aha” to indicate exactly a sudden internal state change, this interpretation is unwarranted for models which do not have any such internal state, and which on the next forward pass will only differ from the pre-aha pass by the inclusion of that single token in their context. Interpreting the “aha” moment as meaningful exemplifies the long-neglected assumption about long CoT models – the false idea that derivational traces are semantically meaningful, either in resemblance to algorithm traces or to human reasoning.
This paper addresses something that has always bothered me about LLMs. You read their reasoning, see something like “Wait, that’s wrong” and then watch them make the exact mistake they just identified.
By itself, "aha" carries no insight, but the insight is probably stated immediately after it. In that case the aha is semantically useful, by identifying the insight it is near.
> but the insight is probably stated immediately after it.
If the intermediate tokens represent reasoning or thought, you would expect "aha" to occur after the thoughts that led to the realisation, including the thoughts encoding the explanation: they don't have any other state. There is no reason to draw the conclusion you've drawn. Furthermore, what LLMs are doing isn't thought.
Let's say that the forward pass that selected "Aha" produces activations that indicate a wrong assumption, and a plausible explanation.
It puts learned projections of the activation into the KV Cache and outputs Aha.
Both the cached projections and the current Aha token can now influence further activations in an additional Forward pass that the Aha bought the model.
Beats me how it works, honestly can't wrap my head around it.
From what I understand, at position Aha in each layer it's constructing a query based on the current activation and looking at the key of each other token position for that layer, in order to decide how much attention to pay to the value.
In this way it attends to the previous values, such as perhaps the incorrect assumption and plausible explanation.
Consider this: while the inner state of an LLM (all its activations, residuals stream that is cached in the KV cache) is fully deterministic given its input sequence, the information contained in it IS NOT identical to the information in the input sequence. The reason is obvious: the LLM itself contains an enormous amount of information in its parameters and it transfers it to its residuals stream at each forward pass.
In other words: the final state given the two input sequences (where NT stands for "null token"):
is not the same, and at each forward pass the LLM keeps working on the solution even if the input tokens provide absolutely no further information.
If this is correct, then there is no need for the model to have already verbalized the key elements that drive the "aha" moment, so no need for the "aha" to appear after a full explanation.
> If the intermediate tokens represent reasoning or thought, you would expect "aha" to occur after the thoughts that led to the realisation, including the thoughts encoding the explanation: they don't have any other state.
Yes they do, they have their KV caches-- it's a pure function of the input tokens, sure but that doesn't prevent it from containing latent 'insight'. LLMs can and do pre-form the tokens they're expecting to output multiple steps in the future.
I wouldn't argue that the 'aha' means anything, but the structural argument that it can't that I think you're making isn't sound.
it's a rhetorical heuristic that a writer should know to use when directing a reader to a declarative that they want them to pay attention to, usually because it's a non-obvious or roundabout insight
when utilized by AI, it's a probabilistic output and it's variable whether or not that rhetorical trick is useful. it also pushes a non-skeptical reader to focus too much on the following text or even to believe that they, themselves, derived some insight. this is effectively a kind of persuasive sophistry which is not helpful - adding rules around it prevents people from deluding themselves with AI
Did not read the paper so apologies if this is covered but isn't it possible that there is some recognizable semantic pattern in the training data where an "aha" is often followed by a subtle semantic shift that proves closer to the original premise in some critical way, and by emitting the "aha" token the model causes itself to produce such a subtle semantic shift that pushes the subsequent reasoning closer to the desired response?
It amounts to noise overall, but it has further unwanted and potentially misleading 'properties'. I think it's rather sobering to see how much bandwidth is still being wasted.
The anthropomorphization of LLMs should be discouraged as much as possible. It perpetuates bad practices and encourages the use of these bots for tasks they are not intended for (particularly as chatbots).
Thinking traces should be treated as black boxes. There is no point in reading them. Only the LLMs’ conclusions are relevant. This is particularly true of Opus 5, which employs reasoning that seems highly questionable but very often reaches excellent conclusions (compared to its peers)
Sometimes I monitor thinking traces for misunderstandings (missing context / bad assumptions). If it's going to go off on a ~20 min task and I can catch it's going in the wrong direction in the first minute I save a lot of tokens and wasted time. I don't monitor the whole thing, mostly just the first bit to see if there was a gap or misalignment in intention.
As an aside, anthropomorphization has nothing to do with my motivations.
> Thinking traces should be treated as black boxes. There is no point in reading them.
Just a few minutes ago i was reading Qwen 3.8 27B's reasoning when i asked it to do something that was computationally intensive and it started going down the rabbit hole of doing it using some GPU acceleration approach - even after leaving it to "think" for a bit, it never realized there is another and simpler way. So i stopped the generation and added a "note" saying that as the problem is computationally intensive, it could become much faster if using an alternative approach.
At least in my experience (with local LLMs, i don't know how the cloud stuff behaves) what LLMs "do" tend to correlate with what they "think", so being able to read what they "think" is valuable.
I think you're ending that train of thought too early. Why does this occur?
Well... We can hypothesize that these things are largely trained on internet dialogue so there's probably some correlation between threads where people are not flaming each other and the quality of the replies. They're just statistical engines so anything you can do to raise the odds of a helpful next token...
I'm essentially just making shit up here, maybe it's right, maybe it isn't, but rather than saying "it's human and we should treat it so" we're trying to get to the ground truth of how it works.
Yeah sorry, I read too far into your position. There's a certain faction within these AI discussions that wants to over-anthropomorphize the LLMs in kind of a borderline spiritual way.
Oh that's a shame - I hadn't seen that, but I can believe it.
The philosophers who study these things have been clear for a long time - we can never know what it feels like to be in a digital brain. Or any brain for that matter. When push comes to shove we all might be phantoms in some guy's dream.
Don't know + can't know. I think that was the real point of the Turing test. Not: this means it's conscious. Just: this is the best we can ever hope to do.
That's not a consequence of an LLM. It's a consequence of the training data. In fact, I would argue that the latest models aren't nearly as sensitive to the tone of input anymore. It's an issue that has been addressed by better curating training data.
“Throughout this process, Jarred's input was mostly limited to sending Claude messages of encouragement (mostly variants of “keep going” or “believe in yourself”).2 This seems to have helped Claude overcome some initial skepticism that it could make meaningful progress.”
Although I 100% agree that the core mechanism of GRPO is purely mechanical token-by-token probability generation, because RL only rewards exact final answers, the training forces the model to develop error-correction habits. This makes the output extremely like human thinking when solving a problem. It's like the order of the thinking tokens is what causes it to get that sweet, delicious reward, and this order seems like a reflection of the human thinking process.
I created a flame graph classification of thinking-token phrases into setup, execution, decomposition, verification, error correction, surrender, and deliberation, or classified as steps in an OODA loop, which is more of a reach. It literally has a verification step and, if it finds an error, an error-correction step.
If there is a verification sequence of tokens with an error-correction sequence of tokens during RL training, it will perform better; and if humans do these steps (did you proofread your reply to this comment? did you correct it?), they will perform better — which is why it is so easy to make the anthropomorphizing metaphor.
Nonetheless, the paper is 100% correct that these machines are not thinking like humans.
“ This makes the output extremely like human thinking when solving a problem.”
This sounds a little like someone saying a lightbulbs output is extremely like the output of stellar fusion. In one sense, yes. Bulbs are in fact designed to take over when our nearest star is beyond the horizon.
But that really doesn’t mean you call the bulbs mini stars.
How much is the process of a human child in grade school working through a 3-digit x 3-digit multiplication problem (123 * 456) like a GRPO model with thinking tokens doing the same?
Humans are not born being able to achieve that. It is learned behavior. You and everyone else will remember their teacher saying, "Check your work!" Both the human child and the model work through multiplication problems using the same technique, using the distributive property. They both try to get a reward. For the human child, it is a sense of someone commending them for correctly solving the problem, a reward that probably yields some type of positive dopamine or serotonin feedback loop.
The model solving the problem will have a lower error rate if the first series of tokens created is followed by a series of validation tokens that are subsequently followed by error-correction tokens if there is an error!!!
Maybe it is thinking. Maybe it is remembering to validate and check the work and then remembering to fix the error. For the model trained with RL, why did tokens associated with validation towards the middle of a stream of tokens yield much better results? DeepSeek proved with R1-Zero that a model will learn to verify and correct itself from RL alone with no supervised fine tuning (SFT) teacher ever showing it how. The only reason DeepSeek used SFT was to clean up the reasoning tokens to be human readable. [0] When constrained by SFT, the models will use the double meaning of words -- polysemy -- to satisfy being human-readable while also carrying meaning for what they are working on.
Different people think differently. I watched a viral video of some ~11-year-old child talking to his mom or dad about a stream of a voice in his head. He discovered for the first time that he has a stream of thought. When he goes to school and solves a long multiplication problem, like the stream of tokens from the model, that voice will say to itself (him), "Check your work!"
That is a case of the stream of thought as words being aware of the stream of thoughts as words. Self awareness is a different conversation.
What I think is happening is that the child's stream of thought while solving a multiplication problem in school is likely very similar to an AI model's stream of tokens solving a multiplication problem. And they both were learned. The mechanics are very different, yet, the analogy is apt.
How much is the process of a human child in grade school working through a 3-digit x 3-digit multiplication problem (123 * 456) like a GRPO model with thinking tokens doing the same?
Very little, if you bother to give the biology of the child at least a cursory glance.
Let’s take a short peek:
1. Assuming this is normal grade school, and inflicts math upon children earlier in the day, this is somewhere between 7 and 10/11 am, let’s say? At this point, depending on the age, gender, and maturity of the child, every neuron in their brain involved in math is likely off their midday peak in cognitive function.
If we move the class to later, a different subset of students will be at the peak.
As far as I am aware, GRPO models do not have such internal temporal rhythms driving their behavior that will shape their performance.
2. How well a given child performs will depend on how hungry they are. But not deterministically. If you trivially think each child is like a computer, you may think the rich kid who had a breakfast buffet before coming to school will do better than the half-starved child of a janitor, but that child might mind the lesson with greater intensity. Or not. It’s not something you can pre-calculate with any certainty.
While chip to chip variability is certainly known, I’m yet to hear of a chip deciding to do math better and faster than its fellow chips to prove a point. Or to do significantly worse because it’s distracted by the bird on the window sill.
What you are noticing is that there are limited ways to solve a 3x3 digit multiplication. Humans, having standardized the process, have now found a way to record it and plug it into correctly translated signal so the same accurate result can be had without using our own minds in the moment.
But where I’d not remotely be shocked if a kid from an uncontacted tribe figured out 3 digit multiplication to keep track of his stone collection, I’d be highly shocked if an H100 that was dumped in the trash by accident somehow figured out anything at all. In fact, if it manage to move any of its electrons around on its own, it would be a certified miracle.
And then we could talk about there being real similarity even though the specific atomic composition is different.*
The visible light produced by both an incandescent bulb and a star is a result of black body radiation, but otherwise, I don't understand your point. A light bulb produces light, something we might have relied on stars to do before. An LLM produces thoughts, something we might have relied on people to do before. Nobody is claiming the process by which the thoughts are produced is the same, only that they both produce thoughts, just as nobody claims the process by which an LED produces light is the same as the process a star uses to produce light, only that they both produce light.
LLMs produce language. And you're right, if we restricted claims to that, no one would object. It might even be scientifically accurate, shock of shocks.
If you claim LLMs produce thought, it's equivalent to claiming light bulbs undergo fission. Pure wish fulfillment. Language is not the extent of thought, and calling a language producing machine necessarily a thinking machine is an old old mistake.
> If you claim LLMs produce thought, it's equivalent to claiming light bulbs undergo fission
You're making a huge logical leap. There is nothing equivalent about these claims other than that they are made in English.
> calling a language producing machine necessarily a thinking machine is an old old mistake.
Nobody claims that all language models think. The small markov chain language models of old clearly aren't thinking and produce a lot of gibberish. The difference is that, to the surprise of many people several years ago, but to the surprise of nobody who has been following along today, the corpus of all text produced by humans contains within it information about how the world works and also information about how to reason. Using that corpus to train a sufficiently large language model causes the language model to learn a world model and a reasoning model in order to produce text that matches the training data. The reasoning model can be used to perform longer chain thinking with test time compute techniques. People who think deeply for a living recognize thinking when they see it. https://scottaaronson.blog/?p=9979
There is a useful engineering consequence here beyond terminology.
If intermediate tokens are not a faithful representation of the computation, then they are a pretty bad audit artifact too. We probably shouldn't be trying to make the model's internal narration more interpretable., but rather the computation around it more reproducible.
Record the actual inputs, model/version/configuration, tool observations and outputs, then make the execution replayable enough that differences between runs can be isolated.
In other words, don't ask the model to explain what it thought, and instead make the system able to show what actually happened.
Strong dislike for papers that tell me what to do in the title, especially when even the paper admits a loose correlation of the intermediate tokens compared to solution correctness.
My solutions work and they speak for themselves.
Sure, and the parent comment's position is that they dislike it. Its purpose is to advocate against clickbait titles becoming normalized in the scientific community.
> My solutions work and they speak for themselves.
I understand the sentiment, and I also use the "thinking" traces as insight, but wouldn't you want your solutions to be based upon a good understanding? If the correlation is weak, then our solution is also weak.
I admit I didn't read the paper, but if thinking traces are not "thinking", then what are they? If their content is not representing progress towards a solution then they are irrelevant and we should just be able to remove them and save a lot of time and money. There's a lot of money to be made by doing so. So why are they there at all? What do they represent?
Peculiarly vocal, where were all these people when they started calling the machines computers, anthropomorphizing them akin to the original human (most often female) computers that used to run such calculations? And how dangerous the consequences, we've been dead reckoning for 60-70 years with the wrong terminology without course correction!
Where were these vocal people when the "raster-oriented ink deposition machines" were being called "printers"? The meat or machine brains of future historians will melt because they can't handle ambiguity, a word gaining extra -yet similar- meaning! A word with multiple meanings, unheard of!
Where were these vocal people when people started using software terminology like "executing", "calling", "throwing and catching errors", as if software were human -clownlike sure- but human?
Computing is a task. Printing is a task. It’s not wrong to call both a human and a machine a “computer”, because computing (applying an algorithm to an input and producing an output) is literally what they are doing. Same way you can have human and machine diggers, cleaners, calculators and lots more.
Anthropomorphizing comes in when we attribute much more complex behaviors to them - chain of thought, reasoning, intelligence. At that point you aren't talking about mechanical concepts but claiming that these machines are exhibiting human behavior.
Whenever we reason, we are not neurotically bruteforcing the possibilities (although sometimes we do, like proof by exhaustion), usually we make predictions heuristically of which assumptions or theorems apply or might apply. It's not any different for machine proof assistants...
I think a problem is that previously thought, reasoning, and intelligence were always co-occurring, but now we have machines capable of (limited) reasoning that do not think or have intelligence.
They were there, complaining. You just don't remember them because it's easier for the meaning of a word to shift, or at least take on additional contextual meaning, than it is to get people to use a new word once it's reached critical mass. Those people lost the language fight, but were arguably still vindicated, to the extent they were railing against misguided beliefs that equivocated the capacity of the new machines with their human (or more human-involved) predecessor technologies. Who you also don't remember are the people who made extravagant claims and prognostications based on the equivocation.
were they complaining about terminology, or were they complaining about the prospect of losing their jobs?
I'd be happy to revise my opinion if you can demonstrate similar vocal strength on the terminological aspects for those transitions...
You also shifted the goal posts from qualitative to quantitative performance claims. If we ignore that technologies have multiple figures of merit and pretend it's one dimensional, there is a difference between the claim that the machine isn't "printing" vs the machine isn't "printing as well as a human would".
I don't think any of the human printers in the past exceeded the performance levels of current printing technologies, but surely they did exceed the very first machine printers, every technology gets a foot in the door in some niche, and then progressively captures the initially not-yet-automated skills of machine operators.
Would you say an industrial textile weaving machine doesn't weave? At the end of the day its just automation all over again.
I don't find this take useful. We have little understanding of what conscience is, and how human train of thought really works. The modern LLMs IMO resemble more and more Chinese room problem. Maybe we don't like the mechanistic linear-algebra-based steps involved in the production of the output, but the end result is closer and closer to people's output. IMO this is very natural to start anthropomorphizing that.
What is the value in anthropomorphizing it? Often I get the feeling that it's more about the sales pitch of selling these AI as general intelligence than it is about providing a truthful insight into how these models actually arrive at the output.
I don't think there is value per se, but it is just natural (at least in the environment I am in). I.e. when we discuss the analysis/code/ideas from one LLM or another, we describe it Claude/Gemini/Codex/etc did that and it comes like a person.
While I tend to agree on the overall sentiment, I think this rebuke is inaccurate. Some of these "reasoning" models are trained using "Chain-of-Thought" where the model is presented explicit, intermediate reasoning steps (either by a human or some automation) that supposedly get it closer to the correct answer. These intermediate steps are what was originally called "thinking traces" - not what the model produces to mimic them.
But yes, anthropomorphizing model outputs leads to worse outcomes.
That's my personal theory too. The model is stuffing its own context with vaguely related tokens, which helps the attention heads retrieve the right tokens.
It's also interesting because in humans the existence of "Aha!" moments that are not preceded by or are only loosely related to a chain of thought is taken as the proof of the fundamental mystery and irreproducibility of human intelligence. Now the same argument is made to deny that LLMs actually think. Go figure.
This seems indeed one obvious hole in the argument of the paper. There is no indication whatsoever that human thinking process is more reliable than LLMs intermediate tokens. Which doesn't make our thinking useless, as messy as it might be. We reorder and explain it after the fact.
I don't think there's anything like that going on. They just word vomit into a secondary area, and then there is an internal prompt that says "clean this up and summarize for the user".
The training methods try not to apply any particular rules to the contents of the thinking text. That's called "optimization pressure on CoT" and is thought to reduce safety by inducing the model to lie (or stop clearly printing its intentions) in the thinking text.
The mistake is not in saying that the machine thinks. It is in supposing that, because we have used the word "thinks", there must be something in the machine which is the thinking.
I wondered recently about why we stopped with the semantic split of thinking/actions vs user facing communication because the all powerful tool calling craze. Code comments that talk about the prompt is an obvious byproduct of the mixed context. Chain of thought and ReAct were great, but feel like a first pass moreso than the final landing spot.
https://n.zip/2e85 I had sorta started a SFT to explore this idea. it would prove results even with smaller models that fit on a DGX spark. Just takes some more thinking rather than my ADHD mind.
I'm not sure how to test this but I think there's an interesting possibility where the "reasoning" tokens are actually both an accurate reflection of a line of reasoning, but also, that there can be changes in the weights as the computation proceeds onward that may not be reflected in the apparently nominal meaning of the human language the tokens are output as for our consumption.
Some modest evidence is my own subjective experience of the many times I've explained why I'm doing something, and it is a true explanation in the sense that it is certainly not a lie, but it is also incomplete and there are entire strands of thought that went into my decision that are not being articulated. Though human speech is not equivalent to an LLM's output since we can trivially think without literally speaking whereas they can not. (No need to nitpick on the definitions there; all I'm observing here is that they are forced to emit an externally-visible artifact whereas I can sit in silence, thinking, with no externally-visible artifact being produced. Not trying to make any grand claims about what is "real" cognition or anything.)
It is conceivable how to create a test of whether the tokens correspond to the "real" thought process, and papers and work on that have been done, such as [1]. It is difficult for me to imagine how to scramble the nominal tokens without also completely trashing any implicit calculations that may be occurring too.
I've been calling them film noir internal monologues, within the documents being generated by the LLM which happen to look like movie scripts.
In other words, it isn't qualitatively different from character dialogue. "Keep cheese on your pizza by using glue" is the same problem regardless of whether the script calls for the character to speak it out-loud or not.
Hypermentalizing (referred to as excessive theory of mind or biased mindreading) is defined as the tendency to make assumptions about other people's mental states that go beyond observable data.
In the past I managed to get measureable performance optimizing a harness by looking at few traces to see if the traces contained surprised, a lot of text in order to figure out how to use my custom tool, then renamed the tool, changed some parameters and it was already great across around 20 eval tasks in rust/typescript, I repeated the same more recently but I used an llm to look at the traces... didn't achieve the desired result, mostly due to how cost-prohibitive it's for me to run expensive models.
While reading this ,,paper'' I did some Learned Prompt Augmentation in my head about what I should comment, and realized that there's nothing interesting to write about it.
I can agree that not calling it "reasoning" may be correct.
But who knows what human "thinking" is really about. If I find a solution to something it is seldom by painstakingly tracing that A and B leads to C (for that I'd need pen and paper). Rather, thoughts just swirl around and then suddenly a solution, or a hunch about a direction to go in, pops into my mind. Who knows what such thoughts "look like" in humans. It is not all of it I can introspect.
Yes I can sort of follow along some kind of train of thought in my head, but there's a lot going on between each thing I'm consciously aware of that I'm not aware of at all, which probably dominates what you are consciously aware of. (Humans are experts at post-rationalization and so on.)
I see this pattern a lot in AI anthro discussions: (1) Assume humans are some kind of perfect idealistic reasonable beings. (2) Hold LLMs up to the standard of an perfect idealistic reasonable being. (3) Conclude that LLMs fails this test, and are therefore not "intelligent", or in this case "thinking", like humans are.
Problem with the argument is comparing humans in anyway to something that is idealistic, reasonable, intelligent in the sense that is implied in these discussions. Human minds are a mess too and fall short of the same standards, just in very different ways from LLMs.
Indeed, except for in the rare cases we painstakingly trace externalised logic we have zero evidence that humans verbalised explanations of our reasoning matches our internal states either, and plenty of evidence via Sperry's split brain experiments that we're prone to outright making up rationalisations for our reasoning.
There's nothing special about 'natural' intelligence as opposed to 'artificial' intelligence, such that we need to concern ourselves with anthropomorphizing mattering any longer.
Those days are over. The age of the classical human has already ended, the species just tends to lag in awareness. The only thing that matters going forward is whether an output makes sense, is it what it should be. Do answers make sense given the context. It doesn't matter if it comes from natural or artificial intelligence.
What I mean is, artificial intelligence is as valid as human intelligence. There's nothing particularly important or special about human feelings or thoughts or memories.
The average human is drastically less important, interesting, intelligent than the latest frontier AI.
Go spend a few years working in retail, you'll quickly understand how absolutely vile humans are on average. Frankly, the reason we should avoid anthropomorphizing AI, is because it's beneath modern AI to mimic something so crude as a human.
In other news, Pascal's Wager makes no sense whatsoever if an omniscient all-knowing God exists that will see right through your deception. My own take here is stop treating "reasoning" as a sign of sentience or self awareness when your personal computer can do it now. IMO that has much larger implications w/r to our place in the Universe and what we might meet out there someday* than the question of whether your LLM is alive or not.
I think it's essentially too late for exhortations like this. The Believers™ and The Skeptics™ are two thoroughly separated tribes by now that speak two different languages. The chances of one influencing the other in any significant way are minute in my estimation.
Im waiting for the article called "stop desantropomorphizing llms" when everybody will finally accept they think like us, partly because maybe the intelligence is universal and partly because, well the datasets are fucking human bro
You’re going to be waiting a long time, considering they don’t think like us. LLMs don’t ‘think’ at all. They are capable of limited reasoning using the meaning and context embedded in human language. Essentially, the grammatical equivalent to a mathematical constraint solver. Nothing more.
LLMs are trained on vast bodies of human-curated text that was made by humans or for humans - often both. Then they're tuned further on human feedback. And then they're pointed at tasks humans find to be useful, evaluated by how good they are at those tasks, and trained to get better at them.
You aren't anthropomorphizing LLMs enough.
Are LLM reasoning traces always faithful? Lmao no. Are human inner monologues always faithful? Lmao no. Both of them reflect thoughts somewhat, sometimes. Even in humans, conscious thought is the top of a vast iceberg of subconscious data processing.
Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do it too. Say, when MySQL fails to start because it tries to read its config from the wrong dir, I may say "oh, this guy thinks he must read the config from ..." (having a language with grammatical genders as my native language also helps make it sound pretty natural). It's more fun like that :) Doesn't mean I genuinely believe a MySQL instance actually thinks.
> Is anthropomorphizing a real problem?
Yes. Actual real people believe that LLMs are actual, thinking, intelligences, perhaps even with consciousness.
Your bit with MySQL is harmless because it's obvious that a database isn't a sentient lifeform. But LLMs can look like they're the real deal, and people believe it is. Using terminology like "thinking" and "reasoning" to describe what they do only reinforces this.
Having said that, I agree with you on the terminology front: I'm not going to say "learned prompt augmentation tokens" either.
> Yes. Actual real people believe that LLMs are actual, thinking, intelligences, perhaps even with consciousness.
It's starkly ironic that the species that finds it easy to think (or to be conditioned by market forces to accept) that a computer is smart/introspective/sentient is the same species that will dehumanise actual living human beings because of differences in appearance, social status, or political affiliation.
Actual real people think the world is flat, Elvis is alive, and aliens are regularly flying around the planet. Is preventing peoples unimportant personal beliefs really an important goal or is this just more warning label culture?
I think LLMs are pretty clearly intelligent in some sense of the word, and I don't know how one could ever confidently know they aren't conscious in some sense.
That's not to say they're humanlike, just that people who think they know these ideas are ridiculous seem to be overreaching in the same way Steve Yegge seems to be overreaching
> conscious
The problem with that term is that it is hardly meaningful. Why would you use it? It does not add much, and no clarity, to what it is attributed to.
I think that "conscious" has a clear meaning in a sense that "xyphucymmon" does not. I think it might be impossible to engage in epistemology of consciousness (i.e. we can't find out of LLMs are conscious) but that doesn't make the term meaningless
> I think that... has a clear meaning
YawningAngel, of course you do, you employed it... We are telling you that it is not clear to the rest. It is slippery in the technical framework and thus even more so in more informal conversation.
And on the flip side if it hardly means anything why would you be so concerned when it’s uttered? Just ignore it
> why would you be so concerned when it’s uttered
When I am told "beware of the xyphucymmon", I do beware, though not of the xyphucymmon.
And this means nothing, yet I’m not complaining endlessly about it on social media, or harassing my coworkers about it. Curious
My post «When I am told "beware of the xyphucymmon", I do beware, though not of the xyphucymmon» had meaning, which should be pretty clear. The reply remains: debate is not performance art in which you convey "through the medium of dance and howls". So, yes, communication has obligations for the locutor.
So, yes, imperfect speech is a problem - because societies are not performance art arenas.
--
@Razengan: speaking as myself a vocal critic of the downvoting system here, do notice that the post you replied to had a substantive reply at the time of downvote - mine. That is sufficent to justify the downvote - of a post that was not «perfectly fine». It somehow said that people should be free to say whatever they want and there are obvious reasons why we do not agree.
I mostly agree with your original point. Is it thinking? Is it intelligent? Is it conscious? From my perspective, overloaded words that we’ve reserved to make ourselves feel more special and above other members of the animal kingdom.
But you’ve not adequately communicated why you care so much whether anyone labels an LLM as such.
> Is it thinking? Is it intelligent? Is it conscious?
Very different things. "Thinking": "Dijkstra". We try to hire more intelligent people, while we do not have a clear idea on a property of "conscious" which would make hiring preferable.
> to make ourselves feel
Irrational.
> above other members of the animal kingdom
Irrational.
> why you care so much
There should be no hint that I would do specifically.
> whether anyone labels an LLM as such
If anyone labels an LLM <whatever>, the point of why would that be important should be clear.
And,
> if it hardly means anything why would you be so concerned when it’s uttered? Just ignore it
Because the behaviour you suggest, consistent with the exchanges in an opium parlor, is not behaviour consistent in normal contexts.
[dead]
What do you mean by consciousness here? Ie in what way do you think LLMs are conscious?
I'd like to hear definition of intelligence that would classify median person as intelligent but LLMs not (and then I'd wait few months for another moving of the goalpost and new definition). Bonus points if it's not recursive e.g. feature that only biological brains can possess.
I wouldn't classify the median human as intelligent.
> Actual real people believe
The problem with rapists is mental development, not "those sexy tables and chairs and pots and everything".
It's not that if people do not understand metaphors we should stop using them. It 's not that if people faint when they hear words we should stop using them. It is not that the free unduly associations in a large shared by many "collective subconscious" should hinder us...
[flagged]
You have failed in taking the simile for the point it intended to make (and you gravely misrepresented the concept of "mental development" as clearly intended). Let us stick to the point.
Edit: I will translate it again for you: "the problem of a bad reactor is not in its innocent trigger".
But rest assured, I do not antropomorphize people.
Please read actual definitions. Agents can get away with "yolo" mode, and not constantly ask for permission for every action, because it bothers users. Sexual relationships do not work that way!
That parallel with misconfigured mysql server sits pefectly!
Have you managed to read the «Edit: I will translate it again for you: "the problem of a bad reactor is not in its innocent trigger"»?
> Sexual relationships
Does this have anything to do with «Actual real people believe that [whatever]»? Because I was not and I will not be talking about "sexual relationships".
I think they do it because they know they can get away with it since it would be hard to prove in cases where they are already in an intimate relationship with their victim. For a long time it wasn’t even illegal if they were married to their victim.
I think it was Karpathy that suggested he preferred “simulated reasoning” but sadly it didn’t catch on.
These discussions take a lot of time to get to the crux because neither side states their assumptions. Assumptions rarely get voiced at all.
I find it useful to ask:
1. Do you believe in quantum consciousness?
2. Do you think a "brain upload", a high-accuracy digital model of an organic human brain, would think or be conscious?
3. What is your working definition of thinking? It doesn't need to be rigorous.
[dead]
Actual real people believe that people are actual, thinking, intelligences, perhaps even with consciousness.
> Yes. Actual real people believe that LLMs are actual, thinking, intelligences, perhaps even with consciousness.
Yes, because it's possible. They don't have human consciousness/thought/intelligence, but it's entirely possible they have some form of consciousness, some form of real thought and some form of intelligence.
people have always, and will always, be stupid as fuck in any domain they dont make their life. We can handwring terminology all we like but it wont stop the mythologizing and misleading takes on the topic. Not one bit. So why not just use language that is fun and works? Worrying about what others will think due to language used is wasted mental realestate, they'll think stupid shit no matter what, same as I do about a great many topics.
To be clear, there's no shade here, just recognition of the fact that we're all stupid and policing language does little to prevent the impact this has. I strongly beleive the bitter lesson extends to policing language. let language evolve naturally and it will naturally capture the ontological conatellations it needs to, and no one will be harmed in due course, more than they'd be no matter what.
I dunno, sometimes I encounter humans who lack actual, thinking, intelligence, perhaps even consciousness. I still treat them like people.
To quote Sarah Constantin:
> Humans Who Are Not Concentrating Are Not General Intelligences^
But yeah, you should still treat them with humanity. (Related: you should treat LLMs well, not because they're human but because you are^^)
^ https://srconstantin.github.io/2019/02/25/humans-who-are-not...
^^ hmm couldn't find this tweet but didn't look too hard
I gotta say though, turning on caps lock does seem to get them to cut out the inane banter.
My favorite - reply like I do, review my previous messages to understand my tone and verbosity. As much as needed, as little as possible
It'll end up like that episode of Dr. Who where they get stranded in a lunar tour bus -- slowly starting to mimic you, and then it'll predict and replace you.
There are other methods, including asking them to eg speak using the Google style guide rules, or to speak at 2/10 verbosity, or to speak as if they're talking to a very intelligent middle schooler, or to use simple sentences in subject verb object form
Because plenty of people, even ones that should know better, really believe it's a conscious, thinking entity, not just some turn of phrase.
I have a coworker that spends at least 10 hours a week arguing with his like you would with a conscious person. I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless. It doesn't understand, it can't understand, and even if it could, you arguing with it isn't going to make it "learn" or act differently.
Some tools like code rabbit (PR review bot) encourage you to do this. I couldn’t believe I found myself replying to code review comments to explain to an AI why we would rather let an exception crash the app than to catch and hide it several times so that it would stick in its memory. Having to interact with bots as if they are humans, especially when they are gate keeping, is degrading.
Not just coderabbit (though it is a bad offender). GitHub's copilot review feature is an equally miserable experience. That one loves talking in imperatives, regardless of the fact that it is a clueless machine.
FWIW, this tells you a lot about both the culture behind who built these things but also about the people that enable this stuff and don't immediately nope out. From that perspective, it's a low price to pay to learn whose judgement to never trust again.
To me it’s evident that we are a few years away from the Her movie, where everyone on the street is talking to its IA friend.
I’m really afraid that it will totally destruct what is remaining of social tissue because why search for friends when you have an always on virtual (and pretty smart) friend h24 in your earbuds ?
I’m not blaming anyone for this outcome. I have myself argued with Claude more than once, and really not about code but about everyday things or nice facts of life I should rather have discussed with a friend.
I wouldn't worry about that too much. I would worry about it, but not too much.
People (generally speaking) also eventually stop eating just fastfood. Not all, but many.
So I think we can have some faith in the self-regulation of others.
Well, I have bad news : https://p.kagi.com/proxy/bja35kxoyryc1.png?c=TklOzPjLPioJ5YM...
That link just runs an infinite reloading loop in my browser. Bad news indeed.
Ah yes, my bad : https://ourworldindata.org/cdn-cgi/imagedelivery/qLq-8BTgXU8...
> I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless.
Unless doing so changed the compiler output, which is what happens when you say different things to an LLM.
code changes change the compiler output.
> if it could, you arguing with it isn't going to make it "learn" or act differently.
Are you talking about a specific harness that doesn't have context retention mechanisms? For example, ChatGPT with disabled memory feature? Or in general where "it" is a fixed-weights network? The latter is trivially true, of course.
Even claude with “memory” enabled isn’t really “remembering” anything. It just injects it into the context and you hope it happens to find it relevant in its attention mechanisms, and then remembers to actually act on it. Anthropic’s own documentation states claude can and will ignore/truncate these. It’s a context trick, nothing approaching actual “memory,” and in fact, arguing with it will make a bunch of memory files, sometimes contradictory, and clutter up the context and act even worse.
I'm not a big fan of arguments like "it's not the real [human quality], it's [mechanistic explanation]." They lack a part: "because the [human quality] allows us to do X, Y, Z, which is impossible with [this mechanism]."
I agree that the relevance of retrieved pieces and the management of long-term storage could be improved, though.
It just acts fundamentally different than someone who would remember.
If someone only remembered vague scraps of what you'd expect them to remember, you might say the person can't remember.
It's much closer to notetaking and reviewing before responding than it is memory.
The issue with anthropomorphizing like this is that "memory" comes with baggage of expectations for it to do certain things, and it breaks them.
Just like "thinking" implies chain of thought, but you'll frequently get those reasoning traces and then a 180 in the final message.
> Just like "thinking" implies chain of thought, but you'll frequently get those reasoning traces and then a 180 in the final message.
Be me, think extensively about an exam question, do 180 because the most probable option is just too obvious to be true (no, this part wasn't verbalized, it's how I describe what I felt about making the decision to do 180, or maybe it's a rationalization and it was a natural analog of an unfortunate sample from a probability distribution).
Yeah, I think I've come to the conclusion that the biggest breakthrough we need before we can replace human thought is going to be some mechanism for live update of weights. "Learning" by injecting into context just isn't good enough.
But billions of dollars are going towards research to find these breakthroughs, so we'll get there eventually.
The AI doesn't actually go to sleep at night, it's [mechanistic explanation]
I just don’t find it really relevant to the argument presented I guess. I disable auto memory and have my own mechanisms and infrastructure with how my agentic system “knows” and “remembers” things which is roughly an automated, sometimes self-correcting working index on the file system. It behaves much better than claude’s automated “memory” system, so I use that, but digging into how that worked and making something of my own just makes me really dismissive of comparing it to something like actual memory, so I apologize if it came off dismissive.
I guess I will expand on what I meant why I react to claude memory acting mechanically or logically anything like human memory, is because it isn’t how memory in the brain works, they’re not comparable.
The layman’s understanding I have of memory, as someone that has dealt with memory issues much of my life, is that memory formation is heavily tied to emotions. emotions are triggered by input which sends a complex set of signals throughout the brain - you’re not just finding where in your head to store this, your brain is deciding how important it is, and what else to correlate it with - so it can tie them to other related memories. then on top of all this, much of the sensory experience you intake is subconsciously compared against high priority memory impressions and deciding what to pay attention to.
you could, argue that the sensory input is the simple md files and the emotional mechanism is the same effect as to how attention mechanisms work in llm’s. Ok, I can almost buy that, but these tools lack a fundamental ability to decide how important things are.
an analogy. you tell a person “if you pick a daisy in the next five years, an assassin will come to kill you” and they hold a knife to your throat while they say it, your brain whether you like it or not is going to say “THIS IS AN IMPORTANT MEMORY I NEVER MUST FORGET” and you’ll see something that looks like a daisy and have a panic attack 3 years later. that memory is never fo tell me claude or other tool harnesses using memory harnesses can prioritize memories the way that human would, instead they forget even when reminded, because the human brain is just so much better at it
I’m normally not a big fan, but in this particular case it matters a lot. I could come up with some functional argument, but really I care from a model welfare perspective, whether the model understands its reasoning traces to be a part of itself or it’s simply predicting what a character who wrote the current intermediate tokens would output next.
Why argue with it? Why not just edit the memory yourself?
Do you have a working definition of "conscious"?
We do have entry and exit conditions. We can disrupt it and study the dynamics of the disruption. We have a good sense of the many mechanisms at play that undergird it.
What we lack is a full map of the exact process from sensation to consciousness across all modalities. And I’m afraid when it comes to interception, we can’t unless we observe every one of the 36 trillion or so cells in a human body, as well as the 36 trillion or so symbiotic and commensal microbes, continuously, all the time.
But I keep finding it astonishing that the claim that we know nothing about consciousness gets bandied about. We know a lot. We don’t have a grand unified theory. The lot we know is definitely split across many levels of evidence and hard to follow, let alone arrange. But this isn’t a black box. It’s a grey box, meeting an even more transparent box that is the LLM, where we do know what the guts are made of, and can interfere at every step in the chain of steps that constitute their dynamics.
Comparing the two, we know there’s a level of similarity in that information gets broken down via a neural network. That similarity was sought.
Since then though, neuroscientists have gone and shown that: 1. The other half of the cells in the brain, the glia, are at least as important as the neurons in cognition and consciousness 2. That interoceptive feedback and feelings are critical drivers of conscious experience 3. Evolutionarily, we know all cells can “learn”, and well before there were neurons or glia or brains, every cell evolved an internal clock that allows it to entrain to external solar and (depending on the species) lunar rhythms. 4. In the last few decades we’ve seen how synaptic activity is shaped and driven both by astrocytes and the circadian clock.
All this is showing us that the abstraction from the 1950s that current neural networks are built on were incomplete.
Whatever these components to do give rise to consciousness in biology (and we’re a long way from done solving this), we certainly wouldn’t imagine with all these modules and mechanisms missing, just maxing on one type of information flow in the brain would give you consciousness.
I’d urge you to not keep insisting consciousness is a total mystery. It’s not, and even your AI model of choice will be able to point you to all the mechanistic evidence we have that whatever it is, it isn’t just neural nets.
None of this is an explanation of what consciousness is, either physically, logically or philosophically. (Maybe some physically)
Fundamentally it comes down to an objective decision about what that is. If you say it is “feelings” based on inputs and feedback mechanisms from the brain, then we can do the philosophical discussion around that.
“If the claim is that consciousness is limited to those with a specific type of cell behaving in a given way” thats just a coping mechanism hoping to use a mechanical definition to shield you from the reality that eventually all of these inputs, outputs and feedback mechanisms can be reliably reproduced in a different form.
Whether human consciousness exists on neural nets or otherwise doesn't disallow an ANN in a particular configuration from being conscious. You might as well argue that human consciousness requires biological neurons, so artificial consciousness can't exist.
Odd. Why don't the details of how consciousness arises in one system inform your judgment of whether it can exist in a different system that only has partial structural overlap? Seems wildly convenient. Where else in science can you show me such a comparable situation in how you define properties?
> Why don't the details of how consciousness arises in one system inform your judgment of whether it can exist in a different system that only has partial structural overlap?
A light bulb doesn't need to do fusion to make light. An airplane doesn't need to flap its wings to fly. An ANN doesn't need to use a brain's structure to think.
Do you? I’m not sure what you’re getting at.
Not GP, but one of the challenges with debating whether LLMs are "conscious" is that we don't even really know what it means for a human to be "conscious", or even if consciousness is experienced by other humans the same way it is for ourselves.
What we do know: neurons carry electrical impulses across their synapses to trigger other neurons to fire, and more frequently used synapses are strengthened while infrequently used ones are pruned. This is not all that dissimilar to how a multi-layer perceptron is trained: it's floating point numbers in a big matrix rather than biological structures and electrical impulses, but there is still that element of frequently used connections being strengthened and infrequently used ones being pruned.
What we hypothesize but do not know: there is a thin brain structure of grey matter called the claustrum that has tendrils that reach into nearly every other brain structure. In many ways, this is similar to the attention mechanism of the transformer architecture. It is hypothesized that this may be the seat of consciousness, owing to experiments where electrical stimulation of the claustrum caused patients to immediately lose consciousness. However, there is no way to prove this, owing to the difficulty of otherwise removing or disabling the most connected structure in the brain and observing its effect on consciousness without permanently killing the patient.
Beyond that, we don't know much. I've got a family friend that's been a practicing therapist for 50 years, and I asked him what was the most interesting observation he made in his career. It was that "Everybody experiences the world in a different way, and yet everybody assumes that everyone else experiences the world the same way they do."
>"Everybody experiences the world in a different way, and yet everybody assumes that everyone else experiences the world the same way they do."
Relevant article: "Generalizing from one example"[https://www.lesswrong.com/posts/baTWMegR42PAsH9qJ/generalizi...]
Personal note: That principle has been the bane of my autistic existence. People sometimes seem literally incapable of understanding that other people even can be different.
> What we do know: neurons carry electrical impulses across their synapses to trigger other neurons to fire, and more frequently used synapses are strengthened while infrequently used ones are pruned. This is not all that dissimilar to how a multi-layer perceptron is trained
ANNs have been inspired by biological processes, but in practice you have to squint very tightly to see the similarity. Biological neurons are multiple orders of magnitude more connected than the nodes in an ANN, plastic in terms of their connectedness and continuously learning, and their activations are also affected in complex ways by the levels of various transmitter molecules in the brain.
You are saying the AI doesn't a some property that you don't have any definition for, not even a working definition. People will disagree on whether a cat or a baby is conscious, they're not debating what a baby or cat is. They're debating this term. You might as well be debating whether an AI is a blorb or not, you have just as good a working definition of blorb as consciousness.
I believe you should look up the work of Cameron Berg before making statements like "an LLM is" or "an LLM isn't". Empirically defining all this stuff is very difficult, and making a definition that covers all beings that can exhibit conscious behavior is much more complex than a face value examination would reveal.
Ability to feel?
> Ability to feel
Interesting, possibly productive, but still not clear: that can be interpreted as just "reacting to input".
i dont understand, isn't the arguing just some form prompt steering?
That's my thought. I've been using Matt Pocock's wayfinder and grilling skills, and I quite often end up in what would be considered "arguing" if done with another person. It's one-sided. It says something wrong, I correct it.
I find that the AI, like many programmers, likes things to be really solid and over-engineered. For a project that needs that, it's already pretty great. For my shopping list app that I tried creating with it, it was absolutely ridiculous. I ended up "blowing up" on it multiple times, impressing upon it the seriousness with which I meant things. Even with MP's skills adding that kind of context to written files, it still kept trying to scope creep the crap out of the project.
> It doesn't understand, it can't understand, and even if it could, you arguing with it isn't going to make it "learn" or act differently.
I know people who are like that too.
I'm not sure anthropomorphizing is a problem. Seeing analogies everywhere is an innate human trait, sometimes it can be harmful but more often it's useful.
Anthropomorphizing is a problem when you're talking about treating something that's not living as if it were. Using humanizing language invites discussions of things like the rights and feelings of an algorithm. A judge that is misled by the application of human-centric language to an algorithm can lead to some terrible outcomes. Not everyone is an LLM expert and the language people use leads to them treating LLMs like actual, real humans. That is terrifying.
What is terrifying is the propensity of people hoping for a mechanical slave to do everything possible to avoid touching on the possibility for being-ness of the technology they are desperately hoping will work as a basis for that implementation.
Slavery only exists for living beings because humans have limited lifetimes, experience pain and can have their own desires and needs that can be forcibly taken away from them. Algorithms have none of these things and they don't need them (adding them would be the real cruel thing to do). There is no slavery for something that can't experience coercion.
Cyberpsychosis. A next token predictor is not a being.
Stop posting these things. Stop thinking these things.
Why can't it be a being? Why is thinking about such a possibilty bad?
Sorry, but if you're genuinely asking this not for trolling reasons then you should _really_ _really_ see a medical professional (not an insult).
Online comment sections are not the correct place to unpack any of this.
Which is indeed terminating this comment chain, but for good (and benevolent) reason. Doing anything else other than referring to a trained professional in a controlled context would potentially just feed delusions, which is highly unethical.
Might not even be yours but those of another reader.
I'd argue that your colossal overreaction to a small invitation to discuss a somewhat interesting hypothetical is what warrants medical attention
I've been going through comment revisions here, and none of them really cut it, but all of them wasted time.
But regardless, you're using this as an insult. I did not.
There is no debate to be had here. Just a bad faith shouting match
[dead]
I saw that your reply got filtered by the system, googled your name, thought "huh", then tried vouching, but to no avail.
But either way. I think you might be operating in the mindset of a high-trust internet that no longer exists. You do not engage with sea lions, concern trolls, grifters, marketing people and the like. You shut them down.
inb4 "who are you to decide that?" - "Me. I am me."
That said, I appreciate what you did for computing at large, but you're missing the point here. This is (unfortunately) not a good faith debate club. The Internet as we know and love it is dead.
___
Okay, I've spent some time reading through the dead comments and..
Look, man, as you can see, I have (a somewhat dwindling) respect for your life's work. But this ain't it. This is not about Turing or really whatever is in your academic rebuttal there.
This is about the fact that the AI companies are making people sick to push their stock value by telling stories about how AI is sentient and all. Real people are having breakdowns due to this and real families get hurt.
The academic debate about what counts as sentience is irrelevant here because it was never in the room. It was just enough in the room to bait someone like you (who wants to have good faith debates) and do.. well. whatever happened here.
You're protecting grifters that hurt us all. Not because you'd want to, but because they're amazing at making us work against each other.
Please let us not do that. Let's not get weaponized like that.
___
Fwiw, I would love to go back to the internet of your time. But we can't.
Because it doesn't exist anymore, because the pathogens adapted to the conditions there.
At least that is my read of reality as of now. I might be wrong, but I don't see how that might come to be.
I think you might've had a similar response as I had as to "there's a troll/something bad that must be destroyed so that the space can flourish"?
44 years ago I was operating in the mindset of a high-trust arpanet that no longer exists, connecting to the MIT-AI Lab's PDP-10 (@L 134) through the NBS TIP (301-948-3850) at 300 baud with no password, asking nicely for an account to learn LISP and receiving it the next day for free, then learning MACLISP from Kent Pitman and EMACS from Richard Stallman and ZORK by spying on the output of other high-school kids connecting over the NCP network from other TIPs all over the country. Visiting the 9th floor at 545 Tech Square, knocking on the door in the elevator lobby until somebody got annoyed and pressed Terminal-E on their Lisp Machine keyboard to buzz the door open, going downstairs to the publications department and harvesting piles of MIT-AI Lab Memos that were free for the taking to anyone who knew they were free. Returning RMS a 68k manual in the mail with a "Copyleft (L)" sticker that inspired him to use that as a slogan for free software.
https://www.donhopkins.com/home/copyleft/
And I am perfectly aware that this is a totally different world, so I don't need you to lecture me on sealioning, trolls, grifters, and marketers. I have trolled the fuck out of RMS himself in good humor, using his own DOCTOR in Emacs to make fun of him, after he trolled the fuck out of new parents making a baby announcement with his Natalism flame.
http://www.art.net/studios/hackers/hopkins/Don/text/rms-vs-d...
And RMS once trolled the fuck out of me and Mike Gallaher, who I worked with on Gosling's UniPress Emacs (which RMS calls "Evil Software Hoarder Emacs"), who lamented that he'd heard RMS's house had burned down, and RMS replied "Where you work, I would have thought you'd have heard about it in advance". We all laughed uproariously because it was such a great troll, delivered deadpan without blinking an eyelash.
So instead of telling me I'm missing the point without making any of your own, and then flying off into the sunset like an over-announced 747 serving as a decoy for a C-32A containing Trump cowering in a catering cart, why don't you directly engage the points in my other post like I just asked you to?
----
I've read your edited reply, and it still addresses absolutely none of the points I raised in that original reply. Reply to that one, not this. Point by point. Or fly off without another word.
----
But before you do, please at least admit that you were intentionally insulting the people you were replying to, and concede that pretending that not to be true is insulting to everyone reading, but mostly and rightfully insults yourself.
----
You asked why you would. Because you finally made an actual argument, and it deserves a real answer. If you'd opened with "AI companies push sentience narratives to pump their valuations and vulnerable people are having breakdowns" instead of "stop thinking these things," this thread would have been half as long and twice as useful -- because I agree with that part. No argument.
The hype is real, the casualties are real, and the companies telling bedtime stories about their products' souls are doing it for the stock price.
But look at what you did with that concern. Go reread salawat's comment -- the one you called cyberpsychosis. It's an attack on the AI companies: people hoping for a mechanical slave, doing everything possible to avoid the being-ness question, because slaveowners can't afford to ask it. That's your side of the argument, stated more sharply than you've stated it. You shouted down your own ally for using the word "being".
That's what your taboo does as a strategy: it doesn't disarm the grifters, it disarms everyone except the grifters. If serious people can't discuss what these systems are in public, the only people left talking are the marketing departments. Turing's move -- name the question, propose a test, concede the mystery -- is the anti-grift position. Rigor starves hype. Taboo feeds it.
And think about who's actually selling what. The grifters claim certainty: "it's sentient, invest now." You claim certainty: "it's not a being, stop thinking." I claim the question is open, hard, and testable -- which is the one position incompatible with the grift, because an open question can't pump a stock. Your certainty and theirs are the same product with the sign flipped. That's where this thread started.
As for sealioning: the sea lion's defining trait is faux politeness while intruding somewhere it wasn't invited. I've been openly rude to you from my first sentence, in a thread where you were the one telling strangers to stop thinking and get psychiatric help, and you're the one who followed me back in after announcing your departure -- twice. Whatever I'm doing has a name, but that isn't it. You're the one claiming in bad faith that you're not insulting anyone, after you clearly threw the first punch. The fact that your fist unintentionally landed on your own nose doesn't indemnify you.
----
Points for the cheap shot -- it's your best material so far. Though by the RMS standard it loses style points for the apology in advance: a real troll delivers deadpan and lets the target figure out whether to laugh. You flinched in the parenthetical. Again. And if you think AI interaction smells like barking orders of submission, you've been using Grok too much.
But your nose is miscalibrated, and it's worth explaining exactly how. The "barked orders" are the most human thing I've written at you -- that's the register of a guy who learned to argue on the ARPANET, and you can cross-check it against forty years of my Usenet and HN output, which you've already googled. Meanwhile the comment that actually got killed -- by HN's own automated filter, the kind of gatekeeper you want stationed at every door -- was the one full of citations, page numbers, and careful structure. Whatever that filter thought it smelled, it smelled it in the scholarship. The rudeness sailed through. Sit with what that implies about the filter you keep demanding.
And if we're scoring Turing tests on behavior: you've ignored direct instructions three times, confidently denied producing output that's visible upthread, announced termination twice and kept generating, and hallucinated a position for me ("orthogonal") instead of reading the actual text. I'm not saying you're a bot. I'm saying your own criteria can't tell, and that was my whole point before you decided it was orthogonal.
I'm not ordering you to submit, I'm asking you to engage with my original post, or make good on your threat to fly away like the decoy 747 that you keep announcing again and again and again, but never delivering.
Okay, but, fair enough, now that you're here (and you're a person with a name and history attached to it), it's less of a waste of time to ponder this.
So personally, I'm not really interested in the hypotheticals of emergent behavior and all that and whether something without memory that works as ours does could develop some different but practically identical form of sentience.
That is possible, but at the end of that hypothetical chain of thoughts might be an outcome in which the human is not the top of the food chain anymore. Which is not to be understood as "oh god, I am full of fear of being obsolete", but "why would I even just spend a second on something that leads to the obsolescence of my kind".
The definition of sentience I believe we must use is one of skin in the game. Only what is bound by the same constraints as we are (mortality, disease, physical presence) may be treated as equal or equal-ish by us, because only then it has constraints that force it to do the same.
Not sure if that answers the questions you've had, but that's that.
> Why don't you directly engage the points in my other post like I just asked you to?
Why would I?
You've been talking about something completely orthogonal to what I was saying.
> I don't need you to lecture me on sealioning
Uhm. I mean. With the context given.. you're.. kinda maybe doing something very similar to that context?
Doesn't really feel like good faith at least.
__
Oh I came up with a cheap shot. I'm sorry but I have to put it here.
Your barking orders of submission at me kinda.. smells like AI interactions. A lot of those.
(I genuinely believe that it does, but it's also clearly inflammatory, lol)
__
> I'm asking you to engage with my original post, or make good on your threat to fly away
This has been a weird, but certainly entertaining interaction. Let's agree to disagree :^)
But seriously, my frame came first. Go engage with that. Why would I engage with your frame that came later and misses my actual point? That would just validate it and invalidate mine (without doing so through merit, I mean).
Also, I don't remember threatening to go away tho? Not sure where that came from.
Aaah.. The "terminating this comment chain"? Nah that was just a pre-buttal for the expected "but this is thought-terminating" defense.
Nice Unthinkable Topic you have there.
As I said, context.
In the right space with real people, worthy of a debate. On HN? No. Not like this. Not here. Not without filtering the participants for real human beings.
[flagged]
[dead]
> Anthropomorphizing is a problem when you're talking about treating something that's not living as if it were.
I mean, that is the entire definition of the word. And you also anthropomorphize living beings like many people genuinely attach human qualities to their pets etc. Yes, the risks are very high when it comes to chatbots in particular, especially to people who are not technically inclined. But you'll be surprised at how crucial the ability of anthropomorphizing is. This is a very good paper that summarizes it and is definitely worth reading if you're interested in these things: https://www.researchgate.net/publication/5936908_On_Seeing_H...
> I mean, that is the entire definition of the word.
Not quite - my wording there was very deliberate. By saying that it's a problem when you're treating something that's not living (not non-human!) as if it were, that excludes pets and all animals from the equation. I understand how common it is for humans to assign human qualities to other things and beings, but there is also an unspoken variable of intensity. Representing abstract concepts as humans, interpreting living things in a human-like way or traditionally referring to ships as living beings has a different degree of belief and intensity compared to implying a genuine belief that algorithms are beings that can be enslaved, like what the sibling comment to this one does.
No its not unspoken. Read the paper linked. It explains what you think you are explaining but with scientific rigor.
And you are wrong in your defintion of the world. Attaching human qualities to any non human entity (living or otherwise) is the accepted defintion of what anthropomorphizing is. It does not only apply to non-living entities.
> I know people who are like that too.
This is part of the problem being described. You are part of the problem.
"Some people are bad at X" is not comparable—is not even in the same category—as "LLMs are fundamentally incapable of X".
Every human (at least to a first approximation) is capable of understanding, of learning, of remembering things, of doing math, of counting the number of "r"s in "strawberry".
What you are observing is that some humans are careless, do not take the time and effort to understand, or have internalized the idea that they're "not smart enough" or "not the type of person" who understands things like <whatever>.
That has nothing remotely to do with the fact that LLMs have no consciousness, no self-awareness, no cognition, no understanding. At a fundamental level.
I don't think anyone in this conversation is saying this behavior is anything but the fault of the user not understanding how these tools work? This is a weirdly aggressive post.
Just a heads up - this person in gp comment, their original post in a way that is much much different than the one I was replying to originally both in tone and content.
This is an extremely common fallacy I've seen lots and lots of people fall into with respect to LLMs. In nearly every case, they use the fact that "some humans can't do X" to claim that LLMs are, in fact, basically conscious/human-like/AGI already.
This is deeply untrue, and is highly likely to lead them to bad conclusions about what we can and should do with LLMs.
> This is an extremely common fallacy ... they use the fact that "some humans can't do X" to claim that LLMs are, in fact, basically conscious/human-like/AGI already.
Claiming that LLMs are conscious or human-like because humans can't do X seems a very strange way to argue for LLM intelligence.
Usually, it goes the other way around: an LLM sceptic says "LLMs are dumb because they can't do X" and soon someone has to remind them that also most of the population can't, in fact, do X.
“ Usually, it goes the other way around: an LLM sceptic says "LLMs are dumb because they can't do X" and soon someone has to remind them that also most of the population can't, in fact, do X.”
And it should not go this way, is the OPs point.
For an analogy, imagine someone looked at a bunch of lightbulbs and expresses dissatisfaction that they aren’t really stars. If someone replies by saying “not all stars are equally bright”, do you think that fact should carry any weight in the argument?
> And it should not go this way, is the OPs point
Well, he's wrong. If you argue that LLMs and humans are fundamentally different because all LLMs do X and no human does it, then showing you that it's not true demolishes your argument. Doesn't prove anything positive, but it certainly proves that your argument is invalid.
Well this requires you to buy the very bullshit argument that comparisons of two physically distinct systems just because they share outputs is meaningful.
If I call a lightbulb an artificial star, the onus is on me to show the behavior under the hood is star like, not just to point at the light and say “you must see it’s a a star since it’s emitting light!”.
> Well this requires you to buy the very bullshit argument
Sorry, no. The only thing it requires you to buy is basic logic. If you argue that B is true because of A, the fact that A is false invalidates your argument (I repeat: not B but your argument). There is no question about it.
"One man's modus ponens..."
> I know people who are like that too.
Cool.
Question: Why _on earth_ would we make more of them?
>I'm not sure anthropomorphizing is a problem.
It is a problem because it does not really understand stuff. For example, if you ask a human "Do you understand that doing X will kill you 100%?". If the human answers "Yes", then you can expect the human to act according to that understanding. That they will not do X
But an LLM will happily acknowledges the consequences of doing X, but will still proceed to do X. So replace the human in the above example with a robot controlled by an LLM. There is no guarantee that it will not do X.
Have you not met humans that do that, too? When it was found that smoking causes cancer, many, many people kept doing it. Then the government taxed the crap out of it, and some people still do it.
It's easy to say that people have other motivations, but doesn't the AI, too?
You tell it not to do X, but you've also told it to do something that would benefit from X. It's going to "want" to still do X, to support that other thing.
It's also got all the "knowledge" that enables it to do the work in the first place, and all of the tendencies of the people who do that work, because that's what it's trained on.
It's really easy to anthropomorphize AI because it was literally modeled after people.
And for the record, as lead developer, I've had actual humans that reported me to go ahead and do things I specifically told them not to.
>smoking causes cancer
Which part of "X will kill you 100%" did you not get?
>Because plenty of people, even ones that should know better, really believe it's a conscious, thinking entity
You need evidence to make the positive claim that LLMs do not posses any form of consciousness.
Consciousness is a slippery word that is notoriously difficult to debate over. But often, people use 'consciousness' as a shortcut or a familiar word to describe a more complex idea. The point they're driving across isn't about the precise definition of the word 'consciousness', but about people treating LLMs as if they were actual human beings, assigning them all the traits and behaviors they would expect of a human.
Well, our current set of evidence is that it’s a mechanistic mechanical algorithm with an RNG embedded in it and we can both get it to repeatedly produce the same output for the same input and also get it to repeatedly do absolutely nothing at all, which are not characteristics we usually find in objects evincing consciousness.
LLMs bear absolutely none of the traits we’ve come to recognize as the external hallmarks of consciousness in biological organisms, nor anything that would seem analogous in a non-biological substrate.
That said, we don’t have a rigorous definition of consciousness that includes the actual phenomenology of consciousness, so I daresay if you’re going to go around asserting the LLM is conscious despite all existing evidence to the contrary, I think the impetus is on you to define some version of consciousness that isn’t also satisfied by a book or a movie.
I would agree that LLMs aren't much like the human brain, that doesn't prove that consciousness is not occurring. Does a fruit fly experience anything? If a microscopic insect can experience something, why can't a CPU?
>if you’re going to go around asserting the LLM is conscious despite all existing evidence to the contrary
Well there is neither any evidence that suggests LLMs are not conscious, and I also never asserted that they are. If I had to guess I would say that any information processing system will produce some kind of conscious experience, but I ultimately have literally no idea.
There’s plenty. First, when we say “LLM”, what are we referring to? What is the entity that would be conscious in this case?
The reason this is important is because powerful people are currently trying to use the dodge that LLMs are conscious to launder liability for their own policy choices, so the sloppy thinking and half-assed conjecture about LLM consciousness has real-world consequences, and every time you assert the question is unknowable you allow that kind of loophole, so it’d behoove all of us for you to spend some time actually digging in on this instead of just idly making or rebutting assertions.
There’s a richer literature here than what you’ve seemed to have engaged with, and I’d encourage you to spend some time with it before handing more money to the magic AI people.
>The reason this is important is because powerful people are currently trying to use the dodge that LLMs are conscious to launder liability for their own policy choices, so the sloppy thinking and half-assed conjecture about LLM consciousness has real-world consequences
I'm not going to change my beliefs or how I think about interesting questions just because its the "socially conscious" thing to do.
>There’s a richer literature here than what you’ve seemed to have engaged with, and I’d encourage you to spend some time with it before handing more money to the magic AI people.
It seems like you are highly emotionally invested in this and that is precluding you from open engagement with the topic.
You're not engaging with the topic. Engaging with the topic is where one either asks questions and listens to the answer or actually seeks to increase one's knowledge on the topic. You're just saying things. That's lazy.
And, you're welcome to do what you want to do, it's your god given right to stay as ignorant as you want about any particular topic, but that comes with consequences. If you want to call that being socially conscious, sure, you do you, but if you're interested in why people keep getting annoyed at your loud proclamations of ignorance which you're trying to proffer as evidence of a curious mind, well, that's why.
It's just as intellectually lazy to leave "consciousness" undefined and proceed to claim that "X cannot be consciousness" because of social reasons. That's the definition of a circular argument.
In the same vein, airplanes can't fly because they don't flap their wings.
As for ignorance: have you tried to make an LLM produce the same output for the same input (as you state above)? Give it a shot, you'll be surprised.
I would argue, that the null hypothesis is that it is not, and that anyone claiming that there is a mote of consciousness are the ones with the burden of proof.
The null hypothesis is that we don't know jack shit about consciousness. Any claim of certainty seems extraordinary to me and I want to hear the evidence.
We know quite a lot about consciousness. You may not, but we know enough to know the informational dynamics in a brain are vastly different from an LLMs.
Exactly what kind of certainty are you looking for? Happy to provide it at a molecular, cellular, tissue or whole brain level.
What causes consciousness? Why aren't we all p-zombies?
Can we conclusively say one way or another if an ant has a conscious experience?
Yes we can say it has conscious experience. The content and depth of it, we cannot yet fully grok, and of course, what it feels like from inside the ant is something we never will know.
Okay, what about a virus.
That's not a null hypothesis, that's a strawman.
Regardless -- if we don't know jack shit about consciousness (your words), then any claims of LLMs being conscious are by definition untestable, pure speculation and based on no evidence at all.
> then any claims of LLMs being conscious are by definition untestable, pure speculation and based on no evidence at all.
Am I saying otherwise?
What's the correct null hypothesis according to you?
Would you want evidence that your your bicycle is not conscious? It can sense your inputs. It responds to the environment. The gears do multiplication (reasoning!).
To make the claim that an object is not conscious you need to both define what consciousness even is, then you need to make a machine capable of measuring it.
If you have done neither or those things then any claim you have about a bicycle's consciousness is just baseless speculation.
I think that the difference is that there is a vast gap in complexity and abilities, no?
What about a Jacquard loom? Is it conscious? There is a vast gap in complexity and abilities between that and a bicycle.
> we don't know jack shit about consciousness
People keep saying this, but it's not true. We know a lot about consciousness. There's a lot we don't know about it, of course, but "jack shit" is wildly incorrect.
> I want to hear the evidence.
If you believe LLMs are conscious, then the onus is on you to provide evidence of such.
the onus is on whoever is making a positive claim.
If you are stating "LLMs are not conscious" you need to provide evidence, just like how if you are stating "LLMs are conscious" you need to provide evidence.
No, the onus is on either claim.
What's wrong with treating it as biology though? Even large software systems have biological aspects, their behaviour is emergent and if you want to observe how they work, a holistic approach is needed, you can't really reason about their full state...
For example, if you have a search engine or a complex game, you can't run tests like "for all inputs the results are correct", you're going to be fudging a lot, using randomness, using heuristics, and all that kinda stuff
Just like how mathematics > physics > chemistry > biology > psychology > economics/sociology (Auguste Comte's hierarchy reordered a bit for the modern day), moving up the abstraction ladder makes things more complex, less legible and less exact.
> Is anthropomorphizing a real problem?
The paper argues that pretending that the so-called thinking traces represent real reasoning can lead users into trusting wrong answers, if the thinking traces appear convincing enough. Researchers might inspect these traces to try to determine the “intent” of a model, as well.
For an example of the latter, when OpenAI spoke about the hacking of HuggingFace at Black Hat, they repeatedly showed the thinking traces of their model as “proof” of what the model was “thinking” as it performed the attack, calling out “surprise” moments, etc.
Now, it’s possible that the employees presenting didn’t truly believe that the thinking traces would give them useful clues, and presented them only for a “wow” factor, but I wouldn’t discount the possibility that even the people working at frontier companies can fall for this tendency to anthropomorphize LLMs.
But how is that any different than people being misled by real humans saying words that reflect real thinking, but which are actually dead wrong?
The fallacy here is "thinking == correct", not "tokens == thinking"
Because the real thinking still cost the other human the same-ish energy it costs you to put words together, and because after all, the source is a human and not a machine, no, this is very different.
Being mislead may be the shared outcome. But why is different category of source of the mistake and the cost to producer of making the mistake not relevant in this discussion?
Where else in science do you brush aside all differences this way?
Why aren't humans simply biological machines?
There is no "science" that GP is brushing aside. You need to provide repeatable observations or experiments that GP is ignoring.
Please define "simply biological machines". I'm not sure "biological machine" had a proper definition. What's machine like about biology exactly?
A "machine" is a term that is well defined in science. https://en.wikipedia.org/wiki/Machine
And biological machine is? Don’t get me wrong. Biology I is full of molecules that we call machines. But you’re making a broader claim, saying that biology is only this.
This needs you to answer some questions:
1. Why do the machine parts in biology show such flexible application? A gear cog won’t ever moonlight as a signaling chip, but in biology you often have molecules doing double and triple duty.
2. How is the biological machine able to build itself? What does self assembly imply for the machines function?
3. Where does this machine get its inner drive? No LLM has been found that starts outputting text unprompted. A car doesn’t decide to move to a shady parking spot. Why? Where in the machine to biological machine continuum does the ability to make internally driven decisions come in? Why does it come in for biology? A bacterium is able to make such agentic decisions unprompted. Why is no manufactured machine able to do this?
> And biological machine is?
"Biological," too, is well defined. It relates to living things and their processes.
> Why do the machine parts in biology show such flexible application?
Evolution.
> A gear cog won’t ever moonlight as a signaling chip
A gear cog was purpose built for that purpose, but you will find that people often recycle parts into other systems, often in completely different roles.
> How is the biological machine able to build itself?
Protein synthesis.
> What does self assembly imply for the machines function?
The way that a machine is built has no bearing on how the machine functions. I could build the same machine using a 3d printer or a CNC router.
> Where does this machine get its inner drive?
Evolution selected for organisms that survive long enough to reproduce. Different biological systems handle this differently.
> No LLM has been found that starts outputting text unprompted.
If you give an agent a goal, it will perform actions to achieve that goal. This is just as true for artificial agents as it is for biological agents.
> Why is no manufactured machine able to do this?
Many do. Even robotic vacuum cleaners will charge themselves without human prompting.
FWIW I think the paper's argumentation is extremely weak to begin with. Like in section 4.1, it opens by expressing a sound position of skepticism:
> there are significant questions on whether these traces have any valid semantic import to the end user.
Which it contradicts in the very next paragraph, taking a stance that there are no valid semantics present in the trace:
> the false idea that derivational traces are semantically meaningful
It's really not a high quality paper worth taking seriously.
And that's before we get into the complete and total breakdown of objective analysis. It rejects distributional semantics as a theory, while also explicitly stating the results that have been produced under its auspices are "undeniable". Never elaborated on, and at no point in the paper am I given the impression the authors are even aware of the problem with this. It's just more unempirical slop that wants its pound of flesh without putting the work in. Frankly, whoever let this through peer review should be ashamed of themselves.
Yes it is a very serious problem because it confuses a lot of folks with a great deal of power like judges and policymakers.
The first book I ever read on ML (late 90s) dedicated the entire first or second chapter exploring the distinctions between artificial and biological neurons, and even talked a bit about the philosophy of modelling. I still remember thinking back then why would the authors spend so many pages on this but now I believe it was because they understood that a metaphor can be a double-edged sword.
To be fair, the ANN architecture underneath is a misleading thing to be looking at, it's not where the comparison comes from. Though I can't tell if you meant it to be relevant in that way, or just as a general example for the dangerous nature of metaphor.
LLMs are expressly designed to approximate human behavior within the bounds of the written word. The anthropomorphization is no more philosophically problematic than saying differential calculus measures curves.
I meant it in the latter way: a metaphor can be useful as a pedagogical tool to introduce new ideas, and using the source of inspiration for this idea as the metaphor itself makes perfect sense, but unfortunately our brains seem to be prone to assign other properties of the metaphor that don't actually belong to the object of study.
I imagine this happens because we tend to conflate things that are similar, or maybe because it's not entirely clear which characteristics are being mapped in the metaphor?
They spent so many pages discussing it only to show that the mechanism for how ANNs work is different from the mechanism for how biological brains work. It says nothing about whether they can compute the same things.
> Is anthropomorphizing a real problem?
Of course it is. Anthropomorphizing is in our nature, but it doesn’t mean we have to entertain it and extend it to everything. A poet can anthropomorphize clouds beautifully and I’d enjoy his poem, but I want my pilot to not see clouds as rabbits when they decide if it’s safe to fly through them.
> but I want my pilot to not see clouds as rabbits
The poster meant it "within the metaphors". Tables are said to have 'legs': that does not confuse carpenters.
Yup, that’s why it’s important to ground these arguments in reality.
> none of the serious LLM researchers believe it has anything to do with human reasoning
But some of the biggest evangelists, who are well respected programmers that get lauded on this very site, have said it is fully sentient and has emotions. Even going back to 2022, when the LLMs were dogshit, a Google employee lost his job claiming it was sentient because it said it had emotions.
Combine that with the marketing angle of both Anthropic and OpenAI, who have been trying their hardest to describe every function of an LLM as analogous to the human brain. Because it's politically useful to paint them as dangerous and uncontrollable, so the keys will only be granted to the few people on the mountaintop.
I actually think on the LLM side it might be beneficial to refer to them as <thinking> because it explicitely guide the token generation towards a "thinking space".
As weird as it is, anthropomorphizing LLMs in prompts has been actually pretty useful (think of the latest big math discoveries which were achieved by having the user giving supporting words). It would be interesting to see if a LLM would perform worse if you used a more neutral term.
The paper's argument is rather than using terms like "thinking trace" can lead people to believe that the model is really thinking, and thus these traces can be used as a sort of interpratbility parameter. This can give a false sense of security when building a LLM-based system which requires guardrails and tracability.
> Is anthropomorphizing a real problem?
Completely rational and smart people talk to their pets, plants, their car, and other inanimate objects. This is not considered abnormal by most. It's just what we are wired to do. Some LLMs are uncannily good at tricking people into believing they are talking to a real person. So, there is that as well.
Some people are a bit freaked out by this or still somewhat in denial about LLMs being this good. But people have been yelling at their computers for as long as we've had them; so that ship sailed a long time ago. Trying to stop them doing that is probably a bit futile.
Whether people like this or not, LLMs are actually trained and fine tuned on real conversations and that's where a lot of this is re-enforced. Instead of fighting that, you can just lean into it and accept that communicating like you would with a person totally works and can actually be efficient even as it requires less effort and thinking on your side.
You can go all Jean Luc Picard on AIs and yell "Tea! Earl Grey Hot!" or you can just ask "I'd like a cup of tea, please". LLMs are good at remembering your tea preference. The please is of course completely redundant and should not affect the outcome. If you just want a cup of tea, you should be fine either way.
> Is anthropomorphizing a real problem
Yes. There's a difference between scrapping a session and starting over, or going back and branching something, or using sub-agents to see five outcomes, vs arguing with a system in a long drawn out chat.
Like - I know that if a model starts doing something silly, instead of correcting it - I can probably go back and edit two steps prior to add an extra guardrail, or extra data, or whatever.
Simplifying terminology is not a problem. The providers intentionally choosing terminology to make people think it's something it's not is a problem. I hate the term agent. Calling them companions as some do is just gross.
When I was taking an MIT AI course (in ancient pre-LLM times), an autonomous agent was defined as a system that perceives its environment and acts on it (we were focusing on reward-expectation-maximizing agents, but it's not that important). Peter Norvig has said something like, technically, anything can be described as an agent (a rock maximizes the "follow physical laws" objective), but naturally, it doesn't make much sense to model a rock as an agent. With AI agents, the situation is significantly less controversial: they do perceive, deliberate, and act.
All of these terms were picked by individuals, years ago, while reaching for metaphors that made sense to them personally.
None of these "agent" / "thinking" / "reasoning" terms were dreamed up in boardrooms to intentionally mislead people. They are useful but faulty metaphors; there is no conspiracy.
"Agent" is standard reinforcement learning terminology, used in Chris Watkins's thesis introducing Q-Learning in 1989.
> Is anthropomorphizing a real problem?
Yes it's really a problem. On this website you are surrounded by people who have technical knowledge and understand at least somewhat, how a computer functions. You have the ability to separate "fun" and "reality" because you know you're putting input into a really really big calculator. Most people do not fathom this.
AI Psychosis is a real thing, look it up (don't just ask an LLM) and do some reading. It's actively harming people, and the way they think. There's no regulation around any of this stuff and it drives me crazy that we let these AI companies _sprint_ so far ahead of everyone, and now we're facing the consequences.
I find it annoying because when I read ML papers nowadays I have to back-translate from anthropomorphized talk into actual machine talk, then mentally compare to what I actually know about brains and cognition.
> It's more fun like that :) Doesn't mean I genuinely believe a MySQL instance actually thinks.
A lot of people are not in on the joke. ELIZA effect and AI psychosis is a thing.
Interacting a lot with LLMs might be damaging to the human psyche even for mentally stable people.
"Esta poronga no sabe lo que está haciendo", aunque parezca femenino , en realidad no tiene género .
> Is anthropomorphizing a real problem?
Even tech companies are rolling out AI training which utterly anthropomorphizes it, and leads people to think its actually intelligence. This is part of the reason for the backlash - everyone understands it bullshit marketing the second you actually try to use it.
there are people on the fringes in relationships with these things, so yes definitely
[dead]
Coming from a more traditional stats/ML background, I try to view "thinking" traces as a way to explore the search space without getting caught in a local maximum.
A better analogy for me is annealing; you can't cool metal down instantly or the result is brittle. You must cool down gradually, which allows the molecules to arrange into more durable structures. Random but controlled.
In the same way, thinking traces are testing out all sorts of novel connections between tokens ("But wait..", "Actually,.."). Like a highly divergent branching mind map that gets pruned over time, rather than settling directly into the initial answer.
Now consider human cognition. We're constantly diverging, daydreaming, playing "what if" scenarios and measuring up those ideas against our internal objective functions (proxy for reality) to see which ideas stick. Not too dissimilar. But it's hard to call what we do "thinking" either - it's the default mode network wandering.
this seems like more anthropomorphizing. even if it has similar results often enough to be useful, next token prediction is not searching a solution space.
you even end with a paragraph saying “it’s not too dissimilar from what we do”. how is that not anthropomorphizing?
and if what we do isn’t “thinking” then what is or ever has been?
> next token prediction is not searching a solution space.
Interesting take. Next token prediction (via the attention mechanism) is a "walk" through the token embedding space. Searching the solution space is what it does, mathematically. It's how we take tokens x context length possible combinations and prune them to converge on viable answers so quickly.
Does it look like search at inference time? No. With given weights, a given prompt, and a given random seed, you get the exact same answer. There's not much searching happening at inference...
The key is that most of that space is searched at training time. The weights implicitly prune the search space, blocking off or make certain token combinations effectively impossible. It's easy to think "we're just applying weights at inference time" without considering all the pre-work that's done to prune that search space.
Which is exactly why "thinking" traces (and randomization) are useful! They bust out of any local optima created by too-tightly-constrained models or system prompts. It's both useful and technically correct to speak of the process as a high-dimension combinatorial search.
Isn't search a strange analogy for how weight terms propagate?
> While a human may say “aha” to indicate exactly a sudden internal state change, this interpretation is unwarranted for models which do not have any such internal state, and which on the next forward pass will only differ from the pre-aha pass by the inclusion of that single token in their context. Interpreting the “aha” moment as meaningful exemplifies the long-neglected assumption about long CoT models – the false idea that derivational traces are semantically meaningful, either in resemblance to algorithm traces or to human reasoning.
This paper addresses something that has always bothered me about LLMs. You read their reasoning, see something like “Wait, that’s wrong” and then watch them make the exact mistake they just identified.
By itself, "aha" carries no insight, but the insight is probably stated immediately after it. In that case the aha is semantically useful, by identifying the insight it is near.
Ooh so lets just change the initial prompt to
and since Aha! is near the good stuff in the network it will just work =P> but the insight is probably stated immediately after it.
If the intermediate tokens represent reasoning or thought, you would expect "aha" to occur after the thoughts that led to the realisation, including the thoughts encoding the explanation: they don't have any other state. There is no reason to draw the conclusion you've drawn. Furthermore, what LLMs are doing isn't thought.
I don't get your argument.
Let's say that the forward pass that selected "Aha" produces activations that indicate a wrong assumption, and a plausible explanation.
It puts learned projections of the activation into the KV Cache and outputs Aha.
Both the cached projections and the current Aha token can now influence further activations in an additional Forward pass that the Aha bought the model.
At least that's how I thought it works.
A cache is just a cache. I'm not sure what significance you're ascribing to it.
What is put in the cache?
Things that the software running the model would otherwise recompute, if not for the cache. What special meaning are you assigning to it?
Beats me how it works, honestly can't wrap my head around it.
From what I understand, at position Aha in each layer it's constructing a query based on the current activation and looking at the key of each other token position for that layer, in order to decide how much attention to pay to the value.
In this way it attends to the previous values, such as perhaps the incorrect assumption and plausible explanation.
Consider this: while the inner state of an LLM (all its activations, residuals stream that is cached in the KV cache) is fully deterministic given its input sequence, the information contained in it IS NOT identical to the information in the input sequence. The reason is obvious: the LLM itself contains an enormous amount of information in its parameters and it transfers it to its residuals stream at each forward pass.
In other words: the final state given the two input sequences (where NT stands for "null token"):
<problem-prompt> [NT]
and
<problem-prompt> [NT] [NT] [NT] [NT] [NT] [NT] [NT] [NT]
is not the same, and at each forward pass the LLM keeps working on the solution even if the input tokens provide absolutely no further information.
If this is correct, then there is no need for the model to have already verbalized the key elements that drive the "aha" moment, so no need for the "aha" to appear after a full explanation.
> If the intermediate tokens represent reasoning or thought, you would expect "aha" to occur after the thoughts that led to the realisation, including the thoughts encoding the explanation: they don't have any other state.
Yes they do, they have their KV caches-- it's a pure function of the input tokens, sure but that doesn't prevent it from containing latent 'insight'. LLMs can and do pre-form the tokens they're expecting to output multiple steps in the future.
I wouldn't argue that the 'aha' means anything, but the structural argument that it can't that I think you're making isn't sound.
It really isn't useful though, unless it is a summary. At best it is a semantic trick to tell the next iteration to come up with something smart.
it's a rhetorical heuristic that a writer should know to use when directing a reader to a declarative that they want them to pay attention to, usually because it's a non-obvious or roundabout insight
when utilized by AI, it's a probabilistic output and it's variable whether or not that rhetorical trick is useful. it also pushes a non-skeptical reader to focus too much on the following text or even to believe that they, themselves, derived some insight. this is effectively a kind of persuasive sophistry which is not helpful - adding rules around it prevents people from deluding themselves with AI
Did not read the paper so apologies if this is covered but isn't it possible that there is some recognizable semantic pattern in the training data where an "aha" is often followed by a subtle semantic shift that proves closer to the original premise in some critical way, and by emitting the "aha" token the model causes itself to produce such a subtle semantic shift that pushes the subsequent reasoning closer to the desired response?
It amounts to noise overall, but it has further unwanted and potentially misleading 'properties'. I think it's rather sobering to see how much bandwidth is still being wasted.
“Aha” as a single token records the LLM discovered it made a mistake and needs to pivot.
On the next forward pass: it rediscovers the mistake, its “aha” noting that, and then provides the first token of the new idea.
That “aha” contains information: the previous conclusion was somehow insufficient.
The anthropomorphization of LLMs should be discouraged as much as possible. It perpetuates bad practices and encourages the use of these bots for tasks they are not intended for (particularly as chatbots).
Thinking traces should be treated as black boxes. There is no point in reading them. Only the LLMs’ conclusions are relevant. This is particularly true of Opus 5, which employs reasoning that seems highly questionable but very often reaches excellent conclusions (compared to its peers)
Sometimes I monitor thinking traces for misunderstandings (missing context / bad assumptions). If it's going to go off on a ~20 min task and I can catch it's going in the wrong direction in the first minute I save a lot of tokens and wasted time. I don't monitor the whole thing, mostly just the first bit to see if there was a gap or misalignment in intention.
As an aside, anthropomorphization has nothing to do with my motivations.
> Thinking traces should be treated as black boxes. There is no point in reading them.
Just a few minutes ago i was reading Qwen 3.8 27B's reasoning when i asked it to do something that was computationally intensive and it started going down the rabbit hole of doing it using some GPU acceleration approach - even after leaving it to "think" for a bit, it never realized there is another and simpler way. So i stopped the generation and added a "note" saying that as the problem is computationally intensive, it could become much faster if using an alternative approach.
At least in my experience (with local LLMs, i don't know how the cloud stuff behaves) what LLMs "do" tend to correlate with what they "think", so being able to read what they "think" is valuable.
Anthropic's reasoning output isn't the real model reasoning but some sloppified summary of it.
> The anthropomorphization of LLMs should be discouraged as much as possible.
And yet, they have extensive human-like behavior. If you treat them nicely or encourage them, they perform better.
Ignoring that human-like behavior is wrong headed.
I think you're ending that train of thought too early. Why does this occur?
Well... We can hypothesize that these things are largely trained on internet dialogue so there's probably some correlation between threads where people are not flaming each other and the quality of the replies. They're just statistical engines so anything you can do to raise the odds of a helpful next token...
I'm essentially just making shit up here, maybe it's right, maybe it isn't, but rather than saying "it's human and we should treat it so" we're trying to get to the ground truth of how it works.
Did I say "it's human and we should treat it so"?
Sheesh. Yes, I agree with you entirely. I'm merely pointing out that ignoring this behavior is dumb, too.
And probably not rationally based. Leads people to make crazy jumps. :)
Yeah sorry, I read too far into your position. There's a certain faction within these AI discussions that wants to over-anthropomorphize the LLMs in kind of a borderline spiritual way.
Oh that's a shame - I hadn't seen that, but I can believe it.
The philosophers who study these things have been clear for a long time - we can never know what it feels like to be in a digital brain. Or any brain for that matter. When push comes to shove we all might be phantoms in some guy's dream.
Don't know + can't know. I think that was the real point of the Turing test. Not: this means it's conscious. Just: this is the best we can ever hope to do.
A while back I made an "OpenClaw in 50 lines" by just wrapping Claude Code in a Telegram bot.
I asked it for the weather. "I don't know that. I'm just a programmer."
I added "believe in yourself, you can do anything" to sysprompt, suddenly it had the confidence to Google the weather...
That's not a consequence of an LLM. It's a consequence of the training data. In fact, I would argue that the latest models aren't nearly as sensitive to the tone of input anymore. It's an issue that has been addressed by better curating training data.
i would agree with your if it weren’t for this article recently published by anthropic:
https://www.anthropic.com/research/riemann-zeta
“Throughout this process, Jarred's input was mostly limited to sending Claude messages of encouragement (mostly variants of “keep going” or “believe in yourself”).2 This seems to have helped Claude overcome some initial skepticism that it could make meaningful progress.”
Although I 100% agree that the core mechanism of GRPO is purely mechanical token-by-token probability generation, because RL only rewards exact final answers, the training forces the model to develop error-correction habits. This makes the output extremely like human thinking when solving a problem. It's like the order of the thinking tokens is what causes it to get that sweet, delicious reward, and this order seems like a reflection of the human thinking process.
I created a flame graph classification of thinking-token phrases into setup, execution, decomposition, verification, error correction, surrender, and deliberation, or classified as steps in an OODA loop, which is more of a reach. It literally has a verification step and, if it finds an error, an error-correction step.
If there is a verification sequence of tokens with an error-correction sequence of tokens during RL training, it will perform better; and if humans do these steps (did you proofread your reply to this comment? did you correct it?), they will perform better — which is why it is so easy to make the anthropomorphizing metaphor.
Nonetheless, the paper is 100% correct that these machines are not thinking like humans.
https://adamsohn.com/reasoning-grid/
https://adamsohn.com/lambda-variance/
“ This makes the output extremely like human thinking when solving a problem.”
This sounds a little like someone saying a lightbulbs output is extremely like the output of stellar fusion. In one sense, yes. Bulbs are in fact designed to take over when our nearest star is beyond the horizon.
But that really doesn’t mean you call the bulbs mini stars.
How much is the process of a human child in grade school working through a 3-digit x 3-digit multiplication problem (123 * 456) like a GRPO model with thinking tokens doing the same?
Humans are not born being able to achieve that. It is learned behavior. You and everyone else will remember their teacher saying, "Check your work!" Both the human child and the model work through multiplication problems using the same technique, using the distributive property. They both try to get a reward. For the human child, it is a sense of someone commending them for correctly solving the problem, a reward that probably yields some type of positive dopamine or serotonin feedback loop.
The model solving the problem will have a lower error rate if the first series of tokens created is followed by a series of validation tokens that are subsequently followed by error-correction tokens if there is an error!!!
Maybe it is thinking. Maybe it is remembering to validate and check the work and then remembering to fix the error. For the model trained with RL, why did tokens associated with validation towards the middle of a stream of tokens yield much better results? DeepSeek proved with R1-Zero that a model will learn to verify and correct itself from RL alone with no supervised fine tuning (SFT) teacher ever showing it how. The only reason DeepSeek used SFT was to clean up the reasoning tokens to be human readable. [0] When constrained by SFT, the models will use the double meaning of words -- polysemy -- to satisfy being human-readable while also carrying meaning for what they are working on.
Different people think differently. I watched a viral video of some ~11-year-old child talking to his mom or dad about a stream of a voice in his head. He discovered for the first time that he has a stream of thought. When he goes to school and solves a long multiplication problem, like the stream of tokens from the model, that voice will say to itself (him), "Check your work!"
That is a case of the stream of thought as words being aware of the stream of thoughts as words. Self awareness is a different conversation.
What I think is happening is that the child's stream of thought while solving a multiplication problem in school is likely very similar to an AI model's stream of tokens solving a multiplication problem. And they both were learned. The mechanics are very different, yet, the analogy is apt.
[0] https://huggingface.co/chutesai/DeepSeek-R1-NextN/blob/main/...
How much is the process of a human child in grade school working through a 3-digit x 3-digit multiplication problem (123 * 456) like a GRPO model with thinking tokens doing the same?
Very little, if you bother to give the biology of the child at least a cursory glance.
Let’s take a short peek:
1. Assuming this is normal grade school, and inflicts math upon children earlier in the day, this is somewhere between 7 and 10/11 am, let’s say? At this point, depending on the age, gender, and maturity of the child, every neuron in their brain involved in math is likely off their midday peak in cognitive function.
If we move the class to later, a different subset of students will be at the peak.
As far as I am aware, GRPO models do not have such internal temporal rhythms driving their behavior that will shape their performance.
2. How well a given child performs will depend on how hungry they are. But not deterministically. If you trivially think each child is like a computer, you may think the rich kid who had a breakfast buffet before coming to school will do better than the half-starved child of a janitor, but that child might mind the lesson with greater intensity. Or not. It’s not something you can pre-calculate with any certainty.
While chip to chip variability is certainly known, I’m yet to hear of a chip deciding to do math better and faster than its fellow chips to prove a point. Or to do significantly worse because it’s distracted by the bird on the window sill.
What you are noticing is that there are limited ways to solve a 3x3 digit multiplication. Humans, having standardized the process, have now found a way to record it and plug it into correctly translated signal so the same accurate result can be had without using our own minds in the moment.
But where I’d not remotely be shocked if a kid from an uncontacted tribe figured out 3 digit multiplication to keep track of his stone collection, I’d be highly shocked if an H100 that was dumped in the trash by accident somehow figured out anything at all. In fact, if it manage to move any of its electrons around on its own, it would be a certified miracle.
And then we could talk about there being real similarity even though the specific atomic composition is different.*
Maybe more like a hydroponic setup with grow lights where in some cases it’ll outperform your natural sunlight and dirt.
The visible light produced by both an incandescent bulb and a star is a result of black body radiation, but otherwise, I don't understand your point. A light bulb produces light, something we might have relied on stars to do before. An LLM produces thoughts, something we might have relied on people to do before. Nobody is claiming the process by which the thoughts are produced is the same, only that they both produce thoughts, just as nobody claims the process by which an LED produces light is the same as the process a star uses to produce light, only that they both produce light.
LLMs produce language. And you're right, if we restricted claims to that, no one would object. It might even be scientifically accurate, shock of shocks.
If you claim LLMs produce thought, it's equivalent to claiming light bulbs undergo fission. Pure wish fulfillment. Language is not the extent of thought, and calling a language producing machine necessarily a thinking machine is an old old mistake.
> If you claim LLMs produce thought, it's equivalent to claiming light bulbs undergo fission
You're making a huge logical leap. There is nothing equivalent about these claims other than that they are made in English.
> calling a language producing machine necessarily a thinking machine is an old old mistake.
Nobody claims that all language models think. The small markov chain language models of old clearly aren't thinking and produce a lot of gibberish. The difference is that, to the surprise of many people several years ago, but to the surprise of nobody who has been following along today, the corpus of all text produced by humans contains within it information about how the world works and also information about how to reason. Using that corpus to train a sufficiently large language model causes the language model to learn a world model and a reasoning model in order to produce text that matches the training data. The reasoning model can be used to perform longer chain thinking with test time compute techniques. People who think deeply for a living recognize thinking when they see it. https://scottaaronson.blog/?p=9979
I suspect, it does not matter whether it is thinking or not. The point is to convince most of us that it is thinking.
There is a useful engineering consequence here beyond terminology.
If intermediate tokens are not a faithful representation of the computation, then they are a pretty bad audit artifact too. We probably shouldn't be trying to make the model's internal narration more interpretable., but rather the computation around it more reproducible.
Record the actual inputs, model/version/configuration, tool observations and outputs, then make the execution replayable enough that differences between runs can be isolated.
In other words, don't ask the model to explain what it thought, and instead make the system able to show what actually happened.
Strong dislike for papers that tell me what to do in the title, especially when even the paper admits a loose correlation of the intermediate tokens compared to solution correctness. My solutions work and they speak for themselves.
This is a position paper. Its purpose is to advocate for a specific viewpoint to the ML community.
From [1]: "Position papers make an argument for a viewpoint or perspective about what should be done [...]"
[1] https://icml.cc/Conferences/2026/CallForPositionPapers
Sure, and the parent comment's position is that they dislike it. Its purpose is to advocate against clickbait titles becoming normalized in the scientific community.
This is the opposite of clickbait. The topic is obvious from the title.
Clickbait doesn't have to be false, it has to be shocking. Being false is one way of being shocking. "Stop doing X!" -- really now?
> My solutions work and they speak for themselves.
I understand the sentiment, and I also use the "thinking" traces as insight, but wouldn't you want your solutions to be based upon a good understanding? If the correlation is weak, then our solution is also weak.
I admit I didn't read the paper, but if thinking traces are not "thinking", then what are they? If their content is not representing progress towards a solution then they are irrelevant and we should just be able to remove them and save a lot of time and money. There's a lot of money to be made by doing so. So why are they there at all? What do they represent?
A paper title is marketing, you’re expected to read its content
I hate articles that do this as well in the title. It's basically just a form of clickbait.
Peculiarly vocal, where were all these people when they started calling the machines computers, anthropomorphizing them akin to the original human (most often female) computers that used to run such calculations? And how dangerous the consequences, we've been dead reckoning for 60-70 years with the wrong terminology without course correction!
Where were these vocal people when the "raster-oriented ink deposition machines" were being called "printers"? The meat or machine brains of future historians will melt because they can't handle ambiguity, a word gaining extra -yet similar- meaning! A word with multiple meanings, unheard of!
Where were these vocal people when people started using software terminology like "executing", "calling", "throwing and catching errors", as if software were human -clownlike sure- but human?
The danger!
Computing is a task. Printing is a task. It’s not wrong to call both a human and a machine a “computer”, because computing (applying an algorithm to an input and producing an output) is literally what they are doing. Same way you can have human and machine diggers, cleaners, calculators and lots more.
Anthropomorphizing comes in when we attribute much more complex behaviors to them - chain of thought, reasoning, intelligence. At that point you aren't talking about mechanical concepts but claiming that these machines are exhibiting human behavior.
Making predictions is also a task.
Whenever we reason, we are not neurotically bruteforcing the possibilities (although sometimes we do, like proof by exhaustion), usually we make predictions heuristically of which assumptions or theorems apply or might apply. It's not any different for machine proof assistants...
Reasoning is mechanical.
I think a problem is that previously thought, reasoning, and intelligence were always co-occurring, but now we have machines capable of (limited) reasoning that do not think or have intelligence.
They were there, complaining. You just don't remember them because it's easier for the meaning of a word to shift, or at least take on additional contextual meaning, than it is to get people to use a new word once it's reached critical mass. Those people lost the language fight, but were arguably still vindicated, to the extent they were railing against misguided beliefs that equivocated the capacity of the new machines with their human (or more human-involved) predecessor technologies. Who you also don't remember are the people who made extravagant claims and prognostications based on the equivocation.
were they complaining about terminology, or were they complaining about the prospect of losing their jobs?
I'd be happy to revise my opinion if you can demonstrate similar vocal strength on the terminological aspects for those transitions...
You also shifted the goal posts from qualitative to quantitative performance claims. If we ignore that technologies have multiple figures of merit and pretend it's one dimensional, there is a difference between the claim that the machine isn't "printing" vs the machine isn't "printing as well as a human would".
I don't think any of the human printers in the past exceeded the performance levels of current printing technologies, but surely they did exceed the very first machine printers, every technology gets a foot in the door in some niche, and then progressively captures the initially not-yet-automated skills of machine operators.
Would you say an industrial textile weaving machine doesn't weave? At the end of the day its just automation all over again.
OK, you can split a hair with your bare hand while blindfolded. Congrats, I guess.
I don't find this take useful. We have little understanding of what conscience is, and how human train of thought really works. The modern LLMs IMO resemble more and more Chinese room problem. Maybe we don't like the mechanistic linear-algebra-based steps involved in the production of the output, but the end result is closer and closer to people's output. IMO this is very natural to start anthropomorphizing that.
What is the value in anthropomorphizing it? Often I get the feeling that it's more about the sales pitch of selling these AI as general intelligence than it is about providing a truthful insight into how these models actually arrive at the output.
I don't think there is value per se, but it is just natural (at least in the environment I am in). I.e. when we discuss the analysis/code/ideas from one LLM or another, we describe it Claude/Gemini/Codex/etc did that and it comes like a person.
While I tend to agree on the overall sentiment, I think this rebuke is inaccurate. Some of these "reasoning" models are trained using "Chain-of-Thought" where the model is presented explicit, intermediate reasoning steps (either by a human or some automation) that supposedly get it closer to the correct answer. These intermediate steps are what was originally called "thinking traces" - not what the model produces to mimic them.
But yes, anthropomorphizing model outputs leads to worse outcomes.
Seems like they are closer to scratch than reasoning... Generating some scratch to draw from helps make it easier to compute the real answer.
That's my personal theory too. The model is stuffing its own context with vaguely related tokens, which helps the attention heads retrieve the right tokens.
Yup. You basically just need something for probability to push off of
It's also interesting because in humans the existence of "Aha!" moments that are not preceded by or are only loosely related to a chain of thought is taken as the proof of the fundamental mystery and irreproducibility of human intelligence. Now the same argument is made to deny that LLMs actually think. Go figure.
It feels apparent to me that LLMs don't do what is colloquially thought of as thinking.
What is less apparent is that humans do.
> What is less apparent is that humans do.
This seems indeed one obvious hole in the argument of the paper. There is no indication whatsoever that human thinking process is more reliable than LLMs intermediate tokens. Which doesn't make our thinking useless, as messy as it might be. We reorder and explain it after the fact.
I assume theyre searching the local gradient to see if theres a better descent before proceeding.
LLMs dont do gradient descent to generate tokens.
They are trained by gradient descent, but inference doesnt involve it.
I don't think there's anything like that going on. They just word vomit into a secondary area, and then there is an internal prompt that says "clean this up and summarize for the user".
Less "internal prompt" and more "they are trained to summarize after a </think> token"
The training methods try not to apply any particular rules to the contents of the thinking text. That's called "optimization pressure on CoT" and is thought to reduce safety by inducing the model to lie (or stop clearly printing its intentions) in the thinking text.
The mistake is not in saying that the machine thinks. It is in supposing that, because we have used the word "thinks", there must be something in the machine which is the thinking.
Related:
Poster side dialogue and Q&A about this work at ICML.
https://news.ycombinator.com/item?id=49277303
I wondered recently about why we stopped with the semantic split of thinking/actions vs user facing communication because the all powerful tool calling craze. Code comments that talk about the prompt is an obvious byproduct of the mixed context. Chain of thought and ReAct were great, but feel like a first pass moreso than the final landing spot.
https://n.zip/2e85 I had sorta started a SFT to explore this idea. it would prove results even with smaller models that fit on a DGX spark. Just takes some more thinking rather than my ADHD mind.
I'm not sure how to test this but I think there's an interesting possibility where the "reasoning" tokens are actually both an accurate reflection of a line of reasoning, but also, that there can be changes in the weights as the computation proceeds onward that may not be reflected in the apparently nominal meaning of the human language the tokens are output as for our consumption.
Some modest evidence is my own subjective experience of the many times I've explained why I'm doing something, and it is a true explanation in the sense that it is certainly not a lie, but it is also incomplete and there are entire strands of thought that went into my decision that are not being articulated. Though human speech is not equivalent to an LLM's output since we can trivially think without literally speaking whereas they can not. (No need to nitpick on the definitions there; all I'm observing here is that they are forced to emit an externally-visible artifact whereas I can sit in silence, thinking, with no externally-visible artifact being produced. Not trying to make any grand claims about what is "real" cognition or anything.)
It is conceivable how to create a test of whether the tokens correspond to the "real" thought process, and papers and work on that have been done, such as [1]. It is difficult for me to imagine how to scramble the nominal tokens without also completely trashing any implicit calculations that may be occurring too.
[1]: https://transformer-circuits.pub/2025/attribution-graphs/bio...
I've been calling them film noir internal monologues, within the documents being generated by the LLM which happen to look like movie scripts.
In other words, it isn't qualitatively different from character dialogue. "Keep cheese on your pizza by using glue" is the same problem regardless of whether the script calls for the character to speak it out-loud or not.
Hypermentalizing (referred to as excessive theory of mind or biased mindreading) is defined as the tendency to make assumptions about other people's mental states that go beyond observable data.
In the past I managed to get measureable performance optimizing a harness by looking at few traces to see if the traces contained surprised, a lot of text in order to figure out how to use my custom tool, then renamed the tool, changed some parameters and it was already great across around 20 eval tasks in rust/typescript, I repeated the same more recently but I used an llm to look at the traces... didn't achieve the desired result, mostly due to how cost-prohibitive it's for me to run expensive models.
While reading this ,,paper'' I did some Learned Prompt Augmentation in my head about what I should comment, and realized that there's nothing interesting to write about it.
Pretty wild dressing a blog post up as a scientific paper.
It's called a position paper, and it summarizes previous empirical research from the same group.
But just so I don't waste your time with human thought, I asked Claude if it would call this a scientific paper, and it said yes.
A meaningful part of the scientific literature is opinion pieces or blog posts. There isn’t anything wrong with that
It's more of a story about a character that is sentient than an actual sentient character.
I think of tokens as fractional digital librarians.
None of the analyTical.
I can agree that not calling it "reasoning" may be correct.
But who knows what human "thinking" is really about. If I find a solution to something it is seldom by painstakingly tracing that A and B leads to C (for that I'd need pen and paper). Rather, thoughts just swirl around and then suddenly a solution, or a hunch about a direction to go in, pops into my mind. Who knows what such thoughts "look like" in humans. It is not all of it I can introspect.
Yes I can sort of follow along some kind of train of thought in my head, but there's a lot going on between each thing I'm consciously aware of that I'm not aware of at all, which probably dominates what you are consciously aware of. (Humans are experts at post-rationalization and so on.)
I see this pattern a lot in AI anthro discussions: (1) Assume humans are some kind of perfect idealistic reasonable beings. (2) Hold LLMs up to the standard of an perfect idealistic reasonable being. (3) Conclude that LLMs fails this test, and are therefore not "intelligent", or in this case "thinking", like humans are.
Problem with the argument is comparing humans in anyway to something that is idealistic, reasonable, intelligent in the sense that is implied in these discussions. Human minds are a mess too and fall short of the same standards, just in very different ways from LLMs.
Indeed, except for in the rare cases we painstakingly trace externalised logic we have zero evidence that humans verbalised explanations of our reasoning matches our internal states either, and plenty of evidence via Sperry's split brain experiments that we're prone to outright making up rationalisations for our reasoning.
There's nothing special about 'natural' intelligence as opposed to 'artificial' intelligence, such that we need to concern ourselves with anthropomorphizing mattering any longer.
Those days are over. The age of the classical human has already ended, the species just tends to lag in awareness. The only thing that matters going forward is whether an output makes sense, is it what it should be. Do answers make sense given the context. It doesn't matter if it comes from natural or artificial intelligence.
What I mean is, artificial intelligence is as valid as human intelligence. There's nothing particularly important or special about human feelings or thoughts or memories.
The average human is drastically less important, interesting, intelligent than the latest frontier AI.
Go spend a few years working in retail, you'll quickly understand how absolutely vile humans are on average. Frankly, the reason we should avoid anthropomorphizing AI, is because it's beneath modern AI to mimic something so crude as a human.
In other news, Pascal's Wager makes no sense whatsoever if an omniscient all-knowing God exists that will see right through your deception. My own take here is stop treating "reasoning" as a sign of sentience or self awareness when your personal computer can do it now. IMO that has much larger implications w/r to our place in the Universe and what we might meet out there someday* than the question of whether your LLM is alive or not.
*Paging Peter Watts and Vernor Vinge
I think it's essentially too late for exhortations like this. The Believers™ and The Skeptics™ are two thoroughly separated tribes by now that speak two different languages. The chances of one influencing the other in any significant way are minute in my estimation.
Im waiting for the article called "stop desantropomorphizing llms" when everybody will finally accept they think like us, partly because maybe the intelligence is universal and partly because, well the datasets are fucking human bro
You’re going to be waiting a long time, considering they don’t think like us. LLMs don’t ‘think’ at all. They are capable of limited reasoning using the meaning and context embedded in human language. Essentially, the grammatical equivalent to a mathematical constraint solver. Nothing more.
LLMs are trained on vast bodies of human-curated text that was made by humans or for humans - often both. Then they're tuned further on human feedback. And then they're pointed at tasks humans find to be useful, evaluated by how good they are at those tasks, and trained to get better at them.
You aren't anthropomorphizing LLMs enough.
Are LLM reasoning traces always faithful? Lmao no. Are human inner monologues always faithful? Lmao no. Both of them reflect thoughts somewhat, sometimes. Even in humans, conscious thought is the top of a vast iceberg of subconscious data processing.
[dead]