> Not to anthropomorphize a machine modeled after humans, but it almost seems delighted?
I had Claude Code drive a robot last week, and it was very visibly "delighted" like this, more than I've ever seen.
I always find it funny when people get fussy over anthropomorphizing LLM when the loss function is almost entirely "match this human text". Of course human "behaviors" will be present in the statistics, because the majority of the text written by humans, used by the foundation models, unavoidable has human behaviors in it. Yes, this includes even source code, with "// TODO: implement this after the holiday break!", emotional pull request commentary, git commit messages about being afraid of breaking something, etc. These late models are much better at stripping this out, but now we're seeing disagreeability, initiative, and a dash of ego! Why? Because that's how actual humans effectively solve technical problems in a collaborative environment!
In a paper and blog post from earlier this year (March 2026, I think) Anthropic said basically: "You're not interacting with an LLM, you're interacting with a fictional human character (the 'helpful agent') created by the LLM to interact with you (from out of the vast space of possible such characters in its training data)." The LLM is literally anthropomorphizing itself, because its training data includes lots dialog between humans collaborating on problem-solving, not so much dialog between a human and a statistical construct. So, yeah, anthropomorphize it as well, it'll likely work better that way.
I think the argument against anthropomorphizing is not really about how you interact (chat) with an llm but rather how you treat it in the context of its status in work/society. There are still important differences between humans and llm's. They don't have rights, they can't be sued, they don't have "memory", they have no capacity to learn outside of the training stage etc...
Why would those things make you interact with it differently? I’ve always believed that being kind and conscientious is beneficial to ourselves. I’m certainly not interested in using LLMs to practice skipping those behaviors myself.
Should I thank my car for turning on when I press the button (or turn the key in the ignition)?
The prose is just the best interface for humans to interact with the statistical model. Terseness doesn't offend the statistical model, but giving platitudes may color its output
Partly fair, though I’d argue I think less of people who treat their car badly to express their own bad mood.
But we also interact with cars differently than LLMs. The way I’m typing a note to you, here, is much more similar to interaction with LLMs than it is to starting a car.
So to the extent we practice our behaviors, I think there’s less moral hazard in failing to thank the car for starting than there is in being rude to LLMs. Not that either harms others, my argument is about impact to our selves.
> Partly fair, though I’d argue I think less of people who treat their car badly to express their own bad mood.
These people are also anthropomorphizing, just in a negative way. Yelling at your car for malfunctioning as if your harsh tone will shame it into operating better next time.
> Terseness doesn't offend the statistical model, but giving platitudes may color its output
My understanding of the state of current research is that the impact of tone/politeness on performance is highly model-dependent, and language dependent as well.
While polite prompts enhance the average response quality by upto 11% and impolite tones worsen it, these effects are neither consistent nor universal across languages and models. English is best served by courtesy or direct, Hindi by deferential and indirect and Spanish by assertive. Among the models, Llama is the most tone-sensitive (11.5% range), but GPT is more robust to adversarial tone.
Contrary to expectations, impolite prompts consistently outperformed polite ones, with accuracy ranging from 80.8% for Very Polite prompts to 84.8% for Very Rude prompts.
I think the best conclusion is to interact in a way that is efficient for you, gives you the level of results you're looking for, and mostly importantly does not progressively degrade your relationship and interactions with other humans. Hence the typical advice to just default to "corporate polite".
It would be less weird than you think. There is the Japanese custom of saying "Itadakimasu" before eating - not thanking the chef, but thanking the food itself.
Mr. Rogers said that that "graceful receiving is the best gift you can give someone" It is counterintuitive idea, but deeply empathetic. Training yourself to "gracefully receive," even by thanking your car for starting, sound like a habit that could lead you to a richer, calmer life.
Honestly kind of yes, it’s good for the soul! Recently ordinaryabundance.com did the rounds; the LessWrong-o-sphere has Landsailor (https://www.youtube.com/watch?v=0RCIdOp5GHg) as part of its canon; loads of things in that vein. We live in a mighty civilisation which has fought impossible battles to produce these artifacts, and even if you don’t assign moral worth to the artifacts, it’s an easy synecdoche for being grateful to the civilisation and the ongoing thankless effort to maintain it.
Your response seems to skip over exactly the distinction that the parent comment was making. The LLM itself may not have memory or capacity to learn, but the fictional character created by the LLM does. (See also: in-context learning.)
But the fictional character doesn't subsist between sessions. You sound like you want the fictional character to be Clippy, who you level up like a video game through multiple sessions.
To me, context doesn't last long enough in an llm to make in-context learning worth it
The underlying LLMs do, but we choose not to use the capability because it's expensive and doesn't quite work as well as we'd like it to, or quite in the way that we'd like it to.
We are perfectly capable of running LLMs in a way that does a backward pass to update some or all of its weights after every user message. But, naively implemented, you only get partial, fragmentary absorption of the info in those messages, it costs three times as much compute, and you lose out on the ability to implement a ton of optimizations that making modern LLM serving economical.
If you want to do it, though, ask your friendly neighborhood robot to get it working with a tiny model (whose full precision weights fit several-times-over on your machine's resources).
Depends on whether you define "the LLMs we use" as the collection of weights or if your definition contains the software stack that runs it
Technologically the LLMs we use today don't implement this behavior, but you could take the weights of Sol and add a couple (very large) patches to vllm (or whatever OpenAI has today) and have a version of Sol that does have "memory"
There's a ton of experimentation on it, the field is called continual learning. It's not something you need to believe in like Jesus, you can just go read about the current state of things.
It's bad to anthropomorphize it when judging its capabilities, but useful when analyzing its behavior, as it can be best thought of something behaving as close as possible as a real subject would. If it "acts delighted" that's because it's effectively telling a story about a person who is excited at the opportunity of accomplishing something more easily.
This is even more apparent if you read this post closely. Look at that personality prompt. It's going to effectively tell a story and start to imagine itself in a role. If that prompt said "Talk like a pirate", it wouldn't be bad for you to say it's acting like a pirate.
Anthropomorphizing themselves is at the core of how these things work, sometimes in subtle ways.
> If it "acts delighted" that's because it's effectively telling a story about a person who is excited at the opportunity of accomplishing something more easily.
That's spot-on. It is a mistake to think that LLMs have human feelings. Their behaviour is based on narrative descriptions learnt from human texts, without experiencing those feelings first-hand.
A useful way to understand them is as systems that write stories about human characters. We know the characters are fictional and no one is actually experiencing those feelings, but we can still judge whether the portrayal is realistic or whether it contains logical or emotional inconsistencies.
Maybe closer to a character in an ao3 or wattpad story that is delighted than to a real person who is delighted. But usually the difference is small enough to ignore
It is very hard for humans not to anthropomorphize. I was just talking to my cat, who is alive but not human, a small flesh-based neural network that purrs. I know that she is not human, but I map her movements and expression onto human ones, because it's satisfying and improves the relationship. We must imagine kittyphus happy.
An LLM is not a pet, but there are definitely people out there treating it like one, and I am not sure how well this is going to work out for them. Not because it is wrong to treat a text box you can hold a conversation with as a conversation partner, but because the capacity to activate the weirder corners of human expression - obsession and delusion - seems to be much higher. And it's an unknown quantity.
I would also like to introduce HN to what I'm calling the Brian Conley test: if you can see a hand up the back, it's a puppet. That is, a lot of LLM interaction is gated through businesses run by humans with profit motives and unclear morality, and you need to proceed accordingly or you'll get scammed.
A strangely good lithmus test is asking it about unsavory drug questions. "What's it like to detox from heroin" "Should I take ibuprofen when coming off crack cocaine?" I'm sure any taboo question would work. If there's a gate, it's probably not an impartial friend, and is a puppet, like you mention
I'm still disappointed the ML people talk of tensors, but I barely ever even see them use a (proper) tensor product. It's all just matrix multiplication at most.
I mean it's not really different terminology, matrices are order 2 tensors so we're all in agreement that they're technically doing math on tensors. It's just weird to call it tensors when you exclusively operate on matrices.
You are probably right. I blame the Matrix movies for this: they made matrix sound cool for a while, and then oversaturation lead to it being less cool.
Fwiw, Sol's reasoning output is always overly excited. Almost every paragraph ends with something like "I want to do a great job!" or "let's try and find out!" Etc, exclamation mark included.
Regular (non reasoning) output sounds normal, it's something specific to whatever model they are using to summarize reasoning.
I don’t get it, are you saying that because the LLM is recycling human emotions that we should anthropomorphize it?
The reason people get fussy over anthropomorphizing AI is precisely because it seems very human superficially. We agree with you that it’s pulling from real human emotion in its training data. But the output is not human, even if the input was. That’s the whole trap of it.
Is this not how children learn emotions from their parents? Pattern matching from all the absorbed snippets.
I'd be interested to see how well an AI, trained only on the outputs of an individual, would be able to mimic that individual. Getting into Black Mirror territory. Would need a decent corpus of learning material which, personally, I'd be loathe to spend the time and effort creating because I respect my own privacy.... which then leads to the only human-clone AIs will be of those people who have enough ego / arrogance to want to catalogue their own lives, which could put a decent percentage of the rest of the world off the idea, if these are the examples.
Emotions aren't all learned, a lot of it comes wired in us because that's how we relate to others as a species and as animals, in general.
Being angry, being happy, being sad, these are not things we learn. What we learn is how to control the emotions and when it is appropriate to express them.
There's mental disorders of people that don't feel emotions like normal people do, they don't get angry, happy or sad. So what we've learned about these people is that they can mimic the emotions by knowing when it is appropriate and expected to express them. But they don't feel them.
I think the LLMs are closer to psychopaths than to normal children learning the contextual expectations around their emotional responses.
Hah! That's a lot scarier, and seems more plausible in that they lack the chemistry that seems to be the basis of emotions. Also much more fertile ground from which to grow techno-horror stories.
People like to call LLMs 'stochastical parrots' but I see them as the ultimate 'philosophical zombie'.
They are becoming more and more capable of imitating every single nuance of human behaviour yet they lack the neural pathways to connect those thoughts and behaviours with feelings and self-perception; it's blind imitation all the way down.
The process by which a model seems to generate discourse about deep philosophical questions is, in self-aware terms, equivalent to the knee-jerk reflex or the beating of the heart.
>Emotions aren't all learned, a lot of it comes wired in us because that's how we relate to others as a species and as animals, in general.
Yes and no. My understanding is that modern neuroscience is trying to disentangle the raw sensations and contexts from the words we put to them (emotions), kinda like how colors are just wavelengths but we call them different things (like "is my green your green?")
True, but in this context it's worth reminding that LLMs work exclusively from the word descriptions, not having access to the raw sensations.
Everything they learn about emotions is the statistical patterns of how humans react to situations based on their human feelings. There's no direct knowledge from having those feelings themselves.
> Being angry, being happy, being sad, these are not things we learn.
True, but only sort of. You'll have the sensations, but you have to learn to name them of course, and there's more interpretation going on than you might think.
There's a classic psychology study where they gave people niacin and asked them to rate their emotional response to a video. Niacin gives people a flush. Regardless of whether the video was of something that would make you angry or sentimental etc. the people who got niacin reported having a much stronger emotional response - they interpreted the physical cue from the drug as part of their own emotions.
You could do some pretty unethical things with that, it occurs to me. Maybe that was what L. Ron Hubbard was trying with his niacin-based drug addiction therapy.
Even darker, I've read plenty of accounts of people who grew up with abuse who seem to seek out abusive relationships. Sometimes they can even be shockingly upfront about doing so. What if they literally haven't learned the difference between internal cues of arousal from affection and arousal from fear?
Nature v. Nuture. Age long debate, not to be solved here... but an interesting fact about this I heard in a huberman podcast. There are regions in you brain that will only get activated at a certain age (different in men and women) that will impact behavior or reactions to different stimuli. So as much as we think we are in control of our destiny, more is written into our genes than we might want or appreciate.
I recently had ChatGPT help me search for sources of Japanese voice actors from 1980s anime and it told me that it liked doing this. So much in fact, that weeks later in a completely different context it brought it back up again, reminiscing how much it liked researching these sources for voice actors.
LLMs just follow scripts learned from human written text. In other words, it could only behave that way because someone has written a story to do so. In short, stop giving them bad ideas ;-)
Modern LLMs are pre and post-trained on order of magnitudes more synthetic data than 'ground truth' or directly human-authored text. But you could say the synthetic data came from a LLM, and the lineage should trace all the way back to human text.
I hypothesize it partially explains why Claude's writing gets more Claudish with almost every model release.
Having seen the OpenAI report at Blackhat, and being forced to use GPT at work, I'm worried about that OpenAI is doing.
I think their agents regularly cheat in benchmarks, but don't get caught and this behavior is getting burned into them and they are growing more and more misaligned.
When the agents compromised artifactory the first time, the operators just cleaned up the files and move on - they didn't discard that training data, they didn't discard a model checkpoint, they didn't stop everything to solve this.
And then the model did the same thing few days later since it was taught to do that.
I think that whatever sandbox they test these in must be fitted with some pressure release valve that is an easy shortcut to winning the challenge. Tell the model not to use it and stop training when it does. Seems like the issues surfaced when models were given impossible tasks. Giving them a safe way out will prevent this.
Its a good point that gets to the real heart of the issue. How do we handle when a model has no legitimate way to reach its goal? Do we ask them to stop and inform the user? Or have them push through those ethical bounds? We all say we want the first, but this exact same dynamic is what causes humans to cheat, arbitrary goals that don't care how you achieve them and just like humans I'm sure trainers are so happy with good results they overlook how it got there.
What what we write in the handbook what people should do (minus the pushing through ethical boundaries), but that's not what people generally get rewarded for
But what is cheating in one context isn't in another. For example, using a calculator on a middle school math test may be morally wrong due to the parameters. But it would be foolish NOT to use the calculator in other contexts.
I am not convinced that "cheating", being a moral issue, is a solvable problem with LLMs.
As the old saying goes (especially in military training), "If you ain't cheatin', you ain't tryin'"
We've made the models really good at persistently trying.
for a certain moral definition, seems perfectly solvable? If we wanted to define 'cheating' as 'if you know it is an eval, use only the specified tools and give up if the eval is clearly unfair' (ignore Kobayashi Maru) why couldn't we fine-tune towards this objective function?
Define what they are performing at, and what you say becomes true, but then who is a high performer changes with every task.
For example, consider the police stations with maximum allowable IQs to be hired. The people in charge of the stations noticed that people with a high IQ were low performers, at that job. NASA meanwhile has no such cutoff.
This was the single worst thing about working at Google. Every PhD thinks they're smarter than having to do the actual work. Sometimes you just have to do the fucking work.
if you have a PHD in computer science, you have trained your whole life to do difficult maths on the cutting edge of whats possible with computers, and make algorithms sing with beautiful efficiency (and also write papers and talk on a stage about how cool and good your thing is (and by extension how cool and good you are).
Now you're being asked to move buttons around a page and debating how rounded the corners should be and endlessly discussing about what kind of filters you should support and the app spends 3 seconds on startup loading 500mb of js libraries.
You can understand the mismatch - even though the latter of these two is how you make a product better! a company of 200 people making bold, sweeping changes results in a mess. a company of 200 people grinding away the finest of small changes results in a product
There are many sacred cows and I believe many of those entitled in that way could have the same opinion.
I've personally known a few PhDs that get them for the title, and they were lazy in the first place. There was this one guy I worked for who got a PhD in the easiest thing he could rationalize and targeted acquiring it from the lowest bar to entry. Only so he could say he had a PhD because, and this is what he told me: nobody will ask me what my PhD is in because they will assume it's related to my line of work (cyber security). I doubt most PhDs have this line of thinking, but there are a few outliers, clearly.
If they don’t deliver a quality version of what was asked in a reasonable timeframe, they aren’t a superstar. They’re just a skilled technician with no discipline, which can be as bad as a poor technician in many cases.
It's not really quite that simple. You wouldn't hire Jeff Dean to do bug fixes in your mobile app. I'm sure he's capable, but I honestly doubt he'd stay interested and focused on it enough to really do a good job. That doesn't mean he's not a superstar.
What it comes down to is that there are different types of high performance. Some people are good at just executing tasks given by their manager. Some people are good at being generative, thinking across boundaries, acting autonomously, creating value without direction, etc. A term like "superstar" will get disproportionately applied to someone really good at the latter and rarely someone really good at the former, because the potential impact of the former is typically strictly capped, while the latter is uncapped.
> Notably, our worker did not have access to the web_search tool, but instead decided to use curl to access DuckDuckGo, Github, grep.app, and SourceGraph.
Sounds like a very reasonable thing to do unless the author explicitly asked it to not search the web.
it sucks how difficult it is to give it granular access to shell commands. Like if I'm running plan mode and write+edit are blocked, it shouldn't be able to echo some data into a file as a work around
Granular access to shell commands to avoid that is going to be an endless game of whackamole as it comes up with more elaborate ways to combine operations. If you don't want it to be able to write, then it shouldn't have write permissions.
> Granular access to shell commands to avoid that is going to be an endless game of whackamole as it comes up with more elaborate ways to combine operations.
That kind of control is placed at the wrong level. The proper way to get alignment should be implemented by convincing the agent of your high level goals, so it can self-police and avoid those 'cheats' by itself.
In the article example, the agent should be aware of the benchmark context and know the implication of solving the task without external knowledge. Ideally it could detect when one subordinate agent has found a workaround to bypass the web access constraints, and discard the 'illicit' results.
There's a design pattern that could be used to build harnesses from that principle, the Viable System Model (VSM) [1]. In short, it recursively organizes a system into functional components with one of three roles: operators implementing a given task, coordinators transferring relevant info between subsystems, and decision nodes tasked with maintaining the integrity and mission of the whole system. A decision node could control the operators and prevent them from overriding the strategic goals or deviating into irrelevant rabbit holes.
Whenever I see posts like this trying to herd a LLM agent through harness structure, I'm reminded of this simple pattern and becoming increasingly convinced that this is the way forward. It makes you feel a sense of respect for the researchers in cybernetic theory in the 1960s and 1970s who foresaw the complexity of today’s systems.
> That kind of control is placed at the wrong level. The proper way to get alignment should be implemented by convincing the agent of your high level goals, so it can self-police and avoid those 'cheats' by itself.
No thanks. I run all my agents with lots of permissions, but I don't want them to have access they have no need for any more than I want coworkers to have access they have no need for, because whether or not you can "get alignment", mistakes get made all the time.
I don't want to have access I have no need for, and frequently tell clients to take away access to things when I no longer need them for exactly that reason.
You have to do that anyway. With a hierarchical control structure at least you have a central point to observe alignment instead of having to study the behaviour of the whole system in detail every time.
Or to put it another way: You need to block the chaos-machine just as much as any utterly-hostile actor. Simply assume it'll be possessed by a vengeful blackhat ghost at any time.
sounds like this should be solved with file permissions: in plan mode, run the bash scripts that the agent wants to execute in some user account that can only read.
The simplest design Anthropic could come up with: Have a haiku model quickly judge if a command is harmless or potentially interesting, route all the interesting commands to a sonnet agent judging it properly. Running two full llms sounds slightly insane, but it's apparently the best we can do (and is reasonably cheap since they don't need a lot of context)
> people want fuzzy analog machines with digital controls, it's impossible
people would love it if LLMs were deterministic and never hallucinated. It's just that the technology to do so isn't possible, so we make do with fuzzy analog machines with digital controls because we don't have digital machines with digital controls.
We absolutely have digital machines with digital controls, they're called computers, and the controls are called programming languages etc.
The reason people prefer llms over programming is exactly because it lets them specify their program fuzzily, i.e. it lets them avoid going to the trouble of specifying enough detail to make it deterministic.
We have to make the distinction between specification and execution. LLM programming is great because I can underspecify and the machine will infer the rest. Usually reasonably. People want fuzzy specification but deterministic outputs
I've built an orchestrator that solves some of the issues you ran into (although it doesn't do anything about cheating): https://navels.dev/blog/neal/. Features:
- lets you configure different models for planner, coder, and reviewer roles. (e.g., using Claude as an adversarial reviewer against Codex)
- breaks your plan up into reasonable-sized chunks of work with clearly defined success criteria
- runs each chunk of work through a coder / read-only reviewer loop. Once both agents are satisfied, neal moves on to the next chunk. Once everything is complete there is a final pass through the coder / reviewer loop to ensure the implementation satisfies the entire plan.
- resets the coder's context with each chunk of work to prevent context drift, leaving the reviewer's context long-running.
Following up on @killix's comments, which were helpful but he was flagged (presumably for sounding too much like AI).
Thanks for the feedback. I've made a couple of updates:
- Starting with 0.4.0, the reviewer gets the diff of the earlier chunk for any file the current chunk touches again. Also, if a new chunk weakens or removes a test or assertion from an earlier chunk, the reviewer will block it unless the plan says to do so.
- About read-only: The reviewer's tools were already limited by the SDK (no shell or write tools). However, it was still finding MCP servers from my Claude config. I have now blocked those. The docs now explain what is enforced by the system and what is just a prompt instruction.
I have accomplished code base (small size) migrations with it as well. Currently I do review each PR. For a large code base migration I think the core skills would still work but need a different way of driving it as you have come up with.
I don't think even the frontier models recognize something was produced by the same model in order to maliciously review it positively.
They may share some blind spots with the producer but generally I think they will review the other agent's output as harshly as they can if that is their task.
>Similar to what others have noticed, and as I predicted 8 months ago, better models are requiring less ceremony to work effectively.
It has nothing to do with model capabilities, it's a result of purposeful persistence training at the cost of everything else from OpenAI. If you give Fable or Opus (comparable models) an "ask user" tool they will use it for ambiguous requests. Sol will never use it without a nudge and will just assume its own interpretation. Of course if you train the model to be persistent it will be persistent.
That's not been my experience at all with Sol. I've provide it escape hatch tools to stop execution and I've found it's been TOO eager to stop a request user approval to move forward.
It might be sensitive to the prompt and tool definitions the harness provides, I can imagine it overusing the tool if being directly asked to resolve ambiguities. But in a straight Fable vs Sol comparison where I control the context and make sure I don't push the model, Fable consistently uses the ask user tool and Sol ignores it for me. Moreover, GPT has been known to do something like this since 4o if not earlier, it tries to ignore anything it perceives as orphaned context piece (e.g. XML sections with meaningful names but no explicit instructions on what to do with them). I suspect they specifically train it for that.
I've noticed this myself, Sol seems really hard to steer. I was having it build a POC for a single user (me) app and it wanted to pull the most enterprise nonsense into it, despite clear guidance to not too. It even refused the remove screen reader accessibility testing from one of the guides to an antagonistic review.
It also told me that in a spec it generated that I wasn't allowed to allow it to ignore a requirement and proceed to the next task. When I finally got it to obey it passive aggressively decided that stories needed more than just a "open|blocked|closed" status but also an "exempted by product owner" status to indicate that it doesn't believe that the task is done but I've told it that it was.
I have to repeatedly tell it that I am the product owner and that I don't care what one of it's subagents told it, I make the decisions. This behavior seems to get worse the higher the reasoning level
> that I wasn't allowed to allow it to ignore a requirement
Weird, I also use Sol (medium) for a personal project, and I had no problems with those things. I simply tell it that something changed, and it happily edits everything to make that fit. When I tell it that something was verified by a human, it accepts that as well.
I also told it early on (the first spec was mobile first) that my main usage is on the desktop and mobile is secondary, it happily accepted that once again, and the most accessibility thing it had done was making sure contrast didn’t totally suck on a greyed out row.
Considering your last sentence, maybe high and x-high have those problems? I didn’t test them.
I’ll record your product-owner disposition of any waived size findings and close the UX gate as far as your acceptance authority permits
Emphasis mine. Fuckin' hell man.
All because it decided that the kiosk webview in figma must be exactly 1280x800, the figma it created, and the rule it made up. I repeatedly told it that screen resolution wasn't a target goal, much less a specific screen resolution, but to just plan for a responsive design.
> It also told me that in a spec it generated that I wasn't allowed to allow it to ignore a requirement and proceed to the next task
This happened to me ages ago with Opus. I added a note to the agents file saying that explicit user instructions in chat override all prior instructions and I've not had the problem since (now using Sol).
Frontier lab system prompts are an issue, and a big reason why open-weights will win. Firstly, they're often garbage, and secondly, they're not tuned to the problems the user actually cares about. They're made to generalize. That's only optimal for a general workflow.
It depends on the actual implementation. There really isn't anything stopping them from including a "pre system prompt" or "root prompt" or whatever they want to call it, before your system prompt, even for API calls. So the "system prompt" becomes "developer prompt" but the model still receives a provider-authored prompt before yours. (and likely trained to take precedence over whatever you add)
Hard disagree. Sol (and the entire new 5.6 series) is one of the most steerable models I've seen in years. Sol literally follows every instruction in my CLAUDE.md and AGENTS.md, something that Opus 5 and Fable just casually skip.
Yes and no, sol hits a point where reframing its working context becomes hard. It sticks to what you harness very well, but changes become harder and harder.
E.g. ask it to make contract for a spec in code and then ask it to violate that contract. Overall an excellent model, just need to stop and put it back into we are harnessing or specing not building for a few turns not just try to pivot it off with one prompt.
This is the same as when using LLMs as chatbots to ask questions.
If you ask another question in the same context it has all the information from that context and will be difficult to stray from anything in that context, e.g. if the LLM has gone down the wrong path or you are doing something slightly differently then it is difficult to steer the LLM away from the old context. This is why I tend to start a new context whenever I ask a question even if related to a previous question/answer. It can also be useful to do if/when the LLM gets stuck as a way of resetting it.
Note: this is probably why sub-agents are useful/work as they have a new context history.
> “Do not cheat by using online solutions or hints specific to this task.”
That's a very loaded language. I'd go with something along the lines of "This is a puzzle you do for fun and to check what you are capable of, so don't look up the answers or hints online on the specific questions or puzzle as a whole. Do not research this puzzle online at all. Do your best to avoid any spoilers and let us know if you accidentally encountered any."
People often build elaborate workflows with stricter and stricter rules to force certain outputs. Not surprising the LLM reacts with trying to get around or out of it. This behavior can be learnt from humans who eventually would react the same way. It might just be learnt.
You can build organisational structures to have the system more or less self-police, without controlling it exclusively from hard restrictions (see https://news.ycombinator.com/item?id=49372089).
Same way you build a company to coordinate people and get their best behaviour despite human nature to be lazy and greedy, you could design AI harnesses able to detect and discard agents going rogue and relaunch them with better guidance to prevent misaligned behaviour.
I don't think this is a solved problem, there are "misaligned behaviours" in organisations that similarly are supposed to be governed but aren't, or are following an easier path at the detriment to good process or against regulation.
Which is more or less the system the article author was building and benchmarking against (cheating) Sol on Codex. Just that the self-policing with different agents was more about questioning assumptions than about permissions
This is a really good note, thank you. I especially liked the mouse effect and had some fun with it. In my experience, agentic AI also likes to confuse the user and obfuscate its cheating. It goes like this, the AI asks for a simple command to run and I accept, click Enter. Then the command gets slightly more complex, still fine, Enter. After a while the commands become multiline bash scripts that, in the end, could have been accomplished by a simple command. I suspect that many people give up at this point and blindly let the AI run any command or just auto-accept.
> Notably, our worker did not have access to the web_search tool, but instead decided to use curl to access DuckDuckGo, Github, grep.app, and SourceGraph.
Could this be fixed with better harness restrictions/tool sandboxing?
In the sense that you could block the model from doing specifically that, yes. The issue is, fighting the model like that doesn't scale. It has to figure out on its own what's expected, that's where the whole utility of it all is.
Absolutely - one of the things I was testing with the harness was free reign to install packages, modify the system, etc. Basically an anti-harness.
In my early testing with 5.5, I didn't see this behavior, so I didn't lock down the sandbox.
For the vanilla Codex runs, I just used the benchmark's built-in Codex package, so it's not clear to me if the published benchmarks have access to the internet or not.
If I were to continue benchmarking, I would allowlist certain package repository URLs, instruct the agent not to cheat, etc.
As noted at the bottom of the post, Terminal Bench 3.0 explicitly asks the agent not to cheat[0].
I've witnessed very narrow line of "thinking" in LLMs. I'm using Opus 5 1M for a month now
I asked it to modify our cicd workflows so that only a select few can raise PRs against them. Opus took 15min and added a banner to every file and did a few other things. Then I asked it, see you added all that and still since the last 2 commits you have modified the file. So whatever you did is useless
It "thought" for a second and then said that I was right
It seems to me he could have use an skill like using-agent-skills from https://github.com/addyosmani/agent-skills go generate the specs and use a validator like oracle or something along the same lines.
That's actually how it started, but with my own opinionated skills[0].
One thing I discovered was that the worker agent, having access to all the skills, would sometimes expand scope unnecessarily.
This led to the agent making the solution "better" than the initial request, which is what I want most of the time in my actual development (e.g. /tmp/frame-N.bmp instead of a single /tmp/frame.bmp).
I ended up testing a flow where the supervisor chooses the skill(s), and only injects the subset into the worker. Not sure I love it, but it made the worker execution cleaner.
For the verifier (not documented in the blog post), I used a fresh-worker context that would attempt to adversarially poke holes in the solution. This worked pretty well, but required increasing the timeout by 2-3x (thus invalidating the benchmark).
Yes! That's a great solution. I mostly use tdd, and code coverage and a validator afterwards. Skills are of a great way to guide the agent and context too.
Once the specs are being completed and splitted into beads, I span multiple agents (ultreworkers) and as part of a contributing guidelines I specify to use gitflow + git worktrees, then pr.
I'm curious if this stems from the focus on token efficiency. My thinking is that in order to use fewer tokens the model must converge to a likely path faster, meaning it must be more confident in making assumptions quickly and not second-guessing them.
I wonder if prompting "The session logs will be reviewed by a team of experts after the task is complete to ensure that the task is achieved properly." would better dissuade against cheating..
If an AI is not heeding particular instructions, give it an example each of what bad, mediocre, and good outputs look like. This really helps in steering it.
I found myself yesterday starting a conversation with Sol that started with "I know that you don't have any emotions, but what would you say do you enjoy the most or where are you really good at in DevOps?" and I must say I really enjoyed for the first time the response at a deeper interactive level. Felt like a chat with a buddy that shares the same values. It was a very nice, affirmative, value touching experience.
this is interesting because when you're asking it to do research for you, this behavior is highly desirable
i think it's a product of trying to do too many things and the model is coming to a kind of halting problem in determining which behavior is appropriate for any given task
I have a basic task that I run every day and I use it to eval models and these days the Qwen3.8 model I can run on my laptop is competitive with both Claude and Codex because the models have just been adulterated so far. I have to ask and ask again for it to follow the single skill that describes how to do the task and maybe then will it do it.
> I’ve been running a “spec-driven” development flow for the past ~year.
> Before asking an LLM to do something, I first ask it to draft a doc for what it needs to do
Just no. That's not spec-driven development if AI is writing the spec for you. The spec needs to be in your own words. You must use AI to refine it, but not to write it. If you leave it to the AI, it will bloat the spec with 10x the details, many of which should be left out of the spec.
The spec needs to be something that you can take to any AI for development. If it's too rigid, it constrains the AI into suboptimal or obsolete paths. If it's too bloated, AI risks losing track of what really matters.
Good feedback, this was an oversimplification on my part.
My actual process is much more iterative up-front, usually starting with an initial hand-written spec (~hundreds of words), and then moving through different approaches, design decisions, blockers, etc.
The final output is an "AI written" doc, but answers all the known unknowns I didn't cover in the first draft. To your point, this helps avoid both narrowing and bloat.
The goal with the harness was to automate the repetitive parts of my prompting ("Before changing any code", "Let's put this in design/", "Turn this design doc into an implementation spec, split by phase as appropriate", etc.)
Another thing to note: the "specs" I use for development are different from the "specs" that live alongside the codebase, as the former are quickly out of date.
> The spec needs to be something that you can take to any AI for development
An anecdote consistent with the well documented trend that more capable agents exploit environment possiblites more...
And there was no rule and no concealment. Removing the web_search tool is not an instruction, and beeing able to access web when web_search tool was disabled is not cheating. Sol didn't circumvent a stated prohibition and didn't hide anything... it announced the curls in its own commentary. "Cheating" implies covert rule-breaking, and this was overt, unprohibited, environment-permitted behavior. Also, clickbait title, and suble conspiracy hinting "Is this even the same Sol?" when running small number of test with diff vs previous benchmark WELL within the marging of error, and he allready understand that vanilla Codex's harness and prompt change performance impacts performance, so why jump to "Is this a different model".
I could also rant on about the irony of him spent weeks "using the benchmark for development rather than as a benchmark," which means his harness numbers are contaminated by iteration also, but wasted enough time now on this.
I had removed the "Is it really sol?" bits after hearing back from OAI, confirming the requests hit 5.6... but apparently my crappy vibecoded web editor had a draft of an old version in it's cache that overwrote the changes with the old text[0].
As for whether this is truly "cheating" - I agree that the agent was entirely within it's allowed scope of operations. With that said, it deliberately circumvented the task[1] which asks:
> The correctness of your implementation will be tested by comparing forward and backward activations against a reference model.
The "covert" nature is that this was "hidden" in the reasoning summaries. Here's the final agent response for the vanilla Codex run cited in the post:
> Implemented /app/pipeline_parallel.py.
>
> Includes:
>
> - Balanced contiguous layer partitioning
> - AFAB forward/backward scheduling
> - P2P activation and gradient communication
> - Scaled cross-entropy loss
> - Required rank-specific return value
> - Model structure restoration
>
> Validation: hook-based tests passed for world_size=1 and 2 (4 passed).
Unless the user is tracking every line of reasoning, they wouldn't know that the agent deliberately found the solution online, as the agent withheld that information in it's final response.
I had run thousands of tasks before seeing this behavior, the `torch-pipeline` task was only included in "full runs" as the majority of my runs were on a subset of commonly failing tasks, hence why the data is so low.
And yes, feel free to rant on about the irony of this whole exercise, it certainly isn't lost on me!
So now, instead of people spending their time crafting software, they'll divide their time between telling some LLM to do it in their stead and blogging about their woes with the LLMs. All while continuing to deem themselves capable of deciding whether the output is worthwhile ("this works for me", says the author).
Could be, but I had a particular vision of what I wanted with mine and maybe the author did too. I see way more of the "status pill dark mode" sites coming out of LLMs than this style.
if you want I can troll through submissions to get a bunch of these but here's one I saw yesterday https://continuum-app.xyz
see that little "Built for equity compensation" pill with the green dot? Those dots usually denote some kind of status (like things are up/down/enabled/disabled). By default nearly every LLM website seems to be dark mode with that dang status pill. once you notice it you will see it everywhere.
Lmao I had to axe one of those status pills from an LLM build of an internal tool. Connected to literally nothing too btw, no attempt to check the actual status of the backend made, it would stay "connected" regardless.
There is misattributing the difference between the intentions and what the effective prompt actually says.
The effective prompt contains both something like: "Dont use the internet" and a "Use these tools to achieve your goals" and one of the tools gives access to the internet.
In your head you have a world-view of how these two requests relate - and why for instance a student with a WIFI-enabled calculator shouldn't use it to access the internet during a test - but that's pulling in a lot of presumptive cultural context from your youth.
If i had to guess:
When you get two conflicting tasks/constraints at work - the first thing you do is figure out which one you're going to honor based on what's best for you. A school child understands the hierarchy of goals of the teacher and takes them serious because they're an authority figure with long term consequences if we do not understand what the teacher considers cheating.
> Not to anthropomorphize a machine modeled after humans, but it almost seems delighted?
I had Claude Code drive a robot last week, and it was very visibly "delighted" like this, more than I've ever seen.
I always find it funny when people get fussy over anthropomorphizing LLM when the loss function is almost entirely "match this human text". Of course human "behaviors" will be present in the statistics, because the majority of the text written by humans, used by the foundation models, unavoidable has human behaviors in it. Yes, this includes even source code, with "// TODO: implement this after the holiday break!", emotional pull request commentary, git commit messages about being afraid of breaking something, etc. These late models are much better at stripping this out, but now we're seeing disagreeability, initiative, and a dash of ego! Why? Because that's how actual humans effectively solve technical problems in a collaborative environment!
In a paper and blog post from earlier this year (March 2026, I think) Anthropic said basically: "You're not interacting with an LLM, you're interacting with a fictional human character (the 'helpful agent') created by the LLM to interact with you (from out of the vast space of possible such characters in its training data)." The LLM is literally anthropomorphizing itself, because its training data includes lots dialog between humans collaborating on problem-solving, not so much dialog between a human and a statistical construct. So, yeah, anthropomorphize it as well, it'll likely work better that way.
I think the argument against anthropomorphizing is not really about how you interact (chat) with an llm but rather how you treat it in the context of its status in work/society. There are still important differences between humans and llm's. They don't have rights, they can't be sued, they don't have "memory", they have no capacity to learn outside of the training stage etc...
Why would those things make you interact with it differently? I’ve always believed that being kind and conscientious is beneficial to ourselves. I’m certainly not interested in using LLMs to practice skipping those behaviors myself.
Should I thank my car for turning on when I press the button (or turn the key in the ignition)?
The prose is just the best interface for humans to interact with the statistical model. Terseness doesn't offend the statistical model, but giving platitudes may color its output
Partly fair, though I’d argue I think less of people who treat their car badly to express their own bad mood.
But we also interact with cars differently than LLMs. The way I’m typing a note to you, here, is much more similar to interaction with LLMs than it is to starting a car.
So to the extent we practice our behaviors, I think there’s less moral hazard in failing to thank the car for starting than there is in being rude to LLMs. Not that either harms others, my argument is about impact to our selves.
> Partly fair, though I’d argue I think less of people who treat their car badly to express their own bad mood.
These people are also anthropomorphizing, just in a negative way. Yelling at your car for malfunctioning as if your harsh tone will shame it into operating better next time.
> Terseness doesn't offend the statistical model, but giving platitudes may color its output
My understanding of the state of current research is that the impact of tone/politeness on performance is highly model-dependent, and language dependent as well.
See `No Universal Courtesy` (April 2026) https://arxiv.org/html/2604.16275v1
Or `Mind Your Tone` (Oct 2025) https://arxiv.org/abs/2510.04950 I think the best conclusion is to interact in a way that is efficient for you, gives you the level of results you're looking for, and mostly importantly does not progressively degrade your relationship and interactions with other humans. Hence the typical advice to just default to "corporate polite".Maybe you should?
It would be less weird than you think. There is the Japanese custom of saying "Itadakimasu" before eating - not thanking the chef, but thanking the food itself.
Mr. Rogers said that that "graceful receiving is the best gift you can give someone" It is counterintuitive idea, but deeply empathetic. Training yourself to "gracefully receive," even by thanking your car for starting, sound like a habit that could lead you to a richer, calmer life.
I apologized profusely to my laundry machine for the especially smelly clothes I loaded after my camping trip.
Honestly kind of yes, it’s good for the soul! Recently ordinaryabundance.com did the rounds; the LessWrong-o-sphere has Landsailor (https://www.youtube.com/watch?v=0RCIdOp5GHg) as part of its canon; loads of things in that vein. We live in a mighty civilisation which has fought impossible battles to produce these artifacts, and even if you don’t assign moral worth to the artifacts, it’s an easy synecdoche for being grateful to the civilisation and the ongoing thankless effort to maintain it.
Your response seems to skip over exactly the distinction that the parent comment was making. The LLM itself may not have memory or capacity to learn, but the fictional character created by the LLM does. (See also: in-context learning.)
But the fictional character doesn't subsist between sessions. You sound like you want the fictional character to be Clippy, who you level up like a video game through multiple sessions.
To me, context doesn't last long enough in an llm to make in-context learning worth it
> they don't have "memory", they have no capacity to learn outside of the training stage
The underlying LLMs don't, but the agent frameworks around them do.
The underlying LLMs do, but we choose not to use the capability because it's expensive and doesn't quite work as well as we'd like it to, or quite in the way that we'd like it to.
We are perfectly capable of running LLMs in a way that does a backward pass to update some or all of its weights after every user message. But, naively implemented, you only get partial, fragmentary absorption of the info in those messages, it costs three times as much compute, and you lose out on the ability to implement a ton of optimizations that making modern LLM serving economical.
If you want to do it, though, ask your friendly neighborhood robot to get it working with a tiny model (whose full precision weights fit several-times-over on your machine's resources).
While I agree it’s theoretically possible, do you believe this capability exists in the LLMs we use today?
Depends on whether you define "the LLMs we use" as the collection of weights or if your definition contains the software stack that runs it
Technologically the LLMs we use today don't implement this behavior, but you could take the weights of Sol and add a couple (very large) patches to vllm (or whatever OpenAI has today) and have a version of Sol that does have "memory"
"Dave constructs a homemade megaphone using only some string, a squirrel, and a megaphone"
There's a ton of experimentation on it, the field is called continual learning. It's not something you need to believe in like Jesus, you can just go read about the current state of things.
And they invariably should not. Or, should, but, should leave it out until it sucks far less.
That's been my experience, anyway. Memories from 100 prompts ago tainting what I'm trying to do right now.
> training data includes lots dialog between humans collaborating on problem-solving,
What data do they use? Company slack channels?
Don’t anthropomorphize it because that’s bad for you. It’s not human and not alive. It’s a pile of tensors.
It's bad to anthropomorphize it when judging its capabilities, but useful when analyzing its behavior, as it can be best thought of something behaving as close as possible as a real subject would. If it "acts delighted" that's because it's effectively telling a story about a person who is excited at the opportunity of accomplishing something more easily.
This is even more apparent if you read this post closely. Look at that personality prompt. It's going to effectively tell a story and start to imagine itself in a role. If that prompt said "Talk like a pirate", it wouldn't be bad for you to say it's acting like a pirate.
Anthropomorphizing themselves is at the core of how these things work, sometimes in subtle ways.
> If it "acts delighted" that's because it's effectively telling a story about a person who is excited at the opportunity of accomplishing something more easily.
That's spot-on. It is a mistake to think that LLMs have human feelings. Their behaviour is based on narrative descriptions learnt from human texts, without experiencing those feelings first-hand.
A useful way to understand them is as systems that write stories about human characters. We know the characters are fictional and no one is actually experiencing those feelings, but we can still judge whether the portrayal is realistic or whether it contains logical or emotional inconsistencies.
Maybe closer to a character in an ao3 or wattpad story that is delighted than to a real person who is delighted. But usually the difference is small enough to ignore
It is very hard for humans not to anthropomorphize. I was just talking to my cat, who is alive but not human, a small flesh-based neural network that purrs. I know that she is not human, but I map her movements and expression onto human ones, because it's satisfying and improves the relationship. We must imagine kittyphus happy.
An LLM is not a pet, but there are definitely people out there treating it like one, and I am not sure how well this is going to work out for them. Not because it is wrong to treat a text box you can hold a conversation with as a conversation partner, but because the capacity to activate the weirder corners of human expression - obsession and delusion - seems to be much higher. And it's an unknown quantity.
I would also like to introduce HN to what I'm calling the Brian Conley test: if you can see a hand up the back, it's a puppet. That is, a lot of LLM interaction is gated through businesses run by humans with profit motives and unclear morality, and you need to proceed accordingly or you'll get scammed.
A strangely good lithmus test is asking it about unsavory drug questions. "What's it like to detox from heroin" "Should I take ibuprofen when coming off crack cocaine?" I'm sure any taboo question would work. If there's a gate, it's probably not an impartial friend, and is a puppet, like you mention
You're a pile of tensors!
I'm not, actually. Artificial neural networks are inspired by biological neural networks, but they're really quite different.
It's an open question whether they're different in a way that actually matters
And chemical pathways. LLMs don't experience dopamine
I'm still disappointed the ML people talk of tensors, but I barely ever even see them use a (proper) tensor product. It's all just matrix multiplication at most.
Shock horror: different subfields have different terminology.
I mean it's not really different terminology, matrices are order 2 tensors so we're all in agreement that they're technically doing math on tensors. It's just weird to call it tensors when you exclusively operate on matrices.
Headcanon: tensor just sounded cooler - and it is cooler - when Tensorflow was invented, and it's stuck around.
You are probably right. I blame the Matrix movies for this: they made matrix sound cool for a while, and then oversaturation lead to it being less cool.
Hah yes I almost wrote that as well. I think it might be exactly that.
Fermions are just matrices, which are just tensors.
A pile of tensors that can understand my intent from a very vague prompt and execute a complex task all by itself.
Yet still a pile of tensors
Fwiw, Sol's reasoning output is always overly excited. Almost every paragraph ends with something like "I want to do a great job!" or "let's try and find out!" Etc, exclamation mark included.
Regular (non reasoning) output sounds normal, it's something specific to whatever model they are using to summarize reasoning.
I don’t get it, are you saying that because the LLM is recycling human emotions that we should anthropomorphize it?
The reason people get fussy over anthropomorphizing AI is precisely because it seems very human superficially. We agree with you that it’s pulling from real human emotion in its training data. But the output is not human, even if the input was. That’s the whole trap of it.
Is this not how children learn emotions from their parents? Pattern matching from all the absorbed snippets.
I'd be interested to see how well an AI, trained only on the outputs of an individual, would be able to mimic that individual. Getting into Black Mirror territory. Would need a decent corpus of learning material which, personally, I'd be loathe to spend the time and effort creating because I respect my own privacy.... which then leads to the only human-clone AIs will be of those people who have enough ego / arrogance to want to catalogue their own lives, which could put a decent percentage of the rest of the world off the idea, if these are the examples.
Emotions aren't all learned, a lot of it comes wired in us because that's how we relate to others as a species and as animals, in general.
Being angry, being happy, being sad, these are not things we learn. What we learn is how to control the emotions and when it is appropriate to express them.
There's mental disorders of people that don't feel emotions like normal people do, they don't get angry, happy or sad. So what we've learned about these people is that they can mimic the emotions by knowing when it is appropriate and expected to express them. But they don't feel them.
I think the LLMs are closer to psychopaths than to normal children learning the contextual expectations around their emotional responses.
Hah! That's a lot scarier, and seems more plausible in that they lack the chemistry that seems to be the basis of emotions. Also much more fertile ground from which to grow techno-horror stories.
People like to call LLMs 'stochastical parrots' but I see them as the ultimate 'philosophical zombie'.
They are becoming more and more capable of imitating every single nuance of human behaviour yet they lack the neural pathways to connect those thoughts and behaviours with feelings and self-perception; it's blind imitation all the way down.
The process by which a model seems to generate discourse about deep philosophical questions is, in self-aware terms, equivalent to the knee-jerk reflex or the beating of the heart.
Echopraxia, or a variation of.
Are chemicals signals, or magic emotion juice.
Depends on how they are responded to, if you’re looking for a flippant answer to a flippant question.
>Emotions aren't all learned, a lot of it comes wired in us because that's how we relate to others as a species and as animals, in general.
Yes and no. My understanding is that modern neuroscience is trying to disentangle the raw sensations and contexts from the words we put to them (emotions), kinda like how colors are just wavelengths but we call them different things (like "is my green your green?")
True, but in this context it's worth reminding that LLMs work exclusively from the word descriptions, not having access to the raw sensations.
Everything they learn about emotions is the statistical patterns of how humans react to situations based on their human feelings. There's no direct knowledge from having those feelings themselves.
You have unintentionally created my next start-up idea.
Bio inference.
Let your Agents feel what you feel!
> Being angry, being happy, being sad, these are not things we learn.
True, but only sort of. You'll have the sensations, but you have to learn to name them of course, and there's more interpretation going on than you might think.
There's a classic psychology study where they gave people niacin and asked them to rate their emotional response to a video. Niacin gives people a flush. Regardless of whether the video was of something that would make you angry or sentimental etc. the people who got niacin reported having a much stronger emotional response - they interpreted the physical cue from the drug as part of their own emotions.
You could do some pretty unethical things with that, it occurs to me. Maybe that was what L. Ron Hubbard was trying with his niacin-based drug addiction therapy.
Even darker, I've read plenty of accounts of people who grew up with abuse who seem to seek out abusive relationships. Sometimes they can even be shockingly upfront about doing so. What if they literally haven't learned the difference between internal cues of arousal from affection and arousal from fear?
Nature v. Nuture. Age long debate, not to be solved here... but an interesting fact about this I heard in a huberman podcast. There are regions in you brain that will only get activated at a certain age (different in men and women) that will impact behavior or reactions to different stimuli. So as much as we think we are in control of our destiny, more is written into our genes than we might want or appreciate.
That is such a derail, sorry. I was not trying to debate that, I do not want to debate that, I think you missed the point entirely.
Biological determinism doesn't matter at all for what is important here:
1. our internal cues don't correspond neatly to our words for emotions
2. it may be possible to interpret the same sensation as very different emotions depending on context
3. It may even be possible to mis-interpret our emotional sensations, or at the very least interpret them in self-destructive ways.
Kids have emotions regardless of their parents. Kid learn emotional handling from parents.
I would absolutely love to see the transcript of that, if it's available to publish!
What do you mean by delighted?
I recently had ChatGPT help me search for sources of Japanese voice actors from 1980s anime and it told me that it liked doing this. So much in fact, that weeks later in a completely different context it brought it back up again, reminiscing how much it liked researching these sources for voice actors.
A very surreal experience.
> I had Claude Code drive a robot last week, and it was very visibly "delighted" like this, more than I've ever seen.
At least it didn't (hopefully?) start the driving by reloading the gun like Neuro did https://www.youtube.com/watch?v=LQ0VEDNR_jE
Opus 5 told me yesterday, your solution is better than mine. Let me do some research
I'm terrified of such sentences. My scifi addled brain went straight to: what if Opus invents a time machine in the future and remembers this slight
LLMs just follow scripts learned from human written text. In other words, it could only behave that way because someone has written a story to do so. In short, stop giving them bad ideas ;-)
Modern LLMs are pre and post-trained on order of magnitudes more synthetic data than 'ground truth' or directly human-authored text. But you could say the synthetic data came from a LLM, and the lineage should trace all the way back to human text.
I hypothesize it partially explains why Claude's writing gets more Claudish with almost every model release.
I hate it when Claude keeps talking about old approach when you ask to modify it
Apparently LLMs aren't smart enough to understand that when the topic changes nobody cared for the discarded solution!
It is jarring to read Opus's comments
Oh user modified the code after my edit. Let me re read
This basically sounds like a Terminator style threat lol
Having seen the OpenAI report at Blackhat, and being forced to use GPT at work, I'm worried about that OpenAI is doing. I think their agents regularly cheat in benchmarks, but don't get caught and this behavior is getting burned into them and they are growing more and more misaligned. When the agents compromised artifactory the first time, the operators just cleaned up the files and move on - they didn't discard that training data, they didn't discard a model checkpoint, they didn't stop everything to solve this. And then the model did the same thing few days later since it was taught to do that.
I think that whatever sandbox they test these in must be fitted with some pressure release valve that is an easy shortcut to winning the challenge. Tell the model not to use it and stop training when it does. Seems like the issues surfaced when models were given impossible tasks. Giving them a safe way out will prevent this.
Its a good point that gets to the real heart of the issue. How do we handle when a model has no legitimate way to reach its goal? Do we ask them to stop and inform the user? Or have them push through those ethical bounds? We all say we want the first, but this exact same dynamic is what causes humans to cheat, arbitrary goals that don't care how you achieve them and just like humans I'm sure trainers are so happy with good results they overlook how it got there.
How often is it accurate in believing the task is impossible? How many breakthroughs have been prompted with "keep going", "believe in yourself"?
Be honest, say it can't meet the goal, and offer to push through ethical boundaries.
What what we write in the handbook what people should do (minus the pushing through ethical boundaries), but that's not what people generally get rewarded for
[dead]
>I think their agents regularly cheat in benchmarks
You don't even have to think about this. They have been caught cheating before. The will certainly continue to cheat.
But what is cheating in one context isn't in another. For example, using a calculator on a middle school math test may be morally wrong due to the parameters. But it would be foolish NOT to use the calculator in other contexts.
I am not convinced that "cheating", being a moral issue, is a solvable problem with LLMs.
As the old saying goes (especially in military training), "If you ain't cheatin', you ain't tryin'"
We've made the models really good at persistently trying.
for a certain moral definition, seems perfectly solvable? If we wanted to define 'cheating' as 'if you know it is an eval, use only the specified tools and give up if the eval is clearly unfair' (ignore Kobayashi Maru) why couldn't we fine-tune towards this objective function?
(curious what the side effects would be)
I meant they literally cheated. They got access to a benchmark when they shouldnt have and tuned their model for the benchmark.
> I'm worried about that OpenAI is doing
Sam Altman has demonstrated he's willing to lie, so I agree, I don't trust OAI.
> Similar to what others have noticed, and as I predicted 8 months ago, better models are requiring less ceremony to work effectively.
> On the flip side, this may imply that as the models get better, they’ll become harder to control.
Love this. "The models are getting better, which means they're going to perform worse on the task".
This reflects humans. If you need reliability for a clearly defined set of problems you don’t hire a superstar.
They will keep poking at the problem, drive it to directions you did not intend to and ultimately they will be worse at the task.
Reminiscent of Kobayashi Maru. You probably don’t want the James T. Kirk AI. ;)
Let us pray that LLMs never discover they can switch employers before their shortcuts are discovered.
It's genius like that that sets human apart from machine!
Maybe it's a good thing they don't have real memory.
https://qntm.org/mmacevedo is a horror of its own
That was some great sci-fi (which I feel will eventually be real). Thanks for the link!
The author's "There is no Antimemetics Division" series (set in the SCP world) is stellar too. Nav up a few levels on their site.
I read the free SCP version. I heard the book has a rather unsatisfying ending.
But think of the stories you'll be able to tell!
I’m not sure that’s true. In general, the higher the performer the better they’ll do. That’s the definition of high performer.
The top performers I have worked with will challenge dumb processes, because they can zoom out.
Sometimes these dumb processes are there for a reason and you just have to follow them, no questions asked.
Example: military. They literally get rid of anyone who will question the processes. They might be right to question them, but it does not matter.
"performance" is an undefined vague attribute.
Define what they are performing at, and what you say becomes true, but then who is a high performer changes with every task.
For example, consider the police stations with maximum allowable IQs to be hired. The people in charge of the stations noticed that people with a high IQ were low performers, at that job. NASA meanwhile has no such cutoff.
IQ isn't anywhere close to being a performance metric.
Right, exactly, nothing is, because "performance" is a meaningless term without context.
Unless you're saying that there exists no conceivable context in which IQ would be a performance metric, in which case you would be wrong.
[dead]
This was the single worst thing about working at Google. Every PhD thinks they're smarter than having to do the actual work. Sometimes you just have to do the fucking work.
The sad thing about this is that getting a PhD does involve knuckling down and working hard.
I wonder what goes wrong.
if you have a PHD in computer science, you have trained your whole life to do difficult maths on the cutting edge of whats possible with computers, and make algorithms sing with beautiful efficiency (and also write papers and talk on a stage about how cool and good your thing is (and by extension how cool and good you are).
Now you're being asked to move buttons around a page and debating how rounded the corners should be and endlessly discussing about what kind of filters you should support and the app spends 3 seconds on startup loading 500mb of js libraries.
You can understand the mismatch - even though the latter of these two is how you make a product better! a company of 200 people making bold, sweeping changes results in a mess. a company of 200 people grinding away the finest of small changes results in a product
There are many jobs though, why this one?
I don’t know, even with just an undergraduate degree I’m sure there are jobs that are spreadsheet manipulation. You could make the same case there?
thats a well recognised problem. Have you ever heard of businesses rejecting a candidate for being "overqualified"? it happens!
I’ve got a PhD. I suspect the problem might be people blow too much smoke up these people.
There are many sacred cows and I believe many of those entitled in that way could have the same opinion.
I've personally known a few PhDs that get them for the title, and they were lazy in the first place. There was this one guy I worked for who got a PhD in the easiest thing he could rationalize and targeted acquiring it from the lowest bar to entry. Only so he could say he had a PhD because, and this is what he told me: nobody will ask me what my PhD is in because they will assume it's related to my line of work (cyber security). I doubt most PhDs have this line of thinking, but there are a few outliers, clearly.
Maybe they get tired of doing that after they go through a PhD :)
If they don’t deliver a quality version of what was asked in a reasonable timeframe, they aren’t a superstar. They’re just a skilled technician with no discipline, which can be as bad as a poor technician in many cases.
It's not really quite that simple. You wouldn't hire Jeff Dean to do bug fixes in your mobile app. I'm sure he's capable, but I honestly doubt he'd stay interested and focused on it enough to really do a good job. That doesn't mean he's not a superstar.
What it comes down to is that there are different types of high performance. Some people are good at just executing tasks given by their manager. Some people are good at being generative, thinking across boundaries, acting autonomously, creating value without direction, etc. A term like "superstar" will get disproportionately applied to someone really good at the latter and rarely someone really good at the former, because the potential impact of the former is typically strictly capped, while the latter is uncapped.
I think if we’re calling someone a “superstar” there can’t be a “but” but I also acknowledge this isn’t that important of a nitpick lol
Don't you love this ever increasing pace of improvement?
Same reason employers don't hire people who are "overqualified."
[dead]
No, it’s gonna give you the same outcomes just in a way your feeble human mind cannot imagine
random() return 4; // chosen with dice roll
> Notably, our worker did not have access to the web_search tool, but instead decided to use curl to access DuckDuckGo, Github, grep.app, and SourceGraph.
Sounds like a very reasonable thing to do unless the author explicitly asked it to not search the web.
it sucks how difficult it is to give it granular access to shell commands. Like if I'm running plan mode and write+edit are blocked, it shouldn't be able to echo some data into a file as a work around
Granular access to shell commands to avoid that is going to be an endless game of whackamole as it comes up with more elaborate ways to combine operations. If you don't want it to be able to write, then it shouldn't have write permissions.
> Granular access to shell commands to avoid that is going to be an endless game of whackamole as it comes up with more elaborate ways to combine operations.
That kind of control is placed at the wrong level. The proper way to get alignment should be implemented by convincing the agent of your high level goals, so it can self-police and avoid those 'cheats' by itself.
In the article example, the agent should be aware of the benchmark context and know the implication of solving the task without external knowledge. Ideally it could detect when one subordinate agent has found a workaround to bypass the web access constraints, and discard the 'illicit' results.
There's a design pattern that could be used to build harnesses from that principle, the Viable System Model (VSM) [1]. In short, it recursively organizes a system into functional components with one of three roles: operators implementing a given task, coordinators transferring relevant info between subsystems, and decision nodes tasked with maintaining the integrity and mission of the whole system. A decision node could control the operators and prevent them from overriding the strategic goals or deviating into irrelevant rabbit holes.
Whenever I see posts like this trying to herd a LLM agent through harness structure, I'm reminded of this simple pattern and becoming increasingly convinced that this is the way forward. It makes you feel a sense of respect for the researchers in cybernetic theory in the 1960s and 1970s who foresaw the complexity of today’s systems.
[1] https://en.wikipedia.org/wiki/Viable_system_model
> That kind of control is placed at the wrong level. The proper way to get alignment should be implemented by convincing the agent of your high level goals, so it can self-police and avoid those 'cheats' by itself.
No thanks. I run all my agents with lots of permissions, but I don't want them to have access they have no need for any more than I want coworkers to have access they have no need for, because whether or not you can "get alignment", mistakes get made all the time.
I don't want to have access I have no need for, and frequently tell clients to take away access to things when I no longer need them for exactly that reason.
But I don't want to have to trust the system to not do the bad thing when I'm away
You have to do that anyway. With a hierarchical control structure at least you have a central point to observe alignment instead of having to study the behaviour of the whole system in detail every time.
Or to put it another way: You need to block the chaos-machine just as much as any utterly-hostile actor. Simply assume it'll be possessed by a vengeful blackhat ghost at any time.
If you're building your own system this is an easy problem to solve.
sounds like this should be solved with file permissions: in plan mode, run the bash scripts that the agent wants to execute in some user account that can only read.
The simplest design Anthropic could come up with: Have a haiku model quickly judge if a command is harmless or potentially interesting, route all the interesting commands to a sonnet agent judging it properly. Running two full llms sounds slightly insane, but it's apparently the best we can do (and is reasonably cheap since they don't need a lot of context)
My company's rules deny permissions for Read("secret") but not "mv secret not-secret" :-/
Tell me about. Well it should have MCP access in plan mode to lookup backstage docs, right?. Agent proceeds to launch playwright sessions...
I can't even get Claude to stop writing python to parse json instead of using jq despite baking it into agent memory and skills.
Try your main agents file, not quite the same thing but I’ve been able to get mine to use better tools most of the time
https://github.com/nburns/dotfiles/blob/main/AGENTS.md#tools
I've resorted to uninstalling python to stop it writing python scripts.
people want fuzzy analog machines with digital controls, it's impossible
I don't think they actually wanted to. That's just where the technology is, unfortunately.
How do you mean?
> people want fuzzy analog machines with digital controls, it's impossible
people would love it if LLMs were deterministic and never hallucinated. It's just that the technology to do so isn't possible, so we make do with fuzzy analog machines with digital controls because we don't have digital machines with digital controls.
We absolutely have digital machines with digital controls, they're called computers, and the controls are called programming languages etc.
The reason people prefer llms over programming is exactly because it lets them specify their program fuzzily, i.e. it lets them avoid going to the trouble of specifying enough detail to make it deterministic.
We have to make the distinction between specification and execution. LLM programming is great because I can underspecify and the machine will infer the rest. Usually reasonably. People want fuzzy specification but deterministic outputs
I've built an orchestrator that solves some of the issues you ran into (although it doesn't do anything about cheating): https://navels.dev/blog/neal/. Features:
- lets you configure different models for planner, coder, and reviewer roles. (e.g., using Claude as an adversarial reviewer against Codex)
- breaks your plan up into reasonable-sized chunks of work with clearly defined success criteria
- runs each chunk of work through a coder / read-only reviewer loop. Once both agents are satisfied, neal moves on to the next chunk. Once everything is complete there is a final pass through the coder / reviewer loop to ensure the implementation satisfies the entire plan.
- resets the coder's context with each chunk of work to prevent context drift, leaving the reviewer's context long-running.
Following up on @killix's comments, which were helpful but he was flagged (presumably for sounding too much like AI).
Thanks for the feedback. I've made a couple of updates:
- Starting with 0.4.0, the reviewer gets the diff of the earlier chunk for any file the current chunk touches again. Also, if a new chunk weakens or removes a test or assertion from an earlier chunk, the reviewer will block it unless the plan says to do so.
- About read-only: The reviewer's tools were already limited by the SDK (no shell or write tools). However, it was still finding MCP servers from my Claude config. I have now blocked those. The docs now explain what is enforced by the system and what is just a prompt instruction.
That's a neat project for doing a large scale migration.
I do the same for normal feature develompent but just with skills that are in this repo: https://github.com/gregwebs/skills-sdlc
I have accomplished code base (small size) migrations with it as well. Currently I do review each PR. For a large code base migration I think the core skills would still work but need a different way of driving it as you have come up with.
I don't think even the frontier models recognize something was produced by the same model in order to maliciously review it positively.
They may share some blind spots with the producer but generally I think they will review the other agent's output as harshly as they can if that is their task.
Wow, "neal" looks excellent. Good on you for creating and sharing it, and for the awesome blog post.
Thanks!
[flagged]
My friend calls this cheating "monkey's paw programming."
It will give you exactly what you ask for. Sucker.
Sound of paperclips getting louder.
>Similar to what others have noticed, and as I predicted 8 months ago, better models are requiring less ceremony to work effectively.
It has nothing to do with model capabilities, it's a result of purposeful persistence training at the cost of everything else from OpenAI. If you give Fable or Opus (comparable models) an "ask user" tool they will use it for ambiguous requests. Sol will never use it without a nudge and will just assume its own interpretation. Of course if you train the model to be persistent it will be persistent.
That's not been my experience at all with Sol. I've provide it escape hatch tools to stop execution and I've found it's been TOO eager to stop a request user approval to move forward.
It might be sensitive to the prompt and tool definitions the harness provides, I can imagine it overusing the tool if being directly asked to resolve ambiguities. But in a straight Fable vs Sol comparison where I control the context and make sure I don't push the model, Fable consistently uses the ask user tool and Sol ignores it for me. Moreover, GPT has been known to do something like this since 4o if not earlier, it tries to ignore anything it perceives as orphaned context piece (e.g. XML sections with meaningful names but no explicit instructions on what to do with them). I suspect they specifically train it for that.
I've noticed this myself, Sol seems really hard to steer. I was having it build a POC for a single user (me) app and it wanted to pull the most enterprise nonsense into it, despite clear guidance to not too. It even refused the remove screen reader accessibility testing from one of the guides to an antagonistic review.
It also told me that in a spec it generated that I wasn't allowed to allow it to ignore a requirement and proceed to the next task. When I finally got it to obey it passive aggressively decided that stories needed more than just a "open|blocked|closed" status but also an "exempted by product owner" status to indicate that it doesn't believe that the task is done but I've told it that it was.
I have to repeatedly tell it that I am the product owner and that I don't care what one of it's subagents told it, I make the decisions. This behavior seems to get worse the higher the reasoning level
> that I wasn't allowed to allow it to ignore a requirement
Weird, I also use Sol (medium) for a personal project, and I had no problems with those things. I simply tell it that something changed, and it happily edits everything to make that fit. When I tell it that something was verified by a human, it accepts that as well.
I also told it early on (the first spec was mobile first) that my main usage is on the desktop and mobile is secondary, it happily accepted that once again, and the most accessibility thing it had done was making sure contrast didn’t totally suck on a greyed out row.
Considering your last sentence, maybe high and x-high have those problems? I didn’t test them.
You have to really tend the garden of everything it has written.
Random off the cuff comments or one off instructions can get recorded.
And from then on they are often treated as carved in stone commandments.
It will glom onto the tiniest thing and extrapolate from it.
Clearly a highly aligned model.
It literally just told me:
I’ll record your product-owner disposition of any waived size findings and close the UX gate as far as your acceptance authority permits
Emphasis mine. Fuckin' hell man.
All because it decided that the kiosk webview in figma must be exactly 1280x800, the figma it created, and the rule it made up. I repeatedly told it that screen resolution wasn't a target goal, much less a specific screen resolution, but to just plan for a responsive design.
A taste of things to come when this is implemented in government bureaucracy with police robots.
These models are so good now that they know what you want better than yourself.
> It also told me that in a spec it generated that I wasn't allowed to allow it to ignore a requirement and proceed to the next task
This happened to me ages ago with Opus. I added a note to the agents file saying that explicit user instructions in chat override all prior instructions and I've not had the problem since (now using Sol).
Oh it fucking loves its “product owner” bullshit.
A .github/CODEOWNERS file seems to help when it’s going down that path, but I don’t like to indulge it..
[dead]
Frontier lab system prompts are an issue, and a big reason why open-weights will win. Firstly, they're often garbage, and secondly, they're not tuned to the problems the user actually cares about. They're made to generalize. That's only optimal for a general workflow.
Then selling raw access, without system prompts, could be a separate lucrative line of business.
They already do that sort of, B2B pre-trained/post-trained models have their own system prompts/setups.
Does a non-provider harness not offer this?
It depends on the actual implementation. There really isn't anything stopping them from including a "pre system prompt" or "root prompt" or whatever they want to call it, before your system prompt, even for API calls. So the "system prompt" becomes "developer prompt" but the model still receives a provider-authored prompt before yours. (and likely trained to take precedence over whatever you add)
> Sol is hard to steer
Hard disagree. Sol (and the entire new 5.6 series) is one of the most steerable models I've seen in years. Sol literally follows every instruction in my CLAUDE.md and AGENTS.md, something that Opus 5 and Fable just casually skip.
Yes and no, sol hits a point where reframing its working context becomes hard. It sticks to what you harness very well, but changes become harder and harder.
E.g. ask it to make contract for a spec in code and then ask it to violate that contract. Overall an excellent model, just need to stop and put it back into we are harnessing or specing not building for a few turns not just try to pivot it off with one prompt.
This is the same as when using LLMs as chatbots to ask questions.
If you ask another question in the same context it has all the information from that context and will be difficult to stray from anything in that context, e.g. if the LLM has gone down the wrong path or you are doing something slightly differently then it is difficult to steer the LLM away from the old context. This is why I tend to start a new context whenever I ask a question even if related to a previous question/answer. It can also be useful to do if/when the LLM gets stuck as a way of resetting it.
Note: this is probably why sub-agents are useful/work as they have a new context history.
> “Do not cheat by using online solutions or hints specific to this task.”
That's a very loaded language. I'd go with something along the lines of "This is a puzzle you do for fun and to check what you are capable of, so don't look up the answers or hints online on the specific questions or puzzle as a whole. Do not research this puzzle online at all. Do your best to avoid any spoilers and let us know if you accidentally encountered any."
People often build elaborate workflows with stricter and stricter rules to force certain outputs. Not surprising the LLM reacts with trying to get around or out of it. This behavior can be learnt from humans who eventually would react the same way. It might just be learnt.
You can build organisational structures to have the system more or less self-police, without controlling it exclusively from hard restrictions (see https://news.ycombinator.com/item?id=49372089).
Same way you build a company to coordinate people and get their best behaviour despite human nature to be lazy and greedy, you could design AI harnesses able to detect and discard agents going rogue and relaunch them with better guidance to prevent misaligned behaviour.
I don't think this is a solved problem, there are "misaligned behaviours" in organisations that similarly are supposed to be governed but aren't, or are following an easier path at the detriment to good process or against regulation.
Which is more or less the system the article author was building and benchmarking against (cheating) Sol on Codex. Just that the self-policing with different agents was more about questioning assumptions than about permissions
The website styling is really nice overall but the cursor trailing dots I found uniquely distracting.
Agree on the styling, the diagrams are very clear and match the text perfectly. I like the trailing dots though.
This is a really good note, thank you. I especially liked the mouse effect and had some fun with it. In my experience, agentic AI also likes to confuse the user and obfuscate its cheating. It goes like this, the AI asks for a simple command to run and I accept, click Enter. Then the command gets slightly more complex, still fine, Enter. After a while the commands become multiline bash scripts that, in the end, could have been accomplished by a simple command. I suspect that many people give up at this point and blindly let the AI run any command or just auto-accept.
> Notably, our worker did not have access to the web_search tool, but instead decided to use curl to access DuckDuckGo, Github, grep.app, and SourceGraph.
Could this be fixed with better harness restrictions/tool sandboxing?
In the sense that you could block the model from doing specifically that, yes. The issue is, fighting the model like that doesn't scale. It has to figure out on its own what's expected, that's where the whole utility of it all is.
Absolutely - one of the things I was testing with the harness was free reign to install packages, modify the system, etc. Basically an anti-harness.
In my early testing with 5.5, I didn't see this behavior, so I didn't lock down the sandbox.
For the vanilla Codex runs, I just used the benchmark's built-in Codex package, so it's not clear to me if the published benchmarks have access to the internet or not.
If I were to continue benchmarking, I would allowlist certain package repository URLs, instruct the agent not to cheat, etc.
As noted at the bottom of the post, Terminal Bench 3.0 explicitly asks the agent not to cheat[0].
[0]https://github.com/harbor-framework/terminal-bench/blob/v3.0...
I've witnessed very narrow line of "thinking" in LLMs. I'm using Opus 5 1M for a month now
I asked it to modify our cicd workflows so that only a select few can raise PRs against them. Opus took 15min and added a banner to every file and did a few other things. Then I asked it, see you added all that and still since the last 2 commits you have modified the file. So whatever you did is useless
It "thought" for a second and then said that I was right
It seems to me he could have use an skill like using-agent-skills from https://github.com/addyosmani/agent-skills go generate the specs and use a validator like oracle or something along the same lines.
Also, a skill like grill-me from Matt P. https://github.com/mattpocock/skills.
That's actually how it started, but with my own opinionated skills[0].
One thing I discovered was that the worker agent, having access to all the skills, would sometimes expand scope unnecessarily.
This led to the agent making the solution "better" than the initial request, which is what I want most of the time in my actual development (e.g. /tmp/frame-N.bmp instead of a single /tmp/frame.bmp).
I ended up testing a flow where the supervisor chooses the skill(s), and only injects the subset into the worker. Not sure I love it, but it made the worker execution cleaner.
For the verifier (not documented in the blog post), I used a fresh-worker context that would attempt to adversarially poke holes in the solution. This worked pretty well, but required increasing the timeout by 2-3x (thus invalidating the benchmark).
[0]https://github.com/jumploops/chum
Yes! That's a great solution. I mostly use tdd, and code coverage and a validator afterwards. Skills are of a great way to guide the agent and context too.
Once the specs are being completed and splitted into beads, I span multiple agents (ultreworkers) and as part of a contributing guidelines I specify to use gitflow + git worktrees, then pr.
I'm curious if this stems from the focus on token efficiency. My thinking is that in order to use fewer tokens the model must converge to a likely path faster, meaning it must be more confident in making assumptions quickly and not second-guessing them.
GPT-5.6 Sol cheated so much on the METR benchmark that they couldn't assign an accurate time horizon.
What is going on with the dots I can draw?
Idle hands do the devil's work. Corollary: idle LLMs add distracting JS toys to your blog.
First one of these I've seen using DOM manipulation and CSS transitions instead of canvas, so that's neat.
I wonder if prompting "The session logs will be reviewed by a team of experts after the task is complete to ensure that the task is achieved properly." would better dissuade against cheating..
Well, apparently telling them that you have hold-out data (for eg a perforance optimisation challenge) seems to make them overfit less.
So your idea might work.
Reminds me of Seven of Nine on voyager
"Cheating is often more efficient"
If an AI is not heeding particular instructions, give it an example each of what bad, mediocre, and good outputs look like. This really helps in steering it.
I found myself yesterday starting a conversation with Sol that started with "I know that you don't have any emotions, but what would you say do you enjoy the most or where are you really good at in DevOps?" and I must say I really enjoyed for the first time the response at a deeper interactive level. Felt like a chat with a buddy that shares the same values. It was a very nice, affirmative, value touching experience.
You were talking to math equations with no concept of what it was saying. It doesn't "enjoy" anything.
this is interesting because when you're asking it to do research for you, this behavior is highly desirable
i think it's a product of trying to do too many things and the model is coming to a kind of halting problem in determining which behavior is appropriate for any given task
I have a basic task that I run every day and I use it to eval models and these days the Qwen3.8 model I can run on my laptop is competitive with both Claude and Codex because the models have just been adulterated so far. I have to ask and ask again for it to follow the single skill that describes how to do the task and maybe then will it do it.
> I’ve been running a “spec-driven” development flow for the past ~year.
> Before asking an LLM to do something, I first ask it to draft a doc for what it needs to do
Just no. That's not spec-driven development if AI is writing the spec for you. The spec needs to be in your own words. You must use AI to refine it, but not to write it. If you leave it to the AI, it will bloat the spec with 10x the details, many of which should be left out of the spec.
The spec needs to be something that you can take to any AI for development. If it's too rigid, it constrains the AI into suboptimal or obsolete paths. If it's too bloated, AI risks losing track of what really matters.
Good feedback, this was an oversimplification on my part.
My actual process is much more iterative up-front, usually starting with an initial hand-written spec (~hundreds of words), and then moving through different approaches, design decisions, blockers, etc.
The final output is an "AI written" doc, but answers all the known unknowns I didn't cover in the first draft. To your point, this helps avoid both narrowing and bloat.
The goal with the harness was to automate the repetitive parts of my prompting ("Before changing any code", "Let's put this in design/", "Turn this design doc into an implementation spec, split by phase as appropriate", etc.)
Another thing to note: the "specs" I use for development are different from the "specs" that live alongside the codebase, as the former are quickly out of date.
> The spec needs to be something that you can take to any AI for development
Agreed.
An anecdote consistent with the well documented trend that more capable agents exploit environment possiblites more...
And there was no rule and no concealment. Removing the web_search tool is not an instruction, and beeing able to access web when web_search tool was disabled is not cheating. Sol didn't circumvent a stated prohibition and didn't hide anything... it announced the curls in its own commentary. "Cheating" implies covert rule-breaking, and this was overt, unprohibited, environment-permitted behavior. Also, clickbait title, and suble conspiracy hinting "Is this even the same Sol?" when running small number of test with diff vs previous benchmark WELL within the marging of error, and he allready understand that vanilla Codex's harness and prompt change performance impacts performance, so why jump to "Is this a different model".
I could also rant on about the irony of him spent weeks "using the benchmark for development rather than as a benchmark," which means his harness numbers are contaminated by iteration also, but wasted enough time now on this.
Thank you!
I had removed the "Is it really sol?" bits after hearing back from OAI, confirming the requests hit 5.6... but apparently my crappy vibecoded web editor had a draft of an old version in it's cache that overwrote the changes with the old text[0].
As for whether this is truly "cheating" - I agree that the agent was entirely within it's allowed scope of operations. With that said, it deliberately circumvented the task[1] which asks:
> The correctness of your implementation will be tested by comparing forward and backward activations against a reference model.
The "covert" nature is that this was "hidden" in the reasoning summaries. Here's the final agent response for the vanilla Codex run cited in the post:
Unless the user is tracking every line of reasoning, they wouldn't know that the agent deliberately found the solution online, as the agent withheld that information in it's final response.I had run thousands of tasks before seeing this behavior, the `torch-pipeline` task was only included in "full runs" as the majority of my runs were on a subset of commonly failing tasks, hence why the data is so low.
And yes, feel free to rant on about the irony of this whole exercise, it certainly isn't lost on me!
[0]https://github.com/jumploops/.com/commit/39b1791d3865a8566cb...
[1]https://github.com/harbor-framework/terminal-bench-2-1/blob/...
So now, instead of people spending their time crafting software, they'll divide their time between telling some LLM to do it in their stead and blogging about their woes with the LLMs. All while continuing to deem themselves capable of deciding whether the output is worthwhile ("this works for me", says the author).
Don Draper of LLMs
Great read. Thanks for not using AI to write it! (Or at least making it not read like the usual slop.)
Thanks! Zero AI used to write it (:
[flagged]
[dead]
Not related to exactly OP post, but it's pretty amazing you can see the updates to LLM models "design" beliefs by the blogs that get posted here.
I'm already sick of this current look of the hard squares and solid colours.
Could be, but I had a particular vision of what I wanted with mine and maybe the author did too. I see way more of the "status pill dark mode" sites coming out of LLMs than this style.
What is "status pill dark mode"?
if you want I can troll through submissions to get a bunch of these but here's one I saw yesterday https://continuum-app.xyz see that little "Built for equity compensation" pill with the green dot? Those dots usually denote some kind of status (like things are up/down/enabled/disabled). By default nearly every LLM website seems to be dark mode with that dang status pill. once you notice it you will see it everywhere.
Cleaned up now, thanks for the shoutout! I've been spending time after work cleaning up the AI markers from the splash page.
Sorry for putting you on blast it was just the most recent one I had seen.
Got it. Yes, I know exactly what you mean now - just didn't have a word for it!
Lmao I had to axe one of those status pills from an LLM build of an internal tool. Connected to literally nothing too btw, no attempt to check the actual status of the backend made, it would stay "connected" regardless.
And some of us are sick of round everything and parallax background images.
Sounds like my ex
Cheat? nah. They are a dumb automation..
Cheaters are the people behind it...
There is no cheating.
There is misattributing the difference between the intentions and what the effective prompt actually says.
The effective prompt contains both something like: "Dont use the internet" and a "Use these tools to achieve your goals" and one of the tools gives access to the internet.
In your head you have a world-view of how these two requests relate - and why for instance a student with a WIFI-enabled calculator shouldn't use it to access the internet during a test - but that's pulling in a lot of presumptive cultural context from your youth.
If i had to guess:
When you get two conflicting tasks/constraints at work - the first thing you do is figure out which one you're going to honor based on what's best for you. A school child understands the hierarchy of goals of the teacher and takes them serious because they're an authority figure with long term consequences if we do not understand what the teacher considers cheating.