I kind of have a different idea of agents. I totally believe in a deterministic scaffold but I really think that an agent should be as deterministic as possible - the more code, the better.
Think of a typical loop we may ask of Claude Code today (assume we are not using TDD): run some test suite with fail fast mode, diagnose if the failure is due to recent feature changes (pass reference to backend/frontend, github issues, PRD,...). Ask CC to decide if test failed due to feature change and then update the test. Perhaps ask CC to use sub-agent to investigate and fix (if deemed so). Commit each fix, move on to next.
I know, this has so many ways to make blunder but I am talking about the agent here, not our error-prone test maintenance. What if we had an agent that had context of your codebase, deterministically ran test suite, linter, hooks, etc. The "English" prompt would become a code loop with the LLM only brought in to decide if a test has failed because of feature change. Also, we can extract git log, JIRA and what not.
Each tool here is real code. Executable code that calls others and only prompts when they meet edge cases. Edge cases are defined but we can now accelerate the maintenance of these tools using agents themselves. But the system is built on "programs that do one thing and do it well" and then reach out to an LLM for its specific edge case. The agent is how these executables work with each other.
What you are describing is precisely my goal with my agent framework.
One of the meta-processes designed in is pushing automated processes, both defined and discovered, down as far as possible. "Down" here means as far towards the metal as reasonable. So automate the automatable stuff, and leave the LLMs to do stuff LLMs are actually good at.
A trivial example is 'handle this bugfix ticket'. Many actions in a bugfix are pre-defined, for example a git commit at the end of the ticket. So Maelstrom will, at the end of a bugfix workflow, will force a git commit from the LLM that did the implementation. The LLM never even sees the git command, it just fills in a JSON field with a commit summary, and the workflow handles the commit.
100% agree that the more deterministic code the better up to the limit where you need the LLM's ability to be non-deterministic to kick in.
There is this ACM blog post called "Manual Work is a Bug" [0] that was originally written to help humans automate processes using code. I find it just as applicable today as when it was written. You and the LLM look at what has to be done and then figure out the scripts/tools to make it happen. You then tie those tools into a system.
The more I use the above the more it makes sense and the worse the whole "just commit the prompt" seems like nonsense.
This is an interesting share, thanks. Yes, that is my mental model. Use coding agents to generate more "programs" (scripts) to automate everything. Have edge case handlers - and these handlers can develop/update the original scripts.
I’m definitely on the deterministic code train as well. All of my success for long running tasks has been around wrapping the agentic harness (cc, codex-cli, etc.) in a deterministic workflow with deterministic gates. We need a name for this outer layer. In my mind that is the true harness because it constrains the agents failure mode. I think flow engineering has been proposed. Maybe it’s the agentic exoskeleton?
I call it tool-response engineering -- the tool-response JSON object itself and any content supplied, plus the available mechanisms after an initial tool call has been made.
From harness? Because people expect a squishy set of things from a harness that is different from what I end up building. I end up with a rigid internal structure that the harness uses in-turn (tests with clear error messages, tools, etc.) and a matched rigid external structure that drives the turn tracking progress and deterministically handles the overall progress. You could call that whole thing a harness but that makes the definition muddy and hard to talk about. So scaffold or skeleton seems more appropriate. The harness constrains the agent. The matched endoskeleton and exoskeleton gives it structure.
In the age of agents, I feel like BDD is more important than ever since it describes the behavior, and if it does so well it’ll make it much easier for these tools to pick up correct behavior.
The best part about LLMs is that, if you write gherkin style tests, you don't even need to automate them yet to get value, the AI can "manually" validate initially and then only codify them into deterministic execution after they've been nailed down. That way you have a bulk set of tests that run automatically, and a few candidate tests that are not automatically run yet, and the boundaries expand over time.
One of my policies for agentic coding is to spend much effort in developing tests, coded tests not LLM based vibes. My projects have around 1:1 LOC between code and tests. Tests are like skin, when the skin is pricked it hurts, agents need to feel pain too.
OP's idea "everything is a text file" is good and I use it too. My plans are saved as task.md files, numbered and named. Work items are checkboxes inside the file, closed work items are checked and a comment is added on the same line to provide feedback about the implementation.
I also keep a current-state-of-the-world document, it should be <20KB of text, keep the essential decisions and intents. Loading it allows resuming in <30s.
Something I never saw anyone else do - I save all user messages in a chat_log.md file which is referenced for intent alignment and state recovery. I consider the chat log on the one hand, and coded tests on the other hand as the two walls, the agent works in the mid section between them.
I don't think we're necessarily in disagreement here: I agree that determinism should be taken as far as possible, but once we zoom out of the software engineering world I think things tend to get a lot less easy to automate if that makes sense.
Not necessarily. I have been building client projects for the last few months only using coding agents. I use way more existing tools to handle a lot more of our digital footprint than I used to before: pdf, images, excel, ocr and many more.
As coding agents have accelerated my work, I just build tons of tooling around existing software. Or in rare cases build new ones. If we zoom out of software engineering, we will still be in the realm of files - text or binary. That does not change.
The question is - do we let agents run the tools or the "programs" call the LLMs. The OS is the new agent, but not the same sense of "agent". I want LLMs to be lightly sprinkled in a future "agent" OS, not the other way around.
It is not blasphemy if langgraph is trying to do that. As I understand langgraph manages orchestration in custom built agents. I usually stay away from systems which already make it seem as if building agents is a ritual.
What I am saying is the opposite - use Claude Code or whatever else - generate actual "programs". Basically scripts. We have tons of ways for "programs" to interact with each other. Then have clearly defined edge case handlers - think "try/catch". How far do you want to go down the rabbit hole in the "catch"? Do you want to re-write a new version of the "program" itself? I do not know, but this type of a system is what Unix already is, with the addition of programs themselves reaching out to LLMs in well defined edge case handlers.
You can drive agents via ACP these days, which I think is the layer you would want for what you’re talking about.
The API is basically what you see as a user of Claude Code or Pi or whatever. You can make new sessions, send messages to sessions, configure which MCPs get started, etc.
I’ve been poking at something similar to what you’re talking about via that route. My client prompts the agent to do a thing, and then afterwards launches deterministic things to check it which can either re-prompt the original session or start a new session.
Eg it automatically runs the tests afterwards, and will send a new prompt in the original chat to fix them if they fail. I also briefly poked at a security analyzer that gets changed files via git and makes a new session to check whether there are security issues and propose a fix that then gets sent to the original session.
If you want a circular loop where the LLM can adjust its own workflow while keeping it deterministic, you can let the agent modify the ACP client that drives it.
I think it all comes down to tight, context efficient, deterministic feedback loops. Pre commit hooks work well for this type of thing. Ideally I think a set of those hooks should run on every file edit, however I haven't gotten around to testing something like that yet.
This post is chock-full of soft ideas. They make no meaningful steps toward "a harness that can do anything". The suggestion is to replace a small node application with specific tooling, with a vm "to give it more capabilities". This is what Agent sandboxes are, already. Making the sandbox the harness, doesn't achieve a concrete goal.
I disagree with the idea that file is a good metaphor for LLM. Files have seek and byte streams, which is just an unneeded abstraction for LLM. The LLM doesn't need to seek or jump to the middle of a file, if you store and organize your data properly.
Why force the LLM to use files over vector database or key-value stores, just because it's a design principal for UNIX (which is designed for human users, not LLMs.)
Feel like there's now a lot of modern software that can do particular things well but there is new promise in finding universal abstractions that can ideally run recursively / safely run any software at their leaves. (Unix + tooling is most of the way there ofc)
I feel like Docker Compose / K8S / VM / Dagger.io layers are close but can't quite always recurse flexibly and aren't always simple to run with. Networking / devices / auth are often awkward choke points
Love the Unix philosophy and the buffs mentioned of the Linux FS. Lean, transparent, and auditability-first is exactly the direction harness' should continue in.
Something I am convinced of though, there probably isn't a single `best` harness for all tasks. Different workloads will likely perform better with certain combinations of model + harness, especially when we are talking about token budgeting and cost tracking.
Ambiance feels like a great base “kernel” to build those variants on top of, rather than the one true harness.
I think tools have to be flexible enough to accommodate a range of expected inputs. Even assuming models are trained to supply a particular schema, they are stochastic and randomly output variants regularly. My experience has been that it's better to try to auto-normalize the range of anticipated outputs and make tools as forgiving as possible rather than force an overly strict schema that might not match training data on the model.
Mapping agent concepts to the Unix environment sounds like a great idea in general, but I get off the train at adopting the FHS, which is an ancient relic that has no place in a green-field system. I don't know exactly what shape it should take, but something along the lines of Nix seems more appropriate. Or maybe Plan 9.
One of the ideas I had (and I'm still tinkering with) is 'agent as a linux user'. All of the perms/groups/access rights are already built in, so it would make sense to leverage these for agents running on the same host. Heck, there's even mail built in for async comms between agents.
What has been the most helpful when developing harnesses:
> When in doubt, simplify. Remove, trim and minimize. Reproduce issues in as small cases as possible, understand the full design completely, there is no shortcuts for this.
I really like this idea and the way you mapped the concepts to unix primitives. Indeed llms are already "unix native". I've been experimenting with similar event driven workflows using k8s primitives but that's one level up the stack. This makes a whole lot of sense to me in terms of organizing a shared mental model. Will definitely check it out. Thanks for the good work.
Actually I was initially going to put this on my Arch machine because utilizing FUSE made a lot of sense to me, but I realized that I use my mac a lot more often so I suppose that's what caused that choice.
I think domain specific harnesses are already surpassing generic harnesses. I also think software development is its own domain.
My harness is a Claude Code plugin with its own brainstorming, adr, and planning skills with associated review and interview skills. Behavioral testing related to acceptance criteria is built in. Everything in my harness is gated to prevent ratholes.
I recently inflated a docker container to execute a set of work with Claude in unsafe mode and immediately saw problems with everything it was doing…and then I realized I had not installed my harness.
Running Claude without an engineering harness is like driving a car without brakes or a steering wheel.
I'm kind of in the same boat and it's been pestering me for months. Every agent is simply a less capable Claude Code.
If it had a lossless, massive context window (100m-1b tokens), then it will squash everything. Give it bash + r/w and it can in theory /goal anything.
I think there's something to be gained in a production environment be siloing agents for reproducebility/auditability, but I suspect that will go away in the future.
There's that video of a silly demo someone made of an OS that was just nested copilot instances that generated the HTML of each window, which allowed you to do whatever you could imagine. It was seen as silly because it was, but that seems truly transformative.
i think the point is less about what agent you should or should not do and more about what is the natural harness for an agent to succeed in. And agents are often autonomously doing things right now, why would you want claude code doing such stuff for you?
I kind of have a different idea of agents. I totally believe in a deterministic scaffold but I really think that an agent should be as deterministic as possible - the more code, the better.
Think of a typical loop we may ask of Claude Code today (assume we are not using TDD): run some test suite with fail fast mode, diagnose if the failure is due to recent feature changes (pass reference to backend/frontend, github issues, PRD,...). Ask CC to decide if test failed due to feature change and then update the test. Perhaps ask CC to use sub-agent to investigate and fix (if deemed so). Commit each fix, move on to next.
I know, this has so many ways to make blunder but I am talking about the agent here, not our error-prone test maintenance. What if we had an agent that had context of your codebase, deterministically ran test suite, linter, hooks, etc. The "English" prompt would become a code loop with the LLM only brought in to decide if a test has failed because of feature change. Also, we can extract git log, JIRA and what not.
Each tool here is real code. Executable code that calls others and only prompts when they meet edge cases. Edge cases are defined but we can now accelerate the maintenance of these tools using agents themselves. But the system is built on "programs that do one thing and do it well" and then reach out to an LLM for its specific edge case. The agent is how these executables work with each other.
What you are describing is precisely my goal with my agent framework.
One of the meta-processes designed in is pushing automated processes, both defined and discovered, down as far as possible. "Down" here means as far towards the metal as reasonable. So automate the automatable stuff, and leave the LLMs to do stuff LLMs are actually good at.
A trivial example is 'handle this bugfix ticket'. Many actions in a bugfix are pre-defined, for example a git commit at the end of the ticket. So Maelstrom will, at the end of a bugfix workflow, will force a git commit from the LLM that did the implementation. The LLM never even sees the git command, it just fills in a JSON field with a commit summary, and the workflow handles the commit.
100% agree that the more deterministic code the better up to the limit where you need the LLM's ability to be non-deterministic to kick in.
There is this ACM blog post called "Manual Work is a Bug" [0] that was originally written to help humans automate processes using code. I find it just as applicable today as when it was written. You and the LLM look at what has to be done and then figure out the scripts/tools to make it happen. You then tie those tools into a system.
The more I use the above the more it makes sense and the worse the whole "just commit the prompt" seems like nonsense.
0 - https://queue.acm.org/detail.cfm?id=3197520
This is an interesting share, thanks. Yes, that is my mental model. Use coding agents to generate more "programs" (scripts) to automate everything. Have edge case handlers - and these handlers can develop/update the original scripts.
I’m definitely on the deterministic code train as well. All of my success for long running tasks has been around wrapping the agentic harness (cc, codex-cli, etc.) in a deterministic workflow with deterministic gates. We need a name for this outer layer. In my mind that is the true harness because it constrains the agents failure mode. I think flow engineering has been proposed. Maybe it’s the agentic exoskeleton?
I call it tool-response engineering -- the tool-response JSON object itself and any content supplied, plus the available mechanisms after an initial tool call has been made.
Why do you need a different name?
From harness? Because people expect a squishy set of things from a harness that is different from what I end up building. I end up with a rigid internal structure that the harness uses in-turn (tests with clear error messages, tools, etc.) and a matched rigid external structure that drives the turn tracking progress and deterministically handles the overall progress. You could call that whole thing a harness but that makes the definition muddy and hard to talk about. So scaffold or skeleton seems more appropriate. The harness constrains the agent. The matched endoskeleton and exoskeleton gives it structure.
I believe they call it "loop engineering"
In the age of agents, I feel like BDD is more important than ever since it describes the behavior, and if it does so well it’ll make it much easier for these tools to pick up correct behavior.
Gherkin style tests also come to mind
The best part about LLMs is that, if you write gherkin style tests, you don't even need to automate them yet to get value, the AI can "manually" validate initially and then only codify them into deterministic execution after they've been nailed down. That way you have a bulk set of tests that run automatically, and a few candidate tests that are not automatically run yet, and the boundaries expand over time.
One of my policies for agentic coding is to spend much effort in developing tests, coded tests not LLM based vibes. My projects have around 1:1 LOC between code and tests. Tests are like skin, when the skin is pricked it hurts, agents need to feel pain too.
OP's idea "everything is a text file" is good and I use it too. My plans are saved as task.md files, numbered and named. Work items are checkboxes inside the file, closed work items are checked and a comment is added on the same line to provide feedback about the implementation.
I also keep a current-state-of-the-world document, it should be <20KB of text, keep the essential decisions and intents. Loading it allows resuming in <30s.
Something I never saw anyone else do - I save all user messages in a chat_log.md file which is referenced for intent alignment and state recovery. I consider the chat log on the one hand, and coded tests on the other hand as the two walls, the agent works in the mid section between them.
https://horiacristescu.github.io/claude-playbook-plugin/docs...
Thanks for the link. I need to evolve my own system in this direction.
Yes, I agree with this. I am not focusing on tests as much and I think that is a big mistake. Agents need to immediately understand something is off.
I don't think we're necessarily in disagreement here: I agree that determinism should be taken as far as possible, but once we zoom out of the software engineering world I think things tend to get a lot less easy to automate if that makes sense.
Not necessarily. I have been building client projects for the last few months only using coding agents. I use way more existing tools to handle a lot more of our digital footprint than I used to before: pdf, images, excel, ocr and many more.
As coding agents have accelerated my work, I just build tons of tooling around existing software. Or in rare cases build new ones. If we zoom out of software engineering, we will still be in the realm of files - text or binary. That does not change.
The question is - do we let agents run the tools or the "programs" call the LLMs. The OS is the new agent, but not the same sense of "agent". I want LLMs to be lightly sprinkled in a future "agent" OS, not the other way around.
I know it's a type of blasphemy here, but deterministic workflows such as what you describe is where langgraph really shines imo.
It is not blasphemy if langgraph is trying to do that. As I understand langgraph manages orchestration in custom built agents. I usually stay away from systems which already make it seem as if building agents is a ritual.
What I am saying is the opposite - use Claude Code or whatever else - generate actual "programs". Basically scripts. We have tons of ways for "programs" to interact with each other. Then have clearly defined edge case handlers - think "try/catch". How far do you want to go down the rabbit hole in the "catch"? Do you want to re-write a new version of the "program" itself? I do not know, but this type of a system is what Unix already is, with the addition of programs themselves reaching out to LLMs in well defined edge case handlers.
You can drive agents via ACP these days, which I think is the layer you would want for what you’re talking about.
The API is basically what you see as a user of Claude Code or Pi or whatever. You can make new sessions, send messages to sessions, configure which MCPs get started, etc.
I’ve been poking at something similar to what you’re talking about via that route. My client prompts the agent to do a thing, and then afterwards launches deterministic things to check it which can either re-prompt the original session or start a new session.
Eg it automatically runs the tests afterwards, and will send a new prompt in the original chat to fix them if they fail. I also briefly poked at a security analyzer that gets changed files via git and makes a new session to check whether there are security issues and propose a fix that then gets sent to the original session.
If you want a circular loop where the LLM can adjust its own workflow while keeping it deterministic, you can let the agent modify the ACP client that drives it.
I think it all comes down to tight, context efficient, deterministic feedback loops. Pre commit hooks work well for this type of thing. Ideally I think a set of those hooks should run on every file edit, however I haven't gotten around to testing something like that yet.
I get what your saying, however we don't need to be stuck with the Claude code harness. You may find this interesting.
https://www.langchain.com/blog/tuning-the-harness-not-the-mo...
This post is chock-full of soft ideas. They make no meaningful steps toward "a harness that can do anything". The suggestion is to replace a small node application with specific tooling, with a vm "to give it more capabilities". This is what Agent sandboxes are, already. Making the sandbox the harness, doesn't achieve a concrete goal.
I disagree with the idea that file is a good metaphor for LLM. Files have seek and byte streams, which is just an unneeded abstraction for LLM. The LLM doesn't need to seek or jump to the middle of a file, if you store and organize your data properly.
Why force the LLM to use files over vector database or key-value stores, just because it's a design principal for UNIX (which is designed for human users, not LLMs.)
What's with this "harness" word people have been trying to adopt lately? Are we all going rock climbing?
A harness is just a piece of software that exposes APIs for reading files, connecting to the internet, etc.
It parses the LLM output for tool calls, executes the command, and puts the output back into the LLM input. That's all there is to it.
I think a better term would be bridle, but most people don't know it. But they do put harnesses on their kids and pets.
Buzzword bullshit bingo is a must for anyone who wants to ride the hype.
We're taming a beast.
framework is too 90s
The awesome power of the AI must be somehow contained by us mere mortals.
“That can do anything” gives me Zombo.com vibes. I would be more interested if it seemed to do particular things well.
Feel like there's now a lot of modern software that can do particular things well but there is new promise in finding universal abstractions that can ideally run recursively / safely run any software at their leaves. (Unix + tooling is most of the way there ofc)
I feel like Docker Compose / K8S / VM / Dagger.io layers are close but can't quite always recurse flexibly and aren't always simple to run with. Networking / devices / auth are often awkward choke points
Or if it had any kind of measurement across a range of tasks
Love the Unix philosophy and the buffs mentioned of the Linux FS. Lean, transparent, and auditability-first is exactly the direction harness' should continue in.
Something I am convinced of though, there probably isn't a single `best` harness for all tasks. Different workloads will likely perform better with certain combinations of model + harness, especially when we are talking about token budgeting and cost tracking.
Ambiance feels like a great base “kernel” to build those variants on top of, rather than the one true harness.
How much do the labs post-train on the harness inputs & outputs? That's a critical piece to understand if a "generic" harness is at all possible
https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/ says already too much to use foreign tools.
I think tools have to be flexible enough to accommodate a range of expected inputs. Even assuming models are trained to supply a particular schema, they are stochastic and randomly output variants regularly. My experience has been that it's better to try to auto-normalize the range of anticipated outputs and make tools as forgiving as possible rather than force an overly strict schema that might not match training data on the model.
Mapping agent concepts to the Unix environment sounds like a great idea in general, but I get off the train at adopting the FHS, which is an ancient relic that has no place in a green-field system. I don't know exactly what shape it should take, but something along the lines of Nix seems more appropriate. Or maybe Plan 9.
One of the ideas I had (and I'm still tinkering with) is 'agent as a linux user'. All of the perms/groups/access rights are already built in, so it would make sense to leverage these for agents running on the same host. Heck, there's even mail built in for async comms between agents.
What has been the most helpful when developing harnesses:
> When in doubt, simplify. Remove, trim and minimize. Reproduce issues in as small cases as possible, understand the full design completely, there is no shortcuts for this.
I really like this idea and the way you mapped the concepts to unix primitives. Indeed llms are already "unix native". I've been experimenting with similar event driven workflows using k8s primitives but that's one level up the stack. This makes a whole lot of sense to me in terms of organizing a shared mental model. Will definitely check it out. Thanks for the good work.
thanks a lot! let me know what you think -- i'd love to share ideas with people doing similar stuff!
I do have a gripe already... The readme says macos, however as of yet I don't see the actual dependency. Sort of defeats the *nix spirit as well, no?
Actually I was initially going to put this on my Arch machine because utilizing FUSE made a lot of sense to me, but I realized that I use my mac a lot more often so I suppose that's what caused that choice.
Yeah, a little... I was mostly building this for myself so I didn't really think about other platforms but I'll get to it asap :)
good news. now you have a feature request.
Or an incoming PR ;)
I think domain specific harnesses are already surpassing generic harnesses. I also think software development is its own domain.
My harness is a Claude Code plugin with its own brainstorming, adr, and planning skills with associated review and interview skills. Behavioral testing related to acceptance criteria is built in. Everything in my harness is gated to prevent ratholes.
I recently inflated a docker container to execute a set of work with Claude in unsafe mode and immediately saw problems with everything it was doing…and then I realized I had not installed my harness.
Running Claude without an engineering harness is like driving a car without brakes or a steering wheel.
Thank you for framing my laziness as being a daredevil
Why are we not just using Claude Code or Codex on our machine and using this thing? Real question...
I'm kind of in the same boat and it's been pestering me for months. Every agent is simply a less capable Claude Code.
If it had a lossless, massive context window (100m-1b tokens), then it will squash everything. Give it bash + r/w and it can in theory /goal anything.
I think there's something to be gained in a production environment be siloing agents for reproducebility/auditability, but I suspect that will go away in the future.
There's that video of a silly demo someone made of an OS that was just nested copilot instances that generated the HTML of each window, which allowed you to do whatever you could imagine. It was seen as silly because it was, but that seems truly transformative.
i think the point is less about what agent you should or should not do and more about what is the natural harness for an agent to succeed in. And agents are often autonomously doing things right now, why would you want claude code doing such stuff for you?
This one is open source, and potentially better, since it responds to (file system) events instead of polling.
Just one more harness bro, one more harness I swear one more harness and we will have solved AGI bro