A lot of harness launches recently but this one actually stood out to me. Just like the Prime Intellect harness that launched recently, this one also leverages RLM (recursive language model). It’s supposed to allow a persistent agent.
The specifics are really interesting. The model itself can change the user input and when you message it, it gets entered into “its stream of consciousness.”
I have a software factory going with Pi, might swap the main orchestrator with Headlong and have it be persistent to drive the other sessions without me.
A software that's optimized for agentic development, varying levels of review necessary depending on the technical choices of the factory developer. Some review may be automated via agents itself etc.
Ymmv obviously. Its definitely one of the funnest toy projects I've been working on for the last 5-7 months now (such a factory, mine is unreleased however - just for myself as of today)
Very fascinating, super interesting engineering. Although i do find it very funny how they just bypass a massive vulnerability, basically zero data isolation (even between good actors, let alone bad ones) with 3 sentences. Only in the llm space you can slap a massive limitation like this in the middle of the article and continue like nothing happened
> Whatever anyone tells Audel becomes part of the single experience that every other conversation draws on. In practice, Audel is bad at keeping secrets. Ask it what it’s been working on with someone else and it will often just tell you, even though we’ve asked it not to. We also haven’t studied what happens when two people give conflicting instructions. For now, we assume anything you tell Audel is shared with everyone on the team.
The part they punt on ("we haven't studied what happens when two people give conflicting instructions") is the interesting part. That's not a memory problem, it's an authz problem. If everyone writes into one shared stream then there's no model of whose instructions bind the agent or who can override whom. It's resolving the conflict that will generate the greatest "learnings" and advance the agent. This basically becomes a tool designed to misbehave rather than a tool that will learn creatively.
We all know how conflicting instructions to AI end - "I'm sorry Dave. I'm afraid I can't do that"
I'm curious, you say "super interesting engineering" but then they say "it will often just tell you, even though we’ve asked it not to" and to me that seems like extremely shit engineering.
Where are the interesting engineering parts at? Seems to be an interesting idea and perhaps design, but to call the implementation/engineering itself bad seems to be an understatement.
The security and the overall engineering were entirely separate items in that comment I think. It was explicitly called out that this is a security problem that you'd really only see treated in this manner in the LLM space. For what it's worth it's effectively unsolvable (AFAIU) short of realizing AGI with an amicable alignment.
What do you mean unsolvable? Don't give the LLM access to stuff it shouldn't, this is like Access Control 101, not sure how anyone can claim that particular problem is unsolvable?
"turn -> FINAL -> schedule wake-up", this is where my excitement has faded unfortunately. Many of us are probably wondering about the same idea: bridging the gap between a reactive agent and my daily workflow or existence. But, this still feels too close to how Claude (or any other agent) runs as a process in the background (always ON), where you can use a custom channel to feed the dialog with external signals like chat, CI/CD events, whatsapp, etc.
Humans aren't scheduling a wake-up to the next thought. Ideally, a sub second agentic loop with no FINAL / wake-up, always "spinning" would get closer. I'm conscious about the waste of resources this would drag with it (because of current architectures), but exciting still.
PS. I love the take on using bash instead of Python (one less abstraction layer!) and using UNIX fundamentals as stepping stone when composing tools as agents are naturally drawn to using it on a box anyways.
Really hard to support these types of projects without subscription options. BYOK API only approach can run huge amounts of money. I budget for $200 Pro subscriptions. Just using Openrouter for a dark factory kanban ticket system using frontier models will run you $50-$150 which is about 1 ticket.
Can we please not normalize telling people to install things by curl piped into bash? I hate this trend. And particularly not for a very new, mostly untested by a wider audience piece of software from a company that few have ever previously heard of.
It even says, quoting from the website: "Headlong is alpha research software."
Yeah that's totally something I want to curl thing.sh | bash , great idea.... Wow.
I understand that people want to get people using their software as quickly as possible and with the absolute minimum of friction, but let's put some more thought into how this could be done in a less sketchy way.
It's like we've regressed to the days when you would download a .exe file from tucows and blindly run/trust it on your windows 98SE PC.
You can always read the script file (orm ore realistically feed it to agent).
If they distributed appimage or rpm would that be safer anyhow? Cannot it run malitious code same as the script would run?
One of the big problems is that no one actually does read these scripts. You could say "Oh but it's their own fault, duh" but theres a very legitimate argument to be made users going the path of least resistance and that you shouldn't offload this responsibility on your users.
Regarding appImage or rpm, attackers need to build and package these to inject these, while this curl | bash pipe opens up the possiblity of payloads simply by taking over the domain. And this isn't really that far fetched, just think about the Notepad++ update payload recently. The regular package was unaffected while the domain used for the update was taken over.
Then theres also the argument about normalization. Just like he said, this isn't just something he said, this is a very real argument. You don't want to teach users bad habits. Even if / you / inspect the code you get, not everyone will. And ultimately, we should strive to make the Internet a safer place, if only to get less botnets.
This is the same as terms of service agreements. There's a crowd mentality around it where only one person with enough of a voice needs to read it and scream. It's brittle because if everyone thinks that then nobody checks it but the probability scales with the attention it gets.
Totally agree we should make the internet safer, this is just a clarification that we don't necessarily need everyone to check it. One person who always checks everything and sends this to their LLM and posts a viral post can be a deterrent.
The installer is one threat. You're then running this on your machine which is the next threat.
andy here (headlong post author).
terminal bench 3 is pretty popular for comparing different harnesses using the same underlying model (it's another laude project actually). artificial analysis has an index. you can look at the model cards of popular model releases- they tend to have the most popular current benchmarks on them.
w/ headlong we decided to announce it before we've benchmarked it. we mostly wanted to informally share our experiences w/ it in this initial post. we plan to do some benchmarking coming up here soon tho
Don't use any public benchmarks, every single one is worthless for your own use cases essentially.
Spend a day or two going through your existing chat sessions, and create your own private benchmark with test cases based on real tasks, that you don't share with anyone nor publicly. Make it easy to add/remove new harnesses and model combinations, make it give you a final score, ideally avoid using other LLMs for scoring, then use this to figure out if the new model/harness actually improves things for you.
I've been doing this for some time, and while most new releases show big increases in the benchmarks/evaluations, my own benchmark usually barely moves.
The language composition is interesting. The source is half Shell, a quarter Python, almost a fifth Typescript. Among the rest is 2.3% Rust and 1.3% Swift.
If you wonder what the Rust is for: It is the Ratatui TUI.
Of course the don't have exactly the same scopes but they are in general all about persistent memory and / or continous agent loops. Like I miss those times where only once a week a new js framework was promoted.
Sub Question : IS there a real successful agent product today that uses a library for harness(like langgraph etc)? Building our own worked for us. Works with our components(postgres, events ...) and scales naturally with our system.
I don't know about real successful. Since you mentioned Langchain, you could look at https://www.langchain.com/dcode which is a CLI harness build off Langchain deep agents.
the webpage says: "Your agent keeps thinking between external interactions in a self-guided loop inspired by human inner monologue"
Since supposedly it keeps thinking when you leave it alone, I wonder what happens if you give it a brief prompt like "research the unicode eggplant emoji" and then ignore it for a week, and come back to find that you've spent thousands of dollars for claude to write a 385 page novel about the eggplant emoji.
> Audel designed experiments to spawn recursive shellm sub-runs to work on subproblems. Most of the experiments failed, because shellm has a safety watchdog that kills any command that stays silent for 30 seconds. Audel fought the watchdog for about 40 minutes and mostly stopped using shellm sub-runs. Results from recursive sub-runs of shellm merged back into Audel’s mind 64 times in its first two days and 12 times in the twelve days since. We’ve since revamped the watchdog, and we’ll see if we can convince Audel to give recursion another shot.
This is why "I made it think in a loop" doesn't result in significant improvement in LLM performance. It's not learning. You need RLAIF, STAR, IDPO, etc to retrain the model to learn from its mistakes. And you need a human to review it so it's not compounding mistakes. It's expensive and time-consuming. Doing it wrong leads to bad outcomes. But not doing it leads to no significant improvement.
A lot of harness launches recently but this one actually stood out to me. Just like the Prime Intellect harness that launched recently, this one also leverages RLM (recursive language model). It’s supposed to allow a persistent agent.
The specifics are really interesting. The model itself can change the user input and when you message it, it gets entered into “its stream of consciousness.”
I have a software factory going with Pi, might swap the main orchestrator with Headlong and have it be persistent to drive the other sessions without me.
What do you mean by "software factory"?
Aka vibecoding platform.
A software that's optimized for agentic development, varying levels of review necessary depending on the technical choices of the factory developer. Some review may be automated via agents itself etc.
Ymmv obviously. Its definitely one of the funnest toy projects I've been working on for the last 5-7 months now (such a factory, mine is unreleased however - just for myself as of today)
Looks very interesting. Best suited with Local LLMs with unlimited tokens.
Very fascinating, super interesting engineering. Although i do find it very funny how they just bypass a massive vulnerability, basically zero data isolation (even between good actors, let alone bad ones) with 3 sentences. Only in the llm space you can slap a massive limitation like this in the middle of the article and continue like nothing happened
> Whatever anyone tells Audel becomes part of the single experience that every other conversation draws on. In practice, Audel is bad at keeping secrets. Ask it what it’s been working on with someone else and it will often just tell you, even though we’ve asked it not to. We also haven’t studied what happens when two people give conflicting instructions. For now, we assume anything you tell Audel is shared with everyone on the team.
The part they punt on ("we haven't studied what happens when two people give conflicting instructions") is the interesting part. That's not a memory problem, it's an authz problem. If everyone writes into one shared stream then there's no model of whose instructions bind the agent or who can override whom. It's resolving the conflict that will generate the greatest "learnings" and advance the agent. This basically becomes a tool designed to misbehave rather than a tool that will learn creatively.
We all know how conflicting instructions to AI end - "I'm sorry Dave. I'm afraid I can't do that"
I'm curious, you say "super interesting engineering" but then they say "it will often just tell you, even though we’ve asked it not to" and to me that seems like extremely shit engineering.
Where are the interesting engineering parts at? Seems to be an interesting idea and perhaps design, but to call the implementation/engineering itself bad seems to be an understatement.
The security and the overall engineering were entirely separate items in that comment I think. It was explicitly called out that this is a security problem that you'd really only see treated in this manner in the LLM space. For what it's worth it's effectively unsolvable (AFAIU) short of realizing AGI with an amicable alignment.
What do you mean unsolvable? Don't give the LLM access to stuff it shouldn't, this is like Access Control 101, not sure how anyone can claim that particular problem is unsolvable?
"turn -> FINAL -> schedule wake-up", this is where my excitement has faded unfortunately. Many of us are probably wondering about the same idea: bridging the gap between a reactive agent and my daily workflow or existence. But, this still feels too close to how Claude (or any other agent) runs as a process in the background (always ON), where you can use a custom channel to feed the dialog with external signals like chat, CI/CD events, whatsapp, etc.
Humans aren't scheduling a wake-up to the next thought. Ideally, a sub second agentic loop with no FINAL / wake-up, always "spinning" would get closer. I'm conscious about the waste of resources this would drag with it (because of current architectures), but exciting still.
PS. I love the take on using bash instead of Python (one less abstraction layer!) and using UNIX fundamentals as stepping stone when composing tools as agents are naturally drawn to using it on a box anyways.
Really hard to support these types of projects without subscription options. BYOK API only approach can run huge amounts of money. I budget for $200 Pro subscriptions. Just using Openrouter for a dark factory kanban ticket system using frontier models will run you $50-$150 which is about 1 ticket.
Can we please not normalize telling people to install things by curl piped into bash? I hate this trend. And particularly not for a very new, mostly untested by a wider audience piece of software from a company that few have ever previously heard of.
It even says, quoting from the website: "Headlong is alpha research software."
Yeah that's totally something I want to curl thing.sh | bash , great idea.... Wow.
I understand that people want to get people using their software as quickly as possible and with the absolute minimum of friction, but let's put some more thought into how this could be done in a less sketchy way.
It's like we've regressed to the days when you would download a .exe file from tucows and blindly run/trust it on your windows 98SE PC.
You can always read the script file (orm ore realistically feed it to agent). If they distributed appimage or rpm would that be safer anyhow? Cannot it run malitious code same as the script would run?
One of the big problems is that no one actually does read these scripts. You could say "Oh but it's their own fault, duh" but theres a very legitimate argument to be made users going the path of least resistance and that you shouldn't offload this responsibility on your users.
Regarding appImage or rpm, attackers need to build and package these to inject these, while this curl | bash pipe opens up the possiblity of payloads simply by taking over the domain. And this isn't really that far fetched, just think about the Notepad++ update payload recently. The regular package was unaffected while the domain used for the update was taken over.
Then theres also the argument about normalization. Just like he said, this isn't just something he said, this is a very real argument. You don't want to teach users bad habits. Even if / you / inspect the code you get, not everyone will. And ultimately, we should strive to make the Internet a safer place, if only to get less botnets.
This is the same as terms of service agreements. There's a crowd mentality around it where only one person with enough of a voice needs to read it and scream. It's brittle because if everyone thinks that then nobody checks it but the probability scales with the attention it gets.
Totally agree we should make the internet safer, this is just a clarification that we don't necessarily need everyone to check it. One person who always checks everything and sends this to their LLM and posts a viral post can be a deterrent.
The installer is one threat. You're then running this on your machine which is the next threat.
Are there any objective metrics/ benchmarks that people test harnesses by?
There are just so many now that it's hard to personally test them all or just trust the vibes.
andy here (headlong post author). terminal bench 3 is pretty popular for comparing different harnesses using the same underlying model (it's another laude project actually). artificial analysis has an index. you can look at the model cards of popular model releases- they tend to have the most popular current benchmarks on them. w/ headlong we decided to announce it before we've benchmarked it. we mostly wanted to informally share our experiences w/ it in this initial post. we plan to do some benchmarking coming up here soon tho
Don't use any public benchmarks, every single one is worthless for your own use cases essentially.
Spend a day or two going through your existing chat sessions, and create your own private benchmark with test cases based on real tasks, that you don't share with anyone nor publicly. Make it easy to add/remove new harnesses and model combinations, make it give you a final score, ideally avoid using other LLMs for scoring, then use this to figure out if the new model/harness actually improves things for you.
I've been doing this for some time, and while most new releases show big increases in the benchmarks/evaluations, my own benchmark usually barely moves.
The language composition is interesting. The source is half Shell, a quarter Python, almost a fifth Typescript. Among the rest is 2.3% Rust and 1.3% Swift.
If you wonder what the Rust is for: It is the Ratatui TUI.
It buries the lede. Prime Agent sounds like a very cool project.
I'm just exhausted. So I've today now learned about four new harness:
https://github.com/exoharness/exo/
https://github.com/laude-institute/headlong
https://github.com/microsoft/agent-lightning
and now https://github.com/PrimeIntellect-ai/prime-agent
Of course the don't have exactly the same scopes but they are in general all about persistent memory and / or continous agent loops. Like I miss those times where only once a week a new js framework was promoted.
Used an llm to re-order my stars on github into appropriate lists ( it became a mess after LLM and the different type of projects).
I had all of those projects already in my favorites except Headlong. Arround 4k+ stars.
And probably WIKI ingested 300 of them for recipes and best practices.
Haven't used the WIKI yet though...
It’s written in 10k lines of bash? Why?
Sub Question : IS there a real successful agent product today that uses a library for harness(like langgraph etc)? Building our own worked for us. Works with our components(postgres, events ...) and scales naturally with our system.
I don't know about real successful. Since you mentioned Langchain, you could look at https://www.langchain.com/dcode which is a CLI harness build off Langchain deep agents.
The Googlers must be vague posting about something internal.
But did it produce anything meaningful? If so, show me.
the webpage says: "Your agent keeps thinking between external interactions in a self-guided loop inspired by human inner monologue"
Since supposedly it keeps thinking when you leave it alone, I wonder what happens if you give it a brief prompt like "research the unicode eggplant emoji" and then ignore it for a week, and come back to find that you've spent thousands of dollars for claude to write a 385 page novel about the eggplant emoji.
The page has a nice little easter egg if you click "Dr. K" in the footer.
> Audel designed experiments to spawn recursive shellm sub-runs to work on subproblems. Most of the experiments failed, because shellm has a safety watchdog that kills any command that stays silent for 30 seconds. Audel fought the watchdog for about 40 minutes and mostly stopped using shellm sub-runs. Results from recursive sub-runs of shellm merged back into Audel’s mind 64 times in its first two days and 12 times in the twelve days since. We’ve since revamped the watchdog, and we’ll see if we can convince Audel to give recursion another shot.
This is why "I made it think in a loop" doesn't result in significant improvement in LLM performance. It's not learning. You need RLAIF, STAR, IDPO, etc to retrain the model to learn from its mistakes. And you need a human to review it so it's not compounding mistakes. It's expensive and time-consuming. Doing it wrong leads to bad outcomes. But not doing it leads to no significant improvement.
Harnesses really are the new 'javascript framework' aren't they?
We need "React" moment, to solve imperative O(n^2) state transitions with declarative O(n) target states, and let the harness do the "diffing"
that imaginary line. this crossed it.
Why compress by recency rather than something else?
> "Headlong is a complete agent harness with a core of less than 10K lines of Bash..."
Wow. So, be nice or I'll replace you with a very large shell script?