In such systems, producing code is the easy part, verification is hard. Verification via test cases just feels like the same agent validating its own assumptions.
Wondering what the actual verification loop looks like once you start taking these systems to production.
In principle I’d agree usually, but I think this specific case, it’s reasonable to highlight that the “experts get value” argument doesn’t really add much to the discourse anymore.
We know this and you just need to look at what’s happening with LLMs in maths to see it’s true. It doesn’t add anything to the verification point however.
I think self-verification can work in some ways. For example, I (well, an LLM) recently wrote some caching optimizations for a particular filesystem intensive workload on a cluster. This is an internal thing, not customer facing, for context.
Deploy to the staging cluster, have the LLM monitor with read access to the logs. Opus is… better than a human at looking at logs and comparing timings and checking for warnings and errors.
If it passes a small load test there, deploy to the live cluster and actively monitor. If it doesn’t fall over under load, is faster, is still correct, not throwing errors it works good enough.
Does much more matter? This produced a fairly large performance improvement. Verification in this case is the process of running it under load and double checking the code isn’t doing anything truly wacky.
User facing apps are harder to validate, sure, but you can easily have an LLM looping looking for errors, log issues, etc.
For home apps, this is more validation than you would have personally done
I've been enforcing TDD with Claude since the start and to be honest it actually works great. It doesn't write bad tests just to fit a bad implementation. And when running against a Pglite in-memory database it can iterate rapidly.
I've had an idea for an automated verification for a while, but too busy to implement it.
For every property of the system, implement a mock that simulates a system that works, and multiple variations with forseeable bugs (models are fairly good at predicting possible bugs). Then, write a unit tests that pass on the working simulation and fail on the buggy simulations.
This way model ensures reasonably well that the unit tests test what they're supposed to. Then, you use those tests as a verification to guide the model to implement the functionality.
This approach relies on the assumption that the model can design a reasonable architecture composed of independently-testable modules. In my experience, frontier models should be able to do that.
Agreed, we had agents writing standard integration/unit/e2e tests but turned out to be a ton of work reviewing just that and hard to wrap one’s head around as a developer.
We ended up building our own test platform to reduce the human bottleneck in the verification loop.
Takes the form of a hermetic test environment that run all our code and infra in a way that matches production. Our tests interact with it from the ”outside”, e.g. by driving a browser, a terminal or making API calls.
We still have coding agents writing those tests but the big difference is that we record the browser and the API calls and can review and verify through a UI without having to do any manual testing. It also serves as a regression test suite with E2E coverage, which is kind of a must to avoid coding agents making assumptions and breaking things.
Not a huge fan of the term ”software factory” but working like this I actually think it’s achievable.
While doing local development where your environment is already setup and dependencies are installed doing something like this seems viable after the initial effort.
But how would this work in a software-factory setup, where coding agents run on fresh VMs or when using something like Claude Code on the web? How do you quickly recreate the required environment and dependencies there?
That’s essentially what we do. The agent writes the browser tests (and the other tests), we just review the replays of them afterwards from the test suite.
I'm building an orchestrator just for the "build" part of building.
The only thing we've truly got left realistically is deciding what to build and making all the product decisions on how it should work.
So that's what I'm trying to enjoy the most now and then when I'm done planning and writing the spec I just delegate to my orchestration engine and walk away while it builds.
It spawns fresh agents to implement, then verifies mechanically (test/lint/build etc.), then fans out multiple reviewers across different providers/models, all those findings go into a triage step with an agent who is given standardized context of previous triage and review rounds and who deduplicates issues and writes guidance for fixing issues fully, then a fix agent gets spawned , then a post-fix verification agent and then back to review until no more issues are found.
For a large feature it can work for a day or more uninterrupted and contrary to just using something like /goal or having an agent orchestrate the whole process, the code stays on track and gets delivered as close to the spec as possible.
All this repetition is necessary if the goal is to be able to generate tons of code and maintain quality while not needing to review everything yourself.
You can test for test vacuity with mutation testing, using tools like Stryker (for JS). In my experience this significantly helps models to write tests that actually test for correctness rather than aligned assumptions.
I think vision models will come more into play for validating things. It’s the most like consciousness, and less like - as you put it an LLM validating its own assumptions.
It’s at least an independent way of analyzing the work (as glyphs and images).
So - apparently it's not fully self-hosted, since I don't see a GPU.
I'm interested in hearing from folks who are hosting their own GPU to run coding models. So far my own results are... not great. Seems like frontier models are needed via the big providers?
My favorite real-world example that I worked on: I created a YouTube documentary series about corruption in a small town in Illinois. This required downloading thousands of hours of government meeting videos from YouTube, transcribing + chunking + embedding, summarizing meeting segments, running a couple passes of validation, then searching for interesting storylines. I also scraped thousands of public documents which exposed campaign finance violations and some truly nefarious stuff going on behind the scenes.
I did 90-95% of the work on this with a local GPU. I still ran Claude and Codex, but they were just writing code to orchestrate API calls against local models, local Whisper, etc. Doing the same with the public cloud would have cost thousands of dollars, instead it cost a few hundred dollars for a frontier lab subscription and tens of dollars for electricity.
I recently got into the DGX Spark and just a month ago, got an AMD Ryzen developer platform. Both are incredibly useful tools for some upcoming projects I'm working on (also related to government corruption), but in my limited experience of trying to run opencode on them, the "good models" are still completely unusable because the system prompt alone requires a minute of thinking.
Do you have a writeup of how you did it, which tools you used? I have a similar project on a much more mundane topic, where I want to build a searchable index of Youtube videos, blog posts, etc. based on their content.
1) I have knowledge DB that I query, spend a lot of time to get "query N models, stream to me all N results, let me pick which one" - only to find out that qwen beat all other contenders (those were picked before I got R9700, so the rest is =<8B parameters)
2) Setup OpenCode to use it and gave it a few tasks:
- first task (new feature to my MCP) took 40 minutes to complete with zero input from me, while it took 20 for sonnet-5 and sonnet-5 kept bugging me. Results are near identical.
- second task (port a specific version of a package to my flake) it got stuck in a hilarious loop where model already been told what hash to use by nix itself, but it wanted to figure our how to get hash another way for some reason.
- third task (another task, but much harder than first one with most of the discovery already done), kept doing discovery and running out of context, went through 3 compactions (256k context is what I can fit on R9700). Room got too hot, so I stopped it.
3) virtual assistant like Hermes but my own: no notes, works great.
I'm pretty sure codding issues are just harness and lack of memory that Claude Code already had. Pretty nice setup, similar to mine but I built my own lightweight PaaS that is highly specific to what I run.
Using a RTX 5090 ($3~4k), I can run Qwen 3.8 27B at ~180 TPS with ninfer [0]. With its thinking maxed out, I can confirm that the quality of output is roughly on par with Opus 4.5~4.6 - that is, this 20GB file really can write software by itself, but the amount of thinking required makes it strictly slower than larger models, even at 180 TPS.
Still, it's largely replaced the cheap tier of the frontiers that I would otherwise be using. It can be run with older GPUs (a 3090 is ~1k), but the time spent thinking will become a fairly noticeable impediment for staying in the flow.
The next step up would be to run DeepSeek V4 Flash 0731 on two DGX Sparks (~$10k), which serve at 60 TPS and sit somewhere around Opus 4.7 level without 3.8-tier thinking.
However, it is worth noting that, if you are buying this hardware just to serve LLMs, it is not cost-effective. It would take over a decade of continuous use to make back the cost of the DGX Spark setup in 0731 tokens. I'm running this setup because I happen to have a 5090, and the two 3090s in my server were cheap enough when amortised over several years.
I have a similar setup to the OP (although I'm not sure if I'd call it a "software factory"), and I utilize a local model for some aspects of the setup. Specifically, I have a single RTX 3090 Ti with 24GB VRAM on my (main) home server, and I've been primarily running Qwen 3.6 35B A3B out of it via Ollama (I just switched to Qwen 3.8 27B, though, and I've also tried other Qwen models as well as Gemma models).
The Qwen models are decent, but they don't come close to the full Claude experience I've come to expect. As such, I only use the local models for specific tasks where it makes sense to do so. Really the setup is that my Claude-powered agents are able to incorporate my local model into work it builds out. The agents can perform inference against the Ollama API as they see fit, and I encourage them to do so for tasks where (a) the low-level capacity of the local models make sense and/or (b) where costs can become a concern.
It seems to work well when it comes into play (like having Claude drive a web browsing session but letting Qwen handle much of the actual browser interactions, image analysis, etc.). Still, Qwen just isn't smart enough (or fast enough on my machine) to handle anything agentic that isn't non-trivial.
Check out some NIXOS communities. People they are doing fun crazy stuff, as NixOS is immutable, thus creates separate sandboxes (building/taking down) per agent, etc.
I don't know most of the stuff there, but at least you can dive there.
I was recently inspired by another article here to start my own. My skills are written and tested, the factory has built the first test project, and I’m setting up the final machine to run it. Here’s my initial post about my motivation and early plans plus some follow-ups, and there are more to come.
People who build software factory are in for a disappointing future. Spoken as someone who uses AI day to day for job and personal projects, I've came across a few super gnarly bugs already, ranging from frontend React apps (yes, believe it or not, frontend is far from solved) to embedded that Claude can't solve. Those bugs are feature breaker. My primary skillset is fullstack/frontend leaning, not embedded. Granted, I might be out of depth in embedded/RTOS but I am very qualified to do frontend. Yet I found bugs in those areas that Claude Opus high can't even solve for hours.
I pointed Claude at those for hours and hours, didn't fucking work, ended up creating more and more slop. Until I stepped in and fixed it in a few lines of code. Thankfully with all my AI assisted work, I took time to understand everything so I was able to debug and fix.
The problem for software factory is, that up to a certain point, you'll encounter some nasty bugs that are really important for your features, but you can't ship it because while the AI deals with the 90%, turns out that last remaining 10% are the one that make or break your project. And you need to deal with that remaining 10%. But how can you deal with the remaining 10% if you don't understand that 90% or majority of it?
Also, don't listen to influencers like Kent Dodds or Uncle Bob or Steve Yegge, who advocated for software factory walk away from the code kinda thing. They are already rich, and they sell teaching content, but most importantly, their time in this industry has passed. They are not in the weeds doing enterprise software development anymore.
Their first and primary concern is trying to be relevant to sell courses. They don't care about you. Yes I used to listen to them (though not often), but now not anymore. The world doesn't get better or worse whether they choose to sip margarita on a beach or spin yet another "so many words to say absolutely nothing" video/blogs.
On the other end of it, people regularly leave out the "why" and "how". Why do software factories not work at scale right now? What are the pieces of building software that agents don't excel at? Is it possible that the 10% that it can't get right, it'll never get right? Is the technology actually improving in the direction that makes this a possible future or we at a local maxima?
I won't admit to knowing the answers, but more discussion needs to happen around the meta of software engineering before anyone can make conclusions about whether we're actually getting there.
My intuition tells me that there's too much nuanced complexity that goes unnoticed for automation to truly take place.
I think software factory is a loaded term that implies it can help you build any software to any complexity which as you correctly noted, is absolutely not true.
However, if you just think of these kinds of LLM driven software factories as having a limit in terms of complexity (ie. some asymtotic / logarithmic increase in cost beyond aome complexity threshold), it still is quite a useful system to have (despite being admittably more niche) because it lets you quickly explore various product ideas, user experiences, and featuresets.
My personal take is that the best approach is to first prototype products as vibe coded slop using a software factory like this to understand the user experience, featureset, and product concepts. Then, once you have a clearer understanding you can properly design out the system and more carefully rebuild it either from scratch with a more costly model and clear architectural details or strip out the core systems and redesign them in large refactors.
I use agents to ship small products and I behave to them like to real human. In real world, would you add the test of the codec to the same engineer who wrote it? No. Same here, just have one agent to write the code (engineer) and another one to test the code (quality assurance) They are alive creatures :)
Is there a name for the infrastructure stack that is designed to be operated by a human instead? Spawn agents each in a separate VM with the repo checked out and the tooling set up, allow them to spawn subagents in the same manner, but pull their changes from their branches (set up as remotes in your own dev environment) and merge into trunk yourself?
I've created a simple project for that, except that by default they open PRs on Forgejo/Github/Gitlab. It's open source and the workflows are config files, so you can configure it to allow you to pull from them instead of creating a PR. Source: https://github.com/smithy-ai/smithy-ai
One way I use it that is similar to what I think you want is to run a local Forgejo instance (just like OP) and have the Dockerised Claude instances work in that, then pull from there to my local git repo, review, test etc and push from my local repo to a remote like Github.
Can anyone that has OpenClaw/Hermex experience tell me what it's like working on this over OpenCode served?
On bigger tasks OpenCode sometimes hangs. I'm not sure if that's on the provider side or on my network side, but i sometimes have to stop a subagent because its done but doesnt clean up. I wanted to try Pi, but now I'm wondering if Hermis is a better fit.
I can relate to this feeling, having encountered it many times in academia. But the inverse would also be true, would you stop reading yet another article that explains for the umpteenth time what library Y does? Not arguing for the author, but there’s certainly something about knowing and writing for your audience.
Neat stuff. I use Hermes with the $20/mo ChatGPT Codex integration, myself, as well as Qwen running locally on a 64GB M1 Max Macbook Pro to help save my ever-quickly shrinking Codex credits. It's been wonderful for stuff like, say, giving Hermes an account on my Home Assistant server (after taking a backup) and having it make a bunch of configurations, or draw conclusions based on historical sensor data (trying to determine if my A/C is undersized by feeding Hermes all the capabilities of my system, size of house, and letting it pull room temp sensor data from HA is one project I did recently), is absolutely fantastic. It also spins me up little webapps for things like a very me-specific RSS reader, a "to do" checklist that references my email (and archives said emails when I mark an item off the list), and various little internal IT tools and report generators.
The Hermes subreddit is a curious place where every second person has quick their $300k a year job and is making a living off of Hermes doing... something? They treat themselves as the CEO of a bunch of agentic employees and have AI generated infographics of their "Stack" (all hail the mighty Stack) and everybody stands in a circle and applauds the most convoluted Hermes setups you've ever seen with not a single word how any of this is supposedly making anybody any money.
Are people developing like this? Are people asking money for something they one-shot instructions to an orchestration agent which delegated to fifteen other agents in Kanban and then spat out something that "works"? Every project I've ever had an LLM do a majority of the work for me has been strictly for my personal use; I'd never let anyone else use this stuff because it doesn't pass the vibe check. When a new frontier model comes out I'll pass the previous frontier model's work past the new model and let it tear it to shreds and see what improvements could be made for shits, giggles, and to waste a week's worth of tokens in the course of thirty minutes, but Reddit is overflowing with seeming non-coders who are passing this 100% organic slop off as sellable product?
In my experience agentic sw development delivers if you are not too picky with the details or if you don't notice what's not done according to your instructions. If you want exactly what you had in mind, the last 10% might last a very long time and cost many tokens. More or less like working with humans.
That’s a great use of Hermes. Had it concluded if your A/C is undersized?
Before this little experiment I generally used it as a very specific search engine for houses, ‘Find me a house in the country but close to amenities with a workshop barn but close enough on the train to Manchester’. It’s great for this!
I’ve never been on the Hermes subreddit I’m gonna go check it out. I agree with the commenter below, it’s fine when I need no record my reps but don’t really care how. It writes tests but I don’t verify. It’s a manual ‘works or it doesn’t’ Thing. I’d never sell the code to anyone.
(soft plug: in case this is really exciting to you but you don't want to own all of this infra for your company, check out our "software factory" factory over at noriagentic.com)
It seems at this stage hermes has the upper hand over OpenClaw in people building multi-agent systems. What about hermes do people like above OpenClaw. I'm building a completely different kind of "software factory" called Abject and I'm curious what people are valuing in Hermes above other systems.
People assemble projects from a collection of buzzwords these days without understanding the underlying technology. Did he need to use tailscale to interact with the box remotely? No. But, it's what a google search will tell you to do.
In such systems, producing code is the easy part, verification is hard. Verification via test cases just feels like the same agent validating its own assumptions.
Wondering what the actual verification loop looks like once you start taking these systems to production.
I just wrote a Twitter thread about how LLMs are like chainsaws [0]
In short:
- they are very powerful
- beginners feel like they are just as good as experts
- but they can cause a LOT of damage if you don't know what you are doing
- experts will get way more done and beginners may cut off a limb or drop a tree on their car
0 - https://x.com/alexpotato/status/2090872063464509912?s=20
People have been making analogies like this since 2023, hardly worth a link to a random twitter thread.
why do people find comments like these necessary? What did you gain by making this comment?
In principle I’d agree usually, but I think this specific case, it’s reasonable to highlight that the “experts get value” argument doesn’t really add much to the discourse anymore. We know this and you just need to look at what’s happening with LLMs in maths to see it’s true. It doesn’t add anything to the verification point however.
I think self-verification can work in some ways. For example, I (well, an LLM) recently wrote some caching optimizations for a particular filesystem intensive workload on a cluster. This is an internal thing, not customer facing, for context.
Deploy to the staging cluster, have the LLM monitor with read access to the logs. Opus is… better than a human at looking at logs and comparing timings and checking for warnings and errors.
If it passes a small load test there, deploy to the live cluster and actively monitor. If it doesn’t fall over under load, is faster, is still correct, not throwing errors it works good enough.
Does much more matter? This produced a fairly large performance improvement. Verification in this case is the process of running it under load and double checking the code isn’t doing anything truly wacky.
User facing apps are harder to validate, sure, but you can easily have an LLM looping looking for errors, log issues, etc.
For home apps, this is more validation than you would have personally done
I've been enforcing TDD with Claude since the start and to be honest it actually works great. It doesn't write bad tests just to fit a bad implementation. And when running against a Pglite in-memory database it can iterate rapidly.
It's really funny. I would read HN headlines about PhD research in the area of https://en.wikipedia.org/wiki/Correctness_(computer_science) and it always seemed really boring
Now it is obvious to me how important this work is
I've had an idea for an automated verification for a while, but too busy to implement it.
For every property of the system, implement a mock that simulates a system that works, and multiple variations with forseeable bugs (models are fairly good at predicting possible bugs). Then, write a unit tests that pass on the working simulation and fail on the buggy simulations.
This way model ensures reasonably well that the unit tests test what they're supposed to. Then, you use those tests as a verification to guide the model to implement the functionality.
This approach relies on the assumption that the model can design a reasonable architecture composed of independently-testable modules. In my experience, frontier models should be able to do that.
Agreed, we had agents writing standard integration/unit/e2e tests but turned out to be a ton of work reviewing just that and hard to wrap one’s head around as a developer.
We ended up building our own test platform to reduce the human bottleneck in the verification loop.
Takes the form of a hermetic test environment that run all our code and infra in a way that matches production. Our tests interact with it from the ”outside”, e.g. by driving a browser, a terminal or making API calls.
We still have coding agents writing those tests but the big difference is that we record the browser and the API calls and can review and verify through a UI without having to do any manual testing. It also serves as a regression test suite with E2E coverage, which is kind of a must to avoid coding agents making assumptions and breaking things.
Not a huge fan of the term ”software factory” but working like this I actually think it’s achievable.
This is a good insight.
While doing local development where your environment is already setup and dependencies are installed doing something like this seems viable after the initial effort.
But how would this work in a software-factory setup, where coding agents run on fresh VMs or when using something like Claude Code on the web? How do you quickly recreate the required environment and dependencies there?
Give pyor.review a try, should help with the human bottleneck
You could let a agent drive the browser verification
That’s essentially what we do. The agent writes the browser tests (and the other tests), we just review the replays of them afterwards from the test suite.
Some people have alternative models review each other's output.
The future might be parallel attempts and selecting the best-of-N solutions.
I'm building an orchestrator just for the "build" part of building.
The only thing we've truly got left realistically is deciding what to build and making all the product decisions on how it should work.
So that's what I'm trying to enjoy the most now and then when I'm done planning and writing the spec I just delegate to my orchestration engine and walk away while it builds.
It spawns fresh agents to implement, then verifies mechanically (test/lint/build etc.), then fans out multiple reviewers across different providers/models, all those findings go into a triage step with an agent who is given standardized context of previous triage and review rounds and who deduplicates issues and writes guidance for fixing issues fully, then a fix agent gets spawned , then a post-fix verification agent and then back to review until no more issues are found.
For a large feature it can work for a day or more uninterrupted and contrary to just using something like /goal or having an agent orchestrate the whole process, the code stays on track and gets delivered as close to the spec as possible.
All this repetition is necessary if the goal is to be able to generate tons of code and maintain quality while not needing to review everything yourself.
You can test for test vacuity with mutation testing, using tools like Stryker (for JS). In my experience this significantly helps models to write tests that actually test for correctness rather than aligned assumptions.
Maybe unpopular prediction:
I think vision models will come more into play for validating things. It’s the most like consciousness, and less like - as you put it an LLM validating its own assumptions.
It’s at least an independent way of analyzing the work (as glyphs and images).
So - apparently it's not fully self-hosted, since I don't see a GPU.
I'm interested in hearing from folks who are hosting their own GPU to run coding models. So far my own results are... not great. Seems like frontier models are needed via the big providers?
My favorite real-world example that I worked on: I created a YouTube documentary series about corruption in a small town in Illinois. This required downloading thousands of hours of government meeting videos from YouTube, transcribing + chunking + embedding, summarizing meeting segments, running a couple passes of validation, then searching for interesting storylines. I also scraped thousands of public documents which exposed campaign finance violations and some truly nefarious stuff going on behind the scenes.
I did 90-95% of the work on this with a local GPU. I still ran Claude and Codex, but they were just writing code to orchestrate API calls against local models, local Whisper, etc. Doing the same with the public cloud would have cost thousands of dollars, instead it cost a few hundred dollars for a frontier lab subscription and tens of dollars for electricity.
I recently got into the DGX Spark and just a month ago, got an AMD Ryzen developer platform. Both are incredibly useful tools for some upcoming projects I'm working on (also related to government corruption), but in my limited experience of trying to run opencode on them, the "good models" are still completely unusable because the system prompt alone requires a minute of thinking.
Sounds amazing! What’s the name of the series if we wanted to find it on YouTube?
I don't know the creator handle or series name, but it's got to be Dixon, IL. lol
Can’t be if they were having to scrub government meetings to put the puzzle together. Unless, of course, they made this before she was caught. :)
Completely coincidentally, I’m typing this from the Aldi at 88 and 26, waiting out a storm.
Do you have a writeup of how you did it, which tools you used? I have a similar project on a much more mundane topic, where I want to build a searchable index of Youtube videos, blog posts, etc. based on their content.
I recently picked up R9700 to run Qwen 3.8-27B.
1) I have knowledge DB that I query, spend a lot of time to get "query N models, stream to me all N results, let me pick which one" - only to find out that qwen beat all other contenders (those were picked before I got R9700, so the rest is =<8B parameters)
2) Setup OpenCode to use it and gave it a few tasks:
I'm pretty sure codding issues are just harness and lack of memory that Claude Code already had. Pretty nice setup, similar to mine but I built my own lightweight PaaS that is highly specific to what I run.Using a RTX 5090 ($3~4k), I can run Qwen 3.8 27B at ~180 TPS with ninfer [0]. With its thinking maxed out, I can confirm that the quality of output is roughly on par with Opus 4.5~4.6 - that is, this 20GB file really can write software by itself, but the amount of thinking required makes it strictly slower than larger models, even at 180 TPS.
Still, it's largely replaced the cheap tier of the frontiers that I would otherwise be using. It can be run with older GPUs (a 3090 is ~1k), but the time spent thinking will become a fairly noticeable impediment for staying in the flow.
The next step up would be to run DeepSeek V4 Flash 0731 on two DGX Sparks (~$10k), which serve at 60 TPS and sit somewhere around Opus 4.7 level without 3.8-tier thinking.
However, it is worth noting that, if you are buying this hardware just to serve LLMs, it is not cost-effective. It would take over a decade of continuous use to make back the cost of the DGX Spark setup in 0731 tokens. I'm running this setup because I happen to have a 5090, and the two 3090s in my server were cheap enough when amortised over several years.
[0]: https://github.com/Neroued/ninfer
I have a similar setup to the OP (although I'm not sure if I'd call it a "software factory"), and I utilize a local model for some aspects of the setup. Specifically, I have a single RTX 3090 Ti with 24GB VRAM on my (main) home server, and I've been primarily running Qwen 3.6 35B A3B out of it via Ollama (I just switched to Qwen 3.8 27B, though, and I've also tried other Qwen models as well as Gemma models).
The Qwen models are decent, but they don't come close to the full Claude experience I've come to expect. As such, I only use the local models for specific tasks where it makes sense to do so. Really the setup is that my Claude-powered agents are able to incorporate my local model into work it builds out. The agents can perform inference against the Ollama API as they see fit, and I encourage them to do so for tasks where (a) the low-level capacity of the local models make sense and/or (b) where costs can become a concern.
It seems to work well when it comes into play (like having Claude drive a web browsing session but letting Qwen handle much of the actual browser interactions, image analysis, etc.). Still, Qwen just isn't smart enough (or fast enough on my machine) to handle anything agentic that isn't non-trivial.
We've put almost back in the title above.
(Submitted title was "A self hosted AI software factory")
Apologies, I was trying to choose between a snappy title and the whole story.
No worries! It's a fuzzy game.
Check out some NIXOS communities. People they are doing fun crazy stuff, as NixOS is immutable, thus creates separate sandboxes (building/taking down) per agent, etc.
I don't know most of the stuff there, but at least you can dive there.
"using Codex for inference" says the article
I was recently inspired by another article here to start my own. My skills are written and tested, the factory has built the first test project, and I’m setting up the final machine to run it. Here’s my initial post about my motivation and early plans plus some follow-ups, and there are more to come.
https://joeldare.com/creating-a-minimal-dark-factory
Just read the intro post, it’s a good read! I’ll read the rest when I get home from work. It’s nice to see others have had the same idea.
People who build software factory are in for a disappointing future. Spoken as someone who uses AI day to day for job and personal projects, I've came across a few super gnarly bugs already, ranging from frontend React apps (yes, believe it or not, frontend is far from solved) to embedded that Claude can't solve. Those bugs are feature breaker. My primary skillset is fullstack/frontend leaning, not embedded. Granted, I might be out of depth in embedded/RTOS but I am very qualified to do frontend. Yet I found bugs in those areas that Claude Opus high can't even solve for hours.
I pointed Claude at those for hours and hours, didn't fucking work, ended up creating more and more slop. Until I stepped in and fixed it in a few lines of code. Thankfully with all my AI assisted work, I took time to understand everything so I was able to debug and fix.
The problem for software factory is, that up to a certain point, you'll encounter some nasty bugs that are really important for your features, but you can't ship it because while the AI deals with the 90%, turns out that last remaining 10% are the one that make or break your project. And you need to deal with that remaining 10%. But how can you deal with the remaining 10% if you don't understand that 90% or majority of it?
Also, don't listen to influencers like Kent Dodds or Uncle Bob or Steve Yegge, who advocated for software factory walk away from the code kinda thing. They are already rich, and they sell teaching content, but most importantly, their time in this industry has passed. They are not in the weeds doing enterprise software development anymore.
Their first and primary concern is trying to be relevant to sell courses. They don't care about you. Yes I used to listen to them (though not often), but now not anymore. The world doesn't get better or worse whether they choose to sip margarita on a beach or spin yet another "so many words to say absolutely nothing" video/blogs.
What most people leave out of the conversation is the element of time. We are early. The technology continues to improve.
Is it perfect now? Absolutely not. Will it continue to get better at a surprising pace? Absolutely yes.
So, just adjust your framing from "we're there" to "we're getting there" and all of your concerns go away.
On the other end of it, people regularly leave out the "why" and "how". Why do software factories not work at scale right now? What are the pieces of building software that agents don't excel at? Is it possible that the 10% that it can't get right, it'll never get right? Is the technology actually improving in the direction that makes this a possible future or we at a local maxima?
I won't admit to knowing the answers, but more discussion needs to happen around the meta of software engineering before anyone can make conclusions about whether we're actually getting there.
My intuition tells me that there's too much nuanced complexity that goes unnoticed for automation to truly take place.
Read "no silver bullet" article.
I like your name, which article is this?
This is a good intro https://cekrem.github.io/posts/there-is-still-no-silver-bull...
I think software factory is a loaded term that implies it can help you build any software to any complexity which as you correctly noted, is absolutely not true.
However, if you just think of these kinds of LLM driven software factories as having a limit in terms of complexity (ie. some asymtotic / logarithmic increase in cost beyond aome complexity threshold), it still is quite a useful system to have (despite being admittably more niche) because it lets you quickly explore various product ideas, user experiences, and featuresets.
My personal take is that the best approach is to first prototype products as vibe coded slop using a software factory like this to understand the user experience, featureset, and product concepts. Then, once you have a clearer understanding you can properly design out the system and more carefully rebuild it either from scratch with a more costly model and clear architectural details or strip out the core systems and redesign them in large refactors.
I use agents to ship small products and I behave to them like to real human. In real world, would you add the test of the codec to the same engineer who wrote it? No. Same here, just have one agent to write the code (engineer) and another one to test the code (quality assurance) They are alive creatures :)
Is there a name for the infrastructure stack that is designed to be operated by a human instead? Spawn agents each in a separate VM with the repo checked out and the tooling set up, allow them to spawn subagents in the same manner, but pull their changes from their branches (set up as remotes in your own dev environment) and merge into trunk yourself?
I've created a simple project for that, except that by default they open PRs on Forgejo/Github/Gitlab. It's open source and the workflows are config files, so you can configure it to allow you to pull from them instead of creating a PR. Source: https://github.com/smithy-ai/smithy-ai
One way I use it that is similar to what I think you want is to run a local Forgejo instance (just like OP) and have the Dockerised Claude instances work in that, then pull from there to my local git repo, review, test etc and push from my local repo to a remote like Github.
Can anyone that has OpenClaw/Hermex experience tell me what it's like working on this over OpenCode served?
On bigger tasks OpenCode sometimes hangs. I'm not sure if that's on the provider side or on my network side, but i sometimes have to stop a subagent because its done but doesnt clean up. I wanted to try Pi, but now I'm wondering if Hermis is a better fit.
Why is it always assumed the reader knows the function of 15 obscure libraries and/or frameworks when posting projects like this?
I don't know most of these tools either, but that is why I find posts like this useful. Don't you end up learning something new in cases like this?
I can relate to this feeling, having encountered it many times in academia. But the inverse would also be true, would you stop reading yet another article that explains for the umpteenth time what library Y does? Not arguing for the author, but there’s certainly something about knowing and writing for your audience.
Maybe it's assumed you're capable of Googling?
I skipped straight to the end to find exactly what I expected the tutorial would cover: security and isolation.
Neat stuff. I use Hermes with the $20/mo ChatGPT Codex integration, myself, as well as Qwen running locally on a 64GB M1 Max Macbook Pro to help save my ever-quickly shrinking Codex credits. It's been wonderful for stuff like, say, giving Hermes an account on my Home Assistant server (after taking a backup) and having it make a bunch of configurations, or draw conclusions based on historical sensor data (trying to determine if my A/C is undersized by feeding Hermes all the capabilities of my system, size of house, and letting it pull room temp sensor data from HA is one project I did recently), is absolutely fantastic. It also spins me up little webapps for things like a very me-specific RSS reader, a "to do" checklist that references my email (and archives said emails when I mark an item off the list), and various little internal IT tools and report generators.
The Hermes subreddit is a curious place where every second person has quick their $300k a year job and is making a living off of Hermes doing... something? They treat themselves as the CEO of a bunch of agentic employees and have AI generated infographics of their "Stack" (all hail the mighty Stack) and everybody stands in a circle and applauds the most convoluted Hermes setups you've ever seen with not a single word how any of this is supposedly making anybody any money.
Are people developing like this? Are people asking money for something they one-shot instructions to an orchestration agent which delegated to fifteen other agents in Kanban and then spat out something that "works"? Every project I've ever had an LLM do a majority of the work for me has been strictly for my personal use; I'd never let anyone else use this stuff because it doesn't pass the vibe check. When a new frontier model comes out I'll pass the previous frontier model's work past the new model and let it tear it to shreds and see what improvements could be made for shits, giggles, and to waste a week's worth of tokens in the course of thirty minutes, but Reddit is overflowing with seeming non-coders who are passing this 100% organic slop off as sellable product?
I need fewer morals.
In my experience agentic sw development delivers if you are not too picky with the details or if you don't notice what's not done according to your instructions. If you want exactly what you had in mind, the last 10% might last a very long time and cost many tokens. More or less like working with humans.
That’s a great use of Hermes. Had it concluded if your A/C is undersized?
Before this little experiment I generally used it as a very specific search engine for houses, ‘Find me a house in the country but close to amenities with a workshop barn but close enough on the train to Manchester’. It’s great for this!
I’ve never been on the Hermes subreddit I’m gonna go check it out. I agree with the commenter below, it’s fine when I need no record my reps but don’t really care how. It writes tests but I don’t verify. It’s a manual ‘works or it doesn’t’ Thing. I’d never sell the code to anyone.
(soft plug: in case this is really exciting to you but you don't want to own all of this infra for your company, check out our "software factory" factory over at noriagentic.com)
Maybe I missed it but what AI model did you use for this?
> All on my home server, without another cloud infrastructure bill. The only ongoing cost specific to this experiment is a £20 Codex sub.
Quite a lot of people are automating the SDLC now and are looking at dark factory solutions.
It seems at this stage hermes has the upper hand over OpenClaw in people building multi-agent systems. What about hermes do people like above OpenClaw. I'm building a completely different kind of "software factory" called Abject and I'm curious what people are valuing in Hermes above other systems.
People assemble projects from a collection of buzzwords these days without understanding the underlying technology. Did he need to use tailscale to interact with the box remotely? No. But, it's what a google search will tell you to do.
Please share a link if its on the internet.