E2E tests are now quick to write due to LLMs, and are then deterministic AND cheap to run. How would this compare to the token costs of running an agent the whole time for each test? How do you make sure results stay stable regardless of the nondeterministic nature? Do customers still need to create test cases - any way to import from test case management system - based on which they could have already generate e2e tests locally?
Unfortunately from our experience tests don’t scale as well as code. First of all, static tests are very brittle: you rely on selectors, need wait times, and can’t really test a lot of dynamic content (think AI chats/interactions). Then it’s all the infrastructure around it: solving captchas, handling auth, handling email OTP (each of our agents has access to its own inbox) and handling video recording and screenshots.
To ensure stable results we do a lot of harness engineering, where we inject trajectories of previous tests to ensure the stability and also the split into smaller steps helps to prevent context overload and decision fatigue.
Regarding test case management, our customers have used our CLI to migrate their existing test cases from whatever system they were using before.
"Traditional E2E tests are slow to set up and expensive to maintain." I don't really understand this. If I'm already using Opus to write the code, surely it would know best what E2E tests to write to be able to verify its own output? This seems like an unnecessary external step.
Unfortunately from our experience tests don’t scale as well as code. First of all static tests are very brittle, you rely on selectors, need wait times and can’t really test a lot of dynamic content (think AI chats/interactions). Then it’s all the infrastructure around it: solving captchas, handling auth, handling email OTP (each of our agents has access to its own inbox) and handling video recording and screenshots. So with the traditional testing approach you end up mocking a lot of services. I highly recommend you to give it a try!
I would respectfully disagree on this. How i write tests right now I ask claude/codex to create an eval and it just spins up a bg LLM agent worker which verifies the tests in the sandbox/internally.
So i would say that atm in house testing is easier than external testing for us
I will say though, I don't really want to "write tests in natural language", I want something to crawl my app and figure out what's there and what's currently broken and then write its own regression tests.
Have you been able to nail down a loop where your tool can take an open pr, guess the code path and do some testing?
We use cypress heavily for our core flows which has a similar ai prompt thing but it’s not quite ad hoc enough for smaller fixes which is where the bottleneck still comes in for us.
Yes! We spent quite a lot of time on this, and we are currently creating a test plan based on PR changes and sending an agent to verify it. We have some customers who are only using this feature.
Was writing E2E tests ever a problem that needs automation? Also E2E tests need to be updated everytime a new feature is added. TesterArmy sounds great. But config overhead and potential security leaks makes it a no go
Yes a lot. So with the acceleration of AI in the software engineering. Features are being shipped faster but causes regressions. The only way to verify is either you write tests with AI and spend hours reviewing them or you do manual QA. agent-qa aims to solve the later. First your product should work for the end user, later you can write clean test etc.
We found gemini 3 flash to be the best model as of now, when it comes to bang for the buck, GPT 5.5 is also a bit more expensive than 5.4, if we run tests at scale it has to be affordable. Once a newer model that fits into the criteria is released, we will update it!
Our approach is heavily focused on agents, both for executing tests and for managing the platform. We want to provide the best and simplest way to conduct agentic testing, with a strong focus on details. It looks like their platform also requires a sales call.
Does anyone still believe there is a long-term future in traditional UI/UX?
It feels like a lot of attention is still going into landing pages, dashboards, and CRUD apps, while overlooking a bigger shift where fewer people will actually need to interact with those interfaces directly when the same tools can perform the underlying tasks automatically, without much UI at all.
So the bigger question is does UI/UX evolve into something else, or does a large part of it simply disappear?
I might be a bit too early. Recently I started a project and decided to skip all of that and focus to make it more friendly to AI agents and frankly so far it has been great purely from user experience but also what it delivers.
How can it perform tasks automatically? It's not magic, there has to be an UI/UX for interacting with it. Will that UI/UX be more optimized and easier to use is the question. Like would you prefer saying "close window computer" or press alt+f4 or just click on the little cross thing or equivalent. Why are we assuming all AI automagic UI/UX will be better for all tasks?
AI agents can perfectly do a lot of the data entry tasks and build dashboards. You practically need to build none of that when you can ask an AI agent to pull the data and build a chart or provide a file or a paste to insert into a database.
Basically that.
If the app requires a mouse then it should have UI, if not, unless critical, it can be driven by an agent.
Is there a long term future in hand-crafted UI/UX? Maybe not.
Is there a future where we still have traditional UX? Absolutely.
I don't want to write a whole dissertation on this topic, so I'm just going to mention that we tried to build AI voice assistants for a decade, and while LLMs have basically solved understanding, they have not solved the UX portion.
Aside from entertainment/marketing purposes, I think UI will become useless. Why should I interact with UI when an agent can (and will) interact with it?
Because you do not know everything and information (possible inputs, actual outputs) still needs to be presented to you. Because text is not the best way to present everything. Because consistency of presentation makes it easier to absorb. Because everything is social and standardization is useful.
I hooked up our Hubspot to Claude and also found it very valuable, but I actually found the most value there in entity linking, data cleanup and general data processing more than the UI changes.
I still had Claude make me charts because I wanted to understand what was going on though.
I'm not routinely involved in sales, but if I was I feel like there are some breakdowns I would want to see regularly.
I also want to note that my startup sells an infra product. It doesn't need a UI, but people fucking love charts, so we built them the charts and they are happier.
Does it work of mobile native applications or expo apps that have native modules?
Pricing question, the usage on the plans seems low considering in the demo you said that you have 25 tests per pr which would mean you get only 10 PRs per month on the hobby plan?
Yes, it works for any framework. We just get the built native binary and run it in the cloud.
Regarding pricing, the self serve options are currently only for lower usage. We will add more plans further down the line. Currently the most popular one is the startup plan. If you need more usage I’m happy to discuss it on a call!
Curious how they frame testing on real devices: “Real devices on demand: spin up iOS simulators and Android emulators”. So those are not real/physical devices?
We support both web and mobile, which is what a lot of companies prefer, just one agent for both. Also, I'm pretty sure Revyl relies only on vision models, which tend to be slower. We built the platform around a hybrid approach that combines vision and accessibility APIs, which is much faster.
Would love to hear your feedback after you try it out!
We use agents to navigate the app, making real-time decisions based on its state. I prefer to compare it more to a manual QA engineer than to static e2e tests. We spent a lot of time on the harness to make sure the results are reliable. This allows you to assert on dynamic content like AI-generated content. We also support validation of email flows since the agent can read its own email.
> We use agents to navigate the app, making real-time decisions based on its state.
This still leads me to my original question of how though. If you're not using locators are you just passing page contents to the LLM? Or using a multi modal model and say screenshotting? My experience with that has been pretty poor and worse than proper e2e scripts, and is fairly expensive to boot.
Sorry for the insistence haha, just interested because it could be pretty groundbreaking if done well.
Fable (rip) is absurdly good at this, great time to build a product around it, you definitely need the harness, but it feels like it just turned the corner to be able to do really in depth and edge case work.
Do you handle heterogenous environments and network connectivity simulation as well? I am working on a mobile app and occasionally having users just lose a request or two can put the state machine into unusual modes.
I feel like new AI model releases will only allow our agents to do more in-depth testing; the space still has a lot of room to grow. Quality assurance is way more complicated than just clicking around a UI.
Regarding the other question: not yet. For now, we have Chromium, iOS, and Android (latest versions of each), but we are working on adding more. Regarding network connectivity, it's coming soon (I have an open PR).
Hey, I just gave it a try and ran a quick test on booking.com. It took ~3 mins for a basic test. Do you cache the test steps so that future runs are faster and they don't call LLMs for the subsequent runs?
Also your current pricing is $300 for 1K tests which means $0.3 for each test. We tried out playwright mcp and it easily consumes 1M+ tokens for a test with ~20 steps (including image input). So with this pricing are you guys default alive?
Also is there a benchmark which you ran to prove the efficacy of your testing agent? because in the current stage it is a trust me bro kinda thing.
We've been doing quite a lot of context engineering and optimizations to make sure it's not as expensive. The subsequent runs are faster because we cache the trajectory of the agent (not the whole test run yet, as we want to keep the agent in the loop, more like a manual QA engineer, not a test script).
We currently do not have any benchmarks; much of the experience depends on the test plan. We've been mostly focusing on the customer experience not benchmarking.
not sure the pain point you mentioned resonate. with LLMs its very easy to do E2E testing. also I feel uneasy about outsourcing this part with all the security issues these days.
Unfortunately from our experience tests don’t scale as well as code.
First of all, static tests are very brittle: you rely on selectors, need wait times, and can’t really test a lot of dynamic content (think AI chats/interactions). Then it’s all the infrastructure around it: solving captchas, handling auth, handling email OTP (each of our agents has access to its own inbox), spinning up simulators and handling video recording and screenshots.
To ensure stable results we do a lot of harness engineering, where we inject trajectories of previous tests to ensure the stability and also the split into smaller steps helps to prevent context overload and decision fatigue.
Regarding security part, the product can operate solely without any access to the codebase, you can just give us a URL or a mobile app build and we will do the testing.
You have the .com? That's a no-brainer imo. I have a domain for a saas where the .com is squatted so I settled for .ai (and other surrounding TLDs / host permutations) and right out of the gate ran into some issues with firewall vendors in corpo environments.
E2E tests are now quick to write due to LLMs, and are then deterministic AND cheap to run. How would this compare to the token costs of running an agent the whole time for each test? How do you make sure results stay stable regardless of the nondeterministic nature? Do customers still need to create test cases - any way to import from test case management system - based on which they could have already generate e2e tests locally?
Unfortunately from our experience tests don’t scale as well as code. First of all, static tests are very brittle: you rely on selectors, need wait times, and can’t really test a lot of dynamic content (think AI chats/interactions). Then it’s all the infrastructure around it: solving captchas, handling auth, handling email OTP (each of our agents has access to its own inbox) and handling video recording and screenshots.
To ensure stable results we do a lot of harness engineering, where we inject trajectories of previous tests to ensure the stability and also the split into smaller steps helps to prevent context overload and decision fatigue.
Regarding test case management, our customers have used our CLI to migrate their existing test cases from whatever system they were using before.
Why can't you test AI chats?
"Traditional E2E tests are slow to set up and expensive to maintain." I don't really understand this. If I'm already using Opus to write the code, surely it would know best what E2E tests to write to be able to verify its own output? This seems like an unnecessary external step.
Unfortunately from our experience tests don’t scale as well as code. First of all static tests are very brittle, you rely on selectors, need wait times and can’t really test a lot of dynamic content (think AI chats/interactions). Then it’s all the infrastructure around it: solving captchas, handling auth, handling email OTP (each of our agents has access to its own inbox) and handling video recording and screenshots. So with the traditional testing approach you end up mocking a lot of services. I highly recommend you to give it a try!
I would respectfully disagree on this. How i write tests right now I ask claude/codex to create an eval and it just spins up a bg LLM agent worker which verifies the tests in the sandbox/internally.
So i would say that atm in house testing is easier than external testing for us
Has anyone tried to build their own version of this?
It's cool, but I'm not super excited about using some 3rd party SaaS as a critical part of my testing.
I did. Checkout the OSS alternative - https://github.com/vostride/agent-qa
Nice!
I will say though, I don't really want to "write tests in natural language", I want something to crawl my app and figure out what's there and what's currently broken and then write its own regression tests.
Have you been able to nail down a loop where your tool can take an open pr, guess the code path and do some testing?
We use cypress heavily for our core flows which has a similar ai prompt thing but it’s not quite ad hoc enough for smaller fixes which is where the bottleneck still comes in for us.
Yes! We spent quite a lot of time on this, and we are currently creating a test plan based on PR changes and sending an agent to verify it. We have some customers who are only using this feature.
Was writing E2E tests ever a problem that needs automation? Also E2E tests need to be updated everytime a new feature is added. TesterArmy sounds great. But config overhead and potential security leaks makes it a no go
Try the OSS alternative - https://github.com/vostride/agent-qa
Another repo that shouldn't exists Do you know anything about how E2E tests are actually written and run?
Yes a lot. So with the acceleration of AI in the software engineering. Features are being shipped faster but causes regressions. The only way to verify is either you write tests with AI and spend hours reviewing them or you do manual QA. agent-qa aims to solve the later. First your product should work for the end user, later you can write clean test etc.
Some digging FAST_MODEL = "google/gemini-3-flash" (fast mode primary) DEEP_MODEL = "openai/gpt-5.4" (deep mode primary) VISION_CLICK_MODEL= "openai/gpt-5.4" (the visual grounder)
fast: gemini-3-flash, falls back to gpt-5.4, 15-min run timeout, max 2 visual calls/step. deep: gpt-5.4, 15-min timeout, max 3 visual calls/step.
Why such a hard timeout, and why not latest models?
We found gemini 3 flash to be the best model as of now, when it comes to bang for the buck, GPT 5.5 is also a bit more expensive than 5.4, if we run tests at scale it has to be affordable. Once a newer model that fits into the criteria is released, we will update it!
Love your approach to product. It feels like TesterArmy will become the "Vercel for testing". Refreshing stuff!
Thank you! That's the goal
I wonder how does it compare to mobileboost.io, which has been used by some companies like Duolingo?
Our approach is heavily focused on agents, both for executing tests and for managing the platform. We want to provide the best and simplest way to conduct agentic testing, with a strong focus on details. It looks like their platform also requires a sales call.
Great presentation
On a slight tangent, since we are all here...
Does anyone still believe there is a long-term future in traditional UI/UX?
It feels like a lot of attention is still going into landing pages, dashboards, and CRUD apps, while overlooking a bigger shift where fewer people will actually need to interact with those interfaces directly when the same tools can perform the underlying tasks automatically, without much UI at all.
So the bigger question is does UI/UX evolve into something else, or does a large part of it simply disappear?
I might be a bit too early. Recently I started a project and decided to skip all of that and focus to make it more friendly to AI agents and frankly so far it has been great purely from user experience but also what it delivers.
How can it perform tasks automatically? It's not magic, there has to be an UI/UX for interacting with it. Will that UI/UX be more optimized and easier to use is the question. Like would you prefer saying "close window computer" or press alt+f4 or just click on the little cross thing or equivalent. Why are we assuming all AI automagic UI/UX will be better for all tasks?
AI agents can perfectly do a lot of the data entry tasks and build dashboards. You practically need to build none of that when you can ask an AI agent to pull the data and build a chart or provide a file or a paste to insert into a database.
Basically that.
If the app requires a mouse then it should have UI, if not, unless critical, it can be driven by an agent.
That's my point.
Perfectly is a wild word to use here
Is there a long term future in hand-crafted UI/UX? Maybe not.
Is there a future where we still have traditional UX? Absolutely.
I don't want to write a whole dissertation on this topic, so I'm just going to mention that we tried to build AI voice assistants for a decade, and while LLMs have basically solved understanding, they have not solved the UX portion.
Aside from entertainment/marketing purposes, I think UI will become useless. Why should I interact with UI when an agent can (and will) interact with it?
Because you do not know everything and information (possible inputs, actual outputs) still needs to be presented to you. Because text is not the best way to present everything. Because consistency of presentation makes it easier to absorb. Because everything is social and standardization is useful.
We started using a CRM that has no UI - the only interface is AI agent.
Frankly, the best experience I've had in a long time. I just ask the agent what I need and it does it. Populating the CRM is even easier.
Personally, again personal take, I cannot see going back to CRUD applications.
I hooked up our Hubspot to Claude and also found it very valuable, but I actually found the most value there in entity linking, data cleanup and general data processing more than the UI changes.
I still had Claude make me charts because I wanted to understand what was going on though.
I'm not routinely involved in sales, but if I was I feel like there are some breakdowns I would want to see regularly.
I also want to note that my startup sells an infra product. It doesn't need a UI, but people fucking love charts, so we built them the charts and they are happier.
Does it work of mobile native applications or expo apps that have native modules?
Pricing question, the usage on the plans seems low considering in the demo you said that you have 25 tests per pr which would mean you get only 10 PRs per month on the hobby plan?
Yes, it works for any framework. We just get the built native binary and run it in the cloud.
Regarding pricing, the self serve options are currently only for lower usage. We will add more plans further down the line. Currently the most popular one is the startup plan. If you need more usage I’m happy to discuss it on a call!
I'm curious how your mobile testing compares to https://revyl.com
I've been experimenting with Revyl and it's really nice. I think this agent-driven testing is the future.
Curious how they frame testing on real devices: “Real devices on demand: spin up iOS simulators and Android emulators”. So those are not real/physical devices?
Physical devices for AI agents is something we at TestingBot do provide: https://testingbot.com/support/ai/mcp
We support both web and mobile, which is what a lot of companies prefer, just one agent for both. Also, I'm pretty sure Revyl relies only on vision models, which tend to be slower. We built the platform around a hybrid approach that combines vision and accessibility APIs, which is much faster.
Would love to hear your feedback after you try it out!
Try the OSS alternative - https://github.com/vostride/agent-qa
How do you differ from othe companies doing automated testing?
Rainforest QA, for example, has been in this space for a while and also happens to be a YC company.
Seems interesting, but I wonder about this
> Traditional E2E tests are slow to set up and expensive to maintain.
Isn't this just using agents to create e2e tests or is there some better new approach I'm missing?
We use agents to navigate the app, making real-time decisions based on its state. I prefer to compare it more to a manual QA engineer than to static e2e tests. We spent a lot of time on the harness to make sure the results are reliable. This allows you to assert on dynamic content like AI-generated content. We also support validation of email flows since the agent can read its own email.
> We use agents to navigate the app, making real-time decisions based on its state.
This still leads me to my original question of how though. If you're not using locators are you just passing page contents to the LLM? Or using a multi modal model and say screenshotting? My experience with that has been pretty poor and worse than proper e2e scripts, and is fairly expensive to boot.
Sorry for the insistence haha, just interested because it could be pretty groundbreaking if done well.
Fable (rip) is absurdly good at this, great time to build a product around it, you definitely need the harness, but it feels like it just turned the corner to be able to do really in depth and edge case work.
Do you handle heterogenous environments and network connectivity simulation as well? I am working on a mobile app and occasionally having users just lose a request or two can put the state machine into unusual modes.
I feel like new AI model releases will only allow our agents to do more in-depth testing; the space still has a lot of room to grow. Quality assurance is way more complicated than just clicking around a UI.
Regarding the other question: not yet. For now, we have Chromium, iOS, and Android (latest versions of each), but we are working on adding more. Regarding network connectivity, it's coming soon (I have an open PR).
Congrats on the launch!
Will this solution work with services protected by cloudflare turnstile or captchas? Does this involve human in a loop?
Love using tester army to validate PRs against my preview environment. Skips the manual check much of the time and helps me ship more confidently.
Happy to hear that!
What are people using to test mobile apps on self hosted infrastructure nowadays? Is there a solution that's not super heavy and/or slow?
Try the OSS alternative - https://github.com/vostride/agent-qa
Jesus man, stop spamming your project in this thread
Hey, I just gave it a try and ran a quick test on booking.com. It took ~3 mins for a basic test. Do you cache the test steps so that future runs are faster and they don't call LLMs for the subsequent runs?
Also your current pricing is $300 for 1K tests which means $0.3 for each test. We tried out playwright mcp and it easily consumes 1M+ tokens for a test with ~20 steps (including image input). So with this pricing are you guys default alive?
Also is there a benchmark which you ran to prove the efficacy of your testing agent? because in the current stage it is a trust me bro kinda thing.
We've been doing quite a lot of context engineering and optimizations to make sure it's not as expensive. The subsequent runs are faster because we cache the trajectory of the agent (not the whole test run yet, as we want to keep the agent in the loop, more like a manual QA engineer, not a test script).
We currently do not have any benchmarks; much of the experience depends on the test plan. We've been mostly focusing on the customer experience not benchmarking.
Does it support testing on all Apple platforms (macOS, iOS, iPadOS, watchOS, tvOS, and visionOS)?
Currently, only iOS, but we can add iPadOS too!
not sure the pain point you mentioned resonate. with LLMs its very easy to do E2E testing. also I feel uneasy about outsourcing this part with all the security issues these days.
Unfortunately from our experience tests don’t scale as well as code.
First of all, static tests are very brittle: you rely on selectors, need wait times, and can’t really test a lot of dynamic content (think AI chats/interactions). Then it’s all the infrastructure around it: solving captchas, handling auth, handling email OTP (each of our agents has access to its own inbox), spinning up simulators and handling video recording and screenshots.
To ensure stable results we do a lot of harness engineering, where we inject trajectories of previous tests to ensure the stability and also the split into smaller steps helps to prevent context overload and decision fatigue.
Regarding security part, the product can operate solely without any access to the codebase, you can just give us a URL or a mobile app build and we will do the testing.
Goodness I really didn't expect such lazy copy-pasting of responses for a YC company.
Really? You didn't expect low-effort hustler-like behaviour from a YC company? :)
This is a solved problem, there are many that do this. Can't believe YC would fund this in 2026.
Try the OSS alternative - https://github.com/vostride/agent-qa
System requirements please
Congratulations on launch, I’ve been tracking your progress since you’ve been accepted for spring batch.
Always happy to see cool products from Poland! :)
Thank you!
.army?
We are thinking whether to change this.. We also have testerarmy.com/.ai
Change it now to .com or get stuck there for years, suffering anti spam filters, potential renewal problems and more in the meantime.
You have the .com? That's a no-brainer imo. I have a domain for a saas where the .com is squatted so I settled for .ai (and other surrounding TLDs / host permutations) and right out of the gate ran into some issues with firewall vendors in corpo environments.
Yeah, we have all of them. I saw it too where in bigger companies our emails were going straight to spam. Will migrate to it soon
[flagged]
[flagged]
[flagged]
[dead]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
Try the OSS alternative - https://github.com/vostride/agent-qa
[flagged]
[flagged]
[dead]
[dead]
[flagged]