Nine of 10 companies now use AI to help create software code, but AI is creating code so fast that testing and quality teams cannot keep up, research finds. 

To solve this constraint problem, more companies are turning to AI to help test software, too.

But knowing when and when not to use AI in testing will be critical to both contain costs and ensure software quality and application integrity, which means that applications work as they’re supposed to—all of the time. 

While AI is still a relatively new force in the software development lifecycle, the key consideration about how and how much to test remains around the age-old tradeoff of speed, cost, and quality. What are you willing to give up at a given point in time? AI agents can do work quickly but, in some cases, they’ll cost more than a human if they burn a lot of tokens to get a job done. However, when the volume of software generation simply outpaces what any team can manually test, agents can help meet coverage goals. In some cases, that may even be worth a higher cost.

Knowing the Difference

First of all, it’s important to know the difference between manual, automated and autonomous testing. 

●  Manual is just that: human. 

●  An automated test follows instructions. A human defines a sequence of steps, a login flow, a checkout path, a form submission. Then, the machine executes those steps as written. 

●  An autonomous test is different. Rather than receiving a sequence of steps, autonomous agents receive a destination. They’re trusted to figure out how to get there. They decide what to test, execute, and adapt based on what they find. Agents operating autonomously aren’t executing instructions. They’re making decisions to achieve an outcome.

To date, 87% of teams have automated at least 21% of testing, yet 92% still test manually, our research shows. This underscores that traditional automation has not been able to adapt quickly enough to the pace and velocity of AI-generated code. As such, almost all companies surveyed said they expected to increase investment this year in autonomous testing.

Deciding What’s Best and When

Testing occurs on a spectrum and the right type of testing will depend on each individual company’s culture, workflow, economics, and even products.  But, in general, here are eight questions QA and software experts can ask to help decide whether manual testing, automated or autonomous is the best option.

How common is the application use case? If the use cases are low risk and prone to high repeatability, they’re likely good candidates for autonomous or automated testing. These are things like log in and check out, basically use cases that occur over and over. Agents can do this pretty well because the different paths the agents can take are limited so the risk of something going wrong is low.

How breakable is the environment? When the environment itself is the source of constant breakage–say mobile applications where frequent OS updates are consistently breaking test steps—autonomous agents that adapt independently may be better suited than pre-defined sequences that require manual repair. 

‍How regulated in the application use case? Regulated environments, compliance-heavy applications, and security-sensitive systems need the predictability and auditability that automated testing provides. In those contexts, knowing exactly what was tested, exactly how it was tested, and exactly what the result was is a hard requirement.‍

How buggy or issue-prone is this piece of my application? The more buggy and issue prone, the more you want a human in the loop.

‍How much flexibility is needed? Manual processes allow testers to have more flexibility. With automated testing, the same code and scripts will be leveraged over and over. Automation also requires establishing test cases, programming them with the automation tool and then running the test, which takes a lot of time. If testers have an idea they want to work on quickly, it might be easier - and faster - to do so manually.

‍How long will the project last?  Because automation requires a lot of upfront investment, it might be too costly to deploy for a short-term project. Manual testing would be cheaper and more bottom line beneficial.

Are you testing for usability? If so, humans will be better than machines at judging how users willinteract with a certain feature or how well people will be able to navigate the program. 

What’s the damage if an application fails? The higher the collateral, the more human in the loop you want. Also, consider the importance of domain expertise. If you can give agents good and continually updated instructions, you can trust autonomous testing more. 

Going forward, look for companies to take more of a hybrid approach to testing, utilizing manual, automated and more fully autonomous solutions. The companies that do so most successfully will continually assess the speed, cost, and quality tradeoffs. And, they’ll be able to quickly and reliably adjust as needed.

‍