Qodo’s fully autonomous agent tackles the complexities of regression testing

Code is continuously evolving in the software development process, requiring ongoing testing for quality and maintainability. This is the root of regression testing, in which existing tests are re-run to ensure that modified code continues to function as intended.

However, regression testing can be time-consuming and complex, and may often be neglected in lieu of other priorities. 

Qodo (formerly CodiumAI) says it can ease headaches around the process with the release today of its new fully autonomous AI regression testing agent, Qodo Cover. Its agent creates validation suites to ensure that software applications are, essentially, behaving. The 2-and-a-half-year-old startup announced its new tool at AWS re:Invent, where it also pitched as a finalist in an AWS Unicorn Tank competition.

“We’re moving toward a place where AI doesn’t just write code — it helps tackle the majority of developers’ workload by proving that code functions correctly,” Qodo CEO Itamar Friedman told VentureBeat.

Supporting the next big leap in software development

Qodo explained earlier this year at VentureBeat Transform that it is approaching AI agents in an incremental fashion — taking on competitors such as Devin that offer more end-to-end suites. The Israeli startup offers numerous small agents that handle specific tasks within software development workflows. 

Qodo Cover is the newest of these. The fully autonomous agent analyzes source code and performs regression tests to validate it as it changes throughout its lifecycle. The platform ensures that each test runs successfully, passes and increases the amount of code it covers — and only keeps those that meet all three criteria. 

It’s estimated that enterprise developers spend only an hour a day actually writing code; the rest of their time goes to crucial tasks such as testing and review, Friedman pointed out. However, “many companies are rushing to generate code with AI, focusing on that one hour while ignoring the rest of the equation.”

Traditional testing approaches simply don’t scale, he noted, which can stall the next leap in software development where AI can reliably generate 80% or more of high-quality code. “Just like how hardware verification revolutionized chip manufacturing a few decades ago, we’re now at a similar inflection point with software. When 25% or more of code is AI-generated, we need new paradigms to ensure reliability.”

Hugging Face-approved

Demonstrating its ability to generate production-quality tests, a pull request generated fully autonomously by Qodo Cover was recently accepted into Hugging Face’s PyTorch Image Models repository. Pull requests are a means of quality control in software development, allowing collaborators to propose and review changes before they are integrated into a codebase. This can keep bad code and bugs out of the main codebase to ensure quality and consistency.

The acceptance by Hugging Face validates Qodo’s offering and exposes it to more than 40,000 projects in the popular machine learning (ML) repository. 

“Getting a contribution accepted into a major open-source project is a signal that AI agents are beginning to operate at the level of professional developers when it comes to understanding complex codebases and maintaining high standards for quality,” said Friedman. “It’s a peek into how software development will evolve.”

Qodo Cover is built on an open-source project that Qodo launched in May. That project was based on TestGen-LLM, a tool developed by Meta researchers to fully automate test coverage. To overcome challenges with large language model (LLM)-generated tests, the researchers set out to answer specific questions: 

• Does the test compile and run properly?
• Does the test increase code coverage?

Once those questions are validated, it’s important to perform a manual investigation, Friedman writes in a blog post. This involves asking: 

• How well is the test written?
• How much value does it actually add?
• Does it meet any additional requirements?

Users provide several inputs to Qodo Cover, including: 

• The source file for code to be tested
• Existing test suite
• Coverage report
• Command for building and running suites
• Code coverage targets and maximum number of iterations to run
• Additional context and prompting options

Qodo Cover then generates more tests in the same style, validates them using the runtime environment (i.e., do they build and pass?), reviews metrics such as increased code coverage and updates existing test suites and coverage reports. This is repeated until code either reaches the coverage threshold or the maximum number of iterations. 

Giving devs full control, providing progress reports

Qodo’s agent can be deployed as a comprehensive tool that analyzes full repositories to identify gaps and irregularities and extend test suites. Or, it can be established as a GitHub action that creates pull requests automatically to suggest tests for newly-changed code. Qodo emphasizes that developers maintain full control and have the ability to review and selectively accept tests. Each pull request also includes detailed coverage progress reports. 

Qodo Cover supports all popular AI models, including GPT-4o and Claude 3.5 Sonnet. The company says it delivers high-quality results across more than a dozen programming languages including JavaScript, TypeScript, C++, C#, Ruby, Go and Rust. It is intended to integrate with Qodo Merge, which reviews and handles pull requests, and coding tool Qodo Gen.