An e-commerce platform with 200,000 monthly active users and a catalog of over 50,000 products had a problem threatening its growth: production bugs. These were not catastrophic errors that took down the service. They were subtle defects that eroded user trust: a search filter returning incorrect results, a discount calculation failing on specific combinations, a checkout flow breaking on certain mobile devices.
Each production bug generated support tickets, returns, and in the worst cases, customers who left without warning. The development team knew the problem existed, but they were trapped in a reactive cycle: they spent so much time fighting fires that they had no capacity to prevent the next ones.
The Challenge
Insufficient Manual Testing
The QA team consisted of two people performing manual tests before each release. With a product that included catalog, search, cart, checkout, payments, order management, discount system, and admin panel, it was physically impossible to test all relevant combinations on every release.
Prioritization was inevitable: main flows were tested, and edge scenarios were left unverified. But the bugs that reached production were almost always in those edge scenarios nobody had time to test.
22% Automated Test Coverage
The product had automated tests, but they covered only 22% of the code. Most were unit tests written during initial development, many of which were outdated or broken. There were no integration tests or automated end-to-end tests. The team had attempted to increase coverage multiple times but always abandoned the effort: writing tests for legacy code without documentation is tedious work that competes for time with the features customers are requesting.
Long Release Cycles
Fear of bugs stretched release cycles. Each deployment required two days of manual testing, which limited releases to one every two weeks. In e-commerce, where promotions, seasonal campaigns, and competitive responses demand agility, releasing every two weeks was a competitive disadvantage.
Business Impact
The bugs were not just a technical problem. Each bug in the checkout flow represented lost transactions. The analytics team estimated that production errors cost between 15,000 and 25,000 euros monthly in lost sales and support costs. That number convinced management to invest in a solution.
The Solution
We designed and implemented an AI-powered testing pipeline that integrated into the existing development workflow without requiring the team to radically change how they worked.
Phase 1: Massive Test Generation with AI Agents (Weeks 1-4)
The first objective was to increase test coverage from 22% to 70% in four weeks. With a human team, that would have taken months. With AI agents, it was feasible.
We used Claude Code to analyze each code module and generate three types of automated tests:
- Unit tests: for individual functions and methods, covering both normal cases and edge cases.
- Integration tests: to verify that modules communicate correctly with each other. The discount system with the cart, the cart with checkout, checkout with the payment processor.
- End-to-end tests: for critical user flows. Product search, add to cart, apply discount, complete purchase, receive confirmation.
The agent did not generate tests blindly. It analyzed the code, identified execution branches, detected the points where bugs were most likely to appear (decimal operations, null state handling, parameter combinations), and prioritized test generation in those areas.
Each agent-generated test went through human review before integration into the suite. Not all were perfect: approximately 15% needed adjustments. But the remaining 85% were functional and correct from the first generation.
Phase 2: Pull Request Review Agent (Weeks 3-5)
We configured an automated agent that reviewed every pull request before it reached a human reviewer. The agent performed four verifications:
- Impact analysis: identified which parts of the system were affected by the changes and verified that tests existed for those areas.
- Known error pattern detection: searched for patterns that had historically caused bugs in the project (decimal operations without rounding, date comparisons without timezone, incorrect handling of empty arrays).
- Coverage verification: checked that new changes came with tests and that coverage did not decrease.
- Regression validation: ran existing tests related to the modified area and reported any failures.
Phase 3: Automated Visual Testing (Weeks 5-7)
For visual bugs, those not detected by logic tests but requiring actually seeing the interface, we implemented a visual testing system with automated screenshots.
Before each release, an agent navigated the platform’s main flows in five different configurations (desktop, tablet, mobile, and two intermediate resolutions), took screenshots, and compared them against reference captures. Any visual difference above a configurable threshold generated an alert for human review.
This captured a category of bugs that was previously invisible to automated testing: buttons overlapping at certain resolutions, text getting cut off, product images not loading in specific contexts.
Phase 4: CI/CD Integration (Weeks 6-8)
The entire testing infrastructure was integrated into the existing CI/CD pipeline. The final flow for each pull request was:
- The developer creates the pull request.
- The review agent analyzes changes automatically.
- Unit, integration, and end-to-end tests run in parallel.
- Visual tests run across all five configurations.
- If everything passes, the pull request is marked as ready for human review.
- The human reviewer focuses on business logic and design decisions, knowing that technical quality is already verified.
Results
After 8 weeks of implementation:
- Production bugs: from 12 per month to 0. The first full month with the pipeline active registered zero production bugs. The following two months maintained the trend.
- Test coverage: from 22% to 85%. The agent generated over 2,400 automated tests in the first four weeks. After human review, 2,040 were incorporated into the suite.
- Release cycles: 60% faster. From one release every two weeks to three releases per week. Confidence in the testing pipeline eliminated the need for extensive manual testing.
- Manual QA time: reduced by 80%. The two QA team members went from dedicating 100% of their time to manual testing to dedicating 20%. The remaining 80% was spent designing more advanced testing strategies and supervising the agents.
- Recovered sales: estimated 18,000 euros monthly in transactions that were previously lost due to bugs in the checkout flow.
Lessons Learned
AI Agents Are Better at Generating Tests Than Writing Production Code
Test generation is one of the applications where AI agents shine most consistently. Tests have a clear pattern (given X, when Y, then Z), and the agent can generate exhaustive variations that a human would not have the patience to write. It is a use case where the agent’s thoroughness complements the human’s creativity.
22% Coverage Is Not a Starting Point, It Is Technical Debt
Many teams become accustomed to low coverage and normalize it. But 22% means 78% of the code has no safety net. When a change is introduced, there is no automated way to know if something broke. Raising coverage to 85% was not a luxury: it was the minimum condition for deploying with confidence.
Visual Testing Captures an Invisible Category of Bugs
Logic tests do not see what the user sees. A test can confirm the product price is correct while that price is displayed overlapping with another element on the user’s screen. Automated visual testing closed that gap.
The Investment in Testing Pays Back in Weeks, Not Months
Management expected a 6-month ROI. They achieved it in 6 weeks. Between recovered sales, reduced support tickets, and freed QA team time, the investment paid for itself before the pipeline was fully stabilized.
If your development team ships bugs to production more often than acceptable, or if your test coverage is a source of stress, we can help. Request a free audit and we will analyze how to implement an AI-powered testing pipeline adapted to your product.