A bug has a clear lifecycle: discover, reproduce, fix, verify. A flaky test has no lifecycle. It passes, then fails, then passes again. Nobody knows if the failure is real. Nobody wants to investigate. The test stays in the suite, burning CI minutes and eroding trust, until someone finally deletes it out of frustration.
The real cost of flaky tests
Most engineering teams measure bugs. Few measure the time lost to flaky tests. The best public numbers come from Google, which has been tracking this at scale for years. In a 2016 write-up on the Google Testing Blog, they reported that about 1.5% of all test runs produced a flaky result, and that almost 16% of their tests showed some level of flakiness. At Google's volume that meant dedicated infrastructure just to re-run and quarantine tests, and engineers who spent a meaningful share of their week deciding whether a red build was real.
Smaller teams do not have that infrastructure. They have the same problem and handle it by hand.
The worst part is not the direct time cost. It is the behavioural change. Once developers learn that test failures are often false positives, they start ignoring all test failures. The test suite becomes a formality. Real bugs slip through because the alert mechanism has been desensitised.
Why E2E tests are especially flaky
Unit tests rarely flake. Integration tests flake occasionally. End-to-end tests flake constantly. The reasons are structural.
Timing dependencies. E2E tests drive a real browser rendering a real application. Page load times vary. Animations finish at different speeds. Network requests resolve in unpredictable order. Every WebDriverWait is a guess about how long "long enough" is.
Shared state. E2E tests often share databases, caches or browser sessions. One test creates data that another test depends on. When the test order changes, through parallel execution or a newly added test, things break.
External services. If your app calls a payment provider, an email service or a third-party API, your tests depend on that service being up and fast. Rate limits, maintenance windows and latency spikes all cause flakes.
Infrastructure variance. CI runners have different CPU, memory and network characteristics from your laptop. A test that passes locally in 2 seconds can time out in CI under load.
The three common fixes that do not work
1. Retry on failure. Most CI systems can auto-retry failed tests. This masks the problem. The test still flakes 20% of the time; you now run it three times and hope one passes. The pipeline takes three times longer, and the system has been trained to hide real failures.
2. Increase timeouts. If a test fails because an element did not appear within 5 seconds, teams raise the timeout to 15. The test stops flaking, and the pipeline slows down. When the app genuinely breaks and the element never appears, the test now waits 15 seconds before reporting what should have been an instant failure.
3. Mark as known flaky. Some frameworks let you tag tests as known-flaky so they stop blocking the pipeline. This is deletion with extra steps. The test no longer protects anything. It just occupies space and CI time.
None of these address the root cause: scripted tests are brittle because they encode exact expectations about UI structure, timing and state.
How agent-based testing avoids flakiness
AI agents do not rely on CSS selectors or fixed timing. They interact with the application the way a person does: look at the screen, identify the relevant elements, and act.
No selectors to break. An agent finds the login button by reading the page, not by looking up #btn-login or .auth-form > button:first-child. When the class name changes or the DOM shifts, the agent still finds the button, because it still says "Log in".
Adaptive waiting. Instead of a hardcoded sleep(5) or WebDriverWait(driver, 10), an agent takes a screenshot, checks whether the page is ready, and proceeds when the content is visible. A slow page gets more time, a fast one gets less. No tuning required.
Independent sessions. Each agent runs in its own isolated browser session with its own state. There is no shared database fixture to conflict with and no test-order dependency.
Goal-oriented. A scripted test asserts on exact intermediate states: after clicking submit, the URL must be /dashboard. An agent evaluates whether the goal was achieved: can the user see their dashboard after logging in? If the app adds a loading spinner, a redirect or an interstitial page, the scripted test breaks and the agent continues.
Agent-based tests still fail. When one does, it means the agent genuinely could not complete the task, which is the signal a test is supposed to give.
What this looks like in practice
Consider a checkout flow: add an item to the cart, fill in shipping details, enter payment info, confirm the order.
A Selenium test for this flow has roughly 40 to 60 assertions, each one checking a specific element on a specific page at a specific moment. If any of them fails because of a slow network, a changed CSS class or a new interstitial modal, the test flakes.
An Aiqaramba journey for the same flow looks like this:
Add the "Pro Plan" to the cart. Complete checkout using the test
credit card (4242 4242 4242 4242). Verify that the order confirmation
page shows the correct plan name and total.
The agent navigates the flow, handles whatever UI it encounters, and reports whether the checkout completed. If the flow genuinely breaks, through a payment validation error, a missing cart item or a broken redirect, the agent reports that with a step-by-step log, screenshots and a recording. If the UI merely changed its layout, the agent adapts.
The result is that test failures become trustworthy again. When a card turns red on the health board, something is actually wrong.
Shifting from test maintenance to test coverage
The deeper problem with flaky tests is opportunity cost. Every hour spent investigating a false positive is an hour not spent covering an untested workflow.
Most B2B SaaS applications have dozens of critical user flows. Teams with Selenium typically cover three to five of them before the maintenance burden makes adding more impractical. The remaining flows are tested manually, slowly and inconsistently, or not at all.
Remove the maintenance burden and you can test all of them. A team that spent 20 hours a week maintaining 50 Selenium tests can describe 50 agent journeys in an afternoon and let them run continuously. The only question left is which flows matter to your users.