A flaky test is a test that fails non-deterministically — it passes sometimes and fails other times when run against identical code. Flaky tests are the single largest source of lost confidence in automated test suites. When a test fails on a CI run, the first question engineers ask is whether the failure is real or flaky. If the answer is probably flaky, the test result is disregarded, the run is re-triggered, and the failure provides no useful signal. This pattern degrades the quality gate that automated testing is supposed to provide: a CI pipeline where engineers routinely dismiss failures as flaky is functionally no better than one with no tests at all.

In 2026, flaky tests are more prevalent than in previous years because test suites are larger, CI runs are more frequent, and the applications under test are more complex — more asynchronous behavior, more microservice dependencies, more dynamic DOM elements. The techniques that worked to control flakiness in simpler applications and smaller suites — fixed sleep statements, test serialization, stable test data — are insufficient at current scale. This guide covers the four root cause categories responsible for the majority of flaky tests, how to detect flaky tests systematically before they accumulate in your suite, how to remediate them by root cause, and which tools have emerged in 2026 to help manage the problem at scale. For teams looking to establish a solid testing foundation, Astaqc's test automation services cover both initial suite design patterns that minimize flakiness and remediation of existing flaky suites.

What Makes Tests Flaky: The Four Root Cause Categories

Flakiness has four root cause categories. Understanding which category a flaky test belongs to is the prerequisite for effective remediation. Tests in different categories require different fixes, and applying the wrong fix wastes effort while leaving the underlying cause in place.

Timing dependencies are the most common root cause. A test that contains a fixed sleep — await page.waitForTimeout(2000) — is betting that the application will reach the expected state within two seconds. If the application is slow due to CI agent load, database query latency, or a network call to an external service, the test fails when the bet is wrong. Dynamic content, animations, and asynchronous state updates make timing-dependent tests brittle. The correct fix is explicit waiting: replacing fixed sleeps with conditions that wait for a specific element to appear, a network request to complete, or a DOM mutation to settle.

Test order coupling occurs when tests share mutable state and depend implicitly on execution order. If test B expects a user account to exist because test A created it, test B fails when run in isolation, when run before test A, or when the order is changed. Database state, browser cookies, local storage, and in-memory caches are all common sources of coupling. Properly isolated tests create and clean up their own test data in before/after hooks rather than depending on side effects from other tests.

Environment variance covers non-determinism introduced by differences in the execution environment rather than the test or application code. Different operating systems render fonts and calculate element positions differently. Different browser versions handle JavaScript timers, network events, and CSS animations with small behavioral differences. Different CI agent hardware produces different execution timings. Environment variance is the most difficult root cause to fix completely because it requires environment standardization rather than test code changes.

External service dependencies create flakiness through variable latency and occasional downtime of services outside the test suite's control. Tests that make real HTTP calls to staging APIs, third-party services, or development databases fail when those services respond slowly or are temporarily unavailable. The standard remediation is service isolation: using mocks or test doubles for external dependencies in unit and integration tests, and using stable, dedicated test environments for end-to-end tests that require real service calls.

Most flaky test suites contain examples of all four categories. A useful diagnostic approach is to categorize a random sample of recently flaky tests by root cause and measure the proportion in each category. This tells you where to focus remediation effort for the largest impact. For teams that need to address a large backlog of flaky tests systematically, Astaqc's software testing services provide structured flaky test triage and remediation engagements. The complete software testing guide provides context on how flakiness affects overall test suite health metrics.

Detecting Flaky Tests: How to Find Them Systematically

The naive approach to flaky test detection is waiting for engineers to notice that a test fails intermittently and report it. This is slow, misses tests that are flaky on CI but not locally, and creates an incentive to simply skip or delete problematic tests rather than fix them. Systematic detection requires running tests multiple times and tracking the pass/fail rate over time.

Re-run detection is the most reliable method. Each test in the suite is run N times (typically 3 to 10) against the same code and environment. A test that fails at least once but passes at least once in the N runs is flagged as flaky. The threshold for flagging can be tuned: flagging on a single failure in 3 runs catches all flakiness but generates more false positives; requiring 2 failures in 10 runs is more conservative but catches genuine flakiness reliably. Most CI platforms support test re-run via framework configuration: --retries 3 in Playwright, --rerun-failing in pytest, or @flaky annotations in JUnit. The challenge is that re-running the entire suite N times multiplies CI costs proportionally.

Historical failure pattern analysis uses CI run history rather than intentional re-runs. By querying the last 30 days of CI test results and identifying tests that fail on some runs but pass on others for the same commit, engineering teams can build a flakiness rate for each test without running extra CI cycles. This requires structured test result storage — most CI platforms can be configured to store test outcomes in a queryable format. Tests with a flakiness rate above a defined threshold (for example, more than 5% of runs fail) are candidates for quarantine and remediation.

Quarantine patterns separate flaky tests from the blocking CI quality gate without deleting them. A quarantined test still runs on CI but its failure does not block the pipeline. Results for quarantined tests are tracked in a separate dashboard. Engineers are expected to remediate quarantined tests within a defined period (typically two weeks) before they re-enter the blocking suite. Without a quarantine process, flaky tests accumulate in the blocking suite and erode trust in CI until someone disables them entirely — which silently removes test coverage.

For teams using test automation infrastructure, detecting flakiness early requires building the historical tracking and re-run infrastructure before flaky tests accumulate. The manual vs. automated testing guide covers how flaky automated tests affect the balance between automated and manual coverage, and performance testing services are relevant when flakiness is traced to timing dependencies caused by performance variability in the application under test.

Remediation Strategies by Root Cause

Remediation depends on root cause. Applying a retry-based fix to a test order coupling problem does not fix the coupling — it masks the failure by re-running until the shared state happens to be in the right condition. Effective remediation identifies the root cause category first and applies the appropriate fix.

Remediating timing dependencies requires replacing fixed waits with explicit wait conditions. The pattern is: identify the condition that must be true for the test step to succeed, express that condition in the wait API, and replace the fixed sleep with the conditional wait. In Playwright: await page.waitForSelector('[data-testid="submit-button"]', { state: 'visible' }) instead of await page.waitForTimeout(3000). In Selenium: WebDriverWait with an expected condition instead of Thread.sleep(). For animation-related timing, waiting for the CSS transition to complete via a polling wait on an element's computed style is more reliable than waiting a fixed number of milliseconds.

Remediating test order coupling requires making each test independent. This means moving state setup into beforeEach hooks and teardown into afterEach hooks. Tests should create their own test users, test records, and test data, and clean them up after they complete. For tests that depend on a database, using a test data factory pattern — where each test creates a minimal, unique dataset via an API or database client — ensures isolation. The signal that coupling is fixed is that the test passes when run in any order and when run in complete isolation.

Remediating environment variance requires standardizing the execution environment. For browser-based tests, pinning the browser version in CI eliminates browser-version-driven flakiness. Running tests in containers ensures consistent OS-level behavior across CI agents and developer machines. For visual regression tests, environment variance is handled by setting a per-environment tolerance threshold in the SSIM comparison, or by using baseline images captured in the canonical CI environment.

Remediating external service dependencies requires service isolation. For unit and integration tests, replace real external HTTP calls with mocks or contract test doubles. For end-to-end tests that require real service behavior, use a dedicated, stable test environment that is not shared with manual QA or development traffic. Where external services have known availability SLAs below 99.9%, tests that depend on them should be explicitly tagged and excluded from blocking pipelines during known downtime windows.

For teams addressing a large flaky test backlog, prioritizing by the estimated impact of each fix is more effective than working through tests chronologically. A single fix that addresses a shared beforeEach hook used by thirty tests is higher priority than fixing thirty individual timing waits. Astaqc's QA team service provides dedicated resource for systematic flaky test triage, including root cause analysis and fix prioritization for large existing suites. See the QA outsourcing guide for how to structure an engagement that covers flaky test remediation within a broader test quality programme.

Flaky Test Management Tools in 2026

A category of tools has developed specifically for flaky test management: detecting flakiness in CI history, surfacing flakiness rates per test, quarantining known flaky tests, and tracking remediation progress. In 2026, these tools have matured from research-grade internal tools at large tech companies to accessible SaaS products that integrate with standard CI platforms.

BuildPulse connects to CI pipeline history and identifies tests that fail intermittently. It tracks flakiness rates per test over time, groups flaky tests by likely root cause based on failure patterns, and integrates with GitHub pull requests to annotate PRs with flakiness warnings for tests that the PR touches. BuildPulse does not fix flaky tests but reduces the discovery time from weeks to days and provides the historical data needed to prioritize remediation.

Trunk Flaky Tests (part of the Trunk CI platform) provides a quarantine workflow integrated with GitHub and GitLab. Tests detected as flaky via re-run analysis are automatically moved to a quarantine group. Engineers receive notifications and a time-boxed window to remediate. Tests that are not remediated within the window are escalated. This automates the quarantine workflow that most teams try to implement manually but abandon due to administrative overhead.

Allure TestOps is a test management and analytics platform that tracks test history across runs and surfaces flakiness rate, mean time to failure, and failure frequency for each test in the suite. It integrates with Allure Report and provides both a historical view per test and aggregate suite health metrics. For teams that need test management capabilities alongside flakiness tracking, Allure TestOps covers both.

Playwright Test's built-in retry and soft assertion features do not detect flakiness systematically but are practical first-line defenses. The --retries flag re-runs failing tests automatically, reducing the rate of flaky test failures that block pipelines. Soft assertions allow a test to continue collecting assertion data after a single assertion fails, providing more context about the failure before the test is marked as failed.

No tool in this category automatically fixes flaky tests. The tools accelerate discovery and make the management overhead of quarantine and tracking practical at scale, but the root cause analysis and code change remain engineering work. For teams with a significant investment in test automation and a growing flaky test count, the value of these tools is in the data they provide — specifically, flakiness rates per test and trends over time — which supports prioritization and accountability in remediation efforts. For guidance on selecting and integrating flaky test tooling as part of a broader automation strategy, Astaqc's test automation services provide advisory and implementation support. The software testing cost guide covers how unmanaged flakiness affects overall QA programme cost through CI retries, engineer time, and reduced pipeline throughput.

Frequently Asked Questions

What is the difference between a flaky test and a failing test?

A failing test fails consistently because the code under test has a bug or because the test expectation is incorrect. A flaky test fails non-deterministically — it passes on some runs and fails on others with no code changes between runs. The distinction matters because a failing test requires a code or test fix, while a flaky test requires a test stability fix or root cause investigation. Conflating the two leads to either ignoring real failures (dismissing them as flaky) or wasting time investigating genuine flakiness as if it were a code defect.

Should flaky tests be deleted if they cannot be fixed quickly?

Deleting a flaky test removes coverage silently. The preferable approach is quarantine: the test continues to run but its failure does not block the pipeline, and it is tracked for remediation. Deletion should be a last resort, used only when the test covers a scenario that is also covered by other tests, or when the coverage provided by the flaky test has been deliberately replaced. A quarantine process with a defined remediation deadline — for example, two weeks — prevents tests from remaining quarantined indefinitely while maintaining visibility into what coverage is at risk.

How many retries are appropriate for a flaky test before it is considered a genuine failure?

One to three retries is the standard range for CI pipelines. A single retry (running the test twice on failure) catches most occasional flakiness without masking genuine failures — a real bug will fail consistently across retries. Three retries is appropriate for tests known to be highly sensitive to timing in a specific environment. More than three retries typically indicates that the test needs remediation rather than more retries, and should be treated as a temporary measure during stabilization, not a permanent fix.

Can AI tools reliably classify the root cause of a flaky test automatically?

In 2026, AI-assisted root cause classification can suggest likely categories (timing, coupling, environment, external service) based on failure patterns, log analysis, and test code inspection. Tools like BuildPulse use pattern matching on failure messages to suggest categories. LLM-based tools can analyze test code and suggest specific waits to replace. However, accurate root cause diagnosis still requires an engineer to verify the suggestion against the actual test behavior and application code. AI classification accelerates triage but does not replace it.

What is the flakiness rate threshold at which a test should be quarantined?

There is no universal threshold, but a flakiness rate above 5% — meaning the test fails on more than 1 in 20 runs without code changes — is a reasonable trigger for quarantine in most CI environments. At 5% flakiness on a suite of 100 tests, approximately 5 tests will fail on any given CI run due to flakiness alone, which is enough noise to erode confidence in the quality gate. Some teams set a more aggressive threshold of 1% for blocking tests. The appropriate threshold depends on CI run frequency and how much false-negative noise the team finds acceptable.

How does test environment containerization reduce flakiness?

Containerization reduces environment variance flakiness by ensuring that every CI run executes in an identical environment: the same OS version, the same library versions, the same filesystem structure, and the same network configuration. Tests that fail on one CI agent but pass on another due to OS-level differences are made consistent by running all agents from the same container image. Containerization does not eliminate timing-dependent or order-coupled flakiness, but it removes one of the four root cause categories from the problem space entirely. For teams using Docker-based CI, building a standardized test container image and using it across all agents is one of the highest-leverage investments for reducing ambient flakiness. See Astaqc's test automation services for support with test environment standardization.