A Green Pipeline That Lies to You Costs Hours
A red build you cannot explain is not a failing test. It is a flaky test. It passed three times yesterday and failed twice this morning on identical code. Now your release is blocked, your CI queue is backed up, and you are re-running jobs hoping the problem resolves itself.
That is the actual cost: decision paralysis, manual re-runs, and a creeping distrust of your entire test suite. Once a team stops trusting green, green stops meaning anything.
Symptoms You Will Recognize
Before you can fix anything, you need to name what you are seeing.
- A test fails on
mainbut passes when you run it locally. - Re-running CI without any code change produces a pass.
- The same test passes in isolation and fails in the full suite.
- Failures cluster around time-sensitive assertions or async operations.
- Test output differs between parallel and serial execution.
If two or more of these match your situation, the root cause is almost certainly one of three things: shared mutable state, timing assumptions, or external I/O that is not controlled.
One Root Cause Underneath All of Them
Flaky tests are determinism failures. The test result depends on something the test does not control.
That something is usually global state (a database row left over from a prior test), a wall-clock dependency (sleep(1) or Date.now()), or a live network call that occasionally times out. The fix is always the same shape: take control of the thing the test was leaving to chance.
This is the same discipline that makes a 400-line controller testable in the first place - see Refactor Story: A 400-Line Controller to Something Testable for a concrete walkthrough of isolating dependencies before you test them.
The Playbook, in Order
1. Quarantine before you debug
Tag every suspected flaky test with a skip annotation or a dedicated @flaky group. Do not delete it. Do not merge a green pipeline that only green because the flaky test is gone.
// PHPUnit 10 - mark it, do not lose it
#[Group('flaky')]
public function test_order_status_updates_correctly(): void { ... }
Run your CI suite excluding that group while you investigate. Your releases unblock immediately.
2. Reproduce it on demand
A flaky test you cannot reproduce locally is a test you cannot fix. Run it in a tight loop.
# Run a single NestJS Jest spec 50 times, stop on first failure
for i in $(seq 1 50); do npx jest --testPathPattern="order.spec.ts" || break; done
If it fails within 50 runs, you have a reproducible failure. If it does not, suspect external I/O or a race condition that only surfaces under CI load.
3. Strip shared state
Wrap every test in a transaction and roll it back. In Laravel 10 with PHPUnit 10, use RefreshDatabase or DatabaseTransactions. In NestJS with TypeORM 0.3, wrap each test in a QueryRunner transaction and call rollbackTransaction() in afterEach. If the failure disappears, shared state was the cause.
4. Replace time and randomness with fixed values
Any test that calls Carbon::now(), Date.now(), or Math.random() without freezing time is a flaky test waiting to surface. Freeze the clock before the act, assert against the frozen value.
This applies to anything with an expiry window, a rate limit window, or a scheduled-job trigger. (The same logic applies to LLM token budgets that shift by time of day - see The Single Detail That Decides Your LLM Bill for a related example of where hidden time-sensitivity costs you.)
5. Mock or contract-test every external call
A test that makes a real HTTP call to Stripe, Twilio, or your own staging API will fail when that service is slow. Use a recorded fixture or a mock at the HTTP layer, not at the service class level. Tools: Laravel's Http::fake(), Nock 13 for Node, WireMock for anything polyglot.
How to Tell the Playbook Worked
Run your flagged tests 100 times in CI with no code changes. Zero failures means deterministic. One or more failures means you have more state to control. Only promote a test out of quarantine when it passes 100 consecutive runs in CI, not just locally.
Track flaky-test count as a metric in your pipeline (most CI platforms expose re-run counts). If the number trends down week over week, the playbook is working.
When This Advice Does Not Apply
If your test suite has no isolation at all - every test shares a single populated database, no transactions, no fakes - fixing one flaky test at a time is the wrong starting point. The suite needs a structural reset before targeted fixes make sense. In that case, start with the database layer: introduce DatabaseTransactions, run the suite, and watch how many tests break on the first honest run. That number is the real size of the problem.
Frequently asked questions
A broken test fails consistently on the same code. A flaky test produces different results across runs without any code change. The failure depends on something outside the test's control - shared state, timing, or an external service.
Neither immediately. Quarantine it first with a group tag or skip annotation so it stops blocking CI. Then reproduce it, find the root cause, and fix the underlying determinism problem. Deleting it hides coverage you may need.
100 consecutive passes in CI is a reasonable bar. Local passes alone are not sufficient because CI environments often differ in parallelism, timing, and available system resources.
For database rows, yes - wrapping each test in a rolled-back transaction eliminates leftover data between tests. It does not help with in-memory singletons, Redis keys, or file-system side effects. Those need their own teardown.
Track the number of test re-runs triggered per pipeline run. Most CI platforms (GitHub Actions, GitLab CI, CircleCI) expose this. A declining re-run count week over week is the clearest signal the flakiness problem is shrinking.
Enjoyed this article?
Get notified when I publish new posts on SaaS, Laravel, and remote engineering.



