What 'All Tests Passed' Actually Promises You (And What It Doesn't)

Part four of the pipeline series — lint and type checking both run early because they're cheap. Tests are next, and they're where the pipeline starts actually costing real time.

Before you read on, test your intuition. Pick how the tests are written, pick a bug hiding in the code, and run the job. Some combinations fail loudly. Others come back green with the bug still inside, which is the whole point of this post. Then read the rest and come back to it:

Two different scopes, one job most teams don't separate

"The test job" is usually really two things running back to back: unit tests (one function or module, in isolation, everything else mocked or stubbed out) and integration tests (multiple real pieces wired together — a real database, a real API call between two of your own services). Unit tests are fast and tell you a piece works in isolation. Integration tests are slower and tell you the pieces actually work together, which unit tests structurally cannot verify no matter how many of them you write, because mocking the other side is exactly what makes them fast — and exactly what makes them blind to integration bugs.

The green checkmark's actual scope

"All tests passed" only means: every test that exists, and was actually run, passed. It says nothing about behavior nobody wrote a test for. A function can have a real, live bug in a code path that's simply never exercised by the suite, and the pipeline will still show green, because green doesn't mean "correct" — it means "consistent with what we decided to check."

This is where coverage numbers get misunderstood. A coverage percentage tells you what fraction of your code ran during the test suite — it says nothing about whether anything meaningful was actually asserted about the result. It's entirely possible to hit 100% line coverage with tests that call every function and check nothing, and the coverage tool has no way to tell you that's what happened. Coverage is a floor on "did we even look here," not a ceiling on quality.

What even a weak suite does catch is the loud stuff: a missing import, or a crash at runtime. Those fail the job outright, and you get the big red cross instead of a quiet wrong answer.

Integration tests: slower, harder, and closer to the truth

Integration tests are the long-running ones. They exercise what your code actually does: reading real files, extracting data from them, calling external APIs. That realism is the point, and it's also the cost.

They are hard to set up, because the environment they run in has to be a close replica of production, with the same services, the same config, and data shaped like the real thing. Some things can't be integration-tested at all. Authentication through a third-party provider is the usual example, since you can't (and shouldn't) log in to someone else's service from CI on every push.

What they give you is that the calls are real. If the file format changed, the API now returns a different shape, or a service is misconfigured, this is where you find out, which no mocked unit test can tell you. The same limit from the green-checkmark section applies, though: they only verify the paths somebody wrote a test for.

Now scroll back up and run the demo again. Which combinations still surprise you?

Flaky tests, and why CI treats them the way it does

A flaky test is one that passes sometimes and fails sometimes with no code change in between — usually a timing assumption, a shared bit of state leaking between test runs, or a real race condition in the code under test dressed up as a "test problem." Some pipelines quietly retry a failed test once or twice before calling it a real failure. That's a genuinely double-edged setting: it keeps a known-flaky suite from blocking unrelated PRs, but it can just as easily paper over a real, intermittent bug that happens to reproduce as test flakiness. A team that retries flaky tests without also tracking and eventually fixing them is choosing convenience over signal, and it's worth knowing which one your pipeline is doing.

Why it runs after lint and type checking, not before

Same fail-fast logic as the series so far: a test suite can take minutes, sometimes much longer with integration tests spinning up real services. Running lint and type checking first means a trivial typo or an obvious type mismatch gets caught in seconds instead of burning several minutes of runner time before failing anyway.

Next: the job most people roll their eyes at — clean-code and static-analysis checks — and why "this is just a vibe metric" is only half true.