My first instructor had a saying about checklists: a checklist you don’t trust is worse than no checklist, because it teaches you to stop looking. She was talking about preflight, but she could have been talking about software. Programmers keep checklists too — automated tests that run over every change before it ships — and somewhere along the way, many teams stopped trusting their own. The response to that problem has collected under a young name, TestOps, and this primer walks through it the way I’d walk a new student through the hangar: plainly, with a glossary at the end and no shame about asking what the words mean.
TestOps, explained from the preflight ritual
Think about what a preflight inspection actually produces. Not a repaired aeroplane — a decision: airworthy or not, made quickly, made the same way every time, and trusted enough that nobody re-checks the fuel sample out of fear. Now picture the opposite: an inspection that flags a phantom oil leak every third walkaround. Pilots start skipping steps. That is precisely the situation many software teams are in — their automated checks cry wolf so often that engineers re-run them without reading — and it is the situation TestOps sets out to fix.
The definition worth memorising: run the test suite the way you would run any critical service. The people it serves are developers; what it sells them is a trustworthy go/no-go answer for each change, delivered while they still remember what they changed. The Ops Atlas overview of TestOps is the best one-stop reference I know, and it makes a point worth repeating: the discipline is not about owning more tests, it is about whether each verdict deserves belief. Notice the aviation parallel — nobody praises a safety program for the thickness of its manuals, only for what its go/no-go decisions are worth.
Weight and balance for test suites
Every student pilot learns that loading is a design problem: too much weight in the tail and the aircraft behaves badly at altitude. Test suites have their own loading problem. The classic healthy shape — most checks fast and cheap at the bottom, a few slow end-to-end journeys at the top — is called the test pyramid, an idea Mike Cohn wrote about back in 2003. Load the wrong end and you get the famous anti-pattern: suites dominated by slow, moody interface tests that take hours and fail for reasons no one can reproduce. Practitioners call that shape the ice-cream cone, and it wobbles exactly the way the name suggests.
Newer shapes exist — the honeycomb and the trophy, which shift weight toward tests around a system’s middle layers — and partisans argue about them the way pilots argue about high-wing versus low-wing. The honest answer from the field: pick by cost and by how reliably each layer gives the same answer twice, not by which diagram is fashionable. A shape is airworthy if the verdicts coming out of it are.
A mini-glossary before you go solo
Flight training works because the syllabus gives every manoeuvre a name, a standard, and a way to fail it honestly. Software teams kept the checking habit and lost the vocabulary, which is half the reason their meetings go in circles. Here is the vocabulary that comes up in every TestOps conversation, translated out of jargon:
- Verdict — the pass/fail answer a suite gives for one change. The entire product of the operation; everything else exists to make it believable.
- Flaky test — one that returns different answers for identical code. The defect is in the test or its surroundings, not in the software being checked.
- Quarantine — a fenced-off status for untrustworthy checks: pulled from gate duty, assigned an owner plus a removal date, and watched until fixed or deleted.
- Burn-in — a probation period for new tests, running nightly without gate power until they prove steady.
- Sharding — splitting a big suite across many machines so the full answer arrives in minutes instead of hours.
- Stage budget — a time ceiling for each pipeline stage, so quick feedback stays quick and the heavy checks get their own slot.
- Merge queue — a holding area that lines up changes to be tested and landed in order, so one broken commit can’t spoil the batch behind it.
One research finding deserves a place in every glossary, because it changes how you read your own suite: when scholars studied flaky tests, the large majority — on the order of four in five — were unreliable from the day they were committed. Instability is usually designed in, not drifted into. A test that depends on timing, on the order other tests run in, or on leftover data from a previous run is born distrustful.
Where a TestOps beginner goes next
If you maintain software for a flight school, a parts shop or an aviation club, none of this requires a reorganisation. Start by timing how long your checks take and counting how often a failure turns out to be false. Put a time budget on the quick stage. Fence off the check that fails every third run, give it an owner and a deadline, and watch what happens to the team’s willingness to read failures. Small, mechanical, boring — the way good maintenance always is. Our own beginner’s guide to aircraft maintenance makes the same argument in hardware terms, and the glossary habit transfers too: nobody wanders around a hangar using the wrong word for a control surface, which is why ailerons, elevators and the rudder get their own plain-language walkthrough on this site. And when someone tries to sell your club a testing platform, apply the same scepticism you’d bring to parts shopping — the overpayment traps in how to stop overpaying for aircraft parts rhyme almost line for line with the software world’s shiny-tool traps. Trust is built verdict by verdict, in the air and in the pipeline.
