Measure the theory before building the fix
Everyone suspected the file server. A canary and a few counters pointed at our own service, and the fix worked on the first try.
In mid-October the background service started to stall. Desks waited, rounds ran late, and the obvious suspect was the file server. The service keeps its source of truth on a network share, and the share felt slow.
The easy move was to ask IT to look at the file server. I decided against it until we had numbers.
My part. I kept the question away from IT until the measurement was in, and I asked for the fix to be proven three separate ways.
A canary for the theory
The first packet added a canary: a tiny job that touched its own files on the share every 10 seconds, independent of everything else. It ran for 21 hours and took 7,488 samples. None of them was slow on the server's side. In the same window, the service's own share work stalled in 666 of 1,223 minutes.
The share was fine. Something in our service was waiting on itself.
Counters for our own work
Next came per-job counters with 0.2% overhead: how often each job touched the share, and how many full reads it made. The numbers ranked the problem within a day. Two jobs each walked the whole store about four times per cycle, and because the service runs its work on one thread, everything else queued behind those walks.
One walk per cycle
The next packet replaced eight walks with one shared walk per cycle. I asked for the proof three ways, because each one catches something the others cannot.
| Proof | Share calls, job A / job B |
|---|---|
| A fixture | −68% / −69% |
| A scratch copy of the real store, real code | −76% / −83%, 0 files different |
| Production, before and after | −59% / −42% |
In production, the time the two jobs spent on the share fell by 95% and 70% per hour: from 3,527 s to 190 s, and from 1,500 s to 447 s.
Production saved fewer calls than the scratch copy because status checks there rose 2.3 times. A copy of real data is close to the real thing. It is never the same.
What I took from it
- A theory is a claim. Before acting on it, find the run that would prove it wrong.
- Instrument first, fix second. The counters from one packet named the lever the next packet pulled, and the fix worked on the first try.
- Prove a performance change on a fixture, on a copy of real data, and in production.