Systems | Development | Analytics | API | Testing

We shipped v2 of a small products API on a Thursday. Green CI. Green replay. The new search endpoint worked. I went home feeling competent. Friday morning I ran the same traffic against both builds with proxymock and compared the SQL. v2 had added 80 queries on the same HTTP script. A per-product audit COUNT was firing inside the list handler. A startup migration had run ALTER TABLE and CREATE TABLE audit_log. Total DB time was up 70 ms on a demo that should have been boring.

A customer emailed on a Tuesday: checkout hung for ten seconds. I opened our tracing tool, punched in the time window, and got nothing. The trace was sampled out. We keep 1% of traces, like most shops with real traffic do. The one request that actually mattered was in the 99% we threw away. I spent twenty minutes admiring our observability stack before admitting it couldn’t answer a first-grader’s question: what happened to this person? Here’s what I know now.

Having maintained a CodePush fork for 18 months and served billions of updates, we decided it was time for an overhaul. We’re big fans of CodePush, and it was a great benefit to the React Native community. But, it was written in 2015 and had some weaknesses. The biggest of these was architectural, with limitations in how update checks and release metadata could scale. The result, Codemagic Patch, is now public and available to self-host.