Systems | Development | Analytics | API | Testing

The dashboards are red, the alerts are firing, and Slack has officially become a war room. Someone has asked, “Anyone else seeing this?” and 14 people have immediately responded with screenshots. Welcome to another day in distributed systems. The recent GitHub outage is a great reminder of how complicated modern applications have become. Services talk to services, infrastructure scales up and down, retries multiply traffic, and dependencies behave in ways nobody expected.

Private cloud and on-premises infrastructure are different. I know this. I used the terms interchangeably anyway because I was trying to compress a pattern into one sentence. That was sloppy, but the underlying pattern still matters. Banks run Kubernetes differently from most other organizations. Not because every bank makes the same choices, but because similar constraints can produce a similar pattern.

Selecting load testing tools for your CI/CD pipeline is a lot like picking between a scalpel and a Swiss Army knife. A scalpel – the focused, specialized solution – excels at one job with surgical accuracy. Think of Apache JMeter or Gatling: these tools are designed for deep, targeted testing, offering powerful scripting and low-level protocol support. They let you dial in precise scenarios and slice through performance bottlenecks with fine control.

Somewhere in your service is a branch that has never executed. Not a rare one, a never one. It handles a dependency being unavailable: it reads from a cache, it returns a stale value, it marks the response degraded so callers know not to trust it too far. It was written carefully. It was reviewed. Whether it works is an open question, because nothing in the test suite makes that dependency fail, and the dependency does not fail on request.

An engineering leader asked me a question a few weeks ago: could we read their business workflows out of API traffic instead of asking people to document them? I said it should be possible. Then I tried it. A few engineers know how the system really works. They know which calls make up a work order and which checks happen after a write. That stuff rarely makes it into the test plan. Usually it’s in somebody’s head. Sometimes it’s in several heads, with slightly different answers.

Self-hosting OTA updates became common after App Center and CodePush shut down in 2025. CodePush had been the popular free option, but the remaining hosted services easily hitting five or six figure bills for popular apps. For staging-only or use with small user bases, self-hosting is fairly straight forwards. But, as the user base grows, the architecture can start to struggle and bolt on pieces are required.

Engineering leaders have been claiming to “shift quality left” for years but production remains stubbornly stuck out of reach of software engineers. The realm of production remains mysterious with tools no one has access to and UIs that wouldn’t make sense to engineers anyway. I’ve noticed a small but growing trend of large enterprises hiring Reliability Engineers instead of Site Reliability Engineers. Dropping one word looks cosmetic but I think it points to a much bigger change.

A metric says latency increased. A log says a request failed. A trace identifies the slow dependency. An APM agent points to the method. Manual instrumentation explains the business operation. Traffic capture shows the exact request and response that triggered it. Each layer answers a question the previous layer could not. Each also introduces a new cost, blind spot, and failure mode.