This post is about Kafka and the process I have been through recently writing a performance test for an application that subscribes to messages from this technology. The test I ended up with was in the end very straightforward but there were several hurdles that took a while to resolve. I hope that reading this post will hopefully help you avoid them.

Have you abandoned a web service, because it took too long to load? I bet you have, everyone does that every now and then with slow applications. There are a lot of services to check load times, but is it the only thing to check about website’s performance worldwide? Certainly not! It is just one thing to check, not the only one. You may have performance tested the service locally and validated that everything works just fine.

I used the K6 load testing framework to benchmark the Compute Engine f1-micro and Caddy web server hosting this site. With CloudFlare caching turned off, the server was able to serve an onslaught 800 virtual users continuously reloading the page (while maintaining a median request duration of <400ms), but started dropping requests when increasing the load further. This is fine.

Apache JMeter is a Java open source tool. If you want to run load or performance tests using Apache JMeter, you can use their Graphic User Interface (GUI) and you don’t need to know any programming language, follow the JMeter documentation and you are ready to go to design your API testing scenarios.

Socrates preached, “To know thyself is the beginning of wisdom.” This ancient Greek anecdote applies to your modern Apache Kafka project: developers, go forth and load test your real-time application to understand the capacity and limitations of your project before deployment. Failure to do so will cost you time and money (e.g. Robinhood’s outage on a historic trading day). Load testing your real-time applications has three main objectives.