Systems | Development | Analytics | API | Testing

Here's why you need to double down on your DataOps before your vacation. In the past few months, everything has changed at work (or at home). Q1 plans were scrapped. Reset buttons were smashed. It was all about cost-cutting and keeping lights on. Many app and data teams sought quick solutions and developed workarounds to data challenges and operational problems as people prepared to work from home for the foreseeable future. And now, it’s time for a holiday.

Next to the healthcare system, COVID-19’s biggest infrastructural burden fell upon the supply chain. Fluctuations in supply and demand of essential goods, along with the oil surplus, led to a freight cliff in mid-April. Outbound tender volume and spot rates bottomed out, which highlighted a massive drop in demand. As the market rebounds, technological investments are key to the industry’s recovery.

Our CTO, Andrew Stevenson was interviewed by Alan Shimel for TechStrong TV. The discussion was all about hot data topics such as DataOps, DevOps and practices to successfully enable Kafka. Andrew narrates his journey from civil engineering to starting Lenses.io with Antonios, our CEO, to help organizations succeed with real-time data.

Apache Kafka has gained traction as one of the most widely adopted technologies for building streaming applications - but introducing it (and scaling it) into your business can be a struggle. The problem isn’t with Kafka itself so much as the different components you need to learn and different tools required to operate it. For those motivated enough, you can invest money, effort and long Friday nights into learning, fixing and streamlining Kafka - and you’ll get there.

Data Engineers are forever flying the flag for open-source technology. But now that we’re safely locked away in our homes - potentially for the rest of the year - a new danger looms: That we get distracted by our new data tools and lose touch with delivering value to the business. Today most Data Engineers around the world are working from home, and at first glance it may seem like this works. After all, a solid internet connection is all we need to carry on doing what we were doing...

Remote working can be a blessing for every parent. In that way, you don’t have to face tough dilemmas like “staying at the office to finish fixing that bug” versus “returning home early to do some karaoke with your children”. Being a working parent used to be all about making tough calls in terms of career or childcare.

Some time ago, the concept of event streaming was not that widespread. In addition to that, the platforms provided were much fewer than today, and required more technical depth to set up and operate. However, as time passed, those platforms matured, community grew, the documentation was improved and the rest of the world started to wake up to the advantages these platforms can bring to address the real-time experiences businesses need. And Apache Pulsar is a great example.

Systems fail from time to time. And there is nothing worse than being unaware a system is down at all. Which is why we wrote the email alerts from Lenses.io tutorial. But what about those evenings where you are enjoying a documentary or just having an email-free evening? After all, system failures rarely happen at a magically convenient time.

Open source is great but sometimes it misses the mark for security at enterprise levels. Take Kafka Connect, I’ve built a few connectors in my time and prior to its introduction to Apache Kafka back in 2017 I used other hand cranked pieces of software and security was always a primary concern. One feature that will quickly put a blocker on your project being successful is not reaching production. There’s a number of reasons for this but high up, usually top, is security.

Events and Conferences are a core element of any growing industry and ours is no different. It’s the easiest way to learn what’s new and to gain a better understanding of any technology. After all, we might love blogs and online courses, but at the end of the day, is there a better way to process any type of information than a passionately delivered speech or a lively discussion?