For the past few years, big data has been all the rage, but it has not delivered on all its promise. We were expecting omniscience at our fingertips, and we’re getting moderately well-targeted advertising instead. Don’t get confused though, big data is not going anywhere. On the contrary, data keeps growing at a dizzying rate, increasing in volume, variety, velocity, and veracity.

I am very excited to introduce Talend Data Streams, a brand new, cloud-native application that enables you to get streaming data integrations up and running in minutes all while providing unparalleled portability powered by Apache Beam.

With the growing adoption of cloud-based IT infrastructures, the proliferation of mobile and IoT devices, and the rise of social media, companies of all sizes, across all industries are amassing huge quantities of data in differing variety, velocity, veracity and validity.

In the past few years, there has been a shift in the data industry, leading to the emergence of a new category of data citizens: the ‘ad hoc’ or ‘citizen’ integrators. With these new personas adding to the (already long) list of data workers having access to corporate information, companies are needing to re-think the way they approach their data security and data governance strategies.

The appearance of Hadoop and its related ecosystem was like a Cambrian explosion of open source tools and frameworks to process big amounts of data. But companies who invested early in big data found some challenges. For example, they needed engineers with expert knowledge not only on distributed systems and data processing but also on Java and the related JVM-based languages and tools.