Yes, I’ve said it. The modern data stack is a pain to work with. But it wasn’t always like that. As companies realized they can leverage data to accelerate growth new data tools were invented. From NoSQL databases that specialize in processing specific data structures (graph anyone?) to the Python-Pandas-like Spark ecosystem that allows you to run queries on Big Data (capital B, mind you). But with every new tool added to the data stack, the complexity increased.

ETL pipelines help companies extract, transform, and load data so it is ready to provide insights and value to the company. But running a smooth data operation depends on building reliable and scalable data ingestion pipelines. SaaS vendors like Keboola and Azure Data Factory take away the heavy lifting.