Systems | Development | Analytics | API | Testing

Too many data teams focus on data movement and creating data pipelines without aligning those activities to business outcomes. Data teams are meant to make data flows work so much of their focus involves managing data connections across the data ecosystem. Pulling data from Salesforce or ensuring reverse ETL from a data warehouse helps support successful data movement and overall data pipeline development. It doesn't always align to a more effective supply chain or cost savings.

The world is full of different types of goals. Consider football. The goal in football is at the end of the field. A runner either crosses the goal or they don't, when trying to make a touchdown. Or, consider basketball. In basketball, when a player shoots the ball, the player’s shot either goes through the net or it doesn’t. Alternatively, consider ice hockey. When a hockey player shoots the puck, it either goes into the net — or it doesn’t.

From databases to data warehouses and, finally, to data lakes, the data landscape is changing rapidly as volumes and sources of data increase. With a growth projection of almost 30%, the data lake market will grow from USD 3.74 billion in 2020 to USD 17.6 billion by 2026. Also, from the 2022 Data and AI Summit, it is clear that data lake architecture is the future of data management and governance.

I used to work with organizations to support their software evaluations, selections, and implementations. What regularly started as a desire for greater visibility into data, generally turned into a need for better management of data across their ecosystems. Basically, organizations cannot look at individual data and analytics projects without evaluating their needs in a cohesive manner. The reality is that selecting a new set of tools for data optimization requires a lot of effort.

When doing long-term planning for an organization, it is really helpful to start with a statement of what the architecture is and what that architecture will encompass. The architectural definition of the technology that will be used will serve as a long-term guide for making technical decisions. In addition, a properly built architecture serves as an instrument of focus, direction, and prioritization for the organization.

According to IDCs Global Datasphere, 64.2 ZB of data was created in 2020 alone. This number is projected to grow by 23% annually from 2020-2025. Therefore, we need data governance frameworks for efficient data management and control. This will help us extract maximum value out of such high volumes of data. Such frameworks would be required for data integrity, data protection, and data security. Indeed, according to BDO, the average data breach cost has been estimated to be around USD 3.8 million.

When I first became an industry analyst, I was fascinated with vendor focus on key capability sets and engineering. The market shifted between acquisition cycles, new start ups, and development cycles for a go-to market strategy based on differentiated tools and capability sets. All of this jockeying for position based on feature sets created strong technology stacks that paved the way for a platform approach and eventual modernization and the transition to cloud.

This is a guest post for Integrate.io written by Bill Inmon, an American computer scientist recognized as the "father of the data warehouse." Inmon wrote the first book and first magazine column about data warehousing, held the first conference about this topic, and was the first person to teach data warehousing classes. Five things you need to know about this topic: The data warehouse is the whack-a-mole of technology.

With the ever-increasing volume of data being generated from a highly diverse set of data sources, organizations have started to increasingly direct their focus on solutions that can help them with data management more efficiently and effectively. Indeed, in the current decade, having a robust data infrastructure is key to an organization’s success, and timely data-driven decision-making is what every management is striving for today.

Data warehousing requires data centralization, whereas data mesh enables a decentralized approach to data access. Organizations might think that the solution to their data management strategy requires a choice between the two, but the reality is that both approaches can and should co-exist.