Much has been written on the growth of machine learning and its impact on almost every industry. As businesses continue to evolve and digitally transform, it’s become an imperative for businesses to include AI and ML in their strategic plans in order to remain competitive. In Competing in the Age of AI, Harvard professors Marco Iansiti and Karim R. Lakhani illustrate how this can be confounding for CEOs, especially in the face of AI-powered competition.

Modern business applications leverage Machine Learning (ML) and Deep Learning (DL) models to analyze real-world and large-scale data, to predict or to react intelligently to events. Unlike data analysis for research purposes, models deployed in production are required to handle data at scale and often in real-time, and must provide accurate results and predictions for end-users.

Data science has come a long way, and it has changed organizations across industries profoundly. In fact, over the last few years, data science has been applied not for the sake of gathering and analyzing data but to solve some of the most pertinent business problems afflicting commercial enterprises.

We’re delighted to announce the release of the Iguazio Data Science Platform version 2.8. The new version takes another leap forward in solving the operational challenge of deploying machine and deep learning applications in real business environments. It provides a robust set of tools to streamline MLOps and a new set of features that address diverse MLOps challenges.

As more and more companies are embedding AI projects into their systems, attracted by the promise of efficiencies and competitive advantages, data science teams are feeling the growing pains of a relatively immature practice without widespread established and repeatable norms.

A Forbes survey shows that data scientists spend 19% of their time collecting data sets and 60% of their time cleaning and organizing data. All told, data scientists spend around 80% of their time on preparing and managing data for analysis. One of the greatest obstacles that make it so difficult to bring data science initiatives to life is the lack of robust data management tools.

Spark is known for its powerful engine which enables distributed data processing. It provides unmatched functionality to handle petabytes of data across multiple servers and its capabilities and performance unseated other technologies in the Hadoop world. Although Spark provides great power, it also comes with a high maintenance cost. In recent years, innovations to simplify the Spark infrastructure have been formed, supporting these large data processing tasks.

Data scientists today have to choose between a massive toolbox where every item has its pros and cons. We love the simplicity of Python tools like pandas and Scikit-learn, the operation-readiness of Kubernetes, and the scalability of Spark and Hadoop, so we just use all of them. What happens? Data scientists explore data using pandas, then data engineers use Spark to recode the same logic to scale or with live streams or operational databases.