Data projects in the form of data warehouse, data lake, big data, cloud data migration, BI reporting and analytics, machine learning are manifesting in every organization. While project timelines are shrinking, the number of data projects are increasing as is the complexity.
We have observed that data-centric applications are lacking the rigors and the discipline required to execute these large and complex projects. While general software projects have adopted the CICD and DevOps principles, the data integration and migration projects are still living under the rock. With the advent of Big data and Cloud technology, this has become a huge problem.
Time-to-market for a data project has become critical in organizations of all sizes. This paper discusses the adoption of DataOps methodologies for data and big data projects, to improve the success of the project as well as speed up the time-to-market. We further analyse some of the bottlenecks such as: organizational culture, data test automation and how they are hindering the implementation of DataOps. Ultimately, we are proposing the DataOps solution to improve both delivery of the data project and data quality.