Why we developed iceDQ
Data Reliability Engineering
Data Reliability Engineering
Data centric projects and systems such as a data warehouse, big data, CRM and data migration are like a factory wherein raw data is ingested and then in a series of orchestrated steps data is cleansed, transformed, integrated to produce the final product, the data. This is very similar to an assembly line in a factory.
Conventionally, enterprises are solely focused on inspecting the final data that is delivered in production. However, the data issues often stem from prior missteps that happened during development, testing, and operations. By the time data defects are found in production data, it’s too late.
Often data defects in production have their root cause much earlier in development and operations of the data platform. In the data factory analogy, organizations navigate two crucial phases before delivering final data: building the data factory and running it effectively. Mistakes at any point in this lifecycle directly impact the final data quality.
Phase 1, Build the Data Factory: In this phase, requirements are collected, processes are coded, and the data pipelines are orchestrated. Data testing in the build phase is needed to capture key issues:
Phase 2, Run the Data Factory: Once the data factory is assembled, it starts receiving data and all the processes start running in an orchestrated manner. Apart from the defects from the build phase, additional data errors are introduced because of lack of operational checks and controls.
Phase 3, Final Production Data: Any Carry-over defects that were missed in the prior two phases along with ongoing challenges can still cause defects in final production data.
We’ve seen that just checking data quality at the very end isn’t enough. Instead, a more comprehensive approach called data reliability engineering is needed.
DQ – Data Quality
Data Quality is an instance in time.
DR – Data Reliability
Data Reliability is consistent data quality over time.
DRE – Data Reliability Engineering
Data Reliability Engineering is a practice of integrating people, processes, and products to deliver reliable data.
Successful implementation of DRE requires people, processes and the iceDQ platform in all phases of data development life cycle.
One of the first steps towards DRE is to automate data testing ensuring that each ETL process is tested before deploying into production.
Instead of sampling data, use the complete data set for testing. This will ensure complete test coverage by discovering any corner test cases missed during analysis.
Ensure that not only quality controls are done on the final data, but also the tooling and data pipelines are tested thoroughly.
Shift the focus from the end of data development lifecycle to the left by ensuring business users are involved in requirements gathering, establishing data audit rules and making sure that the checks and controls are established in operations.
As part of the code deployment not only deploy the ETL code but also embed checks as part of deployment. This will allow the operations team to monitor the production pipelines with minimum effort.
Store all the testing, monitoring and data observability rules in a centralized repository that can be accessed by teams over a long period of time.
Establish checks and controls in the production pipeline to capture any data issues before they can cause damage in the downstream systems.
Observe the production data for anomalies and notify incidents on real-time basis as they happen with supporting information of triage and root cause analysis.
Measure and report both the frequency and magnitude of data errors.
By adopting a holistic data reliability approach, companies can significantly reduce the amount of hidden work that goes in correcting data issues. There is a direct savings on costs, time, improved delivery timelines, and most importantly, build a reputation for reliable, high-quality data.
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.