Production Data Monitoring & Compliance
Scenario
Data is used for decision making; hence it is imperative that business user gets complete and accurate data in a timely manner. Beyond the business needs, there are compliance and regulatory needs that must be satisfied. Therefore, data that is used for decision-making must have checks and controls to ensure the reliability of data.
However, we should understand that data quality is a by product of the system that has been put in place – data inputs and the ETL processes that populate the data. The existing enterprise scheduling tools can only monitor the success of the batch processes and cannot verify if the data has been transformed correctly.
The Challenge
An industrial approach for data quality governance is needed that goes beyond the profiling and address fixing solutions. We need to monitor the correctness of data transformation and if necessary check actions.
How to find data issues before the business users complain?
Most of the data transformation or ETL batches run in the night. If the data is not validated in processes than most likely a problem will not be noticed until the business users start using the data. Not only will it make the technology team look bad, but also if the data exceptions go unnoticed; it will result in wrong decisions and financial or reputation loss.
How to establish checks and controls on data for legal requirements?
With the current SOX, BASEL II and Solvency II requirements we cannot get away by claiming ignorance on data correctness. It is much easier to implement an audit based data monitoring solution within the workflow and continuously logging the audit activity thus producing an audit trail that can be used to pass the certification requirements.
How to lower operations & support cost by proactive monitoring?
The cost of fixing the problem is less if found earlier. As the bad data is created, because of the compounding effect of the subsequent processes it can make the data problem even worse. Even if the problem is noticed it is usually too late and expensive in terms of time and resources. Sometimes it may be even impossible to undo the changes without rerunning the whole nightly batch.
Our Solution
iCEDQ is a purpose built ETL Testing, Data Migration Testing and Production Data Monitoring platform for the enterprise. It identifies an error in data integration processes against technical and functional requirements written in data mapping documents. iCEDQ provides a complete automated solution with a built in ability to audit, validate and reconcile data within/ across heterogeneous data sets.
Audit Rules Engine
With our audit rules engine users can create rules to monitor the data on a daily basis. These rules can generate exception reports highlighting the data issues that technology team can fix, so that the business users can have correct data for making key decisions.
Compliance Reporting
The rules are created to make sure that the data is in compliance with SOX, FINRA or other regulations and can be executed at any point of time. The results of these executions are stored inside the iCEDQ repository that allows users to create custom reports for compliance purposes.
Workflow Integration
iCEDQ allow users to integrate or embed the execution of the audit rules in their current workflow using the command line or web service interface. Based on the results returned by the rules it can be decided if the workflows need to be stopped. This integration allows you to proactively monitor the system for data issues on a daily basis.
