Data Quality Assurance and Testing

After 9 months of effort, Nomura Securities celebrated the successful implementation of a new Fixed Income processing system, and integration of its data into a data warehouse.

The project was a major effort that touched almost all areas of the US business and impacted operations in other regions. The overall size and scope of the project was exemplified by more than 400 people who were involved either directly or peripherally. New security reference data platforms, new interfaces for the Fixed Income front office, separation and partitioning of the existing system for the Equities business, major re-plumbing of finance and regulatory and general ledger feeds, new interfaces for risk, a new suite of reports for compliance, and significant changes to trade confirmations and settlement, were just some of the requirements of the project.

One of the key tasks was to route data from the new source system into the existing Data Warehouse. Downstream systems, such as front offices / trading desk, relied on the Data Warehouse, as it consolidated information from the Fixed Income and Equities systems. As this task was data-centric, and it was understood that data quality would be the key to success of the overall project. That is why the bank turned to iceDQ Soft for their data quality and monitoring solution.

For the multi-million-dollar project, “How can we test hundreds of new data-feeds and thousands of existing ETL processes and at the same time monitor progress in the most effective way?”, was the key question remaining to be answered.

In a data-centric project, data quality management is as important as project management. Data, not process, is the focus of such projects. Testing has to prove that the quality profile of the production data is maintained across all components of the project. This means that a dedicated framework for testing data-centric project is a must. The iQA solution provides multi-step action plan. The first step is to operationalize iQA Data Quality Assurance and Testing Framework.

Nomura Securities implemented the Testing Framework. Next, iQA was configured to support the Testing Framework to achieve goals of communication, accountability, global collaboration and visibility to management.

Once the actual testing began, iQA quickly proved to be up to the job. Over a period of nine months, testers, developers, and SMEs implemented roughly five thousand data testing rules of different types, including:

On each release cycle, many data issues were discovered. iQA routed those data issues and reports to developers, SMEs, and even source system users.

Initial runs quickly discovered thousands of critical and non-critical data quality defects. As the testing cycles proceeded, these issues were resolved and users could sign off on the ETL processes.

As new code was developed, previous rules were recombined to provide regression tests. Also, the same rules were reused in different environments such as UAT and integration.

iQA’s web-based interface was effective for global collaboration. Access to inexpensive resources meant low cost. The simplicity and centralized control made it easy to onboard new resources and outsource testing.

iQA’s automated data comparison and testing permitted testing on complete datasets instead of restricted sample test data.

iQA’s automated data comparison and testing permitted testing on complete datasets instead of restricted sample test data.

Integration of iQA with HP Quality Center, which had been acquired by Nomura Securities, allowed use of this infrastructure and its accompanying methodology.

The Rules Knowledge Repository stored all the rules discovered for data testing so that they are accessible for future use. There was no need to maintain documents for testing as both descriptive metadata and actual testing results were stored in the repository.

The testing results Dashboard provided visibility to status of testing at any time. Rapid feedback in the form of reports of data with material errors together with drill-down abilities enabled quick decisions and responses.

For business users, it was easy to sign off on the ETL process based on success of predetermined lists of rules.

It is a typical characteristic of data-centric projects that information requirements are not clear at the beginning. Many of them are discovered as more data is reviewed. This gradual crystallization of information requirements is aided by iQA, with its Rules Knowledge Repository. Another characteristic of data-centric projects is that data tests must be carried into production. There is no guarantee that production data will remain stable into the distant future. Again, iQA has this capability.

The downstream data was used by trading, and any bad data could have had serious financial impact. Because of this, the key success factor for the success of the project was obtaining the sign-off on data quality. And with iQA, the Bank was successful in proving this success within given parameters of time, money, and resources.

iceDQ empowers organizations to ensure data trust and reliability throughout the data life cycle.

Our comprehensive platform combines data testing, data monitoring, and data observability into a single solution, enabling data engineers to proactively manage data quality and eliminate data issues before they impact business decisions.

Leading companies across industries, including prominent players in banking, insurance, and healthcare, rely on iceDQ to continuously test, monitor, and observe their data-driven systems. This ensures trustworthy data that fuels informed decision-making and drives business success.