Data Testing

With the exponential growth in data, there is a surge of data centric projects. And these data projects require testing as a part of DataOps practice.

What is Data Testing?

Data Testing is a QA activity performed by testers in a non-production environment of a data-centric system involving a large volume of data.

The primary goal of data testing is to ensure that the data processes are certified before deploying them into production. Here are some of the systems where data testing is required:

• Customer Relationship Management – CRM
• Master Data Management – MDM
• Data Warehouse /Data Marts – DWH
• Reference Data Management – RDM
• Cloud Data Migration Projects
• Big Data Projects
• Data Lake Projects

Data Testing Types & Methods

There are various types of data testing. Depending on the use cases they vary slightly but the fundamentals remain the same.

1. ETL testing
2. Data Warehouse Testing
3. Data Migration Testing
4. Big Data Testing
5. Cloud Data Testing
6. Data Lake Testing
7. BI Report Testing

Data Testing Techniques

Data testing essentially requires that a data process runs and then output generated by the data process is compared with the expected data. Data testing generally involves two fundamental techniques, data validation and data reconciliation. To get expected values:

1. Static Values: There are predetermined values the users expect. These are essentially predetermined static values.
2. Dynamic Values: Calculated values based on the data feed and the data transformation specifications. Such values are dynamic since the input keeps changing.

For more details, please refer to ETL Testing.

Data Testing Solutions

A data testing solution must have the following capabilities:

1. 1. Data Testing: Almost all applications, including mobile apps, involve data. But here we are referring to is not a few transactions but millions or billions of records.
      • Input data: While data processing does the transformation, if the input data itself is not correct then the data output will also be affected. So, it is essential to verify the quality of input data while in the development phase itself.
      • Reference Data: Reference data in systems like account types and employee types drive a lot of processing logic. Hence it is critical to ensure that the reference data values are exactly as expected.
      • Referential Integrity: Compare the parent and child table for values that are in child table but not in parent table.
   2. Data Processes Testing: Data-centric systems usually have data processes often called ETL or data pipelines. These processes extract, transform and load millions of records in batch or in real time. If these processes are not developed or tested properly, it will greatly impact when the jobs are deployed in production.
   3. Data Process Orchestration Testing: Data pipelines are not standalone processes. But in fact, work in synchronization with many other tasks.
      • Process Schedule Testing: Tests to ensure tasks are executed based on predetermined schedules and events.
      • Process Dependency Testing: Any mistake in the execution order will corrupt the data. Hence it is essential to test the process dependencies before deploying and running them in production.
   4. Data Models and Schema Testing: The data is held in a data model. For example, developers can accidentally change column data types or length. This will affect the data populated in the system.
      Hence the data testing solution usually has capabilities to test and verify the model based on reference schemas.
   5. BI Report Testing: While data in tables is commonly understood, data is also pushed into reports. Just because data in the database is correct does not guarantee the accuracy of the generated report. The data testing solution must also have the capabilities to test reports.

Manual vs Automated Data Testing

Manual Data Testing involves sampling and visual inspection of data. However, manual data testing is impossible to do because of the inability of humans to process data beyond a few records. Developers might build some scripts to augment some of the testing activities. They try to build data in excel but it is not scalable or repeatable. Usually, they end up doing some sampling or limiting the scope of test cases.

The flaws in manual data testing are:

1. 1. 1. Limit on records.
      2. Data sampling.
      3. Not repeatable due to the Ad hoc nature of testing.
      4. Requires skilled worker.
      5. Limited use cases.
      6. Lacks scalability.
      7. No reusability.

Automated data testing on the other hand is based on program templates. There are literally no limits on data volumes. As the data scales so does the computing power. Hence, there is no need to do data sampling. The data testing software has the capabilities to go across environments that enables comparisons.

What is the difference between Data Testing and Application Testing?

A web or mobile application usually has screens with which a user interacts. On the other hand, the systems involving data processing almost never have any screens with which a user interacts. In data testing, large data volume and the ETL processes are the core that drives everything. Application testing is usually focused on UI. But what about testing data-centric systems involving large amounts of data processing? Instead of screens, they have data processes running in the background.

Here’s a summary of the differences.

Conclusion

With the advent of data centric projects, it has become critical for enterprises to focus on data testing during the development phase itself. If the data process is not tested as part of quality assurance, then there will be many more defects in production data. Also, there is an exponential increase in the volume of data as well as complexity of data processing; these factors necessitate automated data testing.