ETL Testing

Automate testing of your ETL process completely

ETL process is a complete black box, only way to do test an ETL process is by comparing the input data with output data of the process. Thus ETL Testing rules need the ability to read the data, apply transformation rules, compare millions of rows and identify issues effectively. And automation of ETL Testing is important more than ever, as the complexity of ETL process and volume of data is increasing on a daily basis.

Challenges in ETL Testing

Testing across Databases & Files

When testing an ETL process users have to compare data across platforms. In such scenarios users perform manual ETL Testing in below manner which is highly error prone.

  • Bring the source & target data into Excel
  • Compare the data by eyeballing/ staring

Testing Millions of Rows

When ETL Testing is done manually, users are only able to test sample set of data because they compare data in excel. Since full volume of data is not compared, ETL process might break in production environment or incorrect data might get loaded because of limited test coverage.

Regression Testing

Today every organization is following the agile methodology for ETL development, so the ETL code is built and deployed continuously. But it is impossible to do regression testing when testing is being done manually. This is turn causes lot of resources spending unnecessary amount of time testing during production deployment.

ETL Testing Solution - iCEDQ

iCEDQ is an automated ETL Testing, Data Warehouse Testing, Data Migration Testing & Production Data Monitoring software. It’s unique ETL testing rules engine allows testers, developers or any other user to compare or validate millions of rows between different databases and files effectively and efficiently. It helps users identify the exact row and column having the data issue between two data sources.

How iCEDQ Works

  • It reads data from any database or file
  • It compares data in memory based on unique columns
  • It evaluates transformation or business expressions
  • It identifies mismatching data based on comparison & expression evaluation
iCEDQ - ETL Testing, Data Migration Testing & Data Quality Governance Platform.

Key Features

Database & File Connectors

iCEDQ can connect to any relational database, flat files or new big data databases like Hive, Impala and many other. Because of this functionality it can perform ETL testing easily as it now compare data across databases and files effectively.

Look at the list of data connectors for hadoop and databases.

In-memory Rules Engine

Our unique auditing rules engine compares data in chunks, in memory. Below is an example of how our ETL testing rules engine works.

  • Reads 10000 rows from source & target
  • Compares the 10000 rows & identify issues
  • Repeats above steps for full dataset

Tests Suites

One of the benefits of automated ETL testing software is ability to do regression testing. iCEDQ enables users to perform regression testing through test suites which allows users to combine all the rules into one test suite.