ETL process is a complete black box, only way to do test an ETL process is by comparing the input data with output data of the process. Thus ETL Testing rules need the ability to read the data, apply transformation rules, compare millions of rows and identify issues effectively. And automation of ETL Testing is important more than ever, as the complexity of ETL process and volume of data is increasing on a daily basis.
When testing an ETL process users have to compare data across platforms. In such scenarios users perform manual ETL Testing in below manner which is highly error prone.
When ETL Testing is done manually, users are only able to test sample set of data because they compare data in excel. Since full volume of data is not compared, ETL process might break in production environment or incorrect data might get loaded because of limited test coverage.
Today every organization is following the agile methodology for ETL development, so the ETL code is built and deployed continuously. But it is impossible to do regression testing when testing is being done manually. This is turn causes lot of resources spending unnecessary amount of time testing during production deployment.
iCEDQ is an automated ETL Testing, Data Warehouse Testing, Data Migration Testing & Production Data Monitoring software. It’s unique ETL testing rules engine allows testers, developers or any other user to compare or validate millions of rows between different databases and files effectively and efficiently. It helps users identify the exact row and column having the data issue between two data sources.
iCEDQ can connect to any relational database, flat files or new big data databases like Hive, Impala and many other. Because of this functionality it can perform ETL testing easily as it now compare data across databases and files effectively.
Look at the list of data connectors for hadoop and databases.
Our unique auditing rules engine compares data in chunks, in memory. Below is an example of how our ETL testing rules engine works.
One of the benefits of automated ETL testing software is ability to do regression testing. iCEDQ enables users to perform regression testing through test suites which allows users to combine all the rules into one test suite.