All About ETL Testing, Data Migration & Data Warehouse Testing

blog - iCEDQ
ETL Testing and Data Quality Governance Software – The Missing Link-iCEDQ

ETL Testing and Data Quality Governance Software – The Missing Link

Data has become critical to the business. Hence, enterprises are investing time, money and resources in data-centric systems such as data warehouse, MDM, CRM & migration projects. However, all research done by independent agencies indicates that, There is such a high failure/delays in implementations of data-centric projects, Users still don’t trust data coming from data warehouses.

Share On :
Read more
How to test an ETL Process – Basic Concept-iCEDQ

How to Test an ETL Process – Basic Concept

A source table has an individual and corporate customer. The requirement is that an ETL process should take the corporate customers only and populate the data in a target table. The test cases required to validate the ETL process by reconciling the source (input) and target (Output) data. The transformation rule also specifies that output should only have corporate customers.

Share On :
Read more
iCEDQ Platform vs Data Quality Tools-iCEDQ

iCEDQ Platform vs Data Quality Tools

iCEDQ is a Quality Assurance and Test Automation platform for data-centric projects and processes such as data warehouse, CRM, data migration & conversion, ETL. It certifies the ETL processes or migration by effective ETL Testing and Data Migration Testing. The product can be further used for monitoring the data processes in production. The product emphasizes mainly process quality.

The major difference between iCEDQ & other Data Quality tools is the purpose they serve. iCEDQ is a test automation platform for process quality whereas other Data Quality tools are a combination of data profiling & fixing/ correction tool used in production.

Share On :
Read more
The Data Migration Process & the Potential Risks-iCEDQ

The Data Migration Process & the Potential Risks

Data migration is the process of transferring data from one system to another system, known as the target system, using a variety of tools and techniques.
Below are the different types of migrations which are encountered in different enterprises: Database Migration: This involves moving from one database software to another. E.g. your organization just bought HP Vertica and is planning to from MySQL to HP Vertica. Database Version Upgrade: This involves upgrading the older version of the database to the latest or most current available database version.

Share On :
Read more
Data Migration Testing Techniques to Migrate Data Successfully-iCEDQ

Data Migration Testing Techniques to Migrate Data Successfully

We discussed the potential risks involved with the data migration process in our last iCEDQ insight. As previously mentioned, data migration is an important process where data from one system is transferred to a new, target system. The threat of data loss, data corruption, extended downtime, and application crashes make the data migration process risky. Amid these potential risks, a proper quality assurance process must be implemented to test the possibility of various risks of affecting the data migration process.

Share On :
Read more
Agile Data Warehouse Testing & Data Migration Testing-iCEDQ

Agile Data Warehouse Testing & Data Migration Testing

Development of a data warehouse, ETL, data migration or conversion always faces an ever-decreasing timeline. These implementations can take years to complete and users are not ready to wait that long.

The waterfall development model has been discarded in favor of the agile or development model. However, this has changed only one component of the Data Development Life Cycle. This does not mean that the quality of these processes or the data they produce is of high quality.

Share On :
Read more
QA Challenges in Data Integration Projects-iCEDQ

QA Challenges in Data Integration Projects

Quality Assurance (QA) is a very important component of any data-centric application project.  Projects such as data warehouse, data migration, ETL, Data Lakes and MDM are no exception. The Majority of these projects are the multi-year and multi-million dollar in nature due to the amount of work and products required. Therefore it’s necessary to have proper planning for QA in place to avoid late discoveries of process and data errors.

While the methodologies of testing have evolved considerably over the years, the science of QA in data integration project has not. In this article, we’ll focus on some of the key challenges with data warehouse testing, data migration testing and ETL testing.

Share On :
Read more
What Are The Challenges Of A Data Factory-iCEDQ

What Are The Challenges Of A Data Factory

DataOps is a set of practices and tools used by Big Data teams to increase velocity, reliability, and quality of data analytics. It emphasizes communication, collaboration, integration, automation, measurement and cooperation between data scientists, analysts, data/ETL (extract, transform, load) engineers, information technology (IT), and quality assurance/governance. It aims to help organizations rapidly produce insight, turn that insight into operational tools, and continuously improve analytic operations and performance.

Share On :
Read more
iCEDQ your Gatekeeper for Data Issues-iCEDQ

iCEDQ your Gatekeeper for Data Issues

The Challenges: Today’s organizations have thousands of data integration (ETL) processes constantly moving silos of data from various operational and/or external data sources to downstream applications.
Since the downstream system doesn’t have control over incoming data or the process, it can cause serious data issues due to:
The quality of the data depends on the upstream systems,
The ETL jobs may not process the data correctly.

Share On :
Read more