Understand Your ETL, Data Warehouse, Data Migration Issues

Apr72016

Agile Data Warehouse Testing & Data Migration Testing Featured - iCEDQ

Agile DW Testing & Data Migration Testing | Blog

Agile TestingBy Sandesh Gawande April 7, 2016

Development of a data warehouse, ETL, data migration or conversion always faces an ever-decreasing timeline. These implementations can take years to complete and users are not ready to wait that long.

The waterfall development model has been discarded in favor of the agile or development model. However, this has changed only one component of the Data Development Life Cycle. This does not mean that the quality of these processes or the data they produce is of high quality.

Share On :

Jun92014

ETL Testing and Data Quality Governance Software – The Missing Link Featured - iCEDQ

ETL Testing & Data Quality Governance Software | Blog

Data WarehouseBy Sandesh Gawande June 9, 2014

Data has become critical to the business. Hence, enterprises are investing time, money and resources in data-centric systems such as data warehouse, MDM, CRM & migration projects. However, all research done by independent agencies indicates that, There is such a high failure/delays in implementations of data-centric projects, Users still don’t trust data coming from data warehouses.

Share On :

Oct92019

DataOps Implementation Guide | Blog

DataOpsBy Sandesh Gawande October 9, 2019

The team is usually divided into development, QA, operations and business users. In almost all Data Integration projects, development teams try to build and test ETL processes, reports as fast as possible and throw the code across the wall to the operations teams and business users. However, when the data issues start appearing in production, business users become unhappy. They point fingers at Operations people, who in turn point fingers at QA people. The QA group then puts the blame on the development teams.

Share On :

Sep302015

Data Migration Testing Techniques to Migrate Data Successfully Featured - iCEDQ

Data Migration Testing Techniques | Blog

Data Migration TestingBy Sandesh Gawande September 30, 2015

We discussed the potential risks involved with the data migration process in our last iCEDQ insight. As previously mentioned, data migration is an important process where data from one system is transferred to a new, target system. The threat of data loss, data corruption, extended downtime, and application crashes make the data migration process risky. Amid these potential risks, a proper quality assurance process must be implemented to test the possibility of various risks of affecting the data migration process.

Share On :

Jan192017

Auditing – The Missing Element in Data Management and Data Governance Featured - iCEDQ

Audit Missing Element in Data Management & Data Governance | Blog

Data ManagementBy Sandesh Gawande January 19, 2017

Perhaps the most astonishing fact, however, is that IT has been blind for so long to the need for monitoring and metering (Auditing) for data health, and yet this fundamental engineering concept. For instance, Figure 1 illustrates a centrifugal steam engine governor.

Share On :

Sep212017

AML Software Implementation & Production Monitoring with iCEDQ DataOps Platform Featured

AML Software Implementation & Monitoring | Blog

DataOpsBy Sandesh Gawande September 21, 2017

AML software is a downstream system that consumes data from multiple sources. AML software analyzes data based on compliance models. This results in suspicious activity reports. Further, they also monitor data for regulations such as FATCA, trade restrictions, sanctions, and watch list.

However, if the incoming data is not good then these results cannot be relied upon. It is basically at the mercy of upstream systems.

iCEDQ is an in-memory data audit rules engine. It sits between the data sources and downstream systems such as ALM. It can validate and reconciliation data coming from multiple data sources. Thus, managing data issues before it affect the downstream system.

Share On :