Snowflake migrations involve moving thousands of tables, complex schema mappings, and billions of records from legacy databases. Manual testing using scripts and spreadsheets does not scale, misses record level issues and lacks the audit trails that regulated industries require.
iceDQ connects directly to your source databases and Snowflake using 50+ connectors and automates your entire migration testing lifecycle from schema validation to data reconciliation to post cutover monitoring, so your business can certify the migration with confidence.
Test ETL by Reconciling Target Data with Source Data
After ETL process execution, iceDQ connects to the source and target systems and applies rules to verify if the data is processed correctly.
- Connect to compare data across systems: Securely connect compare data across databases, files, cloud platforms, on premise platform, or data lakes using 100+ connectors without loading data into iceDQ.
- Compare at row and column level: Detect missing records, duplicates, and mismatched data values pinpointing every issue before going live.
- Business Reconciliation: Compare related business entities like orders vs shipments or accounts vs transactions to uncover functional data issues.
Validate Business logic and Data transformation by ETL
iceDQ validates input data before it enters the ETL pipeline, ensuring bad data is caught early and does not impact downstream systems.
- Validate Data Transformation: Verify complex transformations, filters and data aggregation with conditional expressions and fuzzy logic.
- Check data completeness: Ensure all expected records, columns, and mandatory fields are present before processing begins.
- Validate formats and types: Enforce expected data types, date formats, numeric ranges, and patterns.
- Verify reference data integrity: Cross check source values against lookup tables or master data to detect invalid codes, categories, or identifiers.
- Detect duplicates: Use iceDQ's dedicated duplicate identification template to find duplicate records and inconsistent data before they impact ETL accuracy.
- Out of the box checks and GenAI: Apply prebuilt checks or use iceDQ's GenAI assistant to generate checks from natural language prompts.
- Thresholds and rule severity: Set acceptable error limits and classify rule outcomes as warnings or failures to control pipeline behaviour.
- Verify transformation logic: Validate concatenations, calculations, aggregations, type conversions, and lookups against mapping specifications.
- User Defined Functions (UDFs): Encapsulate complex transformation logic in reusable UDFs across multiple rules.
- Validate business rules: Define rules based on organizational policies and business requirements to certify ETL output before deployment.
Validate Business logic and Data transformation by ETL
iceDQ validates input data before it enters the ETL pipeline, ensuring bad data is caught early and does not impact downstream systems.
- Validate Data Transformation: Verify complex transformations, filters and data aggregation with conditional expressions and fuzzy logic.
- Check data completeness: Ensure all expected records, columns, and mandatory fields are present before processing begins.
- Validate formats and types: Enforce expected data types, date formats, numeric ranges, and patterns.
- Verify reference data integrity: Cross check source values against lookup tables or master data to detect invalid codes, categories, or identifiers.
- Detect duplicates: Use iceDQ's dedicated duplicate identification template to find duplicate records and inconsistent data before they impact ETL accuracy.
- Out of the box checks and GenAI: Apply prebuilt checks or use iceDQ's GenAI assistant to generate checks from natural language prompts.
- Thresholds and rule severity: Set acceptable error limits and classify rule outcomes as warnings or failures to control pipeline behaviour.
- Verify transformation logic: Validate concatenations, calculations, aggregations, type conversions, and lookups against mapping specifications.
- User Defined Functions (UDFs): Encapsulate complex transformation logic in reusable UDFs across multiple rules.
- Validate business rules: Define rules based on organizational policies and business requirements to certify ETL output before deployment.
Generate Data Defect Report
For every rule executed, iceDQ generates a granular exception report that identifies the defective records and the specific cells where the data fails.
Types of Exceptions Identified:
- Partial Match: Records exist in both source and target, but one or more values do not match.
- Only in Source: Records that were not loaded by the ETL process.
- Only in Target: Extra records that were incorrectly processed into the destination.
- Defective Cells: Values with failed checks
Share with Stakeholders: Export exception reports to developers, business users, and management for collaborative issue resolution.
Agentic ETL Testing: From Discovery to Results
Describe your ETL testing needs in plain English. iceDQ's AI Agent discovers your pipeline, profiles your data, generates rules, schedules execution, and delivers results, all from a single conversation.
- Discover & Explore: Navigate connections, schemas, and tables automatically. Fetch sample data and metadata without opening a single screen.
- Profile & Recommend: Profile source and target tables and receive intelligent check recommendations based on your actual data.
- Author Rules in Bulk: Describe testing scenarios or upload mapping documents to generate hundreds of rules in a single interaction.
- Build Workflows & Schedule: Assemble rules into regression packs and schedule daily or release-cycle runs.
- Execute & Get Results: Trigger executions, track progress, and retrieve pass/fail status, row counts, and exception breakdowns through natural language.
Integrate iceDQ Rules into Your CI/CD Pipeline
iceDQ is built with an API first approach, enabling you to embed data testing directly into your CI/CD pipelines for continuous validation across development, QA, and production.
- Trigger rules from CI/CD tools: Execute iceDQ rules and workflows from Jenkins, Azure Pipelines, Bamboo, or any CI/CD tool using out of the box REST APIs.
- Automate testing on every deployment: Run data tests as part of your build and release pipeline so ETL changes are validated before reaching production.
- Regression test packs: Combine multiple rules into regression packs and trigger them on every release cycle to ensure new changes do not break existing pipelines.
- Report to test management tools: Push test results and exceptions to JIRA, Xray, HP ALM, Azure Test Plans, and ServiceNow for complete traceability.
- Flexible scheduling: Use iceDQ's built in scheduler for daily, weekly, or monthly runs, or trigger on demand from external orchestration tools like Airflow, Control M, Tidal, and AutoSys
- Defect management integration: Automatically log exceptions in tools like JIRA, HP ALM, and ServiceNow for end to end traceability.
- Track trends across executions: Compare exception counts across test runs to measure improvements and spot recurring issues.
PRODUCT HIGHLIGHTS
 |
Performance & Scalability |  |
Scale with multi-threaded parallel rule execution and Kubernetes cluster support for auto-scaling. No limitation on data volume. |
 |
Out-of-the-Box Connectivity |  |
Connect to 100+ databases, files, APIs, cloud platforms, and data lakes with an extensible connector framework. A full list of supported connectors is available in the product documentation. |
 |
Bulk Rule Generator |  |
Generate hundreds of rules in a single interaction using iceDQ's Agentic Assistant, reducing test creation effort by up to 90%. |
 |
AI-Assisted Check Creation | |
Describe your testing needs in natural language and let iceDQ's GenAI assistant auto-generate checks and SQL with no coding required. |
 |
File vs Database | |
Test file loads and extracts against database tables with automated record matching and column level exception reporting. |
 |
Database vs BI Reports | |
Compare data warehouse or database values against BI report outputs from Tableau, Power BI, Qlik, Cognos, or MicroStrategy to certify report accuracy. |
 |
Complex Transformations | |
Validate advanced transformation logic using SQL expressions, Groovy scripting, and reusable User Defined Functions (UDFs) for consistent, repeatable checks. |
 |
Script Rules | |
Automate pre and post processing tasks, make custom API calls, and build end to end integrations using Apache Groovy or Java scripts. |
Automate your ETL testing with iceDQ today.
Side CTA - Request a demo
FAQs
Is iceDQ dependent on a specific ETL tool?
No. iceDQ is ETL tool agnostic. It validates the data output produced by the ETL, not the ETL process itself. Customers use iceDQ with Informatica, Azure Data Factory, DataStage, Talend, SSIS, and more.
Is there a limitation on record processing?
No. iceDQ can test millions of records with no data volume limits. Performance scales with multi-threaded parallel execution and Kubernetes cluster support.
How is iceDQ different from other ETL testing tools?
iceDQ does not use database for processing. Most tools load data into a database for comparison, creating performance bottlenecks. iceDQ processes data in memory without loading it into any database, resulting in faster execution and not constrained by database limitations.
Can iceDQ handle complex transformation testing?
Yes. Multiple transformation logic within an ETL process can be recreated using iceDQ’s rules engine with SQL expressions, Groovy scripting, and User Defined Functions (UDFs).
How does iceDQ handle corner cases and edge scenarios?
iceDQ’s rules engine operates independently of any database. The scripting component using Groovy or Java can handle nearly 100% of testing scenarios including edge cases.
Does iceDQ support pre-processing and post-processing?
Yes. Script Rules enable data preparation before test execution and cleanup or notification workflows after completion.
How many connectors are supported?
50+ connectors including databases, files, APIs, cloud platforms, and data lakes with an extensible framework. A full list is available within the product documentation.
Can iceDQ test across databases, files, cloud, and on-premises environments?
Yes. iceDQ connects to both cloud and on-premises data sources, enabling cross-environment reconciliation across databases, files, APIs, and BI platforms.
Can iceDQ tests be triggered via API?
Yes. iceDQ provides REST APIs to trigger rule and workflow execution programmatically.
Can iceDQ integrate with external schedulers?
Yes. iceDQ has a built-in scheduler and supports integration with Airflow, Control-M, Tidal, and AutoSys through REST API and CLI.
Does iceDQ integrate with QA and ticketing platforms?
Yes. iceDQ integrates with Jira, Xray Test Management, Azure Test Plans, HP ALM, and ServiceNow.
Does iceDQ support CI/CD integration?
Yes. iceDQ supports CI/CD through its Jenkins plugin, REST APIs, and compatibility with Azure Pipelines, Bamboo, and Git.
Does iceDQ support rule versioning?
Yes. iceDQ maintains version history for rules, enabling teams to track changes and roll back when needed.
Can iceDQ show exceptions at the record and column level?
Yes. iceDQ generates granular exception reports at both record and column level, highlighting exactly where mismatches occurred.
Can iceDQ push exception reports into a database?
Yes. iceDQ can export exception data into an external database for further analysis or downstream integration.
Does iceDQ support Medallion architecture ETL testing?
Yes. iceDQ validates data across bronze, silver, and gold layers by reconciling between each layer and applying checks at every stage of the pipeline.