A data lake stores data before it enforces structure on it. A lakehouse adds table formats, ACID transactions, and schema enforcement on top of that same low-cost storage – but the underlying risk doesn’t disappear, it just moves. Schema drift that would fail loudly in a database fails silently in a lake.
iceDQ tests both layers of this architecture, the lake’s raw, schema-on-read storage and the lakehouse structured tables built on top of it. This ensures data is validated at ingestion, at every transformation, and at every medallion layer boundary, not just when it finally reaches a report.
OVERVIEW
Data Lake Testing
Data lakes store raw data in its native format, often before structure, ownership, or quality has been fully established. Testing has to validate the data on those terms:
- Schema-on-read validation: Confirm that data conforms to expected structure at the point of query or consumption, since a data lake doesn't enforce schema at write time the way a database does.
- File format & structure validation: Verify Parquet, Avro, ORC, JSON, and CSV files are well-formed, correctly partitioned, and free of corruption before they're consumed downstream.
- Ingestion validation: Check incoming files and streams for completeness, expected arrival, and structural consistency as they land in raw storage.
- Partition & directory structure checks: Confirm data lands in the correct partitions, since misplaced or mispartitioned files are a common source of silent query failures.
- Schema drift detection: Catch structural changes in incoming data - added, removed, or retyped fields - before they break downstream jobs that assume a fixed schema.
Data Lake Testing
Data lakes store raw data in its native format, often before structure, ownership, or quality has been fully established. Testing has to validate the data on those terms:
- Schema-on-read validation: Confirm that data conforms to expected structure at the point of query or consumption, since a data lake doesn't enforce schema at write time the way a database does.
- File format & structure validation: Verify Parquet, Avro, ORC, JSON, and CSV files are well-formed, correctly partitioned, and free of corruption before they're consumed downstream.
- Ingestion validation: Check incoming files and streams for completeness, expected arrival, and structural consistency as they land in raw storage.
- Partition & directory structure checks: Confirm data lands in the correct partitions, since misplaced or mispartitioned files are a common source of silent query failures.
- Schema drift detection: Catch structural changes in incoming data - added, removed, or retyped fields - before they break downstream jobs that assume a fixed schema.
Data Lakehouse Testing
A lakehouse layers table formats like Delta Lake, Apache Iceberg, and Apache Hudi on top of lake storage, adding the transactional guarantees and schema enforcement a lake lacks on its own. Testing here validates that structure is actually holding:
- Table format integrity: Validate that Delta, Iceberg, or Hudi tables are internally consistent - metadata, transaction logs, and underlying data files agree with each other.
- ACID transaction validation: Confirm that inserts, updates, deletes, and merges (upserts) complete correctly and leave the table in a consistent state.
- Schema enforcement & evolution testing: Verify that schema changes - adding a column, widening a type - are applied correctly and don't silently break existing consumers.
- Time travel & versioning checks: Validate that historical table versions are queryable and accurate, where the platform supports time travel.
- Query engine consistency: Confirm that the same lakehouse table returns consistent results whether queried through Spark, Databricks SQL, Snowflake, or another engine reading the same underlying files.
Medilion Layer Reconciliation Testing
Most lakehouse pipelines follow a medallion architecture - raw data refined in stages, from bronze through silver to gold. Every hop is a place where data can drift from the truth. iceDQ reconciles each transition, not just the final layer:
- Bronze layer validation: Confirm raw, ingested data is complete and matches the source system before any transformation is applied.
- Bronze-to-silver reconciliation: Validate that cleansing, deduplication, and conformance logic applied in the silver layer preserves record counts and field-level accuracy from bronze.
- Silver-to-gold reconciliation: Confirm that aggregations, business rules, and joins applied in the gold layer produce results that reconcile back to silver - not just directionally, but to the row or to the aggregate value.
- Cross-layer lineage checks: Trace a record or value from gold back to its bronze origin, so a discrepancy at the reporting layer can be diagnosed instead of just detected.
- End-to-end pipeline certification: Run all three layer transitions as a single connected test suite, so a pipeline is certified bronze-to-gold, not layer by layer in isolation.
PRODUCT HIGHLIGHTS
 | Lakehouse table formats: |  | Delta Lake, Apache Iceberg, Apache Hudi |
| Distributed processing & lakehouse platforms: | | Databricks, Apache Spark-based systems, Hive, Impala. |
| Cloud object storage: | | Amazon S3, Azure Data Lake Storage (ADLS Gen2), Google Cloud Storage. |
| Cloud data warehouses (lakehouse-adjacent): | | Snowflake, Google BigQuery, Amazon Redshift, Azure Synapse Analytics, Amazon Athena. |
| File formats: | | Parquet, Avro, ORC, JSON, XML, CSV, flat files — read and validated directly, without staging into a database. |
| Pushdown execution: | | Run validation and reconciliation logic natively inside the lakehouse or processing engine, avoiding the cost of extracting large tables for comparison. |
| Six rule types: | | Validation, Duplicate, Pushdown, Recon, Checksum, and Script cover everything from single-file checks to full bronze-to-gold reconciliation. |
| Row-level exception reports: | | Every failed check identifies the exact records and fields responsible — including which medallion layer introduced the discrepancy. |
| Low-code, no-code, agentic test authoring: | | Generate lakehouse testing rules from natural language prompts, mapping documents, or pipeline code using iceDQ's AI agent and MCP Server. |
| Workflow orchestration & scheduling: | | Chain layer-by-layer tests into a single pipeline-aligned workflow, triggered by schedule or pipeline event. |
| CI/CD integration: | | Gate deployment of lake and lakehouse pipelines on test results using exit codes, from any orchestration tool. |
Automate your Data Lake Testing with iceDQ.
Side CTA - Request a demo
FAQs: Data Lake Testing with iceDQ
What's the difference between a data lake and a data lakehouse?
A data lake stores raw data in its native format with schema applied only at read time. A lakehouse adds a transactional table schema format such as Delta Lake, Iceberg, or Hudi.
Can iceDQ test raw files before they're loaded into a lakehouse table?
Yes. iceDQ can validate files directly in cloud storage – checking format, structure, and completeness – before they’re loaded into a Delta, Iceberg, or Hudi table.
Can iceDQ read directly from Storage and skip the table format?
Yes, iceDQ treats it as a regular data lake and reads the files, but you lose lakehouse capabilities.
Can iceDQ read directly from the table format layer?
Yes. iceDQ can directly read Delta Lake, Apache Iceberg and Apache Hudi.
Why do iceDQ prefer reading data by referencing data catalog such as Unity Catalog , AWS Glue and Microsoft Fabric?
While iceDQ can refer the physical location “s3://prod-data/gold/customer/v4”, with a catalog it can read like a real table “SELECT * FROM gold.customer”
Why does a lakehouse need testing if it already enforces schema?
Lakehouse schema enforcement catches structural mismatches, not data mismatches. iceDQ validates incomplete, duplicated, or incorrectly transformed data.
What is medallion layer reconciliation testing?
It’s the practice of validating and reconciling data at each stage of a medallion architecture between bronze, silver, and gold making sure no data is lost.
Can iceDQ trace a data quality issue back to its origin across layers?
Yes. Cross-layer lineage checks let you trace a data quality issue in the gold layer back through silver to its bronze source, identifying root cause instead of just the symptom.
Does iceDQ support Delta Lake, Iceberg, and Hudi specifically?
Yes. iceDQ connects to lakehouse platforms built on these table formats and can validate table integrity, transaction consistency, and schema evolution alongside standard reconciliation and validation rules.
Do I need to move data out of the lakehouse to test it?
Not necessarily. Using pushdown execution, iceDQ can run tests natively inside the processing engine, so testing scales with the data instead of requiring extraction.
Can this run automatically as part of our pipeline deployment?
Yes. Every rule returns an exit code, so CI/CD pipelines and orchestration tools can gate downstream steps including promotion from one medallion layer to the next.