Your organization is exchanging thousands of data files in multiple formats every day across applications, cloud platforms, and business processes. Every data issue, wrong file format, or data corruption can lead to reporting errors, failed integrations, compliance risks, and costly business disruptions.
iceDQ validates, reconciles with databases, and tests data files to ensure quality and format compliance in both development and production environments.
OVERVIEW
File Validation Testing
Check the content of the file itself, independent of format or source:
- Duplicate checks: Catch duplicate records that would distort downstream analysis and reporting.
- Completeness checks: Detect nulls and missing values so incomplete records don't silently propagate.
- Format verification: Confirm data types, precision, and length match what's expected, field by field.
- Integrity checks: Flag anomalies like unexpected binary characters or corrupted content.
- Reference data consistency: Verify that codes, categories, and reference values match the expected list, so nonstandard or invalid values get caught before they reach reporting.
File Validation Testing
Check the content of the file itself, independent of format or source:
- Duplicate checks: Catch duplicate records that would distort downstream analysis and reporting.
- Completeness checks: Detect nulls and missing values so incomplete records don't silently propagate.
- Format verification: Confirm data types, precision, and length match what's expected, field by field.
- Integrity checks: Flag anomalies like unexpected binary characters or corrupted content.
- Reference data consistency: Verify that codes, categories, and reference values match the expected list, so nonstandard or invalid values get caught before they reach reporting.
File Reconciliation Testing
Confirm the file agrees with the other places the same data lives:
- File to file: Compare two files against each other - useful for validating extracts, exports, or vendor deliveries.
- File to database: Reconcile file content against the database or warehouse it's meant to load into or originate from.
- File to previous file: Compare against a prior version of the same file to catch unexpected drift over time.
File Format Testing
Confirm the file's structure is what your pipeline expects before it's processed:
- Column structure checks: Detect column additions, removals, or reordering that would otherwise break downstream parsing.
- File format checks: Validate delimiters, encodings, and format conventions, which can silently drift between deliveries - especially from external or third-party sources.
- Binary content checks: Identify unexpected binary data inside what should be a plain-text file.
File Delivery Testing
Confirm the file arrived, and arrived intact:
- Arrival time: Verify the file landed within its expected window - late files delay downstream decisions and processes.
- Empty file check: Flag files that arrived with no data, which is functionally the same as a missing file.
- File completeness: Check for truncation or partial data loss during transmission.
- Manifest / control file match: Compare file content against its footer record or control file to confirm nothing was dropped in transit.
- Duplicate delivery check: Compare against previously received files to avoid processing the same file twice.
PRODUCT HIGHLIGHTS
 | SQL for Files: |  | Write standard SQL directly against file data - no need to load files into a database first. |
| In-Memory Processing: | | Files are processed in real-time without staging into an RDBMS. |
| No Size Limit: | | Process files from a few megabytes to multiple terabytes without performance degradation. |
| Broad Format Support: | | Avro, Excel, flat files, CSV, fixed-width files, Google Sheets, JSON, Parquet, SAS data sets, SAS XPT, XML, EBCDIC, COBOL, and more - including legacy and mainframe formats. |
| Cloud Storage Connectivity: | | Connect directly to Amazon S3, Azure Data Lake Storage (ADLS Gen2), and Google Cloud Storage, alongside FTP, SFTP, shared drives, and SharePoint. |
| Cross-Source Reconciliation: | | Reconcile files against other files, databases, or APIs - with no restrictions on source-type combinations. |
| Low-Code, No-Code, and GenAI: | | Use prebuilt checks for completeness, range, pattern, datatype, and length, or auto-generate checks and SQL with iceDQ's GenAI assistant. |
| Enterprise-Grade Security: | | Supports authentication and access protocols including LDAP, Active Directory / ADFS, Kerberos, SSO, OAuth 2.0, OIDC, SAML 2.0, MFA, Azure Managed Identity, AWS IAM, GCP service accounts, SSH keys, SSL/TLS and client certificates, and SAS tokens. |
Automate your File Testing Tool with iceDQ.
Side CTA - Request a demo
FAQs: File Testing Tool with iceDQ
What types of files does iceDQ support?
Nearly all modern and legacy file formats, including mainframe formats like EBCDIC and COBOL, alongside standard formats such as CSV, JSON, XML, Parquet, and Excel.
Can I run SQL queries against files?
Yes. iceDQ provides Text-JDBC adapters that let you run SQL directly against supported file formats, without loading them into a database first.
What file storage platforms are supported?
iceDQ connects to on-premises, cloud, and data lake storage, including Amazon S3, Azure Data Lake Storage (ADLS), Google Cloud Storage (GCS), FTP, SFTP, shared drives, and SharePoint.
Does iceDQ support authentication when accessing files in storage?
Yes. iceDQ supports enterprise and cloud authentication protocols, including LDAP, Active Directory, ADFS, Kerberos, SSO, OAuth 2.0, OIDC, SAML 2.0, MFA, and cloud-native identity mechanisms such as AWS IAM, Azure Managed Identity, and GCP service accounts.
Are there any file size limitations?
No. iceDQ processes files incrementally, so it can handle files ranging from gigabytes to terabytes without loading the entire file into memory.
Can I create validation rules for files the same way I would for a database?
Yes. iceDQ’s validation engine is independent of the underlying data source, so any rule that applies to database data can be applied to file-based data.
Can iceDQ reconcile files against databases, not just other files?
Yes. iceDQ supports reconciliation across any combination of source types – file-to-file, file-to-database, database-to-database, and file-to-API.