Before initiating production downloads, the CDI Data Acquisition System validates repository workflows using a representative test manifest.
Download Manifest
↓
Test Manifest
↓
ENA Validation
↓
NCBI/SRA Validation
↓
Production ReadinessThe objective of repository validation is to confirm that:
Successful repository validation provides confidence that the workflow is functioning correctly before large-scale acquisition begins.
Example validation checks:
wc -l data/manifests/test-manifest.tsvfind data/raw/fastq \
-name "*.fastq.gz" \
| wc -lValidation summary:
Test Manifest Records 6
Downloaded FASTQ Files 6
Status PASSOverall result:
Repository Validation Status
ENA: PASS
NCBI/SRA: PASS
READY FOR PRODUCTION DOWNLOADThe CDI Data Acquisition System uses a multi-layer validation strategy.
flowchart TD
A[Repository Validation]
--> B[File Count Validation]
B --> C[Integrity Validation]
C --> D[Metadata Validation]
D --> E[FASTQ Structure Validation]
E --> F[Validation Report]flowchart TD A[Repository Validation] --> B[File Count Validation] B --> C[Integrity Validation] C --> D[Metadata Validation] D --> E[FASTQ Structure Validation] E --> F[Validation Report]
Each validation layer addresses a different source of acquisition risk and contributes to overall dataset reliability.
data/
├── metadata/
├── manifests/
├── logs/
├── raw/
│ ├── sra/
│ └── fastq/
│
├── inventory/
│ └── fastq-inventory.tsv
│
└── validation/
├── file-validation.tsv
├── metadata-validation.tsv
└── validation-report.tsvThe validation directory stores outputs generated during the validation process.
File-level validation focuses on verifying the integrity and completeness of downloaded sequencing assets.
Common checks include:
These checks help identify incomplete, missing, or corrupted downloads before downstream analysis begins.
Count downloaded FASTQ files:
find data/raw/fastq \
-name "*.fastq.gz" \
| wc -lCompare against the expected count recorded in the download manifest:
wc -l data/manifests/ena-fastq-urls.txtExpected outcome:
Manifest Files = Downloaded FilesAgreement between expected and observed file counts provides the first indication that the download process completed successfully.
Whenever repositories provide checksums, downloaded files should be verified before proceeding to downstream analyses.
Common checksum methods include:
Checksum validation confirms that downloaded files match the versions distributed by the repository.
Linux:
md5sum sample.fastq.gzmacOS:
md5 sample.fastq.gzValidation principle:
Repository Checksum
↓
Local Checksum
↓
Match
↓
Validated FileChecksum verification helps detect:
Most sequencing files are distributed as compressed archives.
Before analysis, compressed files should be tested for corruption.
gzip -tv data/raw/fastq/*.gzExpected output:
OKFiles failing this check should be re-downloaded before further processing.
Researchers should verify that expected sample counts match downloaded data.
Examples:
Expected Runs: 188
Downloaded FASTQ Files: 188
Status: PASSor for paired-end studies:
Expected Runs: 54
Expected FASTQ Files: 108
Downloaded FASTQ Files: 108
Status: PASSDiscrepancies should be investigated before proceeding.
Validation should compare downloaded files against metadata records and download manifests.
Questions to consider:
Consistency between metadata and sequencing data is critical for reproducible dataset construction.
Duplicate samples may occur when:
Duplicate detection helps prevent inflated sample counts and biased analyses.
For sequencing projects, validating FASTQ structure is particularly important.
Each FASTQ record should contain four lines:
@SEQ_ID
SEQUENCE
+
QUALITYQuick inspection:
zcat sample.fastq.gz | headThis confirms that the file can be decompressed and follows the expected FASTQ structure.
A more robust validation can be performed using SeqKit:
seqkit stats data/raw/fastq/*.fastq.gzSuccessful execution confirms that the files can be parsed by downstream bioinformatics tools.
Many sequencing studies generate paired-end reads.
Each sample should contain:
sample_1.fastq.gz
sample_2.fastq.gzVerify that forward and reverse files exist for every sample:
ls data/raw/fastq/*_1.fastq.gz | wc -l
ls data/raw/fastq/*_2.fastq.gz | wc -lExpected outcome:
Forward Reads = Reverse ReadsThis validation is particularly important for paired-end datasets such as PRJNA802976.
Different repositories may provide data in various formats.
Common examples include:
Researchers should verify that files are in the expected format and can be successfully opened, decompressed, or parsed.
To support reproducibility, validation should be automated whenever possible.
Example system component:
scripts/bash/05d-verify-downloads.shWorkflow:
Repository Validation
↓
File Count Check
↓
FASTQ Inspection
↓
SeqKit Validation
↓
Repository Comparison
↓
Validation ReportAutomated validation minimizes manual errors and ensures consistency across projects.
Validation results should be recorded as part of the acquisition workflow.
Recommended records include:
Documentation improves transparency, reproducibility, and auditability.
To validate the CDI Data Acquisition System, paired-end amplicon sequencing data from BioProject PRJNA802976 were retrieved independently from both ENA and NCBI.
Validation workflow:
Expected Sequencing Runs
↓
ENA Download
↓
NCBI Download
↓
FASTQ Verification
↓
Repository Comparison
↓
Validated DatasetOnly after successful validation should samples be considered for reference dataset assembly.
Researchers frequently encounter:
A structured validation workflow helps address these challenges systematically.
Data validation is not merely a technical exercise.
It reduces the risk of:
Validation protects the integrity of downstream scientific work.
The CDI Data Acquisition System was validated using three paired-end sequencing runs retrieved independently from ENA and NCBI.
Repository validation summary:
Repository Validation Summary
ENA FASTQ files: 6
NCBI FASTQ files: 6
Verification complete.SeqKit validation confirmed successful parsing of all FASTQ files.
Example results:
Run Accession Reads Read Length
SRR17868090 52,037 301 bp
SRR17868091 102,787 301 bp
SRR17868092 97,511 301 bpInterpretation:
FASTQ File Count
Expected: 6
Observed: 6
Status: PASSPaired-End Validation
Forward Reads: 3
Reverse Reads: 3
Status: PASSFASTQ Validation
Readable Files: 6
Status: PASSRepository Comparison
ENA Read Counts = NCBI Read Counts
ENA Read Lengths = NCBI Read Lengths
Status: PASSOverall result:
Repository Validation: PASS
File Validation: PASS
FASTQ Validation: PASS
Repository Comparison: PASS
Overall Status:
VALIDATEDThe validation report provides a reproducible record of dataset integrity and serves as the final quality gate before cloud transfer, reference dataset assembly, and downstream analysis.
Data validation transforms downloaded sequencing files into trusted analytical assets.
Repository Validation
↓
File Validation
↓
FASTQ Validation
↓
Repository Comparison
↓
Validated DatasetA validated dataset provides confidence that acquisition objectives have been achieved and that downstream analyses are based on complete and reliable data.
Once datasets have been validated, the next challenge is managing storage, transfers, and scalable access to acquired data.
In the next chapter, we explore cloud storage and transfer strategies that support reproducible and collaborative data acquisition workflows while preserving the integrity of validated datasets.