Once studies have been identified, accession systems understood, and metadata evaluated, the next step is acquiring the sequencing data.
Data download transforms study selection and metadata assets into locally accessible sequencing files that can be validated, organized, transferred, and assembled into downstream reference datasets. A reproducible acquisition workflow requires more than simply downloading files. Researchers must understand where files reside, how they are accessed, how downloads can be resumed, and how file integrity can be verified.
By the end of this chapter, you will be able to:
Metadata acquisition tells us what data exist.
The Data Download System retrieves the actual sequencing files required for downstream analysis.
Metadata Assets
↓
Download Manifest
↓
Repository Selection
↓
Download Execution
↓
Integrity Verification
↓
FASTQ Inventory
↓
Data Validation SystemThe quality of downstream analyses depends on the quality and completeness of acquired data. A successful download workflow should ensure that:
flowchart TD
A[Metadata Assets]
--> B[Download Manifest]
B --> C[Repository Selection]
C --> D[Download Execution]
D --> E[Integrity Verification]
E --> F[FASTQ Inventory]
F --> G[Data Validation System]flowchart TD A[Metadata Assets] --> B[Download Manifest] B --> C[Repository Selection] C --> D[Download Execution] D --> E[Integrity Verification] E --> F[FASTQ Inventory] F --> G[Data Validation System]
The Data Download System begins with metadata assets generated in Chapter 04 and ends with a documented inventory of downloaded files.
The CDI Data Acquisition System uses a single Conda environment throughout the workflow.
conda activate cdi-data-acquisitionThis avoids unnecessary environment switching and ensures that metadata acquisition, data download, validation, and dataset assembly operate within a consistent software environment.
The environment used throughout this book is defined as:
name: cdi-data-acquisition
channels:
- conda-forge
- bioconda
dependencies:
- entrez-direct
- pysradb
- sra-tools
- wget
- curl
- seqkit
- csvtk
- jq
- parallel
- pigz
- pythonVerify the key tools required by the Data Download System.
python --version
prefetch --version
fasterq-dump --version
vdb-validate --version
pysradb --help
seqkit version
csvtk version
jq --version
parallel --version
which esearchSuccessful execution confirms that the shared CDI data acquisition environment is ready for metadata acquisition, data download, and data validation workflows.
Public repositories provide access to a variety of downloadable file types.
| Repository | Typical Data | Common Role in This System |
|---|---|---|
| SRA | Raw sequencing reads | Run accession source and SRA Toolkit download |
| ENA | Raw sequencing reads and FASTQ URLs | Direct FASTQ retrieval and checksum validation |
| GEO | Processed and supplementary data | Study-level context and supplementary files |
| MGnify | Processed microbiome outputs | Processed microbiome reference outputs |
| GWAS Catalog | Summary statistics and study information | Association study discovery and summary statistics |
For large public sequencing datasets, the two most important download sources are usually ENA and NCBI SRA.
Researchers frequently encounter:
For raw sequencing acquisition, the most common target output is FASTQ. In microbiome, RNA-seq, metagenomics, and many other NGS workflows, FASTQ files serve as the starting point for downstream quality control and analysis.
A download manifest represents the authoritative list of files to retrieve from a public repository.
The manifest is generated from metadata assets acquired in Chapter 04 and serves as the primary input to the Data Download System.
A simple accession manifest may contain only run identifiers:
SRR3725378
SRR3725379
SRR3725382However, download workflows are often more reliable when manifests contain file-level information.
Example ENA-derived manifest:
run_accession fastq_ftp fastq_md5
SRR3725378 ftp.sra.ebi.ac.uk/.../SRR3725378.fastq.gz e9757b1bf2ad...
SRR3725379 ftp.sra.ebi.ac.uk/.../SRR3725379.fastq.gz e5ce06d20ddb...
SRR3725382 ftp.sra.ebi.ac.uk/.../SRR3725382.fastq.gz 1fb16c5b50ab...File-level manifests provide several advantages:
Recommended project structure:
data/
├── metadata/
│ ├── runinfo-PRJNA322554.csv
│ ├── ena-PRJNA322554.tsv
│ └── srr-accessions.txt
│
└── manifests/
└── download-manifest.tsvWithin the CDI Data Acquisition System, the download manifest acts as a system contract.
Metadata Assets
↓
Download Manifest
↓
FASTQ Retrieval
↓
File Validation
↓
FASTQ InventoryThe manifest defines what should be downloaded, where files are located, and how completeness and integrity will later be verified.
The CDI Data Acquisition System uses a staged download strategy that validates repository workflows before initiating large-scale data retrieval.
Metadata Assets
↓
Download Manifest
↓
Create Test Manifest
↓
Repository Validation
├── ENA
└── NCBI/SRA
↓
Production DownloadThis approach reduces risk by ensuring that:
Rather than immediately downloading an entire dataset, a small representative subset is first used to validate repository access and download workflows.
download-manifest.tsv
↓
test-manifest.tsv
↓
Repository Validation
↓
Production DownloadSuccessful test downloads provide confidence that the acquisition workflow is functioning correctly before scaling to the complete dataset.
This strategy is particularly valuable for large studies containing hundreds or thousands of sequencing runs, where failed downloads can consume substantial time, storage, and bandwidth resources.
Only after successful repository validation should production downloads be initiated.
Several tools are commonly used to retrieve public omics datasets.
Different repositories expose data through different interfaces, and no single tool is optimal for every situation. The CDI Data Acquisition System therefore supports both ENA-based FASTQ retrieval and NCBI SRA workflows.
The SRA Toolkit is commonly used for retrieving sequencing data from the NCBI Sequence Read Archive.
| Tool | Purpose |
|---|---|
prefetch |
Download SRA files |
fasterq-dump |
Convert SRA files to FASTQ |
vdb-validate |
Validate downloaded SRA files |
The SRA Toolkit is particularly useful when direct FASTQ files are unavailable or when repository-specific SRA formats are preferred.
ENA and other repositories often expose direct file links that can be retrieved with common command-line tools.
| Tool | Purpose |
|---|---|
wget |
Download files from HTTP, HTTPS, or FTP links |
curl |
Retrieve files and web resources |
md5sum |
Verify MD5 checksums on Linux |
md5 |
Verify MD5 checksums on macOS |
Within this guide, ENA serves as the preferred FASTQ download source because it provides direct access to sequencing files together with file-level metadata such as checksums.
Some repositories also provide access through cloud infrastructure and object storage systems.
Cloud-based acquisition can reduce transfer times for large datasets and may simplify integration with downstream analysis environments.
Cloud transfer workflows are introduced later in the book after local download, validation, and inventory-generation concepts have been established.
Local Download
↓
Data Validation
↓
Cloud Storage and Transfer
↓
Reference Dataset AssemblyActivate the shared CDI data acquisition environment:
conda activate cdi-data-acquisitionInstall SRA Toolkit if it is not already available:
conda install -c bioconda sra-toolsVerify installation:
prefetch --version
fasterq-dump --version
vdb-validate --versionExample output:
prefetch : 3.4.1
fasterq-dump : 3.4.1
vdb-validate : 3.4.1If the commands return version information, the toolkit is available in the active environment and ready for use.
The Data Download System relies on several command-line utilities for file retrieval and validation.
Verify that curl and wget are available:
curl --version
wget --versionIf wget is missing, install it within the shared environment:
conda install -c conda-forge wgetVerify checksum utilities.
Linux:
md5sum --versionmacOS:
which md5The exact checksum command differs across operating systems, but the purpose remains the same: verify that downloaded files match the expected repository checksums.
Downloaded files should be organized systematically to support validation, inventory generation, cloud transfer, and dataset assembly.
Recommended structure:
data/
├── metadata/
│ ├── runinfo-PRJNA322554.csv
│ ├── ena-PRJNA322554.tsv
│ └── srr-accessions.txt
│
├── manifests/
│ ├── download-manifest.tsv
│ └── test-manifest.tsv
│
├── raw/
│ ├── sra/
│ └── fastq/
│
├── logs/
│ ├── download-ena.log
│ ├── download-ncbi.log
│ └── checksum.log
│
└── inventory/
└── fastq-inventory.tsvA consistent directory structure improves reproducibility, simplifies validation, and creates a clear handoff between components of the CDI Data Acquisition System.
Create the directories:
mkdir -p data/manifests
mkdir -p data/raw/sra
mkdir -p data/raw/fastq
mkdir -p data/logs
mkdir -p data/inventoryThe Data Download System begins with the run accession manifest generated during metadata acquisition.
Copy the manifest into the dedicated manifests directory:
cp data/metadata/srr-accessions.txt \
data/manifests/download-manifest.tsvPreview the manifest:
head data/manifests/download-manifest.tsvCount accessions:
wc -l data/manifests/download-manifest.tsvExpected output:
188The accession count becomes one of the first validation checks performed during data download and validation workflows.
A smaller test manifest can then be generated from the production manifest for repository validation.
download-manifest.tsv
↓
test-manifest.tsv
↓
Repository Validation
↓
Production DownloadThe CDI Data Download System is implemented through a set of reusable Bash scripts.
scripts/bash/
├── 05a-build-download-manifest.sh
│ └── Create production and test manifests
│
├── 05b-download-ena-fastq.sh
│ └── Download FASTQ files from ENA
│
├── 05c-download-ncbi-sra.sh
│ └── Download and convert SRA accessions
│
├── 05d-verify-downloads.sh
│ └── Verify downloaded FASTQ assets
│
└── 05e-build-fastq-inventory.sh
└── Generate FASTQ inventory reportsThese scripts operate on the metadata assets, manifests, and directory structure established in the previous sections and provide a reproducible implementation of the Data Download System.
Together they support:
Metadata Assets
↓
Download Manifest
↓
Repository Validation
↓
Production Download
↓
FASTQ Verification
↓
FASTQ InventoryBefore executing large-scale downloads, it is good practice to validate the environment, repository access, and download workflow using a small test manifest. The following sections demonstrate this validation process before scaling to production acquisition.
Before initiating production downloads, validate the workflow using a small test manifest derived from the full download manifest.
The CDI Data Acquisition System uses test downloads to verify:
download-manifest.tsv
↓
test-manifest.tsv
↓
Repository Validation
↓
Production DownloadPreview the test manifest:
head data/manifests/test-manifest.tsvCount test records:
wc -l data/manifests/test-manifest.tsvExample output:
6The test manifest should contain a small representative subset of the full dataset.
Execute the ENA download workflow:
bash scripts/bash/05b-download-ena-fastq.shExample output:
Production URL count:
188
Creating test URL list...
Test URL count:
6
Downloading test files...Successful execution retrieves a small collection of FASTQ files and validates repository access before large-scale acquisition.
Inspect downloaded files:
find data/raw/fastq -name "*.fastq.gz"Example output:
data/raw/fastq/SRR3725378.fastq.gz
data/raw/fastq/SRR3725379.fastq.gz
...Perform a preliminary validation using SeqKit:
seqkit stats data/raw/fastq/*.fastq.gzExample output:
file format type num_seqs
SRR3725378.fastq.gz FASTQ DNA ...
SRR3725379.fastq.gz FASTQ DNA ...Successful execution confirms that downloaded files are valid FASTQ assets and can be parsed by downstream tools.
This test confirms that:
At this stage we have demonstrated:
Metadata Assets
↓
Download Manifest
↓
Test Manifest
↓
FASTQ Retrieval
↓
Basic ValidationThis preliminary verification provides confidence that the acquisition workflow is functioning correctly.
Rather than immediately downloading hundreds of sequencing runs, validating the workflow on a small representative subset helps confirm that repository access, directory structure, software installation, and validation procedures are functioning as expected.
Once the workflow has been verified, the same approach can be scaled to the complete download manifest through the CDI Data Download System scripts.
The following sections describe two common strategies for large-scale data acquisition: direct FASTQ retrieval from ENA and SRA-based retrieval using the NCBI SRA Toolkit.
The recommended strategy is to use NCBI for study discovery and metadata acquisition, then use ENA for direct FASTQ retrieval whenever suitable file URLs and checksums are available.
Study Discovery
↓
NCBI
↓
Metadata Acquisition
↓
ENA
↓
Download Planning
↓
Data Download| Scenario | Preferred Repository |
|---|---|
| Metadata discovery | NCBI |
| FASTQ download with direct URLs | ENA |
| Checksum validation | ENA |
| Missing FASTQ URLs | NCBI SRA Toolkit |
| SRA-native retrieval | NCBI SRA Toolkit |
| Controlled-access data | Repository-specific process |
| Supplementary processed files | GEO, ENA, project archive, or journal supplement |
This strategy avoids treating all repositories as interchangeable. Each repository contributes differently to the overall acquisition system.
Throughout this guide, ENA serves as the preferred FASTQ download source, while NCBI provides study discovery, metadata retrieval, accession relationships, and alternative download pathways when direct FASTQ access is unavailable.
ENA is often the preferred source for sequencing data acquisition because it provides direct FASTQ download links together with file-level metadata and checksums.
ENA Metadata
↓
FASTQ URLs
↓
Download Manifest
↓
wget
↓
FASTQ Files
↓
Checksum VerificationThe ENA metadata file may contain fields such as:
run_accessionfastq_ftpfastq_md5fastq_bytesInspect the metadata structure before generating download manifests:
head -n 1 data/metadata/ena-PRJNA322554.tsvExample output:
run_accession sample_accession fastq_ftp fastq_md5The ENA workflow is implemented through:
bash scripts/bash/05b-download-ena-fastq.shThe script performs the following steps:
ENA Metadata
↓
Extract FASTQ URLs
↓
Create Production Manifest
↓
Create Test Manifest
↓
Test Download
↓
Production DownloadGenerated outputs include:
data/manifests/ena-fastq-urls.txt
data/manifests/ena-fastq-test-urls.txt
data/raw/fastq/
data/logs/download-ena.logThe script uses:
wget --continuewhich allows interrupted downloads to resume when supported by the repository.
Because ENA provides direct FASTQ files and associated checksums, it serves as the preferred download pathway throughout this guide whenever suitable metadata are available.
When direct FASTQ files are unavailable or SRA-native retrieval is preferred, the NCBI SRA Toolkit workflow can be used.
Run Accessions
↓
prefetch
↓
SRA Files
↓
fasterq-dump
↓
FASTQ FilesThe NCBI SRA workflow is implemented through:
bash scripts/bash/05c-download-ncbi-sra.shBy default, the script operates on the test manifest:
data/manifests/test-manifest.tsvThis design makes the workflow safe for testing and system validation.
To process the full dataset:
bash scripts/bash/05c-download-ncbi-sra.sh \
data/manifests/download-manifest.tsvThe workflow performs three major steps:
Download SRA Files
↓
Convert to FASTQ
↓
Compress FASTQ AssetsOutputs are written to:
data/raw/sra/
data/raw/fastq/
data/logs/download-ncbi.logThis test-first strategy helps prevent accidental large-scale downloads before repository access, storage locations, and software installation have been validated.
For large datasets, FASTQ compression can require substantial time and storage resources. Utilities such as pigz can accelerate compression through parallel processing.
Regardless of repository source, downloaded files should be validated before entering downstream workflows.
Common post-download checks include:
The next section focuses on checksum verification, one of the most important safeguards against incomplete or corrupted downloads.
Whenever repositories provide checksums, downloaded files should be verified before proceeding to downstream analyses.
ENA commonly distributes MD5 checksums alongside FASTQ download links.
Example checksum manifest:
e9757b1bf2adf589d3d599dc2ecfe982 SRR3725378.fastq.gz
e5ce06d20ddb091c1fe4463be981412a SRR3725379.fastq.gzLinux:
cd data/raw/fastq
md5sum -c \
../../manifests/ena-md5.txt \
| tee ../../logs/checksum.log
cd -macOS:
md5 SRR3725378.fastq.gzThe exact command varies by operating system, but the underlying principle remains the same:
Repository Checksum
↓
Local Checksum
↓
Match
↓
Validated FileChecksum verification helps detect:
A file should only proceed to downstream validation and analysis if its checksum matches the value reported by the repository.
A FASTQ inventory provides a reproducible record of sequencing assets acquired during the download process.
The inventory serves as the bridge between data acquisition and data validation by documenting the files that were successfully retrieved from public repositories.
Recommended output:
data/
└── inventory/
└── fastq-inventory.tsvThe inventory workflow is implemented through:
bash scripts/bash/05e-build-fastq-inventory.shThe script scans the FASTQ directory and generates a tabular inventory containing file names and file sizes.
Preview the inventory:
head data/inventory/fastq-inventory.tsvCount inventory records:
wc -l data/inventory/fastq-inventory.tsvExample output:
file size_bytes
data/raw/fastq/SRR3725378.fastq.gz 8234567
data/raw/fastq/SRR3725379.fastq.gz 9156789
data/raw/fastq/SRR3725382.fastq.gz 8012345The inventory provides a concise summary of acquired sequencing assets and supports:
Inventory generation provides a simple but effective validation checkpoint.
Downloaded FASTQ Files
↓
FASTQ Inventory
↓
File Counts
↓
Validation ChecksThe expected number of FASTQ files, total storage requirements, and sample coverage can all be assessed from the inventory before formal validation begins.
For example:
PRJNA322554
188 runs
188 FASTQ files
PRJNA802976
54 runs
108 FASTQ filesThis illustrates why inventory generation is important: expected file counts depend on sequencing layout and must be verified before downstream analysis.
The FASTQ inventory becomes a key input to the Data Validation System introduced in Chapter 06.
Suppose a healthy reference microbiome project identifies hundreds of eligible samples across multiple public studies.
The acquisition workflow proceeds as follows:
Candidate Studies
↓
Metadata Acquisition
↓
Download Manifest
↓
Test Manifest
↓
Repository Validation
↓
Production Download
↓
Download Verification
↓
FASTQ Inventory
↓
Data Validation SystemAt this stage, the objective is not biological analysis. The objective is to acquire a complete, validated, and reproducible sequencing dataset suitable for downstream validation and reference dataset assembly.
Researchers frequently encounter:
Planning for these challenges improves acquisition reliability and reduces the likelihood of downstream data quality issues.
Every download process should be documented.
Important records include:
The goal is not only to download files, but also to make the acquisition process auditable, reproducible, and repeatable.
The Data Download System transforms metadata assets into validated sequencing files ready for downstream quality assessment.
Metadata Assets
↓
Download Manifest
↓
Test Manifest
↓
Repository Validation
↓
Production Download
↓
Download Verification
↓
FASTQ Inventory
↓
Data Validation SystemA reliable download system should answer five questions:
After sequencing files have been downloaded, verified, and inventoried, the next challenge is determining whether the acquired dataset is complete, internally consistent, and suitable for downstream analysis.
In the next chapter, we implement the Data Validation System to evaluate:
before proceeding to cloud transfer, reference dataset assembly, and downstream analysis.
Data Download
↓
FASTQ Inventory
↓
Data Validation
↓
Cloud Storage and Transfer
↓
Reference Dataset Assembly