Once studies have been identified, accession systems understood, and metadata evaluated, the next step is acquiring the sequencing data.
Data download transforms study selection and metadata assets into locally accessible sequencing files that can be validated, organized, transferred, and assembled into downstream reference datasets. A reproducible acquisition workflow requires more than simply downloading files. Researchers must understand where files reside, how they are accessed, how downloads can be resumed, and how file integrity can be verified.
By the end of this chapter, you will be able to:
Metadata acquisition tells us what data exist.
The Data Download System retrieves the actual sequencing files required for downstream analysis.
Metadata Assets
↓
Download Manifest
↓
Repository Selection
↓
Download Execution
↓
Integrity Verification
↓
FASTQ Inventory
↓
Data Validation SystemThe quality of downstream analyses depends on the quality and completeness of acquired data. A successful download workflow should ensure that:
flowchart TD
A[Metadata Assets]
--> B[Download Manifest]
B --> C[Repository Selection]
C --> D[Download Execution]
D --> E[Integrity Verification]
E --> F[FASTQ Inventory]
F --> G[Data Validation System]flowchart TD A[Metadata Assets] --> B[Download Manifest] B --> C[Repository Selection] C --> D[Download Execution] D --> E[Integrity Verification] E --> F[FASTQ Inventory] F --> G[Data Validation System]
The Data Download System begins with metadata assets generated in Chapter 04 and ends with a documented inventory of downloaded files.
The CDI Data Acquisition System uses a single Conda environment throughout the workflow.
conda activate cdi-data-acquisitionThis avoids unnecessary environment switching and ensures that metadata acquisition, data download, validation, and dataset assembly operate within a consistent software environment.
The environment used throughout this book is defined as:
name: cdi-data-acquisition
channels:
- conda-forge
- bioconda
dependencies:
- entrez-direct
- pysradb
- sra-tools
- wget
- curl
- seqkit
- csvtk
- jq
- parallel
- pigz
- pythonVerify the key tools required by the Data Download System.
python --version
prefetch --version
fasterq-dump --version
vdb-validate --version
pysradb --help
seqkit version
csvtk version
jq --version
parallel --version
which esearchSuccessful execution confirms that the shared CDI data acquisition environment is ready for metadata acquisition, data download, and data validation workflows.
Public repositories provide access to a variety of downloadable file types.
| Repository | Typical Data | Common Role in This System |
|---|---|---|
| SRA | Raw sequencing reads | Run accession source and SRA Toolkit download |
| ENA | Raw sequencing reads and FASTQ URLs | Direct FASTQ retrieval and checksum validation |
| GEO | Processed and supplementary data | Study-level context and supplementary files |
| MGnify | Processed microbiome outputs | Processed microbiome reference outputs |
| GWAS Catalog | Summary statistics and study information | Association study discovery and summary statistics |
For large public sequencing datasets, the two most important download sources are usually ENA and NCBI SRA.
Researchers frequently encounter:
For raw sequencing acquisition, the most common target output is FASTQ. In microbiome, RNA-seq, metagenomics, and many other NGS workflows, FASTQ files serve as the starting point for downstream quality control and analysis.
The Data Download System consumes metadata assets generated during metadata acquisition.
data/
└── metadata/
├── runinfo-prjna477349.csv
├── ena-prjna477349.tsv
└── srr-accessions.txtThese files provide accession identifiers, sample-level metadata, and potential download locations required for data retrieval.
The most important handoff from Chapter 04 is the run accession list:
SRR accession list
↓
Download manifest
↓
FASTQ retrievalA download manifest represents the authoritative list of files or accessions to retrieve.
A simple accession-only manifest may look like this:
SRR1234567
SRR1234568
SRR1234569A richer manifest may include repository links and checksums:
run_accession fastq_ftp fastq_md5
SRR1234567 ftp.sra.ebi.ac.uk/.../file_1.gz abc123...
SRR1234568 ftp.sra.ebi.ac.uk/.../file_1.gz def456...Recommended output location:
data/
└── manifests/
└── download-manifest.tsvThe manifest becomes the system contract: it defines what should be downloaded, from where, and how completeness will later be checked.
The CDI Data Acquisition System uses a staged download strategy that validates repository workflows before initiating large-scale data retrieval.
Build Manifest
↓
Create Test Manifest
↓
Validate Repository Workflows
├── ENA
└── NCBI/SRA
↓
Production DownloadThis approach reduces risk by ensuring that:
The test manifest generated from the full download manifest serves as a representative subset for repository validation.
download-manifest.tsv
↓
test-manifest.tsv
↓
Repository Validation
↓
Production DownloadSuccessful test downloads provide confidence that the acquisition workflow is functioning correctly before scaling to the complete dataset.
Only after successful repository validation should production downloads be initiated.
Several tools are commonly used to retrieve public omics datasets.
The SRA Toolkit is commonly used for retrieving sequencing data from the NCBI Sequence Read Archive.
| Tool | Purpose |
|---|---|
prefetch |
Download SRA files |
fasterq-dump |
Convert SRA files to FASTQ |
vdb-validate |
Validate downloaded SRA files |
ENA and other repositories often expose direct file links that can be retrieved with common command-line tools.
| Tool | Purpose |
|---|---|
wget |
Download files from HTTP, HTTPS, or FTP links |
curl |
Retrieve files and web resources |
md5sum |
Verify MD5 checksums on Linux |
md5 |
Verify MD5 checksums on macOS |
Some repositories also provide access through cloud infrastructure and object storage systems. Cloud transfer workflows are introduced later in the book, after local download and validation concepts are established.
Activate the shared CDI data acquisition environment:
conda activate cdi-data-acquisitionInstall SRA Toolkit:
conda install -c bioconda sra-toolsVerify installation:
prefetch --version
fasterq-dump --version
vdb-validate --versionIf the commands return version information, the toolkit is available in the active environment.
Many systems already include curl. Some may require wget installation.
curl --version
wget --versionIf wget is missing, install it in the same environment:
conda install -c conda-forge wgetFor checksum verification, Linux commonly uses:
md5sum --versionOn macOS, use:
md5 --versionThe exact checksum command can differ by operating system, but the purpose is the same: confirm that downloaded files match the expected repository checksums.
Downloaded files should be organized systematically.
Recommended structure:
data/
├── metadata/
│ ├── runinfo-prjna477349.csv
│ ├── ena-prjna477349.tsv
│ └── srr-accessions.txt
│
├── manifests/
│ └── download-manifest.tsv
│
├── raw/
│ ├── sra/
│ └── fastq/
│
├── logs/
│ ├── download-ena.log
│ ├── download-ncbi.log
│ └── checksum.log
│
└── inventory/
└── fastq-inventory.tsvA consistent structure improves reproducibility, simplifies validation, and provides a clear handoff between chapters.
Create the directories:
mkdir -p data/manifests
mkdir -p data/raw/sra
mkdir -p data/raw/fastq
mkdir -p data/logs
mkdir -p data/inventoryA minimal accession manifest can be copied from the Chapter 04 output:
cp data/metadata/srr-accessions.txt data/manifests/download-manifest.tsvPreview the manifest:
head data/manifests/download-manifest.tsvCount accessions:
wc -l data/manifests/download-manifest.tsvThe expected count becomes one of the first validation checks after download.
The CDI Data Download System is implemented through a set of reusable Bash scripts.
scripts/bash/
├── 05a-build-download-manifest.sh
│ └── Create production and test manifests
│
├── 05b-download-ena-fastq.sh
│ └── Download FASTQ files from ENA
│
├── 05c-download-ncbi-sra.sh
│ └── Download and convert SRA accessions
│
├── 05d-verify-downloads.sh
│ └── Verify downloaded FASTQ assets
│
└── 05e-build-fastq-inventory.sh
└── Generate FASTQ inventory reportsThese scripts operate on the directory structure and manifest created in the previous sections and provide a reproducible implementation of the Data Download System.
Before executing the full workflow, it is good practice to validate the environment, directory structure, and download process using a single accession. The following section demonstrates this verification step before scaling to larger manifests and automated script execution.
Before downloading an entire dataset, it is good practice to validate the workflow using a single accession from the download manifest.
Preview the first accession:
head -n 1 data/manifests/download-manifest.tsvOutput:
SRR7450741Download the accession using SRA Toolkit:
prefetch SRR7450741 \
--output-directory data/raw/sraSuccessful execution created:
data/raw/sra/
└── SRR7450741/
└── SRR7450741.sraVerify the downloaded file:
find data/raw/sra -type fOutput:
data/raw/sra/SRR7450741/SRR7450741.sraConvert the SRA file to FASTQ:
fasterq-dump \
data/raw/sra/SRR7450741/SRR7450741.sra \
--outdir data/raw/fastqOutput:
spots read : 79,065
reads read : 158,130
reads written : 158,130Inspect the generated FASTQ files:
ls -lh data/raw/fastqOutput:
total 115344
-rw-r--r--@ 1 tmbmacbookair staff 28M Jun 11 15:35 SRR7450741_1.fastq
-rw-r--r--@ 1 tmbmacbookair staff 28M Jun 11 15:35 SRR7450741_2.fastqThe presence of two FASTQ files indicates that this sequencing run is paired-end.
Perform a preliminary validation using SeqKit:
seqkit stats data/raw/fastq/*.fastqOutput:
file format type num_seqs sum_len min_len avg_len max_len
data/raw/fastq/SRR7450741_1.fastq FASTQ DNA 79,065 11,938,815 151 151 151
data/raw/fastq/SRR7450741_2.fastq FASTQ DNA 79,065 11,938,815 151 151 151This simple test confirms that:
At this stage we have demonstrated:
Metadata Assets
↓
Download Manifest
↓
SRA Download
↓
FASTQ Generation
↓
Basic ValidationThis preliminary verification provides confidence that the acquisition workflow is functioning correctly.
Rather than immediately downloading hundreds of sequencing runs, validating the workflow on a single accession helps confirm that repository access, directory structure, software installation, FASTQ conversion, and basic file inspection are all functioning as expected.
Once the workflow has been verified, the same approach can be scaled to the complete download manifest through the CDI Data Download System scripts.
The following sections describe two common strategies for large-scale data acquisition: direct FASTQ retrieval from ENA and SRA-based retrieval using the NCBI SRA Toolkit. Rather than executing individual commands manually, these workflows are implemented through the reusable Bash scripts introduced above.
The recommended strategy is to use NCBI for study discovery and accession resolution, then use ENA for direct FASTQ download when suitable links and checksums are available.
Study Discovery
↓
NCBI
↓
Metadata Acquisition
↓
ENA
↓
Download Planning
↓
Data Download| Scenario | Preferred Repository |
|---|---|
| Metadata discovery | NCBI |
| FASTQ download with direct URLs | ENA |
| Missing FASTQ URLs | NCBI SRA Toolkit |
| Need SRA-native retrieval | NCBI SRA Toolkit |
| Controlled-access data | Repository-specific process |
| Supplementary processed files | GEO, ENA, project archive, or journal supplement |
This strategy avoids treating all repositories as interchangeable. Each repository contributes differently to the acquisition system.
ENA is often preferred when direct FASTQ links and MD5 checksums are available.
ENA Metadata
↓
FASTQ URLs
↓
wget or curl
↓
FASTQ Files
↓
Checksum VerificationThe ENA metadata file may contain columns such as:
run_accessionfastq_ftpfastq_md5fastq_bytesThe exact column names should be inspected before writing download commands:
head -n 1 data/metadata/ena-prjna477349.tsvThe ENA download workflow is implemented by:
bash scripts/bash/05b-download-ena-fastq.shThis script extracts FASTQ URLs from the ENA metadata file, writes them to:
data/manifests/ena-fastq-urls.txtand downloads the files into:
data/raw/fastq/Download logs are written to:
data/logs/download-ena.logThe script uses wget --continue, which allows interrupted downloads to resume when possible.
When direct FASTQ links are unavailable, the SRA Toolkit workflow can be used.
SRR Accessions
↓
prefetch
↓
SRA Files
↓
fasterq-dump
↓
FASTQ FilesThe NCBI SRA workflow is implemented by:
bash scripts/bash/05c-download-ncbi-sra.shBy default, this script uses the small test manifest:
data/manifests/test-manifest.tsvThis makes the workflow safe for testing and book development.
To run the workflow on the full dataset, provide the full manifest explicitly:
bash scripts/bash/05c-download-ncbi-sra.sh \
data/manifests/download-manifest.tsvThe script performs three steps:
Download SRA files
↓
Convert SRA to FASTQ
↓
Compress FASTQ filesOutputs are written to:
data/raw/sra/
data/raw/fastq/
data/logs/download-ncbi.logThis test-first pattern prevents accidentally launching a full dataset download before the workflow has been validated.
For large datasets, compression can take time. Faster alternatives such as pigz can be used when parallel compression is required.
The previous single-accession test demonstrated that the download workflow is functioning correctly on a representative sequencing run. After scaling to the complete download manifest, additional checks should be performed to confirm file integrity and completeness before proceeding to downstream validation.
Common post-download checks include:
The following section focuses on checksum verification, one of the most important safeguards against incomplete or corrupted downloads.
If ENA provides MD5 checksums, they should be used to verify that downloaded FASTQ files match the files distributed by the repository.
A checksum file may look like this:
abc123... SRR1234567_1.fastq.gz
def456... SRR1234567_2.fastq.gzOn Linux:
cd data/raw/fastq
md5sum -c ../../manifests/ena-md5.txt | tee ../../logs/checksum.log
cd -On macOS, checksum verification may require a slightly different command or installing GNU coreutils.
The important principle is that expected checksums from the repository should match the local downloaded files.
A FASTQ inventory records the sequencing files acquired during the download process and provides a reproducible record of downloaded assets.
Recommended output:
data/
└── inventory/
└── fastq-inventory.tsvThe inventory workflow is implemented through:
bash scripts/bash/05e-build-fastq-inventory.shThis script scans the FASTQ directory and generates an inventory containing file names and file sizes.
Preview the inventory:
head data/inventory/fastq-inventory.tsvCount inventory records:
wc -l data/inventory/fastq-inventory.tsvExample output:
file size_bytes
data/raw/fastq/SRR7450741_1.fastq.gz 12123456
data/raw/fastq/SRR7450741_2.fastq.gz 12098765The inventory provides a concise summary of acquired sequencing assets and supports:
The expected number of FASTQ files, total storage requirements, and sample coverage can all be assessed from the inventory before formal validation begins.
The inventory becomes a key input to the Data Validation System introduced in Chapter 06.
Suppose a healthy reference microbiome project identifies 850 eligible samples.
The workflow proceeds as follows:
850 Eligible Samples
↓
Metadata Acquisition
↓
Download Manifest
↓
Single-Accession Test
↓
Workflow Verification
↓
ENA or NCBI Download Workflow
↓
Download Verification
↓
FASTQ Inventory
↓
Data Validation SystemAt this stage, the objective is not analysis. The objective is to acquire a complete and reproducible sequencing dataset suitable for downstream validation and reference dataset assembly.
Researchers frequently encounter:
Planning for these challenges improves acquisition reliability.
Every download process should be documented.
Important records include:
The goal is not only to download files, but to make the download process auditable and repeatable.
The Data Download System converts metadata assets into verified sequencing files.
Metadata Assets
↓
Download Manifest
↓
Single-Accession Test
↓
Workflow Verification
↓
ENA or NCBI Download Workflow
↓
Download Verification
↓
FASTQ Inventory
↓
Data Validation SystemA reliable download system should answer five questions:
After sequencing files have been downloaded, verified, and inventoried, the next challenge is determining whether the acquired dataset is complete, internally consistent, and suitable for downstream analysis.
In the next chapter, we implement the Data Validation System to evaluate file integrity, sample completeness, metadata consistency, and overall dataset readiness before reference dataset assembly and analysis.