Appendix A — CDI Data Acquisition System Overview

• ID: DAS-APP
• Type: Appendix
• Audience: Omics Data Scientists, Bioinformaticians, and Research Teams
• Theme: CDI Data Acquisition System Reference Materials

The CDI Data Acquisition System consists of a series of interconnected systems that transform public omics data into reusable reference datasets.

Study Discovery
        ↓
Metadata Acquisition
        ↓
Data Download
        ↓
Data Validation
        ↓
Cloud Storage and Transfer
        ↓
Reference Dataset Assembly
        ↓
Reusable Reference Dataset

Repository Accession Reference

Common accession types encountered during data acquisition:

Resource	Example
BioProject	PRJNA322554
BioSample	SAMN12345678
SRA Run	SRR12345678
GEO Series	GSE123456
ENA Study	ERP123456

Example Validation Projects

The CDI Data Acquisition System was validated using two representative human gut microbiome BioProjects.

BioProject	Sequencing Layout	Platform
PRJNA322554	Single-end amplicon sequencing	454 GS FLX Titanium
PRJNA802976	Paired-end amplicon sequencing	Illumina MiSeq

Together these projects demonstrate metadata acquisition, download, validation, and dataset assembly workflows across multiple sequencing designs.

Directory Structure Reference

Example CDI Data Acquisition System project structure:

project/
├── data/
│   ├── metadata/
│   ├── manifests/
│   ├── inventory/
│   ├── raw/
│   └── validation/
│
├── scripts/
│   └── bash/
│
├── library/
├── docs/
└── environment.yml

Bash Scripts Reference

The following scripts are developed throughout this guide.

scripts/bash/
├── 04a-download-ncbi-runinfo.sh
├── 04b-download-ena-metadata.sh
│
├── 05a-build-download-manifest.sh
├── 05b-download-ena-fastq.sh
├── 05c-download-ncbi-sra.sh
├── 05d-verify-downloads.sh
├── 05e-build-fastq-inventory.sh
│
├── 06-validate-downloads.sh
│
└── 07a-sync-to-s3.sh

Common Repository Tools

Metadata Acquisition

EDirect
pysradb
curl
wget

Data Download

SRA Toolkit
wget
curl
pigz

Validation

gzip
seqkit
csvtk
jq

Cloud Storage

AWS CLI
rsync
Aspera

Conda Environment

Example environment:

conda env create -f environment.yml

conda activate cdi-data-acquisition

Core tools used throughout the CDI Data Acquisition System include:

entrez-direct
pysradb
sra-tools
wget
curl
seqkit
csvtk
jq
parallel
pigz

Recommended Workflow

Research Question
        ↓
Study Discovery
        ↓
Metadata Acquisition
        ↓
Metadata Review
        ↓
Data Download
        ↓
Data Validation
        ↓
Cloud Storage
        ↓
Reference Dataset Assembly
        ↓
Dataset Release

Key Principles

The CDI Data Acquisition System is built upon the following principles:

• Reproducibility
• Transparency
• Metadata-first design
• Validation before analysis
• Provenance preservation
• Cloud-ready workflows
• Reusable reference datasets

Bibliography and Resources

The references cited throughout this guide are maintained in the bibliography database (library/references.bib) and are included automatically during book generation.

This appendix also provides a quick reference to commonly cited resources and repositories used throughout the CDI Data Acquisition System.

Suggested Citation Keys

The following citation keys are commonly used throughout the CDI Data Acquisition System:

• (Leinonen, Sugawara, and Shumway 2011)
• (Kodama, Shumway, and Leinonen 2012)
• (Barrett et al. 2013)
• (Arita, Karsch-Mizrachi, and Cochrane 2021)
• (Brazma et al. 2001)
• (Yilmaz et al. 2011)
• (Wilkinson et al. 2016)
• (Sandve et al. 2013)
• (Mitchell et al. 2020)
• (Knight et al. 2018)

Major Public Data Resources

• NCBI — https://www.ncbi.nlm.nih.gov (National Center for Biotechnology Information 2026d)
• Sequence Read Archive (SRA) — https://www.ncbi.nlm.nih.gov/sra (National Center for Biotechnology Information 2026e)
• BioProject — https://www.ncbi.nlm.nih.gov/bioproject (National Center for Biotechnology Information 2026a)
• BioSample — https://www.ncbi.nlm.nih.gov/biosample (National Center for Biotechnology Information 2026b)
• Gene Expression Omnibus (GEO) — https://www.ncbi.nlm.nih.gov/geo (National Center for Biotechnology Information 2026c)
• European Nucleotide Archive (ENA) — https://www.ebi.ac.uk/ena (European Nucleotide Archive 2026)
• MGnify — https://www.ebi.ac.uk/metagenomics (European Bioinformatics Institute 2026b)
• GWAS Catalog — https://www.ebi.ac.uk/gwas (European Bioinformatics Institute 2026a)
• DNA Data Bank of Japan (DDBJ) — https://www.ddbj.nig.ac.jp (DNA Data Bank of Japan 2026)

Arita, M., I. Karsch-Mizrachi, and G. Cochrane. 2021. “The International Nucleotide Sequence Database Collaboration.” Nucleic Acids Research 49 (D1): D121–24.

Barrett, T., S. E. Wilhite, P. Ledoux, et al. 2013. “NCBI GEO: Archive for Functional Genomics Data Sets—Update.” Nucleic Acids Research 41 (Database issue): D991–95.

Brazma, A., P. Hingamp, J. Quackenbush, et al. 2001. “Minimum Information about a Microarray Experiment (MIAME)—Toward Standards for Microarray Data.” Nature Genetics 29 (4): 365–71.

DNA Data Bank of Japan. 2026. “DDBJ.” https://www.ddbj.nig.ac.jp.

European Bioinformatics Institute. 2026a. “GWAS Catalog.” https://www.ebi.ac.uk/gwas.

———. 2026b. “MGnify.” https://www.ebi.ac.uk/metagenomics.

European Nucleotide Archive. 2026. “European Nucleotide Archive (ENA).” https://www.ebi.ac.uk/ena.

Knight, R., A. Vrbanac, B. C. Taylor, et al. 2018. “Best Practices for Analysing Microbiomes.” Nature Reviews Microbiology 16 (7): 410–22.

Kodama, Y., M. Shumway, and R. Leinonen. 2012. “The Sequence Read Archive: Explosive Growth of Sequencing Data.” Nucleic Acids Research 40 (Database issue): D54–56.

Leinonen, R., H. Sugawara, and M. Shumway. 2011. “The Sequence Read Archive.” Nucleic Acids Research 39 (Database issue): D19–21.

Mitchell, A. L., A. Almeida, M. Beracochea, et al. 2020. “MGnify: The Microbiome Analysis Resource in 2020.” Nucleic Acids Research 48 (D1): D570–78.

National Center for Biotechnology Information. 2026a. “BioProject.” https://www.ncbi.nlm.nih.gov/bioproject.

———. 2026b. “BioSample.” https://www.ncbi.nlm.nih.gov/biosample.

———. 2026c. “Gene Expression Omnibus (GEO).” https://www.ncbi.nlm.nih.gov/geo.

———. 2026d. “NCBI.” https://www.ncbi.nlm.nih.gov.

———. 2026e. “Sequence Read Archive (SRA).” https://www.ncbi.nlm.nih.gov/sra.

Sandve, G. K., A. Nekrutenko, J. Taylor, and E. Hovig. 2013. “Ten Simple Rules for Reproducible Computational Research.” PLoS Computational Biology 9 (10): e1003285.

Wilkinson, M. D., M. Dumontier, I. J. Aalbersberg, et al. 2016. “The FAIR Guiding Principles for Scientific Data Management and Stewardship.” Scientific Data 3: 160018.

Yilmaz, P., R. Kottmann, D. Field, et al. 2011. “Minimum Information about a Marker Gene Sequence (MIMARKS) and Minimum Information about Any Sequence (MIxS) Specifications.” Nature Biotechnology 29 (5): 415–20.