Preface and Overview

Published

Jun 2026

  • ID: DAS-000
  • Type: Preface
  • Audience: Omics Data Scientists, Bioinformaticians, and Research Teams
  • Theme: From Public Data to Reference Datasets

The journey from public omics repositories to a usable reference dataset is often longer and more complex than the downstream analysis itself. Study discovery, metadata curation, validation, and reproducible organization are not administrative tasks; they are foundational scientific activities that determine the quality and credibility of subsequent analyses.

The CDI Data Acquisition System presents public data acquisition as a reproducible workflow. It helps researchers move from public data discovery to reference dataset assembly through a structured, transparent, and auditable process.

Whether the goal is educational, scientific, clinical, or commercial, the ability to identify, evaluate, acquire, validate, organize, and transfer public datasets is an essential skill for modern omics research.

CDI Data Acquisition System

Many omics analysis guides begin with a dataset that is already available and ready for analysis. In practice, however, obtaining a suitable dataset is often one of the most challenging stages of a project.

Researchers, analysts, and organizations routinely spend substantial time searching public repositories, evaluating studies, collecting metadata, validating files, and assembling reference datasets before any statistical analysis can begin.

The CDI Data Acquisition System was developed to provide a structured framework for this process.

Rather than focusing on a single omics domain or a single repository, this guide focuses on the common activities required to transform publicly available omics data into reproducible datasets suitable for downstream analysis.

The system is designed to answer practical questions such as:

  • Where can suitable public omics datasets be found?
  • How are accessions organized across NCBI, ENA, BioProject, BioSample, SRA, and related systems?
  • How can metadata be retrieved and evaluated reproducibly?
  • How can FASTQ files be downloaded reliably from public repositories?
  • How can downloaded files be validated against expected metadata?
  • How should raw data, metadata, manifests, checksums, logs, and inventories be organized?
  • How can datasets be transferred to cloud storage or shared storage in a reproducible way?
  • How can a final reference dataset be assembled for downstream analysis?

System Philosophy

The CDI Data Acquisition System treats data acquisition as a scientific workflow rather than a one-time download step.

A reproducible data acquisition workflow should make it possible to answer:

  • What study was selected?
  • Why was the study selected?
  • Which samples and runs were included?
  • Which samples and runs were excluded?
  • Which files were expected?
  • Which files were downloaded?
  • Were the downloaded files complete?
  • Were file checksums validated?
  • Where were the final files stored?
  • Can another person repeat the same process?

This guide therefore emphasizes transparency, traceability, and validation at every stage.

Core Workflow

The CDI Data Acquisition System follows a staged workflow:

Study Discovery
      ↓
Accession Interpretation
      ↓
Metadata Acquisition
      ↓
Download Planning
      ↓
Data Download
      ↓
Data Validation
      ↓
Cloud Storage and Transfer
      ↓
Reference Dataset Assembly
      ↓
End-to-End Case Study

Each chapter focuses on one part of this workflow and produces outputs that become inputs for the next stage.

Example Datasets Used Throughout This Guide

To demonstrate the flexibility of the CDI Data Acquisition System, two representative human gut microbiome BioProjects are used throughout the guide.

These examples were selected because they allow the system to demonstrate both single-end and paired-end sequencing workflows while remaining biologically relevant to human gut microbiome data acquisition.

PRJNA322554

PRJNA322554 is used as the single-end example dataset.

This BioProject represents a human early-life gut microbiome study with stool and meconium samples. It provides a useful example of public microbiome data acquisition because NCBI RunInfo metadata, ENA metadata, FASTQ URLs, and MD5 checksums are available.

Key characteristics:

  • Human gut microbiome study
  • Amplicon sequencing
  • Single-end reads
  • 454 GS FLX Titanium platform
  • ENA FASTQ files available as single FASTQ files per run
  • ENA MD5 checksums available for validation

Typical FASTQ pattern:

SRR3725378.fastq.gz

PRJNA802976

PRJNA802976 is used as the paired-end example dataset.

This BioProject represents a human gut microbiome amplicon sequencing study generated using Illumina MiSeq. It provides a useful paired-end example for testing FASTQ URL extraction, paired-end file counting, download validation, and manifest-based organization.

Key characteristics:

  • Human gut microbiome study
  • Amplicon sequencing
  • Paired-end reads
  • Illumina MiSeq platform
  • ENA FASTQ files available as paired FASTQ files per run
  • ENA MD5 checksums available for validation

Typical FASTQ pattern:

SRR17868091_1.fastq.gz
SRR17868091_2.fastq.gz

Dataset Summary

BioProject Study Type Sequencing Layout Platform Example Use
PRJNA322554 Human gut microbiome Single-end 454 GS FLX Titanium Single-end FASTQ acquisition and validation
PRJNA802976 Human gut microbiome Paired-end Illumina MiSeq Paired-end FASTQ acquisition and validation

Together, these examples demonstrate that the CDI Data Acquisition System can acquire, validate, transfer, and assemble public sequencing datasets regardless of sequencing layout.

The goal is not to present these two BioProjects as the only possible choices, but to use them as reproducible test cases that demonstrate how the system behaves across different sequencing designs.

Expected System Outputs

By the end of the workflow, the system produces a structured project directory containing metadata, manifests, raw files, logs, validation summaries, inventories, and assembled reference dataset components.

A simplified output structure is shown below:

data/
├── metadata/
│   ├── runinfo-PRJNA322554.csv
│   ├── ena-PRJNA322554.tsv
│   ├── runinfo-PRJNA802976.csv
│   └── ena-PRJNA802976.tsv
├── manifests/
│   ├── ena-fastq-urls.txt
│   ├── ena-fastq-test-urls.txt
│   └── checksums.md5
├── raw/
│   └── fastq/
├── logs/
├── inventory/
└── reference-dataset/

The exact files may vary depending on the selected BioProject, repository availability, and whether the workflow is run in test mode or production mode.

Intended Audience

This guide is intended for:

  • Bioinformaticians building reproducible public data workflows
  • Omics data scientists preparing reference datasets
  • Research teams acquiring FASTQ files from public repositories
  • Students learning how public sequencing data are organized
  • Consultants and analysts supporting client-facing data acquisition projects
  • Organizations moving public data into cloud storage for downstream analysis

The guide assumes basic familiarity with the command line, but it explains the workflow logic step by step.

What This Guide Is Not

This guide does not attempt to perform downstream microbiome analysis, taxonomic profiling, functional profiling, or statistical modeling.

Instead, it focuses on the upstream system required before analysis can begin:

Public Data
    ↓
Metadata
    ↓
FASTQ Files
    ↓
Validation
    ↓
Organized Reference Dataset

Downstream analysis systems can then use the validated dataset as a reliable input.

Chapter Roadmap

The guide is organized as a complete end-to-end system.

Chapter Focus
DAS-001 Public data landscape
DAS-002 Study discovery
DAS-003 Accession systems
DAS-004 Metadata acquisition
DAS-005 Data download
DAS-006 Data validation
DAS-007 Cloud storage and transfer
DAS-008 Reference dataset assembly
DAS-009 End-to-end case study
Appendix Supporting commands, checks, and reference material

Each chapter builds toward a reproducible public data acquisition system that can be adapted to different omics domains, repositories, projects, and downstream analysis goals.

Looking Ahead

The next chapter introduces the public omics data landscape and explains how major repositories such as NCBI and ENA fit into the broader process of study discovery, metadata retrieval, file acquisition, and reference dataset assembly.