End-to-End Case Study

Published

Jun 2026

  • ID: DAS-009
  • Type: Case Study
  • Audience: Omics Data Scientists, Bioinformaticians, and Research Teams
  • Theme: Applying the CDI Data Acquisition System from Study Discovery to Reference Dataset Assembly

Introduction

This chapter demonstrates the complete CDI Data Acquisition System workflow using representative human gut microbiome studies.

The objective is to show how publicly available sequencing studies can be transformed into a validated and reusable reference dataset by integrating the workflows developed throughout the previous chapters.

Rather than focusing on a specific organization or project, this case study presents a generalized workflow that can be adapted to a wide range of omics data acquisition scenarios.

The workflow spans the complete acquisition lifecycle:

Study Discovery
        ↓
Metadata Acquisition
        ↓
Download Manifest Generation
        ↓
Data Download
        ↓
Data Validation
        ↓
Storage and Transfer
        ↓
Reference Dataset Assembly
        ↓
Reusable Reference Dataset

Case Study Objective

Objective:

Build a healthy reference gut microbiome dataset from publicly available sequencing studies.

The goal is to demonstrate how the CDI Data Acquisition System transforms public repository data into a curated analytical resource suitable for downstream biological analysis.

Expected outputs include:

  • Metadata assets
  • Download manifests
  • Sequencing files
  • Validation reports
  • Storage assets
  • Reference dataset components

End-to-End System Workflow

Code 
flowchart TD

A[Study Discovery]
--> B[Metadata Acquisition]

B --> C[Download Manifest Generation]

C --> D[Data Download]

D --> E[Data Validation]

E --> F[Storage and Transfer]

F --> G[Reference Dataset Assembly]

G --> H[Reference Dataset Package]

flowchart TD

A[Study Discovery]
--> B[Metadata Acquisition]

B --> C[Download Manifest Generation]

C --> D[Data Download]

D --> E[Data Validation]

E --> F[Storage and Transfer]

F --> G[Reference Dataset Assembly]

G --> H[Reference Dataset Package]

Each stage contributes a reproducible component to the final dataset.

Study Discovery
        ↓
Metadata Acquisition
        ↓
Download Manifest Generation
        ↓
Data Download
        ↓
Data Validation
        ↓
Storage and Transfer
        ↓
Reference Dataset Assembly
        ↓
Reference Dataset Package

Step 1: Study Discovery

The process begins by identifying candidate studies relevant to the dataset objective.

Example workflow:

Literature Review
        ↓
Public Repositories
        ↓
Candidate Studies

Example candidate studies:

PRJNA322554
PRJNA802976

These projects were selected because they represent two common sequencing layouts encountered during microbiome data acquisition and provide publicly available human gut microbiome sequencing data suitable for demonstrating the CDI Data Acquisition System.

BioProject Sequencing Layout Platform
PRJNA322554 Single-end amplicon sequencing 454 GS FLX Titanium
PRJNA802976 Paired-end amplicon sequencing Illumina MiSeq

Together, they demonstrate that the CDI Data Acquisition System can support metadata acquisition, data download, validation, storage, and reference dataset assembly workflows across multiple sequencing designs.

Key activities:

  • Review scientific literature
  • Identify relevant studies
  • Evaluate study eligibility
  • Record accession information

Output:

Candidate Studies

Step 2: Metadata Acquisition

Once studies have been identified, metadata are retrieved and organized into reusable acquisition assets.

Workflow:

BioProject
        ↓
NCBI RunInfo
        ↓
ENA Metadata
        ↓
Run Accession Manifest
        ↓
Metadata Assets

NCBI RunInfo Retrieval

bash scripts/bash/04a-download-ncbi-runinfo.sh

Example outputs:

data/metadata/
├── runinfo-PRJNA322554.csv
└── runinfo-PRJNA802976.csv

Observed results:

runinfo-PRJNA322554.csv
134 lines

runinfo-PRJNA802976.csv
55 lines

ENA Metadata Retrieval

bash scripts/bash/04b-download-ena-metadata.sh

Example outputs:

data/metadata/
├── ena-PRJNA322554.tsv
└── ena-PRJNA802976.tsv

Observed results:

ena-PRJNA322554.tsv
189 lines

ena-PRJNA802976.tsv
55 lines

Run Accession Manifest Generation

bash scripts/bash/04c-build-srr-accessions.sh

Example output:

data/metadata/
└── srr-accessions.txt

The accession manifest provides a reusable inventory of sequencing runs available for downstream acquisition and validation.

Example output:

SRR3725374
SRR3725375
SRR3725376
...

Output:

Metadata Assets
data/metadata/
├── runinfo-PRJNA322554.csv
├── runinfo-PRJNA802976.csv
├── ena-PRJNA322554.tsv
├── ena-PRJNA802976.tsv
└── srr-accessions.txt

Metadata Acquisition Workflow

BioProject Accessions
        ↓
04a-download-ncbi-runinfo.sh
        ↓
RunInfo Metadata
        ↓
04b-download-ena-metadata.sh
        ↓
ENA Metadata
        ↓
04c-build-srr-accessions.sh
        ↓
Run Accession Manifest
        ↓
Metadata Assets

Example outputs:

data/metadata/
├── runinfo-PRJNA322554.csv
├── ena-PRJNA322554.tsv
├── runinfo-PRJNA802976.csv
├── ena-PRJNA802976.tsv
└── srr-accessions.txt

The metadata acquisition workflow transforms BioProject accessions into reusable metadata assets that support sample selection, download planning, repository validation, provenance tracking, and downstream reference dataset assembly.

Output:

Metadata Assets

Step 3: Metadata Review and Sample Selection

The metadata assets generated in Step 2 are reviewed against predefined inclusion and exclusion criteria to identify samples eligible for downstream acquisition.

Example inclusion criteria:

  • Human samples
  • Healthy individuals
  • Stool specimens
  • Publicly available sequencing data

Example exclusion criteria:

  • Disease status unknown
  • Missing metadata
  • Incomplete records
  • Duplicate samples

The resulting set of eligible samples is then used to generate download manifests for data acquisition.

Workflow:

Metadata Assets
        ↓
Inclusion Criteria
        ↓
Exclusion Criteria
        ↓
Eligible Samples
        ↓
05a-build-download-manifest.sh
        ↓
Download Manifest

Example outputs:

data/metadata/
└── srr-accessions.txt

data/manifests/
├── download-manifest.tsv
└── test-manifest.tsv

The SRR accession manifest provides the inventory of sequencing runs available for evaluation and selection.

The production manifest contains all eligible sequencing runs selected for acquisition, while the test manifest contains a small representative subset used to validate repository workflows before large-scale downloads.

Output:

Download Manifest

Step 4: Data Download

Validated metadata and download manifests are used to retrieve sequencing data from public repositories.

Example workflows:

ENA Retrieval

bash scripts/bash/05b-download-ena-fastq.sh

NCBI Retrieval

bash scripts/bash/05c-download-ncbi-sra.sh

Example outputs:

data/raw/
├── ena/
│   ├── SRR17868090_1.fastq.gz
│   ├── SRR17868090_2.fastq.gz
│   ├── SRR17868091_1.fastq.gz
│   ├── SRR17868091_2.fastq.gz
│   ├── SRR17868092_1.fastq.gz
│   └── SRR17868092_2.fastq.gz
│
├── ncbi/
│   ├── SRR17868090_1.fastq.gz
│   ├── SRR17868090_2.fastq.gz
│   ├── SRR17868091_1.fastq.gz
│   ├── SRR17868091_2.fastq.gz
│   ├── SRR17868092_1.fastq.gz
│   └── SRR17868092_2.fastq.gz
│
└── sra/

Workflow:

Download Manifest
        ↓
05b-download-ena-fastq.sh
        ↓
ENA FASTQ Files
        ↓
05c-download-ncbi-sra.sh
        ↓
NCBI FASTQ Files
        ↓
Sequencing Data

The ENA workflow retrieves FASTQ files directly from repository download links, while the NCBI workflow retrieves SRA files and converts them into FASTQ format using the SRA Toolkit.

Together, these workflows demonstrate that the CDI Data Acquisition System can support data acquisition from multiple repository ecosystems while producing comparable FASTQ outputs for downstream validation and analysis.

Output:

Sequencing Data

Step 5: Download Verification

Downloaded files are verified against repository expectations before being accepted for downstream analysis.

Verification activities:

  • File count verification
  • FASTQ inspection
  • Repository validation
  • SeqKit validation
  • Inventory generation
  • Repository reconciliation

Repository Validation

bash scripts/bash/05d-verify-downloads.sh

Example output:

Repository Validation Summary

ENA FASTQ files:  6
NCBI FASTQ files: 6

Verification complete.

FASTQ Inventory Generation

bash scripts/bash/05e-build-fastq-inventory.sh

Example output:

data/inventory/
└── fastq-inventory.tsv

Workflow:

Sequencing Data
        ↓
05d-verify-downloads.sh
        ↓
Repository Validation
        ↓
FASTQ Verification
        ↓
05e-build-fastq-inventory.sh
        ↓
FASTQ Inventory
        ↓
Verified Sequencing Assets

The verification workflow confirms that:

  • Expected FASTQ files were downloaded.
  • Paired-end files are present and consistent.
  • FASTQ files can be parsed successfully.
  • ENA and NCBI retrieval workflows produce comparable outputs.
  • Downloaded assets are ready for formal validation.

Example verification results:

ENA FASTQ files:  6
NCBI FASTQ files: 6

Forward reads: 3
Reverse reads: 3

Repository Validation: PASS

Output:

FASTQ Inventory

Step 6: Data Validation

After download verification, sequencing files undergo systematic validation to assess dataset integrity, completeness, and readiness for downstream analysis.

Example:

bash scripts/bash/06-validate-downloads.sh data/raw/ena

Validation workflow:

FASTQ File Count
        ↓
Compression Validation
        ↓
Paired-End Validation
        ↓
FASTQ Structure Validation
        ↓
Validation Report

Example output:

check                       status   value
fastq_file_count            PASS     6
compression_validation      PASS     0
paired_end_validation       PASS     3
fastq_validation            PASS     data/raw/ena/SRR17868090_1.fastq.gz

Interpretation:

FASTQ File Count
Observed Files: 6
Status: PASS
Compression Validation
Failed Files: 0
Status: PASS
Paired-End Validation
Forward Reads: 3
Reverse Reads: 3
Status: PASS
FASTQ Validation
Readable Test File:
SRR17868090_1.fastq.gz

Status: PASS

Validation report:

data/validation/
└── validation-report.tsv

Workflow:

Verified Sequencing Assets
        ↓
06-validate-downloads.sh
        ↓
Validation Report
        ↓
Validated Dataset

The validation process confirms that:

  • Expected FASTQ files are present.
  • Compressed files are not corrupted.
  • Forward and reverse reads are balanced.
  • FASTQ files can be successfully read and decompressed.
  • The dataset is suitable for downstream processing.

Output:

Validated Dataset

Step 7: Storage and Transfer

Validated datasets are transferred to durable storage systems appropriate for the project requirements.

Possible destinations include:

  • Local storage
  • Institutional storage
  • Shared research servers
  • Cloud object storage
  • Reference dataset repositories

Local Storage Transfer

bash scripts/bash/07a-transfer-local.sh

Cloud Storage Transfer

bash scripts/bash/07b-transfer-s3.sh \
  data \
  reference-datasets

Storage Transfer Verification

bash scripts/bash/07c-verify-storage-transfer.sh

Workflow:

Validated Dataset
        ↓
07a-transfer-local.sh
        ↓
Local Storage

Validated Dataset
        ↓
07b-transfer-s3.sh
        ↓
Cloud Storage

Storage Assets
        ↓
07c-verify-storage-transfer.sh
        ↓
Verified Storage Assets

Example storage options:

Local Storage
├── Workstation
└── External Drive

Institutional Storage
├── Research Server
└── HPC Cluster

Cloud Storage
├── AWS S3
├── Google Cloud Storage
└── Azure Blob Storage

Example local storage structure:

storage/reference-datasets/
├── metadata/
├── manifests/
├── inventory/
├── raw/
├── validation/
└── reference-dataset/

Example cloud storage structure:

s3://reference-datasets/
├── metadata/
├── manifests/
├── inventory/
├── raw/
├── validation/
└── reference-dataset/

The objective is to preserve validated acquisition assets while maintaining provenance, reproducibility, and accessibility for downstream analysis and reference dataset assembly.

Output:

Verified Storage Assets

Step 8: Reference Dataset Assembly

Validated sequencing data, metadata, manifests, inventories, and validation records are assembled into a reusable reference dataset package.

Build Reference Manifest

bash scripts/bash/08a-build-reference-manifest.sh

Build Reference Inventory

bash scripts/bash/08b-build-reference-inventory.sh

Package Reference Dataset

bash scripts/bash/08c-package-reference-dataset.sh

Workflow:

Validated Dataset
        ↓
08a-build-reference-manifest.sh
        ↓
Reference Manifest
        ↓
08b-build-reference-inventory.sh
        ↓
Reference Inventory
        ↓
08c-package-reference-dataset.sh
        ↓
Reference Dataset Package

Example structure:

reference-dataset/
├── metadata/
│   ├── reference-manifest.tsv
│   ├── ena-PRJNA322554.tsv
│   ├── ena-PRJNA802976.tsv
│   ├── runinfo-PRJNA322554.csv
│   ├── runinfo-PRJNA802976.csv
│   └── srr-accessions.txt
│
├── manifests/
│   ├── download-manifest.tsv
│   ├── test-manifest.tsv
│   ├── ena-fastq-urls.txt
│   └── ena-fastq-test-urls.txt
│
├── inventory/
│   └── reference-inventory.tsv
│
├── validation/
│   └── storage-transfer-validation.tsv
│
├── documentation/
│   └── README.md
│
└── dataset-release/
    └── release-summary.txt

Key activities:

  • Reference manifest generation
  • Reference inventory generation
  • Metadata integration
  • Provenance preservation
  • Validation record preservation
  • Documentation
  • Dataset packaging

The assembly process transforms validated acquisition assets into a reusable analytical resource suitable for benchmarking, comparison, interpretation, and downstream biological analysis.

Output:

Reference Dataset Package

Final Deliverables

The completed reference dataset consists of multiple components generated throughout the CDI Data Acquisition System.

Example:

reference-dataset/
├── metadata/
├── manifests/
├── inventory/
├── validation/
├── documentation/
└── dataset-release/

These assets collectively provide:

  • Metadata assets
  • Download manifests
  • Reference inventories
  • Validation reports
  • Provenance records
  • Dataset documentation
  • Release-ready dataset packages

A reference dataset is more than a collection of sequencing files. It is a curated analytical resource that preserves the information required for reproducibility, auditing, interpretation, and future reuse.

System Outputs

The CDI Data Acquisition System transforms public repository data into reusable analytical resources.

Study Discovery
        ↓
Metadata Acquisition
        ↓
Metadata Review and Sample Selection
        ↓
Data Download
        ↓
Download Verification
        ↓
Data Validation
        ↓
Storage and Transfer
        ↓
Reference Dataset Assembly
        ↓
Reference Dataset Package
        ↓
Reusable Reference Dataset

Each stage contributes traceable and reproducible outputs that support downstream biological analysis.

The final result is a validated, documented, and reusable reference dataset package that can support benchmarking, comparative analyses, biological interpretation, and future data integration efforts.

Reproducibility Considerations

A reproducible reference dataset should preserve:

  • Source repositories
  • Study accessions
  • Sample identifiers
  • Run accessions
  • Download manifests
  • Reference inventories
  • Validation reports
  • Storage locations
  • Provenance records
  • Dataset documentation

These records enable future users to reconstruct, verify, and extend the dataset.

Example provenance chain:

BioProject
        ↓
Metadata Acquisition
        ↓
SRR Accession Manifest
        ↓
Download Manifest
        ↓
FASTQ Inventory
        ↓
Validation Report
        ↓
Storage and Transfer
        ↓
Reference Dataset Package

The resulting provenance trail provides a transparent record of how public repository data were transformed into a validated and reusable reference dataset.

Optional Workflow Orchestration

All CDI Data Acquisition System components can be executed independently. However, for demonstration purposes, an orchestration script is provided that executes the complete workflow from metadata acquisition through reference dataset assembly.

Example:

bash scripts/bash/workflow/09-run-end-to-end-workflow.sh

Workflow:

Metadata Acquisition
        ↓
Sample Selection
        ↓
Data Download
        ↓
Verification
        ↓
Validation
        ↓
Storage and Transfer
        ↓
Reference Dataset Assembly

The orchestration script demonstrates how the individual CDI Data Acquisition System components can be combined into a complete end-to-end workflow while preserving the modular design of the system.

Common Challenges

Researchers frequently encounter:

  • Missing metadata
  • Inconsistent annotations
  • Duplicate samples
  • Repository updates
  • Large file transfers
  • Incomplete downloads
  • Storage limitations
  • Study-specific biases

The CDI Data Acquisition System provides a structured framework for addressing these challenges through metadata-driven acquisition, validation, provenance tracking, storage management, and reproducible documentation.

Summary

The End-to-End Case Study demonstrates how the CDI Data Acquisition System transforms public omics data into a validated and reusable reference dataset package.

Public Repository Data
        ↓
Study Discovery
        ↓
Metadata Acquisition
        ↓
Metadata Review and Sample Selection
        ↓
Data Download
        ↓
Download Verification
        ↓
Data Validation
        ↓
Storage and Transfer
        ↓
Reference Dataset Assembly
        ↓
Reference Dataset Package
        ↓
Reusable Reference Dataset

Throughout this workflow, we demonstrated:

  • Study discovery and evaluation
  • Metadata acquisition from NCBI and ENA
  • Run accession manifest generation
  • Download manifest generation
  • FASTQ acquisition through ENA and NCBI workflows
  • Repository verification and inventory generation
  • Dataset validation and readiness assessment
  • Storage and transfer strategies
  • Reference dataset assembly and packaging

The case study also demonstrated acquisition workflows across multiple sequencing designs, including both single-end and paired-end microbiome sequencing studies.

By combining structured workflows, validation procedures, provenance tracking, storage management, and reproducible documentation, researchers can construct high-quality reference datasets suitable for downstream biological analysis, benchmarking, interpretation, and future data integration.

The resulting reference dataset package preserves the metadata, manifests, inventories, validation records, provenance information, and documentation required to support reproducible biological research and long-term scientific reuse.

Concluding Remarks

The CDI Data Acquisition System provides a reproducible framework for discovering, acquiring, validating, storing, and assembling public omics datasets.

The system was demonstrated using representative human gut microbiome studies spanning both major sequencing layouts encountered during public microbiome data acquisition:

BioProject Sequencing Layout Platform
PRJNA322554 Single-end amplicon sequencing 454 GS FLX Titanium
PRJNA802976 Paired-end amplicon sequencing Illumina MiSeq

Together, these studies demonstrate that the CDI Data Acquisition System can support metadata acquisition, data download, verification, validation, storage, and reference dataset assembly across multiple sequencing designs.

Although the examples presented throughout this guide focus primarily on microbiome sequencing studies, the same principles can be applied across a broad range of omics domains, including:

  • RNA-Seq
  • Single-cell sequencing
  • Population genomics
  • Metagenomics
  • Multi-omics integration

The combination of systematic study discovery, metadata-driven sample selection, validated data acquisition, durable storage, provenance preservation, and reference dataset assembly creates a foundation for reproducible biological research and data-driven discovery.

Study Discovery
        ↓
Metadata Acquisition
        ↓
Metadata Review and Sample Selection
        ↓
Data Download
        ↓
Download Verification
        ↓
Data Validation
        ↓
Storage and Transfer
        ↓
Reference Dataset Assembly
        ↓
Reference Dataset Package
        ↓
Reusable Reference Dataset

The resulting reference dataset package preserves metadata, manifests, inventories, validation records, provenance information, and documentation required for reproducibility and long-term scientific reuse.

This completes the CDI Data Acquisition System and establishes a reusable framework for transforming public omics data into validated, documented, and reproducible analytical resources.