Reference Dataset Assembly System

Published

Jun 2026

  • ID: DAS-008
  • Type: Foundations
  • Audience: Omics Data Scientists, Bioinformaticians, and Research Teams
  • Theme: Transforming Validated Data into Reusable Analytical Resources

Acquiring, validating, and storing data are essential steps in the data acquisition lifecycle. However, the ultimate objective is to transform these assets into a reference dataset that can support downstream analysis, benchmarking, interpretation, and decision-making.

The Reference Dataset Assembly System integrates validated sequencing data, metadata, inventories, and provenance records into a coherent and reproducible analytical resource.

What Is a Reference Dataset?

A reference dataset is a curated collection of samples and associated metadata assembled for a specific scientific objective.

Examples include:

  • Healthy gut microbiome references
  • Disease-specific cohorts
  • RNA-Seq benchmark datasets
  • Single-cell reference atlases
  • Population genomics resources

Unlike raw repository data, reference datasets are intentionally constructed, documented, validated, and maintained.

Why Reference Dataset Assembly Matters

Public repositories contain data generated by different studies, laboratories, populations, and experimental designs.

Simply combining datasets without careful evaluation can introduce:

  • Bias
  • Duplicate samples
  • Inconsistent metadata
  • Confounding variables
  • Reproducibility challenges

Reference dataset assembly helps ensure that the final dataset is appropriate for its intended objective.

Reference Dataset Assembly in the CDI Data Acquisition System

Code 
flowchart TD

A[Metadata Acquisition System]
--> B[Data Download System]

B --> C[Data Validation System]

C --> D[Cloud Storage and Transfer System]

D --> E[Reference Dataset Assembly System]

E --> F[Reusable Reference Dataset]

flowchart TD

A[Metadata Acquisition System]
--> B[Data Download System]

B --> C[Data Validation System]

C --> D[Cloud Storage and Transfer System]

D --> E[Reference Dataset Assembly System]

E --> F[Reusable Reference Dataset]

The Reference Dataset Assembly System integrates validated acquisition assets into a reusable analytical resource.

Metadata Assets
        ↓
Validated FASTQ Files
        ↓
Validation Reports
        ↓
Reference Dataset Assembly
        ↓
Reusable Reference Dataset

Define the Dataset Objective

Every reference dataset should begin with a clearly defined objective.

Examples:

Objective Dataset Type
Healthy Gut Reference Healthy cohort
IBD Comparison Dataset Disease cohort
RNA-Seq Benchmark Expression reference
Single-Cell Atlas Cell-state reference
Population Genomics Resource Population reference

The objective determines:

  • Which studies are included
  • Which samples are eligible
  • Which metadata fields are required
  • Which exclusion criteria are applied

A clearly defined objective provides the foundation for dataset assembly.

Healthy Reference Microbiome Example

A healthy reference microbiome dataset may require:

  • Human samples
  • Stool specimens
  • Healthy individuals
  • Adequate metadata
  • Successfully validated sequencing files

The objective should be defined before acquisition begins because it influences study discovery, metadata acquisition, validation, and final dataset assembly.

Sample Selection Framework

Reference datasets should be constructed using predefined inclusion and exclusion criteria.

Inclusion Examples

  • Human samples
  • Healthy individuals
  • Stool specimens
  • Publicly available metadata
  • Available sequencing files
  • Successful validation status

Exclusion Examples

  • Missing metadata
  • Unknown disease status
  • Duplicate samples
  • Incomplete sequencing files
  • Failed validation checks

Consistent criteria improve transparency, reproducibility, and dataset quality.

Metadata Harmonization

Studies often use different terminology to describe the same concepts.

Example:

Study A Study B Harmonized Value
Healthy Control Healthy
Male M Male
Female F Female

Metadata harmonization creates a common representation across studies.

This process improves:

  • Dataset integration
  • Comparability
  • Reproducibility
  • Downstream analysis

A harmonized metadata table often becomes one of the most valuable outputs of the assembly process.

Dataset Integration

Multiple studies may contribute to a single reference dataset.

Code 
flowchart LR

A[Study 1]
--> D[Reference Dataset]

B[Study 2]
--> D

C[Study 3]
--> D

flowchart LR

A[Study 1]
--> D[Reference Dataset]

B[Study 2]
--> D

C[Study 3]
--> D

Dataset integration combines validated samples while preserving study-level provenance information.

For example:

PRJNA322554
        ↓
Validated Samples
        ↓
Reference Dataset

PRJNA802976
        ↓
Validated Samples
        ↓
Reference Dataset

This approach allows datasets generated using different sequencing runs, laboratories, and collection efforts to contribute to a common analytical resource.

Provenance Preservation

Reference datasets should retain information about:

  • Source studies
  • BioProjects
  • BioSamples
  • Run accessions
  • Download dates
  • Validation status
  • Storage locations

Provenance enables datasets to be traced, verified, and reconstructed throughout their lifecycle.

Example provenance chain:

Public Repository
        ↓
Metadata Acquisition
        ↓
Download Manifest
        ↓
FASTQ Inventory
        ↓
Validation Report
        ↓
Reference Dataset

Maintaining provenance records supports transparency, auditing, reproducibility, and future dataset updates.

Reference Dataset Project Structure

A reference dataset should preserve all components required for reproducibility.

Example:

reference-dataset/
├── metadata/
├── manifests/
├── inventory/
├── raw/
├── validation/
├── harmonized-metadata/
├── inclusion-exclusion/
├── documentation/
└── dataset-release/

This structure separates acquisition assets from dataset-specific assembly outputs while preserving the complete provenance chain from public repository to final dataset release.

The resulting reference dataset becomes a reusable research asset that can support benchmarking, comparative analyses, interpretation, and future analytical workflows.

Reference Dataset Assembly Workflow

The CDI Data Acquisition System assembles validated acquisition assets into a reusable reference dataset package.

Workflow:

Validated Dataset
        ↓
Reference Manifest
        ↓
Reference Inventory
        ↓
Reference Dataset Package
        ↓
Reusable Reference Dataset

The assembly process consists of three reproducible components:

Step Script Output
Manifest Generation 08a-build-reference-manifest.sh Reference Manifest
Inventory Generation 08b-build-reference-inventory.sh Reference Inventory
Dataset Packaging 08c-package-reference-dataset.sh Reference Dataset Package

Step 1: Build Reference Manifest

Validated metadata are assembled into a reference manifest representing samples eligible for inclusion in the final dataset.

Example:

bash scripts/bash/08a-build-reference-manifest.sh

Output:

data/reference/
└── reference-manifest.tsv

Example result:

Building reference manifest...

Saved:
  data/reference/reference-manifest.tsv

Records:
      55 data/reference/reference-manifest.tsv

Workflow:

Validated Metadata
        ↓
Reference Manifest

Step 2: Build Reference Inventory

The reference manifest is transformed into an inventory describing the sequencing assets included in the dataset.

Example:

bash scripts/bash/08b-build-reference-inventory.sh

Output:

data/reference/
├── reference-manifest.tsv
└── reference-inventory.tsv

Example inventory fields:

run_accession
sample_accession
fastq_ftp
fastq_md5

Workflow:

Reference Manifest
        ↓
Reference Inventory

The inventory provides a reproducible record of the sequencing assets included in the reference dataset.

Step 3: Package Reference Dataset

The final step assembles validated metadata, manifests, inventories, and validation outputs into a reusable reference dataset package.

Example:

bash scripts/bash/08c-package-reference-dataset.sh

Output:

reference-dataset/
├── metadata/
├── manifests/
├── inventory/
├── validation/
├── documentation/
└── dataset-release/

Example result:

Packaging reference dataset...

Reference dataset package:
reference-dataset/dataset-release/release-summary.txt
reference-dataset/documentation/README.md
reference-dataset/inventory/reference-inventory.tsv
reference-dataset/manifests/download-manifest.tsv
reference-dataset/metadata/reference-manifest.tsv
reference-dataset/validation/storage-transfer-validation.tsv

Package complete.

Workflow:

Reference Inventory
        ↓
Dataset Packaging
        ↓
Reference Dataset Package

Reference Dataset Structure

The completed package contains all assets required for reproducibility.

reference-dataset/
├── metadata/
│   ├── reference-manifest.tsv
│   ├── ena-PRJNA322554.tsv
│   ├── ena-PRJNA802976.tsv
│   ├── runinfo-PRJNA322554.csv
│   ├── runinfo-PRJNA802976.csv
│   └── srr-accessions.txt
│
├── manifests/
│   ├── download-manifest.tsv
│   ├── test-manifest.tsv
│   ├── ena-fastq-urls.txt
│   └── ena-fastq-test-urls.txt
│
├── inventory/
│   └── reference-inventory.tsv
│
├── validation/
│   └── storage-transfer-validation.tsv
│
├── documentation/
│   └── README.md
│
└── dataset-release/
    └── release-summary.txt

Summary

Reference dataset assembly transforms validated acquisition outputs into a reusable analytical resource.

Validated Dataset
        ↓
08a-build-reference-manifest.sh
        ↓
Reference Manifest
        ↓
08b-build-reference-inventory.sh
        ↓
Reference Inventory
        ↓
08c-package-reference-dataset.sh
        ↓
Reference Dataset Package
        ↓
Reusable Reference Dataset

The resulting package preserves metadata, manifests, inventories, validation records, provenance information, and documentation required for reproducible biological research and future dataset reuse.

By combining structured sample selection, metadata harmonization, validation procedures, provenance tracking, and reproducible documentation, researchers can construct high-quality reference datasets that support downstream biological analysis and long-term scientific reuse.

A reference dataset is more than a collection of sequencing files. It is a curated, documented, and validated scientific asset that preserves the context necessary for reproducibility, interpretation, and future discovery.

Looking Ahead

The previous chapters introduced the individual components of the CDI Data Acquisition System and demonstrated their implementation through reproducible workflows.

In the next chapter, we integrate these components into an end-to-end case study that follows the complete acquisition lifecycle:

Study Discovery
        ↓
Metadata Acquisition
        ↓
Data Download
        ↓
Data Validation
        ↓
Storage and Transfer
        ↓
Reference Dataset Assembly

This case study demonstrates how public omics data can be transformed into a validated and reusable reference dataset using the CDI Data Acquisition System.