Metadata Acquisition

Published

Jun 2026

Introduction

Once relevant studies have been discovered and their accession systems understood, the next step is metadata acquisition.

Metadata provide the context needed to understand biological samples, sequencing experiments, and associated data files. They help researchers determine whether a study is relevant to a particular objective and whether individual samples should be included or excluded from downstream analyses.

Without metadata, sequencing files become difficult to interpret, compare, validate, or integrate into a reproducible dataset.

Metadata acquisition is the process of discovering, retrieving, validating, evaluating, and organizing information that describes sequencing studies, samples, experiments, sequencing runs, and associated files.

Before any sequencing data are downloaded, metadata provide the information needed to understand:

  • What was studied
  • Which samples were collected
  • How sequencing was performed
  • Which sequencing runs are available
  • Which files can be downloaded
  • Whether a dataset meets project requirements

For many projects, metadata acquisition is the most important stage of the data acquisition workflow because it determines which data ultimately enter the analysis pipeline.

What Is Metadata?

Metadata are data about data.

In public sequencing repositories, metadata describe the context surrounding biological data rather than the sequencing reads themselves.

Data Metadata
FASTQ file File size, sequencing platform, run accession
Sequencing run Run date, read count, instrument model
Biological sample Organism, body site, disease status
Study Title, description, publication

Why Metadata Matter?

Metadata support:

  • Study discovery
  • Sample selection
  • Quality assessment
  • Download planning
  • Dataset assembly
  • Reproducible research

Metadata Across the Data Acquisition Lifecycle

Study Discovery
      ↓
Metadata Acquisition
      ↓
Metadata Evaluation
      ↓
Download Planning
      ↓
Data Download
      ↓
Reference Dataset Assembly

Metadata Categories

Metadata can be organized into several categories that describe different aspects of a sequencing study.

Study Metadata

Study metadata describe the overall research project.

Examples:

  • BioProject accession
  • Study title
  • Study description
  • Associated publication

Sample Metadata

Sample metadata describe individual biological specimens.

Examples:

  • BioSample accession
  • Organism
  • Body site
  • Disease status
  • Geographic location

Experiment Metadata

Experiment metadata describe how samples were prepared and sequenced.

Examples:

  • Library strategy
  • Library source
  • Library selection
  • Sequencing platform

Run Metadata

Run metadata describe individual sequencing runs.

Examples:

  • Run accession
  • Read counts
  • Base counts
  • Instrument model

File Metadata

File metadata describe downloadable sequencing files and associated validation information.

Examples:

  • FASTQ locations
  • File sizes
  • MD5 checksums
  • Download URLs
  • Sequencing layout (single-end or paired-end)

Together, these metadata categories provide the information required to evaluate studies, select samples, plan downloads, validate files, and assemble reproducible datasets.

Metadata Sources

Metadata can be obtained from multiple sources, each providing different levels of detail and context.

Repository Metadata

Repository metadata are retrieved directly from public databases and typically provide study, sample, experiment, run, and file-level information.

Examples:

  • NCBI
  • ENA
  • DDBJ

Publication Metadata

Publications often provide important biological and experimental context that may not be fully captured in repository records.

Examples:

  • Study objectives
  • Experimental design
  • Cohort descriptions
  • Inclusion and exclusion criteria

Supplementary Metadata

Many studies provide additional metadata files alongside publications.

Examples:

  • Sample sheets
  • Metadata spreadsheets
  • Clinical annotation tables

Curated Metadata

Curated metadata are generated during the data acquisition process by integrating information from multiple sources.

Examples:

  • Download manifests
  • Integrated metadata tables
  • Project inventories

Combining metadata from multiple sources often produces a more complete and reliable representation of a study than relying on a single repository alone.

Metadata Completeness

Not all studies provide the same level of metadata.

Rich metadata may include:

  • Age
  • Sex
  • Disease status
  • Treatment information
  • Body site
  • Collection date

Limited metadata may include only accession identifiers and basic sequencing information.

Metadata completeness often determines whether a dataset can be reused for a particular analysis.

For example, a study may contain sequencing data from human stool samples, but without information on disease status or treatment history, those samples may not be suitable for constructing a healthy reference dataset.

Inclusion and Exclusion Criteria

Metadata are frequently used to determine which studies and samples should be included in a project.

Examples of inclusion criteria:

  • Healthy individuals
  • Human gut microbiome samples
  • Shotgun metagenomic sequencing
  • Adequate metadata availability

Examples of exclusion criteria:

  • Disease cohorts
  • Intervention studies
  • Incomplete metadata
  • Non-target sample types

Clearly defined inclusion and exclusion criteria improve reproducibility and ensure consistent sample selection across studies.

Healthy Reference Microbiome Example

Building a healthy reference microbiome dataset requires careful metadata review before any sequencing files are downloaded.

Samples may be excluded because of:

  • Disease status
  • Antibiotic exposure
  • Missing metadata
  • Inappropriate sample types

A sequencing run may be technically valid and publicly available, yet still be excluded if the associated metadata do not satisfy project requirements. This illustrates why metadata evaluation is a critical step in the data acquisition workflow.

Metadata Challenges

Although public repositories contain vast amounts of information, metadata are often imperfect and require careful review.

Common metadata challenges include:

  • Missing metadata
  • Inconsistent terminology
  • Ambiguous sample descriptions
  • Repository-specific differences
  • Incomplete publications

For example, the same biological concept may be described differently across studies (e.g., “healthy”, “control”, or “non-diseased”), requiring additional interpretation during dataset curation.

These challenges often require manual review, metadata harmonization, and careful documentation.

Metadata as a Scientific Asset

Metadata are not merely supporting information.

Well-curated metadata become scientific assets that:

  • Enable reproducible research
  • Support dataset integration
  • Improve study interpretation
  • Facilitate downstream analyses

As projects grow, curated metadata often become as valuable as the sequencing data themselves because they provide the context required for meaningful biological interpretation.

Healthy Reference Microbiome Example

In projects involving healthy reference microbiome construction, metadata become the foundation for:

  • Study selection
  • Sample selection
  • Download planning
  • Dataset integration
  • Reference dataset assembly

For example, metadata can be used to identify healthy human gut microbiome samples, distinguish between single-end and paired-end sequencing studies, locate downloadable FASTQ files, and generate manifests for downstream acquisition workflows.

Installing Entrez Direct (EDirect)

EDirect provides command-line access to NCBI databases and is widely used for metadata retrieval from BioProject, BioSample, SRA, and PubMed.

Throughout this chapter we use three core commands:

  • esearch — search NCBI databases
  • efetch — retrieve records
  • xtract — extract fields from structured outputs

Environment Setup

Create an environment.yml file containing the required metadata acquisition tools.

name: cdi-data-acquisition

channels:
  - conda-forge
  - bioconda

dependencies:
  - entrez-direct

Create the environment:

mamba env create -f environment.yml

Activate the environment:

conda activate cdi-data-acquisition

Verification

Confirm that EDirect tools are available:

which esearch
which efetch
which xtract

Example output:

/Users/tmbmacbookair/anaconda3/envs/cdi-data-acquisition/bin/esearch
/Users/tmbmacbookair/anaconda3/envs/cdi-data-acquisition/bin/efetch
/Users/tmbmacbookair/anaconda3/envs/cdi-data-acquisition/bin/xtract

Next, test communication with NCBI:

esearch -db sra \
-query "PRJNA322554[bioproject]"

Example output:

<ENTREZ_DIRECT>
  <Db>sra</Db>
  <WebEnv>MCID_6a292bec9551de53c901708f</WebEnv>
  <QueryKey>1</QueryKey>
  <Count>188</Count>
  <Step>1</Step>
  <Elapsed>3</Elapsed>
</ENTREZ_DIRECT>

Interpretation:

  • Db indicates the database queried (sra).
  • WebEnv is an NCBI session identifier used internally by EDirect.
  • QueryKey identifies the result set within the session.
  • Count reports the number of matching sequencing runs (188 for PRJNA322554).
  • Step indicates the workflow step executed by EDirect.
  • Elapsed reports the query execution time in seconds.

A successful query confirms that EDirect is installed correctly and can communicate with NCBI. The result count also provides a useful first validation of the dataset before retrieving detailed metadata.

Tip

Using an environment file ensures that metadata acquisition workflows remain reproducible and portable across systems.

Worked Example: From BioProject to Download Manifest

The following example demonstrates how metadata acquisition transforms a BioProject accession into structured metadata assets that support downstream download and validation workflows.

BioProject:

PRJNA322554

Retrieve Run Metadata

esearch -db sra \
-query 'PRJNA322554[bioproject]' \
| efetch -format runinfo \
> data/metadata/runinfo-PRJNA322554.csv

This command retrieves the SRA RunInfo table and stores it locally for further inspection.

Validate Metadata

wc -l data/metadata/runinfo-PRJNA322554.csv

Expected:

189

Interpretation:

  • 188 sequencing runs
  • 1 header row
  • 189 total lines

Verifying record counts provides a simple but effective first validation step.

Inspect Metadata

head -5 data/metadata/runinfo-PRJNA322554.csv

The RunInfo table contains study, sample, experiment, run, and sequencing metadata that can be used to evaluate the suitability of a dataset.

Extract Run Accessions

cut -d',' -f1 data/metadata/runinfo-PRJNA322554.csv | head

Explanation:

  • -d',' specifies comma-delimited fields.
  • -f1 extracts the first column.

Remove the header and count sequencing runs:

tail -n +2 data/metadata/runinfo-PRJNA322554.csv | wc -l

Expected:

188

Create Download Manifest

cut -d',' -f1 data/metadata/runinfo-PRJNA322554.csv \
| tail -n +2 \
> data/metadata/srr-accessions.txt

The resulting manifest contains one run accession per line and can be used for downstream download workflows.

Metadata Validation

Metadata validation helps ensure that retrieved metadata are complete, internally consistent, and suitable for downstream analysis.

Validation activities include:

  • Record counts
  • Accession completeness
  • Field verification
  • Metadata inspection
  • Manifest generation

Typical outputs include:

  • runinfo-PRJNA322554.csv
  • srr-accessions.txt

These assets form the foundation of downstream download planning and sequencing data retrieval.

Metadata Acquisition Beyond NCBI

The INSDC Ecosystem

Public sequencing metadata are distributed across the International Nucleotide Sequence Database Collaboration (INSDC), a partnership between three major repositories:

INSDC
│
├── NCBI (USA)
├── ENA  (Europe)
└── DDBJ (Japan)

These repositories regularly exchange data and often contain the same underlying studies, samples, experiments, and sequencing runs.

As a result, metadata retrieved from one repository can often be validated against metadata obtained from another repository, providing an additional layer of confidence in the acquisition process.

ENA Example

In addition to NCBI, metadata can be retrieved directly from the European Nucleotide Archive (ENA).

curl -o data/metadata/ena-PRJNA322554.tsv \
"https://www.ebi.ac.uk/ena/portal/api/filereport?accession=PRJNA322554&result=read_run"

Example output:

% Total    % Received % Xferd  Average Speed  Time    Time    Time   Current
                                 Dload  Upload  Total   Spent   Left   Speed
100  24873   0  24873   0      0  17362      0           00:01              0

Interpretation:

  • 100 indicates that the download completed successfully.
  • 24873 bytes of metadata were retrieved.
  • The metadata were saved to:
data/metadata/ena-PRJNA322554.tsv

The ENA metadata table provides download-oriented information that may include:

  • Run accessions
  • Sample accessions
  • FASTQ locations
  • File sizes
  • MD5 checksums

These attributes are particularly useful for download planning, integrity verification, and downstream data acquisition workflows.

Validation

Confirm that the metadata file was created successfully:

wc -l data/metadata/ena-PRJNA322554.tsv

Example output:

189

Interpretation:

The ENA metadata file contains:

  • 188 sequencing run records
  • 1 header row

Cross-Repository Validation

Confirm the number of unique run accessions:

cut -f1 data/metadata/ena-PRJNA322554.tsv \
| tail -n +2 \
| sort -u \
| wc -l

Example output:

188

Compare this with the NCBI RunInfo metadata:

tail -n +2 data/metadata/runinfo-PRJNA322554.csv \
| wc -l

Example output:

188

The agreement between ENA and NCBI confirms that metadata retrieval was successful and complete.

Tip

Cross-repository validation provides additional confidence that metadata assets accurately represent the sequencing runs available for downstream download and validation workflows.

NCBI Versus ENA

Although NCBI and ENA exchange data through the INSDC partnership, they provide complementary strengths during the data acquisition process.

For BioProject PRJNA322554:

EDirect Search
      ↓
188 Runs

NCBI RunInfo
      ↓
188 Records

ENA File Report
      ↓
188 Records
Repository Primary Strength
NCBI Study discovery, metadata retrieval, accession relationships, and repository navigation
ENA Download-oriented metadata, FASTQ locations, file sizes, and MD5 checksums
DDBJ Alternative INSDC access point and data mirror

Repository Selection Strategy

A practical metadata acquisition workflow often combines NCBI and ENA.

Study Discovery
      ↓
NCBI
      ↓
Metadata Validation
      ↓
ENA
      ↓
Download Planning
      ↓
Data Download

In this workflow:

  • NCBI is used for study discovery and metadata retrieval.
  • ENA is used for download-oriented metadata such as FASTQ locations, file sizes, and MD5 checksums.
  • DDBJ serves as an additional access point within the INSDC ecosystem when needed.

This approach leverages the strengths of multiple repositories while maintaining a reproducible acquisition workflow.

Metadata Acquisition Outputs

The workflows presented in this chapter generate reusable metadata assets that support downstream data acquisition activities.

data/
└── metadata/
    ├── runinfo-PRJNA322554.csv
    ├── ena-PRJNA322554.tsv
    └── srr-accessions.txt
File Purpose
runinfo-PRJNA322554.csv Run-level metadata retrieved from NCBI SRA
ena-PRJNA322554.tsv Download-oriented metadata retrieved from ENA
srr-accessions.txt Download manifest containing sequencing run accessions

Together, these assets provide the foundation for reproducible download planning, sequencing data retrieval, validation, and reference dataset assembly.

Summary

In this chapter, we transformed a BioProject accession into validated metadata assets suitable for downstream acquisition workflows.

Study Accession
      ↓
Metadata Discovery
      ↓
Metadata Retrieval
      ↓
Metadata Validation
      ↓
Metadata Evaluation
      ↓
Download Manifest

Along the way, we explored:

  • Metadata categories and sources
  • Metadata completeness and quality
  • Inclusion and exclusion criteria
  • Metadata retrieval from NCBI using EDirect
  • Metadata retrieval from ENA using the ENA API
  • Cross-repository validation within the INSDC ecosystem
  • Download manifest generation
  • Metadata as a scientific asset

We also demonstrated how metadata from multiple repositories can be integrated and validated to support reproducible data acquisition workflows.

The resulting metadata assets provide the information needed to identify relevant samples, plan downloads, validate acquisitions, and support reference dataset construction.

Looking Ahead

Metadata acquisition determines what data should be downloaded.

The next stage of the CDI Data Acquisition System focuses on retrieving the sequencing data themselves.

In Chapter 05, we use the metadata assets and download manifests generated here to locate sequencing files, retrieve data from public repositories, verify download integrity, and prepare datasets for downstream validation and reference dataset assembly.

Metadata Acquisition
      ↓
Download Manifest
      ↓
Data Download
      ↓
FASTQ Files
      ↓
Data Validation