Acquiring and validating data are only part of the data acquisition lifecycle. Once datasets have been validated, they must be stored, transferred, organized, and shared in a manner that supports reproducibility, collaboration, and long-term accessibility.
The Cloud Storage and Transfer System provides the infrastructure layer connecting validated datasets to downstream analytical workflows and reference dataset assembly.
Modern omics datasets can range from gigabytes to terabytes in size.
Examples include:
Managing these datasets requires storage systems that are scalable, accessible, and resilient.
Examples:
Advantages:
Limitations:
Examples:
Advantages:
Examples:
Advantages:
flowchart TD
A[Metadata Acquisition System]
--> B[Data Download System]
B --> C[Data Validation System]
C --> D[Cloud Storage and Transfer System]
D --> E[Reference Dataset Assembly System]flowchart TD A[Metadata Acquisition System] --> B[Data Download System] B --> C[Data Validation System] C --> D[Cloud Storage and Transfer System] D --> E[Reference Dataset Assembly System]
Cloud storage serves as the bridge between validated acquisition outputs and downstream analytical workflows.
Only validated datasets should be transferred into long-term storage environments.
Metadata Assets
↓
Data Download
↓
Data Validation
↓
Cloud Storage
↓
Reference Dataset AssemblyThis approach ensures that incomplete or corrupted files are not propagated into downstream workflows.
A structured storage layout improves reproducibility, traceability, and collaboration.
The Cloud Storage and Transfer System should preserve the outputs generated during metadata acquisition, download, validation, and inventory generation.
Local project structure:
data/
├── metadata/
│ ├── runinfo-PRJNA322554.csv
│ ├── ena-PRJNA322554.tsv
│ └── srr-accessions.txt
│
├── manifests/
│ ├── download-manifest.tsv
│ └── test-manifest.tsv
│
├── inventory/
│ └── fastq-inventory.tsv
│
├── raw/
│ ├── sra/
│ └── fastq/
│
├── validation/
│ └── validation-report.tsv
│
└── reference-dataset/After transfer, the same structure can be preserved in cloud storage.
Example S3 layout:
s3://reference-datasets/
├── metadata/
├── manifests/
├── inventory/
├── raw/
├── validation/
└── reference-dataset/Preserving directory structure during transfer ensures that metadata, manifests, inventories, validation reports, and sequencing files remain linked throughout the data lifecycle.
Consistent organization simplifies:
Cloud storage systems support a variety of transfer mechanisms.
Common approaches include:
The choice depends on:
For large sequencing datasets, cloud-native transfer tools and object storage services are often preferred because they provide scalability, fault tolerance, and efficient handling of large file collections.
Typical workflow:
Validated Dataset
↓
Transfer Tool
↓
Cloud Storage
↓
Analysis EnvironmentReliable transfer methods help ensure that validated datasets remain complete, accessible, and reproducible throughout the data lifecycle.
Several cloud platforms are commonly used for omics data storage and distribution.
| Platform | Example Service |
|---|---|
| Amazon Web Services | S3 |
| Google Cloud Platform | Cloud Storage |
| Microsoft Azure | Blob Storage |
| Institutional Infrastructure | Object storage and research clusters |
All of these platforms support large-scale storage of sequencing datasets and can be integrated into reproducible acquisition workflows.
Object storage systems are particularly well suited for sequencing datasets because they provide:
These capabilities are especially valuable for large microbiome, RNA-seq, single-cell, and multi-omics projects where datasets may contain hundreds or thousands of sequencing files.
Cloud storage is not merely a backup location.
It becomes part of the reproducibility infrastructure.
Validated Dataset
↓
Cloud Storage
↓
Shared Access
↓
Reference Dataset Assembly
↓
Downstream AnalysisBy preserving validated datasets together with their metadata, manifests, inventories, and validation reports, cloud storage enables reproducible reuse of acquired data across projects, collaborators, and analytical environments.
After datasets have passed validation, they can be transferred to cloud storage for long-term preservation, collaboration, and downstream reference dataset assembly.
Example:
bash scripts/bash/07a-sync-to-s3.sh \
data \
reference-datasetsWorkflow:
Validated Dataset
↓
Cloud Transfer
↓
Cloud Storage
↓
Reference Dataset RepositoryCloud transfer scripts automate the movement of validated datasets while preserving directory structure, metadata assets, inventories, and validation reports.
Only validated datasets should be transferred into long-term storage environments.
Every transfer should preserve provenance information.
Important records include:
Provenance enables datasets to be traced, verified, and reconstructed throughout their lifecycle.
Example provenance chain:
Public Repository
↓
Metadata Acquisition
↓
Download Manifest
↓
FASTQ Inventory
↓
Validation Report
↓
Cloud StorageMaintaining provenance records supports:
Not all datasets are publicly accessible.
Researchers may encounter:
Storage systems should support appropriate access controls to ensure that datasets are accessed only by authorized users and handled according to applicable policies and agreements.
Common mechanisms include:
Although many public omics datasets are openly available, secure storage and access management become increasingly important when working with collaborative or controlled-access projects.
A healthy reference microbiome project may follow:
Public Repositories
↓
Metadata Filtering
↓
Validated Downloads
↓
Cloud Storage
↓
Reference Dataset AssemblyCloud storage becomes the staging area where validated datasets are organized, preserved, and prepared for integration into a reference dataset.
For example, metadata tables, download manifests, FASTQ inventories, validation reports, and sequencing files can all be preserved together within a shared storage environment.
Researchers frequently encounter:
Planning storage workflows early helps reduce these risks and improves long-term dataset management.
Before transferring data to cloud storage, verify that:
Checklist workflow:
Metadata
↓
Downloads
↓
Validation
↓
Inventory
↓
Cloud TransferCompleting these checks helps ensure that only validated and reproducible datasets are transferred to long-term storage.
The Cloud Storage and Transfer System extends the CDI Data Acquisition System beyond the local workstation.
Validated Dataset
↓
Cloud Transfer
↓
Cloud Storage
↓
Collaborative Access
↓
Reference Dataset AssemblyBy combining structured storage, reliable transfer methods, provenance tracking, validation records, and appropriate access controls, researchers can create durable and reproducible datasets suitable for long-term analysis and collaboration.
Once datasets have been acquired, validated, transferred, and organized, the next challenge is transforming them into coherent and reusable reference datasets.
In the next chapter, we explore the Reference Dataset Assembly System and the process of integrating validated data into analytical resources that support reproducible biological research.
Validated Datasets
↓
Cloud Storage
↓
Reference Dataset Assembly
↓
Reusable Research Asset