Datasets and data sources#

The datasets used by scdiffeq are downloaded on first use and cached locally. They are hosted on Zenodo:

DOI: 10.5281/zenodo.21947161

Downloads require no authentication, and each file is verified against the md5 checksum published in the record before it is written to the cache.

Note

The datasets were previously served from Figshare. Its download host now sits behind a web application firewall that rejects programmatic requests, so downloads are served from Zenodo instead. Figshare remains a fallback.

Citing the data#

The Zenodo record redistributes data published by other groups. Please cite the original publications, not only the record:

Dataset	Publication
LARRY in vitro	Weinreb C, Rodriguez-Fraticelli A, Camargo FD, Klein AM. Lineage tracing on transcriptional landscapes links state to fate during differentiation. Science (2020). 10.1126/science.aaw3381
Human hematopoiesis	Qiu X, Zhang Y, Martin-Rufino JD, et al. Mapping transcriptomic vector fields of single cells. Cell (2022). 10.1016/j.cell.2021.12.045
Pancreatic endocrinogenesis	Bastidas-Ponce A, Tritschler S, Dony L, et al. Comprehensive single cell mRNA profiling reveals a detailed roadmap for pancreatic endocrinogenesis. Development (2019). 10.1242/dev.173849

Where files are cached#

Each loader takes a data_dir argument (default: the current working directory) and writes beneath <data_dir>/scdiffeq_data/:

scdiffeq_data/
  larry/
    _larry.raw.h5ad                  # as downloaded
    larry.processed.h5ad             # after preprocessing
    scaler.pkl, pca.pkl              # fitted models
    larry.ct_obs_df.csv              # CytoTRACE annotations
    larry.ct_var_df.csv

The raw download and the preprocessed result are separate files. This means a dataset obtained by any route – including a manual download – is still preprocessed on first use, and a preprocessed file that fails a validity check is regenerated from the raw file rather than re-downloaded.

Note

Earlier versions stored both under a single name (larry.h5ad). Existing caches are detected and renamed into the new layout automatically; multi-GB files are not re-downloaded.

Preprocessing is re-run when the processed file is missing or fails validation. To force it explicitly:

import scdiffeq as sdq

# regenerate the processed file from the cached raw file (no re-download)
adata = sdq.datasets.larry(force_preprocess=True)

# re-download the raw file as well
adata = sdq.datasets.larry(force_download=True)

Reproducibility of the PCA#

Dimension reduction fits a 50-component PCA with random_state=0, so repeated runs produce an identical basis.

Warning

This makes results reproducible going forward. It does not reproduce the basis distributed with the original publications, which was computed with an unseeded randomized SVD and now exists only as a stored array. If you need that exact basis, use the X_pca shipped inside the dataset rather than recomputing it.

Working without network access#

Files can be fetched by hand from the Zenodo record and placed in the cache directory under the names above. The loaders will pick them up and preprocess them normally.