Datasets (sdq.datasets)#

The datasets used by scdiffeq are downloaded on first use and cached locally. They are hosted on Zenodo:

Downloads require no authentication, and each file is verified against the md5 checksum published in the record before it is written to the cache.

Note

The datasets were previously served from Figshare. Its download host now sits behind a web application firewall that rejects programmatic requests, so downloads are served from Zenodo instead. Figshare remains a fallback.

Loaders#

sdq.datasets.larry
scdiffeq.datasets._larry_in_vitro.larry(data_dir: str = '/home/runner/work/scDiffEq/scDiffEq/docs/source', variant: str | None = None, filter_genes: bool = True, reduce_dimensions: bool = True, cytotrace: bool = True, force_download: bool = False, force_preprocess: bool = False) → AnnData[source]

LARRY in vitro dataset

The raw download and the preprocessed result are cached as separate files, so a dataset obtained by any means (including a manual download) is still preprocessed on first use.

Parameters:

  • data_dir – str, default=os.getcwd() Path to the directory where the data will be saved.

  • variant –

    Optional[str], default=None Dataset variant.

    None (default) is the biology-rich object: 130,887 x 2,492, already gene-filtered, shipping a precomputed X_pca, X_umap and X_scaled.

    "unprocessed" is the upstream input: 130,887 x 25,289 with no X_pca, carrying a use_genes column that preprocessing filters by (down to 2,447 genes). X_pca is computed locally, or downloaded prebuilt when available.

    variant="fate_prediction" is a deprecated alias for "unprocessed" and will be removed in a future release. Despite its upstream filename (...in_vitro.gene_filtered.h5ad), that object is the unfiltered one.

  • filter_genes – bool, default=True Whether to subset to adata.var['use_genes']. A no-op for variants that are already gene-filtered.

  • reduce_dimensions – bool, default=True Whether to scale and run PCA (50 components, random_state=0).

  • cytotrace – bool, default=True Whether to annotate with precomputed CytoTRACE values. Applied independently of the other preprocessing flags.

  • force_download – bool, default=False Re-download the raw file and regenerate the processed file.

  • force_preprocess – bool, default=False Regenerate the processed file from the cached raw file without re-downloading it.

Returns:

Preprocessed AnnData object.

Return type:

AnnData

More: sdq.datasets.larry

sdq.datasets.human_hematopoiesis
scdiffeq.datasets._human_hematopoiesis.human_hematopoiesis(data_dir: str = '/home/runner/work/scDiffEq/scDiffEq/docs/source', skip_scaling: bool = False, force_download: bool = False, download_unprocessed: bool = False) → AnnData[source]

Human hematopoiesis dataset

Parameters:

  • data_dir – str, default=os.getcwd() Path to the directory where the data will be saved.

  • skip_scaling – bool, default=False Whether to skip scaling.

  • force_download – bool, default=False Whether to force download the data.

  • download_unprocessed – bool, default=False Whether to download the unprocessed data.

Returns:

Preprocessed AnnData object.

Return type:

AnnData

More: sdq.datasets.human_hematopoiesis

sdq.datasets.pancreatic_endocrinogenesis
scdiffeq.datasets._pancreatic_endocrinogenesis.pancreatic_endocrinogenesis(data_dir: str = '/home/runner/work/scDiffEq/scDiffEq/docs/source', filter_genes: bool = True, reduce_dimensions: bool = True, force_download: bool = False, download_unprocessed: bool = False) → AnnData[source]

Pancreas dataset

Parameters:

  • data_dir – str, default=os.getcwd() Path to the directory where the data will be saved.

  • filter_genes – bool, default=True Whether to filter genes.

  • reduce_dimensions – bool, default=True Whether to reduce dimensions.

  • force_download – bool, default=False Whether to force download the data.

  • download_unprocessed – bool, default=False Whether to download the unprocessed data.

Returns:

Preprocessed AnnData object.

Return type:

AnnData

More: sdq.datasets.pancreatic_endocrinogenesis

Citing the data#

The Zenodo record redistributes data published by other groups. Please cite the original publications, not only the record:

Dataset

Publication

LARRY in vitro

Weinreb C, Rodriguez-Fraticelli A, Camargo FD, Klein AM. Lineage tracing on transcriptional landscapes links state to fate during differentiation. Science (2020). 10.1126/science.aaw3381

Human hematopoiesis

Qiu X, Zhang Y, Martin-Rufino JD, et al. Mapping transcriptomic vector fields of single cells. Cell (2022). 10.1016/j.cell.2021.12.045

Pancreatic endocrinogenesis

Bastidas-Ponce A, Tritschler S, Dony L, et al. Comprehensive single cell mRNA profiling reveals a detailed roadmap for pancreatic endocrinogenesis. Development (2019). 10.1242/dev.173849

Where files are cached#

Each loader takes a data_dir argument (default: the current working directory) and writes beneath <data_dir>/scdiffeq_data/:

scdiffeq_data/
  larry/
    _larry.raw.h5ad                  # as downloaded
    larry.processed.h5ad             # after preprocessing
    scaler.pkl, pca.pkl              # fitted models
    larry.ct_obs_df.csv              # CytoTRACE annotations
    larry.ct_var_df.csv

The raw download and the preprocessed result are separate files. This means a dataset obtained by any route – including a manual download – is still preprocessed on first use, and a preprocessed file that fails a validity check is regenerated from the raw file rather than re-downloaded.

Note

Earlier versions stored both under a single name (larry.h5ad). Existing caches are detected and renamed into the new layout automatically; multi-GB files are not re-downloaded.

Preprocessing is re-run when the processed file is missing or fails validation. To force it explicitly:

import scdiffeq as sdq

# regenerate the processed file from the cached raw file (no re-download)
adata = sdq.datasets.larry(force_preprocess=True)

# re-download the raw file as well
adata = sdq.datasets.larry(force_download=True)

Reproducibility of the PCA#

Dimension reduction fits a 50-component PCA with random_state=0, so repeated runs produce an identical basis.

Warning

This makes results reproducible going forward. It does not reproduce the basis distributed with the original publications, which was computed with an unseeded randomized SVD and now exists only as a stored array. If you need that exact basis, use the X_pca shipped inside the dataset rather than recomputing it.

Working without network access#

Files can be fetched by hand from the Zenodo record and placed in the cache directory under the names above. The loaders will pick them up and preprocess them normally.