`just_dna_enricher.download`¶

just_dna_enricher.download ¶

HuggingFace snapshot provisioning — the bulk-fetch side of the resolver chain.

Two reference snapshots share one download body: the Ensembl parquet slice (popular rsIDs) and the ClinVar parquet slice (clinically-curated GRCh38 records). Neither is a canonical reference — each is a static slice with the usual pains (incompleteness, versions, reachability; HF has gone dark mid-demo) — so each is one resolver-chain source downloaded into a cache the resolver reads directly (<cache>/data/*.parquet), with a miss falling through to the next link.

huggingface_hub is imported lazily and guarded: it is the one HuggingFace dependency in the whole workspace, permitted only in this network tier (the 0.5 Constitution amendment scopes the HF ban to format + compiler). The download logic (footer-checked, atomic .part rename) is inherited from just-dna-lite's pipelines byte-for-byte so no drift is born.

ConstraintReferenceError ¶

Bases: FileNotFoundError

Raised when the gnomAD constraint snapshot cannot be provisioned or has no usable parquet.

GatedSnapshotError ¶

Bases: FileNotFoundError

Raised when a licence-gated snapshot (ClinPGx, CPIC) cannot be provisioned.

One class for both because a caller's recovery is identical — build it locally, or point at a cache — and because the two are the same act: reaching a source that forbids sale through bytes the operator took once instead of live per request.

OpenSnapshotError ¶

Bases: FileNotFoundError

Raised when an openly-licensed snapshot (CIViC, STRchive) cannot be provisioned.

GatedSnapshotError's counterpart, and one class for both of these for its reason: the caller's recovery is identical — build it locally, or point at a cache. What separates the two classes is not the failure but the question a reader asks next. A gated snapshot failing may mean the operator never accepted the terms; these two are CC0 and MIT, so nothing here is ever a licence problem and a reader chasing one would be chasing nothing.

SnapshotNotPublished ¶

Bases: FileNotFoundError

The repo or prefix a lane publishes to does not exist yet — asked, and absent.

A distinct type because it is a distinct answer, and the two were folded into one until the default cache pull started exiting 1 on a fresh machine. Three new lanes gained an ensure_* in RM176 whose repos nobody has created, and the bare fs.ls failure reached cache pull's blanket except Exception as a failure — so the command that provisions a deployment reported an error for a snapshot that simply has not been published. Nobody-published is the same third state as nobody-asked (@unreachable-not-absent), and a caller has to be able to tell it from a download that broke, which means a class rather than a message (@client-exception-contract: the transport's own exception type is not this module's contract).

It subclasses FileNotFoundError and not GatedSnapshotError/OpenSnapshotError, deliberately: a caller that catches either of those to mean "this lane is unusable" still catches this, and one that wants to distinguish asks for this type by name.

ensure_snapshot ¶

ensure_snapshot(ensembl_cache: Path | None = None) -> Path

Provision the Ensembl parquet cache from HuggingFace Hub, returning the cache directory.

Source code in enricher/src/just_dna_enricher/download.py

def ensure_snapshot(ensembl_cache: Path | None = None) -> Path:
    """Provision the Ensembl parquet cache from HuggingFace Hub, returning the cache directory."""
    cache_dir = Path(ensembl_cache) if ensembl_cache is not None else default_ensembl_cache_dir()
    return _provision_snapshot(
        cache_dir,
        _ENSEMBL_HF_PREFIX,
        label="Ensembl",
        error_cls=EnsemblReferenceError,
        filename_glob=SNAPSHOT_FILE_GLOBS["ensembl"],
    )

ensure_clinvar_snapshot ¶

ensure_clinvar_snapshot(
    clinvar_cache: Path | None = None,
) -> Path

Provision the ClinVar parquet cache from HuggingFace Hub, returning the cache directory.

Source code in enricher/src/just_dna_enricher/download.py

def ensure_clinvar_snapshot(clinvar_cache: Path | None = None) -> Path:
    """Provision the ClinVar parquet cache from HuggingFace Hub, returning the cache directory."""
    cache_dir = Path(clinvar_cache) if clinvar_cache is not None else default_clinvar_cache_dir()
    return _provision_snapshot(
        cache_dir,
        _CLINVAR_HF_PREFIX,
        label="ClinVar",
        error_cls=ClinVarReferenceError,
        filename_glob=SNAPSHOT_FILE_GLOBS["clinvar"],
    )

ensure_constraint_snapshot ¶

ensure_constraint_snapshot(
    constraint_cache: Path | None = None,
) -> Path

Provision the gnomAD constraint parquet cache from HuggingFace Hub.

The third caller of one download body — the plumbing generalized when ClinVar landed, so this is parameterization rather than new machinery.

Source code in enricher/src/just_dna_enricher/download.py

def ensure_constraint_snapshot(constraint_cache: Path | None = None) -> Path:
    """Provision the gnomAD constraint parquet cache from HuggingFace Hub.

    The third caller of one download body — the plumbing generalized when ClinVar landed, so this is
    parameterization rather than new machinery.
    """
    cache_dir = Path(constraint_cache) if constraint_cache is not None else default_constraint_cache_dir()
    return _provision_snapshot(
        cache_dir,
        _CONSTRAINT_HF_PREFIX,
        label="gnomAD constraint",
        error_cls=ConstraintReferenceError,
        filename_glob=SNAPSHOT_FILE_GLOBS["constraint"],
    )

ensure_clinpgx_snapshot ¶

ensure_clinpgx_snapshot(
    clinpgx_cache: Path | None = None,
) -> Path

Provision the ClinPGx clinical-annotation snapshot from HuggingFace Hub (RM38).

The builder shipped a release ahead of this, so the snapshot existed and no code path could reach it: enrich_clinpgx skipped itself unless a caller passed --snapshot by hand, which on a hosted deployment means the check simply never ran. Publishable because ClinPGx's recorded terms permit redistribution — and the LICENSE.txt the builder extracted travels with the parquet, which is what makes license_sha256 pin anything for whoever downloads it.

Source code in enricher/src/just_dna_enricher/download.py

def ensure_clinpgx_snapshot(clinpgx_cache: Path | None = None) -> Path:
    """Provision the ClinPGx clinical-annotation snapshot from HuggingFace Hub (RM38).

    The builder shipped a release ahead of this, so the snapshot existed and no code path could reach
    it: `enrich_clinpgx` skipped itself unless a caller passed `--snapshot` by hand, which on a hosted
    deployment means the check simply never ran. Publishable because ClinPGx's recorded terms permit
    redistribution — and the `LICENSE.txt` the builder extracted travels with the parquet, which is what
    makes `license_sha256` pin anything for whoever downloads it.
    """
    cache_dir = Path(clinpgx_cache) if clinpgx_cache is not None else default_clinpgx_cache_dir()
    return _provision_snapshot(
        cache_dir,
        _CLINPGX_HF_PREFIX,
        label="ClinPGx",
        error_cls=GatedSnapshotError,
        filename_glob=SNAPSHOT_FILE_GLOBS["clinpgx"],
    )

ensure_cpic_snapshot ¶

ensure_cpic_snapshot(
    cpic_cache: Path | None = None,
) -> Path

Provision the CPIC snapshot from HuggingFace Hub (RM38).

CPIC is open and unauthenticated, so the cache is not about access — it is about a host not spending one shared per-IP allowance on every caller's request, and about the terms being accepted once by the operator who built it rather than implicitly per request.

There is deliberately no ensure_pharmvar_snapshot beside this. PharmVar's bulk data is pulled under a key its terms §2 make personal and non-transferable, and no axis SourceTerms records covers passing that on — an unestablished permission is not a permission. That snapshot stays operator-built and inject-only (locations.resolve_pharmvar_reference).

Source code in enricher/src/just_dna_enricher/download.py

def ensure_cpic_snapshot(cpic_cache: Path | None = None) -> Path:
    """Provision the CPIC snapshot from HuggingFace Hub (RM38).

    CPIC is open and unauthenticated, so the cache is not about access — it is about a *host* not
    spending one shared per-IP allowance on every caller's request, and about the terms being accepted
    once by the operator who built it rather than implicitly per request.

    There is deliberately **no `ensure_pharmvar_snapshot`** beside this. PharmVar's bulk data is pulled
    under a key its terms §2 make personal and non-transferable, and no axis `SourceTerms` records
    covers passing that on — an unestablished permission is not a permission. That snapshot stays
    operator-built and inject-only (`locations.resolve_pharmvar_reference`).
    """
    cache_dir = Path(cpic_cache) if cpic_cache is not None else default_cpic_cache_dir()
    return _provision_snapshot(
        cache_dir,
        _CPIC_HF_PREFIX,
        label="CPIC",
        error_cls=GatedSnapshotError,
        filename_glob=SNAPSHOT_FILE_GLOBS["cpic"],
    )

ensure_civic_snapshot ¶

ensure_civic_snapshot(
    civic_cache: Path | None = None,
) -> Path

Provision the CIViC snapshot from HuggingFace Hub (RM176).

The cache roster recorded this one's absence as a gap, in those words, beside PharmVar's and PubMind's refusals: CIViC is CC0 on every axis, so nothing ever barred publishing a snapshot and none existed only because nobody had run civic publish. Closing it needed a repo, not a permission — so the first civic publish is what makes this function find anything.

Source code in enricher/src/just_dna_enricher/download.py

def ensure_civic_snapshot(civic_cache: Path | None = None) -> Path:
    """Provision the CIViC snapshot from HuggingFace Hub (RM176).

    The cache roster recorded this one's absence as a **gap**, in those words, beside PharmVar's and
    PubMind's refusals: CIViC is CC0 on every axis, so nothing ever barred publishing a snapshot and
    none existed only because nobody had run `civic publish`. Closing it needed a repo, not a
    permission — so the first `civic publish` is what makes this function find anything.
    """
    cache_dir = Path(civic_cache) if civic_cache is not None else default_civic_cache_dir()
    return _provision_snapshot(
        cache_dir,
        _CIVIC_HF_PREFIX,
        label="CIViC",
        error_cls=OpenSnapshotError,
        filename_glob=SNAPSHOT_FILE_GLOBS["civic"],
    )

ensure_drug_labels_snapshot ¶

ensure_drug_labels_snapshot(
    drug_labels_cache: Path | None = None,
) -> Path

Provision the regulator drug-label snapshot from HuggingFace Hub (RM176).

Publishable on the annotation lane's grounds and gated on its terms — ClinPGx's CC BY-SA permits redistribution and forbids sale, which is why the error type is GatedSnapshotError and why cache pull puts this lane through check_declared_use before fetching. LICENSE.txt rides along with the parquet, because a share-alike snapshot whose terms did not travel pins nothing for whoever holds the bytes.

Source code in enricher/src/just_dna_enricher/download.py

def ensure_drug_labels_snapshot(drug_labels_cache: Path | None = None) -> Path:
    """Provision the regulator drug-label snapshot from HuggingFace Hub (RM176).

    Publishable on the annotation lane's grounds and gated on its terms — ClinPGx's CC BY-SA permits
    redistribution and forbids sale, which is why the error type is `GatedSnapshotError` and why
    `cache pull` puts this lane through `check_declared_use` before fetching. `LICENSE.txt` rides
    along with the parquet, because a share-alike snapshot whose terms did not travel pins nothing
    for whoever holds the bytes.
    """
    cache_dir = Path(drug_labels_cache) if drug_labels_cache is not None else default_drug_labels_cache_dir()
    return _provision_snapshot(
        cache_dir,
        _DRUG_LABELS_HF_PREFIX,
        label="ClinPGx drug labels",
        error_cls=GatedSnapshotError,
        filename_glob=SNAPSHOT_FILE_GLOBS["drug_labels"],
    )

ensure_mitomap_snapshot ¶

ensure_mitomap_snapshot(
    mitomap_cache: Path | None = None,
) -> Path

Provision the MITOMAP snapshot from HuggingFace Hub (RM171).

Publishable on the source's own terms — CC BY 3.0, with commercial and clinical use stated free — so this is a gap-closing ensure_* like CIViC's rather than a permission being claimed. The glob is mitomap-*.parquet rather than *.parquet for the reason the constants above record: a published dataset keeps files from every earlier layout, and the reader would otherwise union a foreign schema into the same directory.

Source code in enricher/src/just_dna_enricher/download.py

def ensure_mitomap_snapshot(mitomap_cache: Path | None = None) -> Path:
    """Provision the MITOMAP snapshot from HuggingFace Hub (RM171).

    Publishable on the source's own terms — CC BY 3.0, with commercial and clinical use stated free —
    so this is a gap-closing `ensure_*` like CIViC's rather than a permission being claimed. The glob
    is `mitomap-*.parquet` rather than `*.parquet` for the reason the constants above record: a
    published dataset keeps files from every earlier layout, and the reader would otherwise union a
    foreign schema into the same directory.
    """
    cache_dir = Path(mitomap_cache) if mitomap_cache is not None else default_mitomap_cache_dir()
    return _provision_snapshot(
        cache_dir,
        _MITOMAP_HF_PREFIX,
        label="MITOMAP",
        error_cls=OpenSnapshotError,
        filename_glob=SNAPSHOT_FILE_GLOBS["mitomap"],
    )

ensure_alphagenome_avi_snapshot ¶

ensure_alphagenome_avi_snapshot(
    alphagenome_avi_cache: Path | None = None,
) -> Path

Provision the AVI snapshot from HuggingFace Hub (RM198).

A pull, never a build. The 88.5 GB source is behind an eligibility gate that bars classes of holder outright, so this tier cannot acquire it — but the re-encoded snapshot is 34 GB of Permissive-Use output (RM195), and redistribution=True records the reading that publishing it openly falls inside prohibition 1's carve-out. So an operator who cannot download the artifact can still pull the lane, which is the whole point of publishing it.

It is also the first lane whose snapshot is incomplete without a root-level file: PHRED is not stored and avi_knots.parquet is what reconstructs it, so a pull that brought only the parquets would hand over scores nobody can rank. That file travels because SNAPSHOT_ROOT_FILENAMES names it, not because this function does.

Source code in enricher/src/just_dna_enricher/download.py

def ensure_alphagenome_avi_snapshot(alphagenome_avi_cache: Path | None = None) -> Path:
    """Provision the AVI snapshot from HuggingFace Hub (RM198).

    **A pull, never a build.** The 88.5 GB source is behind an eligibility gate that bars classes of
    holder outright, so this tier cannot acquire it — but the *re-encoded* snapshot is 34 GB of
    Permissive-Use output (RM195), and `redistribution=True` records the reading that publishing it
    openly falls inside prohibition 1's carve-out. So an operator who cannot download the artifact can
    still pull the lane, which is the whole point of publishing it.

    It is also the first lane whose snapshot is **incomplete without a root-level file**: `PHRED` is
    not stored and `avi_knots.parquet` is what reconstructs it, so a pull that brought only the
    parquets would hand over scores nobody can rank. That file travels because
    `SNAPSHOT_ROOT_FILENAMES` names it, not because this function does.
    """
    cache_dir = (
        Path(alphagenome_avi_cache)
        if alphagenome_avi_cache is not None
        else default_alphagenome_avi_cache_dir()
    )
    return _provision_snapshot(
        cache_dir,
        _ALPHAGENOME_AVI_HF_PREFIX,
        label="AlphaGenome AVI",
        error_cls=OpenSnapshotError,
        filename_glob=SNAPSHOT_FILE_GLOBS["alphagenome_avi"],
    )

ensure_strchive_snapshot ¶

ensure_strchive_snapshot(
    strchive_cache: Path | None = None,
) -> Path

Provision the STRchive catalogue from HuggingFace Hub (RM176).

Not _provision_snapshot, and the difference is the snapshot rather than the plumbing. That body is parquet all the way down — it globs data/*.parquet, trusts a populated directory, and re-fetches anything failing the PAR1 footer check. This snapshot is one JSON catalogue at the repo root: there is no data/, no footer to check, and no way to tell a truncated JSON file from a short one without parsing it. So the catalogue is fetched to a .part path and parsed before it is renamed into place, which is the same guarantee the footer check gives the parquet lanes — an interrupted download never lands under the real name.

Source code in enricher/src/just_dna_enricher/download.py

def ensure_strchive_snapshot(strchive_cache: Path | None = None) -> Path:
    """Provision the STRchive catalogue from HuggingFace Hub (RM176).

    **Not `_provision_snapshot`, and the difference is the snapshot rather than the plumbing.** That
    body is parquet all the way down — it globs `data/*.parquet`, trusts a populated directory, and
    re-fetches anything failing the `PAR1` footer check. This snapshot is one JSON catalogue at the
    repo root: there is no `data/`, no footer to check, and no way to tell a truncated JSON file from
    a short one without parsing it. So the catalogue is fetched to a `.part` path and parsed before it
    is renamed into place, which is the same guarantee the footer check gives the parquet lanes —
    an interrupted download never lands under the real name.
    """
    cache_dir = Path(strchive_cache) if strchive_cache is not None else default_strchive_cache_dir()
    return _provision_root_file_snapshot(
        cache_dir,
        _STRCHIVE_HF_REPO,
        payload=STRCHIVE_CATALOGUE_FILENAME,
        label="STRchive",
        error_cls=OpenSnapshotError,
    )

just_dna_enricher.download¶