Skip to content

just_dna_enricher.upload

just_dna_enricher.upload

HuggingFace upload — publisher surface of the network tier.

Two publish shapes share one create-or-update pathway (ensure_repo):

  • module — a compiled module's manifest.json and every file it declares (the attested parquets, the logo, the readme, provenance.json and each log; RM347, RM353) to datasets/<repo>/data/<name>/, matching the layout just-dna-lite's discovery scans, and to data/<name>/v<version>/ when the manifest states a version, so the discovery path can address a particular release rather than only "whatever is there now" (upload_module; RM84).
  • reference snapshot — a built ClinVar (or Ensembl) parquet snapshot + release.json to the root of a dataset repo, matching the download.ensure_*_snapshot layout (publish_reference_snapshot).

Both require a HuggingFace token with write access (hf auth login or HF_TOKEN). This is the dev/publisher half of the enricher's HuggingFace use: snapshot download is a runtime enrich path (download.ensure_snapshot); upload is for authors republishing modules (e.g. the Gen-I v1-port recreation) or publishing a rebuilt reference. Install as just-dna-enricher[dev].

Extracted from just_dna_pipelines.v1_port.publish (just-dna-lite); create_repo was added here (the origin assumed the repo pre-existed) so a brand-new dataset repo can be created on first push.

LayoutShift dataclass

LayoutShift(
    repo_id: str, retires: str, introduces: str, reason: str
)

A retirement declared by the commit that causes it, so the publish carries out the migration.

The rule (maintainer, 2026-09-03). A change that retires one published file and introduces another must also carry a conditioned update to the publication procedure: if the new file is absent from the repo and the old one is present, upload the new and delete the old. The condition is a predicate over the remote, so it fires exactly once per repo and is a no-op forever after — nothing is deleted on a repo that has already moved, and re-running a publish cannot re-delete.

This is the only deletion a publish performs. General cleanup is cache prune, which prints what it would remove and asks; a deletion nobody declared is not a side effect a publish may have. What makes this one safe is not that it is small but that it is named: the pair is in the commit that changed the layout, where a reviewer sees both halves at once.

Deleting is recoverable in a way this repository's @snapshot-accumulates note did not credit — a HuggingFace dataset repo is git-backed, and a superseded revision still resolves (three of them were read off the hub while auditing this on 2026-09-03). The reason to declare rather than sweep is not that bytes are lost; it is that a retired file goes on answering 200 to whoever still asks for it, which is how a lane's default archive stayed frozen for a year (CLINPGX_ARCHIVES).

PublishRefused

Bases: Exception

A publish refused on what it would send — the publisher's one error family (RM368).

Every refusal this module raises is a subclass, so a caller catches the family once and tells the members apart by type: the local module or snapshot is not publishable (ModuleNotPublishable, SnapshotNotPublishable), a planned file changed before it was sent (PlanChangedError), the remote already holds something else (PublishCollisionError, ToolchainDiffersError, OrphanedSidecarError), or the remote moved after the plan read it (RemoteMovedError). Two things stay outside it because they say nothing about what would be sent: a missing write token (PermissionError), and whatever the hub client itself raises.

A base class, so a caller's except order is load-bearing (@client-exception-contract): a handler for a member goes before the handler for the family, as the CLI's do.

ModuleNotPublishable

Bases: PublishRefused, FileNotFoundError

The module directory is not publishable as it stands (plan_upload's refusals, RM368).

It used to be a bare FileNotFoundError, which a caller could not tell from a missing directory. It is still one within 0.8 — a second base kept so an existing except FileNotFoundError keeps catching it; catch PublishRefused or this type instead.

SnapshotNotPublishable

Bases: PublishRefused, FileNotFoundError

The snapshot directory has nothing built to publish (plan_reference_snapshot, RM368).

Still a FileNotFoundError within 0.8, for the reason ModuleNotPublishable is.

PlanChangedError

Bases: PublishRefused

A planned file's bytes are not the ones the plan read and checked (RM368, RM386).

The module send reads each planned file once more and sends exactly the bytes it hashed; a file whose hash is not the plan's (a recompile landing between the check and the send, an edit) or that cannot be read any more refuses the publish before anything is committed. A snapshot's payload is too large to hold, so the hub reads it from the directory: what the hub then holds is checked against the plan's hashes, and a difference withholds the description (release.json), so nothing ever describes bytes nothing checked.

RemoteMovedError

Bases: PublishRefused

The repo is not at the revision the plan read, so the write would act on a state nobody checked (RM386).

Every publish and prune reads the repo at one revision (_head) and commits on top of exactly that revision (parent_commit), and the hub refuses a commit whose parent is no longer the head. A concurrent writer between the read and the write — another publish, a prune — therefore refuses this one instead of being overwritten unseen. A prune plan that states no revision read none and refuses too. Nothing was committed by the write that refused; re-run to read the repo again.

OrphanedSidecarError

Bases: PublishRefused, RuntimeError

The publish would overwrite a release.json describing sidecars it is not carrying (RM185).

Its own type for the reason PublishCollisionError is one: the CLI has to tell it from the refusals plan_* raises. Those say the local snapshot is not publishable; this one says the local snapshot is fine and the remote holds bytes this publish would silently stop describing.

The incident. A ClinVar snapshot is two halves on two cadences — the per-chromosome parquet and citations/citations.parquet — and release.json describes both because build_citations merges a block into it. A rebuild that built only the VCF half published a citations-free release.json over a repo whose sidecar it had not replaced, and the publisher adds without deleting: the sidecar outlived its own description, and the published artifact carried records from one ClinVar release beside citations from another while saying nothing. RM179 fixed the lane; this is the general shape, refused at the boundary where any lane could repeat it.

PublishCollisionError

Bases: PublishRefused, RuntimeError

The versioned path already holds a different release, and no --force was given (RM88).

Its own type rather than a ModuleNotPublishable, because the CLI has to tell it apart from the refusals plan_upload raises: those say the local module is not publishable, this one says the local module is fine and the remote already has that version.

reason is None on the plain collision — nothing recorded says why the bytes differ — and a subclass sets it when something does (ToolchainDiffersError).

ToolchainDiffersError

ToolchainDiffersError(
    message: str,
    *,
    published_toolchain: dict[str, str],
    local_toolchain: dict[str, str],
    content_unchanged: bool | None,
)

Bases: PublishCollisionError

The plain collision's one explained case: the two manifests recorded different toolchains (RM342).

From polars 2 the parquet footer names the polars release that wrote it, so an unchanged spec recompiled beside another polars mints another digest. That is not corruption, and the maintainer's decision (2026-11-04) is that it is never reported as the plain mismatch. The publish still refuses unless --force: the versioned path's bytes stay the ones that version was cut as (Principle 4), and the reason changes what the refusal says, not whether it refuses.

A subclass, so every handler of PublishCollisionError still refuses. A sibling type would turn a handled refusal into an uncaught one in any caller that catches only the collision; the cost is that a caller wanting the distinction must catch this type first (@client-exception-contract), which the CLI does. Tell the two apart by type or by reason, never by the message.

The reason names a co-occurring difference, not a proven cause. A content edit recompiled under another polars raises this too, which is why content_unchanged rides beside it: True when both manifests carry one content_signature (the toolchain alone can account for the digest), False when the content moved as well, None when either side states none.

Source code in enricher/src/just_dna_enricher/upload.py
def __init__(
    self,
    message: str,
    *,
    published_toolchain: dict[str, str],
    local_toolchain: dict[str, str],
    content_unchanged: bool | None,
) -> None:
    super().__init__(message)
    self.published_toolchain = published_toolchain
    self.local_toolchain = local_toolchain
    self.content_unchanged = content_unchanged

UploadPlan

Bases: BaseModel

What an upload would send (also the dry-run result).

SnapshotPlan

Bases: BaseModel

What a reference-snapshot publish would send (also the dry-run result).

PruneCandidate

Bases: BaseModel

One remote file cache prune would remove, and the reason it is nameable as removable.

PrunePlan

Bases: BaseModel

What a prune would delete from one repo. The dry run and the deletion read the same object.

ensure_repo

ensure_repo(repo_id: str, token: str | None = None)

Create-or-update: ensure the dataset repo exists, returning the authenticated HfApi.

create_repo(..., exist_ok=True) is a no-op when the repo already exists, so create and update are one pathway. The returned api is reused by the caller's upload_folder so only one HfApi is constructed per publish.

Source code in enricher/src/just_dna_enricher/upload.py
def ensure_repo(repo_id: str, token: str | None = None):
    """Create-or-update: ensure the dataset repo exists, returning the authenticated ``HfApi``.

    ``create_repo(..., exist_ok=True)`` is a no-op when the repo already exists, so create and update
    are one pathway. The returned api is reused by the caller's ``upload_folder`` so only one
    ``HfApi`` is constructed per publish.
    """
    api = _hf_api(repo_id, token)
    api.create_repo(repo_id=repo_id, repo_type="dataset", exist_ok=True)
    return api

layout_shifts_to_apply

layout_shifts_to_apply(
    repo_files: Iterable[str], repo_id: str
) -> list[LayoutShift]

The declared shifts whose condition the remote currently satisfies. Reads; decides nothing else.

Separate from the publish so it can be asserted directly: the whole safety of this mechanism is that the predicate is false the moment it has run once, and that is a property of this function rather than of the caller that acts on it.

Source code in enricher/src/just_dna_enricher/upload.py
def layout_shifts_to_apply(repo_files: Iterable[str], repo_id: str) -> list[LayoutShift]:
    """The declared shifts whose condition the remote currently satisfies. Reads; decides nothing else.

    Separate from the publish so it can be asserted directly: the whole safety of this mechanism is
    that the predicate is false the moment it has run once, and that is a property of this function
    rather than of the caller that acts on it.
    """
    remote = list(repo_files)
    due: list[LayoutShift] = []
    for shift in LAYOUT_SHIFTS:
        if shift.repo_id != repo_id:
            continue
        if shift.retires not in remote:
            continue  # already retired, or never there
        if any(fnmatch(name, shift.introduces) for name in remote):
            continue  # the repo has already moved: not this one's job
        due.append(shift)
    return due

plan_upload

plan_upload(
    module_dir: Path, name: str, repo_id: str | None = None
) -> UploadPlan

Resolve the upload plan and validate the compiled artifacts are present.

Two destinations, both planned here (RM84): the flat data/<name>/, which is what the reference consumer's discovery scans and therefore has to keep working and keep meaning latest, and data/<name>/v<version>/ inside it — nested rather than a sibling, because the flat path is the one deployed readers are pointed at — which is the only thing on that path that can name a particular release. A module with no readable version gets the flat path alone, never a vNone directory, and version_unknown_reason says which of the four reasons applies. The reason is a field rather than a log line for the reason every other withheld answer here is (clin_sig_not_checked, skipped_offline): a caller that has to parse prose to learn what happened has not been told.

The files are what the manifest declares (RM353, @publisher-allowlist-derived). When manifest.artifact.files is legible, the plan is manifest.json plus every file the manifest names in the module directory — the attested parquets, the logo, the readme, provenance.json and each log — and nothing else: a parquet an earlier compile left behind, or a logo the manifest does not name, would arrive unattested, and a stale parquet this publish put on the flat path would read to discovery as part of the module. (One already on the remote stays; a publish deletes only what it declares.) Each declared file sent must still hash to the sha256 the manifest states, or the plan refuses: the manifest is the guard's comparator, so it has to be true of the bytes. A declared optional file that leaves the module directory is skipped with a warning naming the reason; one absent from disk is skipped quietly, since every optional file is optional (verify skips an absent log or logo the same way). When the manifest cannot say — absent, unparseable, or with no artifact.files list — the plan falls back to _ALLOW_PATTERNS found on disk: an unknown withholds, and a directory with no manifest has always been publishable.

What it refuses, in order, most specific first so a refusal names the actual fault — each a ModuleNotPublishable (RM368):

  • A declared field in a shape the compiler never writes (artifact.files that is not a list, a sha256 that is not a string, …), quoting each. Absent is unknown and withholds; present and unreadable used to read as absent too and dropped the gate that field feeds.
  • An attested entry naming no file — anything but an object with a non-blank string name.
  • Every attested artifact file is sent. artifact.digest is a Merkle root over exactly the files manifest.artifact.files names, so an attested file missing from disk, or one whose name leaves the module directory, would make the published manifest a false claim about bytes that are not there.
  • A declared file whose bytes are not the ones declared (the sha256 differs).
  • weights.parquet never travels alone. A SNP core compiles to all three, so a missing annotations/studies beside it is an interrupted compile.
  • At least one lead table. A directory with a manifest, a readme and no annotation rows is not a module; publishing one is the silent failure a bare widening of the old rule would have produced, and it is what discovery would then be unable to see.

The plan is one reading of the disk (RM368). Every planned file is hashed once, here, into UploadPlan.sha256 — manifest.json from the very bytes it was parsed from — and the upload sends exactly those bytes or refuses (_verified_payload).

Source code in enricher/src/just_dna_enricher/upload.py
def plan_upload(
    module_dir: Path,
    name: str,
    repo_id: str | None = None,
) -> UploadPlan:
    """Resolve the upload plan and validate the compiled artifacts are present.

    Two destinations, both planned here (RM84): the flat `data/<name>/`, which is what the reference
    consumer's discovery scans and therefore has to keep working and keep meaning *latest*, and
    `data/<name>/v<version>/` **inside it** — nested rather than a sibling, because the flat path is
    the one deployed readers are pointed at — which is the only thing on that path that can name a
    particular release. A module with no readable version gets the flat path alone, never a `vNone`
    directory, and `version_unknown_reason` says which of the four reasons applies. The reason is a
    field rather than a log line for the reason every other withheld answer here is
    (`clin_sig_not_checked`, `skipped_offline`): a caller that has to parse prose to learn what
    happened has not been told.

    **The files are what the manifest declares (RM353, `@publisher-allowlist-derived`).** When
    `manifest.artifact.files` is legible, the plan is `manifest.json` plus every file the manifest
    names in the module directory — the attested parquets, the logo, the readme, `provenance.json` and
    each log — and nothing else: a parquet an earlier compile left behind, or a logo the manifest does
    not name, would arrive unattested, and a stale parquet this publish put on the flat path would read
    to discovery as part of the module. (One already on the remote stays; a publish deletes only what
    it declares.) Each declared file sent must still hash to the sha256 the manifest states, or the
    plan refuses: the manifest is the guard's comparator, so it has to be true of the bytes. A
    declared optional file that leaves the module directory is skipped with a warning
    naming the reason; one absent from disk is skipped quietly, since every optional file is optional
    (`verify` skips an absent log or logo the same way). **When the manifest cannot say** — absent,
    unparseable, or with no `artifact.files` list — the plan falls back to `_ALLOW_PATTERNS` found on
    disk: an unknown withholds, and a directory with no manifest has always been publishable.

    **What it refuses, in order, most specific first so a refusal names the actual fault** — each a
    `ModuleNotPublishable` (RM368):

    - **A declared field in a shape the compiler never writes** (`artifact.files` that is not a list,
      a `sha256` that is not a string, …), quoting each. Absent is unknown and withholds; present and
      unreadable used to read as absent too and dropped the gate that field feeds.
    - **An attested entry naming no file** — anything but an object with a non-blank string `name`.
    - **Every attested artifact file is sent.** `artifact.digest` is a Merkle root over exactly the
      files `manifest.artifact.files` names, so an attested file missing from disk, or one whose name
      leaves the module directory, would make the published manifest a false claim about bytes that
      are not there.
    - **A declared file whose bytes are not the ones declared** (the sha256 differs).
    - **`weights.parquet` never travels alone.** A SNP core compiles to all three, so a missing
      `annotations`/`studies` beside it is an interrupted compile.
    - **At least one lead table.** A directory with a manifest, a readme and no annotation rows is not
      a module; publishing one is the silent failure a bare widening of the old rule would have
      produced, and it is what discovery would then be unable to see.

    **The plan is one reading of the disk (RM368).** Every planned file is hashed once, here, into
    `UploadPlan.sha256` — `manifest.json` from the very bytes it was parsed from — and the upload sends
    exactly those bytes or refuses (`_verified_payload`).
    """
    read = _manifest_document(module_dir)
    return _plan_from_document(module_dir, name, repo_id, read)

rehearse_upload

rehearse_upload(
    module_dir: Path,
    name: str,
    repo_id: str | None = None,
    token: str | None = None,
    force: bool = False,
) -> UploadPlan

What upload_module would do, short of writing: the plan, and the versioned-path check (RM368).

The dry run. It runs every check the real run runs — the plan's refusals and, when the module states a version and force is not set, the collision check against the published manifest, read with a read-only client — and raises what the real run would raise. It creates no repo and commits nothing. A hub it cannot reach is the real run's unknown: a warning, and the plan.

plan_upload alone stays the offline half, for a caller that wants the plan with no remote read.

Source code in enricher/src/just_dna_enricher/upload.py
def rehearse_upload(
    module_dir: Path,
    name: str,
    repo_id: str | None = None,
    token: str | None = None,
    force: bool = False,
) -> UploadPlan:
    """What `upload_module` would do, short of writing: the plan, and the versioned-path check (RM368).

    The dry run. It runs every check the real run runs — the plan's refusals and, when the module
    states a version and `force` is not set, the collision check against the published manifest, read
    with a read-only client — and raises what the real run would raise. It creates no repo and commits
    nothing. A hub it cannot reach is the real run's unknown: a warning, and the plan.

    `plan_upload` alone stays the offline half, for a caller that wants the plan with no remote read.
    """
    read = _manifest_document(module_dir)
    plan = _plan_from_document(module_dir, name, repo_id, read)
    if not force and plan.versioned_path_in_repo is not None and read.document is not None:
        _refuse_a_collision(_read_only_api(token), plan, read.document, name, revision=None)
    return plan

upload_module

upload_module(
    module_dir: Path,
    name: str,
    repo_id: str | None = None,
    token: str | None = None,
    commit_message: str | None = None,
    force: bool = False,
) -> UploadPlan

Upload the compiled module to a HuggingFace dataset collection.

Ensures the dataset repo exists (create-or-update), then writes the same files to the flat path and — when the manifest states a version — to the versioned subdirectory under it.

What is sent is the plan's bytes (RM368). The plan hashed every file it names; after the collision check, each is read once more, checked against that hash, and sent as those bytes through create_commit — never re-read from the directory by the hub. A file that changed in between (a recompile still writing, an edit) is a PlanChangedError before either commit.

And the revision it read (RM386). The repo is read once at its head, the versioned-path check reads at that revision, the flat commit is pinned on it and the versioned commit on the flat one (_commit), so a version another publisher lands after the check is a RemoteMovedError rather than an overwrite nobody saw.

The two writes are two commits, not one. Each create_commit is its own commit, so a reader can briefly see the flat path refreshed while the versioned copy is not there yet; the flat path goes first because it is the one anything reads today. One commit carrying both paths' operations is now a one-line change and was not taken here: two commits is the documented behaviour, and a re-run, which is idempotent, is the documented recovery when the second fails.

The versioned path refuses to be overwritten with different bytes (RM88, RM353, RM368). Before either write, the published manifest.json at data/<name>/v<version>/ is read and compared with this module's: its artifact.digest, the sha256 of every file it declares, and every other field (_versioned_conflict). A difference is a PublishCollisionError unless force=True. The flat path is deliberately not guarded: it means latest, so overwriting it is what it is for, and the whole point of the versioned copy is that it does not.

The policy is refuse-unless---force rather than warn-and-proceed or refuse-outright, decided 2026-08-18. The flag's existence is itself the claim that overwriting is sometimes right — a curator re-cutting a draft release is a real workflow, and a gate with no override becomes a gate people route around.

Recompiling under a newer compiler also moves the digest, and will trip this. That is correct rather than a false positive: P4 pins byte-reproducibility to compiler_version, the recorded toolchain and the resolved reference, so the versioned path really would come to hold different bytes than the ones it was published with. The refusal says so, because "I changed nothing" is the first thing its first user will think.

When the artifacts differ and both manifests recorded a toolchain and the two differ, the refusal is a ToolchainDiffersError (RM342), which names both toolchains. Either side unrecorded (a manifest compiled before 0.8), the same toolchain on both, or a difference in declared files or other fields alone, keeps the plain PublishCollisionError.

Raises PermissionError if no token is available.

Source code in enricher/src/just_dna_enricher/upload.py
def upload_module(
    module_dir: Path,
    name: str,
    repo_id: str | None = None,
    token: str | None = None,
    commit_message: str | None = None,
    force: bool = False,
) -> UploadPlan:
    """Upload the compiled module to a HuggingFace dataset collection.

    Ensures the dataset repo exists (create-or-update), then writes the same files to the flat path
    and — when the manifest states a version — to the versioned subdirectory under it.

    **What is sent is the plan's bytes (RM368).** The plan hashed every file it names; after the
    collision check, each is read once more, checked against that hash, and sent as those bytes through
    `create_commit` — never re-read from the directory by the hub. A file that changed in between (a
    recompile still writing, an edit) is a `PlanChangedError` before either commit.

    **And the revision it read (RM386).** The repo is read once at its head, the versioned-path check
    reads at that revision, the flat commit is pinned on it and the versioned commit on the flat one
    (`_commit`), so a version another publisher lands after the check is a `RemoteMovedError` rather
    than an overwrite nobody saw.

    **The two writes are two commits, not one.** Each `create_commit` is its own commit, so a reader can
    briefly see the flat path refreshed while the versioned copy is not there yet; the flat path goes
    first because it is the one anything reads today. One commit carrying both paths' operations is
    now a one-line change and was not taken here: two commits is the documented behaviour, and a
    re-run, which is idempotent, is the documented recovery when the second fails.

    **The versioned path refuses to be overwritten with different bytes (RM88, RM353, RM368).** Before
    either write, the published `manifest.json` at `data/<name>/v<version>/` is read and compared with
    this module's: its `artifact.digest`, the sha256 of every file it declares, and every other field
    (`_versioned_conflict`). A difference is a `PublishCollisionError` unless `force=True`. The flat
    path is deliberately **not** guarded: it means *latest*, so overwriting it is what it is for, and
    the whole point of the versioned copy is that it does not.

    The policy is refuse-unless-`--force` rather than warn-and-proceed or refuse-outright, decided
    2026-08-18. The flag's existence is itself the claim that overwriting is sometimes right — a
    curator re-cutting a draft release is a real workflow, and a gate with no override becomes a gate
    people route around.

    **Recompiling under a newer compiler also moves the digest**, and will trip this. That is correct
    rather than a false positive: P4 pins byte-reproducibility to `compiler_version`, the recorded
    toolchain and the resolved reference, so the versioned path really would come to hold different
    bytes than the ones it was published with. The refusal says so, because "I changed nothing" is the
    first thing its first user will think.

    **When the artifacts differ and both manifests recorded a toolchain and the two differ, the refusal
    is a `ToolchainDiffersError` (RM342)**, which names both toolchains. Either side unrecorded (a
    manifest compiled before 0.8), the same toolchain on both, or a difference in declared files or
    other fields alone, keeps the plain `PublishCollisionError`.

    Raises PermissionError if no token is available.
    """
    read = _manifest_document(module_dir)
    plan = _plan_from_document(module_dir, name, repo_id, read)
    api = ensure_repo(plan.repo_id, token)
    revision = _head(api, plan.repo_id)
    if not force:
        _refuse_a_collision(api, plan, read.document, name, revision)
    payload = _verified_payload(module_dir, plan)
    flat = _commit(
        api,
        plan.repo_id,
        revision,
        commit_message or f"Add {name} module",
        operations=_commit_operations(plan.path_in_repo, payload),
    )
    if plan.versioned_path_in_repo is not None:
        segment = plan.versioned_path_in_repo.rsplit("/", 1)[-1]
        _commit(
            api,
            plan.repo_id,
            flat,
            commit_message or f"Add {name} module {segment}",
            operations=_commit_operations(plan.versioned_path_in_repo, payload),
            landed=f"the flat path {plan.path_in_repo}/ at {flat[:12]}, without its versioned copy",
        )
    return plan

plan_reference_snapshot

plan_reference_snapshot(
    snapshot_dir: Path,
    repo_id: str | None = None,
    *,
    payload: str | None = None,
) -> SnapshotPlan

Resolve a snapshot publish plan and validate the built artifacts are present.

payload names a snapshot whose content is one file at the root rather than data/*.parquet — STRchive's STRchive-loci.json is the case, and ACMG's acmg_sf.csv is the shape's second member. The parameter is the caller's, deliberately: a lane knows the name of the file it builds, and putting a roster of lane filenames in the publisher would make this function the fourth place that has to learn about a new snapshot kind.

A payload snapshot is refused when the file is missing, exactly as a parquet one is when data/ is empty — the refusal is the same claim either way, that there is nothing built here to publish.

Source code in enricher/src/just_dna_enricher/upload.py
def plan_reference_snapshot(
    snapshot_dir: Path,
    repo_id: str | None = None,
    *,
    payload: str | None = None,
) -> SnapshotPlan:
    """Resolve a snapshot publish plan and validate the built artifacts are present.

    `payload` names a snapshot whose content is **one file at the root** rather than
    `data/*.parquet` — STRchive's `STRchive-loci.json` is the case, and ACMG's `acmg_sf.csv` is the
    shape's second member. The parameter is the caller's, deliberately: a lane knows the name of the
    file it builds, and putting a roster of lane filenames in the publisher would make this function
    the fourth place that has to learn about a new snapshot kind.

    A `payload` snapshot is refused when the file is missing, exactly as a parquet one is when
    `data/` is empty — the refusal is the same claim either way, that there is nothing built here to
    publish.
    """
    resolved_repo = repo_id or DEFAULT_CLINVAR_REPO_ID
    if payload is not None:
        if not (snapshot_dir / payload).is_file():
            raise SnapshotNotPublishable(
                f"no {payload} in {snapshot_dir} — build the snapshot first "
                f"(e.g. `just-dna-enricher strchive build`)"
            )
        # Same registry as the parquet branch below, and for the same reason: this branch had its own
        # copy of the pair, so a payload-only lane that gained a root file would have dropped it
        # exactly the way the parquet branch dropped `LICENSE.txt`.
        files = [payload]
        files.extend(name for name in SNAPSHOT_ROOT_FILENAMES if (snapshot_dir / name).is_file())
        return SnapshotPlan(repo_id=resolved_repo, files=files)
    data_dir = snapshot_dir / SNAPSHOT_DATA_DIRNAME
    parquet = sorted(p.name for p in data_dir.glob("*.parquet")) if data_dir.is_dir() else []
    if not parquet:
        raise SnapshotNotPublishable(
            f"no {SNAPSHOT_DATA_DIRNAME}/*.parquet in {snapshot_dir} — build the snapshot first "
            f"(e.g. `just-dna-enricher clinvar build`)"
        )
    files = [f"{SNAPSHOT_DATA_DIRNAME}/{name}" for name in parquet]
    # Sidecars, when the snapshot has them. Only ClinVar does, and only when `clinvar citations` was
    # run, so absence is normal rather than a reason to refuse.
    for sidecar in SNAPSHOT_SIDECAR_DIRNAMES:
        directory = snapshot_dir / sidecar
        if directory.is_dir():
            files.extend(f"{sidecar}/{p.name}" for p in sorted(directory.glob("*.parquet")))
    # Walked from `locations.SNAPSHOT_ROOT_FILENAMES`, never listed here. A file this function does
    # not name is a file the publish silently drops, and the snapshot still looks complete on the
    # other side — which is how a share-alike snapshot went out without its terms
    # (`@publisher-allowlist-derived`). Absence stays normal: only ClinPGx carries a licence and only
    # the AVI lane carries a knot table.
    for name in SNAPSHOT_ROOT_FILENAMES:
        if (snapshot_dir / name).is_file():
            files.append(name)
    return SnapshotPlan(repo_id=resolved_repo, files=files)

check_publish_orphans_no_sidecar

check_publish_orphans_no_sidecar(
    plan: SnapshotPlan,
    api=None,
    token: str | None = None,
    *,
    revision: str | None = None,
) -> None

Refuse a publish that would leave a remote sidecar undescribed. Reads the repo; writes nothing.

The remote tree, not the remote release.json. The block is a description of the bytes and the bytes are what a puller gets, so the description is the half that can already be missing — which is exactly the state just-dna-seq/clinvar was found in on 2026-09-03: the citations parquet present, the block gone. A guard reading the block would have passed the second bad publish as happily as the first.

Scoped to publishes that carry release.json, because a publish that carries none overwrites no provenance. A repo that does not exist yet lists nothing and passes — that is a first publish, not an orphan.

api is the caller's when it has one (a publish has just built an authenticated client and there is no reason to build a second); a dry run passes none and gets an anonymous reader, since listing a public repo needs no token and a dry run must not require write access to say what a publish would do.

revision is the commit a publish read and will commit on (RM386); a dry run passes none and reads the head.

Source code in enricher/src/just_dna_enricher/upload.py
def check_publish_orphans_no_sidecar(
    plan: SnapshotPlan, api=None, token: str | None = None, *, revision: str | None = None
) -> None:
    """Refuse a publish that would leave a remote sidecar undescribed. Reads the repo; writes nothing.

    **The remote tree, not the remote `release.json`.** The block is a *description* of the bytes and
    the bytes are what a puller gets, so the description is the half that can already be missing —
    which is exactly the state `just-dna-seq/clinvar` was found in on 2026-09-03: the citations parquet
    present, the block gone. A guard reading the block would have passed the second bad publish as
    happily as the first.

    Scoped to publishes that carry `release.json`, because a publish that carries none overwrites no
    provenance. A repo that does not exist yet lists nothing and passes — that is a first publish, not
    an orphan.

    `api` is the caller's when it has one (a publish has just built an authenticated client and there
    is no reason to build a second); a dry run passes none and gets an anonymous reader, since listing
    a public repo needs no token and a dry run must not require write access to say what a publish
    would do.

    `revision` is the commit a publish read and will commit on (RM386); a dry run passes none and reads
    the head.
    """
    if RELEASE_FILENAME not in plan.files:
        return
    if api is None:
        api = _read_only_api(token)
    try:
        remote = list(api.list_repo_files(repo_id=plan.repo_id, repo_type="dataset", revision=revision))
    except Exception as exc:
        # Nobody has published here yet, or the listing failed. Neither is an orphan, and a publish
        # that cannot read the repo will fail on its own terms a moment later with a better message.
        logger.info(
            "Could not list %s (%s); publishing without the sidecar check.", plan.repo_id, type(exc).__name__
        )
        return
    carried = {path.split("/", 1)[0] for path in plan.files if "/" in path}
    for sidecar in SNAPSHOT_SIDECAR_DIRNAMES:
        if sidecar in carried:
            continue
        orphaned = sorted(f for f in remote if f.startswith(f"{sidecar}/") and f.endswith(".parquet"))
        if not orphaned:
            continue
        raise OrphanedSidecarError(
            f"{plan.repo_id} carries {', '.join(orphaned)}, and this publish would replace "
            f"{RELEASE_FILENAME} without carrying {sidecar}/ — so the published artifact would hold "
            f"two vintages and describe one. Build the missing half into the snapshot before "
            f"publishing (for ClinVar: `just-dna-enricher clinvar citations --out <snapshot>`), or "
            f"publish from a directory that has it."
        )

rehearse_reference_snapshot

rehearse_reference_snapshot(
    snapshot_dir: Path,
    repo_id: str | None = None,
    *,
    payload: str | None = None,
    token: str | None = None,
) -> SnapshotPlan

What publish_reference_snapshot would do, short of writing: the plan and the orphan check.

The dry run of every snapshot publish command (RM368). A rehearsal that skips the check the real publish refuses on is a different operation — two commands ran the orphan check in their dry run and the rest did not, so their --dry-run promised a publish the real run could refuse. Reads the published repo's file list; uploads nothing.

Source code in enricher/src/just_dna_enricher/upload.py
def rehearse_reference_snapshot(
    snapshot_dir: Path,
    repo_id: str | None = None,
    *,
    payload: str | None = None,
    token: str | None = None,
) -> SnapshotPlan:
    """What `publish_reference_snapshot` would do, short of writing: the plan and the orphan check.

    The dry run of every snapshot publish command (RM368). A rehearsal that skips the check the real
    publish refuses on is a different operation — two commands ran the orphan check in their dry run
    and the rest did not, so their `--dry-run` promised a publish the real run could refuse. Reads the
    published repo's file list; uploads nothing.
    """
    plan = plan_reference_snapshot(snapshot_dir, repo_id, payload=payload)
    check_publish_orphans_no_sidecar(plan, token=token)
    return plan

plan_prune

plan_prune(
    repo_id: str, filename_glob: str, api=None
) -> PrunePlan

What the published repo carries that this lane's snapshot is not made of. Reads; deletes nothing.

Two sources of a name, and neither is a guess. A file under data/ that the lane's own glob excludes is not part of the snapshot by the same definition provisioning uses — _provision_snapshot already refuses to download it and warns when it finds one locally. And a file a LayoutShift declares retired is nameable even where the glob cannot see it: for the four lanes whose glob is *.parquet the exclusion set is empty by construction, so declaration is the only way a retired file there is ever identified.

Everything else is left alone, including files this tier never wrote — README.md, .gitattributes, release.json, LICENSE.txt, and any sidecar directory. A pruner that removed what it did not recognise would be the sweep this design exists to avoid.

Source code in enricher/src/just_dna_enricher/upload.py
def plan_prune(repo_id: str, filename_glob: str, api=None) -> PrunePlan:
    """What the published repo carries that this lane's snapshot is not made of. Reads; deletes nothing.

    **Two sources of a name, and neither is a guess.** A file under `data/` that the lane's own glob
    excludes is not part of the snapshot by the same definition provisioning uses — `_provision_snapshot`
    already refuses to download it and warns when it finds one locally. And a file a `LayoutShift`
    declares retired is nameable even where the glob cannot see it: for the four lanes whose glob is
    `*.parquet` the exclusion set is empty by construction, so declaration is the only way a retired
    file there is ever identified.

    Everything else is left alone, including files this tier never wrote — `README.md`,
    `.gitattributes`, `release.json`, `LICENSE.txt`, and any sidecar directory. A pruner that removed
    what it did not recognise would be the sweep this design exists to avoid.
    """
    if api is None:
        api = huggingface_hub.HfApi(token=env_value("HF_TOKEN") or huggingface_hub.get_token())
    # The listing is pinned to one revision, and the plan carries it, so the deletion can refuse a repo
    # that changed after a person was shown the list (RM386).
    revision = _head(api, repo_id)
    entries = list(
        api.list_repo_tree(
            repo_id=repo_id, repo_type="dataset", recursive=True, expand=True, revision=revision
        )
    )
    sizes = {getattr(e, "path", ""): getattr(e, "size", None) for e in entries}
    remote = [path for path in sizes if path]

    candidates: dict[str, PruneCandidate] = {}
    prefix = f"{SNAPSHOT_DATA_DIRNAME}/"
    for path in sorted(remote):
        if not (path.startswith(prefix) and path.endswith(".parquet")):
            continue
        if fnmatch(path[len(prefix) :], filename_glob):
            continue
        candidates[path] = PruneCandidate(
            path=path,
            size=sizes.get(path),
            reason=f"under {SNAPSHOT_DATA_DIRNAME}/ but outside this snapshot ({filename_glob})",
        )
    for shift in LAYOUT_SHIFTS:
        if shift.repo_id != repo_id or shift.retires not in sizes:
            continue
        # A declared retirement wins the reason slot: it says *when* and *why* the file stopped being
        # part of the snapshot, which "outside the glob" does not.
        candidates[shift.retires] = PruneCandidate(
            path=shift.retires,
            size=sizes.get(shift.retires),
            reason=f"declared retired — {shift.reason}",
        )
    return PrunePlan(
        repo_id=repo_id, candidates=[candidates[k] for k in sorted(candidates)], revision=revision
    )

prune_repo

prune_repo(
    plan: PrunePlan,
    token: str | None = None,
    commit_message: str | None = None,
) -> int

Delete a prune plan's files. Called only after the caller has shown the plan and been told yes.

Returns the number of paths deleted. Deleting is a commit on a git-backed repo, so a superseded revision still resolves — but that is a reason to be able to undo a mistake, never a reason to remove something nobody looked at, which is why this takes a plan rather than a repo id.

Exactly the listed paths, at the listed revision (RM386). One delete operation per candidate — a path, never a pattern the hub matches (RM353 escaped the patterns; there are none now) — in one commit whose parent is the revision the listing was read at. A repo that changed after the plan was shown (a file republished under a listed name) is a RemoteMovedError and nothing is deleted; so is a plan that states no revision, since it read nothing a deletion could be held to.

Source code in enricher/src/just_dna_enricher/upload.py
def prune_repo(plan: PrunePlan, token: str | None = None, commit_message: str | None = None) -> int:
    """Delete a prune plan's files. Called only after the caller has shown the plan and been told yes.

    Returns the number of paths deleted. Deleting is a commit on a git-backed repo, so a superseded
    revision still resolves — but that is a reason to be able to undo a mistake, never a reason to
    remove something nobody looked at, which is why this takes a plan rather than a repo id.

    **Exactly the listed paths, at the listed revision (RM386).** One delete operation per candidate —
    a path, never a pattern the hub matches (RM353 escaped the patterns; there are none now) — in one
    commit whose parent is the revision the listing was read at. A repo that changed after the plan
    was shown (a file republished under a listed name) is a `RemoteMovedError` and nothing is deleted;
    so is a plan that states no revision, since it read nothing a deletion could be held to.
    """
    if not plan.candidates:
        return 0
    if plan.revision is None:
        raise RemoteMovedError(
            f"this prune plan for {plan.repo_id} read no revision, so there is no repo state its "
            f"candidates were listed from; nothing was deleted. Plan it with `plan_prune`."
        )
    api = _hf_api(plan.repo_id, token)
    _commit(
        api,
        plan.repo_id,
        plan.revision,
        commit_message or f"Prune {len(plan.candidates)} file(s) that are not part of this snapshot",
        operations=[CommitOperationDelete(path_in_repo=c.path) for c in plan.candidates],
    )
    return len(plan.candidates)

publish_reference_snapshot

publish_reference_snapshot(
    snapshot_dir: Path,
    repo_id: str | None = None,
    token: str | None = None,
    commit_message: str | None = None,
    *,
    payload: str | None = None,
) -> SnapshotPlan

Create-or-update a dataset repo and upload a built reference snapshot to its root.

Uploads data/*.parquet + any parquet sidecars (ClinVar's citations/) + release.json, so the tree matches download.ensure_*_snapshot and a provisioned snapshot is the same artifact a built one is — PMIDs included, which is what a drafted gene panel needs to compile.

Tied to the bytes it read and the revision it read (RM386). The repo is read at one revision and both commits are pinned on it (_commit), so a concurrent writer refuses the publish (RemoteMovedError) rather than being overwritten. Every planned file is hashed once; the payload goes up through the hub's own uploader, and what the hub then holds is checked against those hashes before the description is sent (PlanChangedError if not); the description is sent as the bytes read with the rest, never read again. The returned plan states the revision and the hashes.

Raises PermissionError if no token is available.

Source code in enricher/src/just_dna_enricher/upload.py
def publish_reference_snapshot(
    snapshot_dir: Path,
    repo_id: str | None = None,
    token: str | None = None,
    commit_message: str | None = None,
    *,
    payload: str | None = None,
) -> SnapshotPlan:
    """Create-or-update a dataset repo and upload a built reference snapshot to its root.

    Uploads ``data/*.parquet`` + any parquet sidecars (ClinVar's ``citations/``) + ``release.json``, so
    the tree matches ``download.ensure_*_snapshot`` and a provisioned snapshot is the same artifact a
    built one is — PMIDs included, which is what a drafted gene panel needs to compile.

    **Tied to the bytes it read and the revision it read (RM386).** The repo is read at one revision
    and both commits are pinned on it (`_commit`), so a concurrent writer refuses the publish
    (`RemoteMovedError`) rather than being overwritten. Every planned file is hashed once; the payload
    goes up through the hub's own uploader, and what the hub then holds is checked against those
    hashes before the description is sent (`PlanChangedError` if not); the description is sent as the
    bytes read with the rest, never read again. The returned plan states the revision and the hashes.

    Raises PermissionError if no token is available.
    """
    plan = plan_reference_snapshot(snapshot_dir, repo_id, payload=payload)
    api = ensure_repo(plan.repo_id, token)
    revision = _head(api, plan.repo_id)
    check_publish_orphans_no_sidecar(plan, api, revision=revision)
    # A declared retirement rides in the same commit as the upload that replaces it (RM186). Listed
    # again at the same revision: both answers are about one pinned state, so they cannot disagree.
    try:
        remote = list(api.list_repo_files(repo_id=plan.repo_id, repo_type="dataset", revision=revision))
    except Exception as exc:
        logger.info(
            "Could not list %s (%s); no declared retirement can apply.", plan.repo_id, type(exc).__name__
        )
        remote = []
    due = layout_shifts_to_apply(remote, plan.repo_id)
    if due:
        logger.info(
            "Retiring %s from %s in this commit: %s",
            ", ".join(shift.retires for shift in due),
            plan.repo_id,
            "; ".join(shift.reason for shift in due),
        )

    # **One `upload_folder`, and the size branch is gone** (RM199, revised 2026-09-11).
    #
    # This used to pick `upload_large_folder` above 5 GB, because `upload_folder` was a single
    # non-resumable commit and 32 GB in one of those starts over on any failure. `huggingface_hub`
    # 1.x settled it the other way: **`upload_folder` is itself multi-commit now**, and
    # `upload_large_folder` emits a `FutureWarning` telling callers to stop using it. Found the way
    # deprecations should be — the warning appeared in a real publish.
    #
    # Two things fall out. The threshold and its two constants are gone, and so is the refusal this
    # function used to raise when a declared retirement met the large path: `upload_folder` takes
    # `delete_patterns`, so RM186's arrival-and-departure stays on one call and there is no longer a
    # collision to refuse. What remains upstream's rather than ours is that a *large* upload is
    # several commits either way — so the one-commit guarantee holds for the payloads that fit in one
    # and is the Hub's business for the payloads that do not.
    #
    # **The description still goes last** (RM199's real content). `release.json` is what a puller reads to learn
    # which release it holds, so it must never arrive before the bytes it describes: a publish that
    # lands the description and then fails leaves a snapshot that *reads as* provisioned and is not.
    # That is `@a-publish-may-not-orphan-the-bytes-it-stops-describing` from the other direction, and
    # it costs one extra commit on a path that is already several.
    payload_files = [f for f in plan.files if f != RELEASE_FILENAME]
    # **The one local read** (RM386). Every planned file is hashed here; the description's bytes are
    # kept and are what is sent, and the payload is held to these hashes on the hub before anything
    # describes it. A file that cannot be read now is a refusal, as at the module send.
    planned: dict[str, _ReadOnce] = {}
    description: bytes | None = None
    for name in plan.files:
        try:
            if name == RELEASE_FILENAME:
                description = (snapshot_dir / name).read_bytes()
                planned[name] = _ReadOnce.of_bytes(description)
            else:
                planned[name] = _read_once(snapshot_dir / name)
        except OSError as exc:
            raise PlanChangedError(f"{name} in {snapshot_dir} could not be read to send it ({exc})") from exc
    plan = plan.model_copy(
        update={"sha256": {name: read.sha256 for name, read in planned.items()}, "revision": revision}
    )

    sent = _commit(
        api,
        plan.repo_id,
        revision,
        commit_message
        or (
            f"Publish reference snapshot ({len(payload_files)} files)"
            + (f", retiring {', '.join(s.retires for s in due)}" if due else "")
        ),
        folder=snapshot_dir,
        # Derived from the plan rather than restated as a pattern list. The two had to agree and did
        # not: `--dry-run` printed a file the patterns then dropped, which is the failure mode that
        # lost `citations/` and `LICENSE.txt` in the first place. One list, computed once, so what a
        # dry run promises is exactly what an upload sends (`@publisher-allowlist-derived`) — each
        # name made literal, since the hub reads a pattern and `data/chr[1].parquet` is one (RM353).
        allow_patterns=_literal_patterns(payload_files),
        # The retirement is a `delete_patterns` on the same call, so the new file arriving and the old
        # one leaving are one operation rather than a window in which the repo has neither or both.
        # `retires` is one name, never a glob, and made literal so the hub cannot read it as one (RM353).
        delete_patterns=_literal_patterns(shift.retires for shift in due) or None,
    )
    _check_on_the_hub(api, plan.repo_id, sent, {name: planned[name] for name in payload_files})
    if description is not None:
        _commit(
            api,
            plan.repo_id,
            sent,
            f"Describe the snapshot ({RELEASE_FILENAME})",
            operations=[CommitOperationAdd(path_in_repo=RELEASE_FILENAME, path_or_fileobj=description)],
            landed=f"the payload at {sent[:12]}, which the previous {RELEASE_FILENAME} still describes",
        )
    return plan