just_dna_enricher.upload¶
just_dna_enricher.upload ¶
HuggingFace upload — publisher surface of the network tier.
Two publish shapes share one create-or-update pathway (ensure_repo):
- module — a compiled module's
manifest.jsonand every file it declares (the attested parquets, the logo, the readme,provenance.jsonand each log; RM347, RM353) todatasets/<repo>/data/<name>/, matching the layout just-dna-lite's discovery scans, and todata/<name>/v<version>/when the manifest states a version, so the discovery path can address a particular release rather than only "whatever is there now" (upload_module; RM84). - reference snapshot — a built ClinVar (or Ensembl) parquet snapshot +
release.jsonto the root of a dataset repo, matching thedownload.ensure_*_snapshotlayout (publish_reference_snapshot).
Both require a HuggingFace token with write access (hf auth login or HF_TOKEN). This is the
dev/publisher half of the enricher's HuggingFace use: snapshot download is a runtime enrich
path (download.ensure_snapshot); upload is for authors republishing modules (e.g. the Gen-I
v1-port recreation) or publishing a rebuilt reference. Install as just-dna-enricher[dev].
Extracted from just_dna_pipelines.v1_port.publish (just-dna-lite); create_repo was added here
(the origin assumed the repo pre-existed) so a brand-new dataset repo can be created on first push.
LayoutShift
dataclass
¶
A retirement declared by the commit that causes it, so the publish carries out the migration.
The rule (maintainer, 2026-09-03). A change that retires one published file and introduces another must also carry a conditioned update to the publication procedure: if the new file is absent from the repo and the old one is present, upload the new and delete the old. The condition is a predicate over the remote, so it fires exactly once per repo and is a no-op forever after — nothing is deleted on a repo that has already moved, and re-running a publish cannot re-delete.
This is the only deletion a publish performs. General cleanup is cache prune, which prints
what it would remove and asks; a deletion nobody declared is not a side effect a publish may have.
What makes this one safe is not that it is small but that it is named: the pair is in the commit
that changed the layout, where a reviewer sees both halves at once.
Deleting is recoverable in a way this repository's @snapshot-accumulates note did not credit —
a HuggingFace dataset repo is git-backed, and a superseded revision still resolves (three of them
were read off the hub while auditing this on 2026-09-03). The reason to declare rather than sweep
is not that bytes are lost; it is that a retired file goes on answering 200 to whoever still asks
for it, which is how a lane's default archive stayed frozen for a year (CLINPGX_ARCHIVES).
PublishRefused ¶
Bases: Exception
A publish refused on what it would send — the publisher's one error family (RM368).
Every refusal this module raises is a subclass, so a caller catches the family once and tells the
members apart by type: the local module or snapshot is not publishable (ModuleNotPublishable,
SnapshotNotPublishable), a planned file changed before it was sent (PlanChangedError), the
remote already holds something else (PublishCollisionError, ToolchainDiffersError,
OrphanedSidecarError), or the remote moved after the plan read it (RemoteMovedError). Two
things stay outside it because they say nothing about what would be sent: a missing write token
(PermissionError), and whatever the hub client itself raises.
A base class, so a caller's except order is load-bearing (@client-exception-contract): a
handler for a member goes before the handler for the family, as the CLI's do.
ModuleNotPublishable ¶
Bases: PublishRefused, FileNotFoundError
The module directory is not publishable as it stands (plan_upload's refusals, RM368).
It used to be a bare FileNotFoundError, which a caller could not tell from a missing directory.
It is still one within 0.8 — a second base kept so an existing except FileNotFoundError
keeps catching it; catch PublishRefused or this type instead.
SnapshotNotPublishable ¶
Bases: PublishRefused, FileNotFoundError
The snapshot directory has nothing built to publish (plan_reference_snapshot, RM368).
Still a FileNotFoundError within 0.8, for the reason ModuleNotPublishable is.
PlanChangedError ¶
Bases: PublishRefused
A planned file's bytes are not the ones the plan read and checked (RM368, RM386).
The module send reads each planned file once more and sends exactly the bytes it hashed; a file
whose hash is not the plan's (a recompile landing between the check and the send, an edit) or that
cannot be read any more refuses the publish before anything is committed. A snapshot's payload is
too large to hold, so the hub reads it from the directory: what the hub then holds is checked
against the plan's hashes, and a difference withholds the description (release.json), so nothing
ever describes bytes nothing checked.
RemoteMovedError ¶
Bases: PublishRefused
The repo is not at the revision the plan read, so the write would act on a state nobody checked (RM386).
Every publish and prune reads the repo at one revision (_head) and commits on top of exactly that
revision (parent_commit), and the hub refuses a commit whose parent is no longer the head. A
concurrent writer between the read and the write — another publish, a prune — therefore refuses
this one instead of being overwritten unseen. A prune plan that states no revision read none and
refuses too. Nothing was committed by the write that refused; re-run to read the repo again.
OrphanedSidecarError ¶
Bases: PublishRefused, RuntimeError
The publish would overwrite a release.json describing sidecars it is not carrying (RM185).
Its own type for the reason PublishCollisionError is one: the CLI has to tell it from the
refusals plan_* raises. Those say the local snapshot is not publishable; this one says the local
snapshot is fine and the remote holds bytes this publish would silently stop describing.
The incident. A ClinVar snapshot is two halves on two cadences — the per-chromosome parquet and
citations/citations.parquet — and release.json describes both because build_citations merges
a block into it. A rebuild that built only the VCF half published a citations-free release.json
over a repo whose sidecar it had not replaced, and the publisher adds without deleting: the
sidecar outlived its own description, and the published artifact carried records from one ClinVar
release beside citations from another while saying nothing. RM179 fixed the lane; this is the
general shape, refused at the boundary where any lane could repeat it.
PublishCollisionError ¶
Bases: PublishRefused, RuntimeError
The versioned path already holds a different release, and no --force was given (RM88).
Its own type rather than a ModuleNotPublishable, because the CLI has to tell it apart from the
refusals plan_upload raises: those say the local module is not publishable, this one says the
local module is fine and the remote already has that version.
reason is None on the plain collision — nothing recorded says why the bytes differ — and a
subclass sets it when something does (ToolchainDiffersError).
ToolchainDiffersError ¶
ToolchainDiffersError(
message: str,
*,
published_toolchain: dict[str, str],
local_toolchain: dict[str, str],
content_unchanged: bool | None,
)
Bases: PublishCollisionError
The plain collision's one explained case: the two manifests recorded different toolchains (RM342).
From polars 2 the parquet footer names the polars release that wrote it, so an unchanged spec
recompiled beside another polars mints another digest. That is not corruption, and the maintainer's
decision (2026-11-04) is that it is never reported as the plain mismatch. The publish still
refuses unless --force: the versioned path's bytes stay the ones that version was cut as
(Principle 4), and the reason changes what the refusal says, not whether it refuses.
A subclass, so every handler of PublishCollisionError still refuses. A sibling type would
turn a handled refusal into an uncaught one in any caller that catches only the collision; the
cost is that a caller wanting the distinction must catch this type first
(@client-exception-contract), which the CLI does. Tell the two apart by type or by reason,
never by the message.
The reason names a co-occurring difference, not a proven cause. A content edit recompiled under
another polars raises this too, which is why content_unchanged rides beside it: True when both
manifests carry one content_signature (the toolchain alone can account for the digest), False
when the content moved as well, None when either side states none.
Source code in enricher/src/just_dna_enricher/upload.py
UploadPlan ¶
Bases: BaseModel
What an upload would send (also the dry-run result).
SnapshotPlan ¶
Bases: BaseModel
What a reference-snapshot publish would send (also the dry-run result).
PruneCandidate ¶
Bases: BaseModel
One remote file cache prune would remove, and the reason it is nameable as removable.
PrunePlan ¶
Bases: BaseModel
What a prune would delete from one repo. The dry run and the deletion read the same object.
ensure_repo ¶
Create-or-update: ensure the dataset repo exists, returning the authenticated HfApi.
create_repo(..., exist_ok=True) is a no-op when the repo already exists, so create and update
are one pathway. The returned api is reused by the caller's upload_folder so only one
HfApi is constructed per publish.
Source code in enricher/src/just_dna_enricher/upload.py
layout_shifts_to_apply ¶
The declared shifts whose condition the remote currently satisfies. Reads; decides nothing else.
Separate from the publish so it can be asserted directly: the whole safety of this mechanism is that the predicate is false the moment it has run once, and that is a property of this function rather than of the caller that acts on it.
Source code in enricher/src/just_dna_enricher/upload.py
plan_upload ¶
Resolve the upload plan and validate the compiled artifacts are present.
Two destinations, both planned here (RM84): the flat data/<name>/, which is what the reference
consumer's discovery scans and therefore has to keep working and keep meaning latest, and
data/<name>/v<version>/ inside it — nested rather than a sibling, because the flat path is
the one deployed readers are pointed at — which is the only thing on that path that can name a
particular release. A module with no readable version gets the flat path alone, never a vNone
directory, and version_unknown_reason says which of the four reasons applies. The reason is a
field rather than a log line for the reason every other withheld answer here is
(clin_sig_not_checked, skipped_offline): a caller that has to parse prose to learn what
happened has not been told.
The files are what the manifest declares (RM353, @publisher-allowlist-derived). When
manifest.artifact.files is legible, the plan is manifest.json plus every file the manifest
names in the module directory — the attested parquets, the logo, the readme, provenance.json and
each log — and nothing else: a parquet an earlier compile left behind, or a logo the manifest does
not name, would arrive unattested, and a stale parquet this publish put on the flat path would read
to discovery as part of the module. (One already on the remote stays; a publish deletes only what
it declares.) Each declared file sent must still hash to the sha256 the manifest states, or the
plan refuses: the manifest is the guard's comparator, so it has to be true of the bytes. A
declared optional file that leaves the module directory is skipped with a warning
naming the reason; one absent from disk is skipped quietly, since every optional file is optional
(verify skips an absent log or logo the same way). When the manifest cannot say — absent,
unparseable, or with no artifact.files list — the plan falls back to _ALLOW_PATTERNS found on
disk: an unknown withholds, and a directory with no manifest has always been publishable.
What it refuses, in order, most specific first so a refusal names the actual fault — each a
ModuleNotPublishable (RM368):
- A declared field in a shape the compiler never writes (
artifact.filesthat is not a list, asha256that is not a string, …), quoting each. Absent is unknown and withholds; present and unreadable used to read as absent too and dropped the gate that field feeds. - An attested entry naming no file — anything but an object with a non-blank string
name. - Every attested artifact file is sent.
artifact.digestis a Merkle root over exactly the filesmanifest.artifact.filesnames, so an attested file missing from disk, or one whose name leaves the module directory, would make the published manifest a false claim about bytes that are not there. - A declared file whose bytes are not the ones declared (the sha256 differs).
weights.parquetnever travels alone. A SNP core compiles to all three, so a missingannotations/studiesbeside it is an interrupted compile.- At least one lead table. A directory with a manifest, a readme and no annotation rows is not a module; publishing one is the silent failure a bare widening of the old rule would have produced, and it is what discovery would then be unable to see.
The plan is one reading of the disk (RM368). Every planned file is hashed once, here, into
UploadPlan.sha256 — manifest.json from the very bytes it was parsed from — and the upload sends
exactly those bytes or refuses (_verified_payload).
Source code in enricher/src/just_dna_enricher/upload.py
rehearse_upload ¶
rehearse_upload(
module_dir: Path,
name: str,
repo_id: str | None = None,
token: str | None = None,
force: bool = False,
) -> UploadPlan
What upload_module would do, short of writing: the plan, and the versioned-path check (RM368).
The dry run. It runs every check the real run runs — the plan's refusals and, when the module
states a version and force is not set, the collision check against the published manifest, read
with a read-only client — and raises what the real run would raise. It creates no repo and commits
nothing. A hub it cannot reach is the real run's unknown: a warning, and the plan.
plan_upload alone stays the offline half, for a caller that wants the plan with no remote read.
Source code in enricher/src/just_dna_enricher/upload.py
upload_module ¶
upload_module(
module_dir: Path,
name: str,
repo_id: str | None = None,
token: str | None = None,
commit_message: str | None = None,
force: bool = False,
) -> UploadPlan
Upload the compiled module to a HuggingFace dataset collection.
Ensures the dataset repo exists (create-or-update), then writes the same files to the flat path and — when the manifest states a version — to the versioned subdirectory under it.
What is sent is the plan's bytes (RM368). The plan hashed every file it names; after the
collision check, each is read once more, checked against that hash, and sent as those bytes through
create_commit — never re-read from the directory by the hub. A file that changed in between (a
recompile still writing, an edit) is a PlanChangedError before either commit.
And the revision it read (RM386). The repo is read once at its head, the versioned-path check
reads at that revision, the flat commit is pinned on it and the versioned commit on the flat one
(_commit), so a version another publisher lands after the check is a RemoteMovedError rather
than an overwrite nobody saw.
The two writes are two commits, not one. Each create_commit is its own commit, so a reader can
briefly see the flat path refreshed while the versioned copy is not there yet; the flat path goes
first because it is the one anything reads today. One commit carrying both paths' operations is
now a one-line change and was not taken here: two commits is the documented behaviour, and a
re-run, which is idempotent, is the documented recovery when the second fails.
The versioned path refuses to be overwritten with different bytes (RM88, RM353, RM368). Before
either write, the published manifest.json at data/<name>/v<version>/ is read and compared with
this module's: its artifact.digest, the sha256 of every file it declares, and every other field
(_versioned_conflict). A difference is a PublishCollisionError unless force=True. The flat
path is deliberately not guarded: it means latest, so overwriting it is what it is for, and
the whole point of the versioned copy is that it does not.
The policy is refuse-unless---force rather than warn-and-proceed or refuse-outright, decided
2026-08-18. The flag's existence is itself the claim that overwriting is sometimes right — a
curator re-cutting a draft release is a real workflow, and a gate with no override becomes a gate
people route around.
Recompiling under a newer compiler also moves the digest, and will trip this. That is correct
rather than a false positive: P4 pins byte-reproducibility to compiler_version, the recorded
toolchain and the resolved reference, so the versioned path really would come to hold different
bytes than the ones it was published with. The refusal says so, because "I changed nothing" is the
first thing its first user will think.
When the artifacts differ and both manifests recorded a toolchain and the two differ, the refusal
is a ToolchainDiffersError (RM342), which names both toolchains. Either side unrecorded (a
manifest compiled before 0.8), the same toolchain on both, or a difference in declared files or
other fields alone, keeps the plain PublishCollisionError.
Raises PermissionError if no token is available.
Source code in enricher/src/just_dna_enricher/upload.py
1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 | |
plan_reference_snapshot ¶
plan_reference_snapshot(
snapshot_dir: Path,
repo_id: str | None = None,
*,
payload: str | None = None,
) -> SnapshotPlan
Resolve a snapshot publish plan and validate the built artifacts are present.
payload names a snapshot whose content is one file at the root rather than
data/*.parquet — STRchive's STRchive-loci.json is the case, and ACMG's acmg_sf.csv is the
shape's second member. The parameter is the caller's, deliberately: a lane knows the name of the
file it builds, and putting a roster of lane filenames in the publisher would make this function
the fourth place that has to learn about a new snapshot kind.
A payload snapshot is refused when the file is missing, exactly as a parquet one is when
data/ is empty — the refusal is the same claim either way, that there is nothing built here to
publish.
Source code in enricher/src/just_dna_enricher/upload.py
check_publish_orphans_no_sidecar ¶
check_publish_orphans_no_sidecar(
plan: SnapshotPlan,
api=None,
token: str | None = None,
*,
revision: str | None = None,
) -> None
Refuse a publish that would leave a remote sidecar undescribed. Reads the repo; writes nothing.
The remote tree, not the remote release.json. The block is a description of the bytes and
the bytes are what a puller gets, so the description is the half that can already be missing —
which is exactly the state just-dna-seq/clinvar was found in on 2026-09-03: the citations parquet
present, the block gone. A guard reading the block would have passed the second bad publish as
happily as the first.
Scoped to publishes that carry release.json, because a publish that carries none overwrites no
provenance. A repo that does not exist yet lists nothing and passes — that is a first publish, not
an orphan.
api is the caller's when it has one (a publish has just built an authenticated client and there
is no reason to build a second); a dry run passes none and gets an anonymous reader, since listing
a public repo needs no token and a dry run must not require write access to say what a publish
would do.
revision is the commit a publish read and will commit on (RM386); a dry run passes none and reads
the head.
Source code in enricher/src/just_dna_enricher/upload.py
rehearse_reference_snapshot ¶
rehearse_reference_snapshot(
snapshot_dir: Path,
repo_id: str | None = None,
*,
payload: str | None = None,
token: str | None = None,
) -> SnapshotPlan
What publish_reference_snapshot would do, short of writing: the plan and the orphan check.
The dry run of every snapshot publish command (RM368). A rehearsal that skips the check the real
publish refuses on is a different operation — two commands ran the orphan check in their dry run
and the rest did not, so their --dry-run promised a publish the real run could refuse. Reads the
published repo's file list; uploads nothing.
Source code in enricher/src/just_dna_enricher/upload.py
plan_prune ¶
What the published repo carries that this lane's snapshot is not made of. Reads; deletes nothing.
Two sources of a name, and neither is a guess. A file under data/ that the lane's own glob
excludes is not part of the snapshot by the same definition provisioning uses — _provision_snapshot
already refuses to download it and warns when it finds one locally. And a file a LayoutShift
declares retired is nameable even where the glob cannot see it: for the four lanes whose glob is
*.parquet the exclusion set is empty by construction, so declaration is the only way a retired
file there is ever identified.
Everything else is left alone, including files this tier never wrote — README.md,
.gitattributes, release.json, LICENSE.txt, and any sidecar directory. A pruner that removed
what it did not recognise would be the sweep this design exists to avoid.
Source code in enricher/src/just_dna_enricher/upload.py
prune_repo ¶
Delete a prune plan's files. Called only after the caller has shown the plan and been told yes.
Returns the number of paths deleted. Deleting is a commit on a git-backed repo, so a superseded revision still resolves — but that is a reason to be able to undo a mistake, never a reason to remove something nobody looked at, which is why this takes a plan rather than a repo id.
Exactly the listed paths, at the listed revision (RM386). One delete operation per candidate —
a path, never a pattern the hub matches (RM353 escaped the patterns; there are none now) — in one
commit whose parent is the revision the listing was read at. A repo that changed after the plan
was shown (a file republished under a listed name) is a RemoteMovedError and nothing is deleted;
so is a plan that states no revision, since it read nothing a deletion could be held to.
Source code in enricher/src/just_dna_enricher/upload.py
publish_reference_snapshot ¶
publish_reference_snapshot(
snapshot_dir: Path,
repo_id: str | None = None,
token: str | None = None,
commit_message: str | None = None,
*,
payload: str | None = None,
) -> SnapshotPlan
Create-or-update a dataset repo and upload a built reference snapshot to its root.
Uploads data/*.parquet + any parquet sidecars (ClinVar's citations/) + release.json, so
the tree matches download.ensure_*_snapshot and a provisioned snapshot is the same artifact a
built one is — PMIDs included, which is what a drafted gene panel needs to compile.
Tied to the bytes it read and the revision it read (RM386). The repo is read at one revision
and both commits are pinned on it (_commit), so a concurrent writer refuses the publish
(RemoteMovedError) rather than being overwritten. Every planned file is hashed once; the payload
goes up through the hub's own uploader, and what the hub then holds is checked against those
hashes before the description is sent (PlanChangedError if not); the description is sent as the
bytes read with the rest, never read again. The returned plan states the revision and the hashes.
Raises PermissionError if no token is available.
Source code in enricher/src/just_dna_enricher/upload.py
1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 | |