The reference-snapshot layout and its provenance record, release.json (RM303).
A reference snapshot is data/*.parquet (or one named payload file), optional parquet sidecar
directories beside data/, LICENSE.txt, and release.json. Six parties have to agree on those
names and on what release.json says: the builder writes it, the publisher uploads it, the
provisioner fetches it, the resolver judges the directory, the reporter prints its label, and a pass
reads its release. Every disagreement so far was silent, so the names and the record live here, in
the tier every party already depends on, and nowhere else.
This module is the contract half: names, a model, a reader, a writer and a payload predicate.
It reads no environment and fetches nothing. Where a snapshot lives on a given machine (the cache
base, the per-lane variables, .env, the pull record) is deployment, and stays in the enricher.
Old spellings are read, never written. Snapshots already published keep their keys. The model
reads rows and licence into the canonical fields, and ClinVar's clinvar_file_date into
dataset, so a republish writes the canonical spelling rather than anyone editing a published file.
Bases: BaseModel
A snapshot's release.json: the keys some reader reads are typed, every other key rides along.
Every typed field is optional, and unstated is None, never a default: a key the file does
not carry is not an answer. Writes go through write_release, which drops the fields the
builder never set and keeps the ones it set to None — a recorded null is a stated unknown.
Values are kept verbatim: source_sha256 is prefixed by some builders and bare in others, and
built_at has two timestamp shapes.
The bytes write_release puts on disk: the set keys only, sorted, newline-terminated.
Source code in schema/src/just_dna_format/snapshot.py
| def to_json(self) -> str:
"""The bytes `write_release` puts on disk: the set keys only, sorted, newline-terminated."""
return json.dumps(self.model_dump(mode="json", exclude_unset=True), indent=2, sort_keys=True) + "\n"
|
SnapshotWrite(directory: Path)
An open write into a snapshot whose record has been withdrawn; commit describes the result.
commit(release: SnapshotRelease) -> Path
Write release as the snapshot's record, after every byte it describes is in place.
Source code in schema/src/just_dna_format/snapshot.py
| def commit(self, release: SnapshotRelease) -> Path:
"""Write `release` as the snapshot's record, after every byte it describes is in place."""
return write_release(self.directory, release)
|
BlockWrite(directory: Path, name: str, readable: bool)
An open write into one block of a snapshot's record; commit merges the new block in.
commit(block: dict[str, Any]) -> bool
Merge block under name into the record; False when the record could not be read.
Source code in schema/src/just_dna_format/snapshot.py
| def commit(self, block: dict[str, Any]) -> bool:
"""Merge `block` under `name` into the record; `False` when the record could not be read."""
if not self.readable or release_is_unreadable(self.directory):
logger.warning(
"the snapshot's release.json in %s is present but unreadable; leaving it alone rather "
"than overwriting its provenance. The %s block was not recorded.",
self.directory,
self.name,
)
return False
existing = read_release(self.directory)
stated = existing.model_dump(exclude_unset=True) if existing is not None else {}
write_release(self.directory, SnapshotRelease.model_validate({**stated, self.name: block}))
return True
|
holds_snapshot_payload(
directory: Path, *, payload_filename: str | None = None
) -> bool
Whether directory holds a snapshot's payload — judged by the payload, never by is_dir().
With payload_filename, the payload is that one file at the root. Without it, the payload is at
least one parquet under data/, or bare at the root (the shape a single-file snapshot was
handed over in before data/ existed). A directory holding only release.json, or nothing, is
not a snapshot: an interrupted build leaves exactly that behind.
Says nothing about whether a pull finished; that is the provisioner's record of one machine's
copy, and it lives with the provisioner.
Source code in schema/src/just_dna_format/snapshot.py
| def holds_snapshot_payload(directory: Path, *, payload_filename: str | None = None) -> bool:
"""Whether `directory` holds a snapshot's payload — judged by the payload, never by `is_dir()`.
With `payload_filename`, the payload is that one file at the root. Without it, the payload is at
least one parquet under `data/`, or bare at the root (the shape a single-file snapshot was
handed over in before `data/` existed). A directory holding only `release.json`, or nothing, is
not a snapshot: an interrupted build leaves exactly that behind.
Says nothing about whether a pull finished; that is the provisioner's record of one machine's
copy, and it lives with the provisioner.
"""
directory = Path(directory)
if not directory.is_dir():
return False
if payload_filename is not None:
return (directory / payload_filename).is_file()
data_dir = directory / SNAPSHOT_DATA_DIRNAME
return (data_dir.is_dir() and any(data_dir.glob("*.parquet"))) or any(directory.glob("*.parquet"))
|
clinvar_release_label(
file_date: str | None, source_sha256: str | None
) -> str | None
ClinVar's release label: clinvar_<##fileDate>, else clinvar_sha256:<digest>, else None.
One formula, called by the builder that writes dataset and by the reader of a ClinVar snapshot
built before the builder wrote one. A VCF whose header states no file date still has the digest
of the bytes, which names the release exactly. With neither, the label is unknown and withheld.
Source code in schema/src/just_dna_format/snapshot.py
| def clinvar_release_label(file_date: str | None, source_sha256: str | None) -> str | None:
"""ClinVar's release label: `clinvar_<##fileDate>`, else `clinvar_sha256:<digest>`, else `None`.
One formula, called by the builder that writes `dataset` and by the reader of a ClinVar snapshot
built before the builder wrote one. A VCF whose header states no file date still has the digest
of the bytes, which names the release exactly. With neither, the label is unknown and withheld.
"""
date = str(file_date or "").strip()
if date:
return f"{CLINVAR_DATASET_PREFIX}{date}"
digest = str(source_sha256 or "").strip().removeprefix("sha256:")
return f"{CLINVAR_DATASET_PREFIX}sha256:{digest}" if digest else None
|
read_release(reference: Path) -> SnapshotRelease | None
A snapshot directory's release.json as a SnapshotRelease, or None when absent or unreadable.
None is the answer for both, as it was when this returned a dict: a caller must not be able to
mistake "this snapshot does not say" for a release, so callers branch on None. Unreadable is a
file that does not parse, or one that is not a JSON object.
A typed key holding the wrong kind of value costs that key, never the record. It is read as
unstated and the warning names it; every other key, the label included, is still read. The
snapshot's own provenance is worth more than one malformed count. release_is_unreadable says
whether a record was read whole, for the writer that must not replace a file it could not read.
Source code in schema/src/just_dna_format/snapshot.py
| def read_release(reference: Path) -> SnapshotRelease | None:
"""A snapshot directory's `release.json` as a `SnapshotRelease`, or `None` when absent or unreadable.
`None` is the answer for both, as it was when this returned a dict: a caller must not be able to
mistake "this snapshot does not say" for a release, so callers branch on `None`. Unreadable is a
file that does not parse, or one that is not a JSON object.
**A typed key holding the wrong kind of value costs that key, never the record.** It is read as
unstated and the warning names it; every other key, the label included, is still read. The
snapshot's own provenance is worth more than one malformed count. `release_is_unreadable` says
whether a record was read whole, for the writer that must not replace a file it could not read.
"""
path = Path(reference) / RELEASE_FILENAME
if not path.is_file():
return None
payload, problem = _load(path)
if payload is None:
logger.warning("Could not read %s (%s); treating the release as unstated.", path, problem)
return None
canonical = _canonical(payload)
try:
return SnapshotRelease.model_validate(canonical)
except ValidationError as exc:
bad = sorted({str(error["loc"][0]) for error in exc.errors() if error["loc"]})
logger.warning("%s: %s do not fit the snapshot record; read as unstated.", path, ", ".join(bad))
try:
return SnapshotRelease.model_validate({k: v for k, v in canonical.items() if k not in bad})
except ValidationError:
return None
|
release_is_unreadable(reference: Path) -> bool
Whether reference holds a release.json that cannot be read whole.
Present and not parseable, not an object, or with a typed key read_release had to drop. Told
apart from absence for the caller that has to act on it: a provenance failure is not a data
failure, so the snapshot stays usable and says so; and a writer merging into the file must leave
it alone rather than replace the only statement of where the data came from. Logs nothing:
read_release is where the warning is said.
Source code in schema/src/just_dna_format/snapshot.py
| def release_is_unreadable(reference: Path) -> bool:
"""Whether `reference` holds a `release.json` that cannot be read whole.
Present and not parseable, not an object, or with a typed key `read_release` had to drop. Told
apart from absence for the caller that has to act on it: a provenance failure is not a data
failure, so the snapshot stays usable and says so; and a writer merging into the file must leave
it alone rather than replace the only statement of where the data came from. Logs nothing:
`read_release` is where the warning is said.
"""
path = Path(reference) / RELEASE_FILENAME
if not path.is_file():
return False
payload, _ = _load(path)
if payload is None:
return True
try:
SnapshotRelease.model_validate(payload)
except ValidationError:
return True
return False
|
release_label(reference: Path) -> str | None
The release a snapshot directory states, its dataset, or None when it states none.
The one reader of the label, for cache status and for every pass that stamps it onto
SourceRow.dataset, so a reader of the cache and a reader of a module see the same string. No
fallback and no guess: a blank, absent or unreadable label is an unknown, and an unknown is
withheld rather than written as a label something could match.
Source code in schema/src/just_dna_format/snapshot.py
| def release_label(reference: Path) -> str | None:
"""The release a snapshot directory states, its `dataset`, or `None` when it states none.
The one reader of the label, for `cache status` and for every pass that stamps it onto
`SourceRow.dataset`, so a reader of the cache and a reader of a module see the same string. No
fallback and no guess: a blank, absent or unreadable label is an unknown, and an unknown is
withheld rather than written as a label something could match.
"""
release = read_release(reference)
label = (release.dataset or "").strip() if release is not None else ""
return label or None
|
write_release(
directory: Path, release: SnapshotRelease
) -> Path
Write release as directory/release.json, atomically, and return the path.
Every snapshot carries the canonical spellings because every record is written here. Atomic
because a release.json truncated mid-write parses as valid JSON that is simply missing keys, and
a reader would believe it. A builder does not call this: it opens a write with
begin_snapshot_write, whose commit calls it after the old record was withdrawn (RM369).
Source code in schema/src/just_dna_format/snapshot.py
| def write_release(directory: Path, release: SnapshotRelease) -> Path:
"""Write `release` as `directory/release.json`, atomically, and return the path.
Every snapshot carries the canonical spellings because every record is written here. Atomic
because a `release.json` truncated mid-write parses as valid JSON that is simply missing keys, and
a reader would believe it. **A builder does not call this**: it opens a write with
`begin_snapshot_write`, whose `commit` calls it after the old record was withdrawn (RM369).
"""
return atomic_write_text(Path(directory) / RELEASE_FILENAME, release.to_json())
|
begin_snapshot_write(directory: Path) -> SnapshotWrite
Withdraw directory's release.json before its payload changes, and return the write handle.
The order is the contract: withdraw, move the bytes, describe them. A builder that swapped its
payload in first and wrote the record after left, on a crash between the two (a full disk, a
killed process), new bytes under the previous build's record, and every reader believed it. With
the record withdrawn first, the worst a crash leaves is a snapshot that states no release, which
every reader already treats as an unknown and withholds.
Call it immediately before the first byte of the live payload moves, never at the top of a
build: a download or a parse that fails before then leaves the old snapshot whole, and it should
stay described. And a build that fails after it is not given its old record back, because a
restored record would describe bytes the failed build may have changed.
The handle's commit is the only way a builder writes release.json, so describing a snapshot
without having withdrawn the old description is not expressible.
Source code in schema/src/just_dna_format/snapshot.py
| def begin_snapshot_write(directory: Path) -> SnapshotWrite:
"""Withdraw `directory`'s `release.json` before its payload changes, and return the write handle.
The order is the contract: **withdraw, move the bytes, describe them.** A builder that swapped its
payload in first and wrote the record after left, on a crash between the two (a full disk, a
killed process), new bytes under the previous build's record, and every reader believed it. With
the record withdrawn first, the worst a crash leaves is a snapshot that states no release, which
every reader already treats as an unknown and withholds.
**Call it immediately before the first byte of the live payload moves**, never at the top of a
build: a download or a parse that fails before then leaves the old snapshot whole, and it should
stay described. And a build that fails after it is **not** given its old record back, because a
restored record would describe bytes the failed build may have changed.
The handle's `commit` is the only way a builder writes `release.json`, so describing a snapshot
without having withdrawn the old description is not expressible.
"""
directory = Path(directory)
directory.mkdir(parents=True, exist_ok=True)
_withdraw(directory)
return SnapshotWrite(directory)
|
begin_block_write(directory: Path, name: str) -> BlockWrite
Withdraw block name from directory's record before the bytes it describes change.
For a second half that writes a sidecar beside a snapshot another build owns (ClinVar's
citations): the half owns one block of the record, never the record, so it withdraws that block
and leaves the rest. Read-modify-write rather than rewrite, and an unreadable record is left
alone, because overwriting it would destroy the only statement of where the data came from.
Source code in schema/src/just_dna_format/snapshot.py
| def begin_block_write(directory: Path, name: str) -> BlockWrite:
"""Withdraw block `name` from `directory`'s record before the bytes it describes change.
For a second half that writes a sidecar beside a snapshot another build owns (ClinVar's
citations): the half owns one block of the record, never the record, so it withdraws that block
and leaves the rest. Read-modify-write rather than rewrite, and an unreadable record is left
alone, because overwriting it would destroy the only statement of where the data came from.
"""
directory = Path(directory)
if release_is_unreadable(directory):
return BlockWrite(directory, name, readable=False)
existing = read_release(directory)
if existing is not None and name in (existing.model_extra or {}):
stated = existing.model_dump(exclude_unset=True)
stated.pop(name)
write_release(directory, SnapshotRelease.model_validate(stated))
return BlockWrite(directory, name, readable=True)
|