Skip to content

just_dna_format.snapshot

just_dna_format.snapshot

The reference-snapshot layout and its provenance record, release.json (RM303).

A reference snapshot is data/*.parquet (or one named payload file), optional parquet sidecar directories beside data/, LICENSE.txt, and release.json. Six parties have to agree on those names and on what release.json says: the builder writes it, the publisher uploads it, the provisioner fetches it, the resolver judges the directory, the reporter prints its label, and a pass reads its release. Every disagreement so far was silent, so the names and the record live here, in the tier every party already depends on, and nowhere else.

This module is the contract half: names, a model, a reader, a writer and a payload predicate. It reads no environment and fetches nothing. Where a snapshot lives on a given machine (the cache base, the per-lane variables, .env, the pull record) is deployment, and stays in the enricher.

Old spellings are read, never written. Snapshots already published keep their keys. The model reads rows and licence into the canonical fields, and ClinVar's clinvar_file_date into dataset, so a republish writes the canonical spelling rather than anyone editing a published file.

SnapshotRelease

Bases: BaseModel

A snapshot's release.json: the keys some reader reads are typed, every other key rides along.

Every typed field is optional, and unstated is None, never a default: a key the file does not carry is not an answer. Writes go through write_release, which drops the fields the builder never set and keeps the ones it set to None — a recorded null is a stated unknown. Values are kept verbatim: source_sha256 is prefixed by some builders and bare in others, and built_at has two timestamp shapes.

to_json

to_json() -> str

The bytes write_release puts on disk: the set keys only, sorted, newline-terminated.

Source code in schema/src/just_dna_format/snapshot.py
def to_json(self) -> str:
    """The bytes `write_release` puts on disk: the set keys only, sorted, newline-terminated."""
    return json.dumps(self.model_dump(mode="json", exclude_unset=True), indent=2, sort_keys=True) + "\n"

SnapshotWrite dataclass

SnapshotWrite(directory: Path)

An open write into a snapshot whose record has been withdrawn; commit describes the result.

commit

commit(release: SnapshotRelease) -> Path

Write release as the snapshot's record, after every byte it describes is in place.

Source code in schema/src/just_dna_format/snapshot.py
def commit(self, release: SnapshotRelease) -> Path:
    """Write `release` as the snapshot's record, after every byte it describes is in place."""
    return write_release(self.directory, release)

BlockWrite dataclass

BlockWrite(directory: Path, name: str, readable: bool)

An open write into one block of a snapshot's record; commit merges the new block in.

commit

commit(block: dict[str, Any]) -> bool

Merge block under name into the record; False when the record could not be read.

Source code in schema/src/just_dna_format/snapshot.py
def commit(self, block: dict[str, Any]) -> bool:
    """Merge `block` under `name` into the record; `False` when the record could not be read."""
    if not self.readable or release_is_unreadable(self.directory):
        logger.warning(
            "the snapshot's release.json in %s is present but unreadable; leaving it alone rather "
            "than overwriting its provenance. The %s block was not recorded.",
            self.directory,
            self.name,
        )
        return False
    existing = read_release(self.directory)
    stated = existing.model_dump(exclude_unset=True) if existing is not None else {}
    write_release(self.directory, SnapshotRelease.model_validate({**stated, self.name: block}))
    return True

holds_snapshot_payload

holds_snapshot_payload(
    directory: Path, *, payload_filename: str | None = None
) -> bool

Whether directory holds a snapshot's payload — judged by the payload, never by is_dir().

With payload_filename, the payload is that one file at the root. Without it, the payload is at least one parquet under data/, or bare at the root (the shape a single-file snapshot was handed over in before data/ existed). A directory holding only release.json, or nothing, is not a snapshot: an interrupted build leaves exactly that behind.

Says nothing about whether a pull finished; that is the provisioner's record of one machine's copy, and it lives with the provisioner.

Source code in schema/src/just_dna_format/snapshot.py
def holds_snapshot_payload(directory: Path, *, payload_filename: str | None = None) -> bool:
    """Whether `directory` holds a snapshot's payload — judged by the payload, never by `is_dir()`.

    With `payload_filename`, the payload is that one file at the root. Without it, the payload is at
    least one parquet under `data/`, or bare at the root (the shape a single-file snapshot was
    handed over in before `data/` existed). A directory holding only `release.json`, or nothing, is
    not a snapshot: an interrupted build leaves exactly that behind.

    Says nothing about whether a pull finished; that is the provisioner's record of one machine's
    copy, and it lives with the provisioner.
    """
    directory = Path(directory)
    if not directory.is_dir():
        return False
    if payload_filename is not None:
        return (directory / payload_filename).is_file()
    data_dir = directory / SNAPSHOT_DATA_DIRNAME
    return (data_dir.is_dir() and any(data_dir.glob("*.parquet"))) or any(directory.glob("*.parquet"))

clinvar_release_label

clinvar_release_label(
    file_date: str | None, source_sha256: str | None
) -> str | None

ClinVar's release label: clinvar_<##fileDate>, else clinvar_sha256:<digest>, else None.

One formula, called by the builder that writes dataset and by the reader of a ClinVar snapshot built before the builder wrote one. A VCF whose header states no file date still has the digest of the bytes, which names the release exactly. With neither, the label is unknown and withheld.

Source code in schema/src/just_dna_format/snapshot.py
def clinvar_release_label(file_date: str | None, source_sha256: str | None) -> str | None:
    """ClinVar's release label: `clinvar_<##fileDate>`, else `clinvar_sha256:<digest>`, else `None`.

    One formula, called by the builder that writes `dataset` and by the reader of a ClinVar snapshot
    built before the builder wrote one. A VCF whose header states no file date still has the digest
    of the bytes, which names the release exactly. With neither, the label is unknown and withheld.
    """
    date = str(file_date or "").strip()
    if date:
        return f"{CLINVAR_DATASET_PREFIX}{date}"
    digest = str(source_sha256 or "").strip().removeprefix("sha256:")
    return f"{CLINVAR_DATASET_PREFIX}sha256:{digest}" if digest else None

read_release

read_release(reference: Path) -> SnapshotRelease | None

A snapshot directory's release.json as a SnapshotRelease, or None when absent or unreadable.

None is the answer for both, as it was when this returned a dict: a caller must not be able to mistake "this snapshot does not say" for a release, so callers branch on None. Unreadable is a file that does not parse, or one that is not a JSON object.

A typed key holding the wrong kind of value costs that key, never the record. It is read as unstated and the warning names it; every other key, the label included, is still read. The snapshot's own provenance is worth more than one malformed count. release_is_unreadable says whether a record was read whole, for the writer that must not replace a file it could not read.

Source code in schema/src/just_dna_format/snapshot.py
def read_release(reference: Path) -> SnapshotRelease | None:
    """A snapshot directory's `release.json` as a `SnapshotRelease`, or `None` when absent or unreadable.

    `None` is the answer for both, as it was when this returned a dict: a caller must not be able to
    mistake "this snapshot does not say" for a release, so callers branch on `None`. Unreadable is a
    file that does not parse, or one that is not a JSON object.

    **A typed key holding the wrong kind of value costs that key, never the record.** It is read as
    unstated and the warning names it; every other key, the label included, is still read. The
    snapshot's own provenance is worth more than one malformed count. `release_is_unreadable` says
    whether a record was read whole, for the writer that must not replace a file it could not read.
    """
    path = Path(reference) / RELEASE_FILENAME
    if not path.is_file():
        return None
    payload, problem = _load(path)
    if payload is None:
        logger.warning("Could not read %s (%s); treating the release as unstated.", path, problem)
        return None
    canonical = _canonical(payload)
    try:
        return SnapshotRelease.model_validate(canonical)
    except ValidationError as exc:
        bad = sorted({str(error["loc"][0]) for error in exc.errors() if error["loc"]})
        logger.warning("%s: %s do not fit the snapshot record; read as unstated.", path, ", ".join(bad))
    try:
        return SnapshotRelease.model_validate({k: v for k, v in canonical.items() if k not in bad})
    except ValidationError:
        return None

release_is_unreadable

release_is_unreadable(reference: Path) -> bool

Whether reference holds a release.json that cannot be read whole.

Present and not parseable, not an object, or with a typed key read_release had to drop. Told apart from absence for the caller that has to act on it: a provenance failure is not a data failure, so the snapshot stays usable and says so; and a writer merging into the file must leave it alone rather than replace the only statement of where the data came from. Logs nothing: read_release is where the warning is said.

Source code in schema/src/just_dna_format/snapshot.py
def release_is_unreadable(reference: Path) -> bool:
    """Whether `reference` holds a `release.json` that cannot be read whole.

    Present and not parseable, not an object, or with a typed key `read_release` had to drop. Told
    apart from absence for the caller that has to act on it: a provenance failure is not a data
    failure, so the snapshot stays usable and says so; and a writer merging into the file must leave
    it alone rather than replace the only statement of where the data came from. Logs nothing:
    `read_release` is where the warning is said.
    """
    path = Path(reference) / RELEASE_FILENAME
    if not path.is_file():
        return False
    payload, _ = _load(path)
    if payload is None:
        return True
    try:
        SnapshotRelease.model_validate(payload)
    except ValidationError:
        return True
    return False

release_label

release_label(reference: Path) -> str | None

The release a snapshot directory states, its dataset, or None when it states none.

The one reader of the label, for cache status and for every pass that stamps it onto SourceRow.dataset, so a reader of the cache and a reader of a module see the same string. No fallback and no guess: a blank, absent or unreadable label is an unknown, and an unknown is withheld rather than written as a label something could match.

Source code in schema/src/just_dna_format/snapshot.py
def release_label(reference: Path) -> str | None:
    """The release a snapshot directory states, its `dataset`, or `None` when it states none.

    The one reader of the label, for `cache status` and for every pass that stamps it onto
    `SourceRow.dataset`, so a reader of the cache and a reader of a module see the same string. No
    fallback and no guess: a blank, absent or unreadable label is an unknown, and an unknown is
    withheld rather than written as a label something could match.
    """
    release = read_release(reference)
    label = (release.dataset or "").strip() if release is not None else ""
    return label or None

write_release

write_release(
    directory: Path, release: SnapshotRelease
) -> Path

Write release as directory/release.json, atomically, and return the path.

Every snapshot carries the canonical spellings because every record is written here. Atomic because a release.json truncated mid-write parses as valid JSON that is simply missing keys, and a reader would believe it. A builder does not call this: it opens a write with begin_snapshot_write, whose commit calls it after the old record was withdrawn (RM369).

Source code in schema/src/just_dna_format/snapshot.py
def write_release(directory: Path, release: SnapshotRelease) -> Path:
    """Write `release` as `directory/release.json`, atomically, and return the path.

    Every snapshot carries the canonical spellings because every record is written here. Atomic
    because a `release.json` truncated mid-write parses as valid JSON that is simply missing keys, and
    a reader would believe it. **A builder does not call this**: it opens a write with
    `begin_snapshot_write`, whose `commit` calls it after the old record was withdrawn (RM369).
    """
    return atomic_write_text(Path(directory) / RELEASE_FILENAME, release.to_json())

begin_snapshot_write

begin_snapshot_write(directory: Path) -> SnapshotWrite

Withdraw directory's release.json before its payload changes, and return the write handle.

The order is the contract: withdraw, move the bytes, describe them. A builder that swapped its payload in first and wrote the record after left, on a crash between the two (a full disk, a killed process), new bytes under the previous build's record, and every reader believed it. With the record withdrawn first, the worst a crash leaves is a snapshot that states no release, which every reader already treats as an unknown and withholds.

Call it immediately before the first byte of the live payload moves, never at the top of a build: a download or a parse that fails before then leaves the old snapshot whole, and it should stay described. And a build that fails after it is not given its old record back, because a restored record would describe bytes the failed build may have changed.

The handle's commit is the only way a builder writes release.json, so describing a snapshot without having withdrawn the old description is not expressible.

Source code in schema/src/just_dna_format/snapshot.py
def begin_snapshot_write(directory: Path) -> SnapshotWrite:
    """Withdraw `directory`'s `release.json` before its payload changes, and return the write handle.

    The order is the contract: **withdraw, move the bytes, describe them.** A builder that swapped its
    payload in first and wrote the record after left, on a crash between the two (a full disk, a
    killed process), new bytes under the previous build's record, and every reader believed it. With
    the record withdrawn first, the worst a crash leaves is a snapshot that states no release, which
    every reader already treats as an unknown and withholds.

    **Call it immediately before the first byte of the live payload moves**, never at the top of a
    build: a download or a parse that fails before then leaves the old snapshot whole, and it should
    stay described. And a build that fails after it is **not** given its old record back, because a
    restored record would describe bytes the failed build may have changed.

    The handle's `commit` is the only way a builder writes `release.json`, so describing a snapshot
    without having withdrawn the old description is not expressible.
    """
    directory = Path(directory)
    directory.mkdir(parents=True, exist_ok=True)
    _withdraw(directory)
    return SnapshotWrite(directory)

begin_block_write

begin_block_write(directory: Path, name: str) -> BlockWrite

Withdraw block name from directory's record before the bytes it describes change.

For a second half that writes a sidecar beside a snapshot another build owns (ClinVar's citations): the half owns one block of the record, never the record, so it withdraws that block and leaves the rest. Read-modify-write rather than rewrite, and an unreadable record is left alone, because overwriting it would destroy the only statement of where the data came from.

Source code in schema/src/just_dna_format/snapshot.py
def begin_block_write(directory: Path, name: str) -> BlockWrite:
    """Withdraw block `name` from `directory`'s record before the bytes it describes change.

    For a second half that writes a sidecar beside a snapshot another build owns (ClinVar's
    citations): the half owns one block of the record, never the record, so it withdraws that block
    and leaves the rest. Read-modify-write rather than rewrite, and an unreadable record is left
    alone, because overwriting it would destroy the only statement of where the data came from.
    """
    directory = Path(directory)
    if release_is_unreadable(directory):
        return BlockWrite(directory, name, readable=False)
    existing = read_release(directory)
    if existing is not None and name in (existing.model_extra or {}):
        stated = existing.model_dump(exclude_unset=True)
        stated.pop(name)
        write_release(directory, SnapshotRelease.model_validate(stated))
    return BlockWrite(directory, name, readable=True)