Skip to content

just_dna_enricher.clinvar_gkm

just_dna_enricher.clinvar_gkm

ClinGen's ClinVar-GKM, cut to its indels: an offline witness for where an indel sits (RM305).

What this lane is for, and what it is not. Three checks need to know whether a placement is right (RM267's two authorities, RM270's respelling, RM292's named witness), and each compares today against a source that shares our failure. ClinVar-GKM republishes ClinVar in GA4GH form, through a pipeline that is not ours, with a spelling-independent VRS id on every allele. That makes it a witness: the lane answers where does an independent party place this indel, and nothing else. It runs no check of its own; a check names it as what it checked against. Clinical significance still comes from the clinvar lane.

Indels only (decided 2026-11-04). The ~4.6 M alleles are mostly substitutions, whose placement is not in doubt and whose ids the format tier mints with the stdlib. What is kept is every allele whose SPDI deletes and inserts a different number of bases: a net insertion or a net deletion. That drops substitutions, same-length MNVs, and the reference-agreeing ReferenceLengthExpression alleles (NC_000021.9:g.45504512_45504520=, deleting and inserting the same sequence), whichever state type carries them. An allele with no SPDI expression (GRCh37- or transcript-only names) and one off the GRCh38 primary assembly (one T2T accession) are dropped too; the build counts each class.

Where the bytes come from. One Parquet set under https://data.clinvar-gkm.org/v1/1.1/parquet/, overwritten in place every week, with no dated or archived copies (docs/probes/LANES_0_8.md §3.1). So a release cannot be pinned by URL. The pin is (source_release, sha256) from the set's own manifest.json, and the builder refuses bytes that do not match it: a file overwritten between the manifest read and the download is drift, and a build from it would describe one release with another's bytes. Both inputs are downloaded whole and read locally, which is what the source asks ("Download the Parquet files you need, then query them locally. The R2 hosting has rate limits and is designed for file downloads, not as a remote query endpoint"). A whole download also keeps the one failure the probe saw off this path: an R2 range request that came back 200 with the whole file. A truncated or partial body cannot pass the size and digest check either way.

Two keys every row has, and one most rows have. The VRS id and the gnomAD-style VCF spelling (4-87310240-T-TA) are on every allele; the rsID is on about 71% of the indels (LANES §3.5), and lives in variation.parquet, joined through the variation's DefiningAlleleConstraint. A reader keys on the first two and treats the rsID as secondary.

The VRS version is declared, not measured. ClinVar-GKM's 1.0.0 changelog says its ids are minted under VRS 2.1.1; neither the Parquet nor its manifest carries a version (LANES §3.6). So release.json records vrs_version with the basis beside it. This tier mints under 2.0 (RM304), and 2.0 and 2.1 move the ids of some ambiguous insertions, so a VRS-id comparison against this witness is only sound when the minor matches (ids_comparable). Until vrs-python ships 2.1, the VCF spelling is the key a comparison uses on every row; the caller supplies it left-normalized, since the gnomad expression is.

ClinVarGkmError

Bases: RuntimeError

The ClinVar-GKM lane could not be built or read.

ClinVarGkmUnavailable

Bases: ClinVarGkmError

The source could not be asked at all: a failed request, never "it published nothing".

A subclass so an except ClinVarGkmError keeps catching it, and a caller that has to tell unreachable from unreadable can (@client-exception-contract): the narrow arm goes first.

ClinVarGkmRetired

Bases: ClinVarGkmUnavailable, EndpointRetired

The source answered HTTP 410: it retired the file asked for (RM354).

A fact about the source, not an outage, and not retried. A ClinVarGkmUnavailable so every existing handler still catches it; the message says the file moved rather than that the host was down.

ClinVarGkmMismatch

Bases: ClinVarGkmError

The bytes are not the ones the manifest states: the set was overwritten, or the body is partial.

ManifestFile dataclass

ManifestFile(name: str, size: int, sha256: str)

One file of the published Parquet set, as manifest.json states it.

GkmManifest dataclass

GkmManifest(
    source_release: str,
    files: dict[str, ManifestFile],
    dataset: str | None = None,
    build_date: str | None = None,
    schema_version: str | None = None,
    license_id: str | None = None,
    license_scope: str | None = None,
    attribution: str | None = None,
    disclaimer: str | None = None,
    data_use_policy: str | None = None,
)

parquet/manifest.json: which release the one Parquet set holds, and each file's digest.

source_release is the ClinVar release the set was built from, and the half of the pin a label is made of. The licence fields are kept verbatim because the snapshot's LICENSE.txt quotes them.

file

file(name: str) -> ManifestFile

The manifest's entry for name, or a refusal: a build cannot verify a file it does not list.

Source code in enricher/src/just_dna_enricher/clinvar_gkm.py
def file(self, name: str) -> ManifestFile:
    """The manifest's entry for `name`, or a refusal: a build cannot verify a file it does not list."""
    entry = self.files.get(name)
    if entry is None:
        raise ClinVarGkmError(f"manifest.json lists no {name}; the set's layout changed")
    return entry

ClinVarGkmClient dataclass

ClinVarGkmClient(
    manifest_url: str = PARQUET_MANIFEST_URL,
    index_url: str = RELEASE_INDEX_URL,
    timeout: float = 30.0,
    client: Client | None = None,
    gate: PacingGate | None = None,
)

The two small JSON documents that say what the source publishes now.

parquet_manifest() is the Parquet set's own record; latest_release() is the release line's index, whose latest_release is the newest weekly release. They differ: the Parquet set is rebuilt from a weekly release a few days after it lands, so on 2026-11-04 the set held 2026-10-24 while the line's newest release was 2026-10-31 (LANES §3.2). @client-exception-contract: transport and status failures are retried, then arrive as ClinVarGkmUnavailable, never as an httpx type.

file_url

file_url(name: str) -> str

A file of the Parquet set: it sits beside the manifest that describes it.

Source code in enricher/src/just_dna_enricher/clinvar_gkm.py
def file_url(self, name: str) -> str:
    """A file of the Parquet set: it sits beside the manifest that describes it."""
    return f"{self.manifest_url.rsplit('/', 1)[0]}/{name}"

parquet_manifest

parquet_manifest() -> GkmManifest

The published Parquet set's manifest. Raises ClinVarGkmUnavailable when unreachable.

Source code in enricher/src/just_dna_enricher/clinvar_gkm.py
def parquet_manifest(self) -> GkmManifest:
    """The published Parquet set's manifest. Raises `ClinVarGkmUnavailable` when unreachable."""
    return parse_manifest(self._fetch(self.manifest_url, "Parquet manifest"))

latest_release_or_none

latest_release_or_none() -> str | None

latest_release, withheld as None when the index cannot be read.

The index answers one informational question (does the Parquet set lag the line?). Its outage may not speak for the manifest's questions (@one-registrys-outage-may-not-speak-for- another), so a build and a currency check go on without it and say the lag is unknown.

Source code in enricher/src/just_dna_enricher/clinvar_gkm.py
def latest_release_or_none(self) -> str | None:
    """`latest_release`, withheld as `None` when the index cannot be read.

    The index answers one informational question (does the Parquet set lag the line?). Its
    outage may not speak for the manifest's questions (`@one-registrys-outage-may-not-speak-for-
    another`), so a build and a currency check go on without it and say the lag is unknown.
    """
    try:
        return self.latest_release()
    except ClinVarGkmError as exc:
        logger.warning("ClinVar-GKM's release index could not be read (%s); the lag is unknown.", exc)
        return None

latest_release

latest_release() -> str | None

The release line's newest weekly release, or None when the index states none.

Source code in enricher/src/just_dna_enricher/clinvar_gkm.py
def latest_release(self) -> str | None:
    """The release line's newest weekly release, or `None` when the index states none."""
    try:
        raw = json.loads(self._fetch(self.index_url, "release index"))
    except ValueError as exc:
        raise ClinVarGkmError(f"the release index is not JSON: {exc}") from exc
    stated = str(raw.get("latest_release") or "").strip() if isinstance(raw, dict) else ""
    return stated or None

current_release

current_release() -> str | None

The currency probe: the label a rebuild today would write, read off the live manifest.

The Parquet set's release, not the line's newest weekly one. A module drafted against this lane can only ever be as current as the set, so comparing it against a release no Parquet carries would report every module behind for the week the set lags, with nothing to re-draft from. The set's own lag is LaneCurrency.source_lags, kept apart.

Source code in enricher/src/just_dna_enricher/clinvar_gkm.py
def current_release(self) -> str | None:
    """The currency probe: the label a rebuild today would write, read off the live manifest.

    The **Parquet set's** release, not the line's newest weekly one. A module drafted against this
    lane can only ever be as current as the set, so comparing it against a release no Parquet
    carries would report every module behind for the week the set lags, with nothing to re-draft
    from. The set's own lag is `LaneCurrency.source_lags`, kept apart.
    """
    return clinvar_gkm_release_label(self.parquet_manifest().source_release)

LaneCurrency dataclass

LaneCurrency(
    pinned_release: str | None,
    pinned_sha256: tuple[tuple[str, str | None], ...]
    | None,
    published_release: str | None = None,
    published_sha256: tuple[tuple[str, str | None], ...]
    | None = None,
    latest_release: str | None = None,
    unchecked: str | None = None,
)

A built snapshot against what the source publishes now — three questions, kept apart.

  • behind: the source has published a newer Parquet set, so a rebuild would get newer bytes.
  • drifted: the source's set names the same release with different bytes. The pin caught an in-place overwrite the label cannot see.
  • source_lags: the published set is older than the line's newest weekly release. Nothing here can fix that (no Parquet exists for the newer release), so it is reported, never folded into behind: a rebuild would change nothing.

Each is tri-state; unchecked names why none could be answered (@currency-asks-the-source-not- the-cache: the published release comes over the wire, never from the snapshot's own record).

sentences

sentences() -> list[str]

What an operator is told: each answered question that found something, and every unknown.

Source code in enricher/src/just_dna_enricher/clinvar_gkm.py
def sentences(self) -> list[str]:
    """What an operator is told: each answered question that found something, and every unknown."""
    if self.unchecked is not None:
        return [f"clinvar_gkm currency unchecked: {self.unchecked}"]
    out: list[str] = []
    if self.behind:
        out.append(
            f"clinvar_gkm: the snapshot holds {self.pinned_release} and the source publishes "
            f"{self.published_release}; rebuild with `clinvar-gkm build`"
        )
    elif self.behind is None:
        out.append(
            "clinvar_gkm: whether the snapshot is behind could not be decided (an unstated release)"
        )
    if self.drifted:
        moved = [
            name
            for (name, ours), (_, theirs) in zip(
                self.pinned_sha256 or (), self.published_sha256 or (), strict=True
            )
            if ours != theirs
        ]
        out.append(
            f"clinvar_gkm: the source's {', '.join(moved)} for {self.pinned_release} is no longer the "
            f"bytes this snapshot was cut from (overwritten in place); rebuild"
        )
    if self.source_lags:
        out.append(
            f"clinvar_gkm: the source's Parquet set holds {self.published_release} while its newest "
            f"weekly release is {self.latest_release}; no Parquet exists for that release yet"
        )
    return out

WitnessPlacement dataclass

WitnessPlacement(
    vrs_id: str,
    chrom: str,
    pos: int,
    ref: str,
    alt: str,
    gnomad: str,
    spdi: str,
    state_type: str | None,
    clinvar_variation_id: str | None,
    rsid: str | None,
)

Where ClinVar-GKM places one allele: one row of the slice.

Flat, one row per (allele, rsID): an allele with two rsIDs is two rows (4 alleles on 2026-10-24), and an rsID naming two alleles (a multi-allelic site) returns both. A lookup never picks one (@multiplicity-is-a-finding).

vcf_key property

vcf_key: str

chrom:pos:ref:alt, the coordinate form variant_key uses for one ALT.

ClinVarGkmWitness dataclass

ClinVarGkmWitness(
    snapshot_dir: Path,
    release: SnapshotRelease,
    frame: DataFrame,
)

The loaded slice, and the three lookups a check makes against it.

[] from any lookup means not witnessed, never placed wrong: the lane holds indels only, a ClinVar-GKM release holds only what ClinVar holds, and 29% of the indels carry no rsID.

vrs_version property

vrs_version: str | None

The VRS version the witness's ids were minted under, as release.json declares it.

ids_comparable

ids_comparable(our_vrs_spec: str | None) -> bool | None

Whether a VRS id we minted under our_vrs_spec may be compared to the witness's ids.

Tri-state at the minor, the grain the id rule changes at (RM304): None when either side is unstated. When it is not True, compare by the normalized VCF spelling instead.

Source code in enricher/src/just_dna_enricher/clinvar_gkm.py
def ids_comparable(self, our_vrs_spec: str | None) -> bool | None:
    """Whether a VRS id we minted under `our_vrs_spec` may be compared to the witness's ids.

    Tri-state at the minor, the grain the id rule changes at (RM304): `None` when either side is
    unstated. When it is not `True`, compare by the normalized VCF spelling instead.
    """
    ours, theirs = vrs_spec_release(our_vrs_spec), vrs_spec_release(self.vrs_version)
    if ours is None or theirs is None:
        return None
    return ours == theirs

by_rsids

by_rsids(
    rsids: Iterable[str],
) -> dict[str, list[WitnessPlacement]]

by_rsid over many rsIDs in one filter, keyed by the rsID as asked; an absent one is absent.

One pass over the slice rather than one per rsID: a module asks about every rsID it places (RM267), and the slice is every ClinVar indel.

Source code in enricher/src/just_dna_enricher/clinvar_gkm.py
def by_rsids(self, rsids: Iterable[str]) -> dict[str, list[WitnessPlacement]]:
    """`by_rsid` over many rsIDs in one filter, keyed by the rsID as asked; an absent one is absent.

    One pass over the slice rather than one per rsID: a module asks about every rsID it places
    (RM267), and the slice is every ClinVar indel.
    """
    asked = {(t if (t := r.strip().lower()).startswith("rs") else f"rs{t}"): r for r in rsids}
    found: dict[str, list[WitnessPlacement]] = {}
    for row in self._rows(pl.col("rsid").is_in(list(asked))):
        found.setdefault(asked[str(row.rsid)], []).append(row)
    return found

by_vcf

by_vcf(
    chrom: str, pos: int, ref: str, alt: str
) -> list[WitnessPlacement]

Exact match on the left-normalized VCF spelling. The caller normalizes; this never does.

Source code in enricher/src/just_dna_enricher/clinvar_gkm.py
def by_vcf(self, chrom: str, pos: int, ref: str, alt: str) -> list[WitnessPlacement]:
    """Exact match on the left-normalized VCF spelling. The caller normalizes; this never does."""
    return self._rows(
        (pl.col("chrom") == normalize_chrom(chrom))
        & (pl.col("pos") == int(pos))
        & (pl.col("ref") == ref.strip().upper())
        & (pl.col("alt") == alt.strip().upper())
    )

lookup

lookup(key: str) -> list[WitnessPlacement]

A variant_key, a VRS id or an rsID → the placements the witness holds for it.

A coordinate key must carry exactly one ALT (chrom:start:ref:alt): a position-only key names no allele, and a multi-ALT key names several, so both are refused rather than half-answered.

Source code in enricher/src/just_dna_enricher/clinvar_gkm.py
def lookup(self, key: str) -> list[WitnessPlacement]:
    """A `variant_key`, a VRS id or an rsID → the placements the witness holds for it.

    A coordinate key must carry exactly one ALT (`chrom:start:ref:alt`): a position-only key names
    no allele, and a multi-ALT key names several, so both are refused rather than half-answered.
    """
    token = key.strip()
    if token.startswith("ga4gh:VA."):
        return self.by_vrs_id(token)
    if re.fullmatch(r"(?i)rs\d+", token):
        return self.by_rsid(token)
    parts = token.split(":")
    if len(parts) == 4 and parts[1].isdigit() and "," not in parts[3]:
        return self.by_vcf(parts[0], int(parts[1]), parts[2], parts[3])
    raise ValueError(
        f"{key!r} is not a VRS id, an rsID or a one-ALT coordinate key (chrom:start:ref:alt)"
    )

parquet_url

parquet_url(name: str) -> str

The URL of one file of the published Parquet set.

Source code in enricher/src/just_dna_enricher/clinvar_gkm.py
def parquet_url(name: str) -> str:
    """The URL of one file of the published Parquet set."""
    return f"{CLINVAR_GKM_BASE_URL}/parquet/{name}"

parse_manifest

parse_manifest(text: str) -> GkmManifest

Read manifest.json. A missing release or file digest is a refusal, never a default.

Source code in enricher/src/just_dna_enricher/clinvar_gkm.py
def parse_manifest(text: str) -> GkmManifest:
    """Read `manifest.json`. A missing release or file digest is a refusal, never a default."""
    try:
        raw = json.loads(text)
    except ValueError as exc:
        raise ClinVarGkmError(f"manifest.json is not JSON: {exc}") from exc
    if not isinstance(raw, dict):
        raise ClinVarGkmError("manifest.json is not a JSON object")
    release = str(raw.get("source_release") or "").strip()
    if not release:
        raise ClinVarGkmError("manifest.json states no source_release, so nothing can be pinned")
    files: dict[str, ManifestFile] = {}
    entries = raw.get("files") or []
    if not isinstance(entries, list):
        raise ClinVarGkmError(f"manifest.json's files is not a list: {entries!r}")
    for entry in entries:
        if not isinstance(entry, dict):
            raise ClinVarGkmError(f"manifest.json has a file entry that is not an object: {entry!r}")
        name, size, sha = entry.get("name"), entry.get("size"), entry.get("sha256")
        sized = isinstance(size, int) and not isinstance(size, bool)
        if not isinstance(name, str) or not sized or not isinstance(sha, str):
            raise ClinVarGkmError(f"manifest.json has a file entry without name/size/sha256: {entry!r}")
        files[name] = ManifestFile(name, size, sha.strip().lower())
    licence = raw.get("license") if isinstance(raw.get("license"), dict) else {}
    source = raw.get("source") if isinstance(raw.get("source"), dict) else {}
    return GkmManifest(
        source_release=release,
        files=files,
        dataset=raw.get("dataset"),
        build_date=raw.get("build_date"),
        schema_version=raw.get("schema_version"),
        license_id=licence.get("id"),
        license_scope=licence.get("scope"),
        attribution=raw.get("attribution"),
        disclaimer=raw.get("disclaimer"),
        data_use_policy=source.get("data_use_policy"),
    )

clinvar_gkm_release_label

clinvar_gkm_release_label(
    source_release: str | None,
) -> str | None

clinvar_gkm_<source_release>, or None when no release is stated.

One formula, called by the builder that writes dataset and by the currency probe that reads the live manifest, so a snapshot and the source name one release with one string.

Source code in enricher/src/just_dna_enricher/clinvar_gkm.py
def clinvar_gkm_release_label(source_release: str | None) -> str | None:
    """`clinvar_gkm_<source_release>`, or `None` when no release is stated.

    One formula, called by the builder that writes `dataset` and by the currency probe that reads the
    live manifest, so a snapshot and the source name one release with one string.
    """
    stated = str(source_release or "").strip()
    return f"{CLINVAR_GKM_DATASET_PREFIX}{stated}" if stated else None

release_older

release_older(
    first: str | None, second: str | None
) -> bool | None

Tri-state first < second over two YYYY-MM-DD dates; None when either is not one.

Source code in enricher/src/just_dna_enricher/clinvar_gkm.py
def release_older(first: str | None, second: str | None) -> bool | None:
    """Tri-state `first < second` over two `YYYY-MM-DD` dates; `None` when either is not one."""
    if not first or not second or not _DATE_PATTERN.match(first) or not _DATE_PATTERN.match(second):
        return None
    return date.fromisoformat(first) < date.fromisoformat(second)

bare_digest

bare_digest(value: str | None) -> str | None

A digest without its sha256: prefix(es), lower case; None for blank.

Source code in enricher/src/just_dna_enricher/clinvar_gkm.py
def bare_digest(value: str | None) -> str | None:
    """A digest without its `sha256:` prefix(es), lower case; `None` for blank."""
    stated = (value or "").strip().lower()
    while stated.startswith("sha256:"):
        stated = stated.removeprefix("sha256:")
    return stated or None

lane_currency

lane_currency(
    snapshot_dir: Path,
    *,
    client: ClinVarGkmClient | None = None,
    offline: bool = False,
) -> LaneCurrency

Ask the source whether snapshot_dir is current. Reads the snapshot; writes nothing.

Source code in enricher/src/just_dna_enricher/clinvar_gkm.py
def lane_currency(
    snapshot_dir: Path,
    *,
    client: ClinVarGkmClient | None = None,
    offline: bool = False,
) -> LaneCurrency:
    """Ask the source whether `snapshot_dir` is current. Reads the snapshot; writes nothing."""
    release = read_release(Path(snapshot_dir)) or SnapshotRelease()
    pinned_release = release.source_release
    pinned = LaneCurrency(pinned_release=pinned_release, pinned_sha256=_pinned_digests(release))
    if offline:
        return LaneCurrency(pinned.pinned_release, pinned.pinned_sha256, unchecked="offline")
    asker = client or ClinVarGkmClient()
    try:
        manifest = asker.parquet_manifest()
    except ClinVarGkmUnavailable as exc:
        # The reason off the exception (RM361): `ClinVarGkmRetired` is a `ClinVarGkmUnavailable`, and an
        # equality-on-constants walk could not see this f-string write `unreachable` about it.
        reason = could_not_ask_reason(exc)
        return LaneCurrency(pinned.pinned_release, pinned.pinned_sha256, unchecked=f"{reason} ({exc})")
    except ClinVarGkmError as exc:
        return LaneCurrency(pinned.pinned_release, pinned.pinned_sha256, unchecked=f"unreadable ({exc})")
    return LaneCurrency(
        pinned_release=pinned.pinned_release,
        pinned_sha256=pinned.pinned_sha256,
        published_release=manifest.source_release,
        published_sha256=_published_digests(manifest),
        latest_release=asker.latest_release_or_none(),
    )

normalize_chrom

normalize_chrom(chrom: str) -> str

A contig as the gnomad expression spells it: no chr, upper case, M read as MT.

Source code in enricher/src/just_dna_enricher/clinvar_gkm.py
def normalize_chrom(chrom: str) -> str:
    """A contig as the `gnomad` expression spells it: no `chr`, upper case, `M` read as `MT`."""
    token = chrom.strip()
    if token[:3].lower() == "chr":
        token = token[3:]
    token = token.upper()
    return "MT" if token == "M" else token