just_dna_enricher.clinvar_gkm_build¶
just_dna_enricher.clinvar_gkm_build ¶
The ClinVar-GKM lane's builder (RM305): two Parquet files in, the indel witness out.
What the lane is, and why its pin is (source_release, sha256), is in clinvar_gkm. This module is
the conversion: download both inputs whole, refuse any byte that is not the file manifest.json
states, keep the net insertions and net deletions on GRCh38, join each to the ClinVar variation and
rsID that name it, and write data/, LICENSE.txt and release.json.
Indels only (decided 2026-11-04). Kept: every allele whose SPDI deletes and inserts a different
number of bases. Dropped, and counted per class in release.json: no SPDI expression (GRCh37- or
transcript-only names), an accession off the GRCh38 primary assembly (one T2T allele), the same length
deleted as inserted (substitutions, MNVs, and the reference-agreeing ReferenceLengthExpression
alleles such as NC_000021.9:g.45504512_45504520=, whichever state type carries them), and a gnomad
expression that does not spell a VCF allele.
Downloaded whole, never range-read, because the source asks for it ("Download the Parquet files you
need, then query them locally. The R2 hosting has rate limits and is designed for file downloads, not as
a remote query endpoint") and because an R2 range request once came back 200 with the whole file
(LANES_0_8 §4). A whole download is checked by size and digest, which a partial body fails either way.
SliceCounts
dataclass
¶
SliceCounts(
alleles: int = 0,
no_spdi: int = 0,
off_grch38: int = 0,
same_length: int = 0,
no_vcf_spelling: int = 0,
indels: int = 0,
)
How every allele read was classified. Each dropped class is counted, never silently lost.
GkmBuildResult
dataclass
¶
GkmBuildResult(
out_dir: Path,
witness_file: Path,
dataset: str,
source_release: str,
row_count: int,
counts: SliceCounts,
with_rsid: int,
distinct_rsids: int,
latest_release: str | None = None,
source_sha256: dict[str, str | None] = dict(),
)
What a build wrote and what it read.
source_lags
property
¶
Whether the Parquet set the build read is older than the line's newest weekly release.
verify_against_manifest ¶
Refuse bytes that are not the file the manifest states: wrong size first, then wrong digest.
The size is what a partial body fails (a range answered with a slice, a cut connection the
transport did not report); the digest is what an in-place overwrite fails. Both are
ClinVarGkmMismatch, because both mean the same thing to a caller: these bytes are not the release
the label would name.
Source code in enricher/src/just_dna_enricher/clinvar_gkm_build.py
download_input ¶
download_input(
name: str,
manifest: GkmManifest,
dest: Path,
*,
url: str | None = None,
) -> StreamedFile
Download one input whole and verify it against manifest. Retries the transport (net).
Source code in enricher/src/just_dna_enricher/clinvar_gkm_build.py
indel_slice ¶
The net insertions and net deletions of allele.parquet, keyed by VRS id and VCF spelling.
Source code in enricher/src/just_dna_enricher/clinvar_gkm_build.py
variation_links ¶
(vrs_id, clinvar_variation_id, rsid) from variation.parquet, by projected read.
The allele → variation link is the variation's DefiningAlleleConstraint (one to one, LANES §3.4);
the rsID is a dbSNP mappings code, prefixed here with rs. A variation with no rsID keeps its
row with rsid null, so the left join below loses no allele.
Source code in enricher/src/just_dna_enricher/clinvar_gkm_build.py
clinvar_gkm_license_text ¶
LICENSE.txt: the source's own licence, scope, attribution and disclaimer, verbatim.
Source code in enricher/src/just_dna_enricher/clinvar_gkm_build.py
build_snapshot ¶
build_snapshot(
allele_path: Path,
variation_path: Path,
out_dir: Path,
*,
manifest: GkmManifest,
source_sha256: dict[str, str | None] | None = None,
source_url: dict[str, str | None] | None = None,
latest_release: str | None = None,
) -> GkmBuildResult
Cut the indel slice out of the two inputs and write data/, LICENSE.txt and release.json.
source_sha256 is each input's digest as the caller established it. When the inputs came off
local disk and the caller passes none, they are hashed here and checked against manifest the
same way a download is, so a stale local copy is refused rather than labelled with today's release.
Source code in enricher/src/just_dna_enricher/clinvar_gkm_build.py
308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 | |
download_and_build ¶
download_and_build(
out_dir: Path,
*,
client: ClinVarGkmClient | None = None,
keep_downloads: bool = False,
manifest: GkmManifest | None = None,
) -> GkmBuildResult
Read the manifest, download both inputs whole, verify them, cut the slice, drop the downloads.
The manifest is read twice: before the downloads, to know what to verify against, and after, so a release that landed mid-build is named as such rather than reported as a corrupt file.