The MITOMAP-minus-ClinVar increment, as a derived cache lane (RM171).
The question this answers is not "does MITOMAP contribute sixteen new expert-panel calls". That
number is a fact about one join against one ClinVar vintage, and a hardcoded list of sixteen alleles
is a snapshot of a diff — stale the next time either parent is rebuilt. The question is what does
MITOMAP publish that the ClinVar cache does not, answered every time both parents are current, so
what a draft appends is the increment rather than an inventory somebody wrote down once.
Three buckets, and only one of them drafts.
- photocopy — the same event is in ClinVar, under either source's spelling. Nothing is drafted. On the measured vintage every
matched bracketed row carried
reviewed_by_expert_panel, so these are the same ClinGen VCEP call
arriving by two routes; drafting them would attribute an adopted source's judgement to the wrong
publisher, and feeding them to a ClinVar concordance check would be a tautology.
- rated miss — absent from ClinVar, and the bracket is one of the five documented VCEP classes.
This is what
draft-panel --source mitomap-miss writes.
- unrated miss — absent, and MITOMAP published no class this tier may map: no bracket at all, or
[VUS*], or a confirmation token on its own. Counted, never given a class.
A fourth bucket sits beside those three because the question cannot be asked of it. unmintable
is a row whose published alleles do not spell a VCF (ref, alt): prose in an allele column, or no
event at all. MITOMAP also writes a deletion with an empty alt (refna="TA", regna=":"), and since
RM293 that row is anchored on the vendored rCRS base at position - 1 and joined like any other,
after checking MITOMAP's refna is what rCRS has there; a row that disagrees with rCRS stays
unmintable. Its row in this lane carries the VCF spelling; allele_defect still records MITOMAP's.
The event join, and no position-level fallback. (start, ref, alt) on chrMT, upper-cased, with
every indel on both sides left-aligned against rCRS first (RM273). An exact join on spelling
called 8 of 23 miss indels new although ClinVar holds each at another anchor (7471 C>CC and
7472 A>CA are both ClinVar's 7465 A>AC), among them 5 of the 6 rated misses RM171 reported. rCRS
is vendored (_rcrs), so the build stays a function of its two parents. A position-level hit is still
a different allele at the same locus, and collapsing onto it would hide a real increment.
Its release.json pins both parents, which is what makes a ClinVar rebuild without a child
rebuild a detectable stale child rather than a silent one. stale_parents re-reads the parents on
disk and names the one that moved (@currency-asks-the-source-not-the-cache, applied to a derived
lane: the "source" is the two parents).
MissBuildResult
dataclass
MissBuildResult(
out_dir: Path,
parquet_file: Path,
buckets: dict[str, int] = dict(),
buckets_by_table: dict[str, dict[str, int]] = dict(),
rated_miss_by_class: dict[str, int] = dict(),
rated_miss_indels: int = 0,
withheld_in_miss: dict[str, int] = dict(),
unmintable: dict[str, int] = dict(),
clinvar_keys: int = 0,
citation_links: int = 0,
parents: dict[str, dict] = dict(),
)
The increment, its parents, and every number the join computed.
accounts_for_every_row
accounts_for_every_row(total: int) -> bool
Every MITOMAP row is in exactly one bucket — the partition, asserted rather than assumed.
Source code in enricher/src/just_dna_enricher/mitomap_miss_build.py
| def accounts_for_every_row(self, total: int) -> bool:
"""Every MITOMAP row is in exactly one bucket — the partition, asserted rather than assumed."""
return sum(self.buckets.values()) == total
|
parent_pin
parent_pin(directory: Path | None) -> dict
The identifying half of a parent's release.json, or {} when there is none to read.
{} is a real answer and is stored as one: a parent built by something that wrote no
release.json cannot be pinned, and recording an empty pin says so, where omitting the parent
entirely would read as "this child has one parent".
Source code in enricher/src/just_dna_enricher/mitomap_miss_build.py
| def parent_pin(directory: Path | None) -> dict:
"""The identifying half of a parent's `release.json`, or `{}` when there is none to read.
`{}` is a real answer and is stored as one: a parent built by something that wrote no
`release.json` cannot be pinned, and recording an empty pin says so, where omitting the parent
entirely would read as "this child has one parent".
"""
if directory is None:
return {}
path = Path(directory) / RELEASE_FILENAME
if not path.is_file():
return {}
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, ValueError):
return {}
if not isinstance(payload, dict):
return {}
return {key: payload[key] for key in PIN_KEYS if key in payload}
|
stale_parents
stale_parents(
miss_dir: Path,
*,
parents: dict[str, Path] | None = None,
) -> dict[str, tuple[dict, dict]]
Parents whose snapshot on disk no longer matches the pin this child was built against.
Returns parent -> (pinned, current) for each one that moved, empty when the child is current.
Absence of a parent is not staleness and is not reported here: a parent that is gone cannot be
compared, and reporting it as "moved" would send a reader looking for a rebuild that happened.
The caller distinguishes the two, because "rebuild the child" and "provision the parent" are
different instructions.
parents names where each parent lives when the caller knows; otherwise the path recorded in the
child's own release.json is used, which is what a later draft run has.
Source code in enricher/src/just_dna_enricher/mitomap_miss_build.py
| def stale_parents(miss_dir: Path, *, parents: dict[str, Path] | None = None) -> dict[str, tuple[dict, dict]]:
"""Parents whose snapshot on disk no longer matches the pin this child was built against.
Returns `parent -> (pinned, current)` for each one that moved, empty when the child is current.
**Absence of a parent is not staleness** and is not reported here: a parent that is gone cannot be
compared, and reporting it as "moved" would send a reader looking for a rebuild that happened.
The caller distinguishes the two, because "rebuild the child" and "provision the parent" are
different instructions.
`parents` names where each parent lives when the caller knows; otherwise the path recorded in the
child's own `release.json` is used, which is what a later `draft` run has.
"""
release = read_miss_release(miss_dir)
pinned = release.get("parents") or {}
if not isinstance(pinned, dict):
return {}
located = {name: Path(path) for name, path in (parents or {}).items()}
moved: dict[str, tuple[dict, dict]] = {}
for name, block in pinned.items():
if not isinstance(block, dict):
continue
recorded = {key: value for key, value in block.items() if key != "path"}
directory = located.get(name) or (Path(block["path"]) if block.get("path") else None)
if directory is None or not directory.is_dir():
continue
current = parent_pin(directory)
if current and current != recorded:
moved[name] = (recorded, current)
return moved
|
read_miss_release
read_miss_release(directory: Path) -> dict
A miss snapshot's release.json as a dict, or {} when it is absent or unreadable.
Source code in enricher/src/just_dna_enricher/mitomap_miss_build.py
| def read_miss_release(directory: Path) -> dict:
"""A miss snapshot's `release.json` as a dict, or `{}` when it is absent or unreadable."""
path = Path(directory) / RELEASE_FILENAME
if not path.is_file():
return {}
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, ValueError):
return {}
return payload if isinstance(payload, dict) else {}
|
miss_dataset_label
miss_dataset_label(release: dict) -> str | None
The increment's own release label — both parents, or nothing.
mitomap_2026-08-24+clinvar_2026-06-27. A derived artifact's identity is the pair it was derived
from: the same MITOMAP dump against a newer ClinVar is a different increment, and a label naming
only the source would say two different artifacts were the same one. None where either parent
is unlabelled, because half an identity is not a shorter identity — it is an unknown, and the
licence row withholds it rather than writing a label that cannot be compared
(@currency-asks-the-source-not-the-cache).
Source code in enricher/src/just_dna_enricher/mitomap_miss_build.py
| def miss_dataset_label(release: dict) -> str | None:
"""The increment's own release label — **both parents, or nothing**.
`mitomap_2026-08-24+clinvar_2026-06-27`. A derived artifact's identity is the pair it was derived
from: the same MITOMAP dump against a newer ClinVar is a different increment, and a label naming
only the source would say two different artifacts were the same one. `None` where either parent
is unlabelled, because half an identity is not a shorter identity — it is an unknown, and the
licence row withholds it rather than writing a label that cannot be compared
(`@currency-asks-the-source-not-the-cache`).
"""
parents = release.get("parents")
if not isinstance(parents, dict):
return None
mitomap = (parents.get("mitomap") or {}).get("dataset")
clinvar = (parents.get("clinvar") or {}).get("clinvar_file_date")
if not mitomap or not clinvar:
return None
return f"{mitomap}+clinvar_{clinvar}"
|
build_miss_snapshot
build_miss_snapshot(
mitomap_dir: Path, clinvar_dir: Path, out_dir: Path
) -> MissBuildResult
Join the MITOMAP snapshot against the ClinVar chrMT parquet and write the increment.
Every MITOMAP row is written, bucketed — not only the misses. Keeping the photocopies makes the
snapshot's central claim checkable from the snapshot itself (a photocopy key is present in the
parent, a miss key is absent), and it costs a thousand rows.
Rows are sorted by (table, start, ref, alt) so a rebuild from the same two parents is
byte-identical (Principle 7).
Source code in enricher/src/just_dna_enricher/mitomap_miss_build.py
| def build_miss_snapshot(mitomap_dir: Path, clinvar_dir: Path, out_dir: Path) -> MissBuildResult:
"""Join the MITOMAP snapshot against the ClinVar chrMT parquet and write the increment.
Every MITOMAP row is written, bucketed — not only the misses. Keeping the photocopies makes the
snapshot's central claim checkable from the snapshot itself (a photocopy key is present in the
parent, a miss key is absent), and it costs a thousand rows.
Rows are sorted by `(table, start, ref, alt)` so a rebuild from the same two parents is
byte-identical (Principle 7).
"""
if pl is None: # pragma: no cover - exercised only where the [dev] extra is absent
raise ImportError(
"polars is required to build the MITOMAP-miss snapshot; install the publisher/dev "
"surface with `pip install 'just-dna-enricher[dev]'` (or `uv sync --group dev`)."
)
mitomap_dir, clinvar_dir, out_dir = Path(mitomap_dir), Path(clinvar_dir), Path(out_dir)
frames = []
for name in VARIANT_PARQUET.values():
parquet = mitomap_dir / "data" / name
if not parquet.is_file():
raise MitomapError(
f"the MITOMAP parent at {mitomap_dir} has no {name}. Refusing rather than joining "
f"the table that is there: a miss count short a source table is measured against a "
f"denominator nobody stated. Build one with `just-dna-enricher mitomap build`."
)
frames.append(pl.read_parquet(parquet))
source = pl.concat(frames)
calls = _clinvar_calls(clinvar_dir)
result = MissBuildResult(out_dir=out_dir, parquet_file=out_dir / "data" / MISS_PARQUET)
result.clinvar_keys = len(calls)
buckets: Counter[str] = Counter()
by_table: dict[str, Counter[str]] = {}
classes: Counter[str] = Counter()
withheld: Counter[str] = Counter()
unmintable: Counter[str] = Counter()
rows: list[dict] = []
# MITOMAP `:` deletions anchored on rCRS and joined (RM293). Logged, not a result field: a new
# field is minor-class, and `main` is patch-only.
anchored_deletions = 0
for row in source.iter_rows(named=True):
defect = row.get("allele_defect")
start, ref, alt = row.get("start"), row.get("ref"), row.get("alt")
if defect == "right_anchored_deletion" and start is not None and ref:
anchored = _anchor_colon_deletion(int(start), str(ref))
if anchored is not None:
start, ref, alt = anchored
defect = None
anchored_deletions += 1
match: dict | None = None
if defect is not None or start is None or not ref or not alt:
bucket = "unmintable"
unmintable[str(defect or "non_nucleotide")] += 1
else:
match = calls.get(_event_key(int(start), str(ref), str(alt)))
if match is not None:
bucket = "photocopy"
elif row.get("clin_sig"):
bucket = "rated_miss"
classes[str(row["clin_sig"])] += 1
if len(str(ref)) != len(str(alt)):
result.rated_miss_indels += 1
else:
bucket = "unrated_miss"
if row.get("withheld_bracket"):
withheld[str(row["withheld_bracket"])] += 1
buckets[bucket] += 1
by_table.setdefault(str(row["table"]), Counter())[bucket] += 1
rows.append(
{
**row,
"start": start,
"ref": ref,
"alt": alt,
"chrom": CONTIG,
"bucket": bucket,
"key_shape": (
None
if defect is not None or not ref or not alt
else ("substitution" if len(str(ref)) == len(str(alt)) else "indel")
),
"clinvar_variation_id": (match or {}).get("variation_id"),
"clinvar_clin_sig": (match or {}).get("clin_sig"),
"clinvar_review_status": (match or {}).get("review_status"),
}
)
result.buckets = {name: buckets.get(name, 0) for name in BUCKETS}
result.buckets_by_table = {
table: {name: counts.get(name, 0) for name in BUCKETS} for table, counts in sorted(by_table.items())
}
result.rated_miss_by_class = dict(sorted(classes.items()))
result.withheld_in_miss = dict(sorted(withheld.items()))
result.unmintable = dict(sorted(unmintable.items()))
result.parents = {
"mitomap": {**parent_pin(mitomap_dir), "path": str(mitomap_dir.resolve())},
"clinvar": {**parent_pin(clinvar_dir), "path": str(clinvar_dir.resolve())},
}
data_dir = out_dir / "data"
data_dir.mkdir(parents=True, exist_ok=True)
frame = pl.DataFrame(rows, schema=_miss_schema()).sort(
["table", "start", "ref", "alt", "record_id"], nulls_last=True
)
frame.write_parquet(result.parquet_file)
result.citation_links = _write_citations(mitomap_dir, data_dir, frame)
_write_release_json(out_dir, result, source_rows=source.height)
logger.info(
"Built the MITOMAP-miss snapshot: %s → %s (%d `:` deletion(s) anchored on rCRS, RM293)",
", ".join(f"{name} {count}" for name, count in result.buckets.items()),
result.parquet_file,
anchored_deletions,
)
return result
|