Skip to content

just-dna-enricher — the network tier

The package reference for just-dna-enricher: the only tier in the workspace allowed to fetch. It produces the source-independent resolution table (resolution.csv) that the compiler consumes, and carries the publisher surface for pushing compiled modules to HuggingFace. New in 0.5.

Enrichment is partly validation, and that is a goal rather than a side effect. Filling in what a module left out is only half of what this tier does; the other half is checking what the module asserted against what the sources actually say. It is the only tier that can: the format and compiler tiers are inject-only by charter (Principle 2) and hold no reference to check anything against. So surfacing a discrepancy — an authored ref that contradicts the genome, a source id that disagrees with a locally-minted one, an rsID that maps somewhere else — is part of the job. Two rules govern every such check: it reports and never repairs (silently rewriting an authored value destroys the evidence that something upstream is wrong, and turns a loud data problem into a quiet one), and its severity follows the mode (best_effort warns and carries on; strict refuses). What exists today:

Check Compares Where Reads Against Attests as
Reference allele authored ref vs the actual reference sequence sequences.verify_reference_alleles authored seqrepo reference_allele
Wrong build a ref-mismatched row vs the same coordinate on GRCh37 (0.6) grch37.diagnose_wrong_build (warns in both modes) authored ensembl-grch37 genome_build_agreement
VRS cross-check a source's own vrs_id vs the locally-minted one vrs.mint_resolution_rows reporting_source seqrepo vrs_allele_id
rsid↔coordinate an authored pair vs what the reference says compiler/resolution.py::_verify over the injected table (warning), and enrich() against the injected Ensembl snapshot (resolver.check_rsid_coordinates, warning in both modes) — one question, two tiers, so one attestation name, and the enricher's half is the one that attests. An indel authored at another anchor of its repeat is the same event (events.names_event, against the reference, 0.8 RM375) authored; abstentions counted in undecided ensembl rsid_coordinate_agreement
Authored spelling a subject's authored coordinate (whole, or CPIC's half) vs each locus written for it, where the two spellings differ (0.8, RM375) events.authored_event, stamped as resolution.csv authored_event by enrich(); the compiler reads same_event as agreement (resolution.recorded_verdicts, bound to judged_coordinate) and judges nothing itself — — — (recorded onto the row, not attested; the reference is read through seqrepo)
Placement agreement where the run's own link placed an rsID vs where ClinVar and ClinVar-GKM place it, and, online with the merge check on, where dbSNP's own record places a snapshot indel neither holds (0.8, RM267; dbSNP since RM398, Ensembl REST before); a disagreement withholds the disputed locus placement.compare_placements, called by enrich() before the table is assembled (warns in both modes; strict refuses through the unresolved gate, since a withheld rsID-only row is unresolved) reporting_source, or ensembl on a record where only REST answered before RM398 (Ensembl against itself); abstentions counted in undecided clinvar, clinvar_gkm, dbsnp, ensembl placement_agreement
Ambiguous back-fill ≥2 rsIDs for one exact allele → recorded, never guessed resolver._lookup_rsid_candidates — — — (recorded onto the row, not attested)
Clinical significance authored clin_sig vs every annotation authority consulted — ClinVar and, since 0.7, PubMind — allele-exactly clinical.verify_clin_sig (warns in both modes), persisted as the N-authority concordance record by clinical.clin_sig_concordance (0.7, RM130 + RM134 § B) authored clinvar clinical_significance
Answered-call currency an authority's call now vs what it said when the author's overrides.csv answer was written (0.7, RM151) clinical.answered_call_shift → concordance.shifted_authority_calls (warns in both modes; the baseline is the previous run's clin_sig_authority_calls.csv, so it is read before the commit rewrites it and a move is observable exactly once) — — — (logged; the record it reads is the attestation)
PGx evidence level authored evidence_level vs ClinPGx's own for that annotation clinpgx.enrich_clinpgx (refuses in strict — the only enricher cross-check that does — on a row's own cited annotation only; RM297) authored clinpgx pgx_evidence_level
Citation existence a cited pmid vs PubMed; a pmid=NONE row's doi vs Crossref's registration agencies (0.8, RM349) literature.enrich_literature authored crossref, pubmed citation_existence
Identifier agreement an authored doi vs the registry's for that PMID literature.enrich_literature authored pubmed citation_identifier
PMC id agreement an authored PMC… in the pmid cell vs PubMed's for that record (0.6) literature._pmcid_conflicts (attested under citation_identifier, the same question one registry over) authored pubmed citation_identifier
Article licence the cited article's own terms, recorded per article (0.6) literature.enrich_literature → licensing.article_terms — — — (a recording pass: it writes a source's answer and compares nothing)
Provenance quote provenance_quote/provenance_regex vs the article's fulltext (Europe PMC's open-access subset, then PMC's BioC service; for a pmid=NONE row, the text Europe PMC holds for its DOI, RM349), else its abstract literature.enrich_literature (warning; partial coverage) authored europepmc provenance_quote
rsID currency an authored rsID vs dbSNP (live / merged / absent) identifiers.check_rsids authored dbsnp rsid_currency
Trait currency trait_efo_id vs OLS4 (obsolete + replacement) identifiers.OntologyClient.trait (attested by check-identifiers since RM72) authored ols4 trait_currency
Gene symbol currency gene vs HGNC approved / previous symbols identifiers.OntologyClient.gene (attested by check-identifiers since RM72) authored hgnc gene_symbol_currency
Gene ↔ locus agreement the row's gene vs the chromosome its variant sits on (0.5.4) identifiers.check_identifiers → GeneLocusConflict (attested since RM72) authored hgnc gene_locus_agreement
PGS accession currency an authored pgs_id vs the PGS Catalog's own record for it (0.7, RM163) — the Catalog answers 200 with {} for a never-assigned id and for a malformed one, so the check reads the body identifiers._check_pgs (attested by check-identifiers; --no-pgs records not_requested) authored pgs_catalog pgs_accession_currency
PGS metadata agreement authored training_ancestry / training_cohort vs the score record's ancestry_distribution / samples_training (0.7, RM163) — its own member, because currency asks whether the id still names a score and this asks whether two cells beside it still match identifiers._check_pgs (attested by check-identifiers) authored pgs_catalog pgs_metadata_agreement
ACMG secondary findings authored acmg_sf vs the published SF gene list (v3.3 via --sf-list; the scraped v3.2 page reports unverifiable) acmg.check_acmg_sf (attested by check-acmg since RM72) authored acmg acmg_secondary_findings
Repeat bands an authored repeat_alleles.csv band table vs STRchive's benign_*/intermediate_*/pathogenic_* (0.7, RM165) strchive.check_repeat_bands (warns in both modes; the catalogue's pathogenic_max is reported and never written) authored strchive repeat_band_agreement
Regulator drug labels a (gene[, allele], drug) claim vs the Testing Level five drug regulators' labels carry, at two join tiers (0.7, RM166) drug_labels.check_drug_labels (warns in both modes; a blank level is unknown and never No Clinical PGx) authored clinpgx regulator_label_agreement
Published refutation an authored direction vs the refutations CIViC publishes for the same variant (0.7, RM170) civic_refutation.compare_refutations → enrich() (warns in both modes; the record names its status_basis, because on the accepted basis the class is empty by construction) authored civic published_refutation
Literature coverage which papers a variant-literature index holds for a module's alleles, and at which tier (0.7, RM167) litvar.check_literature_coverage (reports only; allele-resolved, position-only and absent are three outcomes) authored litvar literature_coverage
Evidence-status currency a recorded StudyRow.confidence — a curation status a citing source published when the row was drafted — vs what that source says about the same item now (0.7, RM160) civic_citations.check_evidence_status_currency → enrich() (warns in both modes; a status accepted or rejected since, or a citation added since. Not dataset_currency: that one asks which release, this one asks whether one judgement moved) civic civic evidence_status_currency
Allele function authored function_status vs PharmVar and CPIC pgx.enrich_pgx (warns in both modes) authored cpic, pharmvar allele_function
Declared use the caller's --use vs a source's terms licensing.check_declared_use (refuses in both modes) — — — (an acquisition gate, not a comparison)
Drafted vs authored rows a source's current row vs the one already in the CSV just_dna_compiler.draft.append_rows (reports differs; never rewrites) — — — (a drafting report; it runs outside enrich())
A drafted table's licence row does sources.csv record what licensed the rows this append is landing? drafting.licence_commit, handed to every append_* as before_commit (RM232) — — refuses the run — the table is not committed, so a module never carries drafted rows with no licence record
Source coverage is the locus inside the source's callset at all? not_covered ≠ not_found gnomad.covers_locus → frequencies.enrich_frequencies (not a strict failure) — — — (a recording pass: coverage is a fact about the callset)
Dataset currency a recorded SourceRow.dataset vs the release that source publishes now (0.7) currency.check_dataset_currency → enrich() (reports; strict refuses over a superseded release, never over an unreachable source) recorded_source recorded_source dataset_currency
Variant impact an authored variant against the score AlphaGenome published for it — the three questions the local AVI artifact provably cannot answer (0.7, RM193) alphagenome_check.check_variant_impact (warns in both modes; non-commercial, so an undeclared run asks nothing. Its own member rather than a second writer of reference_allele, which the Atlas also answers — two registries answering an overlapping question get two names, or one source's outage writes a skip against the other's) authored alphagenome_avi variant_impact_agreement

The last column is the join, and it is asserted rather than promised. vocab.py's own comment says VALID_VERIFICATION_CHECKS was audited against this table — so a member with an emitter and no row here is a check a reader of verification.json cannot look up, which is what happened to variant_impact_agreement for a release. Every member that has an emitter names a row above (test_enricher_doc_registries.py walks it), and the two the vocabulary marks RESERVED — gene_disease_validity and dosage_sensitivity — deliberately have none: their passes record a source's verdict and compare nothing authored, and the member is held for a future pass that does. The containment runs one way only: rows whose last column is — are real work that attests nothing, either because it is an acquisition gate, a recording pass, or a report that runs outside enrich().

Reads and Against are the check's witness, and they are data (RM292, 0.8). Each is the checked/witness of the check's entry in verification.CHECK_WITNESSES, which ran() and skipped() read to fill every record's checked_source and witness_source — so a check with no entry cannot be recorded at all, and test_check_witness.py asserts the roster equals the emitting members. When the two are one source the record's independence reads self_consistent, never independent: the currency checks are that by construction (civic against civic, and recorded_source — each recorded release against the source that published it). authored is then settled against the module's licence table by record_verification, the one seam every command writes through — over the records carried from earlier runs as well as this run's, since the licence table is outside the authored binding — and it is three-valued: one source whose drafter digest still matches over the cells the check reads makes them that source's copy (a ClinVar-drafted, unedited clin_sig reads clinvar against clinvar), unless the witness is also named at the annotation layer and is not that source, since a digest covers rows an earlier draft put there; a licence row naming the witness with nothing establishing which cells it supplied withholds the side (null), as do two holding digests; no such row keeps authored; an unreadable table withholds. A checked side that is one of several witnesses (cpic against cpic+pharmvar) withholds independence too. A check with a tolerance publishes what it abstained on: rsid_coordinate_agreement counts the indel pairs whose position matches no snapshot locus into undecided, outside subjects, because a re-anchoring can move one legitimately (RM31) and an uncounted abstention reads as agreement (@tautology-zero). The roster says which checks have a tolerance, and ran() refuses a tolerant check's record without the count. Two honest limits: the reference-allele check is independent and still blind to a misplaced indel anchor, because the anchor base is a real genome base; and a CIViC-drafted module's refutation check withholds rather than reads self_consistent, because CIViC's drafter is a judgement drafter and records no digest that could establish the copy.

Every check that runs records what it did — verification.json (RM45, 0.6). Until 0.6 the table above described work whose result died with the process: a check's findings reached a log line and an EnrichmentResult field, and the compiled module could not say whether the check had been put at all. just_dna_enricher.verification.record_verification is the load-merge-write that closes that, in the same shape licensing.record_source_terms has and for the same reason — a count of call sites goes stale, one function does not. What to hold onto when wiring a new pass into it:

  • A new check starts with its roster entry (RM292). ran() and skipped() raise for a check with no CHECK_WITNESSES entry, so the first edit is the witness: what the check reads (authored, or the source whose earlier answer it re-asks), what it reads it against, and, if it abstains on a class, the tolerance whose count it must then publish as undecided.
  • A record carries two counts and a closed skip key. ran(check, subjects=…, findings=…) when it ran (subjects=0 is a legitimate answer meaning nothing was in scope) and skipped(check, reason, detail=…) when it did not. Both vocabularies are closed (VALID_VERIFICATION_CHECKS, VALID_VERIFICATION_SKIPS); the human sentence rides in detail, beside the machine key and never instead of it.
  • The denominator comes from the check, never from the caller. verify_reference_alleles returns a RefCheck and compare_clin_sig a ClinSigComparison (or None), so the count travels with the finding it belongs to: a count recomputed beside a check can disagree with it, and then the manifest's own two halves disagree. _verification_records deliberately takes neither variants nor rows — a function that cannot see the tables cannot be tempted to count them. Wiring this up surfaced a real hole in two passes: each had an internal skip (no sequence access; a snapshot present but not queryable) returning an empty list indistinguishable from a clean pass, which is S4's defect surviving inside the machinery S4 built.
  • The denominator is what was EXAMINED, not what existed. The wrong-build pass is bounded (DEFAULT_DIAGNOSIS_LIMIT), so on a panel authored wholesale on hg19 it asks about a sample; recording total there would claim rows it chose not to ask about. sampled is why the two can differ, and the record's detail says so when they do.
  • A downstream check inherits the reason its upstream did not run. The wrong-build pass reads the ref-mismatch list, and diagnose_wrong_build([]) answers no_ref_mismatches for an empty list whatever emptied it — a ref check that ran clean, or one that never ran. Recording the first unconditionally publishes "no authored ref disagreed with the reference" beside a reference_allele record saying nothing was compared: one document contradicting itself, and the false half is the answered-absence-versus-unasked-question collapse S20 exists to prevent.
  • Every early return records its skip — as long as the check APPLIES. enrich_clinpgx is the worked example: a licensing refusal and a missing snapshot both go through one _attest helper, because each says "this check applies to your module and did not run", and a pass that records its findings while staying silent about not having run leaves the manifest unable to tell those apart. Two paths deliberately attest nothing. A strict refusal raises, so no artifact was produced and there is nothing to attest a check against. And a module carrying no pharm_variants.csv is not a skip at all — the check does not apply, there is no claim to have an opinion about, and recording one would mine a nonce and create a verification.json on a module that never asked for one. nothing_to_check stays for a table that is present with no row in scope, which is a real answer. RM72's two commands follow that line exactly: no variants.csv attests nothing (acmg_sf, gene and trait_efo_id are all columns of it, so the check does not apply), while a failed lookup attests unreachable — the run with no report to print is the one where a reader most needs the record. check-acmg distinguishes the two ways it can raise for that purpose: AcmgListUnavailable carries the skip member (unreachable for a request that never answered, no_reference for a source that was there and carried no readable list), while a strict refusal stays the plain exception and attests nothing — the list was read and the question was answered, so recording a skip would say it was never put. When an attestation fails on the way out of an already-failing run it is reported, not raised: the reason the source could not be read is the sentence the author needs, and a layout complaint replacing it points at the wrong problem.
  • Two authorities answering one check still make one record, and the counts have to follow. allele_function is compared against PharmVar and CPIC, so three things differ from the single-source passes. subjects is the alleles an authority named back, never the authored rows: a curator may state a function for an allele a point-in-time slice does not list, and counting that row would publish a comparison nobody put (the shortfall is named in detail instead). findings counts alleles in dispute, not conflicts — one authored status contradicted by both panels is one allele reported twice, and VerificationRecord rightly refuses findings > subjects. And source is filled only when exactly one authority is implicated, because it is a single join key into the licensing table: naming one of two would hide the other, and a comma-joined value would break the join the column exists for. With no leg answering, the reason is picked by precedence (not_permitted → offline → unreachable → not_requested) and the sentence names every absent leg — a licensing refusal is cleared by a declaration, so reporting it as offline would send a reader hunting a network problem that does not exist.
  • A check whose input has no producer records the skip, not the pass's own numbers. vrs mint ends with a number out of a number, and that number is coverage, not a comparison: vrs_allele_id names the cross-check of a source's own ga4gh:VA.… against the locally minted one, whose input is mint_resolution_rows(source_ids=…) — and resolution.csv records ids without recording where an id came from, so no caller in the workspace fills that map. The command therefore records nothing_to_check with the coverage in detail, grouped by reason class. Recording the alleles it named as subjects would assert a comparison that was never made, which is exactly the trap the reference-allele pass fell into on an unbuilt assembly. (The coverage numbers are published already, by the compiler, as manifest.compilation.vrs_alleles / vrs_alleles_identified — this record is not their second home.)
  • A check with no lookup of its own still needs a reason ladder. The rsid↔coordinate pass costs no request: it widens the rsID batch the resolver chain was already sending and compares the authored pairs against what came back (resolver.check_rsid_coordinates, one direction — the converse needs a position→id lookup at chrom:start:ref granularity, and the reverse map enrich() holds is allele-exact, so asking it would report a spelling difference as a contradiction). The verdict is three-valued and compared at chrom:start: ref is optional on the PGx models — pgx_draft writes exactly that shape — and a ref that disagrees with the reference is the reference-allele check's finding, so including it would give one defect two names. A differing position is a contradiction only where every side is a substitution; an indel re-anchors legitimately (RM31), so that pair is undecided rather than reported. What the cheapness pays back is ways not to run: no row authoring both halves (nothing_to_check, tested first — a module with no pair has no assembly question either), a non-GRCh38 module (unsupported, since every resolver link is gated on the build), no snapshot opened (offline or no_reference), and a snapshot that settles none of the pairs — no_reference again, never ran(0, 0), because a record of a check that ran over nothing reads as a clean bill. Everything not compared stays outside the denominator and inside the record's detail, grouped by reason.
  • One proof-of-work per call, so a pass collects its records and writes once. enrich() writes all of its checks at the end of the run, and enrich_literature and check-identifiers do the same. The merge is what keeps every command in one document, replacing per check and never erasing a check this run did not put — and until literature was wired in there was no second writer at all, so the merge was machinery tested only against a document nothing produced.
  • …while the authored bytes stand still (RM423, a 0.8.x patch). Once they have moved (VerificationDoc.module_hash against the current binding), the earlier records are about rows that no longer exist, and the run keeps its own records only: carrying the others would have it sign them under the binding of bytes they never saw, a finding on a removed row or a findings=0 pass over rows the check never read. RM45's "not asked this time" is not "never asked" here, because about these bytes the check never was asked. record_verification logs one warning naming every check it drops. A record's own binding, which would keep the history, is a new field, parked as a minor (RM424).
  • A skip does not displace an answer the document already holds (RM72, 0.6) — while the authored bytes stand still. The merge was unconditional newest-wins, so check-acmg --offline after a real run, or check-identifiers --no-traits after a full one, rewrote a true subjects=13, findings=0 verdict to subjects=0, skipped=offline: an answer turned into "never asked" by a run that learned nothing. The argument is the merge's own, one step further out — a skip is a run that did not put the question, the same fact as an absent check spelled as a record instead of as a silence, so the protection that already covered the absent check covers this one. Newest-wins still holds between two records of the same disposition. This mattered more than it looked: every check wired after it merges through the same function, so wiring four more members into a rule that can silently downgrade them multiplies the defect by four. The condition is what keeps the fix from being a regression: an answer earns protection by still describing this module, so once the authored bytes have moved (VerificationDoc.module_hash against the current binding) a fresh "could not ask" wins — which is exactly what literature relies on when a module's citations change, and a stale finding no authored edit could clear is a defect this project has removed elsewhere. The counter-argument — that a reader may want to know today's run could not reach the source — is answered rather than dismissed: that is a fact about the run, and this is a per-check document, so it would need a run-level place, which is a different question and is not opened.
  • The merge carries the author's CLOSURE across, and only while the authored bytes stand still (RM73, 0.6). just-dna-compiler close writes a closure block into the same document, so this pass — which rebuilds it rather than editing it — would have dropped a field it did not know about, silently and on every run. Enrichment writes only derived sidecars, which are outside the binding, so the ordinary case is that nothing the author closed has changed and un-closing it would train an author to stop closing. Where the authored bytes have moved, the closure is dropped rather than re-bound: carrying it would have this pass assert a human declared these bytes final about bytes that human never saw. Only the author may make that claim again. This is the never-clobber trap one column over from SourceRow.dataset and draft_digest, and the rule generalizes — when another tier starts writing into a document this one rebuilds, ask what the rebuild silently discards.
  • A pass has ONE subject set, and the gate and the record both read it. literature is the worked example and it got this wrong three ways at once, all the same shape. literature.csv is the pin — merge-not-clobber means a re-run refetches nothing — so anything counted inside the fetch loop is a count of this run's requests: the strict gates saw an empty list and refused nothing while the attestation, reading the rows, reported the finding, so whether --strict refused depended on how many times --best-effort had run first, and the module could ship a manifest naming a defect its own strict compile had just blessed. The subject set is now the citations the module makes now, answered by the rows the sidecar holds now, and every number — CLI report, refusal, record — comes off it. Two consequences worth copying. A comparison the sidecar does not pin is re-made over every row rather than over the fetched ones (that is how an existing literature.csv came to hide every DOI/PMC id conflict). And the set is the cited rows, not the whole file: the sidecar keeps a row for a citation the author has since deleted, and counting it produced a finding no authored edit could clear, which this project treats as a defect wherever else it appears.
  • A no-op run records nothing; it does not record a zero. literature --offline fetches nothing and re-examines nothing, so on a module that already carries a literature.csv pinning the citations it makes now, it writes no records at all and the earlier ones stand: a run that has said nothing has nothing to record. With no pin covering those citations there is no answer this run could protect, so the three skips are written and say why — and after an online run that state is reached only by the citations having changed, which is the case the merge rule above deliberately lets a skip win. (This bullet used to argue the silence from the merge replacing per check. That was the merge's defect rather than an argument, and RM72 fixed it.)
  • A pmid=NONE citation is asked about by its DOI, and never read as "no citation" or "no quote" (0.8, RM336, RM349). Such a row cites by DOI alone, so _citations has no PubMed key for it; LiteratureResult.citations_without_pmid and quotes_without_pmid count those rows, and a module citing only by DOI is not refused as citing nothing. Online, each distinct DOI (spec.doi_key) is put to Crossref and its quotes to the text Europe PMC holds for it (The literature pack below), and the answers join both records' denominators: citation_existence counts a DOI with a verdict as a subject and one no agency knows as a finding (source="crossref" when every citation is DOI-only, never ran(0, 0)), and provenance_quote counts what a retrieved text settled. Each way of not having an answer is its own skip when it is all there is: not_requested (--no-doi, --no-fulltext), unreachable (Crossref never answered), no_reference (no text could be read), offline. unsupported is left for a row whose doi carries no DOI token, which a loaded NONE row cannot be.
  • literature puts three questions with three different denominators, and one record cannot carry them. Every citation gets an existence verdict; only some carry an authored DOI or PMC id to compare; fewer still carry a quote whose article could be read. So provenance_quote counts the quotes a retrieved text settled — never the quotes authored — with the remainder in detail, and it is a skip (no_reference) rather than subjects=0 when nothing was settled at all, since a zero out of zero on a check that could not run is exactly the clean-looking pass this whole item exists to stop. An abstract-only search settles a hit and nothing else. The remainder itself splits in two and the two must not be one sentence: an article whose text could not be read, and a quote nobody went looking for because the sidecar's pinned counts do not describe the quotes the module carries now. Only the second is cleared by deleting the sidecar, and calling it a retrieval failure is flatly false for an open-access article that was read on the previous run. The pinned count has to match, not merely be non-zero — a row pinned at two quotes says nothing attributable about the one that is left, so the whole row goes unexamined rather than carrying a finding about a quote the module no longer makes. Same rule on the identifier check: a citation drops out of the comparison because PubMed named no identifier or because a curator wrote that row, and reporting the second as the first is a claim about PubMed nobody established. And a DOI has three states rather than two: resolved, a pinned verdict about a DOI the module no longer cites (LiteratureRow.doi_checked is what makes that visible — without it, correcting a bad DOI made the finding follow the correction), and one no run ever put the question for, which is what a --no-doi run followed by a plain one leaves behind. Both ride in detail rather than shrinking the denominator, since the PubMed half of those citations really was answered.
  • The attestation is bound to the module's authored bytes. Edit variants.csv afterwards and the compiler drops the block with a warning — correctly, because the checks were put against rows that no longer exist. Re-running the pass re-attests. Currency of the source is a different question and is read off each record's own release.

Which of these attest, and which are recording passes rather than checks. Every member of VALID_VERIFICATION_CHECKS either has an emitter or says RESERVED beside itself, and test_verification_record.py walks the vocabulary and asserts exactly that — no total is stated here, because the two that used to be were wrong twice each and a number in prose is a registry nothing iterates.

Nor is a per-command one, and that is RM243's repair rather than a stylistic preference. This paragraph used to name four commands with a count beside each. They summed to 17 of the 24 emitting members — in the paragraph that had just refused to state a total, on the grounds that a number in prose is a registry nothing iterates. Two of the four were stale in the same way: enrich() gained published_refutation and evidence_status_currency, check-identifiers gained pgs_accession_currency and pgs_metadata_agreement with the PGS Catalog (RM163), and no addition moved a figure five lines up. Three commands — clinpgx check-labels, litvar coverage, alphagenome check — were never in the sentence at all. The figures are not restated here even as history, because a stale number in quotation marks beside the rule reads, to a skimming reader, exactly like the rule (RM218 learned that from its own repair).

So the attribution lives in the check table above, where it was already richer than the sentence, and test_enricher_doc_registries.py walks it: every emitting member has a row, no row names a member the vocabulary does not hold, no reserved member is shown as attested, and each row's Where cell names a module that exists. Note what that column is — the site of the comparison, which is often not the module that writes the record: reference_allele is compared in sequences and attested by enrich. Where a command rather than a function is the useful pointer, the cell says so in prose. What this paragraph keeps is the part no walk can say: why a pass is or is not in the vocabulary. The last four members were wired in RM72 (0.6): the two check commands put a real authored-versus-source question, reported it to stdout, and let the record die with the process, which is the sentence RM45 exists to end. What blocked them was their own printed promise to write nothing; that promise is now the narrower and truer one — writes no authored cell, and records that the question was put — because there is no other home for was this check run. It cannot be offloaded downstream (a consumer holding the artifact cannot tell "asked and clean" from "never asked", which is the whole of RM45) and it cannot go in the authored section (an author does not attest to a check they did not run). The record is unconditional, with no --attest flag: an optional record is ambiguous between the check was not run and it ran without the flag, which reintroduces the two-readings-of-one-absence defect the skip vocabulary was built to end. Filling acmg_sf or a gene symbol from the registry being asked about it stays refused whatever the attestation does — that is hints.REDUNDANCY_BEARING, and it is why these are open-ended checks rather than an apply route. The two remaining members are reserved, and each says so in the code (vocab.VALID_VERIFICATION_CHECKS): gene_disease_validity and dosage_sensitivity have no emitter because enrich_gene_validity and enrich_dosage_sensitivity record ClinGen/GenCC verdicts into a derived table and compare nothing authored. Wiring them would report a check where no question was put. The line that decides whether a pass belongs in VALID_VERIFICATION_CHECKS at all is whether it compares something the module asserts — so gene_validity.csv and clinical_assertions.csv, which record what ClinGen and ClinVar say and adjudicate nothing, have no check name and must not gain one: a member for them would let a manifest report a check where no question was put. frequencies.csv, gene_metrics.csv and the per-article licence columns are the same class. The three rows in the table above that are recording rather than comparing — Ambiguous back-fill, Article licence and Source coverage — are kept here because a reader wants the whole surface in one place, and they are named as the exception rather than left to be inferred.

Two of these break the severity rule in opposite directions, and both are deliberate. The allele-function check joins the clinical cross-check in warning under strict too: PharmVar and CPIC are different expert panels — one assigns a molecular function, the other a clinical one — and they genuinely disagree about some alleles, so failing a compile would make the format arbitrate between the two authorities it depends on. The declared-use gate goes the other way and refuses in both modes, because it is not a finding about the data at all: it is a statement that the fetch is not permitted, and best_effort means "resolve what you can", never "take what you may not".

The clinical cross-check warns in strict too. Every other check compares an authored value against a fact — the genome's bases, a deterministic digest, a registry's own id — where the source is simply right. A clin_sig disagreement is two opinions differing, and ClinVar is not truth: a curator who has read the primary literature and disagrees with a one-star submission is doing their job. Failing the compile would make the format arbitrate a clinical dispute, which the data-agnostic charter forbids. The finding therefore carries ClinVar's review-star count so a reader can weigh it themselves.

The division of labour with the compiler is a consequence of Principle 2, not a coincidence: a check that can be settled by computation over injected data belongs in the compiler (it runs on every compile, offline, and cannot be bypassed), while a check that needs a reference can only live here. ref validation is the clean example — the compiler can catch two rows contradicting each other about a reference base, and only the enricher can catch a row contradicting the genome. See COMPILER.md § what the compiler can and cannot validate for the full division, including the blind spots neither tier can close.

The dependency arrow points inward — enricher → compiler → format — so httpx / tenacity / huggingface_hub never enter the compile path. just-dna-format and just-dna-compiler stay strictly inject-only (CONSTITUTION Principle 2). HuggingFace is permitted only here (the 0.5 amendment scopes the Non-goal HF ban to format + compiler); the compiler reaches this package solely through a guarded lazy import on its deprecated ensembl_cache path — it declares no dependency on the enricher.

Companion docs: COMPILER.md (what consumes resolution.csv), SCHEMAS.md (ResolutionRow and the three hashes), CONSTITUTION.md (the 0.5 amendment). Import from the submodule where a symbol lives; __init__.py has no re-exports.

Read beside this: the 2026-09-11 code-first re-derivation

A second reading of this tier, written from the code alone on 2026-09-11, is in audit/ENRICHER_FROM_CODE.md — every command counted from --help, the resolver chain with what --offline changes per pass, every cache lane and every client's exception contract. The 2026-08-18 round found RM97–RM100 (0.6.1); this one found RM208 and RM209. Evidence, not contract: this document is the maintained one. The method is BLIND_REDERIVATION.md.

Install

pip install just-dna-enricher          # runtime: enrich (caches, live Ensembl, live gnomAD, VRS minting)
pip install 'just-dna-enricher[atlas]' # + the AlphaGenome Atlas client (grpcio + protobuf)
pip install 'just-dna-enricher[dev]'   # + publisher surface (module/reference upload), snapshot builders (polars), tests

[atlas] is two packages, and it is checkout-only today (RM192, RM196). The AlphaGenome Atlas serves precomputed variant scores over gRPC; its score fields are plain bytes and its request filter is an AIP-160 string, so grpcio + protobuf reach every RPC and struct.unpack from the standard library decodes the payload — 19 MB and +2 packages, measured in a clean venv on 2026-09-10, against 550 MB and 47 packages for uv add alphagenome (measured 2026-09-11). The bindings are generated from the Apache-2.0 .proto sources, which since RM196 are not vendored: the repository carries a commit id and a sha256 per file, atlas generate fetches and verifies them, and a file that does not match its pin is refused rather than used. docs/vendor/alphagenome_protos/ is where the copy used to be and now holds only a README saying so (RM221 — this sentence still said "vendored in" five weeks after the sources left). Run just-dna-enricher atlas generate once per checkout (it needs grpcio-tools, which is in [dev], not in [atlas] — the runtime imports the bindings without it). An installed wheel carries neither the sources nor the generated tree, so the client's import is guarded and says so; RM196 is where that trade gets decided.

ga4gh.vrs is a core dependency, not an extra. Minting a substitution's VRS allele id is stdlib and lives in the format tier, but justifying an indel needs the reference sequence — and reading sequence is network access, which is this tier's whole job. The plan for this work budgeted for ga4gh.vrs[extras] (seqrepo + pysam + hgvs: a compiled extension plus a multi-gigabyte local sequence store) on the assumption that a local seqrepo was required. Probing showed it is not — core ga4gh.vrs with the seqrepo REST data proxy normalizes indels over HTTP for 14 pure-Python packages and no compiled dependencies. At that price there is no reason to make complete allele identity opt-in, so it is the default and --offline is the only thing that turns it off.

Requires Python >=3.13 (the compiler's floor — deliberately not ensembl-mcp's >=3.14; the query core was ported, not depended on, dropping fastmcp/eliot). In the workspace: uv sync --package just-dna-enricher --group dev.

Module map

Module Role Notable deps
enrich A package since RM260 — __init__ re-exports every name enrich.py carried; the code is in subjects, outcome, build_declaration, orchestration, verification_records and resolution_csv. Orchestration: enrich() runs the resolver chain, writes resolution.csv compiler _load_csv_rows, format ResolutionRow
transaction 0.7 (RM128): the run's durability — staged answers beside the target, the advisory flock over the read-modify-write window, and the (done, total) progress unit format atomic_writer, stdlib fcntl
resolver the DuckDB rsid↔coord resolver (moved from the compiler in 0.5) duckdb, format
events 0.8 (RM375): the one tri-state judge of whether two spellings of a position name one event (same_event, names_event; covers, one direction, since RM394), asked by the placement check, the rsid↔coordinate check, --rederive's respelling test and the authored_event stamp sequences, format
clinvar the DuckDB ClinVar resolver link (lookup_loci) + the annotation reader (lookup_clin_sig) duckdb, format
clinical the clin_sig cross-check over the ClinVar snapshot (offline, reports only) format
concordance 0.7 (RM130): the other half of that check — the two verdicts, the per-authority calls behind them, and the writer that puts clin_sig_concordance.csv + clin_sig_authority_calls.csv beside the spec format concordance, compiler load_csv_rows
clin_sig 0.7: the one raw-significance → VALID_CLIN_SIG normalizer, shared by every source that reports one. Dependency-free on purpose, so a runtime pass reads it without the [dev] extra format VALID_CLIN_SIG
net shared HTTP politeness: PacingGate, batched, dedupe, attempt_floor, and stream_to_file — the one body every bulk download in this package uses, atomic, translated and retried (RM187), with a 120 s timeout by default (RM421) httpx, tenacity
eutils NCBI E-utilities client (esummary), shared by the literature and rsID checks httpx, tenacity
literature pass 4: a module's citations (studies.csv + binning pmids) → literature.csv (PubMed + Europe PMC), fulltext quote match, per-article licence, PMCID→PMID httpx, tenacity
identifiers rsID / trait-CURIE / gene-symbol / PGS-accession currency (dbSNP, OLS4, HGNC, PGS Catalog) httpx, tenacity
currency RM85: has the source a module was drafted from published since? Reads SourceRow.dataset against what the source says now — which release, not whether one judgement moved the per-source release records
provenance RM73: telling a value still copied from a source from one a human has edited — the axis every tautology skip reads compiler draft, format
verification RM45: the load-merge-write that records what each pass checked into verification.json, so the manifest can say it format verification
verdict 0.7 (RM235): a gate's yes/no with the reason codes that made it a no — the one place the house tri-state is deliberately not used, because --strict must choose an exit code —
producers 0.7: which pass fills each machine-produced table, from which source, checking what — the derived counterpart to drafting.DRAFT_PROVIDERS, and what lets the docs site's per-table reference name a writer instead of hand-keeping one licensing
pgs 0.7 (RM163): the PGS Catalog REST client, its release record, and the per-score shape identifiers compares against httpx, tenacity
licensing per-source terms + the declared-use gate; emits SourceRow format SourceRow
clingen ClinGen dosage sensitivity → gene_metrics.csv rows (CC0, so a module stays sellable) httpx, format
gene_validity RM24: curated gene–disease assertions → gene_validity.csv (ClinGen expert panels, GenCC's aggregate; both CC0) httpx, format
assertions RM25: resolution.csv + the ClinVar snapshot → clinical_assertions.csv (the call and the review tier) duckdb via clinvar, format
gwas RM90: the GWAS Catalog REST API (v2 since RM354) → gwas_effects.csv (published effect sizes with their units). Fills no weight httpx, format
expression RM194/RM200: the Atlas ListDenseVariantScores RNA_SEQ interval → expression_effects.csv (per-gene direction, with the distance beside it). Non-commercial, so an undeclared run writes nothing atlas_client, gene_spans, format
ld_pairs 0.8 (RM331): Ensembl REST pairwise LD → ld_pairs.csv, r² between each study's proxy_rsid and its lead in the study's one superpopulation. Online only httpx, tenacity, format
opentargets 0.8 (RM333/RM239/RM236): the Open Targets snapshot → locus_to_gene.csv, colocalisations.csv, consequences.csv, and the fine-mapping posterior on gwas_effects.csv. Reads only the snapshot; coordinates from resolution.csv duckdb, opentargets lane, format
gene_spans RM194: gene symbol → GRCh38 span, out of the operator-built MANE snapshot. A module with a plain name rather than a private helper, so the second caller can find it. RM259 added the reverse, genes_covering: a GRCh38 position → every gene whose span plus the 512 kb attribution horizon covers it, a query hint for a row with no authored gene, never an attribution (50 candidates at HFE H63D, 34 at APOE) polars, mane lane
atlas_client RM192: the AlphaGenome Atlas gRPC client — ListDenseVariantScores and the interval RPCs, on [atlas] (grpcio + protobuf, 19 MB measured) rather than the 550 MB upstream wheel grpcio, protobuf (extra)
atlas_protos RM192/RM196: fetch the Apache-2.0 .proto sources at a pinned commit and generate the bindings. The repository carries the pin — a commit id and a sha256 per file — and neither the sources nor the generated code grpc_tools (extra)
alphagenome_check RM193: the Atlas as a resolver, for the three questions the local AVI artifact provably cannot answer. Attests variant_impact_agreement; reports, never repairs atlas_client, format
alphagenome_avi_build RM191/RM197/RM198 builder: the 88.5 GB AVI artifact the operator already holds → the alphagenome_avi lane. --input is required and has no default — acquisition is the operator's act polars (lazy)
pgx_draft the first drafting provider: CPIC → haplotypes/allele_function/diplotypes rows cpic, compiler draft
clinpgx_draft RM26: ClinPGx snapshot → pharm_variants.csv rows (offline, inject-only) clinpgx, compiler draft
drafting RM228: the shared drafting scaffold every *_draft.py goes through — the DRAFT_PROVIDERS registry, the derived identity verdict (constructing the row model, because authoring_requirements' any_of grammar cannot express ref/alts require a position), a provider's own SourcePrecondition with its reason as a field, and the licence-row/stale-dataset pair that two providers had implemented as one half each compiler draft, licensing
clinvar_draft RM26: ClinVar snapshot → variants.csv partial rows; genotype left to a human clinvar, compiler draft
pubmind RM134 § B: the runtime reader over the snapshot pubmind_build writes — duckdb, core install, fetches nothing duckdb
pubmind_draft RM134 § C: PubMind snapshot → variants.csv rows, as draft-panel --source pubmind rather than a second command pubmind, compiler draft
clingen_allele RM153: the ClinGen Allele Registry — a CAID resolved to an identity this format can carry. The route that places a GRCh37-only row without lifting anything over httpx, tenacity
grch37 RM48: the old assembly, online half — rs-number recovery and the wrong-build diagnosis over a ref-mismatched row. Warns in both modes httpx, ensembl
acmg RM72: the acmg_sf check against the published secondary-findings list; the scraped v3.2 page reports unverifiable httpx, format
acmg_build [dev] builder: ACMG's published v3.3 workbook → the acmg lane, because NCBI's adaptation is a year behind polars (lazy), httpx
civic_build [dev] builder (RM152): a dated CIViC bulk release → snapshot parquet + release.json. CC0, so unlike PubMind this one may be published polars (lazy), httpx
civic_vcf RM169: the dated accepted_and_submitted VCF beside the TSV pair — the submitted half, pinnable and byte-reproducible, with no API read polars (lazy)
civic_api RM160: CIViC's GraphQL API — the one surface carrying evidence attached to a variant no dated file can place, because a VCF record needs a POS httpx, tenacity
civic_identities RM159: identities CIViC states in a variant's name and never in its identifier columns (N150fs (c.448delA), IVS2+1G>A) clingen_allele, ensembl
civic_citations RM160: the citations those variants carry, and the canary that says when the dated file has caught up civic_api, format
civic_draft RM152: CIViC snapshot → direction-axis rows. The axis is direction, not clin_sig — the measurement refused the latter polars, compiler draft
civic_refutation RM170: an authored direction over a variant CIViC has published a refutation of — the pair contested_variants cannot see. Warns in both modes polars, format
mitomap RM171: MITOMAP's published pg_dump reader, and the two-token grammar its status column is written in duckdb
mitomap_build RM171 builder: the 63 MB gzipped SQL dump → snapshot parquet + release.json. CC BY 3.0 with commercial use stated free, so a deployment may publish it polars (lazy), httpx
mitomap_miss_build RM171: the derived lane — MITOMAP minus the ClinVar cache, recomputed from both parents rather than frozen as a diff polars (lazy), mitomap + clinvar lanes
clinvar_gkm 0.8 (RM305): ClinGen's ClinVar-GKM as an indel-placement witness: the manifest/index client, the release label, the three-arm lane_currency, and ClinVarGkmWitness, the lookup by VRS id, VCF spelling or rsID a check calls. Runs no check of its own httpx, tenacity, polars
clinvar_gkm_build RM305 builder: allele.parquet + variation.parquet downloaded whole and verified against the set's manifest.json, cut to the GRCh38 net insertions and deletions, joined to ClinVar variation and rsID polars (lazy), httpx
_rcrs RM273: the revised Cambridge Reference Sequence (NC_012920.1, GRCh38's chrM) as a vendored constant, so mitomap_miss_build left-aligns both sides' indels without a fetch none
mitomap_draft RM171: the rated misses → variants.csv + studies.csv. Only the rated ones; the photocopies and VUS* are each refused for their own stated reason mitomap_miss, compiler draft
clinvar_build [dev]: VCF → snapshot parquet; var_citations.txt → citations/ (+ its own release.json block) polars, httpx
lookup authoring lookups — rsID validity/loci, ref/alts + populations, which paper a PMID names. Writes nothing every client above, compiler hints
pharmvar star-allele definitions + function (Api-Key header, 2 rps) httpx, tenacity
cpic allele function, diplotype→phenotype, defining variants (PostgREST) httpx, tenacity
cpic_notation CPIC's DEL<bases>/INS<bases>/UNIT(n) → a VCF-anchored, left-normalized REF/ALT (RM253); never reads on its own, the caller passes the reference reader sequences
pgx pass 5: cross-check star-allele tables, write the licence table the three above
clinpgx_build [dev]: summaryAnnotations.zip → snapshot parquet + pinned LICENSE.txt; refuses the retired clinicalAnnotations.zip by name polars, httpx
clinpgx pass 6: evidence-level cross-check over the snapshot (offline) duckdb (core, not polars)
gnomad live gnomAD GraphQL: batched + paced rsid resolution, frequency, gene constraint httpx, tenacity
frequencies pass 2: resolution.csv → frequencies.csv (per-ancestry-group AC/AN) compiler load_csv_rows, format
gene_metrics pass 3: the module's genes → gene_metrics.csv (snapshot first, live API second) duckdb, format
constraint_build [dev] builder: gnomAD constraint TSV → gene-level parquet + release.json polars (lazy), httpx
vrs GA4GH VRS allele-id minting onto resolution.csv (substitutions stdlib, indels normalized) ga4gh.vrs
sequences reference-sequence access (cached), the reference-allele check, and the one left-alignment every pass uses (left_normalize, normalize_locus, spelling_verdict; RM270 writes a fill left-aligned with it) ga4gh.vrs
placement 0.8 (RM267): where the run's own link placed each rsID against where ClinVar and ClinVar-GKM place it, and, online, where Ensembl REST places a snapshot indel neither holds — by VRS id, else the left-aligned spelling; a disagreement withholds the rsID sequences, clinvar, clinvar_gkm
locations cache-location resolution — one resolve/default_dir/env_var triple per lane in CACHE_LANES, paired by a test rather than counted here — plus .env and the pull record. The snapshot layout names and read_release are imported from just_dna_format.snapshot since 0.8 (RM303) and stay importable from here platformdirs, python-dotenv
download HuggingFace snapshot download, footer-checked and atomic — one ensure_* per lane whose publish_repo is set, which is the registry's own answer to which lanes may be pulled rather than a second list. A lane with no repo is operator-built; see The caches huggingface_hub (lazy)
cpic_build [dev] builder (0.5.1): the whole CPIC PostgREST database → parquets + release.json polars (lazy), cpic
pharmvar_build [dev] builder (0.5.1): /genes → alleles + defining variants. Operator-built, never published polars (lazy), pharmvar
pubmind_build [dev] builder (0.7, RM134): the ANNOVAR-distributed PubMind table → one parquet + release.json. Operator-built, and pubmind publish refuses polars (lazy), httpx, clin_sig
mane_build [dev] builder (0.7, RM168): one pinned MANE release → three parquets + release.json. Operator-built; there is no mane publish because NCBI states a policy rather than a licence polars (lazy), httpx
opentargets_build [dev] builder (0.8, RM333): one pinned Platform release (26.09) mirrored from EBI's FTP over https → the flattened lane snapshot + release.json; subset_snapshot cuts a regional or test slice polars (lazy), httpx
strchive RM165: the STRchive repeat-locus catalogue — the repeat_alleles.csv band cross-check (offline, reports only) format binning, compiler load_csv_rows
strchive_build RM165 builder: download STRchive-loci.json + release.json. Core httpx only, no parquet and no [dev] extra httpx
strchive_draft RM165: STRchive → repeat_alleles.csv partial rows — identity only, bands left to a human strchive, compiler draft
ensembl live Ensembl REST (the beta GraphQL leg was retired on 2026-11-04), tenacity httpx, tenacity
upload publisher surface — push a compiled module or a reference snapshot to HF ([dev]) huggingface_hub (lazy)
litvar RM167: LitVar2/PubTator3 literature coverage per locus, with the tier that answered (allele node / position node / absent). Reports only; writes no row and no SourceRow httpx, tenacity, clingen_allele
drug_labels RM166: five regulators' drug labels — the (gene[, allele], drug) cross-check at two join tiers (offline, reports only, writes no SourceRow) duckdb, format pgx, compiler load_csv_rows
drug_labels_build [dev] builder (0.7, RM166): ClinPGx's drugLabels.zip → one parquet + LICENSE.txt + its own release.json, dated from the archive's own CREATED_*.txt polars (lazy), httpx
caches The cache registry (CACHE_LANES) and the rebuild adapters behind cache rebuild — one entry per lane carrying its three stages and, for each stage it lacks, the reason as a field. Walked by a test against the *_build modules on disk (RM176) —
cli A package since RM260 — __init__ assembles the app and re-exports every name cli.py carried, one *_commands submodule per command group. Typer app: the runtime commands (enrich, frequencies, gene-metrics, gene-validity, assertions, enrich-and-compile), the authoring ones (hint, draft-*, the check-* family), upload, cache status/pull/rebuild/prune, vrs mint, and one builder group per CACHE_LANES lane carrying a build_command — a publish exists exactly where the lane names a publish_repo, and pubmind publish is the one that exists in order to refuse with its reason. The per-command flag tables are --help and the dated listing in audit/; a hand-kept copy here would be the second thing to update typer

Rate limits (public APIs)

Every live client that can fire more than a handful of requests goes through net.PacingGate (injectable clock, pace-before-retry). Snapshot / one-shot downloads are listed too so the full egress surface is in one place. Published is what the service documents; our pace is what the client actually waits; when the source publishes nothing, the gate is a courtesy, not a claim that the ceiling is known.

PacingGate.spent is what the gate admitted (S95, RM203). A host metering egress per upstream had no number to meter: nothing downstream reported the calls actually made, so a proxy charged by the shape of a request, an upper bound that bills a caller for a call that never happened. Every egressing client waits on its gate once per attempt, inside its retry loop, so one increment is one upstream attempt — a 429 retried three times counts three, a snapshot hit counts nothing. Monotonic, bumped under the slot lock, never reset; a rate is two readings apart. It is the one thing this counter says; what the call cost stays the client's to know. Five clients keep their gate private as _gate (pgs, litvar, civic_api, pharmvar, clingen_allele), so there the reading is client._gate.spent; the rest expose client.gate.

One PacingGate is safe to share across threads, and that is now a stated contract rather than an accident of who happened to call it (S15). It matters because the injection API asks for sharing: LookupClients tells callers to hold a client and reuse it — a fresh one per question would discard exactly this state — so a server running blocking work through a thread pool ends up with several workers on one gate by following our own advice. Until 0.5.4 wait() read last, slept, then wrote it with no lock, so two workers could both find the interval elapsed, both skip the sleep, and turn a published 3/s budget into 6/s — a budget somebody else enforces by blocking the operator's IP. The lock covers the bookkeeping only: each caller reserves the next free slot and waits for it alone, so N callers get N slots spaced one interval apart and no worker is blocked by another's sleep.

NCBI's clients share one gate per process (RM418). NCBI limits E-utilities per key or per address, never per client, and the rsID check and the literature pass each built a client with a gate of its own, so two enrichments on two threads paced each to the full rate. An EutilsClient built without a gate now takes net._service_gate("ncbi-eutils", interval), one gate per (service, interval) in the process; a gate handed in is used as handed. The other clients still build a gate each, the two Ensembl REST ones included (RM419), and separate processes still pace apart (RM404). Single-threaded behaviour is unchanged, and test_net.py proves the spacing on a frozen clock without really sleeping. It is a pace, not a concurrency limit. If what you need is one request in flight per service, put a semaphore around your call site; the gate will not give you that.

Service Used by Published budget Our pace / batching Auth / identity
gnomAD GraphQL gnomad (resolve, frequencies, live constraint) 10 req / IP / 60 s min_request_interval=6.0 (exactly that budget); GraphQL alias batches of 20 (25 worked live; 29 → HTTP 400) ≈ 200 variants/min none
NCBI E-utilities eutils → literature + rsID currency, whose esummary db=snp record is also the dbSNP placement witness (RM398; one fetch serves both) 3 req/s without a key; 10 req/s with NCBI_API_KEY 1/3 s or 1/10 s from whether the key is present; esummary batches of 200 tool=just-dna-enricher; email from JUST_DNA_CONTACT_EMAIL when set; key optional
PharmVar pharmvar / pgx 2 req/s (OpenAPI “Limitations”) PHARMVAR_MIN_INTERVAL=0.5 Api-Key header from PHARMVAR_API_KEY (personal; never written to a module)
Crossref literature.CrossrefClient (DOI existence, and the work it names since RM262) polite pool: 10 req/s single-DOI (5 public); concurrency 3 polite / 1 public (Crossref docs, Dec 2025 revision) min_request_interval=0.1 (10/s — polite single-DOI ceiling) User-Agent just-dna-enricher (mailto:…) when JUST_DNA_CONTACT_EMAIL is set → polite pool; omitted rather than invented
Europe PMC literature.EuropePmcClient (OA fulltext + abstracts + per-article licence; a preprint's JATS through europepmc.org/api/fulltextRepo, RM349, on the same gate) no durable official figure on the developer pages (community reports vary) min_request_interval=0.5 (2/s), batches of 25 on search none
PMC ID converter literature.PmcIdConverterClient (PMCID → PMID, reporting only) no published figure; the service documents a 200-id batch ceiling min_request_interval=0.5 (2/s), batches of 200 tool=just-dna-enricher; email from JUST_DNA_CONTACT_EMAIL when set
OLS4 + HGNC identifiers.OntologyClient neither publishes a documented limit min_request_interval=0.2 (courtesy — GET-per-id, unbatched) Accept: application/json
PGS Catalog REST (www.pgscatalog.org/rest) pgs.PgsCatalogClient → identifier currency + the release probe none published _REQUEST_INTERVAL=0.2 (courtesy — GET-per-accession, unbatched; one extra request for /rest/info) Accept: application/json
LitVar2 / PubTator3 (ncbi.nlm.nih.gov/research/litvar2-api) litvar → literature coverage (0.7, RM167) NCBI asks for no more than 3 req/s without a key _REQUEST_INTERVAL=0.34 — that budget with a little air in it; a 388-locus corpus sweep ran at it without a refusal none
CIViC GraphQL (civicdb.org/api/graphql) civic_api → civic citations, evidence-status currency (0.7, RM160) none published _REQUEST_INTERVAL=0.34 (courtesy) none
ClinGen Allele Registry (reg.clinicalgenome.org/allele) clingen_allele → the identity-from-a-name procedure (0.7, RM153) none published _REQUEST_INTERVAL=0.1 (courtesy) none
Ensembl GRCh37 REST (grch37.rest.ensembl.org) grch37.diagnose_wrong_build (0.6), hint recover Ensembl publishes 15 req/s _REQUEST_INTERVAL=0.1 — keeps a whole panel inside it while still interactive none
ClinVar release probe currency.ClinVarReleaseClient — one bounded read of the VCF's header (HEADER_PROBE_BYTES) per --verify-datasets run (0.7) NCBI asks for no particular pacing on the FTP-over-HTTPS mirror CLINVAR_MIN_INTERVAL=0.5 — this tier paces every service it touches, and one probe per run is inside any budget none
Ensembl REST (rest.ensembl.org) ensembl (the live link); until RM398 also the placement witness (0.8, RM267), for each snapshot indel no loaded witness holds. The link asks 20 rsIDs per POST /variation/:species since RM401 (ensembl._REST_BATCH_SIZE, fixed in the 0.8 patch line; 0.9 makes it a setting with RM404), with the POST's own read timeout (_REST_BATCH_TIMEOUT, 120 s), and one GET for an rsID a batch leaves out 15 req/s per IP, ~55 000 / rolling hour; 429 + Retry-After / X-RateLimit-* _REQUEST_INTERVAL=0.1 (EnsemblSettings.interval), one PacingGate per resolver, waited once per attempt (since 2026-11-04); tenacity on transport, 5xx and 429, waiting at least the answer's Retry-After (net.wait_retry_after); one longer than net.RETRY_AFTER_CEILING (10 s) is not retried, and the resolver asks nothing until it passes none
CPIC (api.cpicpgx.org) cpic / pgx_draft unpublished no PacingGate — coarse PostgREST GETs (gene-scoped), not per-allele loops none
ClinPGx clinpgx / clinpgx_draft n/a at runtime offline snapshot only for the check/draft path — no live poll budget none (live API retired → snapshot)
seqrepo REST (services.genomicmedlab.org) sequences / VRS indel mint unpublished no PacingGate; in-process memo of window reads, one window per indel mint (RM407) none
GWAS Catalog REST v2 (www.ebi.ac.uk/gwas/rest/api/v2) gwas 15 queries a second, published on the v2 docs page ("Throttling", read 2026-11-11); an excess slows later calls rather than refusing them, unlike gnomAD's load-bearing 10/60s DEFAULT_REQUEST_INTERVAL=1.0 — a courtesy well under that budget, not a transcription of it. Cost is one request per page of associations plus one per distinct study (its ancestries); measured at 190 requests for hfe_hemochromatosis on v2 (11 saved by the cache), where the legacy API's 1 + 2N cost 382. --no-study-facts drops it to the pages alone none
AlphaGenome Atlas (gdmscience.googleapis.com) alphagenome expression, alphagenome check none published — the Additional Terms bar classes of holder outright rather than metering requests, which is an eligibility bar and not a budget measured ~1,091 SNVs/s end to end; a gene plus its ±512 kb flanks is ~3.3 M SNVs ≈ 50 min, which the pass prints before the query rather than after ALPHAGENOME_API_KEY
ClinGen dosage TSV clingen n/a (one file) single download, then local parse none
ClinGen gene-validity CSV gene_validity n/a (one file, ~1 MB) single download, then local parse none
GenCC submissions CSV gene_validity n/a (one file, ~28 MB) single download, then local parse; the client's timeout is 180 s because one response is the whole export none
ACMG SF list acmg n/a one HTML GET (~75 KB) or a local --sf-list workbook none
Hugging Face Hub download (snapshots), upload (modules / references) 5-minute fixed windows, three buckets — see below no custom gate; huggingface_hub handles 429 via RateLimit / RateLimit-Policy headers (smart retry in 1.2+) HF_TOKEN / hf auth login — anonymous shares a per-IP pool; a free token is the usual fix

Hugging Face Hub tiers (as of Sep 2025)

Quotas are per 5-minute window. Snapshot provisioning (ensure_*) and publisher upload both hit API (listing / commits) and Resolvers (/resolve/ file bytes). Pages is the website and is not on our path. Numbers marked * are subject to change with platform health.

Plan API Resolvers Pages
Anonymous (per IP) 500 * 3,000 * 100 *
Free user 1,000 * 5,000 * 200 *
PRO 2,500 12,000 400
Team org 3,000 20,000 400
Enterprise 6,000 50,000 600
Enterprise Plus 10,000 100,000 1,000
Enterprise Plus + org IP ranges 100,000 500,000 10,000
Academia Hub org 3,000 20,000 400

Org limits apply per member, not shared. Source of truth: huggingface.co/docs/hub/rate-limits. Always pass HF_TOKEN for snapshot download when possible — anonymous traffic is the usual cause of a “stuck” ensure_snapshot (the client is sleeping on a 429, not hanging).

Rules that follow from the table

  • Pace before retry. tenacity backs off on transport / 429, but a blind retry spends the same budget that caused the 429; every gated client waits first.
  • Reuse clients. A PacingGate is per-client state, except NCBI's since RM418 — constructing a fresh gnomAD client per lookup throws the interval away (lookup holds them for that reason).
  • Batch where the API allows it. gnomAD aliases (20) and NCBI esummary (200) exist because the published ceilings make one-id-per-request unusable; PharmVar is the opposite — 2 rps forces gene-scoped endpoints, never per-allele. Ensembl REST batches since RM401 (POST /variation/:species, EnsemblResolver._resolve_rsids_rest), and a batch answer is read by what the probe of 2026-11-07 showed it does: an rsID with no record is left out (an all-unknown batch is 200 {}), and a merged rsID is keyed by its current id with the asked one in synonyms (rs8192284 → rs2228145). So the asked rsID is matched by name, then by the one record whose synonyms name it, and an rsID still unmatched is asked by GET, whose 400 is the only "no record" (RM361). An omission is never a negative. A batch that fails, a POST 400 included, is could-not-ask for every rsID in it. A batch size of 1 is one GET per rsID, as before. The batching is private in the 0.8 patch line, so only EnsemblResolver batches and an injected resolver is asked one rsID at a time; 0.9 publishes the capability and the knobs with RM404.
  • No response cache for the live clients. NCBI / PharmVar / gnomAD GraphQL / Crossref / Europe PMC / OLS4 / HGNC / PGS Catalog / LitVar2 / the CIViC API / the ClinGen Allele Registry / GRCh37 Ensembl / the ClinVar release probe / live Ensembl are paced only. Persistence is the authored sidecars (resolution.csv, frequencies.csv, …) and the published snapshots cache pull fetches (the CACHE_LANES rows with a repo — see "The caches") — delete a sidecar to force a refetch. Note which entries in that list are also the licence-gated ones: CPIC and PharmVar are paced-only and forbid sale, so they are the two RM38 gives a snapshot (see On a host, or in a service below).
  • A shared IP shares one budget. Every figure above is per-IP or per-token, never per caller, so a hosted deployment multiplies its users onto one allowance rather than getting one each. This is a separate reason from licensing for reaching a source through a snapshot, and it applies to ungated sources too — it just happens that the gated ones are where both reasons land at once.
  • --offline clamps to local caches / sidecars; it does not invent a budget for a live API.
  • The retry ceiling is a floor a deployment can raise — $JUST_DNA_HTTP_RETRY_ATTEMPTS (RM42). Three attempts is right for the audience the CLI was written for: a person who would rather see a failure in ten seconds than wait out a flapping upstream. It is wrong for the other shape the 0.5 tiering created — a server running enrich() inside an unattended publish, where giving up on a transient 502 costs the publisher a whole re-upload rather than ten seconds. Two callers wanting opposite things from one constant is a knob, and it was an import-time decorator argument with no parameter, so a consumer's only route was to walk the package and reassign policy.stop.

net.attempt_floor(n) resolves per call. It raises each client to at least the configured value and never lowers one, so the deliberate per-client tuning survives — gnomAD and eutils sit at 4 because their budgets are tightest, and a value that set every client would flatten that. Safe to raise precisely because every gated client paces before it retries: an extra attempt spends a slot of the published budget rather than bursting past it. Only a bare stop_after_attempt is replaced — a composed stop_after_attempt(3) | stop_after_delay(60) means both, and raising one term would silently change a policy whose author meant the conjunction. None is composed today — and the number of policies is deliberately not stated here: net.py deleted "the nine" from its own comments after the tree carried twelve, because a count in prose is a registry nothing iterates.

The Atlas client reads it too since RM280, and it had to lose grpc's retry to do so. Its channel used to retry under the vendored grpc_service_config.json, which grpc caps at five attempts, the number upstream already asks for, so the variable could not raise it. A tenacity layer on top would have made 5 × N attempts instead of a floor. atlas_client.channel_options() now switches grpc's retry off (grpc.enable_retries = 0, and no retryPolicy in the config it hands the channel), and AtlasClient._call retries under attempt_floor(5) with upstream's backoff. What it retries is derived from _translate: every status that becomes AtlasUnavailable, which adds INTERNAL to upstream's three. Its pacing gate has a zero interval, on a measurement (RM307): no throttle appeared up to 21 calls/s (ALPHAGENOME_ATLAS.md § 6.7), so the "paces before it retries" reasoning holds trivially. The gate is still waited on once per attempt, so spent meters Atlas calls, and connect(gate=…) takes a shared one with a real interval. - Every retry says so, and every give-up says how long (RM403). Each policy names net._log_retry as its before_sleep and net._log_give_up as its retry_error_callback, so the just_dna_enricher.net logger writes, at WARNING:

EnsemblResolver._rest_rsid(rs1801133): attempt 1 of 3 failed (ReadTimeout: The read operation timed out); retrying in 0.6 s
EnsemblResolver._rest_rsid(rs1801133): gave up after 3 attempt(s) over 46.2 s (ReadTimeout: The read operation timed out)

The ceiling is the one the stop reads, so a raised $JUST_DNA_HTTP_RETRY_ATTEMPTS shows in it. The give-up line precedes the caller's own warning, which names the error and what the run did about it. An error a policy does not retry (a 400, a 404) writes neither line. WARNING because nothing in these tiers configures logging, and Python's last-resort handler drops anything lower. An attempt that timed out used to write nothing, so a sweep's timeouts read as a slow server (S133). test_retry_logging.py walks the package and refuses a retry( that names neither hook.

The author's overlay, read but never written (RM136, 0.7)

The compiler applies overrides.csv before any check reads a row — a check must report on what the module asserts. Until 0.7 this tier did not, so an author who corrected a resolution.csv cell through the overlay went on being told the same finding on every run, with no way to clear it and nothing saying the correction had been recorded and honoured one tier over.

Three rules, and each has a refused alternative worth knowing:

  • Read-only. The enricher never writes through the overlay. An overlay row is the author's answer to a difference, never this tier's (RM83).
  • Input reads only. licensing.overlaid_input_rows is called where a pass reads a derived table as an input to something else — frequencies, assertions, and identifiers' gene-locus check all read resolution.csv that way. Every merge baseline stays raw: a pass that reads its own output file to merge against it writes that file back, and post-overlay rows would bake the correction into the derived table. Read the file you write, and write what you read.
  • Per field. licensing.overlay_answers(spec_dir, table) returns the (subject, field) pairs an update names. A finding is answered when the overlay corrects the very cell the finding is about, so a coordinate correction silences the coordinate check and leaves an unrelated clin_sig finding standing. Per row was cheaper and was refused — it would silence findings the author never looked at, which is the silent-suppress hole the overlay's own design calls its worst case.

There is no second implementation of the overlay, deliberately: compiler.load_overlay is public for this, and overlaid_input_rows calls the same apply_overrides the compiler calls. Two copies would drift on the normalization seam that produced a silent P7 break in this feature's first week, and a test compares the helper's output against apply_overrides directly so a future copy would show.

Answered is not agreed. The rsid↔coordinate check is the one wired to this today, and an answered pair leaves disagreements while staying inside subjects, counted by PairCheck.answered and reported as one INFO line naming what happened. The comparison ran and found a difference; removing it from the denominator would publish a cleaner module than there is. Only update counts — an insert answers no prior finding, and a suppress is already reported in its own right by RM131.

Wiring a further check to the answered set is deliberate rather than automatic, because which cells does this comparison read is a per-check fact (resolver._COORDINATE_FIELDS is the first one written down) and guessing it wrong silences a finding nobody answered.

A second answered-set reader arrived with RM151, and it reads the overlay the other way round. licensing.overlay_answered_subjects(spec_dir, table) returns the (subject, member) pairs the overlay names — every operation and every field, which is the opposite rule from overlay_answers above and is right for the opposite direction. overlay_answers decides whether a finding may be silenced, so it insists the overlay touched the very cell the finding is about; anything looser silences findings the author never looked at. overlay_answered_subjects raises a finding, and what goes stale there is the reason — mandatory on every overlay row whatever the row does — so a per-field rule would have to name a column the reason does not live in. A finding raised too widely costs a reader one line; one silenced too widely costs them the finding.

What --offline promises, and the one axis the two readings differ on (RM220)

--offline means no egress. Every pass that takes it turns into a no-op with a warning rather than a failure where it has no snapshot to fall back on, and the warning says which of the two it is (@unreachable-not-absent — nobody-asked is not asked-and-absent).

An injected client is where the tier held two readings, and the axis is the source's licence.

reading passes why
offline outranks an injected client pgx, enrich, frequencies, expression the source is licence-gated. A live client under a flag documented as making no egress is exactly the loophole RM38 closed — test_pgx_licensing.py asserts it by name, and the AlphaGenome Atlas joined that column in RM220 because its Additional Terms bar classes of holder outright
an injected client wins gwas the GWAS Catalog is ungated, and handing over a transport you already hold is not egress. Stated in enrich_gwas's own docstring, deliberately

Neither is wrong and the difference was nowhere written down, which is the defect RM220 actually repaired: expression had gwas's shape against pgx's situation, so an injected client fetched from a gated source under --offline. @flag-means-same is the rule, and this table is what makes the flag mean the same thing given the licence, rather than differing by which module you happened to call.

gwas keeps its behaviour on purpose. Changing a contract its docstring states, for the one ungated source, is a decision rather than a repair — recorded here so the next reader meets a written rule instead of an apparent inconsistency.

Injecting text is a different thing entirely and is not part of this: clingen, gene_validity and acmg accept an already-downloaded export, which cannot egress by construction. That is the documented "inject what you already hold" escape.

enrich() — the resolver chain

enrich(spec_dir, *, mode="best_effort", offline=False, ensembl_cache=None,
       clinvar_cache=None, use_clinvar=True, use_gnomad=True, download=True,
       genome_build=None, write=True, mint_vrs=True,
       verify_ref=True, verify_clinsig=True, verify_rsids=True, verify_datasets=True,
       keep_par_twin=False, rederive=False, keep_staging=False, progress=None,
       resolver=None, gnomad_client=None, grch37_client=None,
       release_probes=None) -> EnrichmentResult

genome_build=None means "read the module's declaration" (spec_genome_build), not "assume GRCh38". It defaulted to the literal "GRCh38" and no caller ever passed anything else, which made every genome_build == "GRCh38" gate below — and the warning saying a non-GRCh38 module resolves nothing — unreachable: a genome_build: GRCh37 module was resolved against GRCh38 Ensembl and the GRCh38 coordinate written into its resolution.csv under the label GRCh38, silently. Enrichment is GRCh38-bound (RM15), so for any other build it now warns, runs no link, and records no lookup result — not even not_found, which would claim the source was asked. Authored coordinates are still transcribed verbatim, under the module's own build. An explicit value stays the inject-only override.

Reads every table that can ask for a coordinate — variants.csv, pharm_variants.csv and haplotypes.csv — computes which rows still need work (need_pos = rsid but no coord; need_rsid = coord but no rsid), runs a first-hit-wins chain, and writes/merges resolution.csv (sorted by (variant_key, locus_index)), stamping each row's source/status. The chain:

  1. Existing / human rows — a resolution.csv already beside the spec is authoritative and never clobbered; a variant_key it already covers is skipped by the chain.
  2. Local cache (offline) — locations.resolve_ensembl_reference locates a cache; resolver.lookup_loci runs the DuckDB lookups (rsid → [loci], pos → rsid). Source cache.
  3. HF snapshot — if no cache is present and not offline/download=False, download.ensure_snapshot fetches the parquet slice (populates the cache read in step 2). The snapshot is a static slice of popular rsIDs, not a canonical reference — same pains as any source (incompleteness, versions, reachability), so a miss falls through.
  4. ClinVar cache (offline, use_clinvar) — for the variants the Ensembl cache/snapshot missed, clinvar.lookup_loci runs the same DuckDB lookups over the ClinVar snapshot (locations.resolve_clinvar_reference, or download.ensure_clinvar_snapshot when absent and online). Source clinvar. It sits after the Ensembl cache on purpose: alts is a resolution fact (it flows into weights.parquet → artifact.digest), and ClinVar carries only its submitted alleles while Ensembl carries every dbSNP allele — so a variant both caches know keeps the Ensembl alts and source="cache", and no already-compiled module's artifact.digest moves. ClinVar is a complementary reference (4.4M clinically-curated records, 1.54M with no rsid) that makes an offline clinical enrich possible without provisioning the 14 GB dbSNP cache.
  5. Live Ensembl — for rsIDs every cache missed, ensembl.EnsemblResolver.resolve_rsid (REST). Source ensembl-rest; a row written before 0.8 may carry ensembl-graphql, the retired leg's. It has four outcomes: loci, an answered [], (None, None) for could-not-ask, and (None, source) for an answer whose one-sided indels could not be anchored (RM268) — see Live Ensembl below, and note that neither None leaves any resolution.csv row at all.
  6. Live gnomAD (use_gnomad) — the last link, for whatever nothing else could resolve. gnomad.GnomadClient.resolve_rsids batches the leftovers. Source gnomad. It goes last for exactly the reason ClinVar goes after the Ensembl cache: gnomAD reports only the alleles observed in gnomAD, not every allele dbSNP knows, so promoting it would narrow some already-compiled module's alts and move its artifact.digest. Last place makes the link strictly additive — it can only ever add variants nothing else had. A failure here is logged and skipped, never fatal.

A rejected answer is a miss (RM382, 0.8). "First hit" means the first answer a subject can use. Each link's answer passes through the allele-aware filter below as it arrives, and an answer that every subject of the rsID rejects (every locus, by the same predicate the table's writer uses), where at least one subject is rsID-only, is set aside, so the next link is asked. Measured on pathogenic_clinvar: the split Ensembl snapshot's only locus for HBB rs613985 is 3:159434329 T>G, which the authored 13 bp deletion cannot sit on, and the ClinVar link now places it at 11:5226724 CATAACAGCATCAG>C. The digest argument for the order is untouched: a variant some subject can host on the Ensembl answer still keeps it, so only rsIDs that used to resolve to nothing change. The decision is per rsID, never per subject, because the placement comparison below takes one authority per rsID; an rsID one subject hosts and another rejects keeps the first answer, and the rejecting subject is reported as below. A pair (rsID and coordinate authored) is never a reason to ask on, since no link is asked live for a pair, but a pair that can host the answer keeps it; the rsid↔coordinate check still compares a pair with the snapshot's own answer. EnrichmentResult.superseded_answers names each set-aside answer and the link that placed the rsID after it (AlleleMismatch.links, placed_by). Under --rederive a recorded placement is kept when this run's links answered only at loci nothing can host (it came from a link this run may not have reached), and a recorded not_found row is dropped.

After the chain settles, vrs.mint_resolution_rows stamps vrs_id/vrs_spec onto every mintable row (mint_vrs=True). An existing id is never overwritten.

The run is a transaction (0.7, RM128)

enrich() used to persist nothing until its tail: a single if write: block at the bottom, everything above it in memory, so a run killed at minute 29 had written zero bytes and the thirty minutes were gone (S66). The obvious repair — checkpoint the table as it goes — trades away a property somebody may be relying on, that a refused strict run leaves the module exactly as it was. The trade is unnecessary. The run is a transaction, which keeps the promise absolutely and recovers the work.

  • Staged answers, in the target's own directory. As each live link answers, the raw answer (rsid → [locus], plus which link said so) is written to .<name>.staging/answers.csv beside the table — transaction.ResolutionJournal, transaction.staging_dir_for. Same directory is the correctness condition, not a convenience: os.replace is atomic only within one filesystem, and shutil.move across a partition degrades to copy-then-delete. Staging beside the target makes a cross-device move structurally impossible rather than merely avoided, and a test asserts the sibling relationship rather than trusting the prose.
  • What is staged is the answer, never the row. Everything downstream of an answer — the allele-aware hosting filter, the pseudoautosomal selection, locus_index, the minted ids — recomputes on the resumed run, so a flag that changed between the kill and the resume changes the table exactly as it would have on a first run, and the journal cannot carry a stale derivation. What it cannot survive is a different genome_build, which is checked and, on a mismatch, discarded whole with a warning. Only positive answers are staged: a request that failed is unchecked rather than absent, and freezing that into the journal would make a transient outage permanent.
  • Seeded between the caches and the live links, so a snapshot provisioned between the kill and the resume still wins the variant it would have won on a first run — and the live request is not made twice. That placement is what makes a resumed run produces the table an uninterrupted run produces a property rather than a hope; it is the item's P7 obligation and it has a test.
  • A staged answer is honoured only if the link that produced it would run this time. The seeding reads the same two booleans that decide whether the live Ensembl and gnomAD blocks execute, so a --no-gnomad or --offline resume drops what that link had already answered instead of stamping a source="gnomad" row a first run with those flags could never have written. alts is in RESOLUTION_FACT_FIELDS, so honouring a switched-off link would move the compiled digest too. The dropped answers are logged, and their subjects go back to the chain.
  • The gate commits. Every refusal (ref mismatches, withdrawn rsIDs, stale rsIDs, unresolved keys) raises before the write block, so a refused strict run changes nothing is now a written promise rather than an accident of statement order. The test asserts it on the bytes on disk of a pre-existing table, not on a return value. The staged answers are left behind by a refusal, which is the point: the next run resumes.
  • keep_staging=True / --keep-staging leaves the staged answers after a successful commit, for debugging, and logs where they are. The default removes them.
  • Not mode-conditional. write=True meaning "at the end" under strict and "as we go" under best_effort was refused in advance — a flag must mean the same thing in every function that takes one. Under a transaction it does, because committing is the only write. write=False stages nothing and takes no lock: with nothing written there is no window to exclude.

The licence row is part of the table's commit, in every pass (S98, RM231). Eight passes wrote their data table and then merged the SourceRow; a scaffold's placeholder licensing.csv made the merge refuse after alphagenome expression had already landed 12,003 non-commercial rows, and the compile gate — keyed on the licence table and nothing else — then passed them as unrestricted. Now each writer takes the merge as before_commit, which layout.atomic_writer runs after the temp file is fsynced and before the rename: a merge that refuses removes the temp, a table that fails to serialize never reaches the merge, and neither file exists without the other. Every pass also reads the table before its fetch (licensing.require_sources_file), so a placeholder fails in a second rather than after a whole-gene query. The one residual — the table's own rename failing after the row's has returned — leaves a licence row for data that never arrived, and the error says so. The drafting surfaces record their row after the compiler's draft append, the same gap one layer over, and the guard names them as RM228's to close.

The advisory lock, and how it degrades

The transaction does not close the concurrency window. Two runs can each stage and each commit, last writer winning over a merge with neither knowing — the reported incident is the sharp form, where a client-side kill did not stop the worker, a zombie run reached the write and overwrote a restored 330-row table with 162 rows, and the module then validated, closed and compiled green. Nothing downstream could see it: the three branches that deliberately write no row for an unanswerable subject make a shorter table indistinguishable from a module whose author resolved less, and those branches are correct.

So the whole read-modify-write window is held under fcntl.flock on the spec directory's own descriptor (transaction.spec_lock). No lockfile, because one left behind by exactly the kill this item is about would block every subsequent run — a worse unattended failure than the one it prevents — and the staleness rule that would fix it is a clock. flock dies with the process, so there is nothing to expire. The lock is non-blocking: a second run refuses with a pinned message rather than waiting half an hour behind a zombie, and the refusal is accurate by construction since the lock is only ever held by a live process.

spec_lock is the first thing enrich() does, ahead of every loader, so a spec_dir that is not an openable directory meets it before anything else. It refuses with EnrichmentError rather than letting os.open's own exception past the pass that owns the call.

The degradation is documented rather than silent. Without fcntl (a non-POSIX platform), or on a filesystem that refuses the call — a network mount may answer ENOLCK/EOPNOTSUPP, or emulate flock in a way that excludes nothing — the run proceeds and logs Advisory locking is unavailable for …, so this run is NOT excluded from a concurrent one. flock is untested here on the network filesystems a consumer may use; serialize enrichment of one spec directory yourself where that matters. Both degradation branches are exercised by tests, because an unreached branch is not an API.

progress — (done, total) over subjects

progress: Callable[[int, int], None] | None = None. Two integers, no object, no event vocabulary to keep working forever. The unit is argued rather than guessed, because P3 keeps a leaf working:

  • The incident is an idle timeout. Both reported runs died at 1800 s with essentially every variant resolved, so what the caller needs first is a keepalive with monotonic progress — which rules out phases, since a 29-minute phase emits nothing and the timeout fires anyway.
  • total must be known up front for the number to mean anything to a caller rendering it. The subject count is; the link count is not, since it depends on what resolution finds.
  • Subjects are the only unit the author's mental model already has. Links are an implementation detail of the batched resolver, and publishing one would make a refactor of resolver.py a contract change.

(0, total) is reported before any work starts. done is the size of a set that only grows, so monotonicity is structural rather than promised, and the assembly loop touches every subject, so the last report is always (total, total). The callback's exceptions are not swallowed — under the transaction an abort there leaves the staged answers exactly as a kill does.

rederive — the drift canary, performed (0.7, RM83's residue)

An ordinary run gap-fills and never re-asks about a recorded subject, so a source that silently revised an answer moves no fetched_at, no fact signature and no digest: MODULE_LIFECYCLE § 5.1's canary was an instrument that could not fire. rederive=True / --rederive re-asks every subject and reports which recorded ones came back different, as EnrichmentResult.rederived.

It composes with the transaction rather than adding machinery: the recorded table is already in memory and the fresh one has not been committed, so both sides exist at the commit boundary and the comparison is free. The comparison is over RESOLUTION_FACT_FIELDS only — read off the registry rather than restated — because the provenance columns move on every run by design.

  • None is not []. None says nobody re-derived; [] says every recorded subject was re-asked and every one still answers the same. Only the second is a clean bill, and only a real difference is printed — an empty comparison must not announce a zero as though it were evidence.
  • A recorded subject this run could not ask about keeps its recorded rows. Otherwise re-deriving would be a way to shorten the table, which is the reported incident wearing a new flag; an offline --rederive would replace a full table with an empty one. Answered-and-absent is an answer and does replace (it writes a not_found row); could-not-ask is not. The carry-forward warns, naming the subjects it kept, grouped by the reason each has no fresh row (RM394, rederive_words.CARRY_REASONS): nobody could be asked (--offline with no snapshot), a build no link serves, no position→rsID map read for a coordinate-only subject (RM393: the map that back-filled its rsID, Ensembl's or ClinVar's, was not read, so the rsID is carried rather than erased), a link asked and no usable answer, or every answer rejected by the hosting filter with a recorded placement standing (RM382). Each reason names its own remedy.
  • The drift label has seven arms (RM394, rederive_words.DRIFT_CASES): respelled, gained_allele (every recorded event still answered and the source names one more: A → A,T, 94 subjects across the examples' 0.8 regeneration), lost_allele, gained_rsid (a coordinate-only subject's rsID newly named), renamed (the same events under another rsID or none), moved (an event on each side the other does not name, or another build) and undecided. SubjectDrift.case carries the arm and SubjectDrift.respelled reads it as before: True respelled, None undecided, False every arm where the source's answer changed. One drift warning counts the changed subjects by arm.
  • A recorded locus on a patch or alt contig is left out of the comparison (RM400). A table live REST filled before RM400 can hold one beside the subject's primary locus, and the fresh answer, from the snapshot or from REST, no longer carries it. Compared whole, such a subject read as lost_allele though its primary placement had not moved. The recorded side gets the same filter, the survivors are renumbered in their recorded order (locus_index is a fact field), and one warning counts the loci left out and the subjects they sat on, since the committed table is that many rows shorter. A subject recorded only off the primary assembly is compared whole, as before. No arm and no carry reason is added.
  • A re-derivation resumes only another re-derivation. After a gap-filling run commits, its staged answers are exactly what produced the recorded table — so a later --rederive seeded from them would compare that table against its own provenance and report a clean bill for precisely the subjects it was asked to re-check. The journal records which run wrote each row, a --rederive run skips the gap-filling ones and says how many it skipped, and the reverse direction is allowed because an answer a re-derivation obtained is still an answer. --keep-staging is the door this closes: without the rule, a file kept for debugging silences the canary.
  • The honest limit. rm resolution.csv plus a re-run re-derives just as correctly and reports nothing: it destroys the old values before the fresh ones arrive, so nothing holds both sides.
  • What is not built, and each looks obvious from the headline: a --refresh command, a diffs file or table, a proposed table beside the current one, a pass that applies the newer value to an authored or curator-set cell (that destroys the evidence of the upstream change — still the rule), and re-asking every subject on every run (it would put the full resolution time on every pass to buy drift detection nobody asked to run continuously).
  • Put the cheap question first. --verify-datasets below asks the same has the world moved about the release label and costs one request per source, so it is what tells you whether re-asking every subject is worth the run. A module whose recorded release is still the one its source publishes has had no upstream revision to detect — and one whose release has moved is exactly the module worth re-deriving.

verify_datasets — has your source published since? (0.7, RM85)

SourceRow.dataset has recorded which release a module's rows came from since RM4, and two things read it — the tautology skip, and licensing.withdraw_stale_dataset when a module ends up mixing two. Neither answers the question an author who has forgotten, or a curator who inherited the module, actually has: has ClinVar published since this was drafted? currency.check_dataset_currency is that comparison, run at the end of enrich() and attested as dataset_currency.

It is a comparison and not a column. The column-shaped repair — a field saying what this module was made from and what would age it — was refused one table over in RM71 on the grounds that it restates dataset and then rots where dataset is maintained. This reads sources.csv and writes nothing at all; repairing a stale label is a re-draft, which is an author's decision and a different command.

  • Three states, and the third is the item. DatasetCurrency.behind is True (the source has published since, with both labels named), False (still current), or None — nobody could ask. None is never False. --offline is where that bites: an offline run makes no request, so every recorded release is unchecked, and a check that reported a clean bill for a source it never reached would be S4's defect wearing the badge of the mechanism S4 built.
  • The denominator is what was compared. subjects counts the legs asked and answered comparably; an unreachable or unaskable source is named in the record's detail instead of being counted, because counting it would publish coverage of a fraction the record does not state. With no leg settled the pass records a skip rather than ran(0, 0).
  • Comparability is tri-state too. clinvar_dataset_label has two forms — clinvar_2026-08-25 from the VCF header and clinvar_sha256:… for a snapshot whose VCF stated no date. They name one release space and equality across the two forms means nothing, so a digest against a stated date is uncomparable (no_reference), not behind.
  • Severity follows the mode, over behind alone. strict refuses on a superseded release and never on an unchecked one: an unreachable source and an --offline run both leave every leg unchecked, and escalating those would make --offline --strict impossible forever over something no author can edit — the unreachable_rsids rule, which warns in both modes and escalates in neither.
  • One probe ships, deliberately: ClinVar. It is the source the item is about and the only one this tier can ask for a release label in the namespace it already records — currency.ClinVarReleaseClient streams the live VCF, reads ##fileDate= through the same reader clinvar_build uses on a downloaded file, and abandons the stream after 256 kB rather than asking for a byte range a server may ignore. One definition of the label, because two spellings would not fail: they would simply never match, and the check would report every ClinVar module as behind forever. Every other source is honestly unsupported. currency.PROBE_SOURCES is the set, derived from default_probes rather than restated beside it; release_probes= injects a wider registry, and the injected one — never PROBE_SOURCES — decides who can be asked.
  • One request per source, not per row. sources.csv is keyed (source, layer) and "what does ClinVar publish now" has one answer whatever a module used it for, so the two layers share it and can never be told different things.
  • It cannot agree with itself. The rows it reads are the ones on disk before this run's commit, and the licence rows enrich() writes are at the resolution layer with no dataset at all — so nothing here is comparing the module against something this same run derived. That is the trap next door: --rederive seeded from its own staged answers would have reported a clean bill for exactly the subjects it was re-checking.

Which tables ask for a coordinate

The chain was never variant-specific; only its input was. Until 0.5 it read variants.csv alone, so a PGx module — which by design carries none (one CSV = one concern) — enriched to an empty resolution.csv and shipped with no coordinates at all. enrich.collect_subjects normalizes every eligible row to a _Subject and feeds it through the unchanged chain, caches, ordering and back-fill:

Table Identity Allele constraint fed to hosting_verdict
variants.csv frozen variant_key (with alts) genotype
pharm_variants.csv variant_key property (without alts) genotype (optional — None keeps every locus)
haplotypes.csv derived the same way, without alts the defining allele
heteroplasmy.csv (0.5.3) variant_key property, with alts — it mints one exactly as VariantRow does none: a measurement band over a locus is not a claim about a genotype
studies.csv proxy_rsid (0.8, RM331) the proxy's rsID, admitted last none: the compiler's effect_allele check judges the allele, and filtering loci here would drop what it reads

A HaplotypeRow reuses the same membership predicate rather than a parallel one: its defining allele is the one-allele form of the question a genotype asks of two. Subjects are deduped by variant_key with variants.csv first, so when two tables name one variant the SNP row wins — it is the only one carrying alts, a resolution fact, and letting a PGx row win would move an already-compiled module's artifact.digest. The PGx tables key without alts deliberately: a pharm annotation or haplotype junction matches a variant at chrom:start:ref regardless of allele.

heteroplasmy.csv joined the list in 0.5.3, and it is the one row here that is build-dependent. Its coordinates are optional exactly as the PGx ones are, so an rsid-authored heteroplasmy module resolved to nothing at all — the same gap PGx had before 0.5, found from the other end when the compiler started reporting which tables a VCF cannot join (COMPILER.md § Scope). Because its variant_key carries alts, it can mint a ga4gh:VA.…, so that load passes the module's genome_build where the two PGx loads rightly do not — the RM36 trap, one call site further on.

A study's LD proxy joined the list in 0.8 (RM331). StudyRow.proxy_rsid names the variant a study measured in place of the row's lead, and the compiler checks the row's effect_allele against the proxy's resolved alleles. A proxy is usually not a module variant (rs4970836 for longevitymap's rs646776), so without a subject of its own it has no resolution.csv row and the check withholds. It is admitted after every table above, so a proxy that is also a module variant keeps that row's subject, and a later run keeps its row in scope. The r² between proxy and lead is the ld-pairs pass's (§ The computed LD pair). The compile carries the proxy's resolved loci on studies.parquet.proxy_loci, so reverse re-emits its resolution.csv row; every table listed here is in SUBJECT_TABLES, which a test holds equal to the tables reverse rebuilds rows from. An invalid study row does not stop the run, unlike an invalid pharm_variants.csv or haplotypes.csv (coordinate sources, so a run that cannot read one has nothing honest to resolve): studies.csv only adds proxies, so the valid rows' proxies are asked for and one warning names the rest (studies.csv has N invalid row(s), and any LD proxy on them is not asked for); the compile refuses those rows anyway. A proxy is not one of the module's loci, so the passes that read resolution.csv as the module's own coordinates — the frequency pass and the clinical-assertion pass — leave out every row keyed by a proxy-only rsID (enrich.subjects.study_proxy_rsids), and the GWAS pass never asks about one. Otherwise each would spend a request on a locus the module does not annotate and write a fact row the compile reports as derived_row_orphan. Naming those rsIDs runs the subject collector, so the frequency and clinical-assertion passes now refuse an invalid pharm_variants.csv, haplotypes.csv or heteroplasmy.csv with their own error type, as enrich already did; before 0.8 they read only resolution.csv.

What the enricher resolves is not what the compiler applies. These subjects all land in resolution.csv, and the compiler applies that table to variants.csv only; a PGx or heteroplasmy table is materialized verbatim. So enriching a PGx module is still worth doing — the table records the coordinates, and a consumer can join them itself — but the parquet keeps the author's nulls until RM43.

Multi-allelic snapshot rows

The Ensembl snapshot has two published spellings of a multi-allelic site. The old repo (just-dna-seq/ensembl_variations) stores it as one row whose alt is pipe-joined (A|C|T); the split repo 0.8 reads (just-dna-seq/ensembl_variations_split, RM372) stores one row per ALT, every ALT of a site padded to one shared ref. Live Ensembl, ClinVar and gnomAD all emit comma-separated lists. resolver._snapshot_alleles normalizes at that one boundary so a locus dict's alts has a single canonical shape whichever spelling the cache holds; test_enrich.py runs the multi-allelic tests over both.

Each ALT is then spelled alone, never in the site's padded frame (RM381). The split repo writes rs1555461597 as 16:23635706 GTT>G, GTT>GT and GTT>GTTT. Grouping those rows by their ref gave one locus, GTT>G,GT,GTTT, which no caller writes: offline nothing could respell it, and online the fill read the module's G/GT in that frame as two deletions and kept the locus whole. Every reader of a snapshot's alt now passes each ALT through alleles.trim_allele (drop the bases both sides end with, then those both start with, while each keeps one: bcftools norm's trim, which reads no reference) and regroups by the trimmed spelling (sequences.per_alt_loci): the three loci G>GT, GT>G and GTT>G. That is the Ensembl and ClinVar rsID lookups, REST's allele_string (TCTT/T/TCTTCTT is one such frame), gnomAD's variant ids, and the three position→rsID readers, which match an authored allele as the row spells it or as trimmed. The old repo's pipe-joined row is trimmed the same way. Left-alignment past the trim still reads the reference (left_normalize), so offline an indel piece is written trimmed with left_aligned unknown. The split repo anchors left, so for its sites the trim is usually the whole normalization. test_split_snapshot_per_alt.py holds the readers to ClinVar's own per-ALT records on real rows, and an AST walk keeps every function whose SQL reads a snapshot's alt in its registry.

This is load-bearing, not tidying. The hosting predicate splits alleles on commas, so an un-normalized A|C|T collapsed into one opaque "allele", no genotype was ever a subset of {ref} ∪ alts, and the allele-aware filter dropped every locus — a cache-resolved rs4244285 with the ordinary genotype A/G resolved to not_found. The reverse back-fill had the mirror bug, comparing an authored alt against the whole joined cell with !=. Both are pinned by tests that fail on the pre-fix code.

A located but unusable cache (a stale snapshot, or a parquet some other tool left in the cache directory) is treated as a miss, with a warning — one optional link's bad data must not sink an enrichment the other links can still complete.

--offline clamps the chain to the local caches (steps 2 and 4 — Ensembl and ClinVar; guaranteed zero egress). Every filled row records the link that won in source, so the compiler can surface resolution_sources in the manifest.

Modes. best_effort fills what it can and records the rest as status="not_found" rows (a warning, not a failure). strict raises EnrichmentError unless every in-scope variant resolves to a position — the network analogue of the compiler's strict=True — and unless every authored ref agrees with the reference sequence. The reference check is raised first, deliberately: a row contradicting the genome is a worse diagnosis than a row the chain could not find, so it should be the error the author sees. EnrichmentResult carries rows, unresolved (variant_keys with no position), sources, mode, and a fully_resolved property.

A not_found row is written only where a source actually answered (S20, 0.5.4). enrich() used to write ResolutionRow(status="not_found", source="ensembl") for a request that failed, stating in the injected table that Ensembl was asked and does not have the rsID. It now writes nothing for that key — the key stays unresolved, so strict still refuses and best_effort still warns, but nothing claims a source said no — and EnrichmentResult.unreachable_rsids names them. That list is deliberately separate from unresolved, which says a key has no position and is silent about why: a key that failed because egress broke and one with genuinely no locus to find are the same entry there, and only one of them is worth re-running. Empty under --offline, since nothing was asked. It warns in both modes and strict does not escalate it — no authored edit clears a failed request (P5), and the not_covered and VRS-coverage findings are the same class. The argument was already four lines below in the same function, where the non-GRCh38 branch declines to write not_found for precisely this reason.

A not_found row also says something false when the source ANSWERED and the alleles did not match (S85, 0.7). That is a fourth state, and until it was named the row was byte-identical to a genuine absence: enrich asked, the source returned loci, the allele-aware filter rejected every one, and the row went out as status="not_found" — this source has no record of your rsID — about an rsID the source demonstrably has. The reported case is the common one: a paper's supplementary published against GRCh37/hg19 spells the submitted strand, so its G/A meets GRCh38's C/T and five subjects of a 64-variant module read as unknown to Ensembl when Ensembl returns all five immediately.

EnrichmentResult.allele_mismatches carries it, as AlleleMismatch(rsid, genotype, loci, offered, strand_flip) — the shape ref_mismatches and stale_rsids already have. It completes the family the RM98 round started: unreachable_rsids means the request was made and failed, unconsulted_rsids that nobody looked, unresolved that a key has no position while staying silent about why, and this one that the asking succeeded and the answer did not match. strand_flip is the single cause this tier can settle from the allele strings alone, and it is False for not established rather than established otherwise — an allele that cannot be complemented withholds it.

Since 0.8 the row is gone too, and the chain asks on (RM382). S85 kept the not_found row beside the finding. Regenerating pathogenic_clinvar showed what that cost: the rejected answer also ended the chain, so ClinVar, which holds HBB rs613985 where the snapshot does not, was never asked. A rejected answer is now a miss (the resolver chain above), the finding names every link that answered (links, in chain order, with loci in the same order), and an rsID-only subject every link rejected gets no row. A later link's placement is recorded in superseded_answers, with placed_by naming it.

Three more states are named since 0.8 (RM272), each an rsID that is also in unresolved and in exactly one of these lists, because each has a different remedy:

  • unanchored_rsids — live Ensembl answered with a one-sided indel whose anchor base could not be read (RM268), so no row is written. Transient: re-run where the GRCh38 reference can be read.
  • non_nucleotide_rsids — every allele the snapshot and live Ensembl gave is not a base (dbSNP_novariation, <.>, an N run; RM271), and the row is written not_found. Permanent: dbSNP records no variation at that mapping, and asking again gets the same answer.
  • reference_rejected_loci — per locus ("<rsid> <chrom>:<start>"), the loci the GRCh38 reference showed are a different indel of the same size from the authored genotype's (RM274). Permanent, and about which locus rather than whether the rsID resolved, so it is not an rsID list.

The log lines that named them before stay as they were.

A mapping off the GRCh38 primary assembly is not a locus, at the REST rung either (RM400, 2026-11-07). REST maps an rsID onto patch and alt contigs too (HSCHR17_1_CTG5, HG76_PATCH). No Ensembl snapshot carries one, a primary-assembly VCF never joins one, and gene_spans, pharmvar and the ClinVar-GKM build drop them, so _loci_from_rest drops them as well, before reading any anchor base, and logs the count. An rsID REST maps only off the primary assembly is the answered-empty [], written not_found, which is what the snapshot answers for it. Before RM400 the rung wrote them, so which link answered decided an rsID's loci.

The row itself is unchanged, deliberately, and that is not a half-fix. Adding a VALID_RESOLUTION_STATUS member would change a wire vocabulary every reader of a published resolution.csv shares; dropping the row would move resolution_signature, since variant_key and rsid are RESOLUTION_FACT_FIELDS while status is provenance and is not. The row was never the untruth — it is honestly unresolved either way — only the reason it gave, so what moved is the reason. The run warns once, in both modes, naming the rsIDs and saying the source has them: an author who greps the artifact for not_found is the person this finding exists for.

The per-locus warning stopped asserting a cause it had not established. hosting_verdict returns a confident False from two arms — a substitution/MNV locus (no flank, so no spelling freedom) and an event length the locus does not offer — and the message gave the second arm's reason for both, so a strand-flipped SNV was reported as "The event sizes differ, which re-anchoring cannot change" about two 1 bp substitutions. compiler.resolution.contradiction_reason is now the False twin of undecided_reason and supplies which, the same repair that function received for None's five causes.

EnrichmentResult.vrs is the MintResult this call computed (RM40, 0.5.1). It carries exactly the two counters compile_module later stamps into manifest.compilation.vrs_alleles / vrs_alleles_identified, plus unmintable_reasons — the grouped-by-reason breakdown that is the actionable half, where "no refget table for build 'GRCh37'" and "needs the reference sequence" live. It used to be logged and dropped, so a consumer reading coverage before a compile — which is what a publish dry run is — had to re-implement the counting, and get two non-obvious rules right to agree with the manifest a publish would produce: count per ALT slot (vrs_id is a parallel array of alts), and treat an absent cell as len(alts) unnamed slots rather than zero, or a table where nothing minted reports flawless coverage out of a denominator of nothing. None when mint_vrs=False — never a coverage of zero.

Resolution is allele-aware in both directions

A rsID is a position/multi-allelic dbSNP tag, so one id routinely names several genuinely different records. Both directions of resolution therefore match on the allele, not just the position — and the forward direction only caught up in this round, which is why the asymmetry is worth naming.

Forward (rsid→loci). A candidate locus whose {ref} ∪ alts cannot host the module's authored genotype is reported and left out of resolution.csv. In the committed HBB example rs281864532 names G>GT, GT>G and GTT>G at one position, and rs613985 names records at two positions 254 bp apart; the authored genotype says which are meant. Recording the rest would hand the compiler a locus it can only drop — and a dropped locus makes the compile unreproducible from the injected table, which --strict refuses. This selects, it does not repair: no authored value is touched, and every skipped record is logged. When every link that answered was rejected, an rsID-only subject gets no row (RM382): until 0.8 it got status=not_found, which says the source has no record of the rsID, about an rsID it has. The key stays unresolved, EnrichmentResult.allele_mismatches names every link's rejected loci, and strict's refusal names the rejection rather than asking for a cache, unless a live link could not be asked for the rsID, in which case it asks for the link. A pair keeps its authored row. (The compiler applies the same predicate as a safety net for hand-authored or stale tables — resolution.hosting_verdict is shared, so the two cannot drift.)

A row that authors both halves is recorded from the reference, not copied (S104, RM251)

A subject carrying rsid and chrom/start needs no resolution, and until 0.7.2 that was the whole of what happened to it: the verbatim branch wrote the authored coordinate into resolution.csv under source="authored", with no ref, no alts, and therefore no VRS id. Yet the same run had already looked the rsID up — the rsid↔coordinate check rides its rsIDs in the Ensembl cache batch — and the answer was discarded. Every CPIC-drafted haplotypes.csv has this shape (allele_definitions gives an rsID, a position and, since the gene.chr join, a chromosome), so every such module compiled with "VRS allele identity covers 0/N" and nothing the author could do about it, and the compiler's _verify compared the module against a photocopy of itself.

Now such a row takes the forward branch when the reference knows its rsID: the loci go through the same allele-aware filter and the same PAR rule, and are recorded with the link's source, ref and alts, so a VRS id mints and the compiler holds two independent values. The authored coordinate is untouched — it is the row's identity, the compiler keeps it, and its positional fill only completes the cells the author left empty (ref, for the CPIC shape). Where the two disagree, that is the finding: rsid_coordinate_agreement records it here, and resolve_from_table._verify warns in best_effort and refuses in strict, exactly as the mishap matrix already said for an authored coordinate that contradicts the table. Three cases still write the authored row: an rsID the snapshot does not know (no link is asked live for a pair, so there is no answer and no negative to invent), a reference whose every locus is rejected by the allele filter (the mismatch is reported, the row keeps its coordinate), and a run with no Ensembl snapshot at all. A table written before this change keeps its authored rows — merge-not-clobber covers them — so delete resolution.csv or run --rederive to record the reference's answer.

A placement is compared with a second authority before it is written (0.8, RM267)

Every link in the chain keeps the first answer and none compares it with another, so until 0.8 every coordinate in resolution.csv had one witness. S117 is why that matters: the Ensembl snapshot (faithful to Ensembl's own VCF dump) places rs8176719, ABO O1, at 9:133257520 G>GC, while ClinVar, dbSNP and Ensembl's own REST place it at 9:133257521 T>TC. GRCh38 reads GGGG T ACC at 517–524, so those are two haplotypes. The wrong anchor is a real base, so the reference-allele check passes it, and RM31's tolerance calls an indel's position difference undecided.

So after the links have run, placement.compare_placements asks the two offline witnesses this tier holds, the ClinVar snapshot and ClinVar-GKM (--clinvar-gkm-cache, or $JUST_DNA_CLINVAR_GKM_CACHE; read locally, never downloaded by enrich), where they place every rsID this run placed. For each allele the first authority gave, the witness's counterparts are its alleles for that rsID on the same contig, within the window an indel can slide in, with the same kind of event (an insertion or deletion of the same length, or a substitution to the same bases). A counterpart is the same event when the spellings are identical, when both are substitutions with the same trimmed spelling, when both mint one VRS id (exact, no tolerance), or when both left-align to one spelling against the reference. Otherwise it is undecided and counted, which is what an offline run gets for an indel respelling. ClinVar-GKM's own ids are VRS 2.1.1 and ours 2.0, so its VCF spelling is minted here rather than its id compared.

  • A disagreement withholds the locus. The locus holding the disputed allele is not written; a locus of the same rsID nobody disputes still is. When every locus is withheld, an rsID-only subject stays unresolved (so strict refuses, and its sentence names the disagreement rather than asking for a cache) and gets no not_found row, since the sources do have it; a row that authored its own coordinate keeps it. The warning names both placements; nothing is rewritten (no corrected spelling in 0.8, PROPOSAL_0_8_PT2 §4 question 3). An author who records the placement they stand behind in overrides.csv (any operation on that resolution.csv subject) is no longer unresolved: the compile applies their answer, and the record still counts the disagreement.
  • A row reads same_event only when every one of its ALTs was placed the same by a witness; an ALT no witness holds makes it unplaced, an unsettled one undecided.
  • A different allele is not a disagreement. ClinVar holds what somebody submitted, so an allele with no counterpart is unplaced, and the rsIDs where nothing had a counterpart are the published one-witness count. A witness allele that is already one of the rsID's placed alleles is no counterpart of the others (RM448, 0.8.9). At a multi-allelic site, ClinVar's A>AC is the same kind and length as Ensembl's other insertion A>AT. Judged against it with the reference read, it was a different event, so the whole locus was withheld and the rsID got no row. That happened online only, since offline nothing judges an indel. The witness's allele is accounted for by the placed allele it matches, and says nothing about the others, which read unplaced. S158 met it on 12 rsIDs of the registry's pathogenic and one of cardio. A witness allele that matches none of the rsID's alleles still disagrees (S117's rs8176719).
  • A ClinVar fill has no independent witness here. ClinVar-GKM is ClinVar's own release, so a row the ClinVar link filled is counted with the one-witness class and its two columns stay empty.
  • Each written row says who was asked and what they answered: placement_witness and placement_verdict (SCHEMAS.md). Both empty means nobody was asked. Every witness a written row names gets its SourceRow at the resolution layer (@write-the-sourcerow), and the compiler's orphan check counts those names as used.
  • It attests as placement_agreement: subjects is the rsIDs compared, findings the rsIDs with a disputed placement, undecided the respellings nothing could settle, and detail the one-witness count over every placed rsID, the rsIDs only dbSNP compared, and the rsIDs dbSNP could not be asked about.
  • Recorded rows are not re-asked. Merge-not-clobber holds, so a resolution.csv written before 0.8 meets the comparison only under --rederive or after the file is deleted. Under --rederive a disputed subject counts as answered, so its recorded placement is not carried forward.

Probed 2026-11-04 on real data: neither loaded witness holds rs8176719 (ClinVar's 2026-06-27 VCF has no record at 9:133257510–529 and no RS=8176719; ClinVar-GKM lacks it, RM305). Catching S117's own case needs a witness that holds it, which is why dbSNP is read:

  • dbSNP is the third witness, online, with the merge check on (RM398). For an rsID the Ensembl snapshot placed with at least one allele that is not a substitution, and that neither loaded witness holds, the run reads dbSNP's own GRCh38 placement out of the esummary db=snp record the rsID merge check fetches anyway (placement._dbsnp_placements). The record's spdi lists every allele as accession:position:deleted:inserted, 0-based: rs8176719 is NC_000009.12:133257521::C, which anchors on GRCh38's own base to 9:133257521 T>TC; the snapshot's 9:133257520 G>GC mints a different id, so the rsID is withheld with both placements named. An accession off the primary assembly is no locus. The witness reads first and the merge check is then served the same records, so the run asks dbSNP nothing it did not ask before RM398. A row the live rung gave itself, a substitution, and an rsID ClinVar or ClinVar-GKM holds are not asked. Answers are never journalled.
  • dbSNP is the snapshot's upstream, so it is self-consistent, not independent (RM292). The witness is named dbsnp, the licensed source placement_witness joins sources.csv on (public domain, NCBI), and it is its own authority, so the licence join stays true; the snapshot is built from dbSNP 157, so placement._UPSTREAM_OF counts dbSNP's agreement with a snapshot placement as self-consistent and leaves the rsID in the one-witness count, which detail names apart. It still catches the snapshot's processing disagreeing with dbSNP's placement, which is S117's mechanism; it cannot catch an error in dbSNP's own placement. The record names no dbSNP build, so the licence row's release is withheld.
  • Not asked is counted, never agreement. A record that could not be fetched (eutils retries a transport error, a 5xx and a 429), or an indel whose anchor base could not be read, leaves dbSNP not asked for that rsID: the row does not name dbsnp, nothing is withheld, and detail counts the rsIDs with the reason. A run with the merge check off (verify_rsids=False) counts every eligible rsID as not asked and says so; --offline asks nothing.
  • It needs nothing of an injected resolver. Until RM398 the third witness was live Ensembl REST, asked through ensembl.RestWitness, an optional method of the resolver, and S133 measured REST's cold requests as most of an online re-derivation's time. RestWitness and placement.ask_live_witness stay as 0.8.0 shipped them, and a recorded row may still name ensembl as its witness; nothing asks it.
  • Measured before it landed (2026-11-07). Over the 169 snapshot indels of the reference examples (5 modules, the 0.8 split snapshot), dbSNP and REST each disagreed with the snapshot on none, so no locus is withheld differently. Their verdicts were identical on 157. On 11, dbSNP confirmed alleles REST left unplaced; on 1 (rs1569493663, a PAR locus) dbSNP's record places the Y copy only, so the X copy reads unplaced. The registry's 46-module replay is the larger re-measure (RM406).

The ClinVar witness is the snapshot the run locates for its link and the clin_sig check, so --no-clinvar --no-verify-clinsig leaves only ClinVar-GKM. The live REST rung is asked only as the third witness above, for a snapshot indel neither loaded witness holds, one request per such rsID.

A filled coordinate is written left-aligned; an authored one is judged (0.8, RM270)

An indel can be spelled more than one legal way, and a caller writes the leftmost parsimonious one (bcftools norm -m -any). The Ensembl snapshot does not always: it places rs72613567 at 4:87310241 A>AA where ClinVar and a caller write 4:87310240 T>TA, rs77944059 seven bases right, and SHOX's rs1569493663 as the multi-allelic X:634690 AGAG>AG,AGAGAG. An exact (chrom, pos, ref, alt) join then misses the row, though nobody is wrong (S120).

  • A coordinate this run fills from an rsID is written left-aligned (sequences.normalize_locus). A fill is outside content_signature (@rm43-positional-fill), so the respelling moves no authored identity. Each ALT takes its leftmost spelling through sequences.left_normalize: a substitution or MNV by trimming its shared flank, with no reference read, and an insertion or deletion through _left_align, the same normalizer RM267's comparison uses (@one-normalizer-two-spellings). ALTs that land on one (start, ref) stay one locus; ALTs that land on two become two loci, the split a caller makes, so SHOX's site becomes X:634689 CAG>C and X:634689 C>CAG and the compiler expands the rsID onto both. It runs after the hosting filter and the PAR choice, which read the link's spelling as before, and each piece keeps the RM267 verdict of the locus it came from. A split writes the piece the row's genotype is about (orchestration._filled_pieces): a genotype spelled in the source locus's own frame names its ALTs exactly (AG/AGAG on SHOX's site is the deletion); one naming ALTs in two pieces (AG/AGAGAG) has no one left-aligned locus, so the source is written whole, as the link gave it, and judged; otherwise every piece that can host it (C/CAG fits either). When any ALT cannot be decided (offline, an unreadable window, a non-nucleotide allele) the locus is written as the link gave it: half a respelling would put two conventions in one cell. Since RM381 the link gives each ALT already trimmed to its own spelling (§ Multi-allelic snapshot rows), and the hosting filter before it reads a genotype authored in a site's padded frame in that frame (resolution.site_verdicts, the same trim on the genotype's alleles): GTT/GTTT names G>GT alone, and the compound het G/GTTT names GTT>G and G>GT, one allele each.
  • An authored coordinate is judged, never rewritten (@enrichment-is-validation). The cell is the row's identity, so every subject that authored a whole coordinate is checked against its leftmost spelling on every run, and a difference is a warning naming both (EnrichmentResult.spelling.findings, over checked, with unknown counted apart). The row's vrs_id is what joins it to a caller's spelling, and since RM270 it is on the parquet. The same holds for a pair's recorded reference locus (S104): the compiler compares it with the authored coordinate by spelling, so it is recorded as the link gave it.
  • Every written row carries the verdict, left_aligned (SCHEMAS.md): true checked left-aligned (every fill the run respelled, and any row already so), false checked and not, empty unknown. Outside resolution_signature.
  • Recorded rows are not respelled. Merge-not-clobber holds (@sidecar-authoritative): a resolution.csv written before 0.8 is respelled only under --rederive or after the file is deleted. The drift canary names a respelled subject apart (SubjectDrift.respelled, logged at INFO; its other arms since RM394 are under rederive above): when every allele on both sides carries a vrs_id and the ids are the same, the source did not move, this release's spelling did. Three-valued since RM375: with an id missing on either side the alleles go to the one spelling judge (events.same_allele_sets, against the reference when the run has one), so a respelling reads True, a different event False (the drift warning), and a change this run cannot settle None, warned apart as undecided rather than blamed on the source. An unmatched substitution is a change; an unmatched indel is undecided, because an offline run keeps indel ALTs the online hosting filter and RM270's split would drop (RM376's rs746071566).

The pass is not a verification.json check: it compares the module's spelling with a convention, not a claim with a witness, and the reference-allele check already attests the bases it reads.

Two spellings of one event: the enricher judges, the compiler trusts (RM375, 0.8)

A pair's locus is recorded as the link gave it, and since the 0.8 snapshot (ensembl_variations_split) the link's indels are left-aligned where the old snapshot sometimes anchored them right. NUDT15's rs761191455 is 13:48041104 G>GG in the old release and 13:48041103 T>TG in the new one, one event in a run of Gs, and reference_examples/cpic_gene_pairs/haplotypes.csv authors the old anchor. The compiler, which holds no reference, called the row positional_identity_contradicted and left it unfilled; check_rsid_coordinates called it a disagreement (an insertion's one-base ref read as a substitution); --rederive called the respelling a source change whenever a row had no vrs_id.

  • One judge, events.py, three-valued: same_event (two whole spellings: identical, trimmed substitutions, VRS ids, left-aligned tuples) and names_event (an authored coordinate that may be partial — a haplotype row authors a start, no ALT — against one allele: its anchors are the substitution's own and trimmed spelling, or every anchor sequences.indel_spellings walks inside the repeat). Different only where that is sharp: a substitution elsewhere, or an indel whose every spelling was read and none touches the authored position. A position inside the event's footprint that is no VCF anchor, a padded ref, and anything with no reference read are undecided. placement.compare_placements, resolver.coordinate_verdict, outcome._same_events and the row stamp all ask it; enricher/tests/test_authored_event.py runs each over one corpus and an AST walk asserts no other function calls it.
  • Recorded per row, resolution.csv authored_event (VALID_AUTHORED_EVENT_VERDICTS: same_event / different_event / undecided), on every locus written for a subject that authored a coordinate whose chrom, start or authored ref differs from the locus's, beside judged_coordinate, the coordinate judged (authored_coordinate_key): the key is the subject's, so the compiler applies the verdict only to a row still authoring that coordinate. Empty where the spellings agree (the compiler sees that on its own) and on a row written before 0.8. A row, not verification.json: the record is rewritten per run over the subjects that run asked, and merge-not-clobber keeps a row's verdict beside the row it judges. Not placement_verdict: that is who else places the rsID, this is whether the module's own cells name the answer (P5).
  • Offline is undecided for an indel: the reference is read through seqrepo, which has no local equivalent here, so an offline run never settles a respelled indel. --rederive online settles it.

The rsID's whole answer beside the narrowed one (0.8, RM387)

The hosting filter, a disputed placement (RM267) and the pseudoautosomal choice narrow the loci written under an rsID's own key, while a positional row's rsID is back-filled from the unfiltered answer. So the forward branch records every place the links offered this run (each link's answer, RM382's set-aside ones included) as source_loci (SCHEMAS.md) wherever that holds a place the rows stand for nowhere; a respelled piece stands for its source locus and a Y pseudoautosomal twin for its X partner (orchestration._wider_answer). RM374's compile check reads the union, so a positional row at a sibling locus is one answer. A subject an ordinary run carries keeps what it recorded; --rederive writes the column, and reports no drift for it, since it is no fact.

Which release answered — per row, and what the table may say (0.8, RM373)

source names the link and authority the licensed source; until 0.8 nothing named the release, so a row the old Ensembl repo answered and one the split repo answered read alike to every tier, and RM372's rsid_coordinate_agreement.release stated the run's snapshot over a table every row of which an older release had answered. Three pieces now, one rule each (enrich/releases.py):

  • Every row this run answers carries source_release (SCHEMAS.md): the release of the link that answered it, read the way the lane states it (CacheLane.release_label): the Ensembl snapshot's manifest.json release_id or release.json dataset, the ClinVar snapshot's label by clinvar_dataset_label (the one spelling every ClinVar reader computes), live REST's data release from /info/data (ensembl_116, asked once per run, only of this tier's own EnsemblResolver, and only when REST answered). The retired GraphQL link stated none; gnomAD's dataset id (gnomad_r4) names a release line, not a release, so a gnomAD row records none; and an answer seeded from an interrupted run's journal records none, because that run never wrote its release down. A row this run did not answer keeps what it had, and a row written before 0.8 keeps none: merge-not-clobber never back-fills (@sidecar-authoritative). --rederive re-answers every row and stamps each; a subject it cannot ask about keeps its rows, release included (@rederive-never-shortens).
  • The licence rows are labelled by the committed table, not the run. record_table_sources writes each source's dataset as the one release every committed row it answered states, and withholds it (blank) when the rows state two, or any row states none. It stamps the cell rather than leaving it to merge_sources_file, which never clobbers and so never filled a row an earlier run wrote blank. A placement witness has no column of its own: a row this run wrote speaks for the witness's release this run (a snapshot witness's own label; none for the dbSNP witness, whose record names no build, so its label is withheld; the REST witness, before RM398, stated /info/data's), and a carried row speaks through the label recorded before this run, which was computed over every row the table then held — so a run with nothing to do does not withdraw a label still true. Only rows of a source with a terms constant are stamped, so a curator's manual row is never touched.
  • A run that answered every row retires the licence row no committed row names (RM383, 0.8). record_table_sources takes answered_every_row (keyword-only, default False, and stated by every call in this package, which a test walks). enrich passes rederive, and the concordance record, rewritten whole on each run that writes it, passes True. Then a row of the pass's own (its layer, a source with a terms constant) that no committed row names is removed inside the table's commit, and an info line names each source,layer retired. Measured on mt_common_deletion: its two point variants moved from ClinVar's back-fill to the split Ensembl snapshot's, and --rederive left clinvar,resolution behind, so the compile warned source_row_unused. Removing it, rather than keeping it with a reason, is the @write-the-sourcerow converse (a row for a source the table does not use is a false statement in a published artifact, and SourceRow has no field a gate could read as "stale"); it is deletion by declaration, because --rederive is the operator saying every row is re-answered (@a-publish-may-not-orphan-the-bytes-it-stops-describing). A routine merging run, and every other derived pass, passes False and retires nothing: merge-not-clobber is its contract, and a deletion it was not asked for is the side effect the rule forbids. A row a subject nobody could ask carries forward still names its source, so an offline --rederive that asked nothing retires nothing.
  • The run's own record withholds its label over a mixed table. rsid_coordinate_agreement.release is the snapshot's label only when every row the snapshot's link (source=cache) answered carries it; otherwise it is null and detail counts why: release withheld: N of M snapshot-answered row(s) in resolution.csv were not answered by this run's snapshot (…): K answered by another release, J recording none. Rows live REST answered carry their own release and are not the snapshot's to speak for. The licence row is the table's, though, and REST spells one Ensembl release ensembl_116 where the snapshot spells it …vep116.0: nothing here can equate the two, so a table both answered names no release on its ensembl licence row.

The same helper writes every derived pass's licence rows (frequencies, gene-metrics, dosage, gene-validity, assertions, gwas, ld-pairs, alphagenome expression, opentargets enrich), from each table's own per-row dataset read through net.stated_release (the bare lane release_dataset writes for an unstated release, gwas_catalog or clinvar, is no release), or, for Open Targets' tables, which carry none, from this run's release on the rows it answered and none on a row it kept: the label recorded before may be a pre-0.8 first run's. A row is labelled by its source's own release, never the day the run asked: gwas reads the Catalog's from /api/search/stats (gwas_catalog_2026-10-29), the AlphaGenome atlas states none (alphagenome_atlas), assertions uses clinvar_dataset_label, and gene-metrics reads the constraint snapshot's release.json. DerivedProducer.release_lanes names the lane each pass reads whose release reaches its licence row; test_producers.py asserts the lanes each command's code reads equal that field plus LANES_READ_WITHOUT_A_LICENCE_ROW, and that no derived pass writes a licence row any other way. Since RM379 the concordance record writes one per consulted authority its call rows name, at the clin_sig_authority_call layer.

A derived table's release moving never makes strict refuse. A filled clinvar/clinvar_gkm licence row is a release --verify-datasets compares, and both have a currency probe. A source publishing past the release that answered is the source working and no authored edit clears it (@a-source-recuring-is-not-a-strict-matter), so on every layer but annotation (licensing.DRAFTED_LAYERS) the finding warns in both modes, escalates in neither, and names its remedy. The remedy depends on which release is behind (RM394, rederive_words.CURRENCY_CASES), read against the label of the snapshot this run read for the source: the snapshot current and the table behind it is enrich --rederive; the snapshot the release the table recorded is cache pull --only <lane> (or cache rebuild) first, because a re-derivation reads that same snapshot and prints the same line; a snapshot that is neither is both; and no comparable snapshot read names cache status and both. One line per source, since each names its own lane. Only a drafted layer's superseded release refuses under strict and says re-draft.

Reverse (position→rsid) back-fill is allele-aware

A coordinate-only variant (rsid None, coordinate authored) can have an rsid back-filled from the reference — but the lookup is allele-aware: it matches the exact allele (chrom, start, ref, alt) via the shared resolver._lookup_rsid_candidates, not just (chrom, start, ref). This matters because an rsid is a position/multi-allelic-level dbSNP tag, not a per-allele one — e.g. rs33922842 at one HBB locus tags C>A (pathogenic), C>G (benign) and C>T (uncertain). An allele-blind match would let an un-rs'd insertion inherit a co-located SNV's rsid. So:

  • 0 allele-exact candidates → rsid stays null, source="authored" (the coordinate is the identity; never guess a label).
  • 1 → attach it (status="resolved").
  • ≥2 for the same allele (a genuine dbSNP merge) → deterministic pick in rsid, status="ambiguous", and the full candidate list in ResolutionRow.rsid_alternates — recorded, never silently chosen.

The candidates are narrowed by the row's alleles before they are counted (RM438). A row with no single alts matches every rsID at (chrom, start, ref), and dbSNP 157 put a 16 bp insertion (rs2517910443) at CYP2D6*4's SNV (rs3892097, 22:42128945), so an A/G row there read as two candidates. One shared rule, resolver._hosting_rsids, now keeps only the rsIDs whose alleles at the position can host what the row states: its alts when stated (with its ref), else every genotype the variant's rows author (a pharm or haplotype subject narrows on its own genotype or allele). The judgement is the forward direction's, site_verdicts; an undecided verdict keeps the candidate (Kleene), and an allele it cannot read (*, symbolic, unparseable, no genotype at all) narrows nothing. The counts above then apply to what is left, so zero hosting is an answer (rsid stays null), and a link whose candidates the row's alleles all refute is a miss, so the next link (ClinVar) is read. The deprecated resolve_variants path asks the same rule and withholds at ≥2 (rsid_ambiguous, naming every one) rather than picking, because it writes no table that could mark the pick; at zero it attaches nothing and says no rsID there can host the row. On this path a row stating one ALT was already allele-exact and is unchanged; resolve_variants never read alts before RM438, so there it now matches the ALT as this path does.

A pseudoautosomal locus is one place, recorded as the X spelling (RM32)

PAR1 and PAR2 are shared between X and Y, so dbSNP maps one rsID to both contigs and the expansion above would emit two rows for one finding. enrich() keeps the X spelling and reports the Y twin it left out; --keep-par-twin (keep_par_twin=True) records both, for a consumer whose reference is not analysis-set masked.

This records the sources' convention, not the consumer's reference — which is what makes it this tier's call rather than a data-agnostic violation (P2 makes the enricher the only tier permitted to hold a source convention). Probed 2026-08-04: ClinVar holds no variant in either PAR on Y (all 677 of its Y records lie outside them), gnomAD v4 excludes the Y PAR from its callset (region(chrom:"X", 640000-641500) serves 880 variants; the same interval on Y serves none), and the ClinGen Allele Registry does mint a separate Y allele id but leaves that record a bare dbSNP cross-reference. Only Ensembl/dbSNP reports both contigs. There is therefore no external place identity to adopt — the Registry minting two CA ids is what closed that direction — and none is invented here.

Three properties worth knowing:

  • The pairing is an offset, not an equality. vrs.par_partner (format tier, stdlib) maps PAR1 at offset 0 and PAR2 at 98,813,480, from the interval table. PAR1 shares coordinates between the contigs and PAR2 does not, so a "same base on X and Y" shortcut would pass a PAR1 module and silently fail a PAR2 one.
  • Allele agreement is required. A twin is dropped only when the partner position carries the same ref/alts. Partner coordinates say "same place"; they do not say "same variant", and a same-place different-allele pair is a real finding that survives whole.
  • The verdict is per locus. XG runs out of PAR1 and SPRY3 runs into PAR2, so a gene- or module-scoped policy would misclassify half of either. reference_examples/par_boundary/ is that case, and it demonstrates the round-trip fixed point — which is exactly why this belongs here and a --par compiler flag would be P7-illegal: resolution.csv travels with the module, a flag would not.

Relatedly, the frozen identity now carries the allele: base.derive_variant_key keys a coordinate variant as chrom:start:ref:alts (normalized) when an alt is present, so distinct alleles at one locus are distinct identities. Together these make the compiler's compile → reverse → compile a full fixpoint (artifact.digest, content_signature, and the provisional resolution_signature). See SCHEMAS.md (derive_variant_key, ResolutionRow.rsid_alternates) and reference_examples/pathogenic_clinvar/README.md (the ClinVar dogfood these fixes came from).

Live Ensembl — REST (ensembl.py)

EnsemblResolver.resolve_rsid(rsid) -> (list[locus] | None, source | None) asks one backend, REST (_rest_rsid, DEFAULT_REST_ENDPOINT = rest.ensembl.org/variation/{species}/{rsid}), and parses its GRCh38 mappings into loci; resolve_rsid_rest is the same request under the name the placement witness asks for (an optional capability of an injected resolver, RestWitness). tenacity (@retry, exponential jitter, 3 attempts) retries a transport error, a timeout, a 5xx and a 429, each attempt paced by the resolver's PacingGate (EnsemblSettings.interval, 0.1 s against Ensembl's 15 req/s), and a retry waits at least what the answer's Retry-After asked for (net.wait_retry_after). A wait past net.RETRY_AFTER_CEILING (10 s, about the longest backoff; Ensembl's hourly budget answers with the time to its reset) is not slept: the request is could-not-ask, and the resolver asks nothing until that time has passed.

The beta GraphQL leg is retired (2026-11-04). beta.ensembl.org/api/graphql/variation, its query leeched from ensembl-mcp, was POSTed before every REST request. Probed by hand that day (19:19Z): the endpoint answers 301 to www.ensembl.org/api/graphql/variation; there the query no longer validates (Cannot query field 'region_name' on type 'Location', HTTP 400); and even a valid query answers a bare rsID with VARIANT_ID_NOT_FOUND, because that API wants a composite region:pos:rsid id. It never answered a bare rsID, so it cost a request per rsID and handed every one to REST. It was removed with its settings (EnsemblSettings.graphql_endpoint, human_genome_id) rather than pointed at the new host: nothing it could say there is a question this resolver asks.

Three outcomes, because two of them used to be one (S20, 0.5.4). A non-empty list is an answer; [] is also an answer — Ensembl was reached and has no GRCh38 locus for this rsID — and None means Ensembl could not be asked at all, so its answer is unchecked rather than empty. Fusing the last two into ([], None) made a failed request read as a definite negative, and loci: [] beside "live Ensembl has no GRCh38 locus for it either" is exactly the fingerprint of a fabricated identifier: a consumer checking which rsIDs in a machine-written document were real put two published variants (rs6567160, a long-standing MC4R BMI locus, and rs13010010) in the fabricated pile, and caught it only because five-of-seven succeeding looked more like flaky egress than a 30%-honest document. This is the tri-state rule the rest of the tree keeps — an unreachable source reports unknown, never the negative.

Two boundaries. A 4xx is an answer, not a failure: Ensembl 400s on rsIDs it cannot resolve (rs3216883, which dbSNP reports as merged), so only a 5xx, a transport error or a timeout returns None. And an answered-empty carries its source, so hint.checked records ensembl-rest when Ensembl was reached and said nothing — the old code's only trace of that case was a missing element in a set, which is unreadable in practice.

A one-sided indel is anchored before it becomes a locus, and a fourth outcome withholds it (RM268, 2026-09-27). REST spells an insertion -/C with start = end + 1 and a one-sided deletion AGTAAG/- over [start, end]; _loci_from_rest used to copy both through, so rs8176719 resolved to 9:133257522 ref='-'. It now prefixes every allele with the GRCh38 base at start - 1 (clingen_allele.anchor_indel, reading through the private EnsemblResolver._read_base, by default a lazy SequenceProxy), giving 9:133257521 T>TC. An allele string with no - is already anchored and reads nothing. When the base cannot be read the locus is withheld, and an answer whose every locus was withheld returns (None, "ensembl-rest"): unchecked like a failed request, but enrich names it in its own warning (a structured field is RM272, minor), and lookup gives it its own finding, because the request did not fail.

A non-base allele is not a locus, at either rung (RM271, 2026-09-27). REST serves the literals dbSNP_novariation and dbSNP_variant as an allele_string; the VCF-dump snapshot serves an empty alt, <.>, IUPAC placeholders (<R>, <Y>), N runs and N-masked chrY refs. One private predicate, ensembl._placeable_alleles, runs at every read of both (REST and the four snapshot reads in resolver): a ref outside ^[ACGT]+$ withholds the locus, an alt outside it is dropped, and a locus with no alt left is withheld. REST with nothing left answers the ordinary [] (it is permanent, so no re-run helps), and the snapshot names such rsIDs in one warning instead of rsid_unresolved.

A row with no position is not an answer (RM452, 2026-11-18). A snapshot row naming an rsID with no chrom or start is left out of the scan's answer, named in one warning, and the next link is asked. A back-fill probe with no start gets no candidates, as one with no chrom does. No published release has shown one.

REST serves an older dbSNP build than the snapshot of the same release (RM447, 2026-11-17). At Ensembl 116, REST's /info/variation names dbSNP 156, and so does the VEP cache line in the split snapshot's header, while the VCF dump the snapshot is built from (2026_07_13) declares ##source="dbSNP" … version="157". So 54 of the 454 reference-example rsIDs that both answered carry alleles in the snapshot that REST lacks, and none go the other way. rs646776 is C>A,G,T in the snapshot and in dbSNP 157, and C/G/T in REST. The 13 rsIDs only the snapshot has are ids new in 157. The enricher widens no REST answer from another source. REST answers a subject the snapshot holds only when the run has no snapshot (download=False on a host without one, or a provisioning that failed), so a row with source=ensembl-rest may hold the older set. The compiler's allele refusal says so for such a row, and the remedy is a re-derivation where the snapshot is provisioned. RM376 found no gap for rs746071566, whose alleles did not change between the builds.

checked carries labels only, and snapshots is where a path lives (S93, RM205). The set used to be mixed — ensembl-rest for the live leg beside an absolute path for a snapshot, and the unreadable-snapshot finding interpolated the same path — so a host serving lookup_variant over HTTP mapped every known path back to a lane name, inside prose too, an audit that has to be repeated on every field added. Now checked holds the lane's name (ensembl, clinvar) or the live source, the finding reads ensembl snapshot unreadable: …, and VariantHint.snapshots maps each label to the path it resolved to for every snapshot the lookup opened or tried to (ensembl, clinvar, pubmind) — the one field a host drops to keep its layout private. duckdb's own first line may still name a file; that is upstream's sentence and stays as evidence.

The REST endpoint, species and timeout are configurable via EnsemblSettings.

The caches

Adding a lane? CACHE_SURFACE.md is the checklist, and every line in it names the test that asserts it — written after a lane shipped with a publish_repo no command could reach.

Parquet snapshots, one base directory, one rule: locate, never download — except where you ask. (The count used to be stated here and went stale twice while the table below grew; a number in prose is a registry nothing iterates. Since RM176 the roster is caches.CACHE_LANES — a walked registry with one entry per lane, compared by a test against the *_build modules on disk, so this table is a rendering of it rather than a third copy. Three lanes were missing from the previous list and nothing could have noticed: cache status reported nine caches on a machine that has twelve.) The Override column below is CacheLane.env_var rendered (S89, RM184), and the constants behind it live in locations as <LANE>_CACHE_VAR beside CACHE_BASE_VAR, the one variable no lane owns. A consumer generating a .env.template, clearing its test environment, or auditing which caches were provisioned by variable derives the whole set as {lane.env_var for lane in CACHE_LANES} | {CACHE_BASE_VAR} rather than keeping the names by hand. The status half is caches.lane_status() (S91, RM204), one LaneStatus per lane in registry order, and cache status renders it — a consumer serving the same answer over HTTP reads the function rather than re-writing the loop, which is two projections of one registry. Its state is three-valued: present, absent, and occupied — the place the lane looks exists, is non-empty and holds no snapshot (a build that failed after its downloads, a payload deleted beside its release.json, a stray .part, a foreign parquet). That third state used to print as absent, which sends an operator to run a pull that prepare is going to refuse, since provisioning never deletes. looked_in says which directory the verdict is about (the lane's env_var if set, else the default), release is what the snapshot names, and release_unreadable is the present-and-unreadable release.json — a provenance failure, not a data failure. A lane declares its size and a present one measures it (S97, RM229). CacheLane.approx_mb is an order of magnitude in whole megabytes, measured on a provisioned box on 2026-09-11 and rounded up (1 means at most a megabyte; Ensembl is ~15 000, the AVI lane ~30 000), so a first-run offer can decide does this fit, is it worth asking without du-ing a box that already has it; None means nobody has measured, which a caller reports as unknown rather than guessing. A test re-measures every lane present on the machine it runs on and refuses a declared number more than an order of magnitude off, which is what keeps the field from becoming the hand-kept table it replaces. LaneStatus.size_bytes is the measured size of a lane you already hold, and cache status prints it. A derived lane's price is its parents': provisioning_closure(lane) is the lane plus its transitive parents in registry order — mitomap_miss is under a megabyte and its closure is a ClinVar download. Every live source this tier reaches has (or can have) a local copy, and the whole reason is in the rate table above: a shared IP shares one budget. An author on their own machine can go live for everything; a host cannot, and for the three licence-gated sources it should not (see On a host, or in a service below). Pre-provisioning is therefore a deployment step, not an optimization.

Cache Subdir Override ensure_* Published at Serves
Ensembl ensembl_variations/ $JUST_DNA_ENSEMBL_CACHE ensure_snapshot just-dna-seq/ensembl_variations_split (since 0.8; built by prepare-annotations) rsID → coordinate (enrich)
ClinVar clinvar/ $JUST_DNA_CLINVAR_CACHE ensure_clinvar_snapshot just-dna-seq/clinvar records + citations/ (enrich, draft-panel)
gnomAD constraint gnomad_constraint/ $JUST_DNA_GNOMAD_CONSTRAINT_CACHE ensure_constraint_snapshot just-dna-seq/gnomad_constraint v4.1 gene constraint (gene-metrics)
ClinPGx 🔒 clinpgx/ $JUST_DNA_CLINPGX_CACHE ensure_clinpgx_snapshot just-dna-seq/clinpgx clinical annotations (clinpgx check)
CPIC 🔒 cpic/ $JUST_DNA_CPIC_CACHE ensure_cpic_snapshot just-dna-seq/cpic alleles / diplotypes / recommendations (pgx, draft)
PharmVar 🔒 pharmvar/ $JUST_DNA_PHARMVAR_CACHE none, by design never published star alleles (pgx)
PubMind 🔒 pubmind/ $JUST_DNA_PUBMIND_CACHE none, by design never published literature-derived verdicts (RM134)
CIViC ✅ civic/ $JUST_DNA_CIVIC_CACHE ensure_civic_snapshot just-dna-seq/civic — CC0, repo not created yet curated cancer interpretations, direction axis only (RM152); --submitted widens the basis from the release's own VCF (RM169)
MANE ❓ mane/ $JUST_DNA_MANE_CACHE none, by design never published the transcript numbering frame — summary, changed Select accessions, excluded genes (RM168)
ClinPGx drug labels 🔒 drug_labels/ $JUST_DNA_DRUG_LABELS_CACHE ensure_drug_labels_snapshot just-dna-seq/clinpgx_drug_labels — repo not created yet what five regulators say about a gene/drug pair (clinpgx check-labels)
STRchive ✅ strchive/ $JUST_DNA_STRCHIVE_CACHE ensure_strchive_snapshot just-dna-seq/strchive — MIT, repo not created yet repeat-locus bands (check-repeat-bands, draft-repeats)
ACMG SF ❓ acmg_sf/ $JUST_DNA_ACMG_CACHE none, by design never published the secondary-findings list (check-acmg)
MITOMAP ✅ mitomap/ $JUST_DNA_MITOMAP_CACHE ensure_mitomap_snapshot just-dna-seq/mitomap — CC BY 3.0, repo not created yet curated mtDNA variants, both mmutation and rtmutation (RM171)
MITOMAP miss ⛓ mitomap_miss/ $JUST_DNA_MITOMAP_MISS_CACHE none — derived not published, deliberately what MITOMAP publishes and the ClinVar cache does not (draft-panel --source mitomap-miss)
AlphaGenome AVI 🔑 alphagenome_avi/ $JUST_DNA_ALPHAGENOME_AVI_CACHE ensure_alphagenome_avi_snapshot just-dna-seq/alphagenome_avi — Permissive Use (RM195), repo not created yet variant-impact scores for 8.8 B SNVs, Int32×10⁵ plus a 466 KB knot table (alphagenome build --input, RM191/RM198)
ClinVar-GKM ✅ clinvar_gkm/ $JUST_DNA_CLINVAR_GKM_CACHE ensure_clinvar_gkm_snapshot just-dna-seq/clinvar_gkm — CC0 (ClinGen's transformation), repo not created yet the GRCh38 indels of ClinVar-GKM with VRS id, VCF spelling, ClinVar variation and rsID: a placement witness a check names (RM305; clinvar-gkm build, currency, lookup)
Open Targets ✅ opentargets/ $JUST_DNA_OPENTARGETS_CACHE _ensure_opentargets_snapshot (private in the 0.8 line, RM408; reached by cache pull) just-dna-seq/opentargets — CC0-marked, published by cache rebuild --only opentargets --publish, repo not created yet Platform 26.09 trimmed: GWAS credible sets and members, L2G, GWAS × molecular-QTL coloc, per-transcript consequences (opentargets enrich, RM333)

🔒 = licence-gated (commercial_use=False). ❓ = terms unestablished (commercial_use=None), which is a different state and not a weaker one: unknown is not permissive. ⛓ = derived: this lane has parents rather than a download. 🔑 = the acquisition itself is gated, which is a fourth thing again (RM191): the AVI artifact is 88.5 GB behind a sign-in whose eligibility clause bars classes of holder — "the AlphaGenome Services aren't available for any commercial entity, even if conducting non-commercial work" — so there is nothing for a rebuild adapter to fetch and the operator supplies the file under their own acceptance (@acquisition-gate-is-not-a-read-gate). It is the second lane with no rebuild adapter and the first with a builder anyway: having a builder module and being rebuildable stopped being the same property here. Since RM198 it is also pullable without being buildable — this tier may not fetch the gated source, but the 34 GB re-encoding is Permissive-Use output (RM195) and redistribution=True records the reading that publishing it openly falls inside prohibition 1's carve-out, so an operator who cannot download the artifact can still provision the lane. Its snapshot is the first that is incomplete without a root-level file: PHRED is not stored, so avi_knots.parquet travels beside data/ and locations.SNAPSHOT_ROOT_FILENAMES is what makes the publisher carry it. The three PGx 🔒 rows are RM38, new in 0.5.1; the PubMind 🔒 row is RM134, ❓ until WGLab answered on 2026-09-28 (RM320), and the MANE one is RM168; the two MITOMAP rows are RM171.

The derived lane, and the parents field (RM171)

mitomap_miss is the registry's first lane that fetches nothing. Its acquire stage is both parents being on disk — the MITOMAP snapshot and the ClinVar one — and its build is an exact (start, ref, alt) join on chrMT, upper-cased on both sides, with no position-level fallback: a hit at the same position on a different allele is a different allele, and collapsing onto it would hide a real increment or invent one where the two sources anchor an indel differently.

CacheLane.parents is a tuple of lane names, empty for every lane that acquires its own bytes. It is a correctness fact — which digests get recorded, what rebuild refuses on — and it is also a cost fact, which its first reader did not expect (S97): the increment is under a megabyte built, and on a blank box its honest price is its parents, a ClinVar download among them. provisioning_closure walks it transitively for exactly that sum. Three things read it and only one is the join:

  • rebuild_lane guards on it. A child whose parents are not on disk is built=None naming which one, with the command that would provision it. The two wrong answers are both silent: a False files another lane's absence as this lane breaking, and an empty increment is the strongest possible claim about MITOMAP — it publishes nothing ClinVar lacks — derived from a comparison that never ran.
  • The registry order is load-bearing, because cache prepare walks it top to bottom. A parent must precede its child, and a test asserts it rather than a comment.
  • cache rebuild hands the child the parents it just cut, under out/<parent>/, so a fresh set is internally consistent. --only mitomap_miss names none and falls back to the resolvers, which is the right answer for a run with no rebuild behind it.

The child pins both parents in its release.json, which is what makes a ClinVar rebuild without a child rebuild detectable rather than silent — mitomap_miss_build.stale_parents re-reads both and names the one that moved. A parent that is gone is deliberately not reported as one that moved: "provision the parent" and "rebuild the child" are different instructions.

It is not published, and the reason is a fourth one. Both parents are redistributable — ClinVar is public domain, MITOMAP is CC BY 3.0 — so nothing in the licensing bars it. What bars it is what the artifact is: a pulled copy would carry a currency check its holder cannot run, having neither parent. Rebuilding it locally is seconds against caches the machine already has, and it cannot be stale by construction.

MANE's ❓ is an unestablished permission, and the third column says so. MANE's bytes carry a policy — NCBI places no restrictions on use or distribution and, in the same paragraph, declines to grant unrestricted permission. That is not permission, so the cache is operator-built. PharmVar's and PubMind's absent ensure_* records a refusal (a personal key; NoDerivatives), CIViC's records a gap, and MANE's records an unestablished question nobody has asked NCBI. PubMind sat beside MANE as the other ❓ until WGLab answered (RM320).

One base, so a single just-dna-lite deployment's cache serves all of them. Each subdir sits under $JUST_DNA_PIPELINES_CACHE_DIR, or platformdirs' user cache for just-dna-pipelines (~/.cache/just-dna-pipelines on Linux) when that is unset. Precedence per cache is explicit argument → its own $JUST_DNA_*_CACHE → the base, and a .env beside the working directory is loaded automatically (locations.load_env, walking up from CWD). Every resolver returns None rather than guessing when nothing is there.

That load writes into os.environ, and it is a library path rather than a CLI one — pass load_dotenv_file=False if that is not wanted. load_dotenv sets every variable the file holds, not only the cache ones, so a process that merely asks where the ClinVar cache is inherits whatever credentials sit in the nearest .env above its working directory. override=False means a variable already present is kept — which reads as the safe direction and has one sharp edge: deleting a variable is what lets the file supply it, so a test isolating itself with del os.environ[...] (or monkeypatch.delenv) is un-isolated by the next resolve. Set it empty instead; that is the same rule the enricher's own tests follow. Every resolver and every default_*_cache_dir takes load_dotenv_file, and since 0.6.3 passing False really does reach the load — before that it reached none of the six, because the default directory is computed as an argument and loaded the file on its own way in (S39). The credential paths export nothing since RM301 (S124). Every client, the PharmVar guard, the AlphaGenome key, the HuggingFace token and the retry floor read their one variable through locations.env_value: the process environment first, then the nearest .env, with load_env's precedence and without writing into os.environ. So constructing a client no longer hands its host the rest of the file. Only the cache resolvers above still export, and they take the switch. RM102 carried the whole question and closed on 2026-08-21 as a decision not to act. It was reopened as RM301 when a host's own configuration record turned out to be the boundary its trigger named.

Inside a cache the layout is fixed, because four parties have to agree on it — builder writes, publisher uploads, provisioner fetches, reader queries — and every past disagreement was silent:

<base>/<subdir>/
  data/*.parquet          # the records; the readers glob exactly this
  citations/*.parquet     # optional sidecar, a SIBLING of data/ (ClinVar only)
  release.json            # which release this is — what reference_sha256 pins against (RM4)
  LICENSE.txt             # the terms, for a snapshot that ships its own (ClinPGx and its labels)

Two caches hold no parquet at all, and that is a second layout rather than a defect in this one: ACMG's list is acmg_sf.csv and STRchive's catalogue is STRchive-loci.json, each at the snapshot root beside release.json. Both are read whole rather than queried, and a parquet for eighty-one rows would be ceremony. This is why they had no resolver until RM176 — every predicate in locations tested for data/*.parquet — and why plan_reference_snapshot takes a payload filename from its caller rather than assuming one shape for every lane.

<base>/acmg_sf/acmg_sf.csv + release.json
<base>/strchive/STRchive-loci.json + release.json

Pre-caching the published snapshots from HuggingFace

just-dna-enricher cache status                       # what is present, where, which release
just-dna-enricher cache pull                         # every published lane with no `--use` gate
just-dna-enricher cache pull --use non-commercial    # …and the gated ones you may hold
just-dna-enricher cache pull --only clinvar --only cpic --use non-commercial

cache pull is re-runnable and cheap: a cache whose pull record says the pull finished costs one small read, the release its repo publishes, and anything else fetches only what it lacks. The record (pull-record.json beside data/) is written from the remote listing before the first download, so an interrupted pull names the files it never got, every resolver reports the lane absent rather than present, and the next run finishes it (RM339).

A pulled lane follows the hub's release (RM420). cache pull and cache prepare compare the cache's release label with the one its repo publishes, both read by the lane's own CacheLane.release_label (release.json's dataset, Ensembl's manifest release_id). Where they differ and the pull record says the cache came from a pull of that repo, it is re-pulled down the repo-switch path: a file is kept only where the new release's manifest attests it, and without a manifest none is; release.json is withdrawn before the first byte moves and fetched last, so a re-pull cut short states no release and the next pull finishes it. No lane owns an ordering over its labels, so a pulled cache follows the hub whichever way the labels differ. A cache built on this machine is left alone and printed with both labels, because between a build and its upload the local lane is the newer one: a builder writes no pull record, and the record states the sha256 of the release.json the pull placed, so a builder's release written over an old pull is not taken for it. A record from before RM420 carries no digest and is read as the pull it was, but not as attesting its label (RM445): the pull that wrote it placed the hub's release.json over whatever it kept, so cache pull re-pulls it even when its label equals the hub's. A hub that cannot be read leaves the cache as it is, with a warning. enrich's own ensure call never reads the hub for a whole cache: a warm cache serves offline, and no enrichment downloads a release mid-run. STRchive's pull writes a record since RM420 too; a catalogue pulled before then carries none, so it is left alone and named with both labels until it is moved aside and pulled once. A cache with no record, built here or pulled before the record existed, is compared with the published listing once; with no route to the Hub it is used as it is, with a warning that its completeness is unknown.

A pull never places the hub's release.json over a file it kept and nothing attests (RM445). A kept file is attested by the release's manifest.json, or by this repo's own pull record saying the files on disk are the release the repo still publishes; the record written before the first download names the release being pulled, so a resumed pull keeps what it fetched only while the hub publishes that same release.json. cache pull and cache prepare fetch an unattested file again (reported as re-pulled); enrich keeps it, places no release.json, and the cache states no release, so its label is withheld rather than false. avi_knots.parquet follows release.json; LICENSE.txt is fetched either way. An unrecorded cache that states a release of its own is a builder's and is never refetched: left alone with another label, kept as it is with the hub's. A pull never fetches beside a local build (a release.json no pull of this repo placed), on either path: a build lacking a part the repo publishes (citations/, say) is left untouched, its release.json kept and nothing removed or fetched, and every run warns that the part could not be reached, naming the remedies (rebuild it with the part, or move it aside and cache pull); cache pull reports it as left alone. A pull's own release.json beside files it no longer vouches for is withdrawn before the first byte, and the next cache pull re-pulls. The record's release_sha256 names only a file the pull placed (null otherwise). A release.json that could not be fetched stops the pull before anything in the cache moves (it is asked before the manifest is placed), as a manifest that could not be fetched does; a repo that publishes none pulls as before. A current label is checked against the listing (RM445). For a pulled cache (never a build) whose label equals the hub's, cache pull and cache prepare compare every kept file with fs.ls(..., detail=True): its size, and for a file of at most 256 MiB the LFS sha256 (lfs → sha256). A contradicted or absent file is fetched again, the label withdrawn first; an unreadable listing leaves the cache as it is (unknown). Above the bound, and for a file stored in git, only the size is compared. enrich does not run the check: a whole recorded cache stays network-free. A download that times out names HF_HUB_DOWNLOAD_TIMEOUT (10 s by default, short for a 1 GB file). One snapshot failing does not sink the rest — each reports its own line, and the command exits 1 if any failed.

Four things worth knowing before you run it on a server:

  • Set HF_TOKEN. Anonymous traffic shares a per-IP pool (500 API / 3,000 resolver calls per 5-minute window); a free token roughly doubles it. The usual symptom of not having one is an ensure_* that looks hung — the client is sleeping on a 429, not stuck. A .env beside the working directory is enough: both provisioners and the publisher call load_env() before get_token(), which otherwise reads only the real environment and ~/.cache/huggingface/token.
  • --use is required for the gated pair, and it is not ceremony. ClinPGx and CPIC forbid sale, and under a data-usage policy the terms are accepted when the data is taken — so downloading is the act being gated. unstated skips them with a reason, commercial refuses, non-commercial proceeds. Same three states as everywhere else; the tool will not assert a purpose for you.
  • Five of the repos have to exist first, and a missing one is not an error. just-dna-seq/cpic and just-dna-seq/clinpgx are new with 0.5.1; civic, strchive and clinpgx_drug_labels with RM176 — none of the five has been published yet. Each provisioner raises SnapshotNotPublished when the repo or its data/ does not exist, and cache pull prints that in yellow without counting it as a failure: nobody-published is the same third state as nobody-asked, and the command that provisions a deployment must not exit 1 on a fresh machine because a snapshot has never been uploaded. A download that breaks is still a failure and still exits 1. Build and publish them once (below), or point at a locally built directory. Since RM395 such a pull leaves no directory behind (an empty one read as occupied to cache status and was refused by cache prepare), and its closing line names what the run left on disk and which lanes had nothing published, never the registry's list of pullable lanes.
  • A published dataset accumulates. Each ensure_* fetches only the files its own snapshot is made of, because the ClinVar repo still carries a 159 MB clinvar.parquet from the single-file era whose columns are raw VCF INFO fields. The readers glob data/*.parquet, so one foreign file puts two schemas under one DuckDB relation and every query dies on Referenced column "clin_sig" not found.

If you prefer the Hub CLI, the layout is plain and the same files are all there is:

hf download just-dna-seq/clinvar --repo-type dataset \
    --include 'data/clinvar-*.parquet' 'citations/*.parquet' 'release.json' \
    --local-dir "$JUST_DNA_PIPELINES_CACHE_DIR/clinvar"

Note the --include: data/*.parquet alone would drag in the stale flat file above. In Python it is from just_dna_enricher.download import ensure_clinvar_snapshot; ensure_clinvar_snapshot(), which does the filtering, the footer check and the atomic rename for you.

Building the ones that are not (fully) published

(Another count that went stale as the table grew — CIViC's builder, documented in its own section below, was already a fifth. The list here is the roster.)

# CPIC — open and unauthenticated, so this is about a host's shared budget, not access.
# Every builder writes data/repro/<lane>/ unless --out names somewhere else (RM177).
just-dna-enricher cpic build --use non-commercial                             # 132 genes, ~120k rows, ~256 KB
just-dna-enricher cpic publish data/repro/cpic --repo <org>/cpic              # optional; redistribution is granted

# ClinPGx — the bulk archive; its LICENSE.txt is extracted and travels with the parquet.
just-dna-enricher clinpgx build --use non-commercial
just-dna-enricher clinpgx publish data/repro/clinpgx --repo <org>/clinpgx

# PharmVar — needs YOUR key, and there is no publish command.
PHARMVAR_API_KEY=… just-dna-enricher pharmvar build --use non-commercial

# PubMind — no key; CC BY-NC-ND, so `--use` gates it; `pubmind publish` exists and refuses (see below).
just-dna-enricher pubmind build --download --use non-commercial                                 # or --table hg38_pubmind_db.txt.gz

# MANE — no key, no `--use`, no publish command. Discovers the newest version from current/ and
# then pins it to release_<version>/ before fetching anything.
just-dna-enricher mane build --download                                     # or --release 1.5 to pin by hand

# The regulator drug labels — a SECOND ClinPGx archive with its own cadence, so its own cache and
# its own repo. Same CC BY-SA terms as the annotation lane, so publishable on the same grounds.
just-dna-enricher clinpgx build-labels --use non-commercial                 # → data/repro/drug_labels
just-dna-enricher clinpgx publish-labels data/repro/drug_labels

# STRchive — MIT, so publishable. Pin a release: the default branch moves, and only a pinned build
# gets a `dataset` label the comparison can name.
just-dna-enricher strchive build --release v2.26.0
just-dna-enricher strchive publish data/repro/strchive

# ACMG SF — nothing is fetched. The workbook is ACMG/Elsevier supplementary material, so you supply
# your own copy; there is no publish command, because nothing grants redistribution of those bytes.
just-dna-enricher acmg build ./acmg_sf_v3.3.xlsx                            # → data/repro/acmg_sf

cache prepare — the whole set, by whichever route each lane has (RM176)

just-dna-enricher cache prepare --use non-commercial        # the deployment's one provisioning step
just-dna-enricher cache prepare --only mane --only acmg     # or a subset

cache pull fetches the published snapshots and stops, and the lanes with no publish_repo are not published, each for a recorded reason — among them PharmVar's personal key, PubMind's NoDerivatives, NCBI's policy over MANE, ACMG's supplementary material, and mitomap_miss, which is derived from two parents on the pulling machine and would only pin somebody else's ClinVar if it travelled. A machine that only pulled is therefore those caches short, and the checks that read them skip themselves with no_reference. prepare runs each lane by the route it has: pull if published, build if not.

  • The route is a property of the lane, never a flag. A published lane pulls, because building it would spend an operator's bandwidth re-deriving bytes somebody already made; an unpublished one builds, because that is the only route there will ever be. PrepareOutcome.route records which answered — present, pulled, built — because a deployment auditing its own caches has to tell a snapshot it fetched from one it made, and release.json names the release but not the route.
  • A present cache is left alone unless it came from a pull and its repo now publishes another release, exactly as cache pull treats one (RM420), so this is idempotent. Re-cutting a snapshot that already exists is cache rebuild, which writes somewhere else on purpose — a build straight into a live cache is visible half-done to anything reading it, and unlike a truncated download there is no footer check to catch it because the file is real.
  • A built lane is staged and moved. It builds into <cache>.incoming and renames across only once the build finished; a failure removes the staging directory rather than leaving a half snapshot a later cache status would call present.
  • In Python: caches.prepare_caches(...), which the command calls. caches.rebuild_caches(...) is its sibling. Both return one outcome per lane in registry order, so a caller can zip against CACHE_LANES without matching on names.

Every bulk download goes through one body, and a failed one is a failed lane (RM187). Each builder streams its source's file through net.stream_to_file, which is atomic (.part, renamed only on success), retries a transport failure — a connection cut mid-body, the failure that produced this item — and translates everything else into the lane's own error type. A status error is not retried, because a 404 from a mistyped release tag is the same 404 four times over. A 410 is the lane's own *Retired, which every caller passes as retired= (required, so no lane can leave it out; RM361's audit round 2): the source moved the file, and the message says so and to upgrade, quoting the body. Every bulk download has a timeout of 120 s unless its caller names its own (RM421; not a setting). It bounds connect and the gap between two chunks, not the whole download, so a slow mirror that keeps sending is never cut off, while a server that accepts the connection and sends nothing raises a read timeout, which is retried as a transport failure and then translated like any other. Before RM421 the default was no timeout at all and eleven of the fourteen downloads named none, so such a stall blocked a build forever.

That last part is what a caller notices. Before RM187 four of the eleven downloads raised httpx's own exception, and each lane adapter catches its own builder's type — so a truncated download was a traceback rather than a built=False, and it escaped rebuild_lane too, aborting every lane after it. If you catch httpx.HTTPError around one of these builders, catch the lane's error instead; the httpx exception is kept as __cause__. A retry re-fetches from byte zero: there is no resume, so a 190 MB download that dies at 180 MB costs the full 190 MB again.

Two refusals a payload-less directory earns (2026-09-09). resolve() answering None means the lane's directory holds no payload, not that it is absent. A directory that exists with none — a build that failed after its downloads, a payload deleted by hand beside its release.json — is refused on the build route before any build is spent, naming the directory and cache prune, because provisioning never deletes and the rename onto it would have raised Directory not empty out of the whole command. And a derived lane judges each parent by that same resolver: an empty out/<parent>/ left by a download cut mid-body is a missing parent (the child could not run), never a present one whose join then fails and files the child as FAILED. prepare_caches isolates each lane the way cache pull always has, so one lane raising is that lane's FAILED line and the rest still run and print.

One endpoint over every builder (cache rebuild, RM176)

Thirteen builders today — eleven when RM176 shipped, plus mitomap and mitomap_miss from RM171 the next day — three stages each — acquire, build, publish — and until RM176 the only way to run them all was one command per lane, each with its own flag shape. cache rebuild is the single endpoint, and it calls the same download_*/build_* functions the per-lane commands call, so there is one conversion algorithm with two callers rather than two that have to agree. The per-lane commands stay: they offer the local-file inputs an operator holds, which a rebuild pass by definition does not.

just-dna-enricher cache rebuild --use non-commercial              # → data/caches/<lane>/
just-dna-enricher cache rebuild --only strchive --only mane --pin mane=1.5
just-dna-enricher cache rebuild --only acmg --source acmg=./acmg_sf_v3.3.xlsx
just-dna-enricher cache rebuild --publish --dry-run               # the plan: reads, takes nothing
scripts/rebuild-caches.sh data/caches                             # the driver, all lanes
just-dna-enricher cache rebuild --out /srv/just-dna/caches-$(date +%F)   # a deployment's own path

--pin lane=release is taken by seven lanes and refused by the rest (RM416): acmg, civic, clinvar_gkm (a check: ClinVar-GKM serves only its current set, and a pin naming another refuses the build), constraint, mane, opentargets and strchive (caches._PINNABLE, held equal to the adapters that read the pin). The other lanes fetch the source's current release only, and a pin for one is refused before the run starts. Until RM416 nine of them ignored it, so --pin constraint=4.1.2 built 4.1 and said nothing.

--dry-run downloads, builds and uploads nothing (RM411; until then it ran the whole rebuild and skipped only the upload, so rehearsing --only opentargets --publish started a 22 GB download). Per lane it says what a run would build, from where (the download, a --source file, or a derived lane's parent caches) and into which directory; not run, refused or skipped in the real run's own words; the release this machine's cache holds; and with --publish, the release the lane's repo states, or nothing published, or could not ask. The upload's own checks need the built bytes, so a lane already built into <out>/<lane> by a run without --publish has its upload rehearsed against that snapshot. The source's own newest release is not asked: the liaison's docs/UPSTREAM_DATA_SOURCES.md keeps it by hand, and RM412 is the opt-in download rehearsal.

Three things about it are worth reading before a deployment runs it nightly.

  • Every lane builds into <base>/<lane>/, never in place over a resolved cache. A rebuild takes minutes, and a short parquet still has a PAR1 footer — so an enrich reading a half-written snapshot mid-flight sees a real but incomplete table and no resolver can catch it. Moving the result into the live caches is a separate, deliberate step.
  • Every --out default is under data/, and none of them is written by hand (RM177). cache rebuild takes data/caches/; every builder takes data/repro/<lane>/ from locations.repro_out. That is not cosmetic: run from a checkout, --out ./cpic drops an untracked snapshot directory in the repository root, which is exactly the state civic reproduce needed its own .gitignore line to paper over. The rule was prose for a release and nine defaults drifted past it — civic, clinvar, pubmind, gnomad_constraint, mane, strchive, acmg_sf, mitomap, mitomap_miss — with four more builders requiring --out and no default at all, which is how the docs came to tell an operator to write --out ./clinpgx into the root. It is derived now and an AST walk over the CLI asserts it, so the tenth builder inherits the rule instead of repeating the defect. A deployment passes its own absolute path and none of this applies.
  • The outcome is three-valued, and not run is not a failure. ACMG needs a workbook that is Elsevier supplementary material, PharmVar a personal key, CIViC a release date to pin, Ensembl is built by prepare-annotations, and a derived lane whose parents are not on disk (mitomap_miss, RM171) names the parent it lacks rather than reporting an empty result. Each prints its own reason — taken from the registry field, not composed here — and the exit code counts only real failures, so a nightly run does not alarm on lanes behaving exactly as designed.
  • Credentials come from the .env too, and until RM176's follow-up two of them did not. $PHARMVAR_API_KEY and $HF_TOKEN are read at the point they are used (@credential-where-read), through env_value since RM301, so a workspace that keeps them in a .env beside the working directory needs nothing exported and the rest of the file is not exported either. An exported variable still wins, as it did under load_env's override=False. The two that were missing it failed in opposite directions: the PharmVar guard read os.environ in front of a builder that does load the file, so the lane reported "no key" and never built on the machine most likely to have one; _hf_api refused a publish outright.
  • export FOO= is stronger than unset FOO, and both messages now say so. override=False keeps a variable that is present, and an empty string is present — so a shell that ran a snippet whose placeholder was edited out reports no key for the rest of the session while the .env holds a working one. Absent and exported-empty are named separately, with different remedies: add it, or unset it. The same trap the tier's own tests exploit deliberately, from the other side.
  • The ACMG lane uses the checkout's own workbook when nobody names one. assets/acmg_sf_v*.xlsx travels with this repository, so a rebuild run from a checkout needs no --source — it is found by walking up from the working directory and, for an editable install run elsewhere, relative to the package. A pinned filename is deliberately not used: the list is versioned, v3.2 became v3.3 in June 2025, and a constant naming one version stops finding the asset the day the next lands — silently, in the direction that falls back to scraping NCBI's stale page. Two matching workbooks are reported rather than ordered, because "the highest version" is an ordering nobody defined and filename sort is not it. assets/ is not in the wheel, so a pip install still supplies its own copy and gets the operator-supplied message: the workbook is ACMG/Elsevier supplementary material, and shipping it in a published package is exactly the redistribution question this lane's registry entry records as unestablished.
  • A --source path is checked before the run starts. Typer's exists=True cannot reach a value embedded in a lane=value string, so a mistyped one used to travel to the lane's builder and surface as a bare [Errno 2] — and acmg is last in the registry, so that was after everything else had downloaded. ~ is expanded, because no shell expands it inside an assignment.
  • --source lane=path is the offline off-switch for clinvar, constraint, clinpgx, drug_labels, pubmind, strchive and mitomap (a pg_dump you already hold; the snapshot is then honestly unlabelled), and the only route for acmg. mane and civic refuse it: each takes three input files, two of three is not a build for either, and a flag that can supply one would be a flag that cannot do its job.

The CIViC adapter fetches the three TSVs and no VCF, which is civic build's own default. RM169 made --submitted opt-in precisely because the release VCF widens the status basis — it admits submitted-but-not-accepted evidence — so a rebuild that fetched it unconditionally would build a different snapshot from the same release than the per-lane command does. Two callers, one release, two artifacts is the fork this endpoint exists to prevent; widening the basis is a curation decision, and civic build --release <date> --submitted is where it is taken. The rebuild's outcome line prints status_basis so the two are distinguishable at a glance.

PharmVar's not run is decided before the request, not from the failure. The service returns an identical 401 for an absent, a malformed and an unrecognised key, and PharmVarError is flat, so an adapter reading the message would be parsing prose. $PHARMVAR_API_KEY unset is the designed third state — the key is personal and non-transferable, so a machine without one is a machine PharmVar never meant to serve — while a key that is configured and then fails is asked-and-failed and counts.

Then point at them, or move them under the base directory so the default resolvers find them:

export JUST_DNA_CPIC_CACHE=data/caches/cpic
export JUST_DNA_PHARMVAR_CACHE=data/caches/pharmvar
just-dna-enricher pgx spec/ --offline --use non-commercial   # zero egress, both legs answered

Why PharmVar has no publish command, and will not get one. Its bulk data is pulled under a key its terms §2 make personal and non-transferable, and no axis SourceTerms records covers passing that on — redistribution=True describes the CC BY-SA grant over the content, not a clause about the account. An unestablished permission is never a permission, the same None ≠ False rule that governs share_alike and commercial_use. So the snapshot is operator-built and inject-only, its release.json says so ("redistributable": false), and the build command prints the same warning.

Why PubMind has no publish command either, and why its reason is a different one (RM134, RM320). PharmVar's bytes arrive under a key whose terms bar passing them on. PubMind's arrive under CC BY-NC-ND 4.0: WGLab answered in writing on 2026-09-28 (WGLab/PubMind#5) that the lightweight release ANNOVAR and Zenodo distribute is under that licence. The software and the full PubMind-DB stay under CHOP's LICENSE.md. The licence allows sharing the table as published, but the snapshot decomposes codon enumerations and normalizes significance tokens, which makes it adapted material, and NoDerivatives forbids sharing that. SourceRow has no axis for NoDerivatives, so it lives in PUBMIND_TERMS.notice.

pubmind publish therefore exists in order to refuse, rather than being absent: a missing command reads as an oversight somebody will helpfully add. It exits non-zero and names the licence, the clause, and the PharmVar precedent.

The terms forbid sale, so pubmind build gates on --use as clinpgx build does. commercial refuses, unstated skips and asks for a declaration, non-commercial builds, and cache rebuild runs the same gate on the lane. Reading the snapshot is not a fetch (@acquisition-gate-is-not-a-read-gate), so draft-panel --source pubmind is not gated. It writes a pubmind licence row with commercial_use=False, and the compile then refuses that module unless it records a non-commercial declaration. Until the answer every axis was None, the gate skipped under every declaration, and the command carried no flag. A module drafted before RM320 keeps its all-null pubmind row, because sources.csv merges never clobber a recorded row: delete that row and re-draft to record the terms.

The Ensembl snapshot's new repo, and a cache that mixes releases (RM372)

prepare-annotations published dbSNP 157 / VEP 116 on 2026-11-04 as a separate repo, just-dna-seq/ensembl_variations_split (tag ensembl-variation-2026_07_13-vep116.0): the same data/homo_sapiens-chr*.parquet names as the old repo, one row per ALT, left-aligned indels, groups/* sidecars outside data/ (not pulled), and a manifest.json with the release id and every file's size and sha256 in place of a release.json. Since the names did not change, three rules keep the two releases apart:

  • A pull record naming another repo is not this snapshot. ensure_snapshot returns a warm cache only when its pull record names the lane's current repo (download.SNAPSHOT_REPO_PREFIXES), and cache pull / cache prepare send a cache recorded from the old repo down the pull route. The resolver still serves such a cache: it is a whole snapshot of an older release, and an offline machine keeps it until someone pulls. enrich alone never migrates it.
  • A switch is recorded until it finishes. The pull record written before the first byte names the new repo and, under replacing, the old one; pull_record_repo reads replacing first and the key is dropped once every file is the new repo's, so an interrupted switch is still the old repo's cache. A sidecar or root file of the old repo is refetched or removed, never kept.
  • A file the manifest does not attest is never kept, and a cache holding one is withheld. The manifest is fetched first and placed before the first byte (it only ever refuses, so an interrupted pull leaves a cache whose old files it disowns). A file of unknown origin is kept only if its size and sha256 match; one this repo's own pull placed was hashed on its way in, so its size is compared; a download is renamed into place only if it matches. A manifest that could not be fetched (a timeout, a refused connection) is not one the repo lacks: the pull stops before anything is written. At read time manifest_coverage checks both directions, by size (a stat, since hashing 14.5 GB per enrich is not a read): a data/*.parquet the manifest does not name, or names at another size, is a mixed release; a data/ file the manifest attests that is not there is an incomplete snapshot (12 of 25 files with no pull record was served before). Either makes the resolver return None for the whole directory, before a prebuilt .duckdb is considered, with its own warning: "mixes releases" and "the snapshot is incomplete" name different fixes. Withheld whole rather than per file, because a dropped chromosome would read as asked-and-absent under --offline. An unreadable manifest attests nothing. No manifest (the old repo) is the unknown arm: the cache is read as it always was.
  • The release has a label. cache status prints the manifest's release_id for the lane (CacheLane.release_label = ensembl_release_label), and the rsid_coordinate_agreement record carries it as release.

Snapshot-first, live second, --offline first-only

Every pass follows one of two shapes, and which one it follows depends on whether a live route exists at all:

Pass With a snapshot Without one, online Without one, --offline
enrich cache provision, else live Ensembl/ClinVar/gnomAD cache only
gene-metrics snapshot (4.1.2 since RM410) provision, else live API (gnomad_api_constraint, no release named: RM417) snapshot only
pgx snapshot live PharmVar/CPIC skipped, with a reason
draft snapshot live CPIC skipped, with a reason
clinpgx check snapshot provision (no live route exists — the API was retired) skipped, with a reason
dosage, literature, frequencies — live no-op + warning (no snapshot exists)

The asymmetry in the middle column is deliberate. clinpgx provisions automatically because there is no live fallback to degrade to; pgx and draft fall back to live because there is one, and pulling a whole database to answer one gene would be the wrong default for an author on a laptop. Neither adds a second flag — --offline is the switch, and an explicit --*-cache / --snapshot path is the inject-only escape hatch, never second-guessed.

Which route actually answered is recorded, not implied: PgxResult.routes reports snapshot or live per source, and a snapshot stamps its own release into SourceRow.dataset (cpic_snapshot_<12 hex>), exactly as the two gnomAD constraint routes already distinguish themselves. A consumer must be able to tell a pinned file from a live API, because the two can differ by a release.

A pass that could run neither way is a third state, never a silent pass: PgxResult.skipped_offline and ClinGenResult.skipped_offline carry the reason, distinct both from "ran and found nothing" and from a failure.

The one cache that is not in the table

The ACMG snapshot used to be listed here, and the reason given was that it is "a single small CSV an author points at, not a shared reference". That reasoning was wrong in the way a design note is wrong when nobody re-checks it: an author points at it because nothing could find it otherwise, and the consequence was that the flagless path fell through to scraping NCBI's page — which serves v3.2 while ACMG published v3.3 in June 2025, so a correctly authored row came back reported as wrong. It has a resolver and a roster row since RM176.

  • No response cache for the live clients. NCBI, PharmVar's live path, gnomAD GraphQL, Crossref, Europe PMC, OLS4, HGNC and live Ensembl are paced only. Persistence is the authored sidecars (resolution.csv, frequencies.csv, …) — delete a sidecar to force a refetch, because enrich() treats an existing one as authoritative and merges into it rather than clobbering it.

When something is wrong with a cache

Symptom Cause Fix
Referenced column "clin_sig" not found a foreign parquet in data/ (stale layout, or an old builder) cache status, then move the file aside and rebuild
"present but not queryable" as above, or a truncated download remove the file; cache pull refetches it
cache status says occupied the directory the lane looks in is non-empty and holds no snapshot — a failed build, a stray .part, a foreign file move it aside (or cache prune --only <lane> for a retired file); prepare refuses to build over it, by design
ensure_* appears to hang anonymous 429 backoff set HF_TOKEN
a pass says a source was "skipped: --offline and no built snapshot" exactly what it says cache pull, or <source> build
repository not found for cpic/clinpgx nobody has published that snapshot yet build it locally and point $JUST_DNA_*_CACHE at it

The drafting scaffold — what every provider shares, and what it is allowed to differ on (RM228)

Seven *_draft.py providers turn a snapshot into authored rows, and they grew one at a time. By 0.7 each had its own copy of the same four decisions, and the copies had drifted: two providers recorded a release label and never withdrew a stale one, one imported another's private _MATCH_ON across modules, and one consumed pydantic's rendered error message as an API. drafting.py is the mechanism those seven were approximating.

The split the scaffold enforces. A provider's skip rule is two rules that were being written as one hand-kept list:

derived or declared why
the model's requirement derived, always VariantRow's rule is one fact; a provider restating it is a copy that drifts. pgx_draft once restated "no rsID and no position" where HaplotypeRow wants rsID or chrom+start, and draft --gene CYP2C9 died on an unhandled pydantic error
the source's precondition declared, with a reason a true fact about that snapshot — ClinPGx carries no coordinate, MITOMAP publishes no rsIDs. Legitimate, provider-specific, and useless to a reader without the reason

Mashed together nobody can tell them apart, which is the state mitomap_draft's clin_sig clause was in: it reads like an identity requirement, it is not one, and the reason it is nonetheless correct (a rated_miss carries one by construction, and the guard buys a named refusal instead of a raw ValidationError) sat three lines below in a comment. SourcePrecondition.reason is a field, so the two halves cannot merge again.

The identity verdict comes from constructing the model, not from authoring_requirements. That was measured rather than assumed. authoring_requirements("variants.csv") answers any_of: [['rsid'], ['chrom','start']], a grammar that cannot express VariantRow's third clause — ref/alts require chrom and start. A guard built on it accepts {"rsid": "rs1", "alts": "G"}, which the model refuses and a compile would refuse, so the partial coordinate rides through (@identity-whole-or-none). authoring_requirements still answers the human-readable which cells are missing; it is not the verdict. Because the probe pre-fills every non-identity field with values the model accepts, any ValidationError reaching it is an identity refusal — no message parsing.

What the registry holds, so no module keeps a private copy: match_on, the table, the kind (projection — it later re-reads a column it drafted — or judgement), the precondition, and for a projection the checked columns. provenance's DRAFT_PROJECTIONS is derived from it; pubmind's projection identity genuinely differs from its match_on (the snapshot has no rsID column) and that is a field with a required reason, refused at construction if absent.

What providers still differ on, deliberately, because unifying either would change behaviour: the covered predicate — each provider's own reading of "this run contributed something" — and the stale-label wording, since a published warning is an API and two providers ship two sentences for this finding (@warning-text-is-api).

Adding a provider. Register it in DRAFT_PROVIDERS, derive the identity verdict through drafting.skip_reason, and write provenance through record_draft_provenance. Two guards make this inherited rather than remembered: test_drafting_scaffold.py asserts the registry equals the *_draft.py modules on disk, and walks each module's AST to refuse a hand-listed identity column or a direct merge_sources_file / withdraw_stale_dataset / record_source_terms call.

One thing that looks open here and is not. clinvar_draft and pubmind_draft write their licence row on any non-dry run rather than gating on covered, which is the shape RM222 found wrong in civic_draft. Both are nonetheless correct: each returns early — "nothing matched; no rows drafted" — before the write, so the property holds upstream of the gate. test_draft_licence_row_needs_coverage.py pins that early return, because it is what actually holds the rule and nothing else asserted it.

CIViC — the direction axis, and a source whose coordinates are all on the wrong build (RM152)

just-dna-enricher civic build --release 01-Aug-2026 --out data/caches/civic
just-dna-enricher draft-panel spec/ --gene VHL --source civic --civic-cache data/caches/civic

It writes direction, never clin_sig, and the inversion is the finding. CIViC was proposed as a second clinical-significance concordance authority. Measured, its germline subset carries five ACMG-tier calls and zero benign-class, so discordant is unsayable about anything and the check would read single or concordant by construction. What it does carry is 1,458 germline rows on Predisposition/Protectiveness — this format's direction (risk/protective) — where the NA count is 0, against 812 on clin_sig. Every number is in CIVIC_SURVEY.md.

The build reads the dated bulk TSVs, not the GraphQL API, and the two are different sources. Only the download side has dated releases, and a snapshot that cannot name its input cannot be reproduced. The cost is that the TSV is accepted-only at 4,903 rows while the API defaults to NON_REJECTED at 11,518 — 2.35× apart, declared by neither — so release.json records status_basis and a count from one surface must never be compared with a count from the other.

Three input files are required and the third is not padding: without MolecularProfileSummaries.tsv, a profile naming two variants and a profile naming none are one blurred drop reason instead of two different facts.

Coordinates are GRCh37 or absent, never GRCh38 — and nothing is lifted over. 2,189 gene variants are GRCh37, 2,433 carry no build, two are GRCh38. The snapshot places rows by reading the identity CIViC publishes beside the coordinate: an rsID, or a GRCh38 RefSeq accession inside ClinVar's HGVS. That is RM48's rule applied rather than circumvented — and better than RM48 hoped, because a published rs-number is an independent value ordinary resolution cross-examines, where a recovered one would have resolution verify Ensembl against Ensembl. Scoring an accession needs the per-chromosome map: NC_000001.11 is GRCh38 while NC_000002.11 is GRCh37. What carries neither identifier is dropped by count, and RM153 is what to do about it.

Identity comes from the registry where CIViC states none, and never from a liftover. A row whose only route is a ClinGen allele_registry_id is kept in the snapshot as identity_derivation="caid" — a route to an identity rather than an identity — and draft-panel --source civic walks that route: the registry returns an rs-number (preferred, because Ensembl then verifies it and two authorities make the check real) or a GRCh38 coordinate. One-sided indels are anchored VCF/Picard-style: the registry states an insertion with an empty reference allele and a deletion with an empty allele, and prefixing both sides with the reference base before the event gives a VCF row at the registry's own position. That position is HGVS's 3′-most one, so inside a repeat the row is then left-aligned against the reference (sequences._left_align, RM273: rs72613567 anchors as 4:87310241 A>AA and is written 87310240 T>TA); a repeat longer than the read window withholds as an unreadable anchor. That takes recovery from 48% of the direction set to 82%. --offline withholds those rows as caid_unresolved — unplaced, not unplaceable — and an unreadable anchor withholds under its own reason rather than being guessed at. Liftover was measured and refused: its ceiling is 13 rows, and the one precise event in the class lifts exactly to the wrong allele.

An identity CIViC states in a variant's name is read out of it (RM159). 53 variants in the 01-Aug-2026 release carry nothing in the identifier columns, and for most of them the identity was published all along in the name itself — N150fs (c.448delA), IVS2+1G>A, D1709N. Thirty-three were resolved by hand and ship as the constant civic_identities.CIVIC_NAME_IDENTITIES, emitted with identity_derivation="curated_name" — a member of its own, because rsid/grch38_hgvs mean "the source stated this in the column for it" and a consumer must be able to exclude the difference without re-deriving it. Coverage goes 237/290 → 270/290 of variants.

They are data rather than a draft-time lookup because four of the 33 needed a judgement no lookup makes: a legacy IVS2 name whose structural conversion lands on the wrong exon (both readings being real registered alleles 9 kb apart), a name pairing a missense protein label with a synonymous cDNA change, a protein consequence standing over an intronic allele, and an rs-number that is position-level where two alleles spell the same substitution. So the answers are shipped, the procedure is written down in docs/probes/CIVIC_IDENTITY_PROTOCOL.md, and the build stays offline. The one identity not adopted is TP53 R72P: it resolves to the reference allele (g.7676154G=), and ref == alt is not a variant row.

Each curated identity is keyed to the exact name it was read from, and lands in one of four counted states in release.json — applied, superseded (CIViC now publishes one of its own; the source always wins, and this is also the cheapest currency signal there is), renamed, absent. The four sum to the table, asserted as an equality. allele_registry_id stays CIViC's verbatim cell: the CAIDs the resolution went through are provenance on the table, never written into the source's column. civic reproduce then cross-examines every placed coordinate against the GRCh38 reference through refget — 57 of 57, 0 mismatches, up from 24 before the adoption.

The unreviewed majority is readable without leaving the dated release (RM169). CIViC publishes <date>-civic_accepted_and_submitted.vcf beside the TSVs, so --submitted widens the basis with no API read and no loss of reproducibility: 507 rows on 270 variants → 1,149 on 397, a rebuild still byte-identical, and the refget cross-check going from 57 coordinates to 129, 0 mismatches. Every row carries evidence_status — CIViC's own word, unconverted — and release.json records status_basis, status_counts, vcf_evidence and unjoinable_submitted.

The VCF is joined onto the TSVs, never substituted for them. A VCF record needs a POS, so the file cannot carry a variant with no GRCh37 coordinate — and 52 of the 54 variants it drops are exactly the class whose identity had to be read out of its name (RM159). Reading it as the row source would discard the hardest-won half of the snapshot.

VariantSummaries.tsv is accepted-only too, which is why identity_derivation="vcf_csq" exists: 112 of the 127 variants the submitted evidence introduces have no row in it, so their identity comes from the same CSQ entry, through the same parsers, on the same published identifiers (57 by CAID, 40 by rs-number, 14 by a GRCh38 accession, 1 by both). The member names the file rather than the route, because the route is already visible in the row's own cells. Nothing is placed from the VCF's own position: it is GRCh37 and lifting it stays refused.

A refutation is kept and its direction withheld. Does Not Support removes a claim without establishing the opposite one, so the row keeps its raw words and states no direction — an unknown is withheld, never negated. The drafter reports how many it held back and why.

And an authored direction beside a refutation is a finding, on two surfaces (RM170). Withholding the refuting row was never the whole answer: a variant CIViC supports and also rebuts still gets a risk row drafted — correctly, because the provider writes what the source said — and until 0.7 nothing then told the author the rebuttal existed. contested_variants cannot see it: that counts a variant whose camps hold both risk and protective, and a refutation enters no camp at all, so the counter is correctly 0 on every basis and the muddy variants are invisible to it.

  • draft-panel --source civic names them at the point the row is written: the variant, the supporting evidence id, the refuting one with its status, and the snapshot's status_basis. The row is still written.
  • enrich folds in the published_refutation check whenever a CIViC snapshot resolves ($JUST_DNA_CIVIC_CACHE), so a hand-authored module that never ran the drafter meets it too. Two finding codes, because they are two sentences: refutation_beside_claim (the source asserts and rebuts) and refutation_without_claim (the module asserts a direction the source has only denied). They are this pass's own keys and travel in the record's detail — deliberately not VALID_WARNING_CODES members, which is the compiler's vocabulary and whose guard asserts every member is built by a compiler check. A compile restates them as verification_findings_recorded.

Warns in both modes and escalates in neither — a source disagreeing with itself is not an authoring error, the same call clin_sig makes. Nothing is repaired: a refutation withholds a claim rather than establishing its opposite, so there is no opposite value to write even if the tier were allowed to write one.

The finding keys on the refuting evidence item, then fans out to the rows it touches. CIViC evidence item 8721 is one statement about the two-variant genotype VHL S183L AND VHL D126N, which the snapshot writes as two single-variant rows (RM174) — so keying on the variant would report one rebuttal as two independent ones.

The record states its basis on every run, including the empty one. Every assert-and-refute pair in CIViC rests on a submitted rebuttal: both accepted refutations in the whole database stand against nothing at all (probes/CONTRADICTION_CORPORA.md). So on the accepted basis this class is empty by construction, and findings: 0 without the basis beside it would read as clear water.

A row names the profile its evidence item actually belongs to (RM174). CIViC publishes molecular profiles as boolean expressions over variants — VHL S183L (c.548C>T) AND VHL D126N (c.376G>A) — and an evidence item on one of those is a claim about a combination genotype. Reaching the builder through the VCF, it is fanned out into one row per variant it names, and molecular_profile_id has to be the variant's own single-variant profile or the row does not join at all. So the item's real profile rides beside the join key in evidence_molecular_profile_id and evidence_molecular_profile_name, on every row rather than only on composites — a column that is null on the common case invites a reader to treat null as "not a composite". A composite is evidence_molecular_profile_id != molecular_profile_id, derived rather than stored, and release.json publishes the count as composite_profile_rows.

The name is null on a TSV-sourced row because MolecularProfileSummaries.tsv publishes none, and filling it from the variant's name would state a profile name the source never wrote. The path asymmetry is deliberate and audited rather than inherited: a multi-variant profile arriving through the TSV is dropped as combination_profile and counted, while one arriving through the VCF is kept and labelled. Making both paths keep it would move an accepted-basis build's numbers, and what the format can honestly represent about a two-variant claim is a different question — that one is RM28's, parked, and RM174 gave it its first counted corpus entry.

Every drop is counted, and the somatic majority is the point of that. Roughly three quarters of CIViC describes tumour tissue no germline genotype can satisfy. A filter whose scope is narrower than its name is what this adoption was designed against, so input_rows == record_count + sum(dropped) is asserted as an equality over a walked registry and every reason lands in release.json.

There is no --use flag on civic build. CIViC is CC0 on every axis, so a declared-use gate would permit every build unconditionally, and a flag feeding a gate that never gates is a flag that does nothing.

civic citations — the citations no dated file can carry, and the canary on them (RM160)

The gap is structural rather than a coverage shortfall. RM169 widened the snapshot as far as a dated file goes, and the wider basis is a VCF — so a VCF record needs a POS, and CIViC publishes no GRCh37 coordinate for a variant it names as a class of event or as a legacy notation. The submitted evidence on those records is published on exactly one surface, the GraphQL API. Ten records' citations are unreachable from every file the builder reads, and one of them is variant 1955 VHL P71fs (c.211insT), whose only reachable evidence for the numbering convention its identity turns on is EID 9969 (PMID 12202531, Dollfus 2002, free full text) — submitted, in the API, in no file. The counts are in probes/CONTRADICTION_CORPORA.md and probes/CIVIC_SURVEY.md.

civic build and civic reproduce are untouched, and that is the whole of the decision. RM160 was settled as shape 3: read SUBMITTED at enrich time, where network reads already live and reproducibility is never claimed. Hashing an API capture as a build input (shape 1) keeps the word reproducible while changing what it is reproducible against, and a second API-built parquet (shape 2) was dissolved by RM169. The read is one request per variant by construction — evidenceItems takes a single variantId — which is why it fits a command an author runs over their own module and would not fit a builder pinning a dated release. Batching it into civic build is the first repair anyone proposes and it is exactly the bargain this shape refused.

Three routes reach a CIViC variant id, and the third exists because the first two miss the class this is for. The snapshot's own coordinate join goes through clinical.comparison_plan, the same resolved-(chrom, start, ref, alt) route the refutation leg and the ClinVar cross-check use, so every CIViC question in this tier is asked about the same alleles. The curated name-identity table (RM159) is how a variant whose identity CIViC publishes only inside its name is reachable at all. Neither reaches 1955 — it is one of the two records that round could not resolve — so --variant-id N asks about an id directly and writes a module-level citation row, which StudyRow has permitted since RM47.

A recovered citation carries the same identity its variant row got — rsID where there is one, the coordinate otherwise, never both. That is clinvar_draft's rule, learned from the compiler's orphan check on the first real panel, and it is load-bearing twice here: a row carrying both has a different match_on signature from the rsID-only rows draft-panel --source civic writes, so the two lanes would each append a row under one (variant_key, pmid) and the compiler would call them duplicates. The canary joins on StudyRow.variant_key for the same reason — the model's own derivation over those cells, rather than a tuple of raw columns this lane spells its own way.

A recovered citation lands in studies.csv and nowhere else. literature.csv is the derived article table, filled by the literature command from the PMIDs studies.csv names, and an article row nothing cites is dropped from the artifact. Writing one here would either duplicate that pass or add a row the compiler discards; drafting the citing row and letting literature fill the article is the pairing that works in both directions. Several evidence items citing one paper are one row — five of CIViC's items cite PMID 17661816 on variant 844 — because (variant_key, pmid) is the grain.

status rides as confidence/confidence_unit, unconverted (civic_evidence_status). CIViC's own instrument named rather than translated into a house grade, so an accepted row and a submitted row are not the same row once both are in the file. Where the live items behind one paper disagree, the confidence is withheld rather than picked. Rejected evidence is not drafted at all: status: ALL returns items CIViC's editors threw out, and a paper whose every item is rejected is content the source repudiated — counted under rejected_by_source, never silently dropped. A citationId on a non-PubMed source (ASCO and ASH abstracts) is a real id in another namespace and withholds rather than becoming a pmid.

The pin is on the SourceRow, and the layer is literature. Every drafted row records when the API was asked (fetched_at) and on what basis (dataset = civic_api:status=ALL) — not a timestamp in each conclusion, which would fold the moment of a read into content_signature. (civic, annotation) stays civic_draft's slot: a second surface of an already-declared source may not claim the lane's row, and a civic_api source would publish a route as a licensed body. literature is what this pass really supplies and is orphan-check exempt. Merge-not-clobber means the pin records the ask that first put a recovered citation into the module, which is a floor on not asked since — and the canary is what closes the gap.

evidence_status_currency is the canary, and it is what makes drafting from a live read honest. enrich() re-asks CIViC about every citation the module records from this lane and reports what has moved: a status accepted or rejected since the draft (civic_evidence_status_moved), or a citation CIViC has added since (civic_citation_added). Two codes because they are two remedies. It warns in both modes and escalates in neither — a source re-curating its own evidence is not an authoring error — and it is deliberately not dataset_currency: that one asks which release a table came from, this one asks whether a per-item judgement has moved, and the two currency findings stay apart.

Its four skip reasons are cleared by four different things, and none of them is a pass:

skip what it means
nothing_to_check the module records no citation from this lane — a civic_draft row carries no confidence_unit, so it is not re-asked against itself
offline the run had no egress. Never ran, findings=0: nobody-asked and the-source-has-nothing-more are different facts
no_reference no recorded citation could be mapped back to a variant id this run — no snapshot, or rows that ground the module rather than a variant. Published with its count rather than silently passed

The two sets the canary reads are deliberately different widths. Which citations have a recorded status to compare is the narrow one — rows carrying confidence_unit = civic_evidence_status, so a civic_draft row is never re-asked against itself and a row whose confidence was withheld is not a subject. Which papers the module already cites is the wide one, over every studies.csv row whatever wrote it, because the added-a-citation arm asks whether there is a row for this paper at all; keying that on the narrow set would report a citation sitting in the file as one CIViC has added, every lap (@a-set-that-silences-is-narrower-than-one-that-raises, taken the other way round). | unreachable | CIViC answered for none of the variants asked about |

Cache internals — locations, resolver, download

  • locations.py (moved from just_dna_compiler.cache) — resolve_ensembl_reference locates a usable reference by precedence (explicit arg → $JUST_DNA_ENSEMBL_CACHE → $JUST_DNA_PIPELINES_CACHE_DIR → platformdirs), and default_ensembl_cache_dir/load_env. It never downloads — location only.
  • resolver.py (moved from just_dna_compiler.resolver) — the DuckDB engine: _connect/ _view_over_parquet over a .duckdb file or a data/*.parquet dir, resolve_variants (fill/expand/ verify with the one-to-many ORDER BY id, chrom, start, ref expansion), and the public lookup_loci the enricher and (until 1.0) the compiler's deprecated path share so they never drift.
  • resolver.probe_table — a batch lookup must HASH its probe, and this is why a panel used to never finish (0.5.2). DuckDB cannot fold a disjunction of equality conjunctions into a hash probe, so WHERE (chrom=? AND start=? AND ref=? AND alt=?) OR … is evaluated against every row of the reference: cost grows with alleles × rows, quadratically in the module. A 297-gene panel ran two hours at 12% CPU with no I/O and looked like a deadlock. Measured on the 4,431,781-record snapshot, same 5,000 alleles, same connection: 88 s OR-chained against 0.21 s joined against a temp table. Three call sites moved (clinvar.lookup_clin_sig, resolver._lookup_rsid_candidates, and clinvar.select_by_gene, which is single-column and became gene IN (…) — 20.9 s → 6.6 s, since IN is pushed into the parquet reader and an OR-chain is not). _lookup_positions_by_rsid and citations_for already used IN and were left alone.

Two things about it that must not be "simplified". The probe rows are rendered as SQL literals, escaped the way _connect escapes the parquet path, because DuckDB's Python parameter binding is where the remaining time goes — same query, same data: literals 0.21 s, a composite-key IN (?, …) 1.04 s, a parameterized UNNEST(?::VARCHAR[]) 3.51 s, executemany 8.6 s. Parameterizing it back gives up most of the win, so re-measure before changing it. And every caller keeps its own ORDER BY: a join reorders nothing by itself, and emitted row order is digest-visible (Principle 7). A regression guard lives in test_query_shapes.py — it asserts the plan contains a hash join (no clock involved) and, separately, times both shapes in one process so a slow runner moves both numbers together. - ClinVar cache location — locations.resolve_clinvar_reference mirrors the Ensembl ladder (explicit arg → $JUST_DNA_CLINVAR_CACHE → $JUST_DNA_PIPELINES_CACHE_DIR/platformdirs, under a clinvar/ subdir), also never downloading. The ClinVar snapshot ships as parquet only (no prebuilt .duckdb). - One resolver body, six callers. _resolve_parquet_cache(explicit, env_var, default_dir) is that ladder, written once: it was copied per snapshot and the copies had already drifted (ClinVar's silently lacked the bare-.parquet case its constraint sibling had). accept_bare_file=True is the one difference, and only the single-file constraint snapshot wants it. _cache_dir(subdir) reads $JUST_DNA_PIPELINES_CACHE_DIR at call time, not at import, so a .env loaded by load_env can still change the answer — and since 0.5.2 it calls load_env() itself, which is the fix for a family of "the cache is right there" reports. _resolve_parquet_cache loads the environment inside itself, but each resolve_*_reference passes default_*_cache_dir() as an argument, evaluated before the call: with the base set only in .env, the first resolve in a process computed its default from platformdirs and returned None, and every later one was correct. That asymmetry produced three separate bug reports — cache pull writing into ~/.cache while cache status looked in the configured directory and called the snapshot absent moments after a successful pull, draft-panel --offline refusing with "no ClinVar snapshot found" for a snapshot cache status reported present, and a test module whose first skip-guard silently skipped. One load, six resolvers, both CLI paths; override=False, so a real environment variable still wins.

The same argument-position evaluation made load_dotenv_file=False a knob that did nothing, in all six resolvers, until 0.6.3 (S39). The load above was unconditional, so it ran before the resolver had looked at its own flag — and a caller passing False still had the whole .env written into os.environ. The flag is now threaded through _cache_dir and the six default_*_cache_dir helpers rather than the load being removed: the unconditional load is the repair described above, and the True path is byte-identical. test_locations.py pins both directions and walks the two families asserting each takes the parameter, so a seventh snapshot's resolver cannot quietly reopen it. Same shape as the watcher's ${BRANCH:-main}: a knob's disabling value is its own case and needs its own probe, not a reading. - locations.read_release(reference) — just_dna_format.snapshot.read_release since 0.8 (RM303): a snapshot's release.json as a SnapshotRelease, or None when it is absent or unreadable (it was a dict until 0.8; a caller reading .get("dataset") now reads .dataset, or snapshot.release_label). Written by every builder and, until 0.5.2, read by nothing but cache status; it is what lets enrich() compare a module's panel: pin against the snapshot in front of it. None for both absence and corruption on purpose: a caller must not be able to mistake "this snapshot does not say" for a release id. - download.py — ensure_snapshot, ensure_clinvar_snapshot and ensure_constraint_snapshot pull the parquet slice from the HF datasets (just-dna-seq/ensembl_variations_split / just-dna-seq/clinvar / just-dna-seq/gnomad_constraint) via one shared footer-checked/atomic body. Every pulled lane's repo id is spelled once, in the public locations.SNAPSHOT_REPO_IDS (lane → org/repo, RM384): each download.SNAPSHOT_REPO_PREFIXES row, keyed like SNAPSHOT_FILE_GLOBS, is derived from it as datasets/<repo id>/data, and so are the upload.DEFAULT_*_REPO_ID aliases and every CacheLane.publish_repo. A consumer pinning ensembl_reference (org/repo@<rev>) takes the repo from SNAPSHOT_REPO_IDS["ensembl"]; an AST walk refuses a repo id written anywhere else. A complete parquet begins/ends with the PAR1 magic; downloads go to a .part temp and rename only after the footer verifies, and a corrupt/truncated file is removed and refetched rather than skipped forever. huggingface_hub is a guarded lazy import — a missing wheel fails with a clear diagnosis pointing at the install or the --*-cache flag. Every pass that wants a snapshot provisions through these: enrich for Ensembl + ClinVar, gene_metrics for constraint. A published dataset accumulates, so each ensure_* fetches only the files its snapshot is made of — clinvar-*.parquet and not the 159 MB clinvar.parquet the repo still carries from the single-file era, whose columns are the raw VCF INFO fields. The reader globs data/*.parquet, so importing that one file would put two schemas under one DuckDB relation and every query would die on Referenced column "clin_sig" not found. A foreign file already in a local cache is reported, never deleted, and the message names it and the fix. release.json comes down with the data, so a provisioned snapshot can state its own release — it is what a drafted module's recorded dataset is derived from (RM4), and a cache that cannot state its release is one a drafted module cannot name. A repo without one still provisions; absence is not an error. LICENSE.txt rides along on the same rule, and it did not until 0.5.1. upload's allow-patterns were data/*.parquet, citations/*.parquet and release.json, so publishing a share-alike snapshot silently dropped the one file the pinned-licence design exists for — clinpgx_build extracts ClinPGx's terms out of the very archive the data came from precisely so a holder of the snapshot can read what governs the bytes, and license_sha256 pins nothing for someone who never received them. Both halves are fixed: the publisher sends it and the provisioner fetches it. Absence stays normal (only ClinPGx ships one).

gnomAD v4.1 — three roles, one endpoint (gnomad.py)

gnomAD enters in three deliberately different kinds of role: a resolution link (above), an allele-frequency pass, and a gene-constraint pass. One GraphQL endpoint serves all three.

Rate limiting is the design constraint. gnomAD allows 10 requests per IP per 60 seconds, so a request per variant is unusable. Everything is batched by GraphQL field aliasing: probed live, batches of 20 and 25 succeeded and 29 returned HTTP 400, so batch_size=20 with a 6 second pacing gate — exactly the stated budget, about 200 variants/minute. Pacing comes before retry: tenacity retries transport errors, timeouts and 429s, but a blind retry spends the same budget that caused the 429. A batch_size below 1 is refused, when the settings are built and again by each batched read (RM455): with 0 nothing would be asked and every id would come back unanswered, silently. The E-utilities settings and the Europe PMC and PMC ID converter clients refuse it the same way.

Partial failures never sink a batch. GraphQL puts per-alias errors in errors[] and still returns data for the rest (a probed 20-alias batch came back resolved=17, errors=3), so every parser reads both halves. An error with no alias path is different — that is our broken query, and it raises.

A multi-allelic rsID cannot be looked up by rsID. variant(rsid: "rs334") answers only "Multiple variants found, query using variant ID to select one." — and rs334 is sickle-cell, exactly the kind of variant a module carries. Those rsIDs are collected and retried through variant_search, which returns every matching variant id. The frequency pass never meets the problem: it keys on the already-resolved chrom-pos-ref-alt.

Pass 2 — allele frequency (frequencies.py, online only)

enrich_frequencies(spec_dir, *, mode, offline, populations, dataset, write, client) reads the coordinates in resolution.csv and writes frequencies.csv: one row per (allele, ancestry group) carrying AC/AN. Existing rows are authoritative and merged, never clobbered.

Three things the raw payload does that the pass undoes, all visible in the committed recording:

  • it carries sex splits (nfe_XX) beside ancestry groups — a second axis (Principle 5), dropped;
  • it lists XX/XY twice — deduplicated;
  • it names the whole-dataset row with a bare empty id — mapped to global.

Server order is not preserved (it is not promised, and the table must be byte-stable): rows are sorted by (variant_key, alt, population_order). Per-population af is not exposed by the API at all — frequency is AC/AN and we compute it, which is why the CSV stores integers and allele_frequency is derived. faf95 is a single value with a named owning group, so it lands on that group's row only.

This is the first online-only link in the whole chain, and it will stay that way: the v4.1 sites VCFs are 58 GB (exomes) and 742 GB (genomes), so there is no slice to ship. --offline makes the pass a no-op with a warning rather than a failure — and that is not a reproducibility hole, because once frequencies.csv is written it is the pin, and every later compile reads it offline.

status has three members, and the third exists because gnomAD's callset has a hole. VALID_FREQUENCY_STATUS is {resolved, not_found, not_covered}:

  • resolved — the source served counts.
  • not_found — the source was asked and has no such allele. A fact about a locus it does cover.
  • not_covered — the source does not cover the locus, so it has no answer and none can be inferred.

The third was added after the pass was found writing not_found for a Y pseudoautosomal locus, whose comment claimed the row was a fact ("gnomAD was asked and does not have this allele"). It is not: gnomAD hard-masks the Y PAR — those bases duplicate the X PAR — so it never looked, and before PAR selection landed a one-to-many expansion handed this pass ten such loci per SHOX panel. That is the None ≠ False rule: an unknown may not be recorded as a negative. gnomad.covers_locus decides it (three-valued, and the source convention lives there while the PAR geometry stays in vrs), and such a locus is now not queried at all — the request would spend a slot of a 10-per-minute budget to learn nothing, and asking is what produced the false absence.

not_covered rather than unchecked, which is this codebase's word for a question that was never put (acmg.py: the row named no gene, the list could not be reached). This is the stronger statement that the source's scope excludes the locus. FrequencyResult reports them in uncovered, kept apart from missing, and they are outside the strict gate on purpose: a locus gnomAD cannot cover is perfectly reproducible, so refusing would make a pseudoautosomal module uncompilable for a reason no authored edit could fix. (FrequencyRow.status also gained a validator here — until 0.5 it was free text on a fact table.)

Pass 3 — gene constraint (gene_metrics.py, offline capable)

enrich_gene_metrics(spec_dir, *, mode, offline, constraint_cache, dataset, download, write, client) takes the module's gene symbols (deduplicated in first-occurrence order) and writes gene_metrics.csv: pLI, LOEUF, missense Z and friends, one row per gene. Snapshot first, live API second.

Since RM157 that is every authored table carrying gene, not variants.csv alone. module_genes is not a report — it is the scope of this pass, of gene_validity and of the ClinGen dosage pass, all three of which call it — so a module whose genes live in its PGx tables had all three quietly do nothing: no rows, no findings, no line saying a question had not been put. Measured on this repo's own corpus, where cyp2c19_star_alleles, apoe_epsilon, cyp2c9_warfarin_grch37 and hfe_compound_het returned an empty gene set while naming CYP2C19, APOE, CYP2C9, VKORC1, CYP4F2 and HFE. The set is derived from the same registry walk the identifier roster uses, so a table kind that gains the column joins by existing; a table that will not parse refuses here rather than being routed to not_read, because half a scope is a silently narrowed one. pgx._GENE_TABLES is deliberately not this roster: it is the pair whose presence decides whether the star-allele cross-check applies at all. One authored table is left out, since 0.8: model_organisms.csv (RM335). Its gene is the gene a mouse experiment altered, not a gene the module's findings are about, so the scope follows manifest.stats.genes, which omits it too (gene_metrics._NOT_THE_MODULES_GENES). check-identifiers still asks HGNC about its symbols.

This is the one gnomAD role that works with zero egress, and the difference from frequency is purely size: gene-level constraint is one row per gene, single-digit MB as parquet.

The snapshot is provisioned, not merely hoped for. With no local snapshot and not offline, the pass calls download.ensure_constraint_snapshot before it considers the API — the same shape enrich() uses for the Ensembl and ClinVar snapshots, with --offline as the only switch (there is deliberately no second flag). That wiring was missing until 0.5: ensure_constraint_snapshot had existed since the download body was generalized and had no caller, so a plain install fell straight through to the live API and quietly recorded v2.1.1 numbers — then warned about the release difference for a snapshot it had never tried to fetch. A provisioning failure degrades to the API rather than sinking the pass (HF has gone dark mid-demo), and the warning names the consequence: older numbers, not no numbers.

The two routes are different releases, and the table says so. Checked against both: for BRCA1 the bulk v4.1 file gives pLI 1.55e-34 / LOEUF 0.885 / mis_z 2.338, while the live API gives 5.52e-38 / 0.928 / 1.734 — same gene, same MANE transcript. The live gnomad_constraint field serves v2.1.1 constraint; v4.1 ships only in the bulk file. So a row records which release it came from: gnomad_v4.1_constraint from the snapshot, gnomad_v2.1.1_constraint from the API, and the fallback logs a warning. dataset is inside the fact set precisely so these cannot be confused — labelling both as v4.1 would record a false fact and make the fact-hash claim two different tables identical.

Refuted on 2026-11-07, fixed by RM417. The API's numbers are not v2.1.1. gnomAD 4.1.2's bulk file gives BRCA1 exactly 5.5171e-38 / 0.92759 / 1.7336, and the live API answers chrX's DMD with 4.1.2's LOEUF 0.2354 to every digit, a gene no release before 4.1.1 has constraint for. So the API serves 4.1.1 or later, and gnomad_v2.1.1_constraint on an API row is a false label. The two routes did differ in August, as 4.1 against 4.1.1; the label named the wrong one. API rows are labelled gnomad_api_constraint since, a route label naming no release (the pm's ruling: a measured "4.1.2" is true only until gnomAD moves the API); a row still labelled v2.1.1 holds whatever the API served on its fetched_at, and a re-run keeps it.

gnomAD constraint snapshot (constraint_build.py, [dev])

download_constraint_tsv + build_snapshot reduce gnomAD's per-transcript constraint TSV to the gene-level parquet the pass reads. The lane builds 4.1.2 since RM410, release/4.1.2/constraint/ gnomad.v4.1.2.constraint_metrics.tsv.bgz (574.5 MB bgzipped, 113 columns, read through gzip because the reader tells a compressed file by its bytes), and cache rebuild --pin constraint=<release> builds another: 4.0 and 4.1 ship a plain .tsv (4.1's is 95.5 MB, not the 4.2 MB some aggregator pages claim), every later release .tsv.bgz. The path is release/<v>/, not release/v<v>/. The snapshot's label, gnomad_v<release>_constraint, is read from the file's own name, gnomAD's gnomad.v<release>.…, and a --source file naming another release than the pin is refused rather than relabelled. Built from 4.1.2 on 2026-11-07: 19,511 genes from 221,898 transcript rows (4.1: 18,111), chrX and chrY included, LOEUF moved for 17,214 of the 18,111 genes 4.1 had. A module's existing rows keep their 4.1 label on a re-run; delete gene_metrics.csv to move it.

The row pick is load-bearing, not cosmetic. The TSV is per-transcript, 55 columns, and mixes RefSeq with Ensembl rows for the same gene — and both carry mane_select=true:

A1BG   1                 NM_130786.4       canonical=true  mane_select=true    <- RefSeq
A1BG   ENSG00000121410   ENST00000263100   canonical=true  mane_select=true    <- Ensembl
A1BG   ENSG00000121410   ENST00000600966   canonical=false mane_select=false

BRCA1 and MYH7 show the same shape, so this is the file's general structure. A naive "first mane_select row wins" returns whichever the file happens to list first — for A1BG the RefSeq row, whose gene_id is the bare NCBI id 1, useless as a stable identity. The rule is mane_select AND an ENSG-shaped gene_id, falling back to canonical on ENSG, else the gene is dropped as unresolved rather than guessed. The test feeds the real rows in both orders and demands the same answer, which is precisely what the naive implementation fails.

The reference-allele check (sequences.py)

verify_reference_alleles(rows, *, sequences, offline) compares each row's ref against the actual bases at its coordinate and returns the disagreements. It runs inside enrich() by default (--verify-ref/--no-verify-ref) and its findings land on EnrichmentResult.ref_mismatches.

Why it has to exist. A VRS allele id is built from which sequence, which interval, and what replaces it. The reference allele is not a component — the refget accession plus the interval already determine it, since sequence[start:end] has exactly one answer. That is correct and deliberate (a content-addressed identity must be a function of the allele, not of the claim about it, or two records for one allele could get two ids and the whole scheme collapses). But it means minting never looks at the authored ref, and VCF's free consistency check — which catches liftover slips, off-by-ones and wrong-assembly errors — is gone. VCF can afford REF because its CHROM is a name rather than a digest, so REF is genuinely load-bearing there. VRS traded that redundancy for canonicality. This check buys it back on the one tier that has the sequence to do it with.

Two failure modes, and the claimed length decides which. The actual bases are always read at the claimed length, so the two cannot be told apart by comparing lengths:

  • a single-base claim — absorbed. 11:5227002 C>A and the true T>A mint the same id, so the minted identity is still correct and nothing downstream could ever notice the bad row. Only this check reveals it.
  • a multi-base claim — corrupting. The claimed length sets the interval, so a wrong ref makes the id span the wrong bases and name an event the author did not intend. RefMismatch.distorts_the_allele_id reports which case a finding is.

RefMismatch.shift is the field to surface first, and it names a third cause. A mismatch does not only mean the ref cell is wrong; far more often in the wild the position is wrong and ref was right all along for the variant the author meant. shift is the offset, in bases, at which the authored ref is the reference sequence — +1 meaning the variant sits one base right of the authored start, which is exactly what subtracting one from a 1-based VCF POS produces. It is None when no neighbour explains it, so nothing is claimed; diagnosis renders whichever cause was established and is the grouping key for a run's summary. The distinction is not cosmetic. Reporting a shifted coordinate as a bad ref sends the author to the wrong column while leaving them to wonder why their coordinate validated against dbSNP — and a shifted row always distorts the allele id whatever its length, because the id is minted at the authored position, so the compiler's VRS pass recomputes the same wrong id and reports it verified.

It reports; it never repairs. The row is left exactly as authored. Rewriting it would destroy the evidence that something upstream is wrong and silence a problem the author needs to decide about.

Rows with no coordinate, and rows whose ref is not plain ACGT (a symbolic allele, RM5), are not checked — abstaining beats inventing a verdict. Reads are cached by (accession, start, end), so a module asking about one locus repeatedly costs one round trip, and the same SequenceProxy is shared with indel minting so a run builds one proxy in total. Needs sequence access, so --offline skips it: a check that cannot run is not a check that passed, and the run says so rather than implying success.

The old assembly: rs-number recovery and a wrong-build diagnosis (grch37.py, RM48)

An author curating from older literature has hg19/GRCh37 coordinates and the module must be GRCh38. Nothing in these four packages converts, so the conversion happens off-tool and lands as an ordinary authored coordinate with no provenance at all. The compiler refuses the coordinates that are provably impossible; this module answers the ones that are merely wrong.

No chain file, no provisioned asset, no new licence. The roadmap's stated blocker was that recovering an rs-number needs "either an hg19-keyed dbSNP surface or a chain file … i.e. the whole snapshot apparatus for one authoring convenience". Probed 2026-08-13 and false: Ensembl runs a permanent GRCh37 REST service at grch37.rest.ensembl.org with the same API shape, serving both dbSNP variants (/overlap/region) and reference bases (/sequence/region). The same request that answers "which rs-numbers sit here on GRCh37" also discriminates the builds outright — 7:140453135..140453137 is CAC on GRCh37 and GTT on GRCh38.

Recovery, never liftover, and the reporter argued their own request down. If the paper gives an rs-number, liftover is unnecessary and strictly worse: authoring the rs-number produces the independent second value resolution._verify cross-examines. So liftover is only reachable where there is no rs-number and only an old coordinate — and in exactly that case the lifted coordinate becomes the row's sole identity with nothing to check it against, a generator of unverifiable-by-construction identities. That is the hazard class behind this tree's 3,038-row off-by-one, where a content-addressed id was a correct digest of the wrong input and every offline gate passed, --strict included.

recover_rsid(chrom, start, *, ref, alts, client, offline) answers with one of four outcomes. Three of them — recovered / ambiguous / none — are the ones pyliftover fuses, reporting "no result" both for a position that maps nowhere and for one that maps to several. The fourth is unchecked, on the other axis: S20 established in this same resolution path that an unreachable source is unchecked rather than absent. A 4xx is an answer (the service 400s on an unknown contig and on a position past the end of one); only a 5xx, a transport error or a timeout is unchecked. --offline reports skipped_offline, never a pass.

The match is anchored on the authored position: a candidate must start exactly there, carry the authored ref, and contain every authored alt. Anchoring is what keeps it honest — at 7:140453136 seven features overlap the base, two merely span it, one is an HGMD record with no rs-number, and four dbSNP records genuinely start there, which is why a position-only query is ambiguous rather than under-specified. The consequence to know: an indel is recorded at one anchor, Ensembl's, which may be neither VCF's left-aligned position nor HGVS's first changed base. CFTR F508del (rs113993960) is TCTT/T/TCTTCTT at GRCh37 7:117199645, one base right of VCF's ATCT>A and one left of HGVS's position, and asked at either of those it comes back none rather than wrong. So the text keeps "records nothing here" for an empty answer only: a none beside overlapping records says no record starts here and counts the overlaps, a reason to retry at a neighbouring anchor. A recovered with no ref says it matched on position alone and makes its advice conditional, because at a padded indel's VCF position the record starting there is the anchor base's (rs1801178 under F508del). The text names the arguments ref/alts; --ref/--alts are only hint recover's spelling of them (RM444).

diagnose_wrong_build(mismatches, *, client, offline) runs only over rows the reference-allele check already rejected, which is the whole cost control — a module whose refs agree makes no request here. verify_reference_alleles skips any build refget_accession has no table for, so a RefMismatch only ever exists for a GRCh38 module, which makes "the other assembly" always GRCh37 rather than a parameter. Three tiers of evidence, and the message says which one it has:

Tier Evidence What it licenses
single_base_match one authored base equals the GRCh37 base there suggestive only — one base in four agrees by chance, and VCF 4.4 §1.6.1.4 requires an ambiguous reference base to be reduced to the first alphabetically, so an authored A may be a lossily reduced R
multi_base_match several consecutive bases agree chance does not explain it
dbsnp_corroborated the bases agree and GRCh37 dbSNP records a variant starting there the strongest, and the only one that names the rs-number to author instead

The two strong tiers supersede the ±1 neighbour reading, and that is not decoration. On the real HFE pair — 6:26093141 and 6:26091179, authored from the GRCh37 literature into a GRCh38 module — _read_with_neighbours reports "coordinate shifted 1 base to the right" for both, confidently and wrongly: the true variants are 228 and 411 bases away, and a neighbouring base equal to the authored ref is a one-in-four event. Two explanations printed side by side with nothing to order them is the shape this codebase keeps fixing, so the summary says which wins. A single-base match does not supersede a shift — both rest on one agreeing base, and ordering them would invent a verdict.

BuildDiagnosisResult.not_checked carries the reason when the pass did not run (skipped_offline, no_ref_mismatches) and is None exactly when it did, for the same reason clin_sig_not_checked exists: an empty list otherwise says both "asked, and nothing points at another build" and "never asked". The diagnosis travels inside the strict refusal rather than beside it, because a strict run raises and returns nothing, so a result-object-only answer would be visible to the mode that does not need it and invisible to the one whose whole output is that sentence.

It writes nothing. just-dna-enricher hint recover --chrom 7 --start 140453136 --ref A --alts T reports rs113488022 as an advisory Alteration with applied=False and refusal="identity_bearing" — the sharpest refusal in the table, because an rs-number is the row's identity and a machine filling one performs an identity migration by network lookup with no authored edit anywhere. Several candidates are reported and never picked. The author types the rs-number into variants.csv and drops the old coordinate; a later enrich places it on whichever build the module declares, and resolution.csv's source column records which link answered. That is where provenance goes — never into an ordinary authored coordinate.

GA4GH VRS allele identity (vrs.py)

mint_resolution_rows(rows, *, minter, offline, source_ids) stamps vrs_id/vrs_spec onto resolved rows by three routes:

  1. stdlib — a substitution, via the format tier's derive_vrs_allele_id. Zero egress, zero heavy dependency, and byte-identical to what ga4gh.vrs and the live gnomAD API produce.
  2. normalized — an indel/MNV, justified against the reference through the seqrepo REST data proxy. Needs the network, so --offline skips it. One read per locus since RM407: normalize is handed a window of 1,024 bases each side, read once through the run's SequenceProxy, and every slice it takes while trimming and rolling through a repeat is served from it (before, each slice was its own HTTP read, about ten per indel). A slice past the window is read from seqrepo as before, so no id changes. The run's one minter keeps what it minted, so the placement comparison and the minting pass normalize an allele once.
  3. null — an indel offline, an unreachable sequence service, an off-assembly contig, or an allele that names no sequence at all. A missing id is honest; an unjustified one would be a ga4gh:VA.… that looks interoperable and is not.

A source's own id (gnomAD serves one) is cross-checked, never trusted over the minted value — the point of a content-addressed identity is that it does not depend on which sources happened to answer.

Each minted row records which VRS minted it (0.8, RM304). vrs.MINTER_VRS_VERSION is read off the library, ga4gh.vrs.VRS_VERSION ("2.0.1" under ga4gh.vrs 2.3.3), and stored at the spec's minor through just_dna_format.vrs.vrs_spec_release, so the vrs_spec cell reads "2.0" as every table written since 0.5 already does. A row is stamped with the rule of the route that minted it: the library's version if any allele was normalized, else the format tier's VRS_SPEC_VERSION for an all-substitution row (the stdlib minted it). The minor is the grain because VRS 2.1 changes the repeat-unit factor an ambiguous insertion is normalized with, which moves that allele's id; a patch never does. Only a row this pass mints is stamped. A row whose id was already recorded keeps its vrs_spec as it is, an empty one included (this run did not mint that id, so it may not claim its version), and one stating a minor other than the minter's is counted in MintResult.version_differs and reported as vrs_version_differs (a CodedWarning, in the compiler's words) by enrich and vrs mint — never re-minted over, never called a mismatch. enricher/tests/test_vrs_version.py asserts the library's minor equals the format tier's VRS_SPEC_VERSION (the stdlib mints the substitutions, the library the indels, and one row may hold both), and pins the insertion rs72613567 (4-87310240-T-TA → ga4gh:VA.Jml7SNku3QQBCVIj78BGiFvR21bNkos7) behind JUST_DNA_NETWORK_TESTS=1: the first test goes red on a library that implements another minor, the second on one that moves the rule. The ga4gh.vrs<2.4 cap stays until vrs-python releases the rle_subunit_mode keyword (PR #665); lifting it is the liaison's, and our call then passes the mode explicitly.

An allele that names no sequence is a permanent class, and it used to be a crash. RM5's symbolic alleles (<DEL:4977>), RM58's . and RM59's * are legal cells the mint pass routed to indel normalization like any other non-substitution — and ga4gh.vrs's LiteralSequenceExpression accepts ^[A-Z*\-]*$, so the first two raised an unhandled pydantic.ValidationError that killed a whole enrich run over one row, while * passed the pattern and would have been handed a content-addressed id for a state that is not a sequence. vrs._sequence_free_reason decides the class once, for both mint and why_not, so the verdict and the sentence cannot drift: the row keeps no id, the run carries on (the UnsupportedBuildError guard beside it is the same shape), and the reason says the id is unreachable on any run rather than offering the --offline re-run that used to be the crash. One constant string per class, because unmintable_reasons groups on it. Still a warning in both modes — no authored edit makes a symbolic allele mintable (P5).

Publisher surface — module upload (upload.py, [dev])

The author/publisher half of the enricher's HF use (snapshot download is a runtime path; module upload is for republishing, e.g. the Gen-I v1-port recreation). Extracted from just_dna_pipelines.v1_port.publish so lite has a canonical home to adopt.

ensure_repo(repo_id, token=None)                                       # create-or-update (create_repo exist_ok=True)
plan_upload(module_dir, name, repo_id=None) -> UploadPlan              # the plan alone, offline; every local refusal
rehearse_upload(module_dir, name, repo_id=None, token=None, force=False) -> UploadPlan   # the dry run: plan + the versioned-path read
upload_module(module_dir, name, repo_id=None, token=None, commit_message=None, force=False) -> UploadPlan
plan_reference_snapshot(snapshot_dir, repo_id=None, *, payload=None) -> SnapshotPlan     # the snapshot plan alone, offline
rehearse_reference_snapshot(snapshot_dir, repo_id=None, *, payload=None, token=None) -> SnapshotPlan   # the dry run: plan + orphan check
publish_reference_snapshot(snapshot_dir, repo_id=None, token=None, commit_message=None, *, payload=None) -> SnapshotPlan

upload_module uploads manifest.json and every file it declares — the parquets artifact.files attests, the logo, the readme, provenance.json and every log manifest.logs names (RM347, RM353) — to datasets/<repo>/data/<name>/ (default repo just-dna-seq/annotators), matching just-dna-lite's discovery layout, and the same files again to data/<name>/v<version>/ (RM84 — see below). When the manifest cannot say what it declares, the fallback allowlist's parquet half is just_dna_compiler.compiler.ARTIFACT_PARQUETS, imported rather than restated — see what publishes, and what refuses below for why that is a rule rather than a tidiness. publish_reference_snapshot uploads a built data/*.parquet + its parquet sidecars + release.json to the root of a dataset repo (default just-dna-seq/clinvar), matching the download.ensure_*_snapshot layout. Both go through ensure_repo — one create-or-update-then-upload pathway (create_repo was added here; the origin v1_port.publish assumed the repo pre-existed).

The sidecar was the gap, and it made downloaded snapshots second-class. ClinVar's citations/ was built and published nowhere, so a consumer who provisioned the snapshot had no PMIDs while one who built it did — and draft-panel cannot produce a compilable module without them, because studies.csv is mandatory. The layout lives once, in just_dna_format.snapshot since 0.8 (SNAPSHOT_DATA_DIRNAME / SNAPSHOT_SIDECAR_DIRNAMES / CITATIONS_DIRNAME / RELEASE_FILENAME, imported by locations), because four parties have to agree on those names — builder, publisher, provisioner, reader — and every disagreement so far has been silent. A sidecar stays a sibling of data/: the readers glob data/*.parquet, so a two-column citations table inside it is the same poisoning a stale single-file clinvar.parquet causes. Absence is normal (only ClinVar has one, only after clinvar citations), so neither end treats it as an error. Each needs a write token (hf auth login or HF_TOKEN) — a missing one raises PermissionError. huggingface_hub is a core dependency, imported at module level. Every refusal is a PublishRefused (RM368, below); the token is the one failure outside that family.

What publishes, and what refuses (RM89)

Until 0.6 the publisher demanded weights/annotations/studies.parquet and uploaded those three and nothing else. Both halves were written when a module meant a SNP core; RM2 made the SNP core optional in 0.4 and the constants stayed. Measured against the sixteen reference examples on 2026-08-17: seven could not be published at all, and eight of the remaining nine published an artifact whose manifest.artifact.files attests parquets that were never uploaded — so the artifact.digest in the manifest cannot be reproduced from what arrived. Only grch37_build, a bare SNP core with no sidecar and no 0.4-family table, was correct. Reported as S35 by just-dna-lite, who found the sources.parquet half of it from the other end: their report footer renders "Not stated" for a module's licence terms because the table carrying them was dropped at upload.

  • The allowlist is derived, never hand-kept. _ALLOW_PATTERNS is just_dna_compiler.compiler.ARTIFACT_PARQUETS plus manifest.json, the two logo spellings and README_CANDIDATES. A new table kind therefore reaches the publisher in the commit that adds it. The hand-kept version of this list is the same defect as a hand-kept fieldnames — @fieldnames-from-model one tier further out, and the consumer named the property they most wanted preserved as "adding a family becomes one edit that discovery and the publisher learn together".
  • Positive rules replace the required triple, ordered most specific first so a refusal names the actual fault:
rule refuses why it is not the old rule
the plan carries every file manifest.artifact.files attests a deleted parquet; an allowlist that has fallen behind the compiler the artifact's own attestation is the comparator, so it needs no list of its own
weights.parquet never travels alone a half-finished SNP-core compile scoped to the weights-led shape — a pharm_variants-led module legitimately has neither companion
at least one lead parquet (LEAD_PARQUETS: weights + the nine 0.4 families) a directory of manifest.json + README with no data this is what discovery actually probes to decide a directory is a module
  • The first rule now catches a missing file, not a lagging list. Since RM353 the parquets sent are the ones artifact.files names, so the rule can no longer fire because the publisher fell behind the compiler; it fires when an attested file is absent from disk or its name leaves the module directory.
  • An unreadable or absent manifest.json withholds, it does not refuse. Same tri-state as version_unknown_reason below, and the same reason: a manifest that cannot say what the artifact contains has said unknown. A directory with no manifest is still publishable, exactly as before.
  • Nothing that published before stops publishing. The change is a widening on both axes — more module shapes accepted, more files sent — and the only new refusals are the two shapes that were never a publishable module (weights alone; no lead table at all).
  • The logs travel by the manifest's own list (RM347). Until 2026-11-04 no logs/ path was in the allowlist, so a published manifest named logs the repository did not hold, and verify --check-logs skips an absent log by design, so nothing said so. The plan now appends each path manifest.logs names, read field by field out of the same JSON the attestation check reads; it is never a glob, because the compiler already decided which files under logs/ ship (RM338) and a pattern here would be a second statement of that decision. Three edges, each a default rather than a refusal:
  • a file under logs/ that the manifest does not name is not uploaded — nothing the published manifest says covers it, and a published file is permanent;
  • a named path that is absolute, climbs out with .., or resolves outside the module directory (a symlink) is skipped with a warning naming the reason, and the rest of the publish goes ahead. So is a hidden path the compiler would not have shipped: RM338's discovery skips a hidden path for every kind except *.log, so a hidden .log uploads and a hidden .env does not. Anything the hub's own default ignore list names (.git, .cache/huggingface, read from huggingface_hub) is skipped even when it is a .log: tool state, never a module file. Since RM368 the send is a create_commit, which applies no ignore list, so this rule is what keeps such a path out;
  • a named log absent from disk is skipped quietly (logs are optional), and an absent or unreadable manifest plans no logs, exactly as before.

The dry run lists exactly what is sent — the rule publish_reference_snapshot already followed. Until RM368 upload_module passed the plan to upload_folder as a glob-escaped allowlist (the hub matches with fnmatch, so logs/fig[1].png unescaped would select logs/fig1.png); it now commits the planned files' bytes themselves, so no pattern is involved. - Every declared file travels, and nothing else does (RM353). provenance.json is named and hashed in manifest.provenance and was in no list, so it was never uploaded. The plan is now manifest.json plus every file the manifest declares in the module directory, read by one helper (_declared_files) from five fields: artifact.files, logo, readme, provenance (a summary carrying file and sha256, not a FileEntry) and logs. inputs and derived describe files beside the spec and are not sent (SPEC_SIDE_FILE_FIELDS, with the reason). The consequences: - a file on disk the manifest does not declare is not sent — a parquet an earlier compile left in the directory, or a logo the manifest does not name. On the flat path a stale parquet reads to discovery as part of the module, so this is narrower than the old allowlist on purpose; - RM347's path guard reads every declared name, so a hand-edited provenance.file of ../x is skipped with the same warning a log gets; a declared optional file absent from disk is skipped quietly; - when the manifest is absent, unparseable or has no artifact.files list, the plan falls back to _ALLOW_PATTERNS on disk, as before: an unknown withholds; - each declared file sent must hash to the sha256 the manifest states, or the plan refuses and asks for a recompile. The guard below compares manifests, so a log edited after the compile would otherwise arrive under a manifest stating its old hash and pass the guard unchanged.

Every name handed to the hub is made literal by one helper (_literal_patterns): the module publisher's allow_patterns, both of the snapshot publisher's (where data/chr[1].parquet unescaped selected data/chr1.parquet and never itself), a declared retirement's delete_patterns (LayoutShift.retires is one name, never a glob). cache prune used to pass delete_patterns too, where an unescaped listed name would delete a file nobody was shown; since RM386 it sends one delete operation per listed path, so it hands the hub no pattern at all. - One reader, one filter, and the dry run refuses as the real run does (RM355). The plan, the publish and the publish's guard read manifest.json once, through _manifest_document, and pass the object on; the version and the file list of one plan come from that one read (the version reader used to open the file a second time). Every file entry, attested or declared, goes through one filter, _named_entry, which keeps an object whose name is a non-empty string, so the attested and the declared list agree about an artifact.files entry named "" (the declared list used to drop it while the attested one kept it). The plan refuses such an entry, quoting it: it is inside artifact.digest, so no upload can carry what the digest covers. An optional declaration naming no file (a logo with an empty name) is outside the digest and is dropped, as before. upload --dry-run sits inside the same try as the real run: a refusal the plan raises prints the same UPLOAD FAILED: … line and exits 1 from either, where the dry run used to end in a traceback. The dry run then still contacted nothing, so the versioned-path guard was a refusal only the real run could give; RM368 closed that (next bullet). - What the publisher plans, checks and sends is one reading of the disk (RM368). They used to be three: the plan hashed the declared files, the guard compared the manifest in memory, and upload_folder read the directory again, so a recompile landing between the guard and the send shipped bytes nothing had checked. The class and its mechanism: - The plan is paths and hashes. UploadPlan.sha256 holds every planned file's sha256 as the plan read it, manifest.json from the very bytes it was parsed from. After the guard, each file is read once more, checked against its entry, and those bytes are what create_commit carries — never a path the hub reads later. A file that changed, went missing or became unreadable in between is a PlanChangedError, naming each, before either commit. The cost is memory: the module's files are held for the two commits, which a module's size (megabytes) affords and a 32 GB snapshot would not, so the snapshot publisher sends its payload from the directory and checks it on the hub afterwards (RM386, next bullet). - A malformed declared field refuses; an absent one withholds (@an-absent-input-is-the-unknown-arm-and-a-malformed-one-is-the-refusal). artifact.files present but not a list used to fall back to the allowlist and drop the attestation check; a sha256 present but not a string used to read as unknown and drop the hash check, and a name (or provenance.file) present but not a string used to read as no file and drop the file. The fields walked are upload.DECLARED_FILE_FIELDS, and a test asserts they and SPEC_SIDE_FILE_FIELDS are exactly the ModuleManifest fields that reach a FileEntry or a Provenance, so a new file-naming field goes red until the publisher decides whether it travels. A missing or null artifact.files is still the unknown arm and falls back as before. - A blank name is no name. _named_entry keeps an object whose name is a string that is not blank; a whitespace-only name used to name a file. The attested refusal is wider than an empty name: it covers every artifact.files entry that is not an object with a non-blank string name (a missing name, a number, a bare string), quoting each. - The manifest itself is compared at the versioned path (below): every key except the ones the digest and declared-file arms compare (digest, files, name, file, sha256, size). - The dry run rehearses. upload --dry-run calls rehearse_upload: the plan plus the versioned-path read, with a read-only client (a token when there is one, anonymous otherwise), creating no repo and committing nothing; it refuses as ALREADY PUBLISHED or TOOLCHAIN DIFFERS exactly as the real run would, and an unreachable hub is the same fail-open warning. Every snapshot publish --dry-run likewise calls rehearse_reference_snapshot (plan plus the orphan check); two commands ran that check in their dry run and the rest did not. - One error family. Every refusal is a PublishRefused: ModuleNotPublishable and SnapshotNotPublishable for the plans' refusals, PlanChangedError, and the existing PublishCollisionError / ToolchainDiffersError / OrphanedSidecarError re-based under it. The plans' refusals were bare FileNotFoundErrors, which a caller could not tell from a missing directory; within 0.8 the two plan types keep FileNotFoundError as a second base, so an existing handler keeps catching them. Every CLI try around a publish catches the family (an AST walk asserts it), and cpic publish / clinpgx publish, which caught nothing, now print one line. - Every write is pinned to the revision its plan read, and a snapshot to the bytes it hashed (RM386). RM368 tied the module send to its bytes; the snapshot publish and cache prune were still a plan of paths acted on later. Each defect needs a concurrent writer, and each now refuses: - One revision per plan. A publish or a prune plan reads the head once (_head), and every remote read after it names that revision: the versioned-path manifest (file_exists, hf_hub_download), the orphan check's and the retirement's listings, the prune listing. Every write goes through one helper, _commit, which passes that revision as parent_commit; the second commit of a pair is pinned on the first. The precondition alone does not hold: the library skips a write that changes nothing and answers with the current head unchecked, and its multi-commit uploader pins the first part only. So each write carries a token in its commit description, and _commit walks the history from the answered head back to the parent: a commit without the token is another writer's, and the write refuses as a RemoteMovedError (a PublishRefused) that counts both kinds and names what this publish already landed (the flat path of a module, a snapshot's payload). A hub error takes the same walk, not the status code. A dry run reads the head and commits nothing, so it pins nothing. - A snapshot is hashed once and held to those hashes. The publish reads every planned file once (SnapshotPlan.sha256, the sha256: of each, plus the git blob id the hub states for a non-LFS file). The payload goes up through upload_folder, which reads the directory after the plan did (kept for RM199's reason: it is the hub's resumable, multi-commit uploader); then the tree at the payload commit is compared with the plan, by lfs.sha256 for an LFS entry and by blob id for any other. A difference is a PlanChangedError and the description is withheld: the payload commit has landed, and the previous release.json stays over it until a re-run, as after any failed second commit. release.json itself is sent as the bytes the plan read, never read again, so a rebuild after the payload lands cannot put the next release's description over this one's rows. The cost is one more full read of the snapshot (the hash), beside the hub's own. - A prune deletes what its listing showed, at that revision. PrunePlan.revision is the commit the listing was read at; prune_repo commits one delete per listed path on top of exactly it, so a file republished under a listed name after the plan was shown refuses the prune. A plan stating no revision read nothing and refuses. - What is left. A refused description leaves the payload landed under the previous release.json until a re-run, as any failed second commit does. The walk reads a commit's description as list_repo_commits answers it (message); were that ever empty, every write would refuse, never pass.

A module is published twice, and the second path is the one that can name a release (RM84)

On the discovery path there is no version in the path, no manifest fetch and no digest check, so a republished module keeps the same URL and a cached copy shadows it — the only invalidation a consumer has is keyed on its own package version, which makes the identity of "the module changed" a property of the reader. The half this tier owns is the layout, and it is written twice now:

path what it means when it is written
data/<name>/ latest — the deployed path, unchanged in meaning, and what discovery scans always
data/<name>/v<version>/ this release — a subdirectory inside the flat path, not a sibling when the manifest states a version
  • v<version> verbatim, never a bare vN. v0.6.0, not v0. A bare major segment throws the rest away, so two patch releases of one module would collide at one path — the defect being fixed rather than a smaller version of it.
  • A module with no version gets the flat path alone, and says which of four reasons applies. Identity.version is str | None and stays null unless module_spec.yaml states a canonical SemVer (the registry stamps one on publish), so most locally-compiled modules have none. UploadPlan carries versioned_path_in_repo=None plus version_unknown_reason — no manifest.json, not readable as JSON, no identity.version, or not MAJOR.MINOR.PATCH — so a caller reads the reason as a field and --dry-run prints it before anything is sent. A data/<name>/vNone/ directory is never constructed.
  • One field is read out of the JSON, not the whole manifest through read_manifest. Validating the full ModuleManifest to reach one string lets an unrelated defect — a hyphen in identity.name, an icon_set outside the vocabulary — withhold a version that is right there and legible, and the refusal then names neither field nor value. The SemVer gate is identity.is_valid_version, the same predicate Identity.version's own validator calls, which is also what keeps a stray / out of a path segment.
  • It never refuses. manifest.json is in the allow-patterns and not in the required set, so a directory without one has always been publishable; RM84 is not a licence to tighten what the publisher accepts. An unreadable manifest is a withheld version, not a failed upload.
  • Two commits, not one — and the docs say so rather than implying an atomicity the code lacks. Each path is its own create_commit, so a reader can briefly see the flat path refreshed while the versioned copy is not there yet. The flat path goes first, because it is the one anything reads today. If the second call fails, the first has already landed: latest is the new release and the versioned copy is absent until a re-run, which is idempotent. Since RM368 the send is already create_commit over an explicit operation list, so one commit carrying both paths is a small change; it was not taken there, because two commits is the documented behaviour and nothing has needed the window closed.
  • A versioned path is only as stable as the author's version: is. Nothing reads the remote before writing, so re-publishing without bumping the version overwrote data/<name>/v<version>/ with different bytes — the same overwrite the flat path has always done, but under a name that invites caching. Closed in 0.6.1 (RM88): the versioned path refuses unless --force. Before either write, the published manifest.json at that path is read and its artifact.digest compared with this module's — and, since RM353, the sha256 of every file each manifest declares, and since RM368, every other field of the manifest. The digest excludes logs, provenance, logo and readme by design, so comparing it alone let a second publish of one version replace its logs without --force; and the published manifest is replaced by the upload too, so the same data under a new title replaced it with no arm noticing. A different digest, a declared file whose hash differs, one declared on one side only, or any other field that differs raises PublishCollisionError, whose message names the arm (a different artifact; the same compiled data but different declared files with the names; a different manifest.json, at the dotted paths). A remote manifest whose bytes hash to this one's is the same release with no field compared; a declaration with no legible hash is not compared. The stamps of the act of compiling (upload.COMPILATION_STAMPS: compiled_at, compiled_by) are left out, so a recompile of an unchanged spec republishes as a no-op; every other Compilation field (COMPILATION_COMPARED, toolchain among them) is compared, and a test holds the two sets equal to the model's fields so a new one must be classified. The CLI reports it as ALREADY PUBLISHED rather than UPLOAD FAILED, because the module is fine and the remote is what disagrees. What about the gate is a decision rather than a detail:
  • Identical bytes are not a collision. The comparator is the digest and the declared hashes, not presence — a presence check would refuse exactly the re-run this section documents as the recovery when the second commit fails.
  • The flat path is not guarded. It means latest; overwriting it is what it is for, and the whole point of the versioned copy is that it is the one that does not move.
  • It fails open. An unreadable published manifest is an unknown, and nothing established a collision, so the publish proceeds with a warning. Failing closed would make a network flake demand --force and train an author to pass it by default. The dry run's read fails open the same way (RM368).
  • A recompile under a newer compiler trips it, correctly: P4 pins byte-reproducibility to compiler_version, the recorded toolchain and the resolved reference, so the path really would come to hold different bytes than the ones it was published with. The refusal says so, because "but I changed nothing" is the first thing its first user will think.
  • A collision the two recorded toolchains explain is ToolchainDiffersError (0.8, RM342). When both manifests carry compilation.toolchain and manifest.toolchains_differ answers True, the refusal is this subclass of PublishCollisionError, with reason == "toolchain_differs" and both toolchains as published_toolchain / local_toolchain, plus content_unchanged (True when both manifests carry one content_signature, False when the content moved as well, None when either states none: the reason names a co-occurring difference, never a proven cause); the CLI prints TOOLCHAIN DIFFERS from a branch placed before the collision's. It still refuses unless --force — the versioned path keeps the bytes it was cut as — and an unrecorded toolchain on either side, or the same one on both, keeps the plain collision with reason None. A subclass so every existing handler of the collision still refuses; tell the two apart by type or by reason, never by the message.
  • upload_folder adds and replaces; it never removes — so a republish leaves a union of two releases. A recompile that stops emitting a table leaves the previous release's parquet at the path beside a manifest that does not attest it, and this happens on the flat path, every time. The format's answer is that an unattested file is not part of the module — manifest.artifact.files says which parquets are, and artifact.digest is a Merkle root over exactly those — so a manifest-first reader never sees it. What stops that being true is the reader: MODULE_LIFECYCLE § 6.8 records that the discovery path fetches no manifest and probes named files, so there a fossil parquet is indistinguishable from a live one and the module reads as the wrong kind. delete_patterns was considered and declined: it cleans nothing on a module nobody republishes, does nothing for a consumer that probes rather than reads, and is one wildcard away from dangerous — HuggingFace filters those patterns with fnmatch, whose * crosses path separators, so a single *.parquet in the allowlist would delete every archived version's parquets. The allowlist is literal basenames today and the archive survives by that accident. The fix that closes it is the reader's, and it is asked explicitly in INTEGRATION_0_6 § 2.8.
  • Nothing already published moves. The flat path keeps being written, so every module published under the old layout stays exactly where it is and keeps resolving. That is the argument for writing both rather than migrating.

  • Nothing prunes the versioned copies. The dual write leaves one full artifact set per release in the collection forever, which a consumer mirroring it pays for. Raised by just-dna-lite in S35 as a consequence rather than an objection — it does not affect discovery — and recorded here rather than fixed, because a retention policy is the collection owner's decision and not the publisher's.

Both questions above were answered by just-dna-lite on 2026-08-17 (S35), and the segment spelling is settled: v<version> verbatim stays. 1. Their scan matches only v-plus-integer (^v(\d+)$, compared with int()), so v1.0.0 does not match and v10 would sort under v9 — but the correction that decides it is that the fallback lives only in their generic fsspec branch. HuggingFace has its own discovery branch with no version fallback at all, so on this path no spelling is read today and the segment cannot be chosen to suit one. A bare vN would collide two patch releases at one path and buy nothing, so verbatim it stays. Both halves of their fix are theirs, and they are unscheduled: teach the HF branch a versioned fallback, and replace the regex with just_dna_format.identity.Version, which already parses and orders. 2. No, and by construction. Both of their discovery branches call fs.ls at exactly one level, never fs.find, a ** glob or a recursive listing, and their probe asks fs.exists on named files rather than listing the directory — so a nested data/<name>/v<version>/ is never enumerated. Verified in their tree by search rather than assumed.

Until their discovery half lands, the versioned directory is written-but-not-yet-read: the flat path is unchanged and remains latest, so following it is still correct.

ClinVar reference snapshot (clinvar_build.py, [dev])

ClinVar is a second, complementary reference beside the Ensembl snapshot: ~4.4M clinically-curated GRCh38 records (~200 MB gz), 1.54M of them carrying no rsid, so a clinical module can enrich offline without provisioning the 14 GB dbSNP cache. Build ([dev], polars) and download (core) split the same way the Ensembl snapshot does:

flowchart LR
  ncbi["NCBI clinvar.vcf.gz GRCh38"] --> build["clinvar_build.py (dev)"]
  build --> pq["clinvar/data/*.parquet + release.json"]
  pq --> hf["HF datasets/just-dna-seq/clinvar"]
  hf --> cache["local cache"]
  cache --> link["clinvar.py lookup_loci (core)"]
  link --> chain["enrich() chain"]
  chain --> res["resolution.csv (source=clinvar)"]
  res --> comp["compiler: weights.parquet"]

build_snapshot(vcf, out_dir) turns the NCBI ClinVar GRCh38 VCF into the per-chromosome parquet snapshot the clinvar link reads (out_dir/data/clinvar-chr{N}.parquet, same layout as the Ensembl snapshot so one DuckDB view shape serves both). One row per ACGT ALT allele (symbolic/structural and >50 bp alleles are skipped and counted), with the columns:

chrom, start, ref, alt, rsid, variation_id, allele_id, gene, genes, clin_sig, clin_sig_raw,
review_status, review_stars, condition, molecular_consequence, variant_type, origin

The resolver link reads only chrom/start/ref/alt; the rest is annotation the parquet carries for RM4 (it never enters resolution.csv — orthogonal axes, P5). clin_sig is folded into vocab.VALID_CLIN_SIG by an explicit severity order (a multi-valued CLNSIG picks the most severe, splitting on |///, so Pathogenic,_low_penetrance is recognised) while clin_sig_raw keeps the verbatim CLNSIG (lossless, auditable). Since 0.7 the fold itself lives in clin_sig.py, not here — see below. A release.json records provenance (clinvar_file_date from the VCF ##fileDate, source_url, source_sha256, record_count, built_at, builder_version) — the values clinvar.clinvar_dataset_label turns into the dataset a drafted module's licence row records (RM4), which is what the clinical cross-check reads back to know it would be comparing a value against itself.

Coordinate convention — no shift. start is the 1-based VCF POS, passed through unchanged; the Ensembl snapshot uses the same convention, so a variant resolved by either reference lands on the same coordinate. (ResolutionRow.start's field doc was corrected from "0-based" to 1-based accordingly.)

The parquet is byte-reproducible across rebuilds (rows sorted per chromosome; only release.json's built_at varies). polars is a [dev], guarded import — the runtime clinvar link is polars-free. download_clinvar_vcf streams the NCBI VCF with the core httpx (atomic .part rename, sha256 while streaming). VCF-parsing idioms are leeched from just-dna-lite's v1_port.clinvar.

One significance normalizer, shared (clin_sig.py, 0.7 — RM134)

normalize_clin_sig(raw) is the single raw-token → VALID_CLIN_SIG fold, and it moved out of clinvar_build the moment a second source started reporting a significance. The reason is not tidiness: a concordance check's entire output is a comparison of two normalized calls, so two hand-written maps make any drift between them read as a disagreement between the authorities rather than between our own tables. The module imports VALID_CLIN_SIG and nothing else, so a runtime pass reads it without the [dev] extra the builders need.

Two defects were fixed on the way out, and neither is visible from ClinVar's side. The map's keys are underscored because that is how ClinVar spells CLNSIG; PubMind spells the same concepts with spaces, so two of its six tokens fell through to other:

Token before after
Uncertain significance other uncertain_significance
Conflicting other conflicting

ClinVar's own Uncertain_significance and Conflicting_classifications_of_pathogenicity mapped correctly all along, so the two sources would have been reported as disagreeing where they agree — on the largest disagreeing class in the measured corpus join. The repair is a whitespace→underscore step in the tokenizer, an identity on every existing key, plus a bare conflicting key. Both registries are asserted as equalities at import: set(CLIN_SIG_SEVERITY) == VALID_CLIN_SIG and set(CLIN_SIG_MAP.values()) == VALID_CLIN_SIG.

A composite is still resolved by severity, and PubMind gets ClinVar's answer. Benign/Likely benign folds to likely_benign, exactly as ClinVar's Benign/Likely_benign does. PUBMIND_ASSESSMENT.md wrote that mapping as benign while it still proposed a second map; teaching one spelling a different answer from the other is the drift the single map exists to remove, so the assessment's line is superseded here rather than implemented.

One more tightening rode along: a whitespace-only or token-less value is not_provided rather than other. "The source states no classification" and "the source stated something we do not model" are different answers, and only the second is a disagreement. No ClinVar CLNSIG takes that shape, so nothing built to date moves.

The clinical cross-check (clinical.py, offline)

verify_clin_sig(variants, resolution_rows, *, reference) compares each authored clin_sig against the ClinVar snapshot's own and returns the disagreements. It runs inside enrich() by default (--verify-clinsig/--no-verify-clinsig) and lands on EnrichmentResult.clin_sig_conflicts.

Tier note, because the obvious reading puts it in the wrong place. This check needs no network at all — the snapshot is local — yet it belongs here rather than in the compiler. The boundary is not online-vs-offline but does the check need a reference: the compiler is inject-only by charter and holds no ClinVar. Reading "offline ⇒ compiler" would put it in the tier that cannot host it.

Allele-exact, never rsID-level, and the snapshot itself shows why: rs334 at 11:5227002 carries T>A as pathogenic (2 stars) and T>G as likely_benign (1 star). One rsID, one locus, two opposite calls. Comparing by rsID would report a module that is simply right. The allele the annotation is about is taken from effect_allele when set, otherwise from the genotype allele that is not the reference; when neither pins it down, the comparison falls back to the whole locus and reports only if no record there supports the authored call.

What counts as a disagreement is coarser than the vocabulary: pathogenic vs likely_pathogenic is a difference of confidence inside one conclusion, not a conflict, and anything paired with uncertain_significance/conflicting/not_provided is not a conflict either — ClinVar has no opinion to disagree with. Opposed calls (pathogenic-class vs benign-class) are the finding worth acting on and are flagged as such.

A module drafted from this very snapshot is not checked, and the run says so (0.5.2). Where the clin_sig came out of draft_gene_panel, the comparison is a value against itself: a consumer measured 27.1 s with the check on and 2.6 s with it off on a 7,818-row panel, byte-identical output, and 0 conflicts either way — necessarily 0. That zero is the problem rather than the cost: it looks like evidence and is none.

A pass that contributes nothing records no terms (RM142, 0.7). Every fact pass writes the licence row for the source it consulted — that rule is unchanged and is what the compile gate reads. What it means is this module uses this source, so the row belongs to a pass that actually put a row in a table, never to one that merely ran. The family already worked that way by construction: gene_metrics, frequencies, assertions and gene_validity pass {row.source for row in out} to record_source_terms, so an empty pass records nothing. clingen.py built a fixed row and wrote it unconditionally, and a single-variant module on a gene ClinGen does not curate therefore shipped a ClinGen obligation and warned about a licence disagreement it had no part in.

The predicate is what this run covered — not the table's contents, which include what an earlier run merged in and already recorded, and not the absence of missing genes, which would drop the declaration from any module carrying one uncurated gene beside a curated one. The compiler cannot check this: its orphan warning exempts the annotation layer deliberately (RM46), so only the pass knows. ClinGenResult.source_row is still returned whatever happened, because the terms of what was consulted are a real fact with a different meaning.

The marker is machine-written, not authored (RM4, 0.6). clinvar_draft stamps the release it copied the rows out of into the dataset column of the clinvar/annotation row it already had to write in the licence table — clinvar_2026-06-27, from clinvar.clinvar_dataset_label, which reads release.json's dataset. Since 0.8 (RM303) clinvar build writes that dataset through snapshot.clinvar_release_label (clinvar_file_date, falling back to source_sha256), and the snapshot model reads an older ClinVar release, which has only clinvar_file_date, through the same formula. clinical.tautology_reason recomputes that same label from the snapshot in hand and compares. Both sides call the one function, so the writer and the reader cannot drift apart — and this drift would be silent, since a disagreement about the label does not fail, it just never matches.

Widening a panel from a newer snapshot withdraws the label rather than re-writing it. merge_sources_csv is never-clobber so a curator's hand-written terms survive a re-run, and dataset inherited that protection the moment RM4 made it load-bearing — leaving the row naming the older release while half the rows came from a newer one, in the column manifest.sources publishes. licensing.withdraw_stale_dataset blanks it instead, and only when rows were actually added: a module carrying two releases has no single release to name, so the honest value is unknown, and an empty dataset skips nothing. The terms on the row are untouched. Re-labelling to the newer release was the other candidate and it is the same false claim pointing the other way.

It keys on dataset rather than on the module's panel: block because the claim is provenance — these rows came from this snapshot — and the tool that copied them is the authority on it. Asking an author to maintain a declaration whose only reader is one skip is bureaucracy the enricher exists to remove. panel: is deprecated in 0.6 and reads nothing here any more; a 0.5 module whose pin matches gets the check run, which is the safe direction. Only an established match skips: no licence table, a ClinVar row with no dataset, a different release, or a release.json that cannot be read all leave the check running. The row must be at the annotation layer — enrich() writes a second clinvar row at the resolution layer for the coordinates it looked up, and a coordinate is not a copied clinical call.

The release label is half the question; the drafter's digest is the other half (RM73, 0.6). A matching release says the rows were copied out of this snapshot. It says nothing about whether they still are — a cell edited by hand after the draft is no longer a copy of anything, and no module-level fact can see that. RM4 shipped that hole knowingly and put an expensive per-row audit behind strict to recover what the marker was blind to.

clinvar_draft now also stamps SourceRow.draft_digest: a hash of variants.csv projected onto (rsid, chrom, start, ref, alts) → clin_sig, the identity cells and the column this check reads. clinical.tautology_reason recomputes it. Skipping requires both — this release and an unmoved digest — and either half missing runs the check in full.

what the module records what happens
no licence row, no dataset, a different release, an unreadable release.json the check runs
this release, no digest (hand-written label, or a module drafted before 0.6) the check runs
this release and a digest that still matches skipped as a tautology, in both modes
this release, digest moved — someone edited a clin_sig the check runs, over the whole table

Same in both modes, and that is the RM4 ladder collapsing. strict used to pay for a per-row look-up because deciding whether a value was still a copy was the look-up; the digest answers it offline, so there is nothing left for a mode to switch on. EnrichmentResult.clin_sig_audit and its copied/authored/no_record split are gone — what remains is clin_sig_comparison, carrying compared, no_record and the conflicts. strict still does not escalate a conflict into a failure; that is this check's standing exception and it is unchanged.

Why the digest is scoped to the column and not the row. A ClinVar-drafted module always has edited rows: genotype is a placeholder the human is required to fill, so a whole-row hash would be invalidated on every drafted module and the skip would never fire once. Scoped to clin_sig, filling the stub leaves it alone and editing the call moves it — exactly as sensitive as the question.

The limit that remains, stated rather than left to be found. The digest covers the whole table, so it means no checked value has changed since the drafter last wrote. A row hand-authored before a later re-draft is covered by the new stamp along with the drafted rows, and escapes the check. That is strictly narrower than what it replaces — the module-level marker let every hand-edit escape — and any later edit re-enables the check.

The same mechanism, the same way, for the other two providers. pgx_draft writes function_status out of CPIC and pgx._function_conflicts compares that column against CPIC; clinpgx_draft writes evidence_level out of ClinPGx and the ClinPGx check compares that. Both were tautologies nobody had filed, both publishing a structurally guaranteed findings=0. In enrich_pgx the skip is per leg: the CPIC leg records tautology while PharmVar, an independent authority, still runs and the record aggregates both.

The skip carries its reason on EnrichmentResult.clin_sig_not_checked, because an empty clin_sig_conflicts says two opposite things on its own ("compared everything, nothing disagreed" and "never compared"), and a consumer reading the first when the second happened has been told a check passed that was never put. Its values are not_requested (the author's own --no-verify-clinsig), no_snapshot, unusable_snapshot (present but not queryable — compare_clin_sig returns None rather than a comparison of zeros), the tautology sentence, or None when the check really ran. Where a human typed the clin_sig, nothing changes — that is the case this check exists for.

The concordance record (concordance.py, offline) — RM130 + RM134 § B

The check above counts its findings and, until 0.7, kept none of them. clin_sig_concordance(variants, resolution_rows, *, reference, pubmind_reference=None, sources=None, spec_dir=None, checked_at=None, clinvar_comparison=None) in clinical.py consults every authority and returns a ConcordanceRecord whose two tables concordance.write_concordance_tables puts beside the spec. enrich() calls it once per run and commits both tables at the gate, so a refused strict run leaves none behind.

file key carries
clin_sig_concordance.csv (variant_key, genotype) authority_concordance, authored_position, opposed, and the module's own call
clin_sig_authority_calls.csv (variant_key, genotype, authority) that authority's normalized clin_sig, its raw token, and its confidence in its own units

None rather than two empty tables when no authority could be consulted at all, and the distinction is the whole tri-state: two empty tables are a claim (nothing here is contested), while None says the question was never put. A caller must not write a file that says the first when the second happened, and enrich() writing nothing on None is what leaves an earlier record readable instead of overwriting it with a comparison nobody made.

An empty record is written only over a record already on disk (RM370). Like every enricher sidecar, the pair goes through layout.nothing_to_write: zero rows and no file there writes nothing, so a module with no clin_sig cell, or one where nothing is contested on its first run, carries no concordance file. A record already on disk is still rewritten to empty, because that is how a subject the authorities stopped contesting leaves it. Asked and found nothing contested is recorded where it always was counted, the clinical_significance record's denominator in verification.json; the compiler reads an empty record and an absent one alike (no manifest block). The one empty record still written is one the module's overrides.csv answers: its emptiness is what compiles to overlay_answer_vindicated, where a missing table would read overlay_targets_missing_table.

The three-way check subsumes the two-way rather than running beside it. With no PubMind snapshot that authority's call reads unchecked on every subject, and the degenerate case is exactly the ClinVar-only finding: the same subjects are contested, the same conflicts are logged in the same words, and no author meets one disagreement twice. What changes is what the record withholds — authority_concordance reads unchecked rather than single, because one authority speaking while another was never asked is not corroboration.

Each authority is a leg, and a leg has three states (clinical.AUTHORITY_LEG_STATES):

state means contributes
consulted a snapshot answered that authority's call per subject
unchecked no snapshot, or one present that would not answer an unchecked call per subject — never an absence, never agreement
tautological this module's rows were drafted out of this very snapshot and have not moved nothing at all; the reason is on the leg and reported separately

AUTHORITY_ORDER is (clinvar, pubmind) and it is walked rather than restated: it fixes the order the detail rows come out in, which matters because the table becomes a parquet whose bytes depend on row order.

PubMind's several PVIDs over one allele fold to one call, and the camp guard runs first. The detail table is keyed (variant_key, genotype, authority), so a subject carries one row per authority and something has to stand for the set. Where the records straddle the pathogenic/benign line there is no representative call: fold_authority_records answers conflicting, the vocabulary's own word for it, which sits in the undecided camp and so opposes nothing. Folding by severity there would silently answer pathogenic — a winner picked by an ordering nobody defined. Within one camp the fold is the shared normalizer's own severity rule, the same one that resolves a composite token, so Benign and Likely benign in two rows fold exactly as Benign/Likely benign does in one cell. Confidence is withheld unless exactly one record stands behind the call: PubMind's 0–3 count is per record, and any function of two of them is an arithmetic nobody defined. The multiplicity is counted on the record (multi_record_subjects, internally_contested) rather than discarded.

Not merge-not-clobber, and it is the one derived sidecar that is not. Every other one gap-fills because a recorded row might carry a curator's judgement; this one carries none, since the judgement about a contested subject goes in overrides.csv. Merging would be actively wrong: a subject the archive stopped contesting has to leave the record, because a conflict that stops being reported is exactly how an author learns the archive caught up with them.

classify_concordance is a pure function over N authority outcomes, and it knows nothing about ClinVar. That is what makes the arity property testable at the three and five authorities the vocabularies were designed against rather than at the one the producer reaches today. CLIN_SIG_CAMP moved here from clinical.py for the neighbouring reason: the record and the two-way check must draw the opposed-versus-differing line in the same place, and two maps for one distinction is how a drift in our own code comes to read as a disagreement between two archives.

The tautology skip reaches the record too, and since 0.7 it is decided per LEG. Where a module's clin_sig column was drafted out of a snapshot it would be compared against and has not moved since, that comparison is a value against itself and is guaranteed to find nothing — so writing two empty tables would publish nothing here is contested on no evidence, which is the findings: 0 defect wearing a new file. But skipping the whole check would be the opposite error once there are two authorities: a module drafted from ClinVar still gets a real comparison out of PubMind, which copied nothing, and throwing that away to suppress the hollow half is what enrich_pgx already learned.

So clinical.is_tautological_leg(sources, authority, dataset, spec_dir) states the conjunction once — the licence row names this release of this source, and the drafter's digest over the checked column still matches — and every leg calls it with its own label. The checked column comes from DRAFT_PROJECTIONS, derived rather than restated, so a drafting provider added later says which cell it copied without this module learning about it. tautology_reason is the ClinVar instantiation and keeps its shipped wording, because a warning's text is an API. Omitting sources/spec_dir establishes nothing and the comparison runs: an unknown is never a permission to skip.

The subject list is built from the comparison that already ran, never from a second pass over the snapshot. A subject that produced a conflict is rendered from that conflict's own record, so the verdict and the finding are one fact told twice rather than two computations that can drift — and re-asking would cost the whole comparison again.

ClinSigConflict no longer names its authority in a field name. It carried clinvar: str until 0.7; authority and authority_clin_sig replace it, with clinvar kept as a read-only alias so an existing caller keeps working. The rename is the point: a finding whose field is named after one archive costs a rename — major-only work — the moment a second one arrives.

One thing the record makes visible that the two-way check cannot. At a single authority every reported conflict is opposed by construction, because the check only reports where both sides are opinionated and their camps differ, and pathogenic/benign are the only two opinionated camps. So the differing-but-not-opposed case had no producer at N=1 — _clin_sig_detail's second group was unreachable. It became reachable the moment PubMind arrived: two authorities can disagree with each other while neither contradicts the module, which is discordant + matches_some and is what the record was shaped for.

Severity: warning-tier in both modes, escalating in neither (@clinsig-never-escalates), with more force at two authorities than at one. A disagreement with a literature miner's aggregate over the field is a statement about that extraction's limits at least as often as about the module — the measured corpus join agreed 62 % of the time — so discordant is a fact about the field rather than a defect to gate on. clinical.concordance_sentences produces the warnings, each carrying its denominator and the authorities that answered; clinical.concordance_notes produces the info-tier lines for the legs that did not run, because a leg that silently did not run reads as a leg that found nothing. A run that found nothing contested emits neither: a check that cannot fail reports no zero, and neither does one that ran and found nothing. The two warning stems a consumer greps, pinned by test because a warning's text is an API:

{n} of {m} subject(s) put to the authorities are contested, {k} of them on opposed calls
{n} of {m} contested subject(s) are a disagreement between the authorities themselves

The leg notes are info-tier and deduped against the two-way skip's own sentence — the ClinVar tautology's prose comes out of tautology_reason on both paths, so a drafted module would otherwise read it twice in one run. Matched on the sentence, not on the skip key, so a reword stays deduped.

Nothing resolves a split, at two authorities or at five. There is no majority, no consensus call and no resolved winner, and module_spec.yaml's optional authority_precedence: is recorded and computed with by nothing — it says whose call the curator weighted while deciding, so a consumer can see the stance, and no tier reads it. Choosing between a declared order and a majority needs a weighting model this workspace has declined to invent three times.

Has the disagreement you answered moved? (RM151, 0.7)

RM117 shipped the first of two signals about an answer an author has already recorded: a subject that has left the record, which the compiler computes offline because the record holds contested subjects only and is rewritten whole. This is the second, and it lives here because it needs the archive's value now against its value at record time — the first of those requires a consultation.

An overrides.csv row against clin_sig_concordance.csv is a judgement about a particular disagreement: the archive said X, the author says Y, and the reason column explains why. If the archive later says Z, that reason was written about a value that is no longer there. Nothing in the record distinguished a justification that still describes the disagreement on file from one that describes a disagreement since replaced by a different one.

The baseline is the previous run's clin_sig_authority_calls.csv, and it is the only one this format keeps. That table records what each authority actually said — clin_sig, the verbatim clin_sig_raw, and the dataset it came from — keyed (variant_key, genotype, authority), so the comparison is recorded-call against fresh-call per authority. Do not expect this for a table that records no prior value (@probe-names-the-table): an overlay row against frequencies.csv or resolution.csv has no recorded baseline at all, so a general the value moved check would be answerable for one table and silently absent for every other. The finding names its table for that reason.

It is read before the commit, and that ordering is the whole feature. write_concordance_tables replaces the file whole, so the previous run's rows exist only until this run commits. clinical.answered_call_shift is called in the staging phase — the same phase every other product of the run is computed in, for the unrelated reason that a refused strict run must change nothing — and an AST guard pins the read above the write, because no assertion over a return value can see statement order.

A move is therefore observable exactly once, by the run that notices; the run after that compares against the new baseline and is silent. That is a limit rather than a bug, and it is the honest shape for an observation: binding an overlay row to the value it justifies is a mechanism this format does not have, and RM117's three objections to giving outranks a severity consequence all turned on exactly that missing binding. None of them is an objection to noticing that the value moved.

Three states, and the third is what the check is for. A call recorded on both sides with the same classification is unchanged, dataset moved or not — a re-released archive saying the same thing has not moved the disagreement. A call that moved, or that went recorded → no_record (or back), is a shift. Everything else is withheld and reported as a note rather than as agreement: no_prior_record (this run is writing the first record, the previous one would not parse, or the authority was unchecked when the answer was written) and unchecked_now (nobody could ask this run). Neither ever reads as nothing moved.

A move this tier's own normalizer made is reported apart from the archive's. When both sides recorded the same verbatim clin_sig_raw and only the normalized member differs, what changed is how clin_sig.py reads the token — a fact about our code, with nothing for an author to do. Folding it in with the archive's revisions would accuse a source of a change we made.

A subject that left the record entirely is not this finding. It is overlay_answer_vindicated, which the compiler reports as good news, and hanging a second and gloomier finding on the same overlay row is exactly the already firing with the wrong words failure RM117 was.

Warning-tier in both modes, escalating in neither, with more force than the record itself: nothing here is even a disagreement, it is a note that the record an author reasoned over was rewritten underneath their reasoning, and gating on it would make an artifact refuse over an archive's release schedule. Silent on a module with no overlay answers, which is every module today. The two warning stems, pinned by test:

{n} of {m} answered subject(s) rest on an authority call that has moved since the answer was recorded
{n} answered call(s) differ only after this release's own normalization

The wording observes and never adjudicates, and that is a test rather than a convention: a word-boundary grep refuses correct, wrong, mistaken, vindicated, confirmed and their siblings in every message. The disagreement you answered is not the one on record now is a statement about the record; your answer may be wrong is a verdict, and this format does not put a verdict under a check that cannot see the reasoning.

PubMind as the second authority (pubmind.py, offline) — RM134 § B

pubmind.py is the runtime reader over the snapshot pubmind_build writes: duckdb, in the ordinary install, mirroring clinvar.py's split against clinvar_build.py (@duckdb-vs-polars). lookup_pubmind_calls(reference, alleles) answers what does PubMind say about this allele, and pubmind_dataset_label(reference) reads the release label back out of release.json.

There is deliberately no lookup_loci beside it. PubMind's coordinates are PyEnsembl back-mappings of text an LLM extracted, so they are annotation and never resolution: nothing it produces may enter resolution.csv, whose authority column is a different word for a different thing (@source-vs-authority).

The snapshot is located by locations.resolve_pubmind_reference — explicit path → $JUST_DNA_PUBMIND_CACHE → the default cache directory — and there is no ensure_pubmind_snapshot to pair with it: the snapshot is operator-built, pubmind publish refuses, and a missing one is unchecked rather than a download a run should attempt. enrich(pubmind_cache=…) overrides the lookup, and just-dna-enricher enrich --pubmind-cache <dir> is the same override from the command line. The flag was owed for a release: § B built the check while cli.py belonged to the sibling lane, so the second authority shipped reachable only from Python or the environment variable.

PubMind snapshot (pubmind_build.py, [dev]) — RM134 § A

PubMind (Wang & Wang, Nat Commun 2026, doi:10.1038/s41467-026-76834-4) extracts variant–disease–pathogenicity assertions from 41.7 M abstracts and 5.4 M full texts with an LLM. It is a source, of the same kind ClinVar is: an authoritative annotation source. Nothing it produces may enter resolution.csv — its coordinates are PyEnsembl back-mappings of extracted text, and resolution.csv's authority column is a different word for a different thing.

There is exactly one per-variant channel. The web API has two endpoints, neither takes a variant, and both state that per-record detail is withheld; the per-record LLM_reasoning and evidence passages — the genuinely novel part — are reachable only through CHOP's licensed full database. So the input is the ANNOVAR-redistributed hg38_pubmind_db.txt.gz, whose columns are VCF-style despite the ANNOVAR packaging: no - alleles, a one-base deletion written 1 1014264 1014265 CC C with the anchor base retained, and Start the 1-based POS. A join needs no coordinate translation.

build_snapshot(table, out_dir) writes one out_dir/data/pubmind.parquet plus release.json:

chrom, start, ref, alt, pvid, clin_sig, clin_sig_raw, pathogenicity_score, confidence, derivation

The names are unprefixed on purpose — clin_sig, not pubmind_sig. The ClinVar snapshot already uses them, the source is the file, and one column vocabulary across every snapshot is what lets a check read N authorities with no per-source mapping. pathogenicity_score is nullable and null means not computed, never 0.0; confidence is PubMind's own 0–3 evidence-depth count and is deliberately not normalized against ClinVar's review_stars — different instruments, and folding them into one number would be three axes in one field.

Two thirds of the file is not a genotypable position, and every dropped row is counted. When PubMind recovers only a protein change from the text it back-maps through the transcript and writes out every codon that could encode it. derivation records what survived:

derivation What it is
direct the source row was already one base against one base
codon an equal-length block differing at exactly one position, decomposed onto that base
indel a length-changing row, kept but marked — upstream left-normalization is unverified

and PUBMIND_DROP_REASONS records what did not: off_target_chrom, non_acgt (16 rows in the 2026-08-24 file whose alt is 0 or N), ref_equals_alt (523), no_pvid, unparsable_position, multi_substitution (a block needing two or three simultaneous changes — a statement about the protein, not a position), and identical_duplicate (two codon rows decomposing onto one row identical in every column). The registry is walked, so input_rows == record_count + sum(dropped.values()) is an equality over it: silent truncation reads as full coverage.

no_pvid earns its place twice over. The PVID is PubMind's record id, and the whole snapshot is organised around record identity — a verdict with no id cannot be attributed, cannot be deduped against its twin, and cannot join the multiplicity accounting; carrying one as a null would also merge distinct records under identical_duplicate, which compares whole rows. A bad cell is different from a bad row: a malformed pathogenicity_score or confidence withholds that one value as null and counts it in unparsable_score / unparsable_confidence, keeping the row. NaN and inf count as unparsable rather than being stored, because float() accepts both and they would otherwise poison every comparison downstream; a non-integral confidence is withheld rather than truncated, since 2 is a definite count the source did not state.

A contested coordinate keeps every PVID as its own row, and that is the finding. Consolidation into a PVID is keyed on the text the model extracted, never on a coordinate, so one physical variant fragments into many records whose verdicts disagree — at chr6:26092913 (HFE C282Y) the table holds eight, split across four verdicts, one of which pairs rs1800562 with TMPRSS6 and is exactly the shape _gene_locus_conflicts catches. Collapsing them would mean choosing a winner by an ordering nobody defined, which is mode() over an unsorted group. release.json records multi_pvid_keys, max_pvids_per_key and contested_keys — the last being how many of those coordinates actually disagree, which is why the multiplicity may not be tidied away.

The parquet is byte-reproducible across rebuilds (rows sorted by chromosome in karyotype order, then start, ref, alt, pvid); only release.json's built_at varies. release.json also carries the source's sha256 and its ETag and Last-Modified — all three available, all three recorded, so an upstream revision becomes a finding rather than a silent change of answer — plus redistributable: false and the reason. A local --table establishes neither header nor the URL, so all three are recorded as null: unknown rather than absent, and never a default the build did not establish. download_pubmind_table translates httpx into PubMindUnavailable, a subclass of PubMindBuildError, so one except arm catches a moved bulk URL as well as a malformed table while a caller who needs to tell an outage from bad data still can (@client-exception-contract; the subclass makes a caller's except order load-bearing). polars is a [dev], guarded import.

MANE snapshot (mane_build.py, [dev]) — RM168

MANE (Matched Annotation from NCBI and EMBL-EBI) publishes one agreed transcript per protein-coding gene, matched base-for-base between a RefSeq and an Ensembl accession. This workspace already leaned on it before it was code: the CIViC identity protocol pins a numbering frame with MANE.GRCh38.v1.5.summary.txt.gz, "downloaded once and cited" — a procedure step in a probe document, where every other reference table here is a cache with a location and a recorded release. The 33 curated name→identity answers that shipped with that protocol were derived in a frame nothing in the code could read. This builder is that frame, cached and pinned.

just-dna-enricher mane build --download --out data/caches/mane      # discover the newest version, then pin it
just-dna-enricher mane build --release 1.5 --download --out data/caches/mane
just-dna-enricher mane build --summary … --changed … --not-in-mane … [--versions README_versions.txt]
export JUST_DNA_MANE_CACHE=data/caches/mane

MANE is the default, not the answer — and the bound is measured, not asserted. The summary carries CDKN2A twice for one GeneID: NM_000077.5 / ENST00000304494.10 marked MANE Select beside NM_058195.4 / ENST00000579755.2 marked MANE Plus Clinical, two CDS numbering frames for one gene. Plus Clinical is a fraction of a percent of the rows, which is exactly the argument: a case occurring in a third of a percent of genes is precisely the case a remembered accession hides and a table shows. So MANE_status is carried as a column and never collapsed — a builder keeping one row per gene would drop those rows and reintroduce the blind spot the item closes.

RUNX1 is the other half of the bound, and it confirms the protocol rather than fixing it. RUNX1 is a single row, and the 27-residue RUNX1c/RUNX1b offset the identity protocol derived by translating each isoform's CDS is not in MANE and cannot be. The table therefore makes the CDKN2A class of problem visible and is silent on the RUNX1 class, and a pass that treated it as an oracle would be wrong in a way the file itself cannot warn about. That sentence is in the module docstring, in release.json's notice, and printed by the build command, because it is the one thing a reader of this lane has to carry away.

Scope is a transcript-identity aid. Generating c./p. notation is a separately deferred feature with its own unanswered questions, and nothing here proposes it.

Three files, and the other two are why this is a snapshot rather than a download

Table (data/…) What it answers
summary.parquet the frame, and a cross-map: NCBI GeneID, Ensembl gene, HGNC id, symbol, both nuc/prot accession pairs, MANE_status, and GRCh38 coordinates on the NC_ accession
changed_select_accessions.parquet every gene whose MANE Select moved, the release it moved from, and Update_Affects_CDS — the numbering-frame axis, stated by the source
protein_coding_genes_not_in_mane.parquet the genes MANE deliberately has no answer for, each with a reason

The second is the currency check, and taking it in the same pass as the summary is the point: shipping the cache without the thing that notices it going stale is the defect this item is about. A MANE Select change that moves the CDS moves every c. and p. derived in that frame; one that does not, does not — and Update_Affects_CDS says which. Old_MANE_Version spans several releases back, so each row also says how long that gene's frame had been stable. A gene absent from that table has had a stable frame, which is a positive statement the cache can make and a memory cannot; RUNX1, CDKN2A and VHL are all absent from it.

The third is @unreachable-not-absent served by the source. "MANE has no answer for this gene" becomes distinguishable from "nobody asked", with the reason attached — and pending MANE review is a third state on its own, neither absent nor decided. The reason vocabulary is derived from the file and recorded in release.json's excluded_reasons; it is not written down beside it, because a roster stated in prose is a registry nothing iterates (@registry-completeness). The test asserts the counts as an equality against the fixture rather than against a list of strings.

Update_Affects_CDS is stored twice, the split clin_sig / clin_sig_raw established: a nullable boolean a consumer queries, and the source's own token so the mapping stays auditable. A cell that is neither Yes nor No withholds the boolean and lands in unparsable_update_affects_cds — the source said something we cannot hold, which is a different finding from the source saying nothing. unparsable_gene_id and unparsable_coordinate are the same shape for the other two cells that can be malformed, and every one of them is in release.json.

A bad cell keeps its row; a bad field count does not. Everywhere above, a cell the column cannot hold withholds its own value and the row survives, because the rest of the row is still the source's own statement. A ragged row is refused outright (@ragged-csv-row): the cells past the break are shifted, so a short summary row would enter the numbering frame carrying null accessions and read as coverage, and a shifted one would put the wrong accession under the right gene. Whole-file damage is a structural failure, and refusing is what a builder may do about one. _open_table also translates gzip/decoding failures into ManeBuildError, because the opener is chosen on the .gz suffix and a file that is named .gz and is not one — a proxy that decompressed the download and kept the name — otherwise escapes past the CLI's handler as a traceback. EOFError is named beside OSError there on purpose: a truncated gzip raises it and it is not an OSError.

The version is pinned, and release.json copies rather than restates

release_1.5/ back to release_0.5/ all exist, so a pin is a URL rather than a hope. current/ is read exactly once, for 96 bytes, to discover which version is newest — discover_current_release() reads its README_versions.txt, and everything is then fetched from release_<version>/. Pinning the mutable path and hoping is what the item is complaining about.

README_versions.txt publishes MANE Version, NCBI RefSeq Annotation Release and Ensembl Release, and two of those three appear in no filename, so the builder copies the file into release.json's versions block instead of parsing a name — parsing one would reconstruct less information than the source hands over (@probe-the-real-file). It is parsed generically, label by label, so a line MANE adds travels through. dataset is mane_grch38_v<version>, and it is None when no README_versions.txt was read: an unknown release, never one reconstructed.

A local build asserts no URL it did not fetch. source_url, source_etag and source_last_modified are per input and null on a local build, and release_url names the directory actually fetched from. A README_versions.txt handed over on disk establishes which release the files claim to be; it does not establish where the bytes came from. release.json is written atomically (@atomic-sidecar-write) — a truncated one parses as valid JSON that is simply missing keys, and locations.read_release would believe it.

The two modes do not mix, and the CLI says so rather than picking: --release without --download is refused (that would be our claim about somebody else's bytes, where --versions is the source's own), and --versions with --download is refused too, because the download fetches the release's own copy and would silently overwrite the flag. A flag that is quietly ignored is worse than one that is turned away.

The source spells its own key two ways — GeneID:1029 in the summary, a bare 1029 in the other two — so parse_ncbi_gene_id normalizes both into one ncbi_gene_id integer and the test runs it over the raw cells of both dialects (@one-normalizer-two-spellings). Without that, the cross-table question has this gene's frame ever moved? is a string comparison that silently never matches. Everything else is verbatim: MANE_status keeps its spaces, GRCh38_chr stays the NC_ accession, and the version cells stay strings because 0.91 is a MANE release rather than a number.

The parquets are byte-reproducible across rebuilds (gene id, then accession — a total order, with unreadable ids sorting last because None cannot be compared to an int); only release.json's built_at varies. Three schemas share data/, on the CPIC precedent — the glob-union hazard (@snapshot-layout-locations) belongs to readers that union data/*.parquet into one view, and this snapshot ships no reader, so anything reading it later reads by filename. resolve_mane_reference deliberately does not accept a bare .parquet the way the constraint resolver does: one file here is a snapshot missing the currency check and the negative roster.

No --use flag and no publish command. NCBI states a policy rather than a licence: it "places no restrictions on the use or distribution of the data contained therein" and, in the same paragraph, "cannot provide comment or unrestricted permission concerning the use, copying, or distribution of the information contained in the molecular databases." No restriction imposed is not permission granted, so MANE_TERMS records None on every axis (@no-named-licence) — and a declared-use gate fed an unknown skips every build unconditionally, which is a flag that does nothing (@acquisition-gate-is-not-a-read-gate). MANE is a joint NCBI/EMBL-EBI product and only NCBI's side was read; nothing here asserts anything about EMBL-EBI's terms for the same tables (@probe-names-the-table). download_mane_file translates httpx into ManeUnavailable, a subclass of ManeBuildError, so one except arm catches an unreachable FTP site as well as a malformed file while a caller who needs to tell them apart still can (@client-exception-contract). polars is a [dev], guarded import.

One thing this lane does not do: it writes no SourceRow. A builder is not a pass over a spec directory — nothing here consults MANE on behalf of a module, so there is no module to account for (@write-the-sourcerow, the converse half). The terms constant exists so that the first pass which does consult it has one to record.

Note for anyone reaching for MANE next: gene located on mitochondrial genome is a MANE exclusion class, so the mtDNA lane has no MANE frame by construction. And the summary is adjacent to identifiers.py's HGNC registry and to the Ensembl lane — worth reading before somebody builds a second gene cross-map beside it.

Open Targets (opentargets_build.py + opentargets.py) — RM333, RM239, RM236

One lane, one licence, four inputs (PROPOSAL_0_8_PT2 §4 question 2): the Platform release's l2g_prediction, credible_set, colocalisation and variant datasets, plus study to name a coloc row's gene and tissue. The probe behind every choice here is probes/LANES_0_8.md § 1, and the hand-run build is its § 5.

Build. just-dna-enricher opentargets build mirrors release 26.09 from https://ftp.ebi.ac.uk/pub/databases/opentargets/platform/26.09/output/<dataset>/ (eight parallel streams through net.stream_to_file, so retried on transport errors and translated to OpenTargetsUnavailable; a part already on disk with a parquet footer is not fetched again; Ctrl-C or a failed part stops the parts in flight at their next chunk and removes each one cut short, RM409) and flattens it into data/: variants.parquet (the allele table, alleles from columns, never from the variantId, plus each allele's rsIds verbatim as the list rs_ids), consequences-*.parquet (transcriptConsequences exploded), credible_sets.parquet and credible_set_members-*.parquet (GWAS sets only, locus exploded), l2g.parquet, and colocalisations-*.parquet (GWAS left × molecular-QTL right, with the right study's geneId and biosampleId joined on). --source <dir> builds from parts already on disk; --parts N reads the first N parts of each dataset, and release.json then records parts_read beside parts_total. The tables are written to data.building/ and swapped in only when the build finishes, so a failed build leaves the previous snapshot whole; a mirror the command downloaded (<out>/source/<release>/, ~22 GB) is deleted once the snapshot is written, and --source parts are left alone. The full build took 15 minutes here and the snapshot is 5.2 GB, 4.6 GB of it coloc; with rs_ids (2026-11-04) cache status measures 5,457.3 MB, inside the lane's declared 5,500. No row is dropped for its value: no L2G, posterior or h4 threshold is applied anywhere.

Fill. just-dna-enricher opentargets enrich <spec> reads the module's resolution.csv (overlaid, GRCh38 only: the release is GRCh38 and its ids carry no assembly) and joins each resolved allele to the release's allele table on (chrom, position, ref, alt). A row that states no ALT is matched another way, described under the table below. Then:

table what a row is the rule that keeps it honest
locus_to_gene.csv one (study locus, gene, module variant) row for every credible set the module's variant is a member of, led or not, with its L2G score several study loci per variant are all kept (@multiplicity-is-a-finding); two module variants in one set each get its rows; lead_variant_id names the set's lead, and the member's own weight is its posterior
colocalisations.csv one (left, right study locus, module variant) row for those same sets at the source's grain, because one gene colocalises in several tissues and QTL kinds: h4 and clpp both kept, never one picked
consequences.csv one (variant, transcript) pair the transcript set is Open Targets' (one canonical transcript per nearby gene); a variant resolving to two of the release's alleles is withheld, since its rows would share a key
gwas_effects.csv RM239's posterior_probability + fine_mapping_method on the rows the gwas pass wrote filled only when exactly one value answers for (variant, study_accession); empty, and counted, for a study with no credible set, a variant not in the 95% set, and a variant in several of the study's sets with different values

Each table's licence row (opentargets, its layer) lands inside that table's own commit, and only when the run contributed a row to it. A variant this run looked up has its rows replaced; one it could not look up (unresolved, off-build) keeps what it had (@rederive-never-shortens). With no snapshot the pass writes nothing and says nobody asked. It never fetches, so it has no --offline; --strict refuses a snapshot built from part of the release, and one that does not record its part totals (unknown completeness is not completeness). best_effort answers only from what it read whole (RM361): READS names the source datasets each table and the posterior come from, and opentargets.read_is_complete asks release.json whether each was read in full (False for a partial one, None when the record does not say). Anything but True answers no key for that table: rows the snapshot did yield are merged in, every existing row is kept, a key that did not join the allele table is not_read rather than not_in_release, and the posterior on gwas_effects.csv is left as it was (posterior.not_read). Two module keys on one release allele have that allele's L2G and coloc rows withheld and the keys named (shared_allele_keys): one allele stated under two keys is a module defect to name, not a fact to write twice. The member-wide grain (decided 2026-11-04) makes coloc large for a heavily fine-mapped variant: rs646776's 2,061 sets carry 309,634 coloc rows.

A row that states an rsID and no ALT (added 2026-11-04). An authored rsID-only row (apoe_epsilon's two, hfe_compound_het's two) used to be skipped without a word. The pass now reads every allele the release holds at the row's (chrom, start) and each allele's rs_ids. The row's rsID is its rsid, or the rsid_current dbSNP merged it into. A candidate is an allele at the position carrying that rsID, on any REF (one rsID can name a SNV and an indel at one position), or an allele on the row's own REF carrying no rsID at all; one allele the release lists under two ids is one candidate.

what the release holds answer where it is named
exactly one candidate, carrying the rsID, on the row's REF that allele's ALT is used for all four tables allele_supplied
several candidates, or the only one states no rsID or sits on another REF withheld, nothing written, earlier rows kept allele_unstated_withheld
no candidate asked and absent not_in_release
(the row's status is ambiguous: its rsID is the back-fill's pick among several) withheld allele_unstated_withheld
(the row has no rsID either) withheld, nothing asked: a position names no allele (@rsid-not-per-allele) allele_unstated_no_rsid
(the snapshot predates rs_ids) nobody asked; --strict refuses allele_unstated_unasked

A key with a stated ALT on another of its rows is still answered for that allele; only a key with nothing else to ask about is held back.

An allele with an empty rs_ids is a candidate, not a refusal. The release does not label every allele: at 6:26091590 G it states rs773443949 on G>T and nothing on G>A, and ClinVar names both under it, so supplying G>T would pick one of two (@multiplicity-is-a-finding). Over the 26.09 release, 101,945 of the 276,003 loci with several alleles at one (chrom, position, ref) have an rsID-less allele beside a labelled one; a no-ALT row at any of them is withheld, and the author states the ALT. An allele carrying only other rsIDs is a definite no. The supplied ALT is never written back to resolution.csv (@enrichment-is-validation), and no derived row says which ALT it used: the pass's key lists, its warning and the CLI's counts are the only record, so a reader of consequences.csv cannot tell a supplied allele from an authored one. A row-level column would say it; that is a P5 question left open. A withheld key's gwas_effects.csv rows keep their posterior and are counted allele_withheld.

The author's remedy for either side is overrides.csv, as for every derived table: a consequence row the fallback wrote is suppressed or corrected by (variant_key, transcript), and a withheld site gets the rows the author vouches for as inserts, one per field (a consequences.csv row by (variant_key, transcript), an L2G row by (study_locus_id, gene_id) with its variant_key among the inserted fields). The study-locus tables keep their two overlay limits (SCHEMAS.md states the overlay's keys): a suppress of an L2G or coloc pair removes every member's row of it, so one supplied key's row of a pair the module shares with another key cannot be withdrawn alone; and two members of one pair cannot both be inserted, since their insert rows share the overlay's own key.

What the lane does not do. It computes no causal or effector confidence label (RM323's cut): the three tables and the posterior are the inputs, a module states its rubric, and a hand-curated input is a correction in overrides.csv. It carries no symbol for an L2G gene (the source's table has none), and no credible-set level beside the posterior (the release publishes the 95% set only).

The computed LD pair (ld_pairs.py, online only) — RM331

just-dna-enricher ld-pairs <spec> fills ld_pairs.csv: for each studies.csv row naming a proxy_rsid, r² and |D′| between the proxy and the row's lead, in the superpopulation the row's ancestry names, from Ensembl REST's pairwise LD (/ld/human/pairwise/<proxy>/<lead>?population_name= 1000GENOMES:phase_3:<CODE>). The hand-run probe behind every rule below is probes/LANES_0_8.md § 2, and the run that recorded the test fixtures is its § 6.

the study row, or the answer the row written
one ancestry code, Ensembl answers a row resolved, with r2, d_prime (parsed from Ensembl's JSON strings) and answered_rsid
one code, Ensembl answers 200 [] not_reported, r² empty: below its floor or not in LD, never r² = 0
one code, Ensembl answers 400 … variation object rsid_unknown: an answer, so neither retried nor read as an outage
several codes, or none ancestry_mixed / ancestry_unstated, no population, nobody asked
a coordinate-authored lead with no rsID, here or back-filled in resolution.csv lead_has_no_rsid, nobody asked

The five superpopulation names are spelled here, because ?filter=LD does not list them although the endpoint answers for them; a 400 … population object is therefore a defect of this spelling and raises LdError, never a withhold. The proxy goes first in the URL, so variation1 is the proxy as Ensembl names it, which is how a merged id is recorded (rs1624712 answers as rs646776); the row stays keyed on what the author wrote. The release is read in band from /info/data on the run that asks, and is the row's dataset (ensembl_116) and the (ensembl, ld_pair) licence row's. That licence row commits inside the table's write, and only on a run where Ensembl answered: a run that wrote withholds alone consulted no source.

The client contract. 5xx (EBI serves its 500 as an HTML page), 429 and transport errors are retried, then raised as LdUnavailable: could-not-ask, cleared by a re-run. Everything a re-run would repeat is the parent LdError: a 404 or other 4xx (a moved path, a wrong species), a 400 that is not JSON or names no variation, and a 200 that is not a JSON array of pairs. The timeout is 90 seconds (the probe measured a 37-second answer) and the pacing gate one request per 100 ms, under Ensembl's 15 per second and its 55,000 per hour.

Who the lead is. The study row's own rsID; for a coordinate-authored lead, the rsID resolution.csv back-filled at the same place (chrom, start, ref), never looked up by key, since a StudyRow keys without alts and the VariantRow resolution answered for keys with them; a place two rows back-fill differently gives no rsID. A repeated ancestry code (EUR;EUR) is one superpopulation.

What a run commits. A pair already in the file is not asked again (@sidecar-authoritative); delete the file to re-ask after a new release. A recorded pair no study row names any more (its ancestry or proxy was edited) is dropped and counted (LdPairsResult.dropped), so a withheld row does not linger beside the answer that replaced it. --offline asks nothing and writes nothing: there is no LD snapshot, and a row for an unasked pair would claim an answer. A pair whose answer this pass cannot use gets no row and costs that pair alone: the queue goes on, the run's other rows are committed, and the pass then raises LdError naming it. When Ensembl stays down past the retries, the queue stops, the pairs answered this run are committed, nothing is written for the rest, and the pass raises LdUnavailable naming how many were not asked; a re-run asks only those. There is no --strict: every arm above is a reproducible answer or a stated reason, so a strict mode would refuse nothing.

Gene–disease validity (gene_validity.py, online only) — RM24

enrich_gene_validity(spec_dir, *, source, mode, offline, write, export_text, url) takes the gene column of variants.csv and writes gene_validity.csv: one row per (gene, disease, mode of inheritance, submitter). just-dna-enricher gene-validity spec/ [--source clingen|gencc].

The question gene_metrics.csv cannot answer. Constraint says how intolerant of variation a gene looks in a population sample; ClinGen's dosage rating says whether losing a copy causes disease; neither says whether variation in this gene causes this disease, which is the claim a clinical module most often rests on without recording anywhere.

Two submitters, and they are different kinds of thing. ClinGen publishes expert-panel curations — one assertion per (gene, disease, MOI), each from a named Gene Curation Expert Panel working to a numbered SOP. GenCC publishes an aggregate of nineteen submitters, ClinGen among them, plus Orphanet, PanelApp and several laboratories; the same gene–disease pair routinely carries several submitters at different strengths, and that disagreement is the data.

Three things established by reading the real files (2026-08-13 downloads: ClinGen 3,659 rows, GenCC 30,410), each of which decided part of the shape:

  • Mode of inheritance is part of the key. 59 (gene, disease) pairs in ClinGen carry two rows differing only there. (gene, disease, moi) has zero collisions; (gene, disease) silently keeps one curation and drops the other.
  • submitter is in the key too, or GenCC collapses to one arbitrary opinion per pair — the bare-triple mistake the ClinPGx cross-check paid for once already.
  • The two vocabularies disagree in spelling and agree in meaning, so both are mapped onto vocab.VALID_GENE_VALIDITY / VALID_INHERITANCE_MODE at this boundary: Disputed and Disputed Evidence are one member, AD and Autosomal dominant are one member. A consumer filtering on one spelling would silently miss the other's rows. The submitter's wording survives in classification_raw, so the mapping stays auditable, and a wording this release does not model is left unset with one aggregated warning naming the distinct values — never one line per row.

A gene the submitter has not curated gets no row, and is reported in missing. That is clingen.py's rule and its reason: a curating body's silence means nobody has assessed the gene yet, which is not a fact about the gene, so a not_found row would state one. (The ClinVar pass below goes the other way, because ClinVar covers the genome — "asked and absent" really is a fact there.)

--offline is a no-op with a warning (skipped_offline), never a failure: neither submitter ships a snapshot, and both files are small enough to fetch whole. An injected export_text= still wins, because handing over bytes you already hold is not egress.

Both sources are CC0, so a module using this table stays sellable — GENCC_TERMS joins CLINGEN_TERMS in licensing.TERMS_BY_SOURCE, and the pass records its SourceRow at the new gene_validity layer. GenCC's attribution names the contributing sources as well as GenCC, because crediting only the aggregator credits nobody who did the work.

HPO ships no route, and both reasons came from probing rather than from taste. Its release declares terms:license https://hpo.jax.org/app/license; that URL answers HTTP 404 with a JavaScript shell, and OBO Foundry records the licence as a bare label hpo with no SPDX id — so the terms cannot be established from any machine-readable source, and an unestablished permission is not a permission (the PharmVar rule). Recording it with commercial_use=None would flip every carrying module's manifest verdict to undetermined, which contradicts the sellability this table was designed to keep. Separately, the file the item named — genes_to_phenotype.txt — is gene × HP feature × frequency, a different grain from this table entirely; genes_to_disease.txt fits structurally, but its association_type (MENDELIAN / POLYGENIC / UNKNOWN, 8,288 of 15,944 rows UNKNOWN) is a mechanism class, not an evidence grade, and putting it in classification would overload the axis (P5). The row shape holds an HPO row perfectly well; what is missing is a link this tier may take the data over.

Clinical assertions (assertions.py, offline capable) — RM25

enrich_clinical_assertions(spec_dir, *, mode, offline, clinvar_cache, download, write) reads resolution.csv and the ClinVar snapshot and writes clinical_assertions.csv: one row per (allele, archive record) carrying the clinical call, ClinVar's own review wording, the 0-to-4 star rating and the VariationID. just-dna-enricher assertions spec/.

The number this workspace was already computing and discarding. clinical.ClinSigFinding.confidence rendered the star rating into a warning string and kept nothing; clinvar_draft.draft_gene_panel used it as a filter (default 2 — multiple submitters, no conflicts) and kept nothing. So a compiled module flattened a one-star single submission and a practice guideline to the same clin_sig, and every consumer that wanted the difference re-derived it. A number recomputed downstream is a place to drift (RM40/RM41, a fourth time).

It records; it does not adjudicate. clinical.verify_clin_sig — comparing the author's clin_sig against ClinVar's — is untouched, and still warns in both modes on purpose, because failing would make the format arbitrate a clinical dispute. Escalating that check stays parked deliberately, and nothing in this pass moves it.

Four mechanics worth keeping straight:

  • Structured like enrich_frequencies, because the input is the same. It consumes resolution.csv rather than variants.csv: a clinical record is per allele at a coordinate, and the resolution table is where an rsID has already become chrom-pos-ref-alt. That also sidesteps the multi-allelic-rsID problem — one rsID at one locus legitimately carries a pathogenic, a benign and an uncertain allele (rs33922842 in HBB), so looking clinical significance up by rsID would manufacture disagreements out of ClinVar agreeing with itself.
  • Snapshot-first and fully offline-capable, unlike the frequency pass: ClinVar ships as a snapshot, so with one provisioned this pass never touches the network. With none found and not offline it calls download.ensure_clinvar_snapshot — the enrich() shape, --offline as the only switch — and with none reachable at all it is a no-op with a warning (skipped_no_snapshot), leaving any existing table as the pin.
  • dataset is the snapshot's own release, read from release.json (the RM38 rule): a consumer must be able to tell a pinned file from whatever happened to be current, and a re-review is only visible against a stated release. A snapshot that cannot state one gets clinvar_unknown rather than a fabricated date.
  • A coordinate on another build is never queried. ClinVar's lookup key is (chrom, start, ref, alt) and carries no assembly, so a GRCh37 coordinate is a well-formed query returning a different variant's clinical call under this module's key — the same failure the frequency pass had against gnomAD. Such rows are reported in off_build, kept apart from missing because nobody asked about them, and are outside the strict gate: a coordinate on another assembly is reproducibly out of this snapshot's reach, so refusing would make a GRCh37 module uncompilable for a reason no authored edit could fix.

clinvar_build.review_stars is the one place the CLNREVSTAT-to-rating convention lives, and it became public and tri-state in 0.6: None for a record that states no review status and for a wording this release does not model, 0 for ClinVar's own "no assertion criteria provided". It used to answer 0 to all three, which files an unread record under the weakest rating available — a claim nobody made. That is also why ClinicalAssertionRow stores the rating as a column instead of deriving it from the prose beside it: the derivation is a ClinVar convention, and Principle 2 keeps source conventions out of the schema tier entirely.

The literature pack (literature.py, online only)

Pass 4: a module's citations in, literature.csv out. Three questions of decreasing coverage — does the citation exist (PubMed esummary), do the identifiers agree (DOI/PMCID arrive in the same response), and does the quoted passage appear in the article (Europe PMC fulltext for its open-access subset, then PMC's BioC text-mining service for any record with a PMCID, then the abstract) — plus the article's own licence, which arrives in the same Europe PMC response.

studies.csv is one citation site of several, and this pass reads every one (RM47, RM132). A pmid on a binning row grounds the threshold it sits on; one on a pharm_variants.csv row grounds that row's own drug and genotype claim, which studies.csv structurally cannot, since a study row keys on (variant_key, pmid) and attaches to the whole variant. A module whose only citations come from those tables is enriched exactly like one with a studies.csv; a module with none at all is refused, since the relaxation is about where a citation may live and not about whether one is needed.

Those pointers are read through just_dna_compiler.load_citing_rows / table_citations — public for the RM41 reason, because the alternatives were importing a private symbol or hand-keeping a second list of the kinds here, and that list goes stale the next time a model declares a pmid. The compiler's set is derived (_CITING_TABLE_KINDS is every table kind whose model declares the column), so this tier gains a new citation site by doing nothing, and a test walks this module's own source to assert no such roster is kept here. load_binning_rows/binning_citations still exist and still mean the binning kinds only — a caller asking for those is asking about thresholds.

A citation reaching the pass only from such a row contributes no quote and no authored DOI (neither row has those columns — provenance_quote deliberately did not follow the pmid to either site), so it reads as nothing to check rather than as an unretrievable fulltext.

The article's licence is recorded per article, and there is no pubmed row in the licence table (RM46). The pass writes source="pubmed" into every row it produces and TERMS_BY_SOURCE has no entry for it — deliberately, and permanently: a literature source's terms are per article, not per source. PubMed's metadata is one thing; the article belongs to its publisher, and Europe PMC's open subset spans CC-BY, CC-BY-NC and bronze, so one pubmed row would be right for a module citing only ids and a false all-clear for one carrying a provenance_quote lifted from a CC-BY-NC article — wrong in the dangerous direction, since that quote is publisher text in the module's own annotation layer. Four mechanics:

  • license is stored verbatim as Europe PMC spells it (cc by, cc by-nc, cc by-nc-nd — probed over 100 records on 2026-08-13), and licensing.article_terms maps it to the three rights at read time, so a mapping correction reaches rows already written. Same rule as cpic_build.
  • The licence is independent of is_open_access and is not derived from it: PMID 28546431 comes back isOpenAccess: N with license: cc by, because the flag describes Europe PMC's OA subset while the licence describes the article.
  • Three orthogonal axes, and None is never False. CC BY-NC forbids sale and expressly allows sharing, which is why redistribution is its own column; a licence this tier has not read leaves all three null rather than guessing in either direction.
  • Quoting a non-commercial article warns and never gates. The compiler reports it (reading the recorded fact, so it still owns no source convention), in both modes, aggregated by licence. It is the same call as the ClinVar clin_sig cross-check: refusing would make the format arbitrate a copyright question. And note the merge rule's consequence — rows written before 0.6 carry no license, and a re-run will not back-fill them, because merge-not-clobber cannot tell an absent value from a curator's deliberate blank. Delete literature.csv to re-derive.

Coverage is partial by nature, and reporting it as a fraction is part of the check. A pass that said "0 quotes found" for an article it could not read would be describing its own reach as a defect in the module, so quotes_found is null (not zero) when no fulltext could be read. The denominator counts only citations that carry an authored quote: one that asks no question was not skipped for lack of an answer. (That distinction is not hypothetical — it was a real bug, found by running the pass against reference_examples/pathogenic_clinvar/, whose single citation is open access and quote-free, and which the first wording therefore described as unretrievable.)

The fulltext has two rungs, and the second reaches the records the first calls closed (RM257, S110). An NIH author manuscript is isOpenAccess: N in Europe PMC, so the pass never asked Europe PMC for its text, and asking anyway answered HTTP 500 for PMC6463297 on 2026-09-24. PMC's BioC service (PmcBiocClient) serves the same manuscript for text mining, body tables included as tab-separated passages, and the pass asks it for any citation with a PMCID that Europe PMC did not serve. The reference list is dropped, since a quote found in a cited title is not in this paper. It is an NCBI host, so it runs on the E-utilities PacingGate. An article outside the service's set answers 200 with [Error] : No result can be found., which counts as absent, the same as a 404. Retrieval here is not a licence: the service's own note says the file is "available for text mining", which names no terms, so is_open_access and license stay what Europe PMC said. A BioC hit is quote_source=fulltext, because that column records how far the search reached and not which host served it.

An outage is asked again; an absent copy is not (RM258, 0.8). Both clients now have a three-way fetch_fulltext: text, None for a service that holds no copy (a 404, or BioC's plain-text "no result" 200 and only that body), and FulltextUnavailable for a request that never got an answer (a 5xx, a 429, any other 4xx, a transport failure, or a 200 that is not the text: a page that is not JSON or not a JATS <article>), after the retries, which cover a 5xx and a 429. A 410 raises FulltextRetired (RM361). The pass records how the question ended in literature.csv's fulltext_status (retrieved | absent | unreachable | retired), and on every later run it asks that one question again for a pinned row that is unreachable or retired, or null and not already matched against a fulltext (a row written before the column existed, asked once). Nothing else on the row is re-fetched: existence, identifiers and the licence were answered and stay merged. An answer rewrites quote_source, quotes_found and the status; a second failure leaves the row as it was. The older fulltext() keeps its one-None contract for callers that rely on it, such as the authoring plugin's fetch_fulltext tool.

A pmid=NONE citation is checked by its DOI (RM349, 0.8). The DOI is the only identifier such a row carries, so it is what both questions are put about, once per distinct DOI however many rows cite it (spec.doi_key: case and a doi.org prefix ignored):

  • Existence, and which work it is. CrossrefClient.work, so the answer names the title Crossref holds (@existence-not-identity). No citing table carries a stated title to compare it with, so it is recorded (DoiCitation.title) and named in the citation_existence detail and on the CLI, never compared. A DOI no registration agency knows joins doi_missing, which strict refuses on.
  • Crossref's /works 404 is not the registries' 404. It answers 404 for every DOI another agency minted, and a NONE row is exactly where a dataset is cited: 10.5281/zenodo.3360392 (DataCite) is real and 404s there. After a 404 the client asks Crossref's /works/{doi}/agency, which answers for every agency (probed 2026-11-04: datacite for that DOI, 404 for a fabricated one and for an unregistered Zenodo number). Another agency is exists=True with CrossrefWork.agency set and no title in hand, said so in the detail; any failed or contradicting answer withholds (None). This holds for an authored DOI on a PubMed row and for hint citation --doi too, which used to call such a DOI absent.
  • The text. EuropePmcClient.record_for_doi searches DOI:"…" over every Europe PMC source and takes only a record whose own doi matches. A record with a PMC id goes the two rungs above (Europe PMC, then BioC). A preprint (source=PPR) is fetched from Europe PMC's preprint repository (fetch_preprint_fulltext: the record's file list, then the JATS file it names), and only when the search record lists a full text, because the repository's answer for a preprint with none is a 500, the same status as an outage. The documented /{id}/fullTextXML is not the route: it answered 500 for every PPR id probed on 2026-11-04, with and without a full text. The abstract in the same search response is the fallback, with the asymmetry above. fulltext_status has its LiteratureRow meaning: retrieved, absent (no record, or a record with no text) or unreachable (a request never answered, which a re-run clears and the record names apart).
  • Nothing about it is written to literature.csv. That table is keyed on a PubMed id and feeds a published parquet the compiler reads by PMID (split_cited_literature, the quote-counter and licence checks, manifest.literature), so a NONE row there would be a widening of a published column's grammar. The answers live in LiteratureResult.doi_citations (one DoiCitation per DOI) and in verification.json, which is the durable half; with no pin, every online run asks again. Offline nothing is asked and both records say offline.

What it does not do: bioRxiv's own API publishes a jatsxml link for every bioRxiv and medRxiv preprint, a wider route than Europe PMC's subset, and is not used (it is a new source, not a client this tier has). The non-commercial quoting notice and the title-as-quote check (S54) read literature.csv rows and so do not reach a preprint quote.

A quote is an attestation, so no tool may write one — and retrieving the fulltext changes what the check proves. provenance_quote/provenance_regex mean a curator read this passage in this paper, which is why both are registered in hints.REDUNDANCY_BEARING and in hints.ATTESTATION_BEARING: the second names the sharper refusal, because filling doi from the registry that checks it merely spends a comparison, while extracting a passage from a fulltext a tool has just fetched states something false. The consequence for the check itself is worth being blunt about: quotes_found is independent evidence only while the author and this pass read the article separately. Once a machine has retrieved the text, a hit shows the quote pairs with the PMID — still worth having, since it catches a passage filed against the wrong paper — but no longer that the claim is in the article, because nothing establishes a human ever looked. (Reported as S11; the map had simply never learned about the comparison this pass performs.)

Existence and retrievability are different questions, and only the second is affected by a paywall. PubMed indexes paywalled work like any other, so exists is answered for it — PMID 12345678 is not in PMC at all and esummary confirms it perfectly well. What a paywall blocks is the fulltext, and two things close most of that gap:

  • The abstract. Europe PMC serves it for non-open-access records, in the same search response the pass already makes — four of five probed non-OA papers carried one (the exception was a 1994 non-research document). A quote found in the abstract is as conclusive as one found in the body; a quote missing from a 200-word abstract says nothing, so the row records quote_source and the miss still counts as unchecked. That asymmetry is the whole reason the column exists.
  • Crossref, for the citations PubMed structurally cannot cover: preprints, books, theses, datasets have DOIs and no PMID. A probed bioRxiv DOI returns type: posted-content; a fabricated one 404s. It checks the authored DOI rather than the derived one — the registry's own exists by construction — and a transport failure records None, never False. This is also what makes the 1.0 doi-first flip low-risk: when pmid becomes optional, existence checking already works without it.

Google Scholar is not an option and it is worth saying so rather than leaving it as an open idea: it publishes no API, and automated querying violates its terms and is blocked in practice.

Both corrections below came from probing:

  • The PMC ID converter is not used by this pass, and the reason is directional. esummary already returns doi and pmc in articleids, and Europe PMC's search returns them too, so calling the converter for PMID → PMCID is a third request for data already in hand. Worse, it answers a different question: for PMID 12345678 — a real, indexed PubMed record — it replies status: error, "Identifier not found in PMC". Wired in as an existence check it would report every paywalled article as a broken citation. None of that says anything about PMCID → PMID, which is the direction the converter exists for and the one a curator has no other route to — see the PMC id section below.
  • Europe PMC is not an existence oracle. Asked about three ids where one does not exist, it returns two results and silently omits the third — no error, no marker. PubMed decides existence; Europe PMC decides retrievability.

PubMed and PubMed Central ids are one letter apart, and the outcome used to turn on a space (RM50). StudyRow.pmid is free-form and validated through spec.extract_pmids, whose pattern is \b(\d{1,8})\b — so PMC3110566 came back empty (no word boundary between C and a digit) while PMC 3110566 came back ['3110566'], and 3110566 is a real PMID for an unrelated article, because PubMed ids are densely allocated. One spelling of one mistake was refused with a message that never said "PMCID"; the other was accepted as a confident citation of the wrong paper. Three things ship for it, all of them diagnosis and none of them repair:

  • The schema refuses a digit run whose immediate context spells PMC in any spacing (spec.PMCID_PATTERN, spec.extract_pmcids) and the message names the id it saw rather than the one it wanted. Narrow by construction: a cell carrying both (21551363; PMC3110566) still yields the real PMID and is accepted, so only a cell whose sole numeric content is a PMC id refuses — and that cell previously resolved to another article entirely.
  • literature._pmcid_conflicts catches what the schema cannot see: a cell like 21551363 (PMC3110567) carries a real PubMed id, so nothing refuses it, while the two halves name different articles. It costs no request — the PMC id is already in the esummary articleids block — and it is the _doi_conflicts shape, including strict refusing.
  • lookup_citation(doi=…) / hint citation --doi names the work as well as confirming it exists (RM262, S113): CrossrefClient.work parses the /works record the existence request already returned into the same four fields the PMID branch fills from esummary, with the matching DOI … names: '…' finding. With a PMID given too, PubMed's record fills the fields and the DOI's title is still reported. A 200 that is not a Crossref record withholds.
  • A DOI with no PMID gets one through Europe PMC (RM263, 0.8): EuropePmcClient.pmids_for_doi searches DOI:"…" AND SRC:MED, which covers all of PubMed, where NCBI's converter answers only for articles in PMC (10.1038/ng826 is 11788828, Enattah 2002, not in PMC). One PMID comes back as an advisory (source="europepmc"), and PubMed is then asked which paper it is, so the lookup ends with Crossref's title and PubMed's as two info findings that nothing compares: a DOI names one paper, yet PubMed bracket-translates a non-English title (10.1055/s-0041-111184), so a comparison would warn on every such paper. Several PMIDs withhold the advisory and are all named in a warning; none is Europe PMC's answer, said as that and never as PubMed's. An authored pmid skips the route.
  • lookup_citation(pmcid=…) / hint citation --pmcid resolves the other direction through NCBI's converter and then asks PubMed which paper that is, because a converter that hands back a number and stops is the same existence-is-not-identity failure one registry over. The resolved id comes back as an advisory (applied=False, refusal="redundancy_bearing"): filling pmid from NCBI would make LiteratureRow.exists compare NCBI with itself, which is the argument already made for doi. Four outcomes, spelled four ways — resolved, in PMC with no PubMed id, not in PMC, and never answered — because collapsing the last two would render a failed request as a definite negative (S20). The address is pmc.ncbi.nlm.nih.gov/tools/idconv/api/v1/articles/; the long-published www.ncbi.nlm.nih.gov/pmc/utils/idconv/v1.0/ 301-redirects to it.

On evaluating provenance_regex here. The charter requires a linear-time/ReDoS-safe engine, written when the match was specified as consumer-side. Here the pattern comes from the module being enriched and the document from a public archive, on the author's own machine, so the risk is a curator writing a slow pattern by accident. That is worth a bound rather than a compiled dependency — but the bound must be a child process, not a thread. The thread version looks correct and is not: re cannot be interrupted, threads cannot be killed, and the interpreter joins pool threads at exit, so a runaway pattern returns on schedule and then hangs the process on the way out. A timeout is recorded as not checked, never as not-found.

Existence is not identity, so a citation lookup says which paper it found (S12, 0.5.4). PMIDs are densely allocated, so a recalled or invented eight-digit number is usually a real record for a different article — which means pmid_exists=True could never catch a fabricated citation, and the surrounding docs had been treating existence as the guard until a consumer's authoring skill had to retract a rule this surface could not enforce. literature.bibliographic(summary) pulls title/journal/first_author/year out of the same esummary response that answers existence — no extra request — and lookup.CitationHint carries them, with an info finding naming the paper and hint citation --json (which hint variant had and this did not). It is public rather than private _identifiers-style for the RM41 reason: two tiers read it, and the alternative is a consumer re-parsing a payload we already hold. Every value is None when the field is absent rather than an empty string, and year is taken only from a leading four digits of the free-form pubdate (2017 Nov-Dec), so nothing is invented. No title column on LiteratureRow: that table records what was checked, not bibliography. Generalize it — when a check answers yes/no about an identifier, ask whether "yes" could be true of the wrong thing.

Identifier currency (identifiers.py, online)

The generalization of COMPILER.md's "is the source stale?" blind spot from datasets to identifiers. A dbSNP merge, an EFO retirement, an HGNC rename and a PGS accession that was never issued all leave a module perfectly well-formed and quietly out of date.

  • rsIDs run inside enrich() (--verify-rsids), because the verdict lands on resolution.csv's rsid_current/rsid_status columns. Three states from esummary db=snp: live (snp_id == requested, merged_sort='0'), merged (snp_id != requested, merged_sort='1'), absent ({'uid': …, 'error': 'cannot get document summary'}). Since 0.8 the pass also asks about every rsID a meta.csv member names (variant_key= or rsid=, subjects.meta_member_rsids): one variant written under two keys, IL6R's rs8192284 merged into rs2228145, hides a disagreement as a difference, and NCBI is where it shows. Those ids are asked and reported (stale_rsids, and the usual ladder), never resolved: no resolution.csv row is written for an id the module does not carry.
  • Traits and gene symbols are module-level with no sidecar column to fill, so they get their own command, just-dna-enricher check-identifiers. HGNC uses the exact fetch/symbol and fetch/prev_symbol endpoints, never search/ — search/BRCA1 is fuzzy and returns 19 hits including ABRAXAS1.
  • Gene ↔ locus agreement (0.5.4) rides on the same command and is the first check here about a relationship between two identifiers rather than the currency of one. See below.
  • PGS accessions (0.7, RM163) are the fourth registry and ride on the same command, under --pgs/--no-pgs. They put two questions — does the accession still name a score, and do the two authored cells beside it still match the record — and the command writes the Catalog's per-score terms into licensing.csv while it is there. See below.

NCBI is the oracle, not Ensembl. Ensembl REST resolves some merges (rs77121243 → rs334) and returns HTTP 400 on others (rs3216883, which dbSNP reports as merged into rs3051860), so Ensembl alone would misclassify a merged rsID as unresolvable.

absent conflates two opposite meanings and no live endpoint separates them. A withdrawn rsID (rs11273140, retracted after a clustering error) returns a response byte-identical to a never assigned one (rs2000000000). For an author these mean opposite things — fix the typo, versus the variant itself was retracted and the annotation resting on it may be worthless — so the message names both readings and asserts neither. Guessing "typo" would send an author to fix the wrong thing.

withdrawn is nevertheless a real vocabulary member, and refuses in both modes. Nothing automated emits it today, which is a limitation of the API rather than of the model, and the member is kept for two reasons: a curator who has established a retraction can record it in resolution.csv and have the tooling honour it, and a future source that can tell the two apart starts producing it without a vocabulary change — which Principle 3 would otherwise make a one-way door. Its severity is deliberately not absent's: a merged or absent rsID leaves the annotation intact (dated, or unserved), while a retracted variant may leave it describing nothing, so withdrawn is the one resolution finding fatal in best_effort too.

Report, never repair, and here that is load-bearing. weights.parquet carries both variant_key and rsid; for an rsid-authored row they are the same label. Writing the merged-into id back would be an identity migration performed by a network lookup — reverse would emit the new rsID, the next compile would key on it, and variant_key would change with no authored edit anywhere. Severity is the usual ladder (warn / fail in strict), and strict failing is the nudge toward the drift-proof key: author the coordinate, and the VRS allele id cannot drift at all.

Two true halves, one false row — gene ↔ locus (S24, 0.5.4)

variants.csv carries a gene column and nothing compared it to anything. The gene check above asks HGNC whether a symbol is approved, which is a different question — FTO is approved whatever variant sits beside it — so a row pairing a real gene with a variant on another chromosome passed everything, because both halves were individually true and only the relationship was false. Four of one reporter's seven rows were exactly that: real symbols beside invented rs numbers, which resolve anyway because dbSNP is dense enough that almost any seven-digit number hits something. Machine-written sources are a real authoring input now, and this is the shape they fail in — which is also why it belongs beside the currency checks rather than among them: staleness is the world moving, and this is a claim that was never true.

check_identifiers reports GeneLocusConflict per row and repairs nothing — which of the two halves is wrong is not something this tier can know, and since RM426 (S141) the message says so. Its fact sentence is unchanged (<variant> is on chromosome <n>, but HGNC puts <gene> on chromosome <m> — the row pairs a gene with a variant that is not in it.); the closing one, which used to guess "a real symbol beside an invented rsID", now reads Either half may be the wrong one: a real symbol beside an invented variant, or a real variant paired with the wrong gene, and a machine-written summary produces both. Look up which gene <variant> sits in, then check the row against the source you took it from. S141's row was the second half: MTHFR beside rs1801394, which sits in MTRR. Four design points:

  • Chromosome granularity only, and the stronger version is refused in the code using the reporter's own argument. rs1421085 sits in an FTO intron and acts on IRX3/IRX5 megabases away, so a row may legitimately name any of the three; an interval check would fire on correct rows until someone switched it off. A test pins that the FTO row stays silent with the variant nowhere near the gene body. Chromosome disagreement has almost no legitimate cause, costs one comparison, and catches the whole fabrication class.
  • The join is against HGNC's cytoband (16q12.2 → 16, mitochondria → MT), and anything unparsed yields None rather than a guess — a guess here becomes a false accusation about a row. Three-valued as everywhere else: an unknown symbol, an unparsed band and a row with no known chromosome all withhold.
  • Nothing is fetched for the coordinate. For an rsID-only row the chromosome comes from an injected resolution.csv beside the spec — the table the compiler already consumes — because a currency check should not depend on a resolver.
  • A pseudoautosomal gene is exempt. XG straddles the PAR1 boundary, so X/Y there is a spelling, not a contradiction (RM32).

IdentifierReport.gene_loci_not_checked carries the reason when the comparison could not run, for the reason EnrichmentResult.clin_sig_not_checked exists — an empty conflict list otherwise says both "compared everything, nothing disagreed" and "never compared". The CLI prints it.

The PGS Catalog — the fourth registry, and a licence that is per score (0.7, RM163)

pgs.csv is a manifest of PGS Catalog accessions and PgsRow is keyed on pgs_id, so until 0.7 the one authored identifier the format keys a whole table on was the one nothing ever asked its registry about. check-identifiers now asks, under --pgs/--no-pgs, and it puts two questions rather than one.

pgs_accession_currency — does the accession still name a score? Three outcomes, and the shape of the source decides how they are reached.

  • The verdict is read off the body, never off the status. GET /rest/score/PGS000001 returns the record; GET /rest/score/PGS999999, a never-assigned accession, returns HTTP 200 with {}; and GET /rest/score/PGSXXXX, which is not an accession at all, returns HTTP 200 with {} too. So the status code carries no existence information, and a 404 from this service means the request went somewhere unexpected rather than that the score is absent. That is the opposite of OntologyClient's rule, where OLS4's and HGNC's 404 is the answer, and the per-client fact is why test_client_exception_contract.py keeps a 404 table rather than a tier-wide rule.
  • The absence is weighted by the measured base rate, and it is the other way round from dbSNP's. The accession range is sparsely assigned — roughly a third of it, measured over a random sample — so an unrecognised accession is overwhelmingly one that was never issued. @rsid-absent-two-readings gives typo and withdrawal equal weight because dbSNP's id space is dense and merges are frequent; here the message names the typo reading first and withdrawal as the rarer one. The Catalog publishes no supersession field, which is stated as a limit of the source rather than resolved by guessing.
  • A malformed accession is settled before any request. PgsRow's own PGS<digits> grammar answers it, and asking would have established nothing — the Catalog's reply is byte-identical to a never-assigned one. In practice the schema refuses such a row at load, so the state is reached only by a caller holding an id from somewhere other than a parsed table; the arm exists so the message can say spelling rather than offering a withdrawal that almost certainly did not happen.
  • A recognised accession says what it found — name, release date, variant count, traits (@existence-not-identity). A PGS accession is exactly the shape of identifier where a wrong-but-real id resolves cleanly to somebody else's score.

pgs_metadata_agreement — do the two authored cells beside it still match? training_ancestry against the score's ancestry_distribution, training_cohort against its samples_training. Reports, never repairs. Its own vocabulary member rather than a second finding under the line above, because the subject is a cell and not an id: a module with three accessions can put six cells, settle two and withhold four, and none of those numbers fits a denominator counted in accessions.

  • Only the dev and eval stages are compared. The Catalog publishes three — gwas is the ancestry of the discovery study the effect sizes came from, dev the samples the score was built on, eval the ones it was evaluated in. training_ancestry exists so a consumer can refuse an out-of-ancestry application, which is a question about where the score was built and tested; folding gwas in would widen the published set until nothing could disagree.
  • The two vocabularies do not meet, and the gap is withheld rather than reported. VALID_TRAINING_ANCESTRY is 1000G superpopulations plus multi; the Catalog's categories add NR (not reported), ASN, GME and OTH, which no member covers. A score whose distribution names one of those has an ancestry this tier cannot spell, so an authored code the published set does not carry might be exactly that one — and a difference nobody can stand behind is not a finding. PGS_ANCESTRY_CATEGORIES is total over the categories the service serves, with the unmappable ones written out rather than left to a .get default, and a category the Catalog invents later is logged.
  • MAE and MAO are bags, and the two directions are not symmetric. Multi-ancestry excluding European and multi-ancestry including European each stand for two or more superpopulations the Catalog did not break down, so both map to multi for a positive answer and both block a negative: an authored code the rest of the distribution does not name may be inside the bag. Without that, MAO reported a false drift against an authored EUR — a population it includes by definition. _UNENUMERABLE_CATEGORIES is that half, kept apart from the None entries because the two are different facts: a None category has no member at all, these have one and still cannot be exhausted.
  • The comparison is one-directional. An authored cell narrower than the published set is not flagged: a score the Catalog evaluated in eight ancestries at a percent each is not thereby validated in all eight, and claiming fewer is the conservative direction.
  • training_cohort is free text, so the disagreement bar is high. The cell agrees when any non-structural word of it appears anywhere in the record's own account of its training samples — cohort names, and the per-sample ancestry prose beside them, because FIN and Ashkenazi are the field's own examples and are ancestries rather than cohorts. Matching is substring rather than whole word, so FIN is vouched for by Finnish. Words that describe what a cohort is (cohort, study, project) are dropped: they match almost every record, and letting one vouch would make the check unable to fail while looking like one that passes.
  • match_rate_floor and research_tier are deliberately not checked, and the reason is written into the skip's own detail so nobody adds them on the symmetry. match_rate_floor is described in its own Field as an author-set floor and research_tier is a two-member curator judgement; the Catalog publishes neither, so a check over them could not fail (@tautology-zero).
  • The overlay has nothing to say here, and that is structural. overrides.csv corrects derived rows and OVERRIDABLE_TABLES is built from the derived registry; pgs.csv is authored, so a disagreeing cell is edited in place rather than answered. There is no suppression path to wire and none is missing.
  • One accession on two rows is one claim only while the two rows agree. PgsRow is keyed (pgs_id, trait_efo_id), so a pleiotropic score is two rows; the cell key carries the authored value, which collapses identical claims and keeps differing ones apart. Keying on the pair alone dropped whichever row came second, so whether a stale value was caught depended on row order.
  • A drift never fails --strict. An accession the Catalog does not hold is a broken identifier and does, exactly like a retired HGNC symbol; a cell two authorities disagree about is not, and the finding's own message tells the author to leave it if the curation is deliberate. Failing a build on that would be a gate with no way to clear it — the shape @clinsig-never-escalates keeps out. The green line is withheld all the same, so a clean exit never reads as agreement.
  • A Catalog outage reaches only these two records. The leg records unreachable on the report rather than raising: check_identifiers puts questions to four registries and the command writes one record per check, so an exception here would stamp unreachable against trait_currency and gene_symbol_currency for registries that answered. The ontology legs run first for the mirror reason — an OLS4 failure aborts before this leg spends requests and writes a licence table the run is about to discard. Called with variants= rather than a spec directory the record is unsupported, because no row model a caller can hold carries a pgs_id at all.

Currency is read, not built. /rest/info publishes latest_release — date, score count, publication and trait counts — beside the REST API's own version, so pgs_catalog joins currency.PROBE_SOURCES as the second shipped probe with no builder and no snapshot behind it. pgs.dataset_label spells the release for the probe and for the SourceRow.dataset the check stamps, which is clinvar_dataset_label's rule: two spellings of one release make the comparison quietly never match, and never-matches renders as unreadable rather than as a bug. A release record that cannot be read leaves pgs_release at None and the accession verdicts standing — it is a fact about the Catalog, not about any accession. The floor row's dataset is paired with withdraw_stale_dataset, exactly as the two drafting providers pair theirs: merge_sources_csv never clobbers, so without it a row written under an older release would hold that label forever and verify_datasets would report it behind on every run with the pass that owns it unable to refresh it. It only ever blanks — one column cannot name two releases.

The terms are per score, and this is a correctness requirement rather than thoroughness. license is a field on the score record. Measured over the first 250 of the corpus: most carry the Catalog's generic sentence deferring to "any licensing restrictions set by the authors", a handful are "freely available to the academic community for research use" — academic-use-only, the class this workspace already classifies as barring redistribution outright — and a couple are CC0. So PGS_TERMS is written as a floor in GWAS_CATALOG_TERMS' shape (EBI's terms-of-use URL, every gating axis None), and the pass writes one further SourceRow per score, source=pgs_catalog:PGS000013, with that score's own licence string in notice and hashed into license_sha256; license itself carries a short name (CC0-1.0, Academic research use only, or null for the generic sentence) because the compiler compares that column against module_spec.yaml's declared licence by equality, and a 250-character sentence there would put a paragraph into manifest.sources.licenses and into a warning the author cannot act on. All rows sit at the annotation layer, which is where taints_commercial_use reads them, and most-restrictive-wins does the rest. Measured end to end: a module naming one academic-use-only score fails the compile licence gate by name until its author records a non-commercial declaration, where a single generic row would only have warned. A single constant covering the source would be a false claim for a measured minority in the permissive direction, which is the direction that matters — the same shape as @per-article-terms and as the SpliceAI note inside GNOMAD_TERMS. A licence string the classifier has not read is unknown on all three axes and logged; unknown is not permission.

No --use flag and no check_declared_use call: every gating axis on the floor is None, so the gate would skip every run unconditionally, and reading a public endpoint to check an identifier is a read rather than an acquisition anyone has gated (@acquisition-gate-is-not-a-read-gate).

ACMG secondary findings (acmg.py + acmg_build.py) — just-dna-enricher check-acmg

VariantRow.acmg_sf has been materialized into weights.parquet since 0.4 and checked against nothing. The compiler cannot hold a gene list (that is the un-injected reference RM21 taught), and no pass here had one, so the column was assertable and unfalsifiable. This closes it.

Read this part first: the scraped list is a release behind, and the check now says so. ACMG published SF v3.3 in June 2025 — 84 genes over 100 gene-condition rows, adding ABCD1, CYP27A1 and PLN — and NCBI still serves its adaptation of v3.2 (81/94). The five guards below all pass on that page, because it is neither truncated nor re-laid-out; it is simply old. So the check reported acmg_sf=true but ABCD1 is not on ACMG SF v3.2 about a row that is right, which is precisely the short list failure the guards exist to prevent, arriving where no guard could see it.

Two things changed, and the --sf-list half is the one to use:

# once, from ACMG's supplementary workbook (assets/acmg_sf_v3.3.xlsx, or your own download)
just-dna-enricher acmg build assets/acmg_sf_v3.3.xlsx            # → data/repro/acmg_sf
just-dna-enricher check-acmg spec/ --sf-list data/repro/acmg_sf --offline
  • The list can be injected. acmg build turns ACMG's workbook into acmg_sf.csv + release.json (declared sf_version, source_sha256, DOI, counts), and load_acmg_snapshot reads it with the standard library — so check-acmg is the first check here that works --offline. The workbook is the better artifact in every way that matters: version-pinned behind a DOI rather than hand-maintained, content-hashable, and carrying four columns the page does not (Inheritance, Phenotype Category, the release that first listed the gene, and ACMG's scope-of-reporting text). Only MedGen concept ids go the other way — they are on NCBI's page and not in ACMG's sheet, and no verdict reads them.
  • The scrape path carries a staleness tripwire. KNOWN_LATEST_SF_VERSION is one version string, not a gene list. When the list actually read is older, every disagreement — both directions, since ACMG can remove entries as well as add them — is demoted to unverifiable: reported as a warning, never a strict refusal. A mismatch against a superseded list is a question, and answering it anyway is worse than saying nothing. The stale-constant risk is asymmetric on purpose: when v3.4 ships the constant under-warns, i.e. degrades to the previous release's behaviour, whereas a hand-kept gene list would make confident wrong claims about specific genes.

The scrape stays, because it is the only zero-setup path. Probed 2026-08-03: ClinGen's FTP publishes gene-curation, region-curation, dosage and recurrent-CNV lists and no secondary-findings list; ClinVar's FTP tree carries no ACMG flag (gene_condition_source_id, 13,478 rows, zero mentions). NCBI's adaptation of ACMG's Table 1 at /clinvar/docs/acmg/ is still the only fetchable form of the list, so it remains the fallback with the guards that branch was conditional on — now with its version checked.

The guards are load-bearing, and the naive parse really is wrong. Splitting the table on <tr> returns 78 of the 81 genes, silently. The page is hand-maintained and shows it: two rows open with a bare <td> after the previous </tr>, four leave a <td> unclosed with a stray trailing </td>, and the gene cell links through three URL shapes (/gtr/genes/324, /gtr/genes/4089/, /gene/3949). The three genes the naive split drops are TP53, COL3A1 and TPM1 — which is the failure mode stated exactly: a short list makes correctly authored acmg_sf=true rows look wrong, and it would have begun with the single most recognizable secondary-findings gene there is. So the parse works in cells, not rows, and refuses rather than returning a short list:

Guard Catches
the page declares ACMG SF vN.N a re-write, and it supplies the dataset label
one table carries all four EXPECTED_HEADERS a re-layout, or a nav/footer table becoming "the list"
<td> count divides exactly by four a column added or dropped
every four-cell group yields exactly one gene link a gene cell that lost its link — a silent drop
at least MIN_GENES distinct genes survive a truncated response, a JS shell, an error page

MIN_GENES is a floor, not the real count. Hard-coding 81 would be the hand-transcribed gene list this module exists to avoid, and it would go stale the day ACMG publishes v3.3 — which it since has, and the version tripwire above is the answer that a count would not have been. The workbook parse reuses the same floor and adds one guard of its own shape: ACMG's trailing disclaimer sits in the Gene column, ~1,200 characters of prose that a naive read counts as an 85th gene. It is skipped only when every other cell in its row is empty; a symbol that cannot be read on a populated row refuses, because that is the <tr> failure again.

The list is richer than a set of symbols, and cells hold more than one of things. Each row is a gene–condition pair (94 pairs over 81 genes; TRDN is listed for two conditions), and a cell can carry several MIMs and several MedGen concepts at once — SDHB names MIM 115310 and 171300 against MedGen C1861848, C0031511, linking to a MedGen search rather than a concept. disease_mims and medgen_ids are therefore tuples; taking the first of each would be a silent truncation of the same family as the <tr> split.

The verdicts are the house tri-state, and a blank cell is never a defect. agree (either way round) and blank are silent; not_listed (claimed true, gene absent) and denied (claimed false, gene present) are findings that warn in best_effort and refuse in strict; unverifiable is either of those two against a superseded list, which warns in both modes; unstated (blank, gene listed) is a note, because blank means "not stated" and turning that into a defect is the None-means-False collapse this codebase refuses everywhere else; unchecked covers a row naming no gene and --offline without a --sf-list, which reports that nothing was asked rather than that nothing was found.

Findings group by gene, because that is what they are about. Found by running the real thing: the HFE reference example is 13 variants in one gene, and a per-row report printed the same 220-character sentence 13 times. Same aggregation rule CPIC already taught (~600 identical lines for CYP2C19). The per-row verdicts stay on the report for a caller that wants them; AcmgReport.by_gene is what a report prints.

Gene-level, and the column says so — but ACMG's own list is not purely gene-level. acmg_sf is documented as "True when the gene is on the ACMG secondary-findings list", so that is what is compared. ACMG itself is finer-grained in at least one place: the HFE entry reads "Hereditary hemochromatosis (c.845G>A; p.C282Y homozygotes only)". The parse keeps that text and the denied message points at it, telling an author to leave the cell blank rather than false when a row is about a variant in a listed gene that is not itself a reportable finding. Reading the column as per-variant reportability would make the format decide disclosure policy, which it does not do.

This pass records no SourceRow, the deliberate exception to "a pass that consults a source writes one". That rule is about a module carrying a source's data. Nothing lands in the module here: acmg_sf was authored by a human before this ran, exactly as a gene symbol was, and this asks a registry whether the authored value is still right. It is check-identifiers' shape (HGNC and OLS4 go unrecorded too), not dosage's. The corollary is in the other direction: acmg_sf joins hints.REDUNDANCY_BEARING, so no lookup or hint may fill it — a cell filled from the list this checks against would make the check vacuous.

Repeat-allele bands against STRchive (strchive.py + strchive_build.py) — check-repeat-bands

repeat_alleles.csv was the one binning kind nothing in this tier had ever asked a question about, and the corpus behind it is two hand-authored modules. STRchive (dashnowlab/STRchive, MIT) publishes 82 tandem-repeat disease loci with coordinates, motifs, the motif structure and three bands each — benign_*, intermediate_*, pathogenic_*.

The item splits the source by column: the identity half is drafted, and the band half is checked and never written. That split is the whole finding, and it was decided by measuring both corpus modules rather than by caution.

just-dna-enricher strchive build --release v2.26.0               # → data/repro/strchive
just-dna-enricher check-repeat-bands spec/ --catalogue data/repro/strchive

What the measurement said

reference_examples/htt_repeat_expansion reproduces the catalogue exactly where the catalogue speaks: STRchive gives benign 6–26 and intermediate 27–35, and the shipped table gives 6,26 and 27,35, independently authored. reference_examples/fmr1_cgg_repeat is where the source is coarser — STRchive runs one intermediate band from 45 to 200 and the module states 45–54 and 55–200, and 55 is the premutation threshold, the line between a carrier who is not at risk and the FXTAS/FXPOI range. A drafting provider would have written the three bands as the answer and erased it.

Building the check turned up a third thing the item had not measured: HTT is finer than the catalogue too. The module divides STRchive's single pathogenic band at 40, which is the reduced-penetrance / full-penetrance line. So both corpus modules refine the catalogue, in two different places for two different reasons, which is a stronger argument for the split than the one that motivated it.

pathogenic_max is reported and never written

STRchive's HTT pathogenic_max is 250 and the module leaves its top band open. A catalogue's pathogenic_max is the longest allele the literature records — an observation, not a clinical bound. Written into measure_max, a 300-repeat allele would match no bin at all, silently and under --strict too, which is the exact silence RM55 shipped a loud warning about. It gets its own finding kind, ceiling_only_in_source, so that "the source states a ceiling the module does not" can never be confused with "the two ceilings disagree", and a test asserts the number reaches no cell.

How the comparison is put

The authored rows are partitioned by binning._bin_groups — _KEY_FIELDS + (trait_efo_id,), with unresolved sentinels dropped — because that is the partition the overlap rule is enforced over, and grouping any other way would answer this question against a partition nothing else uses.

A group joins a catalogue locus on gene plus the motif, matched against both orientations the source publishes: repeat_unit is one string with no column beside it to say which strand it is written on, so a minus-strand locus is legitimately authored either way.

The comparison reads lower bounds, which is the one representation both tilings share: adjacent bins are [6,26] [27,35] under quantised tiling and [6,26] [26,35] under continuous (@dense-bin-boundary), so an upper bound means two things while the next bin's lower bound is the division either way. A source division is a closed window [previous.max, next.min] rather than a point, so the same division stated in either spelling matches; resolve_tiling decides only which endpoint the message names, and whether a cut at a bin's own top edge divides it.

The finding kinds, one sentence each, asserted equal to the arms compare_bands can emit: a boundary only the module has or only the source has, a floor that differs or is open on one side, and the same three for the ceiling.

What it withholds, and what it never does

Three absences get three different sentences, because a reader chasing a skipped locus looks in three different places: the catalogue carries no such gene; it carries the gene under no matching motif; or it states no bands for it at all (twelve of the 82 loci state none). A (gene, motif) key several loci claim — ARX has two and HOXA13 three in the published file — is reported as contested and compared against nothing, because picking one would make the verdict depend on record order.

--strict never escalates a band difference, in either direction and on either corpus module. Two expert bodies drawing a threshold in different places is a difference between authorities, and a compile that refused would have this format pick the winner — the rule the ClinVar clin_sig and PGx allele-function checks already follow. What strict still refuses is structural: a repeat_alleles.csv that will not load raises in both modes.

This check records no SourceRow, on acmg's rule rather than as an oversight: sources.csv exists so a module can account for what it carries, and nothing from STRchive lands in the module on this path — the bands were authored by a human before it ran. The drafting half does carry the source's data and does write one.

It is also deliberately absent from provenance.DRAFT_PROJECTIONS. That map is for a source that drafts a column one of its own checks later compares; the split means the columns this check reads were never copies of anything, so a projection over measure_min/measure_max would let the tautology skip fire on cells no drafter ever wrote.

The snapshot

strchive build streams STRchive-loci.json and writes it beside a release.json carrying source_url, source_sha256, locus_count, dataset, license, built_at and builder_version. The catalogue is copied rather than re-serialized, so source_sha256 describes bytes a reader can verify with sha256sum. There is no parquet and no [dev] extra: the shape the readers want is the shape the source publishes.

Pin a release. --release v2.26.0 builds from the tag and records dataset=strchive_v2.26.0, which is what the verification record then names. Building from the default branch is allowed and records no label at all rather than inventing one out of a date — an unlabelled snapshot is honestly unlabelled. There is no --offline (the off-switch is passing --catalogue instead of downloading) and no --use (MIT grants the fetch, so the gate would answer the same on every run).

Drafting the identity half (strchive_draft.py) — draft-repeats

just-dna-enricher draft-repeats spec/ --catalogue data/repro/strchive --gene HTT

Without --catalogue the provisioned lane is read ($JUST_DNA_STRCHIVE_CACHE, else the shared cache base), as check-repeat-bands reads it; until RM395 the drafter skipped with no STRchive catalogue was provisioned on a machine holding one. With neither, the skip is nobody-asked, never an empty source.

A drafted row is thinner than the item that ordered it expected, and that is this module's own finding. The proposal listed chrom/start_hg38/stop_hg38, locus_structure, ref_copies and the disease identifiers as the identity half. RepeatAlleleRow has a column for exactly one of them: trait_efo_id. Repeat coordinates are RM65 and the motif structure is RM66, both deferred, so what a row can actually carry is the gene, the motif as the catalogue spells it, the trait CURIE and the fixed measure_kind, with conclusion left as <<REPLACE>>.

The rest is counted and reported, not dropped: how many loci state a fractional ref_copies (a reference allele that is not a whole number of motif copies — RM55's case arriving in a source rather than in a caller's VCF), and how many publish a locus_structure. Neither is rounded into a cell and neither is silently discarded.

STRchive's own evidence grade is named, since RM276. Every locus carries a ClinGen-style grade (Definitive … Refuted), and no authored column holds it, so a locus STRchive grades Refuted (DMD) or Disputed (DIP2B, NIPA1, POLG) used to draft exactly like HTT. A drafted row carrying one of those two grades is now named in a note, in both modes and never as a gate; a Provisional locus (STRchive's not yet curated) is named apart and never called weak. The grade is read off the catalogue file by a private reader; with no file to read, the note says the grades were unread rather than implying none is doubted. Whether the grade should travel into the module is RM294.

What the thin row is still worth is the part an author cannot get from anywhere else: the list of loci, the motif in the orientation the catalogue publishes, and the MONDO id. On HTT the provider derives MONDO_0007739 from mondo and the shipped module's human author wrote the same value — independent agreement, which is the strongest evidence a drafting provider gets. On FMR1 the catalogue names three MONDO ids (fragile X syndrome, FXTAS, FXPOI) and the provider writes none, which is what the shipped module does too.

Never drafted: the band columns. measure_min, measure_max, measure_tiling and the per-band direction/clin_sig/phenotype. The withheld set is derived from the drafted one, so a column added to the kind later is withheld by default rather than silently written empty, and a test asserts the two partition the model's authored fields.

A (gene, repeat_unit) key several loci claim is drafted for none of them — ARX and HOXA13 are the published cases — and contestation is decided over the whole admitted set before the gene filter runs, so a filter cannot leave one claimant looking uncontested. The skip guard for a locus with no usable identity is read out of authoring_requirements, not restated beside the model.

Rows match on (gene, repeat_unit) and go at the end. trait_efo_id is deliberately outside the match key even though the bin-group key includes it: it is a cell the author may clear, and a re-draft after they did would otherwise append the locus twice. A re-draft after the human has written the bands and split one drafted row into four still reports already_present.

STRCHIVE_TERMS is MIT — share_alike=False, commercial_use=True, redistribution=True, all facts rather than the None every other axis in this round had to say — so a module drafted from it stays sellable and check_declared_use answers None on every declaration. The row is written at the annotation layer with the snapshot's release in dataset, because this half does carry the source's data into the module.

Pharmacogenomics and data-source licensing (pgx.py, licensing.py, pharmvar.py, cpic.py)

Pass 5 cross-checks a module's star-allele tables against the nomenclature authorities and records what was consulted and on what terms into the module's licence table. It is the first pass whose primary output is provenance rather than facts.

That file is licensing.csv since 0.6, and sources.csv is the deprecated spelling (RM51, removed at 1.0). No pass names either by hand: licensing.sources_path — and sidecar_path for the other machine-written tables — resolves it through just_dna_format.layout, which also accepts a derived/ subdirectory (RM49). Two rules every pass inherits from that, and both matter: write to the file you read (writing the current spelling onto a module carrying the older one, or the root onto a module that is split, leaves two copies), and two copies is a refusal naming both paths, raised as the calling pass's own error. Never a merge and never newest-wins — these tables are hand-overridable, so two copies are two claims.

The bottom line first: a PGx module is non-commercial only

Every PGx upstream forbids sale, and PGx tables are the layer that taints, so one drafted row settles it for the whole module. ClinPGx, CPIC and PharmVar each carry CC BY-SA 4.0 plus a contractual bar on selling the data, which licensing.{CLINPGX,CPIC,PHARMVAR}_TERMS each record as commercial_use=False. The PGx tables — haplotypes.csv, allele_function.csv, diplotypes.csv, pharm_variants.csv — are the module's own authored annotation, so their SourceRow sits at the annotation layer, and that is the one layer sources.taints_commercial_use treats as tainting. The verdict is most-restrictive-wins, module-wide: mixing in a permissive source cannot launder a restricted one, and the compile refuses in both modes unless the licence table records declared_use=non_commercial for every tainting source. unstated is not a loophole — it is the absence of a declaration, which is exactly what the gate is looking for.

reference_examples/cyp2c19_star_alleles/licensing.csv is the shape: one CPIC row, commercial_use=false, declared_use=non_commercial. reference_examples/pgx_slco1b1_simvastatin/licensing.csv is the same with a licence hash pinned from the bytes it was read out of (license_sha256, dataset=clinpgx_2025-07-05).

And declaring is asserting, not proving — the gate's own closing sentence says so. Recording non_commercial states how the module will be used; nothing in the compiler can check that, and it does not pretend to.

Two things the flat "non-commercial" summary does not say, both of which matter:

  • It is not "unrestricted if you give it away." Sale and distribution are different rights. redistribution is a third recorded axis and it is deliberately not gated (RM27) — all three PGx sources record redistribution=true, since CC BY-SA expressly permits sharing under share-alike plus attribution, so nothing in this workspace trips it today. But the axis exists precisely because academic-use-only sources (OMIM, dbNSFP) permit neither, and "non-commercial" is the reading that would hide the difference.
  • PharmVar is stricter than any column can express. Its recorded notice reads research use only … not intended for direct diagnostic use or medical decision-making, and its API key is personal and non-transferable under its terms §2. commercial_use=False is the nearest a column gets; the rest stays prose in notice, on purpose rather than by omission — a fourth licensing axis means a new SourceRow column, and that is a design round rather than a drive-by addition, so it is a 1.0 change and not a minor one. The restriction is therefore recorded and legible; it is not machine-enforced, and nobody should file the column as cheap.

The licensing picture, probed 2026-08-02

Source Endpoint Auth Licence Sellable
ClinPGx (ex-PharmGKB) api.clinpgx.org none CC BY-SA 4.0 + no-sale clause ❌
CPIC api.cpicpgx.org/v1 (PostgREST) none same policy ❌
PharmVar www.pharmvar.org/api-service Api-Key header, 2 rps CC BY-SA 4.0 + research-use-only ❌
Ensembl / dbSNP already in the chain none unrestricted ✅
PubMind (read 2026-08-28, answered 2026-09-28, RM134/RM320) openbioinformatics.org/annovar/download/hg38_pubmind_db.txt.gz none CC BY-NC-ND 4.0 ❌

The PubMind row is dated separately because it was read almost four weeks after the others and was ❓ for a month. No page stated terms for the ANNOVAR-distributed table: LICENSE.md covers the software, and the article's Zenodo code archive was labelled CC BY 4.0. WGLab answered in writing (WGLab/PubMind#5) that the table is CC BY-NC-ND 4.0, and corrected the Zenodo label. See The caches for what that gates.

Three things follow, and each shaped the code:

api.pharmgkb.org is gone. Retired 2026-07-20; the successor is api.clinpgx.org with paths and formats unchanged. ClinPGx is the umbrella that merged PharmGKB, CPIC and PharmCAT.

CPIC is not an escape hatch. cpicpgx.org/license/ 302-redirects to the ClinPGx data usage policy, so CPIC carries ClinPGx's terms. Preferring PharmVar for the star-allele layer is a data-authority choice — it is the naming authority for CYP star alleles — not a licensing one.

No PGx source is sellable — the consequence for a module is drawn out above. Each layers a contractual bar on sale on top of the CC grant, so a bare "CC BY-SA 4.0" line is not permission to sell; read the surrounding terms. Since the coordinate layer is already covered by Ensembl/dbSNP, ClinPGx and CPIC are deliberately never wired as resolution links — that keeps coordinates unrestricted and leaves nothing to declare there.

The complete roster — TERMS_BY_SOURCE, every source this tier has terms for

The table above is the PGx picture and is scoped to it. This one is the registry: every key in licensing.TERMS_BY_SOURCE, which is what a pass reaches for when it writes a SourceRow. A source absent from it has no recorded terms, which is a different state from having permissive ones — @no-named-licence, and the reason every column below can read —.

Source Licence Sellable Share-alike Redistributable Note the column cannot hold
clinvar public domain ✅ — ✅
clinvar_gkm CC0-1.0 ✅ — ✅ CC0 covers ClinGen's transformation only; ClinVar underneath is public domain, and its attribution and disclaimer are owed (RM305)
dbsnp public domain ✅ — ✅ the placement witness since RM398, read off the merge check's esummary db=snp record; NCBI asks that dbSNP be cited
ensembl Apache-2.0 ✅ — ✅
gnomad CC0-1.0 ✅ — ✅
clingen CC0-1.0 ✅ — ✅
gencc CC0-1.0 ✅ — ✅
civic CC0-1.0 ✅ — ✅ the only annotation snapshot that may be published
strchive MIT ✅ — ✅
mitomap CC BY 3.0 ✅ — ✅ a floor: the host grants it "unless otherwise noted", so a per-record note outranks it
alphagenome_avi AlphaGenome Services Additional Terms ✅ — ✅ Permissive Use (RM195); the sign-in bars classes of holder, which no column says
clinpgx CC BY-SA 4.0 ❌ ✅ ✅ + a contractual bar on sale, on top of the CC grant
cpic CC BY-SA 4.0 ❌ ✅ ✅ cpicpgx.org/license/ 302s to ClinPGx's policy
pharmvar CC BY-SA 4.0 ❌ ✅ ✅ research-use-only, and the key is personal and non-transferable
pubmind CC BY-NC-ND 4.0 ❌ — ✅ NoDerivatives: adapted material may not be shared, which is why the snapshot is never published (RM320)
alphagenome_atlas AlphaGenome Output Terms of Use ❌ — ✅ a second name for one service, because one (source, layer) key cannot carry two licence classes
mane — ❓ ❓ ❓ NCBI states a policy rather than a licence — nothing to refuse and nothing to permit
pgs_catalog — ❓ ❓ ❓ a floor: terms are per score, so PGS_LICENSE_CLASSES is consulted per record
gwas_catalog — ❓ ❓ ✅ redistribution is stated; the other two are not
opentargets CC0-1.0 ✅ — ✅ the release is marked CC0 and the terms of use defer to the original data owners; notice carries that clause and the third-party-rights one verbatim (RM333, the owner axis is RM345)
clingen_allele_registry — ❓ ❓ ❓

❓ is None and never False. A source whose terms could not be established has not been shown to permit anything, and "we could not read the terms" is not a finding that they forbid anything either — so it is skipped in every --use column with a message saying which of the two it is. That is the house algebra landing on a licence: unknown never becomes permission, and it never becomes a refusal either.

Four of the nineteen are a floor rather than a total, and that is a property of the source, not of the constant: mitomap grants site-wide "unless otherwise noted", pgs_catalog is per score, gwas_catalog states one axis of three, and alphagenome_avi's permissive Output terms sit behind an eligibility clause about the holder that no column models (@a-hosts-terms-are-not-its-contents-terms).

declared_use — a third axis, not a mode

--use is unstated (default) | non-commercial | commercial, threaded to enrich_pgx(declared_use=). It is orthogonal to mode on purpose (Principle 5): mode says how hard to fail on a finding, declared_use says who is using the data and why. A three-state string rather than a bool pair, because a bool cannot express the default and defaulting either way would have the tool assert a purpose on the user's behalf.

Source terms --use unstated --use non-commercial --use commercial
forbids sale skip + warning fetch, record the declaration refuse, fetch nothing
unknown (None) skip + warning skip + warning skip + warning
permits fetch fetch fetch

Unknown never becomes permission. A source whose terms could not be established has not been shown to permit anything, and "we could not read the terms" is not a finding that they forbid anything either — so it is skipped in every column, with a message that says which of the two it is.

The refusal lives here, at acquisition, because under a data-usage policy that is when the terms are accepted, and because refusing here means nothing is fetched rather than merely nothing written.

A module's own recorded declaration counts when the flag states none (S105, RM252). Every gate that has a module directory — pgx's two legs, the CPIC/ClinPGx/ClinVar/AlphaGenome drafters and checks — goes through licensing.effective_declared_use(spec_dir, terms, declared_use) before check_declared_use: the flag when it states one, otherwise the row an earlier run recorded for that source at that layer in the licence table, otherwise unstated. So pgx on a module drafted under --use non-commercial no longer says "cpic forbids sale and no use was declared" about the file it has just read; it runs the CPIC leg, reports recorded_use={"cpic": "non_commercial"} (the summary line says where the declaration came from), and PharmVar — which has no row — still asks. Three things it is not: not a default (unstated on disk is not a declaration, and with no row the tool asserts nothing); not per module (a declaration for CPIC says nothing about PharmVar); and not a change to the flag, which outranks the file in both directions — --use commercial against a recorded non_commercial still refuses. The cache lanes (cache pull, the builders) have no module and gate on the flag alone, and test_declared_use_recorded.py walks every check_declared_use call site to keep the two sets exact.

Where the terms come from

The per-source SourceTerms constants (CPIC_TERMS, PHARMVAR_TERMS, …, collected in licensing.TERMS_BY_SOURCE) hold only the residue that cannot be read from a payload. Where a source ships its own licence — ClinPGx bundles a LICENSE.txt inside every archive — the pass reads it from the same bytes it took the data from and records license_sha256, which makes the recorded terms provably contemporaneous with the recorded data instead of a lookup that was true once. Both halves of a static table went stale inside one release (the retired hostname, the moved licence page); a hash turns the next such change into a finding.

The compiler holds no source→licence map — that would give it a source convention (Principle 2) and an un-injected reference. It reads only what the enricher recorded.

A grant may be a floor rather than a total, and two constants are written that way. The PGS Catalog's terms are per score. MITOMAP's page grants CC BY 3.0 for all of Mitomap.org "unless otherwise noted" (MITOWIKI/HelpTerms r5, read from a browser on 2026-09-03 — the data surface serves the dump to plain curl while the web surface 403s), so MITOMAP_TERMS states what the host grants and never that every cell in the dump carries it: a per-record note outranks the site default (@a-hosts-terms-are-not-its-contents-terms). Its commercial_use=True is stated rather than inferred from the CC grant — the page names individuals, clinical labs and commercial services, with no permission and no fee — which is why that axis is a True and not the None an unestablished permission gets. One trap, since a search reproduces it: MITOMAP's NAR article is CC BY-NC, and that is the paper's licence, not the database's.

On a host, or in a service — the gated sources need a cache (RM38)

Everything above assumes the shape this tier was written for: an author runs the enricher on their own machine. That case is fine as it stands, and the reason is worth naming, because it is exactly what stops being true elsewhere. The author accepts the source's terms themselves, spends their own rate budget, and holds their own PharmVar key.

A hosted enricher — the same functions behind an HTTP endpoint, enriching modules for whoever is calling — is a different act, for two independent reasons. Either one alone justifies the conclusion, and they have different consequences, so they are worth keeping apart:

  • Terms and identity. The operator's acceptance stands in for every end user's, and the operator's personal, non-transferable PharmVar key becomes the key third parties query on. There is no per-user switch: the presence of PHARMVAR_API_KEY in the server's environment is the switch.
  • One shared budget. Every published figure in the rate table is per IP, so a server does not get a budget per caller — it multiplies its callers onto one allowance. PharmVar's 2 rps and gnomAD's 10-per-60s are the whole deployment's, and an overspend limits the service rather than the caller who caused it.

The resolution half always handled this, which is why only the gated sources were at issue. Ensembl, ClinVar and gnomAD constraint each have a snapshot, a locations resolver and an ensure_*, so a hosted enrich() is cache-served and --offline is genuinely zero-egress — and all three are commercial_use=True, so there is nothing to gate anyway. The gated set is precisely the three PGx sources, and until 0.5.1 none of them had a working cache path:

Gated source Snapshot builder locations resolver ensure_* Runtime pass
ClinPGx clinpgx_build.py (shipped 0.5) ✅ 0.5.1 ✅ 0.5.1 resolve → provision → skip with a reason
CPIC ✅ cpic_build.py (0.5.1) ✅ 0.5.1 ✅ 0.5.1 snapshot → live → skip with a reason
PharmVar ✅ pharmvar_build.py (0.5.1) ✅ 0.5.1 none, by design snapshot → live → skip with a reason

Before that, --offline was a no-op for pgx (it warned and returned, because there was nothing to fall back to) and was absent entirely from draft, so a hosted PGx path had two options — fetch, or skip the check — and neither was the one it wanted. See The caches above for the operator's side.

The rule this tier follows: a hosted surface reaches a gated source through a snapshot the operator built once, never live per request. --offline is the only switch and an explicit --snapshot / --*-cache path is the inject-only escape hatch — the same shape the ClinVar and constraint snapshots already use, and deliberately not a second flag.

One asymmetry in publishing such a snapshot. The recorded terms permit redistribution for all three, so ClinPGx and CPIC snapshots follow the full ClinVar pattern — build, publish, ensure_*. PharmVar cannot: bulk data pulled under a personal, non-transferable key is not covered by any axis the terms record, and an unestablished permission is never a permission here — the same None ≠ False rule that governs share_alike and commercial_use, applied to a clause no column models. A PharmVar snapshot therefore stays operator-built and inject-only: a resolve_pharmvar_reference and a builder, and deliberately no ensure_pharmvar_snapshot and no pharmvar publish.

And a coordinate bug the snapshot turned from latent into written. PharmVar publishes each defining variant against both assemblies and lists GRCh37 first; _merge_variants was first-wins over any NC_ row, so 451 of the 739 rsID-keyed defining variants would have carried a GRCh37 position (DPYD rs868235016 as chr1:97547910 rather than its GRCh38 place). The accession version cannot separate them — chr10 is .10/.11 and so is chr22 — but referenceCollections does, exactly. Nothing consumed PharmVarAllele.variants before, which is why it never bit; a snapshot stores them. See pharmvar.PHARMVAR_GENOME_BUILD.

CPIC does publish a chromosome, on gene.chr. An earlier probe read sequence_location alone — which genuinely has none — and concluded CPIC publishes none at all, so the drafting provider skipped every defining variant CPIC gives no rsID for: 18 in CYP2C9, 14 in TPMT, 4 in NUDT15. Joining gene.chr onto the symbol the location row already carries is a lookup in CPIC's own tables, not the inference that probe rightly refused, and draft --gene CYP2C9 now writes 17 coordinate-only haplotype rows it used to drop.

licensing.record_source_terms(names, layer, path) is the one place that turns "this pass consulted these sources" into sources.csv rows; since 0.8 a derived pass reaches it through record_table_sources, which labels each row's dataset by the committed table (§ Which release answered, RM373). Three passes were missing it entirely, which is why VALID_SOURCE_LAYERS had members nothing ever wrote:

Pass Layer Sources recorded
enrich (resolution) resolution whichever of ensembl / clinvar / gnomad answered
frequencies frequency gnomad
gene_metrics gene_metrics gnomad (and clingen writes its own dosage row)

None of these layers can taint a module — only annotation does, because a coordinate or an AC/AN is a fact the source reports rather than expression it owns. So what these rows carry is attribution, which gnomAD, Ensembl and ClinVar each request and none enforces, and that is as much what the table is for as the prohibitions are. declared_use is unstated: none of them forbids sale, so these passes never have to ask.

resolution.csv records both a link and an authority. source names which link answered (ensembl-rest, cache, …) and authority names the licensed source it speaks for (ensembl), which is what sources.csv joins on. RESOLUTION_AUTHORITY_BY_LINK is that map, and it lives here rather than in the compiler for the reason stated one paragraph above. A link with no entry — authored, reversed, manual — keeps an empty authority, because the module's own bytes are not a licensed source. gene_metrics.csv had the same overloading and was fixed the other way: it records gnomad, and which release answered stays in dataset, where the two-constraint-routes distinction already lived.

A gene panel is drafted, never decided

clinvar_draft.draft_gene_panel is the provider RM4 waited for, in the shape the charter allows: it drafts rows a human then owns, with no compile-time reference materialization. It was blocked on a real problem rather than on effort. VariantRow.genotype is required and ClinVar publishes alleles, not genotypes — whether carrying a pathogenic allele once is informative (a carrier, an affected proband, neither) follows from the condition's inheritance mode, which ClinVar does not state. Writing A/G because the alt is G would be a clinical claim the source never made; reference_examples/pathogenic_clinvar/ is a human having made that call by hand, row by row.

So the provider writes a partial row: everything ClinVar publishes, with genotype carrying vocab.TEMPLATE_PLACEHOLDER, which no mode compiles. The panel is authored, in place, in gene order, and loudly incomplete until someone decides. A re-draft after those decisions adds nothing, because a partial row matches on the identity columns rather than on the natural key — that key runs straight through the column still holding the stub.

The alleles the author writes that genotype from stay in the report, and since 0.7 the report can be asked again (RM71). A drafted row is rsID-only, so rs118203998 arrives with empty ref/alts and the pair is stated only in the warning stream — one line per open row, uncapped, because each is a task. Every place the alleles could legally be written is the wrong place: identity is filled whole or not at all, alts is redundancy-bearing (filling it makes the compiler compare ClinVar with ClinVar), a comment column is full cost on the most expensive table and rots at the next release, and an unread sidecar becomes a permanent resident of every drafted module. So the repair is scope, not a column. The worklist covers every row in the file whose genotype is still the placeholder, not only the rows this run added, so a second draft-panel reprints it instead of falling silent — and --dry-run obtains it without appending anything, meaning exactly what it means on draft. The earlier narrowing survives for a better reason than scoping gave it: a row the model refused was never written, and a row the author has settled no longer carries the stub. The state line moves with it, since it reads as "these rows also". Where the run holds no record for an open row — usually another gene, or a tighter --clin-sig/--min-review-stars, though the pass cannot tell which — the alleles are genuinely unknown, so they are withheld and the rows are named by gene in one aggregated line rather than guessed at or dropped from the count. The line states only that nothing this run selected covers them; naming a cause it did not establish would be a guess dressed as a finding.

A scaffolded template row is not drafting work. scaffold writes a variants.csv row carrying the placeholder in rsid as well as in genotype, and file scope would otherwise sweep it in — as an inflated count, as a row supposedly belonging to some other gene, and as <<REPLACE>> in a list of row labels. A row whose identity is itself the placeholder is skipped here: it has no identity to match, so no run can ever hold its alleles, and the compiler already refuses it by name.

Two rules worth keeping in view. Identity is filled whole or not at all: the rsID, else the complete coordinate, never a subset, because a lone alts on a position-only row makes derive_variant_key mint a VRS ga4gh:VA.… id instead of chrom:start:ref — a partial coordinate silently changes which variant the row is.

An rsID is only usable as an identity when it names one allele event here, and the predicate that decides that is multi_allelic_rsids — which keyed the site on ref until S41. It grouped by (rsid, chrom, start, ref) and fired on more than one alt inside that group, which reads as more than one alt at one position and is not: an ordinary ClinVar dup/del mirror pair, A>AT beside ATT>A at one position, is two groups of one alt each. So the rsID was never flagged, both records took the same rsid-only identity, and append_partial_rows dropped the second as already_present — a silent loss, since dedup is the normal case and nothing distinguishes it from a re-draft. The event is now the whole (chrom, start, ref, alt), so a differing ref and a differing position both flag. Measured on the 2026-06-27 snapshot over BRCA1/BRCA2/ATM/MLH1/MSH2: 942 rsIDs flagged before and 1,589 after, the 647 new ones exactly the 647 collapsing identities, 725 records recovered, no record made unkeyable. 187 of those collapses had dropped the better-reviewed record — a consequence worth stating separately, because select_by_gene orders by ref before review_stars DESC, so which of the pair survived was decided by allele spelling rather than by evidence. Distinctness is over the allele event and not over records: a re-submission under a second variation_id is one claim written twice, and coordinate identity could not separate those anyway.

A re-draft repairs the omission and cannot retract the collapse — so the drafter now names what it supersedes (S45). "Your module needs a re-draft" reads as a complete instruction and is not one.

The distinction that decides it: a drafting fix either skipped rows or wrote them under an identity that has since moved, and only the second leaves anything behind. S44's widened genotype gate is the first kind — measured on the same afternoon, a stale ClinPGx module of 18,691 rows re-drafts to 18,895 with 0 missing and 0 stale, byte-for-byte a fresh draft, so it needs no caveat at all. S41 is the second kind, and the two shipped in the same release, which invites a reader to generalise the wrong remediation. Drafting appends and never mutates, so re-running the fixed drafter over an existing spec adds the coordinate-keyed rows beside the collapsed rsid-only row rather than replacing it: the module then states both the right answer and the wrong one for the same locus, and the stale row is the one carrying the mislabelled expansion. Reproduced on MLH1 at min_review_stars=2: a stale module of 996 rows re-drafts to 1,061 against a fresh draft's 1,030 — 0 identities missing (every dropped record recovered) and 31 present that a fresh draft does not contain.

Those 31 are invisible from inside the module: _row_cells writes no rsid on a coordinate-identity row, so the obvious predicate — an rsid-only row whose rsID also appears on a coordinate row — finds 0 of 31. What can see it is the pass itself, which holds ambiguous: an rsid-only row carrying an rsID this run is deliberately writing by coordinate is a row no current draft would produce. _superseded_rsid_rows reports them, aggregated through examples, after the append.

Reported, never removed, and that is the reporter's own preference for the reason they gave: a drafted row is authored material by the time a re-draft runs, a human may have curated its genotype, state and conclusion, and deleting curated work to repair a drafting defect is a trade only the author can make. Drafting-appends-never-mutates is the rule one file over, and a provider that started deleting rows would be the exception to it. The cleaner remediation is still a fresh directory reconciled against the old one; the notice exists for the author who did the other thing, which is what the instruction invited. And min_review_stars defaults to 2: a panel that mixes a 0-star "no assertion criteria" submission with a 3-star expert-panel review without saying so is worse than one that names its floor.

What it does not fill is as deliberate as what it does — no weight, direction or effect statistic (ClinVar publishes none), no trait_efo_id (its condition is free text and MedGen, not EFO), no acmg_sf, and no curator/method (the spec's defaults: block owns those).

A non-diploid contig gets its genotype written, because there is no judgement there to protect (0.5.2). MT is haploid and chrY outside PAR1/PAR2 is hemizygous: exactly one genotype is expressible per allele, so sole_expressible_genotype writes the ALT and the row arrives complete. The rule the placeholder encodes is unchanged — it exists for a decision the source does not make — and this is the case where that decision does not exist. Y is decided per locus through the same three-valued vrs.in_pseudoautosomal_region the compiler's ploidy guard uses (XG and SPRY3 straddle a boundary), and both True (diploid) and None (no PAR table for this build) keep the stub, which is the house rule about unknowns. The run reports what it committed to in one aggregated line naming the contigs: those rows read as homoplasmic/hemizygous, and a heteroplasmic level is a different question with its own table kind. Without this every consumer rediscovered it the hard way — one wrote A/G and A/A across 264 mitochondrial loci in a genome-wide panel and 260 in a cardiac one, each asserting a second copy that is not there.

A citation ClinVar files under PubMed that is not a PMID is skipped and counted, never raised. 218 of the 3,952,341 PubMed rows in the 2026-06-27 file carry a nine-digit id (Variation 12606 cites 168335863; PubMed is at eight), and StudyRow.pmid rightly refuses them — but the refusal used to surface as an unhandled ValidationError that aborted a 297-gene draft over one row in one gene. The builder now drops them at the snapshot boundary using the format's own extract_pmids grammar rather than a second opinion restated here, and the drafter survives one anyway, since every snapshot already published carries them. The two shortfalls are reported apart: --max-citations is a choice this run made, an unusable id is a defect in the source.

The snapshot is found, then provisioned — --snapshot used to be required (0.5). _resolve_snapshot runs the ladder enrich() uses: an explicit path is taken as given (the inject-only escape hatch, and what an air-gapped run passes), else the cache locations, else the published snapshot is downloaded unless --offline. Until this, the published snapshot could not reach an author at all — they had to build 4.4M records from a 200 MB VCF or already know the cache path — which mattered most for the citations, since they are what makes a drafted panel compilable and they only started travelling with the snapshot in the same release. Verified end to end from an empty cache: draft-panel --gene HFE provisions data/ + citations/ + release.json and drafts 12 variant rows with 33 grounded study rows carrying real PMIDs, then refuses to compile on the genotype placeholders — which is the designed state, not a failure. No snapshot and --offline raises rather than drafting nothing: an empty draft would read as "ClinVar has nothing for this gene".

Drafting from PubMind — a source with no gene column (0.7, RM134 § C)

draft-panel --source pubmind writes the same rows into the same table from the same gene argument, so it is a flag on the existing command rather than a draft-pubmind beside it. The parts that are hard to get right — the genotype worklist, the placeholder guard, the dedup-against-the-file pass, the refusal summary — are the parts a twin command would have to carry a second copy of, and the worklist's file-scoped seam above is the one this release had to repair. draft-clinpgx is a separate command because it writes different tables; this does not, so pubmind_draft imports the ClinVar provider's machinery instead of restating it.

PubMind names no gene, and this pass does not invent one. The snapshot is (chrom, start, ref, alt) and nothing else locational. Turning --gene BRCA1 into positions needs a gene→locus map, and this repo deliberately holds none — the compiler's own gene/locus check is chromosome-granular for exactly that reason. So the map is ClinVar's own per-record gene attribution matched at the exact position: every position ClinVar records for the gene, with no clinical or review filter, because it is a locus universe rather than a selection and filtering it would narrow what PubMind is even asked about, invisibly. A min/max span over those positions was refused: it invents a boundary nobody defined and writes a gene cell that is a false claim wherever two genes overlap. Both snapshots are therefore required, and each absence names its own switch — $JUST_DNA_PUBMIND_CACHE or --pubmind-cache for one, the ClinVar ladder for the other. A missing ClinVar snapshot raises this pass's PubMindDraftError rather than leaking ClinVar's, and the message says what the second snapshot is wanted for: told "no ClinVar snapshot" by a PubMind command, an author has otherwise been handed a puzzle.

The cost of that choice is stated rather than counted: a PubMind verdict at a position ClinVar has no record for cannot be reached by gene at all, and that class is not countable, since attributing it to a gene is precisely what there is no map for. It includes many of the codon-decomposed offsets.

Identity is the whole coordinate or nothing. The snapshot has no rsID column and most of the source's rows carry no rs-number, so chrom/start/ref/alts go in together. The row still matches on the same five identity columns a ClinVar-drafted row does, so a coordinate both sources speak about is one row in the file rather than two.

A position two requested genes both claim leaves gene empty, counted and named. Both other readings are wrong to write: BRCA1, BRCA2 in that cell is not a symbol check-identifiers can resolve, and picking one is the gene model this pass went to ClinVar precisely to avoid inventing. The row is still drafted — gene is optional and the coordinate is the identity — and the author is told which coordinates and which genes, so filling it stays their call.

Five classes never become a row, and each is named at draft time. A contested key, a length-changing row, a call outside --clin-sig, a confidence below --min-confidence, and a confidence the source never stated — that last one its own class, because None is not 0 and reading an unstated confidence as 0 invents a reading. Every candidate key is drafted or withheld under exactly one reason, and candidates == drafted + Σ withheld is an equality over the walked reason set. A class that withheld nothing reports no zero.

Contestation is decided over every PVID at the key, before either dial runs. PubMind's record id keys on the text a model extracted rather than on a coordinate, so one position carries several records and their calls can disagree. Choosing one needs an ordering nobody defined — mode() over an unsorted group. A --clin-sig or --min-confidence applied first would remove the dissenting record and pick that winner just as effectively, so the filters run second and the coordinate, its PVIDs and its competing calls are all reported instead. Where the records agree, one row is written and the record count, the PVIDs and the best stated confidence stay visible in the transcription.

No study row is drafted, and that is said out loud. The ANNOVAR-redistributed channel carries no PMID and their API withholds per-record detail, so a PubMind draft grounds nothing — studies.csv is mandatory, so the run names the gap rather than leaving the author to meet it at compile.

Unknown terms warn and never gate. check_declared_use is a gate on fetching and its unknown branch skips, which is right for a pass that would go and get data whose terms nobody can state. Nothing is fetched here: there is deliberately no ensure_pubmind_snapshot, the operator built the snapshot with pubmind build, and refusing to read it would make that command's output a file nothing may consume. The reason is reported in the source's own words instead, and the licence row records None on every term — which does not taint, because taints_commercial_use requires an explicit False. What the unknown answer governs is publishing such a module.

A module drafted from PubMind must not be able to confirm itself. pubmind is in DRAFT_PROJECTIONS, projected onto clin_sig, so the drafter stamps a digest of the column the concordance check later reads and the check can establish the copy rather than assume it (@draft-digest). Its identity is the coordinate and not the provider's match_on: the source states no rs-number, so an rs-number an author later adds is a change to the row's spelling and not to the call.

What it does not fill is as deliberate as what it does. No clinvar, pathogenic or benign — all three are ClinVar flags by their own field descriptions, and a position appearing in ClinVar's gene map says nothing about whether this allele is in ClinVar. No phenotype, because the channel carries no condition. state is folded from the source's own call exactly as it is on the ClinVar path, and left as a stub for any call the fold does not cover.

Drafting from MITOMAP — the increment, never the photocopies (0.7, RM171)

just-dna-enricher mitomap build --out data/caches/mitomap                 # the pg_dump → parquet
just-dna-enricher mitomap miss  --out data/caches/mitomap_miss            # the join, from both parents
just-dna-enricher draft-panel spec/ --source mitomap-miss --use non-commercial

The fourth --source on draft-panel, and the first that is not asked for by gene. The other three draft a panel: ClinVar's snapshot is 4.4 M records and CIViC's is a cancer corpus, so an unfiltered draft from either is not a panel but the source. This one's snapshot is the increment, so --gene filters where it exists and is not required; the other three still refuse an empty one.

What it writes. Identity as MITOMAP publishes it (chrom=MT, start, ref, alts), the gene where locus names exactly one, the disease string verbatim as phenotype, and clin_sig from the bracketed ClinGen mtDNA VCEP rating through the one shared normalizer. state is folded from that call where STATE_BY_CLIN_SIG has an answer. Study rows come from MITOMAP's own reference.nlmid links and are position-keyed, like ClinVar's, because a study is evidence about a locus.

What it refuses, and each for its own reason — three sentences rather than one skipped count, because they send an author somewhere different:

bucket why nothing is written
photocopy the same event is in ClinVar under either spelling (indels left-aligned against a vendored rCRS on both sides since RM273), so that VCEP call already reaches this repository with ClinVar's own provenance. Drafting a second copy attributes it to the wrong publisher, and hands a ClinVar concordance check a copy of ClinVar to agree with (@tautology-zero)
unrated miss absent from ClinVar, and MITOMAP published no class this tier may map — no bracket, a bare confirmation token, or [VUS*]. A real identity increment, counted, with no significance invented for it
unmintable the published alleles do not spell a VCF pair: prose in an allele column, or no event. A : deletion is anchored on the vendored rCRS base at position-1 since RM293 and joined like any other row, unless MITOMAP's deleted bases disagree with rCRS

genotype is a placeholder, and the reason is not the contig's. clinvar_draft.sole_expressible_genotype fills the ALT on chrMT — a haploid contig leaves no zygosity open, so there is no decision for a placeholder to protect (S6). That is right about ClinVar, whose record is a claim about an allele. MITOMAP's row is a claim about a literature corpus: homo and hetero say whether the variant has been reported in each state, and a share of the increment is reported only heteroplasmically. Writing genotype=<ALT> there states the homoplasmic reading, which is exactly what reference_examples/mt_heteroplasmy keeps in variants.csv and separates from its heteroplasmy.csv bins. So the cell is stubbed, conclusion with it, and the draft prints one uncapped worklist line per row carrying the alleles and the flags MITOMAP did publish. A MITOMAP-drafted module does not compile until a human writes those cells; that is the cost of the adoption rather than a defect in it.

Four more things the draft says out loud. A drafted row keying on an indel is named. Since RM273 the lane compares events, so such a row is absent from ClinVar under any spelling; an increment built before that compared spellings, and on the real lane five of the six rated misses were ClinVar's own calls at another anchor, so the note tells an author holding an older build to rebuild it. A row whose allele name states a variable number of copies while the allele columns state one definite pair is named too: the source disagreeing with itself, kept rather than repaired, because rewriting it needs a rule for what (n) means that MITOMAP has not given. A parent that has moved since the increment was built is reported and the rows are still drafted — a stale increment is the increment against the older parent, which is a fact worth stating and not an authoring error. And the SourceRow names mitomap, never the derived lane: the computation is this repository's, the content and the attribution duty are the source's, and dataset carries both parents because a derived artifact's identity is the pair it came from.

Lookups answer, they never fill

lookup.py is the authoring counterpart to the passes above: same clients, same offline-capable snapshots, and no writes at all — not a sidecar, not a cell. It answers what an author actually asks. For an rsID: is it live, merged or absent (dbSNP is the oracle; Ensembl returns HTTP 400 on some merged ids and would misclassify them), which coordinates it maps to, and — on demand — whether that answer is ambiguous. For a coordinate: ref, alts, gnomAD populations with the frequency computed as allele_count / allele_number (the API deliberately exposes no af), ClinVar's own call, and — since 0.7 — PubMind's (RM134 § D). For a citation: whether PubMed has the record, plus the DOI and PMC id that arrive free in the same response, with Crossref covering what PubMed does not index at all.

The PubMind leg is three-valued, and it may not fill the cell it reports on. clin_sig is what the concordance check cross-examines, so a hint filling it from one of the authorities being compared would make the check agree with the source it is checking (@hint-redundancy-bearing) — the same defect @draft-digest handles one layer down, and here there is no digest to rescue it. So every record comes back as an advisory with applied=False. No snapshot is nobody asked and says so, naming $JUST_DNA_PUBMIND_CACHE: it is operator-built and there is nothing to download, so an empty answer would otherwise read as "PubMind states nothing here". A snapshot holding no record at the allele is the third state and is reported as an absence in their corpus — no paper survived their triage, which is not a benign call and not a disagreement. Where their own records disagree, every one is reported and none is picked. Unlike the ClinVar leg it also answers for a coordinate the caller typed rather than only for one an rsID resolved to, because their channel is coordinate-keyed and most of its rows carry no rs-number at all.

Every one of those comes back as an Alteration with applied=False and a refusal naming why the value is the author's to type. That is not fastidiousness. resolution._verify compares an authored coordinate against the table, sequences.verify_reference_alleles compares an authored ref against the genome, literature._doi_conflicts compares an authored DOI against the registry — each has force only because the human wrote the value independently of the oracle. Filling the cell from the same source the checker consults makes the check vacuous, and for an rsid-only row _verify does not run at all, so the row would move from honestly unverified to apparently verified. This package had already made the argument for one field: Crossref is asked about the authored DOI because a derived one "exists by construction".

Two operational notes. Clients are injected and reused (LookupClients) because each owns its own PacingGate — gnomAD is one request per six seconds — and a fresh client per question discards both the pacing state and the connection pool. And --offline yields unchecked, never absent: a check that could not run is not a check that passed, and None is not False anywhere in the file.

An unfilled field is filled the same way on every leg (S92, RM206). LookupClients.ensure(name, factory) builds a client under the bundle's lock on first use, stores it and returns it, and close() walks CLIENT_FIELDS — derived from the dataclass, not listed — to close what was built. Until then two legs assigned back onto the bundle and six built a per-request client they closed in a finally, which is exactly the pacing state the paragraph above says to keep; a host filling seven fields and leaving pmc_idconv unset had unpaced egress on one leg and no way to tell from the call site. Ownership follows construction: hold a bundle for a session, and a lookup_* call given none closes the one it built. A hosted CPIC draft is deliberately not a field here — pgx_draft.draft_gene takes client=, so a host shares pacing by holding one CpicClient and passing it.

A cache miss falls through to live Ensembl (0.5), and until it did this surface was silently weaker than the pass it advises on. hint variant --rsid rs1799945 answered "not found in Ensembl, position remains unset" for HFE H63D — which live Ensembl serves at 6:26090951 — because the only thing it had ever searched was a local snapshot that did not contain it. Two things were wrong and both are fixed: the live link (V2 GraphQL → V1 REST) now runs on a miss, in enrich()'s own order so a provisioned snapshot still costs no egress; and the snapshot's own warning stopped speaking for Ensembl, saying "not in the injected Ensembl snapshot" — the thing it actually searched. Naming the searched thing is the difference between "we did not look there" and "it is not there".

That warning kept a second clause it had no standing to write, and RM125 removed it. It read "not in the injected Ensembl snapshot, position remains unset", and the trailing half is a claim about a leg that has not run: online the live link follows and usually fills the coordinate, so lookup_variant returned rs4988235 at 2:135851076 beside a finding saying the position was unset. That is the ordinary three-valued failure — at emission the outcome is genuinely unknown, and the house rule is to withhold rather than guess. Each link now reports only what it searched, and lookup_variant, the one caller that sees both halves, states "{rsid}: position remains unset" once at the end when nothing placed the variant, guarded on there being an rsID at all (a position-only question fills rsid_candidates, never loci). The snapshot miss stays on the record either way, because knowing a local cache is incomplete is what tells an author whether to warm it.

enrich() is untouched by this: it discards both links' warning lists, so lookup_variant was the only reader either sentence ever had. The ClinVar twin needed the same repair and had not had the first one — clinvar.lookup_loci is documented as signature-identical to resolver.lookup_loci, "one implementation, no drift", yet it still said "not found in ClinVar, position remains unset", speaking for the source exactly as the Ensembl half had before it was fixed. Both now say "not in the injected <source> snapshot". A pair described as one implementation is still two strings, and only one of them was reviewed.

A live locus is labelled live. The advisory rows were hard-coded to source="snapshot", which became a lie the moment the live route landed: a network answer claimed to come from a pinned file, in the one field an author reads to judge how reproducible the answer is. They now carry ensembl-graphql/ensembl-rest, and a finding says re-running may differ as Ensembl advances. It is still advisory — a live answer is not a licence to fill a redundancy-bearing cell.

Generation is not automatic

The PGx tables are authored _TABLE_KINDS, not fact sidecars: they carry AuthoredModel semantics, the reserved-namespace guard and raw-byte input hashing. A network pass writing them would blur the authored/derived line the 0.5 rework drew, and hand the human author a file they never wrote but are accountable for. So the automatic pass only ever reads, and scaffolding is an explicit, separate step.

PharmVar and CPIC gotchas

  • The header is Api-Key, no X- prefix, documented per-endpoint in the service's own OpenAPI document (docs/vendor/pharmvar_api_docs.json) rather than in a securityDefinitions block. Every wrong spelling returns the same 401 as no key at all, so a wrong header is indistinguishable from a bad key — the error message says so rather than guessing.
  • The key is personal (PharmVar terms §2), so it comes from PHARMVAR_API_KEY and is never written into a module, fixture, log or snapshot.
  • 2 rps, enforced by the shared net.PacingGate on an injectable clock. The unfiltered /alleles collection is ~25 MB and silently ignores a geneSymbol parameter it does not define; use /genes/{symbol}.
  • CPIC variantallele carries two shapes that are not plain bases, and they are different findings. IUPAC ambiguity codes (R at CYP2C19 *2, Y at *4) are an uncertainty CPIC recorded — expanding R would invent two defining variants where CPIC recorded one — and will never be expressible; they are skipped and reported by the client. Deletion/insertion and repeat notations (DELTCT, INSCGGG, AAAGGGGCG(2), UGT1A1's TA(8)) are spelled by the drafter since 0.8 (RM253), as the VCF-anchored, left-normalized indel every measured caller writes: UGT1A1 *28 is 2:233760233 C>CAT, *36 is CAT>C, *37 is C>CATAT, and the reference count (*1's TA(7)) is written C>C at the same anchor, the reference state, as *1's SNP rows already are. cpic_notation.spell reads the anchor base and the run from the GRCh38 reference through the same SequenceProxy and _left_align the CIViC drafter uses (RM273). Where CPIC's position sits was measured on the snapshot, not assumed: a DEL<bases> position is the first deleted base (every one matches the reference there), an INS<bases> position is the base the insertion follows (NUDT15 *14 is Ensembl's 48037826 G>GCGGG), and a repeat's position is the first base of the run, whose reference count is read, never assumed. Three traps:
  • A DEL beside an INS at one locus is a delins. RYR1 writes c.13994_13995delTCinsCT as DELTC on Reference and INSCT on the variant; read one row at a time, Reference would carry a deletion. Read by locus (cpic_notation.delins_partner), the pair is TC>CT and the DEL half is the reference state — only when the DEL sits on the reference allele (*1/Reference), and judged from every row CPIC states there, never from the rows an --allele filter kept.
  • A repeat's position must be the run's first base. A whole copy of the unit just before it means the position sits inside the run, where a rightward count comes out short; that row is withheld as a disagreement rather than spelled one unit too long.
  • A repeat row carries no rsID. CPIC names UGT1A1's run rs3064744, which no caller writes in its ID column (arrays and dbSNP-annotated VCFs carry rs34983651 for +TA; one imputation server stamps rs606231201, a different allele in dbSNP). The row is keyed by chrom:start:ref. A DEL/INS row keeps CPIC's rsID beside the coordinate: it names one event, and coordinate_verdict leaves an indel's re-anchoring undecided rather than calling it a disagreement.
  • What stays skipped is named by reason (cpic_notation.NOT_SPELLED_EXPLANATION): a bare DEL (no bases; kept skipped by the RM253 decision — on the snapshot it is always the reference allele's row at an insertion site), alternatives (GAGTCG(3) OR GAGTCG(4), MT-RNR1's ; list — an uncertainty, like an IUPAC code), a reference that does not bear the notation out, and a reference that could not be read. --offline reads nothing, so every notation row is withheld with that last reason, and *28 is undefined again until a run with sequence access.

RM5 was not this gap: 0.6 widened the grammar to hold VCF's five symbolic alleles (<DEL:1500>), a deliberately unspelled length, while CPIC's notation names the bases. - HGVS-named alleles are written with spaces dropped and & for the comma (RM253). DPYD's c.1129-5923C>G, c.1236G>A (HapB3) is c.1129-5923C>G&c.1236G>A(HapB3) and c.1003G>T (*11) is c.1003G>T(*11); HAPLOTYPE_NAME_PATTERN (^\S+$) accepts both on all three PGx models with no widening. The comma goes because MULTI_SEP would split it, and & because every other candidate already means something in an allele name (: HGVS's reference separator, + an intronic offset, _ a range, / a diplotype). pgx_draft._draft_name writes a name only when every part is HGVS (c./g./m./n.), plus CPIC's Reference: G6PD's Mediterranean, Dallas, Panama, … separates aliases with its comma, and an & there would claim a compound, so G6PD, CFTR's legacy names, rs… reference (G) and HLA's *57:01 stay skipped. DPYD, RYR1, CACNA1S and MT-RNR1 draft. MT-RNR1's "diplotypes" are single haploid calls, reported in their own bucket rather than as unwritable names. - A large star-allele gene needs --allele (RM34). draft --gene CYP2D6 unfiltered is 16,290 diplotype rows, 73% Indeterminate: faithful, and unreadable. --allele takes the set the consumer's caller can actually emit (n alleles is n(n+1)/2 pairs, so six make 21 diplotypes) and applies it to all three tables at once, because a module that names an allele it never defines is what _cross_validate_haplotype_definitions exists to warn about. *1 is always kept — it is defined by carrying no variants, so it costs nothing, and without it *1/*2 could not be drafted; so is CPIC's Reference where it publishes one, and a name is matched as CPIC spells it or as it is written. An unknown allele refuses with the list CPIC publishes; the flag needs a single --gene, since *2 in CYP2C9 and *2 in CYP2C19 are different alleles. - CPIC activity scores are inequality strings ("≥3.0", "n/a"), not numbers, so they do not drop into MeasureBinRow's numeric bounds; the raw string is carried and the parsing left to a human. - Coordinates are 1-based in both (verified against Ensembl for rs4244285 → chr10:94781859, which PharmVar, CPIC and our own resolution all agree on). Do not convert. - CPIC recommendations are keyed by (gene phenotype, drug, clinical context) and the contexts disagree — and since 0.5 (RM29b) that is no longer a refusal. draft --drug used to stop and list the choices when CPIC scoped a pair to several settings, because DiplotypeRow had nowhere to put the distinction: writing all of them collided on the duplicate-row key, and writing one asserted a clinical setting the author never chose. DiplotypeRow.clinical_context is now part of that key, so every setting is drafted as its own row and the consumer picks — which indication a patient is being treated for is knowable at query time and not at authoring time. Drafted live, --gene CYP2C19 --drug clopidogrel yields 1,998 rows across CVI ACS PCI, CVI non-ACS non-PCI and NVI, and the disagreement is visible in them: *2/*2 Poor Metabolizer is strong in the first and moderate in the other two, with different prescribing text for NVI. --population survives as a filter for an author who wants one setting; an unknown value is still an error, since drafting nothing on a typo would look like "CPIC has no recommendations here". Three of CPIC's sixteen live context values carry trailing whitespace, so the column strips on load — unstripped, 'CVI ACS PCI ' and 'CVI ACS PCI' are two rows describing one setting. recommendation_strength is still CPIC's and evidence_level still PharmGKB's; a provider fills only its own.

Pass 6 — ClinPGx summary annotations (clinpgx.py, offline capable)

pgx.py asks the nomenclature authorities about star alleles over the network; this pass asks ClinPGx about summary annotations — which variant, which drug, at what evidence level — from a local snapshot, exactly as the ClinVar cross-check does. clinpgx_build is the [dev] builder. ClinPGx called these clinical annotations until 2025-07-29 and both names are still in use on its own site; the archive, the members and the id column all say summary now.

The snapshot is read with duckdb, not polars, and that is deliberate: polars is a [dev] dependency here (only the builders need it) while duckdb is core, so reading with polars would leave this runtime pass unusable on a plain pip install just-dna-enricher. clinvar.py reads its snapshot the same way for the same reason — the builder may be dev-only, the pass may not.

The snapshot pins its own licence. ClinPGx ships a LICENSE.txt inside summaryAnnotations.zip, so the builder extracts it, records its sha256 in release.json, and the pass stamps that hash onto the SourceRow. The recorded terms are provably the ones shipped with the recorded data — the property a static source→licence map cannot offer.

The snapshot's grain is (annotation, genotype), joining summary_annotations.tsv to its per-genotype child summary_ann_alleles.tsv. CREATED_<date>.txt is the release id, because ClinPGx publishes no version number and does not refresh its archives in lockstep — relationships.zip was a year newer than clinicalAnnotations.zip when this was written, which RM175 later explained: that archive had stopped being rebuilt at all.

The archive is identified before it is read, and the retired spelling is refused (RM175). ClinPGx renamed clinical annotations to summary annotations on 2025-07-29 and the archive followed — clinical_annotations.tsv → summary_annotations.tsv, clinical_ann_alleles.tsv → summary_ann_alleles.tsv, Clinical Annotation ID → Summary Annotation ID, with the evidence and history siblings renamed the same way (this builder reads neither). clinicalAnnotations.zip was last written on 2025-07-05, is on no downloads page, and the API still answers it 200 through a 303 to that frozen object — so until this item every snapshot the lane built came out of a database fourteen months old, and nothing in the download said so. require_current_archive therefore reads the archive's member names first and answers in three arms: the current spelling builds, the retired one is refused with the rename, its date, the retired filename and the URL to use instead, and an archive that is neither says that instead. just-dna-enricher clinpgx build prints the refusal as CLINPGX BUILD FAILED: … and exits 1.

A build that downloads also asks how old the archive is against its siblings (RM283, 0.8). The refusal above catches the retired name; nothing caught the current archive itself going quiet. probe_archive_dates sends HEAD to the lane's archive and to SIBLING_ARCHIVES (variantAnnotations.zip, clinicalVariants.zip, from RM166's inventory) on the same host, following the 303 to S3, and build_snapshot records every Last-Modified in release.json under one key, archive_last_modified (name → ISO date, null for an archive whose HEAD did not answer). When the lane's archive trails the newest sibling by STALE_AFTER_DAYS (180) or more, the build logs one warning, cache rebuild appends it to the lane's detail and clinpgx build prints it; it never refuses. A sibling that did not answer is named and never treated as old. The date is the S3 object's, so it sits beside dataset and is never the label, which stays CREATED_*.txt. A build from --zip asks nobody and writes no key. Auditing each default URL against the downloads page stays a manual rebuild step, since every ClinPGx HTML route is a JavaScript shell.

Nothing about the table itself moved: the other fourteen columns are identical in name and order, and Phenotype Category carries the same values with the same ; separator, so no vocabulary member, no model field and no parquet column changed. The data moved, which is the point — 16,087 → 16,117 snapshot rows, 22 (annotation, genotype) keys gone and 52 new, 30 rows changing evidence_level and every shared row rehosting its URL from pharmgkb.org to clinpgx.org — so a module drafted from this lane can now see an evidence level move under it. That is the check working, not a regression.

The cross-check keys on the annotation, not the triple, and that is a bug fix rather than a nicety. (rsid, drug, genotype) is not unique: rs4149056 + simvastatin is Metabolism/PK at 1A, Efficacy at 3 and Toxicity at 1A. The first implementation indexed on the triple and reported all three of the reference example's correctly-authored levels as stale. The lookup is now annotation_id → (rsid, drug, genotype, category) → the bare triple, and when the bare triple matches several annotations at different levels the row is reported as unchecked rather than compared against an arbitrary one.

Severity follows the mode ladder, unlike the allele-function check beside it. An evidence level is ClinPGx's own metadata about its own annotation, so a difference means the module is stale — not that two expert panels disagree.

That holds only for a row compared with the annotation it cites (RM297, from S122). Nothing marks a row as ClinPGx-derived, and annotation_id is the source's own accession whatever the source. So two cases are reported in both modes and never refused under strict, in ClinPgxResult.withheld:

  • clinpgx_annotation_not_in_snapshot: the row cites an id the snapshot does not hold. The lookup stops there instead of falling through to a neighbour sharing the category. The warning names three readings (ClinPGx withdrew it, it is mistyped, it was never a ClinPGx accession) and what the snapshot holds at the row's triple.
  • clinpgx_level_differs_from_uncited_annotation: the row cites no id, and the annotation its category or triple reached carries another level.

A cited id that is present but lacks the row's genotype keeps the fall-through and can still refuse. Both codes are lane-local, not VALID_WARNING_CODES members. They are counted in the record's findings and named in its detail, so an unchanged module publishes the same number it did before. Knowing per row that a row cites ClinPGx is RM298's.

The declared-use gate still applies even though nothing is fetched: the terms were accepted when the snapshot was built, and using it is the same act.

del/del is skipped for a different reason since 0.6, and the old one became false. clinpgx_draft used to report ClinPGx's structural genotypes as something the format could not spell. RM5 widened the grammar — <DEL:1500> is authorable now — so the block moved: ClinPGx publishes no length, and a lengthless symbolic allele is a rule the compiler drops. The pass therefore still declines to write those rows, because a provider must not hand an author work the next command in the documented workflow undoes, and the warning now names the length rather than the grammar. _CLINPGX_SYMBOLIC maps the source's dialect (del → DEL) and lives here, at the boundary, not in the schema — the CC → C/C rule beside it is the precedent, and a grammar that accepted every source's spelling would owe every consumer the union of them.

But the rule beside it was narrower than the schema it writes into, and that cost two whole families (S44). _authored_genotype took only CC — two unseparated bases — on the argument that the general case needs the resolved ref/alt to disambiguate. True of an unseparated cell; false of the two shapes ClinPGx actually publishes beside it, and validate_allele accepts any ^[ACGT]+$ allele:

  • An already-separated call, CTT/CTT. ClinPGx writes / wherever an allele runs past one base, so the source has already made the split and no decision arises. Declining it cost CFTR F508del among others — those annotations carry a del-spelled genotype and a pure-nucleotide one under the same annotation_id, so dropping the annotation for the first discarded the second with it. A module shipped 176 CFTR rows and the drug elexacaftor / tezacaftor / ivacaftor while omitting the most common CF variant.
  • A single haploid allele, A / CCCCCCC. The hemizygous/homoplasmic form the grammar already holds, and how ClinPGx spells an mtDNA call. Declining it cost every MT-RNR1 annotation — 24 annotations, 48 rows, 32 of them at evidence level 1A: aminoglycoside-induced hearing loss, a CPIC guideline. The reasoning is clinvar_draft.sole_expressible_genotype's, one source over: the placeholder protects a zygosity decision, and on a haploid contig there is none to protect.

Measured on the provisioned snapshot: 158 rows recovered, 36 of them at 1A, across MT-RNR1 (48), HTR2C, ACE, TYMS, IFNL4, GSTM3 and CFTR. The unseparated multi-base cell the original rule guarded against does not exist in the snapshot — every multi-base call arrives slashed — and it is still declined. The general rule is now a test rather than a comment: every spelling this pass declines must be one PharmVariantRow would also refuse, walked over the accepted set, so a provider can never again be narrower than the format. The converse stays allowed, which is what keeps del/del skipped.

And the terms are pinned now. SourceTerms.row has taken license_text= all along; this caller passed only declared_use and dataset, so a share-alike source was recorded with a null license_sha256 — the module named ClinPGx's licence without tying it to the text that governed the bytes, which is the one thing that field exists for. clinpgx_build already extracts LICENSE.txt beside the parquet, so the fix is to read it. Hashed from the file rather than copied from release.json's stated hash: the file is what the module is claiming, so a truncated copy cannot pin to a value it does not have. Absent stays None and warns — an older snapshot predates the extractor, and inventing a hash would be worse than the null it replaces.

Exception contract — what a caller catches (RM97 + RM101)

Two layers, and a caller only ever touches the outer one. A client raises its own error type, never httpx's (RM97). A pass raises its own type, never the client's (RM101). A consumer calls a pass, so the pass's type is the one to write in an except; before RM101 five call sites let the client's type through and the documented handler was silent for exactly the failure it was written for.

you call catch and for "the source could not be reached"
enrich_frequencies FrequencyEnrichmentError FrequencyUnavailable
enrich_literature LiteratureEnrichmentError LiteratureUnavailable
enrich_gene_metrics GeneMetricsEnrichmentError GeneMetricsUnavailable
check_rsids / check_identifiers IdentifierCheckError IdentifierUnavailable
enrich_dosage_sensitivity ClinGenError ClinGenUnavailable
enrich_gene_validity GeneValidityError GeneValidityUnavailable
verify_acmg_sf AcmgSfError AcmgListUnavailable (carries skip)
enrich_gwas GwasError — client and pass share the type; a 410 is GwasRetired
enrich_pgx PgxEnrichmentError — degrades per leg instead of raising
enrich() — — degrades and withholds; see below

Three layers, really, and the third is where two clients were wrong (RM208). Inside a client the retrying half is its own function and the translation sits outside it — eutils._request/_get, cpic._request, gnomad._request, and since RM208 literature.CrossrefClient and gwas.GwasCatalogClient too. Get that order backwards and one of two things happens, and the tier had one of each:

  • Translate inside the retry and the retry never runs. CrossrefClient.exists caught httpx.HTTPError — the superclass of the two types its own @retry matched — so a ConnectError became the None withhold before tenacity saw it. Measured: one upstream request where attempt_floor(3) asked for three, and the knob a deployment raises moved nothing.
  • Re-raise for the decorator and translate nowhere and the last attempt leaks. GwasCatalogClient re-raised the transport leg bare, correctly, and reraise=True then handed the raw httpx.ConnectError to a caller told to expect GwasError.

A permanent 4xx is never retried, and a 410 is the source moving (RM354). Found when the GWAS Catalog retired its legacy API: every path answered HTTP 410 and GwasCatalogClient filed it as the same GwasError an outage raises, so a source that had moved read as one that was down. The class was wider than that lane. No client named a 410, and six (gnomad, eutils, cpic with its row_count, pharmvar, currency) retried every HTTPStatusError, so a 400, a 404 and a 410 were each asked three or four times first. Since RM354:

  • Retried: a transport failure, a timeout, a 5xx and a 429 — net.retry_if_transient, which the six now use, and so does gwas, whose v2 pass makes one request per page and per study (one passing 503 used to sink all of them). The other clients that retried transport only keep that predicate. test_no_retry_policy_in_the_tier_retries_a_permanent_4xx hands every @retry policy in the package a synthetic 400, 404 and 410 and refuses any that would try again.
  • A 410 raises the client's own *Retired, a subclass of its could-not-ask type, so every existing handler still catches it. The message carries the URL, the status and the source's own text, which is usually where it says the endpoint went. net.raise_if_retired is the one place the message is built. A handler records the reason with net.could_not_ask_reason(exc) (RM361): retired for a 410 (every *Retired carries net.EndpointRetired), unreachable for an outage, because the two are cleared by different things (an upgrade, a re-run). An AST walk refuses an except arm that writes an unreachable reason without it, an f-string opening with the word included. The owners are walked by what they call: every module-level class and function that opens an httpx transport names its 410 (a function-style fetcher such as clingen.fetch_curation_list raises its lane's *Retired too; audit round 2). A client whose contract is a withheld value rather than an exception mixes in net.Withholding, keeps the withhold, and says retired through its sticky retired message (net.withheld_reason). Sticky for the client, never for the run (RM446): a caller may reuse one client across runs (RM404's LookupClients), so every 410 is also counted in retired_met. A run reads net.retired_mark(client) before it asks and records withheld_reason(client, since=mark), so an earlier run's 410 never tells this run's timeout to upgrade. enrich's live-Ensembl warning and literature's crossref_unanswered read it so. The client is never reset, since a caller's client is not the run's to alter. A host's own client with no counter reads as before.
  • What a 404 means stayed per client, and it is not the same everywhere:
client a 404 is so
identifiers.OntologyClient an answer: no such term / symbol absent
gwas.GwasCatalogClient.follow an answer: no such study record that row's ancestry withheld
gwas.GwasCatalogClient.associations_for a failure since v2 — the search answers an unknown rsID with an empty 200 GwasNotFound, a GwasError
literature.EuropePmcClient.fulltext no retrievable fulltext None, the withhold
literature.EuropePmcClient.lookup a failure LiteratureUnavailable
pgs.PgsCatalogClient a failure — the Catalog says "no such score" with an empty 200 PgsCatalogUnavailable
litvar.LitvarClient a failure — its absence is a 400 whose body opens Variant not found LitvarUnavailable
gnomad, eutils, cpic, pharmvar, currency, civic_api a failure the client's base or *Unavailable type

test_retry_is_reachable.py walks every @retry-decorated function in the package and refuses any whose body catches an ancestor of a type its decorator retries. It walks the package rather than the client roster on purpose: both offenders sat in test_client_exception_contract.py's exempt set, and a guard that iterates a roster inherits the roster's exemptions.

alphagenome check raises VariantImpactError / VariantImpactUnavailable (RM193), the pass half of the pair above — a consumer calls the pass, so this is the type to write in an except. Note that a transport failure inside it is neither: the pass records the variant as unreachable and produces no finding, because a service that did not answer has said nothing about the caller's data.

The Atlas client has its own ladder, and it is a client rather than a pass (RM192). It is listed apart because its third arm is not a failure at all:

what happened type remedy
transport failed AtlasUnavailable retry later: the client has already spent attempt_floor(5) attempts (RM280)
REF disagrees with GRCh38 AtlasRefMismatch fix the caller's data — the server names the real base, which @va-omits-ref says only this tier can discover
an indel AtlasNotScored none: the answer does not exist
a quantile saturated at 1.0 AtlasNotScored read PHRED from the downloaded artifact instead

AtlasNotScored deliberately does not derive from AtlasRefused, so an except AtlasRefused cannot swallow it — a caller that recorded "no score" for a variant the service never claimed to have scored would be @unreachable-not-absent in one line. AtlasRefMismatch is a subclass of AtlasRefused, so the same handler-order rule below applies to it, and test_handler_order_is_not_load_bearing_by_accident enumerates the ladder rather than leaving it to a reviewer.

Every *Unavailable is a subclass of the type beside it, so except <Pass>Error keeps catching everything it did (P3, additive within a major) and the narrower catch is new capability rather than a migration. The client's exception is chained, so __cause__ still carries it — but it is no longer the only way to tell the two apart, which is what the split is for.

Being a subclass makes handler order load-bearing, and that is the one thing the split costs. Two separate except arms with the parent first leave the narrow arm dead — Python takes the first matching clause — so a caller who wrote except <Pass>Error above except <Pass>Unavailable reports an outage as an ordinary failure and never sets whatever an outage sets. It fails silently: nothing raises, and a test asserting a status code sees a clean run. One tuple is unaffected, since both names route to the same block. just-dna-registry lost three of four handlers this way on 0.6.2 (S38), which is what enricher/tests/test_shadowed_handlers.py now walks the workspace for.

What the distinction means. The subclass says the source was asked and never answered — nothing was established either way, so a caller records unchecked rather than a negative (@unreachable-not-absent). The plain parent means the question was put and the answer is a problem: a strict run with a variant that genuinely has no frequency, or a local sidecar that will not parse. ClinGenError and GeneValidityError covered both histories with one type until RM101, and the only way to separate them was reading __cause__ — chained for the fetch, bare for the table.

Do not separate them by message. Neither string is in the pinned warning-text catalogue, so a reword would silently flip a verdict from "unchecked" to "your table is broken". That is the argument just-dna-registry made in S37 against their own first option, and it is why this is a type question.

Two passes deliberately do not raise at all. enrich() catches GnomadError on its last-resort gnomAD link ("a last-resort link must not sink the whole enrichment") and answers an unreachable rsID with None from the Ensembl leg; enrich_pgx degrades per leg so one dead source does not take the other's answer down. Both are the tri-state withhold, which is the correct shape where a pass has already produced work worth keeping — the opposite end of the same rule, not an exception to it.

If you are adding a pass or a client, enricher/tests/test_client_exception_contract.py and test_pass_exception_contract.py both discover by walking the package rather than by a list, and will fail naming your addition until it is covered or explicitly exempted. That is deliberate: RM97's guard walked a hand-written tuple of eight module names, identifiers was not one of them, and OntologyClient leaked raw httpx for a whole release as a result.

The complete roster — every error type this tier defines (RM216)

The table at the top of this § is the one a consumer usually needs; this one is the registry. Fifty-one of the tier's error classes were named nowhere in this document, in a section titled what a caller catches — so a consumer meeting ClinPgxUnavailable or GatedSnapshotError had nothing to look it up in. Walked by test_enricher_doc_registries.py against the package, so a new error type joins this table by existing (@registry-completeness).

Read the third column as a ladder, not a set. A subclass makes a caller's except order load-bearing (@client-exception-contract), so catching the base type first silently swallows every narrowing beside it. Two levels deep: AtlasRefMismatch under AtlasRefused, and since RM354 every *Retired that sits under its client's *Unavailable.

The *Unavailable convention is the tri-state, spelled as a type: it means the source could not be asked, which is unknown and never a finding against the module. A base type with no narrowing either has one failure mode or answers with a withholding value instead of raising — Grch37Client and EnsemblResolver are the latter, and are exempt from the contract suite for exactly that reason.

Runtime passes — the type a consumer calling a pass writes in its except

module base type narrowed by
acmg AcmgSfError AcmgListUnavailable, AcmgRetired (under the one before it; a 410, RM361)
alphagenome_check VariantImpactError —
assertions ClinicalAssertionError —
civic_citations CivicCitationsError —
clingen ClinGenError ClinGenUnavailable, ClinGenRetired (under the one before it; a 410, RM361)
clinpgx ClinPgxEnrichmentError —
clinvar_gkm ClinVarGkmError ClinVarGkmUnavailable (could not be asked), ClinVarGkmRetired (under the one before it; RM354), ClinVarGkmMismatch (bytes that are not the file the manifest states; RM305)
currency ReleaseProbeError ReleaseUnavailable, ReleaseRetired (under the one before it)
drug_labels DrugLabelError DrugLabelUnavailable, DrugLabelRetired (under the one before it; a 410, RM361)
enrich EnrichmentError —
expression ExpressionError ExpressionUnavailable
frequencies FrequencyEnrichmentError FrequencyUnavailable
gene_metrics GeneMetricsEnrichmentError GeneMetricsUnavailable
gene_validity GeneValidityError GeneValidityUnavailable, GeneValidityRetired (under the one before it; a 410, RM361)
gwas GwasError GwasNotFound, GwasRetired
identifiers IdentifierCheckError IdentifierUnavailable, IdentifierRetired (under the one before it)
ld_pairs LdError LdUnavailable, LdRetired (under the one before it)
literature LiteratureEnrichmentError LiteratureUnavailable, FulltextUnavailable (RM258: the fulltext clients' unanswered request), FulltextRetired (under the one before it; a fulltext route's 410, RM361), LiteratureRetired (under LiteratureUnavailable)
opentargets OpenTargetsError — (it never fetches)
litvar LitvarError LitvarUnavailable, LitvarRetired (under the one before it)
mitomap MitomapError MitomapUnavailable, MitomapRetired (under the one before it; a 410, RM361)
pgx PgxEnrichmentError —
strchive StrchiveError StrchiveUnavailable, StrchiveRetired (under the one before it; a 410, RM361)

Three surfaces raise a declared family instead of one type (RM288, 0.8). cpic_build.ERRORS, pharmvar_build.ERRORS and pgx_draft.ERRORS are tuples the surface owns, and a caller writes except cpic_build.ERRORS (and so on) rather than listing the members, so a new member reaches every caller and except order never matters. RM310 was the list shape failing: cache rebuild named CpicBuildError and not CpicError. pgx_draft.ERRORS includes LicenseRefusal, which used to escape draft --use commercial as a traceback.

Clients — the transport layer, which a consumer calling a pass never meets

module base type narrowed by
atlas_client AtlasError AtlasNotScored, AtlasRefused, AtlasRefMismatch (under the one before it), AtlasUnavailable, AtlasRetired (under the one before it; UNIMPLEMENTED on anything but an indel, never retried, RM361)
net EndpointRetired — (a marker every *Retired mixes in, read by net.could_not_ask_reason, RM361)
civic_api CivicApiError CivicApiUnavailable, CivicApiRetired (under the one before it)
clingen_allele ClingenAlleleError —
cpic CpicError CpicRetired
ensembl EnsemblError —
eutils EutilsError EutilsRateLimitedError, EutilsRetired
gene_spans GeneSpanError —
gnomad GnomadError RateLimitedError, GnomadRetired
pgs PgsCatalogError PgsCatalogUnavailable, PgsCatalogRetired (under the one before it)
pharmvar PharmVarError PharmVarRetired

Builders and the publisher ([dev]) — what cache rebuild and a build command raise

module base type narrowed by
alphagenome_avi_build AlphaGenomeBuildError —
atlas_protos ProtoFetchError —
civic_build CivicBuildError CivicUnavailable, CivicRetired (under the one before it; a 410, RM361)
civic_vcf CivicVcfError —
clinpgx_build ClinPgxArchiveError ClinPgxUnavailable, ClinPgxRetired (under the one before it; a 410, RM361)
clinvar_build ClinVarBuildError ClinVarUnavailable, ClinVarRetired (under the one before it; a 410, RM361)
constraint_build ConstraintBuildError ConstraintUnavailable, ConstraintRetired (under the one before it; a 410, RM361)
cpic_build CpicBuildError —
mane_build ManeBuildError ManeUnavailable, ManeRetired (under the one before it; a 410, RM361)
opentargets_build OpenTargetsBuildError OpenTargetsUnavailable, OpenTargetsRetired (under the one before it; a 410, RM361)
pubmind_build PubMindBuildError PubMindUnavailable, PubMindRetired (under the one before it; a 410, RM361)
upload PublishRefused (RM368: every refusal a publish raises) ModuleNotPublishable, SnapshotNotPublishable (both still a FileNotFoundError within 0.8), PlanChangedError, OrphanedSidecarError, RemoteMovedError (RM386: the repo moved after the plan read it), PublishCollisionError → ToolchainDiffersError (RM342: the two recorded toolchains differ)

Snapshot readers — an absent or unreadable cache, deliberately a FileNotFoundError

module base type narrowed by
clinvar ClinVarReferenceError —
download ConstraintReferenceError —
download GatedSnapshotError —
download OpenSnapshotError —
download SnapshotNotPublished —
pubmind PubMindReferenceError —
resolver EnsemblReferenceError —

Drafting providers, and the licence gate

module base type narrowed by
civic_draft CivicDraftError —
clinvar_draft ClinVarDraftError —
licensing LicenseRefusal —
mitomap_draft MitomapDraftError —
pubmind_draft PubMindDraftError —
strchive_draft StrchiveDraftError —

CLI

just-dna-enricher enrich spec/ --strict            # write spec/resolution.csv, fail if unresolved
just-dna-enricher enrich spec/ --offline           # cache-only (Ensembl + ClinVar), zero egress
just-dna-enricher enrich spec/ --no-clinvar        # Ensembl links only
just-dna-enricher enrich spec/ --no-verify-ref     # skip the reference-allele check
just-dna-enricher enrich spec/ --no-verify-clinsig # skip the ClinVar clin_sig cross-check
just-dna-enricher enrich spec/ --no-verify-rsids   # skip the dbSNP merge/withdrawal check
just-dna-enricher enrich spec/ --no-verify-datasets # skip the recorded-release currency check
just-dna-enricher enrich spec/ --pubmind-cache pm/ # add PubMind as a second authority to the
                                                  # clin_sig check; without it that leg is unchecked
just-dna-enricher enrich spec/ --keep-par-twin   # record both contigs of a pseudoautosomal locus
just-dna-enricher enrich spec/ --rederive          # re-ask every recorded subject; report what moved
just-dna-enricher enrich spec/ --keep-staging      # keep the staged answers after a successful commit
just-dna-enricher literature spec/                 # pass 4: write spec/literature.csv (online only);
                                                   #   reads studies.csv AND any binning row's pmid
just-dna-enricher literature spec/ --no-fulltext   # existence + identifiers, skip the quote match
just-dna-enricher check-identifiers spec/          # trait CURIEs (OLS4) + gene symbols (HGNC) + gene/chromosome agreement
just-dna-enricher litvar coverage spec/            # RM167: which papers LitVar holds per locus, and at which tier
just-dna-enricher litvar coverage spec/ --offline   # every locus recorded as unchecked, nothing asked
just-dna-enricher litvar gene HFE                   # every node under a gene, split by tier (rsID / CAID / gene / mention)
just-dna-enricher dosage spec/ --offline           # no-op with a warning (ClinGen has no snapshot)
just-dna-enricher gene-validity spec/              # RM24: ClinGen expert-panel gene-disease assertions
just-dna-enricher gene-validity spec/ --source gencc  # …or GenCC's aggregate of nineteen submitters
just-dna-enricher assertions spec/                 # RM25: ClinVar's call + review tier per allele
just-dna-enricher assertions spec/ --offline       # snapshot only; no snapshot → no-op with a warning

# Caches — provision once, then every gated pass runs with zero egress. See "The caches".
just-dna-enricher cache status                     # what is present, where, which release
just-dna-enricher cache pull                       # the ungated published lanes, from HuggingFace
just-dna-enricher cache pull --use non-commercial  # …and ClinPGx, CPIC and the drug labels, which forbid sale
just-dna-enricher cache pull --only clinvar
just-dna-enricher cpic build --use non-commercial                 # whole CPIC → data/repro/cpic ([dev])
just-dna-enricher cpic publish cpic/ --repo org/cpic              # redistribution is granted
just-dna-enricher clinpgx publish cp/ --repo org/clinpgx          # LICENSE.txt travels with it
just-dna-enricher pharmvar build --use non-commercial             # YOUR key; never published
just-dna-enricher pubmind build --download                       # literature verdicts; never published
just-dna-enricher pubmind publish                                # exits 1 and says why (by design)
just-dna-enricher mane build --download                           # the transcript numbering frame
just-dna-enricher pgx spec/ --offline --use non-commercial        # both legs off snapshots
just-dna-enricher pgx spec/ --cpic-cache cpic/ --pharmvar-cache pv/
just-dna-enricher draft spec/ --gene CYP2C9 --offline --use non-commercial
just-dna-enricher acmg build assets/acmg_sf_v3.3.xlsx               # once: the SF v3.3 snapshot
just-dna-enricher check-acmg spec/ --sf-list acmg/  # acmg_sf vs the ACMG SF gene list (offline-capable)
just-dna-enricher strchive build --release v2.26.0                  # the repeat-locus catalogue
just-dna-enricher check-repeat-bands spec/ --catalogue strchive/  # repeat_alleles.csv bands vs STRchive (reports only)

# Authoring — templating and drafting (the compiler owns the offline half; see COMPILER.md)
just-dna-enricher template repeat_alleles.csv       # header + required/one-of/never-empty defaults
just-dna-enricher draft spec/ --gene CYP2C19        # CPIC → haplotypes/allele_function/diplotypes
just-dna-enricher draft-repeats spec/ --catalogue strchive/  # STRchive → repeat_alleles.csv identity rows
just-dna-enricher draft spec/ --gene CYP2D6 --allele '*1' --allele '*4' --allele '*10'  # 21 not 16,290
just-dna-enricher draft spec/ --gene CYP2C19 --drug clopidogrel   # + every clinical context, as rows
just-dna-enricher draft spec/ --gene CYP2C19 --drug clopidogrel --population NVI  # one context only
just-dna-enricher draft-clinpgx spec/ --snapshot cp/ --drug simvastatin --use non-commercial
just-dna-enricher draft-panel spec/ --gene MTHFR --gene BRCA1   # ClinVar gene panel (snapshot auto)
just-dna-enricher draft-panel spec/ --gene MTHFR --snapshot cv/ --offline   # a snapshot you built
just-dna-enricher draft-panel spec/ --gene MTHFR --no-download   # use a cached snapshot; fetch none
just-dna-enricher draft-panel spec/ --gene MTHFR --dry-run   # the genotype worklist, appending nothing
just-dna-enricher draft-panel spec/ --gene BRCA1 --source pubmind --pubmind-cache pm/  # literature verdicts
just-dna-enricher draft-panel spec/ --gene BRCA1 --source pubmind --min-confidence 2   # a deeper floor
just-dna-enricher mitomap build                 # MITOMAP's pg_dump → both curated mtDNA tables (data/repro/mitomap)
just-dna-enricher mitomap miss                  # the join against ClinVar chrMT; both parents required
just-dna-enricher draft-panel spec/ --source mitomap-miss   # the increment only; --gene filters, never required
just-dna-enricher clinvar citations --out data/repro/clinvar --download   # add PMIDs so a panel can compile (--out names the EXISTING snapshot)
just-dna-enricher clinvar publish data/repro/clinvar                     # data/ + citations/ + release.json

# Authoring — lookups. These WRITE NOTHING: every answer comes back advisory, with a reason.
just-dna-enricher hint variant --rsid rs1801133              # validity, loci, ref/alts
just-dna-enricher hint variant --rsid rs334 --ambiguity      # warn when the answer is not unique
just-dna-enricher hint variant --rsid rs1801133 --frequencies  # + gnomAD populations (paced ~6s)
just-dna-enricher hint variant --rsid rs1801133 --offline --json
just-dna-enricher hint variant --chrom 17 --start 43093220 --ref C --alts T --pubmind-cache pm/
just-dna-enricher hint citation --pmid 9545397               # which paper it is + the DOI/PMC id it carries (--json)
just-dna-enricher hint citation --pmcid PMC3110566           # the PubMed id for a PMC id — reported, never written
just-dna-enricher hint trait EFO_0004340                     # current | obsolete | absent
just-dna-enricher hint gene MTHFR                            # approved | retired | unknown
just-dna-enricher enrich-and-compile spec/ out/    # enrich, then compile from resolution.csv (offline)
just-dna-enricher upload out/coronary --dry-run    # rehearse a module HF upload ([dev]); names both paths, reads the published version
just-dna-enricher upload out/coronary              # push to data/coronary/ and data/coronary/v<version>/
just-dna-enricher clinvar build --vcf clinvar.vcf.gz            # VCF → snapshot parquet ([dev]), data/repro/clinvar
just-dna-enricher clinvar build --download                      # fetch the NCBI VCF first, then build
just-dna-enricher clinvar publish data/repro/clinvar --dry-run  # rehearse the reference-snapshot upload
just-dna-enricher clinvar publish data/repro/clinvar            # create-or-update datasets/just-dna-seq/clinvar

enrich takes the mode and cache flags (--strict/--best-effort, --offline, --ensembl-cache, --clinvar-cache), the per-link toggles (--clinvar/--no-clinvar, --gnomad/--no-gnomad, --vrs/--no-vrs), the four verify toggles above (--verify-ref, --verify-clinsig, --verify-rsids, --verify-datasets), --pubmind-cache, --keep-par-twin, and the two transaction flags --rederive / --keep-staging. enrich-and-compile takes the mode and cache flags, the ClinVar and gnomAD link toggles, and the --frequencies / --gene-metrics pass toggles — not --vrs, not the verify toggles, not the transaction flags — it is the offline convenience wrapper, so those are enrich's alone. That distinction is the rule worth writing down; the enumeration is not, because a flag list restated in prose rots the moment a flag is added, and --help is the authority. literature takes --strict/--best-effort, --offline, --fulltext/--no-fulltext; check-identifiers takes --strict, --traits/--no-traits, --genes/--no-genes, --pgs/--no-pgs (RM163) and, like check-acmg, writes no authored cell and records that the question was put: there is no sidecar column for a module-level identifier and filling one from the registry being asked about it would make the comparison vacuous (hints.REDUNDANCY_BEARING), so the report is the whole output apart from the verification.json attestation — five records for check-identifiers (the two PGS ones since RM163), one for check-acmg, unconditional, and an attestation rather than a value; upload takes --repo, --name, --message, --dry-run; clinvar build takes --vcf/--download/--out; clinvar publish takes --repo, --message, --dry-run. enrich-and-compile runs enrich then compile_module(..., ensembl_cache=None, strict=…), so compilation consumes the just-written resolution.csv (path 1) with no reference and no network.

Command → Python API

Every command is a thin shell over a library function; nothing here is CLI-only logic. Use the API directly to compose passes, inject clients, or run in-process.

Command Python API
enrich enrich.enrich
frequencies frequencies.enrich_frequencies
gene-metrics gene_metrics.enrich_gene_metrics
dosage clingen.enrich_dosage_sensitivity
litvar coverage litvar.check_literature_coverage + litvar.verification_records
litvar gene litvar.LitvarClient.gene_nodes
literature literature.enrich_literature
pgx pgx.enrich_pgx
check-identifiers identifiers.check_identifiers
check-acmg acmg.verify_acmg_sf (+ AcmgReport.by_gene for the grouped view)
check-repeat-bands strchive.check_repeat_bands
cache status / cache pull / prepare / rebuild locations.resolve_*_reference / download.ensure_*_snapshot / caches.prepare_caches / caches.rebuild_lane (+ publish_reference_snapshot under --publish)
cpic build / publish cpic_build.build_snapshot / upload.publish_reference_snapshot
pharmvar build pharmvar_build.build_snapshot (no publish — see The caches)
pubmind build / publish pubmind_build.download_pubmind_table + build_snapshot / refuses, cli.PUBMIND_PUBLISH_REFUSAL
mane build mane_build.discover_current_release + download_mane_file + build_snapshot (no publish — see The caches)
opentargets build / enrich opentargets_build.download_release + build_snapshot (published through cache rebuild --publish, RM408) / opentargets.enrich_opentargets
acmg build acmg_build.build_acmg_snapshot → acmg.load_acmg_snapshot
draft pgx_draft.draft_gene
draft-clinpgx clinpgx_draft.draft_pharm_variants
draft-panel clinvar_draft.draft_gene_panel / pubmind_draft.draft_gene_panel_from_pubmind / civic_draft.draft_panel_from_civic / mitomap_draft.draft_panel_from_mitomap_miss, by --source
mitomap build / miss / publish mitomap_build.download_mitomap_dump + build_snapshot / mitomap_miss_build.build_miss_snapshot (+ stale_parents) / upload.publish_reference_snapshot
draft-repeats strchive_draft.draft_repeat_loci
strchive build strchive_build.build_strchive_snapshot → strchive.load_strchive_catalogue
clinpgx build / clinpgx check clinpgx_build.download_clinpgx_zip + build_snapshot / clinpgx.enrich_clinpgx
clinpgx build-labels / check-labels drug_labels_build.download_drug_labels_zip + build_drug_label_snapshot / drug_labels.check_drug_labels
civic build / citations / publish / reproduce civic_build.build_snapshot / civic_citations.draft_civic_citations (+ check_evidence_status_currency, folded into enrich) / upload.publish_reference_snapshot (CC0; RM176) / the CLI's own checks
clinvar build / citations / publish clinvar_build.download_clinvar_vcf + build_snapshot / download_var_citations + build_citations / upload.publish_reference_snapshot
gnomad constraint build / publish constraint_build.download_constraint_tsv + build_snapshot / upload.publish_reference_snapshot
vrs mint vrs.mint_resolution_rows
hint variant / citation / trait / gene lookup.lookup_variant / lookup_citation / lookup_trait / lookup_gene
upload upload.plan_upload / upload_module
template just_dna_compiler.draft.blank_template (cross-package convenience)
enrich-and-compile enrich.enrich → just_dna_compiler.compiler.compile_module

Two library functions have no command, on purpose. clinical.verify_clin_sig and sequences.verify_reference_alleles are checks that run inside enrich (--verify-clinsig, --verify-ref), because their verdicts land on resolution.csv and on the enrichment report. Running either standalone would compute a finding with nowhere to put it.

And two take rows rather than a spec_dir — both now take either (RM41, 0.5.1). acmg.verify_acmg_sf and identifiers.check_identifiers are the only passes whose input is a list of VariantRow, which left every caller turning variants.csv into models itself — and the only thing that does that correctly is just_dna_compiler.load_csv_rows, which was private until 0.5.1. Re-implementing it is a trap rather than a chore: an empty cell becomes None with the key kept (so a defaulted-but- not-Optional field like MeasureBinRow.measure_kind fails on type rather than taking its default), and genome_build is told to each row, so a loader that omits it mints GRCh38 identities for a GRCh37 module. compiler.load_spec_variants(spec_dir) does the load, the injection and the _restamp_for_build in one call, and both checks accept spec_dir= alongside the existing variants=. Exactly one, never both — two answers in mind, and silently preferring one is a guess.

Since 0.5.4 the two forms are no longer equivalent for check_identifiers, and the report says which you got: the gene ↔ locus comparison needs a chromosome per row, and an rsID-only row has one only after resolution, so spec_dir= lets it read the injected resolution.csv beside the spec while variants= alone limits it to authored coordinates. Where that leaves nothing to compare, gene_loci_not_checked names the reason rather than reporting a clean zero.

Since 0.7 the roster is every authored table carrying the column, and only spec_dir= can reach them (S86). check_identifiers built its roster from variants.csv alone while eleven authored models declare trait_efo_id or gene — StudyRow has carried the trait column since 0.3 — so a module whose traits live in studies.csv reported nothing checked and nothing flagged, and could ship a retired CURIE with every gate green. The set is derived from DRAFTABLE rather than listed (@registry-completeness): every table carrying trait_efo_id or gene is in it, and a table kind added later joins by existing. The three derived models that also carry these columns — GeneMetricsRow, GeneValidityRow, GwasEffectRow — are deliberately outside it: they are machine-written, so a stale id in one is the source's currency and no author can act on it.

The unreadable 0 was the item, not the omission. traits checked: 0 said this module declares no trait and its traits are in a table nobody read in the same breath, and a reader took the second for the first. IdentifierReport now carries trait_tables_read/trait_tables_not_read and the gene pair beside them, the CLI count names the tables it is out of, and a table that exists and will not parse is reported as unread — an absent optional table is every module's normal shape and says nothing, while an unreadable one means ids the module carries went unchecked. report.clean no longer prints "all identifiers current" over an empty roster, which was the same vacuous pass one level up. A caller passing variants= still gets the narrow roster and is told so in *_tables_not_read, since rows in hand are all that form has.

Since 2026-09-13 an unreadable id-bearing table also fails the gate, and clean says so. The report was honest and the verdict was not: the five stale lists are empty when a table carrying identifiers will not parse for exactly the reason they are empty when everything agreed, so --strict printed the unreadable table and then exited 0 beneath all identifiers current. A terminal reader saw both halves; a library caller reading the dataclass — which is how this arrived, as an MCP tool — saw only the verdict. clean is now a Verdict: falsy when it carries any code, pass when empty, and if report.clean: is unchanged across the change. AcmgReport.clean is the same type, so the two gates in this tier answer in one kind.

The set holds errors, not non-answers, which is the line that decides membership. offline and tables_unreadable are errors and fail. A check the caller switched off, and a module with no row a check applies to, are not: --strict --no-traits still exits 0, and check-acmg over a module stating no acmg_sf cell now passes with its denominator printed rather than withholding, which is the one behaviour RM234 set differently. unreachable is not a member and could not be — a registry outage raises IdentifierUnavailable before a report exists, and that path already exits 1 with an unreachable attestation for every check it runs.

A prose filter was the defect underneath it. unreadable_tables kept everything in *_tables_not_read except the literal "not present", so the row-taking call form's reason — a statement about how the check was invoked, not about a file — counted as eight tables that would not parse. Nothing caught it because the CLI always passes spec_dir. Both benign reasons are now one constant each (NOT_PRESENT, ROWS_PASSED_IN), written by the producer and read by the filter. That is still prose matched at two sites; the structural repair is to carry the roster's read_errors onto the report, which is a field and therefore not a patch.

And since RM156 a module carrying no variants.csv can reach that roster at all. The widening left two gates in front of itself, both keyed on the one table it had stopped depending on: check_identifiers(spec_dir=) loaded variants.csv unconditionally and raised variants.csv is invalid: ... not found, and the command returned "no variants.csv — nothing to check" before ever calling it. The table has never been mandatory, and four of the tables that carry gene are the PGx kinds a module is built entirely out of — this repo ships four such reference examples, and cyp2c19_star_alleles names CYP2C19 on every row it has. The rows are wanted for one thing, placing a symbol against the chromosome its variant sits on, so an absent variants.csv is now no rows and gene_loci_not_checked says why; a variants.csv that exists and will not parse still raises, because that is a module whose rows exist and cannot be read. The command's guard is the roster instead of a filename: nothing to check means no id-bearing table was read, and only then is no attestation written, since with no question put there is nothing to record having asked.

template is the one command that duplicates the compiler's, and the other four offline authoring commands (stub, requirements, describe, hint, scaffold) are deliberately not mirrored. The offline authoring surface has an owner — just-dna-compiler — and the single mirror exists so a PGx author working through this binary does not have to switch tools for a CSV header. See COMPILER.md § CLI for the compiler's own table, including the three schema-tier functions (keygen, sign, reference) that surface there because just-dna-format ships no CLI.

A quote that is the article's own title (S54)

quotes_found checks a provenance_quote against the article's fulltext, and a title appears in its own fulltext — so a quote copied from the article's metadata makes the check pass while establishing nothing about whether any claim is in any paper. Four published modules are in exactly that state: 3,668 rows, one distinct quote per PMID, each verbatim the title. A passage located for a specific claim varies row to row, because different rows cite the same paper for different findings; one string per paper repeated across every citing row is structurally a property of the article, not of the claim.

The pass reports it. LiteratureResult.titles_as_quotes lists the PMIDs whose every provenance_quote is the title, and the CLI prints it in yellow — a warning, never an exit code, because whether a title is an acceptable locator for a claim is the author's decision and what the tool can honestly say is that quotes_found is not evidence here.

The discriminator is the metadata, not the shape of the string. A minimum length does not separate a title from a passage — seventeen words is an ordinary title and an ordinary sentence — and requiring a provenance_regex instead is no help, since a regex is as copyable as a quote. The comparison is against the title already held for that article (it arrives in the same esummary response that answers existence), so it costs no request and it answers for a paywalled article too, which is where quotes_found stays null and a reader has nothing else to go on.

Two deliberate narrownesses. Normalisation is case, whitespace and a trailing period and nothing more — a quote that merely contains the title is a real quote of a paper that names itself, and an aggressive normalizer would start rejecting located passages. And the report fires only when every quote for a citation is the title: a module quoting the title on one row and a passage on another has an author doing the work.

It answers for a pinned row too, and that is the whole difference between working and not. literature.csv is merge-not-clobber, so a row an earlier pass wrote is not refetched while it still counts the module's quotes (since RM277 a pin at another count is re-fetched and re-checked online; offline keeps it) — and on the four modules that motivated this, every row is pinned, so a check living only in the fetch loop would have fired on none of the 3,668 quotes. The summary is therefore fetched for any cited PMID carrying a quote, pinned or not; esummary batches, so it costs no extra round trip in the common case. The pinned row itself stays authoritative and untouched — the merge rule is not the thing that was wrong.

One correction to the report, found against the recorded fulltext for PMC5753237. A title appears in its own fulltext nearly always rather than always: esummary gives it with a trailing period, the JATS body carries it without, and quote_matches does not strip one — so that exact pair misses. The substance is unaffected, since the miss is punctuation rather than evidence, and a module whose two spellings agree gets the green check the report describes. Both states are pinned.

What makes two rows of a sidecar the same row (S51)

Every derived sidecar is merge-not-clobber, so each pass holds an existing map and must decide when an incoming row is one it already has. That decision is published: each row model declares _KEY_FIELDS, just_dna_format.base.merge_key(row) is what every pass keys its map on, and just_dna_compiler.hints.key_fields(csv_name) reports the same columns to a consumer holding only a filename. The pass and the surface are one statement rather than two, which is the half that keeps them from drifting — before 0.6.5 the key existed only as a dict-key expression inside each pass's body, so a tool re-deriving a sidecar had to guess it, and the guess was coarse on exactly the two tables where one subject legitimately carries several rows.

file key rule note
resolution.csv variant_key subject one rsID resolves to several loci (locus_index orders them); the pass replaces the group whole, so several rows sharing this key is the normal shape and not a duplicate
frequencies.csv (variant_key, population) equality one variant carries a row per ancestry group
gene_metrics.csv (gene, dataset) equality two writers (gnomAD constraint, ClinGen dosage); one gene legitimately carries a row per authority
gene_validity.csv assertion_id, falling back to (gene, disease_id, moi, submitter, dataset) equality the source's own id where it published one. Never (gene, disease) alone, which collapses 59 real ClinGen curations, and never without submitter, which collapses the disagreement GenCC exists to publish
clinical_assertions.csv (variant_key, variation_id) equality a null variation_id is a value, carrying the not_found row that states the archive was consulted and holds no record
gwas_effects.csv association_id equality per record, not per variant — a variant gains associations as papers publish
literature.csv pmid equality one article, one row
sources.csv / licensing.csv (source, layer) equality one source legitimately appears at two layers

What a re-run does since 0.7, and what it no longer costs (RM124)

Merge-not-clobber's behaviour is unchanged and deliberately so: a re-run gap-fills — it fills subjects with no row and leaves recorded rows alone. Re-asking every subject on every pass was rejected, because it would put the full resolution time on every run to buy drift detection nobody asked to run continuously.

What changed is what leaving a recorded row alone risks, and the answer is now nothing. A curator's correction to a covered derived table lives in overrides.csv beside the spec, not inside the sidecar; the compiler applies it on every build. So every derived file this tier writes is a pure build product, hand-editing one is not expected, and rm plus a re-run — the crude form of a full re-derivation — costs nothing. Each writer's docstring says so at the point a reader outside this repo actually meets it.

Two consequences for a pass author. A row this tier writes carries no authored content, so a pass may reason about it as the source's answer and nothing else. And a difference between a fresh row and a recorded one now means the source revised, full stop — the ambiguity that made "re-derive the machine rows and keep the overrides" unimplementable is gone, because the tier never has to tell a curator's edit from what the source said last time.

Read rule as well as columns. resolution.csv is the only subject table today, and treating its key as a uniqueness constraint would report a legal one-to-many file as a duplicate. And read fallback — gene_validity.csv is the only table with one, and a consumer ignoring it is right about the other seven and wrong about the one where it matters. The two levels are tagged ("id" / "grain"), so a grain tuple can never collide with an id that happens to equal it.

A check's findings, and where each one's survive (S70)

verification.json records counts, and until S70 clinical_significance was the only one of the checks enrich() puts whose findings then survived nowhere at all:

check where its findings survive
reference_allele detail=, via summarize_ref_mismatches — grouped by diagnosis, three keys named per group plus "and N more"
rsid_coordinate_agreement detail= — up to DETAIL_LIMIT disagreements, plus what was not compared and why
placement_agreement detail= — up to DETAIL_LIMIT disagreements naming both placements, the one-witness count over every placed rsID, and the undecided respellings; and per row in resolution.csv as placement_witness / placement_verdict
rsid_currency per row in resolution.csv — rsid_status and rsid_current are columns of the written file
genome_build_agreement count only, but its subjects are reference_allele's mismatches, so the candidate rows are reachable
clinical_significance detail= since S70, grouped on opposed. Before that: the logger, and nothing else

The conflicts always existed at runtime — compare_clin_sig returns them and every one is logged — but stderr survives the process and nothing else does, so a record reading findings: 20, detail: null left an author able neither to defend the twenty nor correct them, and re-running to see the log again costs the full ClinVar comparison. _clin_sig_detail groups on ClinSigConflict.opposed because that is the distinction that decides what to do (an opposed pair is worth acting on, a difference is worth knowing) and names both values, since the standing instruction on a mismatch is to check both sides and a bare key cannot be checked at all.

The compiler now says so too. _findings_warning reports a non-zero findings at validate and compile; nothing read VerificationRecord.findings before, so the counts reached manifest.verification.checks[] and an author running the validator saw a green result. A record reporting zero says nothing — a check that could not fail must not report one.

The other half shipped in 0.7 as RM130: a detail makes the rows nameable to a human and not joinable, and the concordance record above is what makes them joinable. RM124's question about whether one record serves both an overlay and outranks was answered as a dated succession, so the record is the overlay's input side — a conflict is a question and an overrides.csv row is the answer.

How a sidecar is written, and why merge-not-clobber makes that load-bearing (S66)

Every sidecar writer goes through just_dna_format.layout.atomic_writer / atomic_write_text — a temp file in the same directory, fsync, then os.replace. Same directory because os.replace is only atomic within a filesystem; fsync before the rename so a machine losing power cannot expose an empty file where a complete one used to be.

This sits under the merge table above rather than beside the writers because the merge is what makes truncation dangerous. A writer that truncates in place leaves, on a kill, a file that is syntactically valid and simply short — and a short resolution.csv is not detected as damaged by anything, it is read back, keyed on subject, and believed. Three branches in enrich() write no row for a subject nothing could answer (@unreachable-not-absent, and each is correct), so fewer rows is a state the table reaches legitimately: a truncated table and a module whose author resolved less are the same bytes. The reported incident is the sharp form — a run killed client-side kept going, reached the write, and replaced a restored 330-row table with 162 rows, after which the module validated, closed and compiled green.

enricher/tests/test_atomic_sidecar_writes.py walks the writers rather than testing one: three of the nine were reported and the other six had the identical shape, so a fix scoped to the report would have left the next writer inheriting whichever neighbour it was copied from.

What this did not fix, and what RM128 then did. Atomic writing left enrich() persisting nothing until its tail, so a run killed before the write had written nothing at all — atomically. The lost-work half, the missing lock over a read-modify-write window that spans the entire run, and the absent progress callback were each a decision rather than a missing line, and all three shipped in 0.7: see The run is a transaction above.

resolution.csv is provisional (0.5)

The table's shape (ResolutionRow — columns, keying, the status vocabulary, how one-to-many expansion is encoded) arrived in 0.5 and is still not frozen at 0.7: SCHEMAS.md carries it as provisional, and the additive-within-a-major / digest obligations have deliberately not been engaged for it. It is free to be refactored wholesale while that status stands, and has taken several passes already. See SCHEMAS.md § the resolution table. The compiler's consumption contract (digest parity between the resolution.csv path and the DuckDB path, offline round-trip) holds regardless of the table's internal shape.

Downstream adoption (cross-repo)

The enricher is the single source of truth for variant resolution. The intended migrations, tracked in ROADMAP.md, live in their own repos:

  • ensembl-mcp keeps only its FastMCP wrapper and imports the query core from here.
  • just-dna-lite / just-dna-pipelines drops its resolver shim + HF download/upload copies and imports the enricher (just_dna_enricher.upload is the canonical publisher API; pipelines still carries a local copy while it is pinned to format/compiler <0.4 and cannot import the 0.5 enricher yet).

Open questions the code does not answer

Written down because each was reached for and not found, and a reader who reaches for the same thing deserves to know it is absent rather than hidden. None is a defect with a filed RMn — a defect is a place where the code and a stated rule disagree, and these are places where nothing states a rule.

  • The Ensembl snapshot's builder. download.ensure_snapshot provisions datasets/just-dna-seq/ensembl_variations_split/data/homo_sapiens-*.parquet and resolver.py reads a fixed column set from it (id, chrom, start, ref, alt), but nothing in this package builds that snapshot, unlike the ClinVar and gnomAD ones: prepare-annotations does. Since RM372 its release is read from the manifest.json it carries (locations.ensembl_release_label), so rsid_coordinate_agreement.release names it; a cache of the old repo states none and stays None.
  • What the ClinVar snapshot's cache status label should be. clinvar_build._write_release_json writes no dataset key and cache status prints release.get("dataset") or "", so a correctly built ClinVar snapshot shows a blank label. Whether that is intended — ClinVar has its own richer clinvar_dataset_label — or an oversight is not decidable from the code.
  • enrich_gwas(mode=…). Accepted, defaulted, never read; the CLI's --strict help promises a "severity ladder" and the pass docstring describes none. Whether the intent was a strict refusal on missing (as the sibling passes have) or no ladder at all is undetermined, and no test covers either. The dead parameter itself is filed as part of RM100; what it should do is this question.
  • identifiers.IdentifierCheckError is declared with a docstring and never raised anywhere, in src or in the tests.
  • CpicClient.knows_drug's return type. Annotated bool | None; the live client can only return bool or raise, and only CpicSnapshotClient returns None. pgx_draft handles all three, so the widening looks deliberate — the live client's docstring just does not say so.
  • Whether check-identifiers is meant to have an --offline. identifiers.verification_records argues it must not; its sibling check-acmg has one. Self-consistent, undecided as policy.

Testing

uv run pytest enricher/tests. All network-free: the cache side uses a tiny synthetic ensembl_variations parquet; the live-Ensembl side uses an httpx.MockTransport; upload is tested against a fake HfApi. Coverage includes offline enrich → compile matching the DuckDB path's facts (every file but the loci RM380 carries), --offline making zero network calls, the V2 503 → V1 REST fallback, tenacity retrying a transient error, strict failure, one-to-many expansion, and the upload plan/token paths. Integration tests that need a real cache are @integration (skipped without JUST_DNA_ENSEMBL_CACHE).

Two files carry an unusual shape on purpose. test_query_shapes.py guards the hash-join probe (0.5.2): its load-bearing assertion is on the query plan, so it needs no clock at all, and its one timing check runs the OR-chained shape and the joined shape in the same process against the same synthetic snapshot — a relative bound, because an absolute one just measures the runner. And test_locations.py runs every probe in a subprocess with a controlled cwd and environment: the bug it pins is the first resolve in a process, and load_env mutates os.environ for the rest of the session, so an in-process test would neither reproduce the defect nor stay isolated from the rest of the suite. It also demonstrates the old arrangement failing rather than asserting that it used to.

The gnomAD tests are driven by real recorded payloads committed under assets/ (gnomad_v4.1_variant_payload.json, gnomad_gene_constraint_payload.json, gnomad_v4.1_constraint_slice.tsv), replayed through httpx.MockTransport. Recording rather than fabricating matters here: the quirks under test — the "Multiple variants found" error sitting beside valid data, the duplicated XX/XY entries, the two mane_select=true rows per gene — are ones a hand-written fixture would have quietly omitted, and the tests would then have passed against the naive implementations they exist to catch. Coverage: batching and the pacing gate (on an injected clock, so the suite never really sleeps six seconds), partial-error resilience, AC/AN against the payload's own exome.af, faf95 landing on one group only, the MANE pick fed in both orders, byte-identical snapshot rebuild, the chain order (a both-links variant keeps source="cache"), --offline making zero gnomAD calls, an unqueryable ClinVar cache degrading instead of crashing, VRS minting and the snapshot-vs-API dataset labelling. Live queries and indel normalization are @integration and opt-in via JUST_DNA_NETWORK_TESTS=1. Without it, every other test is fenced off the network at the transport (RM391): the root conftest.py refuses any host but this machine at socket.getaddrinfo/connect and gRPC's channel constructors, raises an error that names the test and the host, and fails the test at teardown even when a client swallowed the raise.

The ClinVar tests (test_clinvar.py) build against a small committed real slice (assets/clinvar_GRCh38_slice.vcf.gz), computing expected values from that VCF at runtime: build correctness (columns, clin_sig ⊆ VALID_CLIN_SIG, clin_sig_raw preserved), the cross-source coordinate agreement / off-by-one guard, byte-identical rebuild, the ClinVar-after-Ensembl chain order (no compiled digest moves), one-to-many expansion, the allele-aware back-fill + ambiguity marking, the compile → reverse → compile fixpoint, and the mocked reference-snapshot publish. A full build against the real local VCF is @integration (skipped when absent).

The clinical cross-check (test_clinical.py) builds a snapshot from that same slice and leans on a coincidence in it that is too useful to be luck: the slice contains both of rs334's opposed records (T>A pathogenic, T>G likely_benign), so allele-exactness is provable on real data rather than on a constructed pair. Coverage: an opposed call reported with its star count, the same call on the other allele correctly staying silent, uncertain_significance not counting as disagreement, the locus-wide fallback in both directions, strict not escalating, and a foreign parquet degrading instead of raising.

The literature and identifier tests replay recorded payloads from the live services (pubmed_esummary_payload.json, europepmc_search_payload.json, europepmc_fulltext_PMC5753237.xml, dbsnp_esummary_payload.json, ols4_terms_payload.json, hgnc_fetch_payload.json). Each recording carries a quirk a hand-written fixture would have smoothed away, and the tests exist to pin exactly those: a nonexistent PMID arriving as a normal-looking record with an error key; a real PubMed record that is simply not in PMC (exists-yes, retrievable-no); Europe PMC silently omitting ids it does not know; and — the one most worth guarding — rs11273140 (withdrawn) and rs2000000000 (never assigned) returning byte-identical responses, asserted on the recordings themselves so that a future dbSNP release which does separate them fails the test rather than silently invalidating the design. The quote matcher is exercised against the real JATS fulltext with a phrase read out of that same document, and the regex bound is demonstrated on a genuinely catastrophic pattern (it must return not checked, never not found).

Each of these files also carries an opt-in live probe (JUST_DNA_NETWORK_TESTS=1) that re-asks the real services the same questions, so a recording that has drifted away from reality fails loudly instead of letting the unit tests pass against a fiction.

GWAS effect sizes (gwas.py, online only) — RM90

just-dna-enricher gwas <spec-dir> fills gwas_effects.csv with the NHGRI-EBI GWAS Catalog's published effect sizes for every rsID the module names. One row per published association, not per variant — rs1800562 alone carries 191 (v2, 2026-11-04) — keyed on the Catalog's own association_id and merged never clobbered.

The subjects come from enrich.collect_subjects since RM158, not from variants.csv alone. Five authored models carry rsid, so a module whose rsIDs live in haplotypes.csv or pharm_variants.csv got no associations and no line saying none had been asked for. The collector is the same one resolution uses and answers the same question — which rows in this spec ask about a variant — so this pass inherits its precedence (variants.csv first, first occurrence winning, so a PGx row cannot take an identity a SNP row minted) rather than restating the loop. studies.csv carries rsid and is deliberately not a subject: a study row references the variant it grounds, which the module already carries as a row of its own. Its proxy_rsid (0.8, RM331) is a resolution subject, but this pass calls the collector with include_study_proxies=False: an association for an LD proxy is not one for the module's variant.

It does not fill weight, and that is the point of the pass rather than a limitation of it. A consumer asked for exactly that (S36); MODULE_LIFECYCLE § Stage 3 names the cell, and every check here reports rather than repairs. The effect sits beside the authored column and a consumer picks one wholesale.

The pass reads REST API v2 since RM354, because the legacy API is gone. Probed by hand on 2026-11-04:

request status what came back
GET /gwas/rest/api/singleNucleotidePolymorphisms/rs4149056/associations (legacy, what the pass read) 410 text/plain: "This legacy GWAS Catalog REST API is no longer available.", pointing at the v2 migration guide. Every legacy path answers the same, /gwas/rest/api/studies/<accession> included
GET /gwas/rest/api/v2/associations?rs_id=rs4149056 200 20 associations of page.totalElements: 386, with _links.next; size=1000 returns all 386 on one page
GET /gwas/rest/api/v2/associations?rs_id=rs1421085&size=501 (2026-11-10, RM428) 400 {"status":400,"error":"Bad Request"}, which does not name the limit; size=500 answers 200. The ceiling dropped from at least 1000 in six days
GET /gwas/rest/api/v2/associations?rs_id=rs111033563 (a rare clinical variant) 200 no _embedded key at all, page.totalElements: 0 — the same for an rsID that does not exist
GET /gwas/rest/api/v2/associationz?rs_id=rs1 (a wrong path) 404 the only 404 the search gave
GET /gwas/rest/api/v2/studies/GCST000386/ancestries 200 _embedded.ancestries[].ancestral_groups[].ancestral_group per stage (initial, replication)

A 410 raises GwasRetired, a GwasError subclass whose message carries the URL and the Catalog's own text, and it is asked once. It used to fall into the generic status arm as a plain GwasError, the could-not-ask an outage raises, so the pass died saying the source had failed rather than that it had moved. The class this belonged to, in every client of the tier, is § Exception contract.

The page size is the server's, and it refuses rather than caps (RM428). The pass asks for 500 associations a page, the ceiling measured on 2026-11-10, and follows _links.next for the rest. A 400 to an rsID's first request is asked once more with no size, so the Catalog's default page (20) decides; the run warns once, naming the refused size and how many rsIDs paid the extra request, and requests_made counts it. A second 400, or a 400 on a later page (the size was accepted by then), is the could-not-ask GwasError as before. No unit test pins the number; the opt-in live test test_live_rm428_the_catalog_accepts_the_page_size_this_pass_asks_for asks with it and expects a 200.

How v2 maps onto GwasEffectRow — the same columns the legacy payload filled, and the association ids carried over: legacy 13069 and 55421052 are v2 13069 and 55421052, so a re-run merges into a table the legacy pass wrote rather than doubling it (a test pins that).

column v2 field note
association_id association_id (an integer) _links.self's last segment as the fallback
effect_allele, rsid the snp_effect_allele entry naming the queried rsID rs4149056-C, or rs4149056-? → null: the legacy riskAlleleName grammar, unchanged. A multi-SNP association lists one entry per SNP; none naming this variant withholds the allele rather than taking the first
effect_size, effect_measure or_per_copy_num → OR; else the number in beta → beta beta is one composite string where the legacy API had three fields
effect_unit, effect_direction the middle and the last token of beta "0.05 umol/l increase"; the unit kept verbatim (one study writes U+2009 in it). "-" is v2's no-value. A string outside <number> [<unit>] <increase\|decrease> is withheld whole and counted (unparsed_effects, and --strict refuses on it)
standard_error — v2 publishes none. Always null; never derived from the CI
confidence_interval range verbatim; v2's "-" → null, a study's NR kept
risk_allele_frequency risk_frequency NR → null
p_value pvalue_mantissa + pvalue_exponent → 3e-493 the pair the Catalog curated; the float beside it is 0.0 past float64's range and 2.9999999999999997e-29 for 3e-29. p_value_num is read from the same string
trait, trait_efo_id efo_traits[].efo_trait, efo_traits[].efo_id inline now. The EFO label, as the legacy efoTraits link served; reported_trait is not read
pmid, study_accession pubmed_id, accession_id inline now; v2's - → null
ancestry /studies/<accession>/ancestries the one fact still behind a request. Every stage, sorted; NR → absent

Everything below came from probing the live API, not its documentation.

  • The request budget fell by half, and the cache finally pays. On the legacy API pmid, study_accession, ancestry, trait and trait_efo_id all sat behind _links.study and _links.efoTraits, so the pass cost 1 + 2N per variant, and _LinkCache — built on the prediction that associations share studies — saved nothing: rs1800562's 189 associations each named their own study, 382 requests, zero cache hits. On v2 only ancestry is behind a request, one per distinct study, and the same module measured 190 requests with 11 saved on 2026-11-04 (10 variants, 191 associations, 181 studies; 3 min 28 s at the 1 s courtesy interval). --no-study-facts skips that follow and costs one request per page; the inline facts are filled either way, because they cost nothing. The loss is permanent for the rows it writes, and this is not the per-run trade the sentence above reads like (S50). The merge is keyed on association_id alone, and the skip happens before the row is built — so a row written with --no-study-facts keeps ancestry null, and a later run with study facts skips it rather than filling it in. Delete gwas_effects.csv to re-derive it. Every other delete-to-regenerate case in this tier is about a stale value; this one is about a value that was never fetched, so the file looks complete and cannot be repaired incrementally. Back-filling instead was considered and refused: it would make the pass rewrite existing rows, which is what merge-not-clobber exists to prevent, and a null ancestry is not distinguishable from a study that published none — the case follow's 404 arm and an all-NR study deliberately produce. (Before RM354 the flag dropped pmid, study_accession, trait and trait_efo_id too; rows written that way keep them null.)
  • The empty answer is a 200, and a 404 is no longer one. The legacy API held only variants with a published association and 404'd on a rare clinical one, which the pass read as the empty answer. v2 is a search and answers that variant with a 200 and zero elements, so associations_for lets a 404 propagate as GwasNotFound — could-not-ask — and reading it as "no associations" is the confident negative this pass refuses. follow keeps its 404 arm: one study's ancestry withheld, the effect kept. Paging is followed, not trusted: the pass asks for 1,000 per page and walks _links.next, and a 200 with no associations that does not say totalElements: 0 is a shape change, which raises.
  • p_value: 0.0 is an underflow the Catalog really publishes. p_value_num is gt=0 because such a value is not a probability; the pass withholds the number and keeps the row — an early version let the ValidationError discard a real published effect over one derived column (six on rs1800562, still six on v2). The verbatim string now says what the study reported (3e-493), where the legacy pass could only write 0.0.
  • snp_effect_allele is rs4149056-? when a study never established which allele carries the effect. Null, never a guess, and the row is kept and counted — 33 of 191 on that module on v2.

Licensing. GWAS_CATALOG_TERMS is the first entry here with no named licence: EBI states its terms in prose (read 2026-08-17). redistribution=True is directly supported; commercial_use stays None because the page permits "use" generally but conditions it on the original data owners' terms, which for an aggregator of thousands of publications are not established. Unknown is neither permission nor refusal — taints_commercial_use requires an explicit False, so a null warns rather than gating. Do not tidy it to True; a test pins it.

LitVar2 / PubTator3 — literature coverage, and the tier that answered (litvar.py, online) — RM167

NCBI's LitVar2 indexes the literature by variant, and it does so at three tiers that sit beside each other as separate nodes. A node id is litvar@<clingen_id>#<rsid>#<gene_id> with an unfilled slot collapsing to a bare #, so litvar@rs1800562## is the position node and litvar@CA113795#rs1800562## is the allele node beside it, named by a ClinGen canonical allele id. just-dna-enricher litvar coverage spec/ asks, per module locus, which of those answered.

The finding is the tier, because the tier a locus is answerable at is a property of the locus and not of the source. Measured 2026-09-01:

locus position node allele node(s) on the position node and no allele node
BRAF rs113488022 32,095 31,276 + 99 + 41, three CAIDs 801 (2.5 %)
HFE rs1800562 3,053 2,693, one CAID 360 (12 %)
APOE rs429358 3,945 328, one CAID 3,617 (92 %)

BRAF's three CAIDs are three distinct ALTs at one codon — V600E, V600G, V600A — and they differ by three orders of magnitude, so allele resolution is doing real work. APOE is the case that decides the shape: 92 % of the literature at that locus is not allele-resolved, so a pass that reported the allele node's count as the answer would understate it twelvefold, and one that quietly substituted the position count would answer an allele-level question with a position-level fact. Allele-resolved, position-only and absent are three outcomes, and for once the source supplies the three states rather than the schema imposing them.

How a module locus reaches an allele node. The position node's own record carries clingen_ids; each CAID goes through RM153's clingen_allele.ClingenAlleleClient, and its GRCh38 allele is compared against the ones the module names — a real second authority rather than a rename. A resolution.csv that already carries a caid short-circuits that, because the module has then stated its allele identity outright. A one-sided indel — the shape the registry states with an empty referenceAllele or an empty allele — is anchored through clingen_allele.anchor_indel using the module's own ref base at its own start, and withheld anywhere the module does not state one: a guessed anchor puts a wrong ref on a right position, which is a false match rather than a missing one. That last leg needed a repair one file over to work at all — _parse computed the one-sided allele and then dropped it on every record that also carried an rs number, which is most of them, so the registry looked as though it held nothing comparable. It is carried now, and the corpus's own hboc_palb2 rows are what the tier verdict turns on.

The asked tier is a property of the module's rows. A module that names an allele at a locus asks an allele-level question; one that names an rsID and nothing else asks a position-level one, and the position node answers it exactly. Without that split a purely positional module would have every locus counted as a shortfall. The allele columns come off AuthoredModel.ALLELE_COLUMNS, and the rsID roster off DRAFTABLE, so a table kind added later joins by existing.

Four outcomes, and the fourth is the house algebra's. allele / position / absent / unchecked, each with its own reason and its own sentence (coverage_reason, one arm per way of getting there, walked by a test):

tier reason what it means
allele allele_node_matched an allele node for the allele this module names
position row_names_no_allele the module asked at position level; this is the answer, not a shortfall
position no_allele_node_at_locus the index holds no allele node here at all
position allele_nodes_name_other_alleles it holds some, and none of them is this module's
absent no_node_for_rsid the index holds nothing for this rsID at any tier — an answered absence
unchecked offline nobody asked
unchecked index_unreachable LitVar could not be reached
unchecked registry_unreachable the CAIDs at this locus were never resolved, so none of them is this module's is not established
unchecked allele_not_comparable the registry answered and holds no allele these columns can compare

The last two are the collapse @answered-is-not-absent names, one tier out: an empty match is an established negative only where every CAID was actually compared.

The residue is counted, never discarded (@dont-discard-computed). position_only_pmids is the papers on the position node that no allele node at that locus claims — a set difference over the union of every allele node, not a subtraction, because the allele nodes overlap each other. It is 92 % of APOE and 2.5 % of BRAF, and both numbers are about the locus rather than about the module's allele.

It writes no row, and that is the whole artifact answer. A PMID list per variant is not a table kind, literature.csv is keyed by article, and 32,095 PMIDs for one BRAF locus is a row-writer arguing against itself. What lands is verification.json: one literature_coverage record whose subjects is the loci the index answered about, whose findings is the loci where an allele-level question came back position-level, and whose detail names the tier breakdown — because a coverage answer that does not say which tier answered is the defect this lane is about.

No SourceRow either, and for the converse of @write-the-sourcerow: sources.csv travels to the registry meaning this module uses this source, and nothing here reaches a module's tables. identifiers.py is the precedent, civic_draft.py — which does put registry-derived values into a module and does write its clingen_allele_registry row — is the contrast. The source is named on the VerificationRecord instead.

What the corpus actually gets, measured 2026-09-01

Run over the 11 reference modules carrying a resolution.csv — 389 loci, one rsID shared by two modules — with one shared client so a locus two modules name is one request:

loci
answered at allele tier 165
answered at position tier only 92
absent from the index at any tier 122
could not be asked 10
with at least one CAID node at the locus 180 (46.3 %)

Every locus in this corpus asks an allele-level question — no module names an rsID and no allele — so all 92 position-tier answers are the finding rather than the question. Of the 180 loci that carry a CAID node, 165 resolve to the module's own allele; the other 15 split three ways, and each keeps its own reason: 10 allele_not_comparable (the registry holds a one-sided indel this module's row cannot anchor), 3 allele_nodes_name_other_alleles, and 2 where the position node lists a CAID the index has no node for.

14,168 papers across the corpus sit on a position node that no allele node claims, and APOE is most of the top of that list: rs429358 contributes 3,617 of 3,945 and rs7412 3,083 of 3,597. The PGx loci run about half — rs4149056 is 575 of 1,342, rs1057910 606 of 1,093. A pass that reported the allele node's count as the locus's answer would have been quietly wrong by those margins on nine of the eleven modules.

The 122 absences concentrate where you would expect a literature index to hold nothing — 105 of them are in pathogenic_clinvar, a panel of rare ClinVar variants. What the measurement establishes is that the index holds no node for those rsIDs; whether that is because no paper names them or because nothing indexed the paper that does is a question this surface does not answer, and it is recorded as absent rather than as either reading.

The bound: it answers which papers discuss an identified allele, not which allele a name meant

Do not reach for this to recover an identity. Those two read as the same question and are not, and the measurement is on the two hardest records in this repository. CIVIC_LEGACY_INSERTIONS works CIViC 1955 (VHL P71fs (c.211insT)) and 2131 (VHL Q73fs (c.214insGCCC)) down to four candidate alleles with registered CAIDs. Asked of LitVar on 2026-09-01:

asked answer
CA2586965638, CA2501268513, CA2573048346, CA2499307076 no node for any of the four
c.211insT, 211insT, c.214insGCCC no node
VHL P71fs one node — litvar@#7428#p.P71fsX, 1 PMID: 19996202

That single hit is none of the four source papers; it is an unrelated paper that happens to write "P71fs" (@existence-not-identity), and its id is the fifth shape — litvar@#<gene_id>#<protein_name>, all three flag_* false, an unnormalized text mention rather than a variant. Nothing here treats one as an identity, and litvar gene prints the mention count precisely so a reader can see how much of a gene's tail is text: of 588 HFE nodes, 220 are rsID-only, 69 carry a CAID, exactly one is the gene node (litvar@#3077#, 3,285 papers) and the remaining 298 are mentions.

The reason is structural, which is what makes it a bound rather than a gap. PubTator3's export for all four source papers returns title and abstract only — two passages, and zero variant annotations in every one. None is in the PMC open-access subset. The alleles live in Table 3 of a paywalled 1996–2007 paper, and text mining over abstracts cannot reach a table. §8 of that probe went around the same wall through UMD-VHL's curated protein column — a curator's tabulation, one step from the primary. LitVar indexes text.

Two API facts, and one of them is a defect

  • variant/search/gene/GENE returns line-delimited Python repr(), not JSON — single-quoted keys, one dict per line. httpx's .json() raises on it. The other endpoints return proper JSON. parse_repr_lines is a literal parser (ast.literal_eval), never eval, and a test walks the module's AST to assert nothing in the lane calls .json() at all. That endpoint also carries no flag_* at all, so the tier there comes from which keys a record has.
  • autocomplete is a prefix search. ?query=rs429358 returns rs42935848 as a second hit — a real node for a different variant. Taking [0] off that list answers confidently about the wrong thing, so both lookups filter on an exact rsID or CAID. And node ids are never constructed from the grammar: every id handed to get or publications came back from a listing verbatim. The first pass at this source read the trailing ## as a suffix on an rsID, asked one endpoint, and concluded there was no allele tier at all — a confident negative about a whole tier, from one misread character.

An absent node is an answer: autocomplete returns 200 [], and variant/get returns 400 with a body opening Variant not found. The discriminator is the body and never the status, because a malformed query is also a 400 — the same rule as Ensembl's 400 on an unresolvable rsID.

Licensing: NCBI publishes a policy, and a policy is not a licence. NCBI states it "places no restrictions on the use or distribution" of molecular data and, in the same passage, that it "cannot provide comment or unrestricted permission concerning the use, copying, or distribution" because submitters may hold rights it cannot assess. ClinVar escapes that through its own maintenance_use page, which is why CLINVAR_TERMS records public-domain; LitVar has no such page, so under @no-named-licence its gating axes are unknown rather than permissive. Recording it as public domain by analogy with ClinVar is exactly the move that rule forbids. This is NCBI's side only — nothing was read about EMBL-EBI's terms for the surfaces EBI co-hosts, and nothing here asserts anything about them. There is deliberately no LITVAR_TERMS constant, and the rule is worth stating because that file has two kinds of entry rather than one: a TERMS_BY_SOURCE member earns its place either through a pass that records it into sources.csv — clingen_allele_registry through civic_draft, pubmind through its drafting provider — or through a snapshot that keeps the source's bytes on disk, where the unknown redistribution axis becomes load-bearing the moment somebody proposes publishing them, which is MANE_TERMS. This lane has neither: nothing it reads reaches a module's tables and nothing is stored. So the finding lives here and in the module docstring, which is where a reader would look.

Regulator drug labels (drug_labels.py + drug_labels_build.py) — clinpgx check-labels — RM166

ClinPGx's drugLabels.zip annotates the pharmacogenomic content of medicine labels published by five agencies: FDA, Health Canada (HCSC), EMA, Swissmedic and PMDA. The item asked for the FDA and the file supplies four more at no extra cost, which changes the shape from module ↔ authority ↔ authority into a lane where the number of authorities is a parameter — exactly what RM134's vocabulary split was built to survive. Nothing in the surface names an agency. Baking one into a published key is the mistake RM134 caught in ClinSigConflict before it shipped, so the check is regulator_label_agreement, the command is check-labels, and the agencies are data in a regulator column.

just-dna-enricher clinpgx build-labels --use non-commercial                       # → data/repro/drug_labels
just-dna-enricher clinpgx check-labels spec/ --snapshot data/repro/drug_labels --use non-commercial

A second archive from a source already adopted, and a broader finding behind it

Enumerating api.clinpgx.org/v1/download/file/data/ properly — a 303 is a real file, a 404 is not — finds at least twelve published archives: clinicalAnnotations, variantAnnotations, clinicalVariants, drugLabels, relationships, variants, genes, drugs, chemicals, phenotypes, occurrences, pathways-tsv. Before this item, clinpgx_build downloaded one of them. So the honest restatement of RM166 is that the PGx lane reads a fraction of a source it has already adopted and gated, and the FDA question was a narrow way into a broad finding. clinicalVariants.zip is the next one that bears on a shipped table kind and is deliberately not built here.

drugLabels.zip was 59 KB on 2026-08-05 and holds LICENSE.txt, README.pdf, drugLabels.tsv (1,433 rows) and drugLabels.byGene.tsv (238 rows). The second is a pivot of the first by gene symbol and carries no fact the label table does not, so nothing reads it.

Its own release.json, never the annotation lane's. clinpgx_build's docstring records relationships.zip a year newer than clinicalAnnotations.zip, so the archives do not refresh in lockstep — and RM175 later found the reason that gap was so wide, which was that ClinPGx had retired the annotation archive under that name. The label snapshot is dated from its own CREATED_<date>.txt and labelled clinpgx_drug_labels_<date> — distinct from clinpgx_<date>, because two surfaces have two denominators (@two-surfaces-two-denominators).

Every cell is stored verbatim. Seven of the fifteen columns are flag-shaped — a blank or one constant string — and coercing them to booleans would have the builder decide that Biomarker Flag is one, which it is not: it carries three values, On FDA Biomarker List and Formerly on FDA Biomarker List beside the blank. A blank becomes None and the reader decides what a cell means.

The separator is ;, and it is not vocab.MULTI_SEP

MULTI_SEP splits on ,;|. In this file a comma inside a Genes / Chemicals / Variants/Haplotypes cell is data: DPYD c.1129-5923C>G, c.1236G>A (HapB3) is one haplotype named by two variants, and Ascorbic acid (vitamin C), combinations is one chemical. Splitting on it reports 604 variant tokens where the file states 601 and turns three real names into six that match nothing. Read with ;, the cell holds 601 tokens over 189 distinct names, 415 of them rsID-shaped and 178 star-allele-shaped.

Two join tiers, because they are not the same claim

Genes is populated on 1,248 of 1,433 rows (87 %) and Variants/Haplotypes on 217 (15 %). So a module's claim is put at two granularities and the tier is a property of the subject:

  • (gene, drug) — the gene tier. What do the agencies say about this gene and this medicine?
  • (gene, allele, drug) — the allele tier, where the allele is a star allele from diplotypes.csv or an rsID from pharm_variants.csv.

A label naming CYP2C19*2 answers both, because those are two questions rather than one asked twice. The alternative — scoring every authored allele against whatever the gene-level labels say — was written first and measured: on reference_examples/cyp2c19_star_alleles it reported the EMA's single disagreement about clopidogrel 34 times, once per star allele, none of which the label mentions. Under the two-granularity shape the same run reports it once at the gene tier and seven times at the allele tier, for the seven alleles the labels actually enumerate.

The star tokens are not haplotypes.csv's key verbatim, which the item's entry says they are. The file writes CYP2C19*2; the module writes *2 in haplotype_name with CYP2C19 in its own column, so the join composes them. It composes them two ways, because the file spells a gene-qualified token two ways: the star alleles run together (TPMT*3A) and the DPYD haplotypes are spaced (DPYD c.2846A>T). Trying only the concatenation told a DPYD module its allele was named by no label while two regulators named it exactly, which is a false coverage claim rather than a miss. The bare spelling is tried last, for an rsID and for an HLA allele already authored with its gene.

Composition is spelling, never an allele-definition mapping, and the limit is worth stating. A DPYD module authoring *2A gets DPYD*2A / DPYD *2A / *2A and matches nothing, because the file spells that allele DPYD c.1905+1G>A (*2A) — a star name and the variant that defines it are the same allele under two naming systems, and knowing that is PharmVar's job rather than this join's. So the allele tier answers where the two sides spell the same allele the same way; where they do not, the gene-tier subject still answers and the allele is counted as named by no label.

An authored allele no label names is neither withheld nor a finding: the gene-tier subject for the same pair is what answers for it, so it is counted and reported as a coverage number.

Testing Level is five members and a third of the file states none

Testing Required 374, Actionable PGx 312, Informative PGx 162, No Clinical PGx 87, Testing Recommended 26 — and 472 rows stating none. A blank is an absence, not a no: it is unknown, it is counted, and it withholds. Kleene, so it cannot un-see a disagreement already witnessed and it never establishes an agreement on its own. The vocabulary is derived from the payload and a test asserts the equality against the fixture, because a sixth member upstream has to be a visible edit rather than a silent .get(x, default).

Two verdicts, and only the ends of the axis are placed

classify_labels is a pure function over an authored action and N calls at one tier, and it answers two orthogonal questions the way classify_concordance does:

  • concordance — concordant / discordant / single / unstated / none. Level equality, with no ordering at all, so a level ClinPGx adds later still classifies correctly here.
  • position — opposed / unplaced / unchecked / absent / no_label.

The module carries no testing-level column, so most of the axis is deliberately unplaced. Mapping Testing Required onto recommendation_strength=strong would be this format inventing an equivalence between a regulator's testing requirement and CPIC's prescribing strength. No Clinical PGx needs no mapping — it is the negative claim by its own name — so the one authored arm fires when a module ships a prescribing recommendation for a pair every agency that spoke calls No Clinical PGx. The three middle levels are stated and unplaced, which is reported as such rather than as an agreement. The reverse direction withholds too: a module declining to recommend for one diplotype is a statement about that genotype, not about whether the medicine's label carries pharmacogenomics (@refutation-withholds).

evidence_level is not read here at all. It is ClinPGx's own metadata about its own annotation and clinpgx check owns it; treating "there is 1A evidence" as "this module recommends" would put two axes in one field.

What the corpus says

  • cyp2c19_star_alleles — clopidogrel and CYP2C19 is Actionable PGx at four agencies and Informative PGx at the EMA. Three of the five name the star alleles and two name only the gene, so the same disagreement is established at both tiers and reported apart.
  • pgx_slco1b1_simvastatin — four labels reach SLCO1B1 + simvastatin and two of them state no level. The two that do are both Swissmedic's, at different levels: one covers simvastatin and one the fenofibrate/simvastatin combination, and both name rs4149056. So the pair is discordant, one agency disagrees with itself, and neither of its rows is picked as its opinion (@multiplicity-is-a-finding). The two blanks are counted into the record and named in the sentence as stating no level, never folded into the count of labels that stated one.
  • cyp2c9_warfarin_grch37 — CYP4F2 + warfarin is a claim no agency labels, at either tier. Withheld at both, named in the record, and never reported as an absence of pharmacogenomics. It is also what keeps the coverage number honest: an allele only counts as "answered at the gene tier" when the gene tier actually answered, and CYP4F2's did not.

Severity, and what is not written

--strict never escalates, and a test asserts the two modes report the identical list. Five expert regulators genuinely disagree with each other and with a curator, and a compile that refused would make this format arbitrate between its own authorities — the rule the ClinVar clin_sig, PGx allele-function and repeat-band checks already follow. What strict still refuses is structural: a diplotypes.csv that will not load raises in both modes.

This check writes no SourceRow, and either of two reasons would settle it. Nothing from the labels lands in the module — it writes no authored cell, and sources.csv accounts for what a module carries (@write-the-sourcerow's converse; check-repeat-bands is the shipped precedent). And the row is not free: merge_sources_csv keys on (source, layer), clinpgx/annotation is already owned by clinpgx check and draft-clinpgx, and that row's dataset is load-bearing — the evidence-level check's tautology guard compares it against the annotation snapshot's label. Stamping clinpgx_drug_labels_<date> into that slot would silently disable a shipped check. Every other layer sits outside the compiler's orphan-check exemption and would warn source_row_unused on every module.

The licence gate still applies, at clinpgx_draft's reading rather than a third one: ClinPGx's terms are accepted when the data is taken, so --use commercial refuses the read as well as the fetch, and the skip is not_permitted — what clears it is a declaration, not egress.

What closed with this item

The entry wanted two things from the FDA: this concordance check, and a PGx lane member whose terms may not gate. The second is refuted by both routes and closed in writing. The ClinPGx route is CC BY-SA + no-sale, the same gate the rest of the lane sits behind, so it diversifies nothing. FDA's own Table of Pharmacogenetic Associations, probed directly on 2026-09-01, is 126 associations in an HTML page with no CSV or XLS download and no copyright or public-domain statement on the page at all — a quarter of the FDA content ClinPGx already carries, in a shape that has to be scraped, on terms that are unestablished. "US government work is public domain" is a rule with exceptions and the page does not settle it. Licence diversification for this lane is still worth doing, and it wants its own entry with candidates chosen for their terms first, which is the opposite of how this one chose.

AlphaGenome — a local artifact, a live service, and two licence classes (alphagenome_*, atlas_*) — RM191–RM200

AlphaGenome is the only source in this tier reached on two surfaces that are not the same source. One is a bulk artifact an operator already holds, re-encoded into a cache lane and read offline; the other is a gRPC service queried per variant. They answer different questions, carry different terms, and — since RM200 — write two different sources.csv rows, because one (source, layer) key cannot carry two licence classes.

Surface Module What it answers Licence class
AVI artifact → the alphagenome_avi lane alphagenome_avi_build how much does AlphaGenome think this variant matters, for 8.8 billion SNVs, offline Permissive Use (RM195), commercial_use=True — so a module drafted from it stays sellable
Atlas API (gdmscience.googleapis.com) atlas_client, atlas_protos the scores the artifact discarded — per gene, per tissue, with direction non-commercial only; an undeclared run writes nothing

The split is not a convenience. ALPHAGENOME_AVI_TERMS and ALPHAGENOME_ATLAS_TERMS are separate SourceTerms constants under separate source names, and a module that used both would declare both rows — which is the only way sources.taints_commercial_use can reach the right verdict. Reading the Atlas under --use unstated is refused at acquisition, not at write time (@acquisition-gate-is-not-a-read-gate).

alphagenome build — the artifact, re-encoded (RM191, RM197, RM198)

Nothing here fetches, and that is not the usual inject-only rule — this is the network tier, so it is allowed to. The AVI artifact is 88.5 GB behind a sign-in whose eligibility clause bars whole classes of holder, so acquisition is the operator's own act under their own acceptance of the terms. --input is required and has no default URL, which is that rule expressed as a flag.

Three properties of the built lane a consumer has to know, each of which is a measured decision rather than a format choice:

  • The scores are integers and should be compared as integers. raw_score is Int32 at a scale of 10⁵ — exactly lossless, since the source prints at most five decimals. Recovering a float with raw_score_e5 / 1e5 disagrees with the printed value on 53% of rows, because the division rounds a second time. score >= 0.1 is raw_score_e5 >= 10_000.
  • PHRED is not stored, and the file that reconstructs it is not optional (RM198). It is an exact within-corpus rank, so avi_knots.parquet carries the curve in 466 KB instead — as an interval per printed score, which makes threshold safety decidable in advance: a threshold is unsafe iff it lands inside a knot's span. Genome-wide exactly one does, at 3. A snapshot without that file holds scores nobody can rank, and alphagenome check refuses it. It is a sibling of data/, which is why locations.SNAPSHOT_ROOT_FILENAMES exists. Both halves walk that tuple — the publisher did from the start and the puller did not until RM209, which is why a pulled lane held scores nobody could rank while two docstrings said the registry was walked.
  • The artifact is wide by position (RM197): one row per locus — chrom, pos, ref, alt0, alt1, alt2 — and no alt column. Which base each column means is {A,C,G,T} − ref ascending, a function of ref alone, so nothing has to travel beside the data. alphagenome_avi_build.to_long() recovers (chrom, pos, ref, alt, score) rows; apply it to a filtered frame, since over the whole corpus it is 2.9 billion loci becoming 8.8 billion rows, which is the shape the layout exists to avoid storing.

The lane is the first with a builder and rebuild=None: this tier cannot fetch the source, but the re-encoded snapshot is publishable, so an operator who may not download the artifact can still cache pull it. Having a builder and being rebuildable unattended are two different properties (@a-cache-lane-has-three-stages-and-a-list-cannot-say-which-are-missing).

alphagenome check — the Atlas as a resolver, for the three questions the artifact cannot answer (RM193)

Attests as variant_impact_agreement, its own member rather than a second writer of reference_allele — the Atlas answers that too, and two registries answering an overlapping question get two names, or one source's outage writes a skip against the other's (@one-registrys-outage-may-not-speak-for-another). Reports, never repairs. Mostly offline: without --threshold there is no question the local snapshot cannot answer, so the live leg is the exception rather than the path. Raises VariantImpactError / VariantImpactUnavailable — see Exception contract.

alphagenome expression — the axis the AVI artifact threw away (RM194 + RM200)

Fills expression_effects.csv: one row per (variant, gene) saying which way a variant moves that gene's predicted expression, how many of AlphaGenome's 371 tissue tracks agree, and how far the variant sits from the gene. A recording pass — it writes a derived sidecar, compares nothing authored, and therefore has no verification-check member.

One scorer of twenty-two, and the other twenty-one are a measured refusal. RNA_SEQ is the only one with a gene axis — its shape is (genes, tracks), the gene count varying with the window — so AlphaGenome makes the gene attribution itself, which is what @gene-map-is-another-sources-attribution requires. CHIP_TF looked like the useful one and was refused on measurement: over 751 factors the top three carry 1–3% of the mass, the concentration does not track effect size, and every leading factor is a singleton track, so the "top TF" is whichever one happens to be measured once. *_ACTIVE is an activity level rather than a variant effect — it describes the locus, not the variant.

--gene is mandatory in both span forms, because the server-side gene filter is a requirement and not an optimisation; an unfiltered interval query is refused before it is sent. The span comes from --chrom/--start/--end when given, otherwise from the MANE lane widened by the ±512 kb horizon — the Ensembl snapshot is eliminated by its own schema, which has no gene column at all.

distance_to_gene is the column that makes the table usable, and null is one of its values. Distal scores run ~10× lower than scores at the gene, so a flat --min-score keeps only the proximal rows while looking like it filtered on effect — the failure this pass exists to prevent. The MANE lane is consulted even when the interval was supplied by hand, because those are two questions and an explicit interval only answers the first; with no lane the column is null and the manifest publishes without_distance, so a whole table of them is visible before a join. Never an interval edge (@a-derived-lane-has-parents-and-an-absent-parent-is-not-an-empty-result).

--max-rows defaults to 50,000 and refuses rather than truncating; raising it is a deliberate act. --offline is a no-op with a warning — this pass reads the Atlas, not a snapshot, so there is no offline answer to give and pretending otherwise would report nobody-asked as nothing-found.

atlas generate — the bindings, and why neither the sources nor the generated code is committed (RM192, RM196)

uv add alphagenome costs 550 MB and 47 packages against a tier whose entire runtime list is httpx/tenacity/huggingface-hub, and six of the twenty dependencies that wheel declares are never imported on any scoring path. The .proto sources are Apache-2.0, so grpcio + protobuf reach every Atlas RPC — 19 MB, the figure enricher/pyproject.toml measured in a clean venv — with score payloads decoding through struct.unpack from the standard library. (22 MB appears in older text and is the grpcio release current at the design round; RM221 swept it.) That is the [atlas] extra; the alphagenome extra is deleted.

The repository carries neither the upstream sources nor the generated bindings. It carries the pin — a commit id and a sha256 per file — which is what makes a fetch verifiable rather than merely convenient, and what keeps a copy of somebody else's file from going stale silently. atlas generate runs once per checkout; a released wheel carries them already. This is also why just-dna-enricher's build backend is hatchling rather than uv_build (RM196): the bindings are generated at build time and uv_build has no build hook. Backends are declared per package, so the other two tiers are untouched.

Three defects only real execution could find, each one a case where every offline fixture agreed with the code that built it: the Atlas wants chr22 on the wire where a module stores 22 (now fixed by the client owning the spelling, as it already owns the 0-based/1-based conversion); a snapshot's summary.parquet lives under data/ rather than at its root; and a lane directory existing is not the same as a lane being built. A fourth is upstream's: the server returns a next_page_token on an exactly-full final page and following it answers INVALID_ARGUMENT, so a faithful client crashes on the one interval width that divides evenly — score_interval follows the token but also stops once the requested interval is covered.

And one that cost a day: Interval.strand has no zero member. STRAND_UNSPECIFIED = 0 is the proto3 default, so an omitted strand goes on the wire as a value the server rejects, reported as a bare INVALID_ARGUMENT naming no field. Neither the x-goog-fieldmask header nor 32 bp chunking was ever required — both were measured and both are the SDK's own choices.