just-dna-compiler — the transform tier¶
The package reference for just-dna-compiler: the reference compiler that turns a validated spec
directory into a multi-parquet artifact + manifest.json, and reverses it back. Since 0.5 it is
pure-Python and duckdb-free — its runtime deps are just-dna-format, polars, pyyaml, typer.
It never fetches (Principle 2): resolution is consumed from an injected, source-independent
resolution.csv; the pre-0.5 DuckDB reference path moved to just-dna-enricher and is
reachable only through a deprecated, guarded shim (removed at 1.0).
The compiler adopts the schema with a C++-standard-style feature-coverage stance — not all-or-nothing conformance but a per-feature table (below). As of 0.4 the validator is complete, the upgrade derivation ships, the artifact round-trips losslessly including phase, and all nine 0.4 table kinds materialize with enforced table-level coherence.
Companion docs: SCHEMAS.md (the models it compiles), ENRICHER.md (what produces
resolution.csv), CONSTITUTION.md (the invariants).
Read beside this: the 2026-09-11 code-first re-derivation¶
A second reading of this tier, written from the code alone on 2026-09-11, is in audit/COMPILER_FROM_CODE.md — the pipeline in execution order, per-parquet column lists, and a validation table with validate/compile/severity columns. It is the instrument that found RM93 and RM94 in 0.6.1 and RM207 in 0.7 — the last of which is why the parity this document asserts is a property of the binaries rather than of its own prose. Evidence, not contract: this document is the maintained one. The method is BLIND_REDERIVATION.md.
Public API¶
Import from just_dna_compiler.compiler.
validate_spec(spec_dir, authority_keys=None, *, strict=False) -> ValidationResult— validate a spec dir without producing output; strips inject-only authority keys pre-validation (dropped keys →.info), runsvalidate_binsand the duplicate/identity checks, populates.stats.strictgrades the mode-ladder findings at the severity astrictcompile would, so a caller can report what strict would refuse without building anything — note it cannot see the unresolved-position gate, which lives incompile_moduleand needs a resolution table.content_signature(spec_dir) -> str— the stable, name-/Ensembl-independent content identity over the raw authored data CSVs (no compile, no resolution); raisesValueErrorif a present data CSV is invalid. See SCHEMAS.md § identity & integrity. Which CSVs, exactly:variants.csv,studies.csv, and any present table kind — the authored roster. The licensing table (sources.csv/licensing.csv) is outside it: it is hashed byintegrity.source_signatureinstead, so neither its presence nor an edit to anoticecell in it moves this digest. That is correct — the licence layer carries its own identity — and it is stated because it is the one authored, hand-editable table a licence audit sends an author looking for (S53).load_spec(path, *, authority_keys=None) -> ModuleSpecConfig— the public route to a parsedmodule_spec.yaml, raisingSpecError(aValueError) on anything wrong. Sibling ofjust_dna_format.read_manifest/read_verification: one loader per file a module carries, same raise-don't-return contract. It exists because the model was public from the start and the only thing producing one was_load_yaml, so a consumer wantingweighting:,authorship:orlicensehand-parsed the file, lost the authority-key handling and every diagnosis, and carried PyYAML purely to work around a private symbol — the enricher itself was doing the same thing across the tier boundary (S74)._load_yamlkeeps its(config, errors, dropped)tuple forvalidate_spec, which accumulates diagnoses from a dozen sources; that difference is why both exist. It is in this tier rather than the format one because parsing YAML needspyyaml, and the format tier ispydantic+cryptographyby charter. It does not folddefaults:— that is per row and isspec_tablesbelow; the pairing isload_specfor the yaml's blocks,spec_tablesfor the rows.spec_tables(spec_dir) -> tuple[dict[str, list[BaseModel]], str]— the parsed, defaults-folded rowscontent_signaturehashes, plus the declared build.content_signatureis exactly this plus the hash, so anything finer than a whole-module digest — per-table, per-row, what moved between two versions — builds on these rows instead of restating the roster and the fold (S53). SameValueError-on-invalid-CSV contract. The fold is why this returns the finished mapping rather than exporting the pieces: a caller hashingload_csv_rowsoutput directly gets a different digest for the same module, because a value written once underdefaults:and the same value written on every row are one content (RM37) and only this path folds them — measured onhfe_hemochromatosis, the raw build reports twelve changed rows where there are none.compile_module(spec_dir, output_dir, compression="zstd", resolve_with_ensembl=True, ensembl_cache=None, compiled_by=None, ensembl_reference=None, log_files=None, provenance_file=None, logo_file=None, readme_file=None, authority_keys=None, strict=False, ba1_threshold=0.05) -> CompilationResult— compile to parquet +manifest.json.ensembl_cacheis deprecated (removed at 1.0) — see the precedence block.resolve_with_ensemblis not deprecated and is misnamed: since 0.5 it is the master switch for consumingresolution.csvat all, so turning it off ignores an injected table and has nothing to do with Ensembl.ba1_thresholdis the ACMG BA1 allele-frequency cutoff — a parameter rather than a constant, because the right value is disease-specific.-
reverse_module(parquet_dir, output_dir, module_name=None, title=None, description=None, report_title=None, icon="database", color="#6435c9", version=None, write_resolution=True, genome_build=None) -> Path— reverse a compiled artifact back to the authored DSL.genome_build=Nonereads it from the artifact's ownmanifest.json(it lives in no parquet column); pass it only for a bare parquet directory carrying no manifest. -
load_csv_rows(path, row_model, file_label, genome_build=DEFAULT_GENOME_BUILD) -> (rows, errors, warnings)— the authored-CSV loader, public since 0.5.1 (RM41). It was_load_csv_rows, and it was public in practice:just-dna-enricherconsumes it across a package boundary in a dozen places, and a consumer wiring the pipeline server-side had the choice of a private symbol or a re-implementation. Re-implementing is a trap, not a chore — it is notcsv.DictReaderplusModel(**row), because an empty cell becomesNonewith the key kept (so a defaulted-but-not-Optionalfield likeMeasureBinRow.measure_kindreceivesNonerather than its default and fails on type) andgenome_buildis told to each row rather than read from it._load_csv_rowsremains as an alias, so nothing that imported it breaks. -
load_spec_variants(spec_dir) -> (variants, errors, warnings)— a spec directory'svariants.csv, loaded with the build the module declares and re-stamped for it. Three steps, not one: readgenome_buildout ofmodule_spec.yaml, inject it into every row, then_restamp_for_build— becauseVariantRow._freeze_identityruns at construction, where the yaml is not in scope, so a loader that skips either step mints GRCh38 identities for a GRCh37 module. Missing or unreadable yaml falls back toDEFAULT_GENOME_BUILD, matching what compiling that directory would assume; this is a read-only check helper, unlike the enrichment path, which refuses rather than choose a build for a module whose declaration cannot be read (it writes facts back). Added for the two enricher checks that take rows rather than aspec_dir—verify_acmg_sfandcheck_identifiers, which now accept both. -
load_citing_rows(spec_dir) -> dict[str, list]andtable_citations(rows_by_csv) -> list[str]— every table kind beside a spec whose rows cite (a model declaring apmid: the four binning kinds since RM47,pharm_variants.csvsince RM132), and the digit-only PMIDs they name, de-duplicated in first-occurrence order. Public for the RM40/RM41 reason: the enricher's literature pass has to check these pointers alongsidestudies.csv, and its alternatives were a private import or a second roster that goes stale the next time a model declares the column. The set is derived from the models, so a new citing kind is read by both tiers with no edit to either. Row errors raise — a pass reading citations out of a table it could not parse would silently under-report; a caller wanting the per-row diagnosis hasvalidate_spec.load_binning_rows/binning_citationsare the narrower pair, unchanged: they answer over the binning kinds only, because a caller asking for those is asking about thresholds rather than about the citations a module makes.
models.py: ValidationResult (valid, errors, warnings, info, stats) — the .stats key
contract is variant_count/unique_rsids/gene_count/genes/categories/study_count/
clinvar_count/pathogenic_count/benign_count/module_name. CompilationResult (success,
output_dir, errors, warnings, stats, manifest — the emitted ModuleManifest, None on failure).
What the compiler can and cannot validate¶
The compiler sits to the enricher roughly as an assembler-plus-linker sits to a source tree: it consumes an authored schema plus injected resolution facts, and its guarantee is that the result is well-formed and self-consistent, not that it is true. A C++ compiler type-checks a program and still ships your off-by-one; a linker proves every symbol resolves and says nothing about what the functions do. The same boundary applies here, and it is worth stating plainly rather than leaving a reader to infer that "it compiled" means "it is right".
There is also a trust boundary. resolution.csv and the derived-fact sidecars — frequencies.csv,
gene_metrics.csv, literature.csv, gene_validity.csv, clinical_assertions.csv — are consumed as
fact. The compiler can re-derive the parts that are self-verifying and cross-examine the
parts that are redundant, but everything sourced — which coordinate, which rsID, which allele
frequency — is taken on trust from whoever produced it. That trust is the price of Principle 2: a tier
that never fetches cannot independently confirm a fetched fact.
Three things it can check, in increasing strength¶
1. Formal conformance — complete within its domain, like type-checking. Columns, types, closed
vocabularies (extra="forbid" plus the reserved-namespace diagnosis), identifier grammars (rsid, DOI,
PMID, CURIE, ga4gh:VA., CA\d+), value bounds, finite floats (no NaN), required-ness, at most one
unresolved bin sentinel per bin-group key, bin overlap/gap, duplicate natural keys, duplicate
(variant_key, genotype). If a module violates one of these it is malformed, full stop.
Note the one-sidedness of the sentinel rule, because it reads like a completeness check and is not one:
_validate_table_kind counts sentinels per group and refuses a second one, and nothing on the
compile path refuses zero. A binning table carrying no sentinel at all compiles green under
--strict — measured. The missing-sentinel finding lives on the authoring surface only
(hints._check_bins warns no unresolved sentinel row), and it is table-level, so a table whose rows
fragment into several bin groups can satisfy it while leaving most groups with no sentinel.
2. Validate-by-redundancy — where two independently-authored things must agree, disagreement is detectable without any reference. This is where most real authoring bugs are caught:
| Check | Redundancy exploited | Severity |
|---|---|---|
| rsid ↔ coordinate | the pair co-identifies one variant | warning / error in strict |
| inconsistent position for one key | one key, one place | error |
inconsistent ref for one key |
the reference base is a single fact | error |
genotype alleles ⊆ {ref} ∪ alts |
a genotype names alleles the locus has | warning / error in strict |
effect_allele ∈ {ref} ∪ alts |
the effect allele is one of them too | warning / error in strict |
clin_sig pathogenic + high AF |
ACMG BA1: common alleles are not pathogenic | warning |
| a citation PubMed has no record of | the enricher already wrote the verdict down | warning |
allele_count ≤ allele_number |
a count cannot exceed its denominator | error |
2 × homozygote_count ≤ allele_count |
each homozygote contributes two alleles | error |
faf95 ≤ the group's own AF |
a CI lower bound sits below its point estimate | warning |
oe_lof_lower ≤ oe_lof ≤ loeuf |
an estimate lies inside its own interval | warning |
obs_lof / exp_lof == oe_lof |
the same quantity, stored three ways | warning |
| direction ↔ weight sign | two encodings of one claim | warning |
p_value string ↔ the mantissa/exponent pair |
two encodings of one number | warning / error in strict |
| MT/Y two-allele genotype | ploidy contradicts the contig | warning |
| study / frequency / gene-metrics orphans | the sidecar describes something the module lacks | warning |
| literature orphans | a citation no study and no citing table row makes — reported, and left out of the artifact (the row stays in literature.csv) |
warning |
sources.csv orphans / undeclared sources |
every source a fact table cites has terms recorded, and vice versa | warning |
| star allele used but not defined | allele_function/diplotypes name it; haplotypes defines it |
warning |
| phase-ambiguous diplotypes (0.5) | two different haplotype pairs whose unphased genotype is identical while their conclusions differ | warning |
Four of these deserve their reasoning rather than just their row.
The phase-ambiguity check is RM28's cis/trans motivation, closed by computation rather than by a
grammar. Compound heterozygosity is the case that most justified a predicate language, and
reference_examples/hfe_compound_het/ shows it needs none: a diplotype is already a statement about
two homologs, so HFE C282Y/H63D (in trans, at-risk) and C282Y-H63D/wt (in cis, a carrier) are
simply two rows. What no table can say is that a consumer without phase cannot tell them apart —
they present the identical unphased genotype (rs1800562 G/A, rs1799945 C/G) and disagree about what it
means, and nearly all consumer data is unphased.
That is derivable from tables the compiler already holds, which is what makes it a check rather than a
requires_phase column: a column would restate what the data determines and go stale the moment a
haplotype is edited. The signature is, per variant, the sorted pair of alleles the two haplotypes
contribute — sorted because that is precisely what losing phase does. An unmentioned variant reads as
the implied reference, and an allele explicitly equal to the row's own ref normalizes to the same
sentinel; neither needs the reference sequence, only that "unmentioned" means one thing. So it runs
unchanged on a CPIC-drafted table where haplotypes are sparse and ref is absent — and correctly finds
nothing there.
Two boundaries. It compares distinct haplotype pairs, never distinct rows: the dedup key admits a
row per drug and per clinical_context for one pair, and grouping on rows reported 595 ambiguities in
a CYP2C19 module that has none. And it is closed-world — it compares the rows a module states, not
the ones it omits, which is why APOE stays quiet despite ε2/ε4 vs ε1/ε3 being the textbook collision:
that module carries no ε1. The neighbouring "used but not defined" warning covers that side.
The allele-membership checks never escalate to an unconditional error, and the tempting version is
wrong. An authored ref+alts contradicting the row's own genotype looks decidable here — but
ref/alts in variants.csv are not necessarily human-authored. reverse_module writes them, and a
one-to-many rsid reverses into N rows that each carry their own locus's alleles beside the one
genotype the author wrote; exactly one of those rows can match. An unconditional error would mean any
module with a one-to-many rsid compiles once and never again — Principle 7's fixed point, broken by a
lint. (Not hypothetical: three variants in reference_examples/pathogenic_clinvar/ have this shape.)
For the resolved case there is a second reason to stay at a warning: alts came from a source, and
ClinVar carries only its submitted alleles while Ensembl carries every allele dbSNP knows, so a short
alt list is a gap in the source at least as often as a defect in the module. The comparison is also made
against the union of every locus a key resolves to, never per-expanded-row, for the same reason.
"Exactly one of those rows can match" is a statement about matching, and a reader does more than
match (S33). The sentence above is right and the expansion is staying, but it was written in the
authoring/validation frame, where an unmatchable row is inert. It is not inert to a consumer reading
the artifact: TA/TA beside ref=TA is a well-formed reference homozygote carrying the module's
conclusion, and a reporting consumer classified 2,579 of them as pathogenic genotypes a subject
carried. Two things came out of that, neither of them a filter. The read-side contract is now stated
where a consumer will meet it — SCHEMAS § the consumer join contract
— and manifest.compilation.expanded_keys/expanded_rows publish whether an artifact contains such
rows at all.
And since RM87 an expanded row says so about itself. weights.parquet carries locus_index and
locus_count, stamped at the expansion loop, which is the only site that knows a row is a member of
anything — after it the row is ordinary, with a real coordinate, a real reference allele and the
module's own conclusion. locus_count > 1 is the predicate a consumer applies while holding a single
row; locus_index is the 0-based ordinal that lines that row up with its resolution.csv row. Four
things about the pair:
- The ordinal counts within
usable, i.e. after_hostable_locidropped any locus that positively contradicts the genotype — not within the injected table's ownlocus_index, which_sorted_locionly orders by. Understricta dropped locus is a refusal, so the two coincide there. locus_countdefaults to1, not0. A row that was never expanded genuinely resolves to one locus. A zero default would make the predicate readlocus_count > 1 or locus_count == 0, which is the same under-determinationlocus_indexalone has and which the columns exist to remove.- Reverse prefers the stored column and keeps its encounter-order recompute as the fallback. An
artifact compiled before 0.6 has no column and Principle 3 requires it to keep reversing, so
_write_resolution_csvreadslocus_indexwhere it is present and counts where it is not. A test reverses each expanding reference example twice — once with the column, once with it stripped — and asserts the tworesolution.csvfiles are byte-identical; that comparison is what pins the sort dependency the recompute silently relies on. The counter itself is maintained either way, because the positional pass below uses membership in it to enforce weights-first. - Neither column moves a
content_signature, and neither is re-emitted intovariants.csv. They areexclude=Truecompiler-managed columns recording what resolution did.
The positional tables' pass still writes a hard-coded 0, and that is honest only while those
tables never expand — true today, since RM43's fill is one locus per row. It stops being true if RM65
ever puts coordinates on the repeat and copy-number tables, and the RM65 entry carries that line.
The expansion warning is one sentence per rsID, over every authored row at it. It used to be
emitted inside the per-row loop, so a site with two authored genotypes published the identical
sentence twice and each copy said "expanded to 2 rows" of an artifact that had gained four. The union
of loci is what "maps to N loci" counts and the sum over authored rows is what "expanded to M rows"
counts — the two come apart exactly when an author writes more than one genotype at a key, since
_hostable_loci judges hostability per genotype and two rows at one key can legitimately reach
different loci. The deprecated ensembl_cache path keeps the old per-row shape deliberately; it is
removed at 1.0 and its modules report the counts as None.
BA1 is a warning in both modes and its threshold is a parameter (compile_module(ba1_threshold=…),
default 5%). The right cutoff is disease-specific — sickle-cell's rs334 sits at a filtering AF of
~0.048 in African-ancestry groups, just under the line, and a common recessive carrier allele
legitimately sits above it. Failing a compile over that would be the format arbitrating a clinical
judgement.
The sources.csv orphan half exempts the layers with no source column to join, and there are two
of them. "No table used it" is decided by reading the generated fact tables' source columns
(resolution.csv contributes its authority, not its link — RM33). The annotation layer is
variants.csv/diplotypes.csv/…, which carry none by design, so an annotation-layer row can never be
corroborated and used to be reported as stale on every drafted module — the exact row the licence gate
keys on. literature joined that exemption in 0.5.4 (S23) whenever the module carries studies.csv
rows, by the identical argument: studies.csv is the hand-curated literature table and has no source
column either, so a module citing a PMID through it can only be corroborated by the enricher-written
literature.csv, and a module with none has nothing to join. The old behaviour inverted the incentive
where it matters most — vocab.MISPLACED_COLUMN_REASONS['source'] tells an author to declare a
hand-read source by adding a row here, doing so earned a warning that the row was unused, and deleting
it (shipping with the provenance unrecorded) was silent. Compliance warned, omission quiet. It stays
narrow: frequency still warns, because frequencies.csv is machine-written with a source column,
so a frequency declaration in a module carrying no frequencies really is stale. The undeclared half —
a source a fact table cites with no row — is unaffected and warns in every case, and neither half ever
escalates: over-declaring terms is the cheap error, and an author talked out of recording theirs is not.
A coordinate that cannot exist is refused in both modes (RM48, 0.6). _check_build_coordinates
asks one arithmetic question of every row carrying a chrom and a start — could this position exist
on this contig in the build it is recorded under? — and two shapes answer no provably, with no
sequence, no network and no provisioned asset:
- a position past the end of its contig. GRCh38's chromosome 1 ends at 248,956,422 and GRCh37's
runs 294 kb further, so an un-lifted hg19 coordinate in that tail names a base that does not exist.
When the position is inside another build's contig of the same name, the message says which, and
points at
just-dna-enricher hint recover— that is the whole diagnosis, and it costs a dict lookup. - a contig only one build names. The 25 primary contigs are spelled identically in both builds, so
this is entirely about unplaced scaffolds:
GL000209.1is GRCh37's andKI270728.1is GRCh38's.variants.csvrefuses either at the model (itschromvocabulary is 1-22/X/Y/MT and always was — what 0.6 added there is that the rejection names the build); this reachesstudies.csv, the PGx tables,heteroplasmy.csvand the injectedresolution.csv, none of which validate the contig.
It is an error in both modes — the inconsistent-reference-allele class, not a mode ladder. strict
means reproducible artifact, and these rows are not unreproducible, they are false. Nothing
downstream catches them either: a VRS id minted at an impossible position is a correct digest of the
wrong input, which is how a 3,038-row off-by-one once passed every gate including --strict.
Three things it deliberately does not do. It says nothing about a low position — VCF writes POS 0
for a telomeric variant, so only the upper bound is consulted. It withholds on every contig the
tables do not settle: a shared scaffold (GL000194.1), an unversioned accession (GL000205, where the
suffix is what separates the builds), a patch or an alt locus. And it judges each resolution.csv row
against that row's own genome_build column rather than the module's, because that column exists to
say which frame the numbers are in. Findings are grouped by reason — a whole panel authored on hg19 is
one line, not one line per variant. The tables live in just_dna_format.vrs
(PRIMARY_CONTIG_LENGTHS, CONTIGS_ONLY_IN) beside PAR_GRCh38, for the same reason: assembly
constants the compiler needs offline. They carry no refget accessions — a second build's identity
is RM15, and refget_accession still raises for GRCh37.
3. Content-addressed self-verification — the strongest class, because the stored value is a pure
function of other stored values, so a disagreement is provable corruption rather than a difference of
opinion. artifact.digest, content_signature, the three fact-signatures, the Ed25519 signature —
and, since 0.5, vrs_id. Moving allele identity into this class is what the VRS work bought: a
ga4gh:VA.… used to be an opaque cross-reference that had to be believed, and is now a checksum the
compiler recomputes from the coordinate with no dependency and no network.
The inescapable blind spots¶
These are not gaps this tier can close. Each follows from what the compiler is — a transform over injected data — and pretending otherwise would be worse than saying so.
That is a claim about the compiler, not about the ecosystem, and the distinction matters now that two
of the rows below have moved. The enricher holds references the compiler never will, so it can
answer questions this table calls unanswerable — an authored clin_sig against ClinVar's, a cited PMID
against PubMed, an rsID against dbSNP. What stays true is the division of labour: the compiler's own
blind spots are permanent, what it cannot validate it makes legible, and a check that needs a
reference lives one tier up. The "What the format does instead" column below records which tier now
covers what.
| Blind spot | Why it is inescapable | What the format does instead |
|---|---|---|
Is a single-sourced number right? An AC/AN, a pLI, a clin_sig — one source, no redundancy to exploit. A transcription error is indistinguishable from a correct value. |
Nothing to check it against without fetching (Principle 2). | Records dataset (which release) and source (which link) so the number is attributable, and fact-hashes it so it cannot change unnoticed. |
Is the reference base right? A wrong single-base ref mints the correct VA, so the artifact is self-consistent and wrong. |
The compiler holds no sequence. | The enricher checks it (sequences.verify_reference_alleles); the compiler catches only two rows contradicting each other. |
Is an indel's vrs_id right? Cannot be recomputed without justification against the sequence. |
Same. | Reported as unverifiable (never as verified), and carried with that said out loud — a warning in both modes, since no authored edit could clear it. |
| Is the coordinate the variant the author meant? A perfectly valid VA for the wrong locus is indistinguishable from the right one. | Requires knowing intent. | provenance.json, authorship, and the studies table make the claim auditable by a human. Narrowed in 0.6 (RM48): a coordinate that could not exist in the declared build is now refused offline, and one that reads as the old assembly is diagnosed by the enricher against the live GRCh37 service. Neither reaches intent — a well-formed GRCh38 coordinate for the wrong GRCh38 locus is still invisible here. |
Is the annotation medically correct? Whether A/T at HBB → sickle-cell carrier is true. |
Out of scope by charter — the format supplies annotation tables and never a gene–disease inference. | authorship.kind lets a consumer route scrutiny (AI vs human-certified); curator/method record who decided. |
Does the cited study support the row? pmid is grammar-checked; nobody reads the paper. |
Requires the literature. | Partly closed by the enricher (0.5). Its literature pass confirms the PMID resolves, cross-fills the DOI/PMCID, and matches provenance_quote/provenance_regex against fulltext — for the open-access subset only, with coverage reported as a fraction so an unread paper is never mistaken for a failed quote. The compiler still reads nothing; it surfaces the recorded verdict from literature.csv. |
| Is the source stale? A v2.1.1 constraint number is well-formed and current-looking. | The compiler cannot see the world move. | dataset names the release; the gene-metrics pass labels its two routes differently and warns on the older one. Generalized to identifiers in 0.5: the enricher checks rsIDs against dbSNP (live/merged/absent), trait CURIEs against OLS4 (obsolete + replacement) and gene symbols against HGNC (approved/retired). All report; none rewrite. Extended in 0.5.4 to the relationship between two identifiers (S24) — a row's gene against the chromosome its variant sits on — because both halves can be individually valid while the pairing is fabricated. |
Is acmg_sf right? A gene-list-membership flag the compiler holds no list for. |
Same shape as clin_sig: the list is not in the module and cannot be, since a gene list inside the compiler is an un-injected reference (RM21). |
Closed by the enricher (0.5). acmg.check_acmg_sf compares the flag against ACMG SF v3.2 as NCBI publishes it. Warns in best_effort, refuses in strict — list membership is a published fact, not a clinical judgement, so unlike clin_sig it does escalate. A blank cell is a note, never a defect. |
Is the annotation medically correct? — the clinical half. Whether the module's clin_sig is the right call. |
Out of scope by charter (below), and ClinVar is not truth either. | Surfaced, never adjudicated (0.5). The enricher compares each authored clin_sig against the ClinVar snapshot's, allele-exactly, and reports opposed calls with ClinVar's review-star count. It is the one check whose severity does not escalate in strict: failing there would make the format decide a clinical dispute. |
Did the author declare every source they copied from? A copied annotation with no sources.csv row is indistinguishable from an original one. |
Provenance of a text is not a property of the text. | The enricher writes the row when it fetches (it is the only tier that knows); sources.csv + manifest.sources make the declaration legible and hashed. The compiler warns on a source a fact table cites with no row, but cannot see what was copied by hand. |
| Did the enricher get it right? The resolution table is consumed as fact. | The trust boundary itself. | source, status, resolution_mode, fully_resolved and resolution_signature make the provenance and the policy legible. |
The through-line: what the compiler cannot validate, the format makes legible. It records who produced a fact, from which release, under which policy, and hashes it so it cannot drift silently — then leaves the judgement to a consumer. That is the data-agnostic north star applied to trust.
One gap that was not inescapable — where a bin boundary came from (S19/RM47, closed in 0.6)¶
Everything above is a limit of the tier. This one was a limit of the schema, and it is kept here because the distinction is the point: a tier limit is permanent, a schema limit is a release away.
studies.csv is required iff variants.csv is present, so grounding was enforced exactly where
citations usually arrive already attached (a ClinVar-drafted variants.csv) and absent where a human
made the judgement: reference_examples/htt_repeat_expansion compiled green under --strict asserting
where Huntington disease becomes fully penetrant, with no citation anywhere. A StudyRow named a
variant — rsid, or a bare chrom — and a repeat_alleles.csv row is keyed (gene, repeat_unit), so
nothing could point at it.
0.6 closed it with a second citation site: MeasureBinRow.pmid, one optional column on the binning
base reaching all four kinds, plus a relaxation of StudyRow's subject requirement so the paper behind
a threshold can be described without inventing a variant for it. The rule for reading the pair is the
bin row cites, the citation table describes — the pointer sits on the row that states the number, and
everything about the paper stays in studies.csv. heteroplasmy.csv had a second route from the
start: its optional rsid/chrom/start columns (0.5.1) give a row a variant identity a study row
can name, which reference_examples/mt_heteroplasmy does — but that route is the study row, so it
grounds a bin only in a module that carries one.
_check_binning_grounding still warns in both modes, over the bins that carry no pmid, in a
module with no study rows at all — and the remedy it names is the same for every kind, since every kind
can now cite its boundary; a kind whose rows can be pointed at is offered the study-row route as a
second one. It used to exempt a bin for merely naming a variant, inside a branch that has already
established the module records no study rows, so the citation clearing the bin was one that does not
exist: a heteroplasmy.csv module stating four thresholds and citing nothing was green and silent while
the identical thresholds on repeat_alleles.csv were reported (D1-3, fixed in 0.6). The same-release
obligation was the reason the
item was filed rather than fixed: _cross_check_literature reads the bin pointers alongside
studies.csv (otherwise every threshold-grounding citation would read as a stale orphan), and so does
the enricher's literature pass, so a bin-grounded citation is checked for existence and identifiers
exactly like a study-grounded one. reference_examples/htt_repeat_expansion is deliberately left
uncited: the example exists to show what the warning looks like.
0.7 added a third site the same way (RM132), and stopped listing them. PharmVariantRow.pmid
grounds one drug/genotype/category claim, which studies.csv cannot: a study row keys on
(variant_key, pmid) and would attach to every claim recorded for that variant at once. The lesson
above is now enforced structurally rather than remembered — _CITING_TABLE_KINDS is every
_TABLE_KINDS model declaring a pmid, and table_citations/load_citing_rows walk it, so the next
kind to declare the column is read by both tiers with no edit to either. _check_binning_grounding is
untouched by this and stays about bins: there is deliberately no equivalent grounding warning for an
uncited pharm row, because a drug-response table is not the interpretive-threshold case that check
exists for.
One paper, several analyses, and the dedup key (RM140)¶
The three sites above are about which row cites a paper. This one is about what a citing row's
numbers mean once it has. A studies.csv row carries p_value and effect_size side by side and
asserts they belong together; before 0.7 nothing recorded, and so nothing could check, whether they
came from the same analysis. study_design names the study — case-control, GWAS, meta-analysis — and a
single study routinely reports several analyses of one association: the motivating module cited a paper
giving OR 1.4, p 0.36 from an allelic Fisher's exact test and OR 1.42, p 0.75 from a univariate
logistic regression of the same variant, and two agents building from it produced rows that differed
only in p_value, one of them pairing the magnitude of one analysis with the p-value of the other.
Everything was green — including quote verification, because the quote grounds the significance
verdict and contains no statistic to witness.
StudyRow.statistical_test (0.7) is the column, and it is deliberately not a gate. Free text like
study_design, no vocabulary, no new check. What it changes is one existing one:
duplicate_study_citation fires on a repeated (variant_key, pmid) because the check's own reading of
that pair is the same claim written twice — which two rows naming two analyses are not.
Only both stated and different suppresses the warning. The rule is Kleene rather than a != b:
an absent statistical_test is unknown, and unknown against a stated value cannot establish that
two rows describe separate work. So a pair where neither row names an analysis, where both name the
same one, or where one names one and the other does not, warns exactly as it did before, with the same
code and the byte-identical message — every module published before 0.7 behaves as it did. A row
repeating an analysis already stated for its key still warns however many distinct siblings sit beside
it.
StudyRow._KEY_FIELDS is not widened, and the divergence is contained in this check. That tuple
drives hints.key_fields and the key.columns an authoring surface publishes; re-keying a shipped
authored table changes what an identity key means, which is major-only under Principle 3. The check
restates (variant_key, pmid) rather than reading _KEY_FIELDS, so splitting it here splits nothing
else.
Three more schema limits, made legible the same way (0.6, the VCF 4.4 audit)¶
Same class as the bin-boundary gap above — limits of the schema, not of the tier — and they are here for the same reason: so they are not mistaken for the other kind, and so the warning a reader meets on a real module has somewhere to point. All three warn in both modes and none changes a verdict.
The two integer measure kinds are not integral (RM55) — warned, and then fixed in the same line.
VCF 4.4 §7.2 redefined CN to support non-integer copy numbers and §3 types RUC as a Float, so the
premise repeat_count and copy_number were placed in binning._INTEGER_KINDS on has been withdrawn
for both. The consequence was RM35's unsatisfiable triangle re-instantiated on the kinds RM35 exempted,
and worse: under a grid a hole of exactly one is not reported at all, so [0,0] [1,1] [2,2] [3,∞) is a
legal, gapless, green tiling under --strict that answers nothing for a CN of 2.4 — and the schema also
refused the tiling that would fix it, since a shared endpoint on those kinds was an overlap error.
The usable fix landed in 0.6 rather than 0.7: an optional measure_tiling column ({quantised,
continuous}) that the shared-endpoint and gap rules read instead of the kind, with absent meaning
the kind's default so no published table is re-read; modifier_copy_number beside the one genuine
int, modifier_cn deprecated; and a fractional value switching a would-be-quantised group to the
continuous rules by itself, saying so in a warning. See SCHEMAS.md for the per-kind
default table and the resolution order. binning.measurement_shape_warnings still says the RM55
sentence once per table, but only where it is still true — a kind VCF 4.4 types as fractional that
still has a group reading as a grid. A table declaring itself continuous, or carrying a fractional
bound, answers its own boundaries and is silent. Only the removal of modifier_cn stays at 1.0, so
1.0 inherits a removal rather than the retype the original route named.
A measurement can span several bins (RM56). The same two fields carry confidence intervals (CIRUC,
CICN) whose missing bound means unbounded, so a real measurement is an interval; htt_repeat_
expansion states three thresholds inside a 14-count window for one to cross, and the consumer contract
has no state for it. 0.6 warns and states the placeholder — withhold — rather than leaving it
silent. The policy vocabulary (withhold / worst bin / point estimate) and its grain wait for a real
caller VCF. Widening the measurement itself into an interval is not on the table: that puts a
measurement in the module, which the data-agnostic north star forbids.
. in an alts cell splits identity (RM58). VCF's MISSING marker means there are no alternate
alleles, and no ref/alts column has a nucleotide grammar (deliberately — adding one would tighten
the field RM5 exists to widen), so the cell loads and derive_variant_key folds it in as though it named
an allele: 1:1:A:. where the same site with an empty cell is 1:1:A, two content_signatures and no
dedup between them. alleles.non_nucleotide_reason now answers a third reason, "missing", distinct
from "ambiguity" (a permanent uncertainty) and "notation" (a grammar gap a release may widen) — .
is neither, and there is nothing to widen to hold it. A diagnosis, not a grammar: the value is still
accepted, and the compiler warns per table with the two keys side by side. It is the only finding of the
VCF round that reaches identity, and it reaches only the key string — is_substitution refuses a
non-nucleotide alt, so no VA is minted and no content-addressed claim is false.
One finding from the same round is not a schema limit and is listed with the pointer columns in
SCHEMAS.md instead: a min_quality floor stated against QUAL on a requires_callable
row inverts, because QUAL changes sign with the record (§1.6.1.6) and such a row is proved against the
reference record. The compiler warns and deliberately does not refuse — the meaning depends on a record
this tier will never see, and the same row read against a variant record is legitimate.
And a fourth — a pointer that does not identify a VCF field (RM53/RM54, 0.6)¶
Same class as the three above: a limit of the schema, closed in 0.6, and worth keeping in the same place because the check that survives it is again a legibility warning rather than a verdict.
Three authored columns point into a VCF (source_field, callable_from, quality_from) and all
three took a bare token. A VCF field is identified by namespace — INFO and FORMAT are two
reserved-key tables that collide on DP, AD, ADF, ADR, MQ, AF and, since 4.4, CN — and
described by cardinality (Number, which decides how many values come back and what each is of).
Where both readings are type-compatible, and for DP, AF and CN they are, nothing detects the
confusion: the consumer reads a well-formed number of the wrong kind and bins it without error. Both
shipped reference examples that used these columns were wrong this way, under --strict, with every
offline gate passing — the same failure geometry as the 3,038-row coordinate incident.
The schema half is in SCHEMAS.md: the pointer grammar accepts INFO/DP/FORMAT/DP (bare
still legal, still meaning unqualified), the spec's own key charset is accepted (1000G, a dotted
key — RM61), and MeasureBinRow.source_element names which element of a multi-valued field the bin is
measured against, from a closed set of named rules rather than an index (P1 refuses AD[1]).
_check_vcf_pointers is the compiler half, and it has two findings, both warnings in both modes:
a bare key that is one of the known collisions, and a pointer at a spec-defined multi-valued field with
no element rule. Neither escalates under strict — the grammar was widened rather than replaced, so
refusing there would break P3, and strict means reproducible artifact, which an unqualified pointer
is (P5). Both are aggregated by reason, since a panel pointing every bin at one field would otherwise
print the same sentence hundreds of times.
Two things it declines to say, both for the reason this section exists. Cardinality is read from a
transcription of the spec's reserved-key tables and nothing else — REPCN is ExpansionHunter's
key, not the spec's, so this tier is not entitled to assert its Number, and a bare CN disagrees
across the two namespaces. Unknown withholds. And the cardinality finding is scoped to pointers that
have a companion column: callable_from/quality_from did not get one in 0.6, and telling an author
to fill a column the schema does not have would be a finding no edit could clear.
And a fifth — a site annotated for some of its genotypes and not the rest (S32, 0.6)¶
Not a schema limit at all: this one is entirely visible in the authored rows, and nothing but a check
was ever missing. A consumer matches a subject on (variant, genotype), so a genotype with no row is a
subject with no answer — and the consumer who reported it found a curated 520-site module authoring no
homozygous-alternate genotype at 208 of them, with their subject homozygous at 74. Every one was
silently unreported.
_check_genotype_coverage fires only at a site the module already annotates for two or more
genotypes, and that scope is the design rather than a tuning choice. One genotype at a site is the
ordinary shape of a drafted-then-curated module — pathogenic_clinvar authors exactly one at 326 of its
327 sites — and it is a legitimate rule, not a gap; reporting those would put a line on almost every
module in existence, which is how a warning stops being read. Two or more is the author demonstrating
that the genotype space at that site is what they are describing, and the missing member is then a
hole in something they started. It expects the reference homozygote, one heterozygote per alternate and
one homozygote per alternate, and never an alternate/alternate pair — RM35's jointly-satisfiable
lesson, since requiring A/T would make a two-alternate site unreachable. The reference allele comes
from the row or from the injected table, never from a guess: with neither, a two-allele site is still
enumerable and is reported by spelling, and a site with three or more alleles is skipped. Sites whose
genotypes are not diploid nucleotide pairs drop out on their own, which is what keeps MT and non-PAR Y
out without a contig list.
It says nothing about any callset, and that boundary is the item's own. Whether a hom-ref row can
ever match is a property of the data a consumer brings — a variant-only VCF emits no record where the
sample matches the reference, a gVCF and an array both do — so that call belongs to the annotator. The
presence of a hom-ref row is therefore never reported: those rows are correct, and on array data they
are the ones that carry the answer. Warning in both modes and never a strict error, joining the small
set of checks that arbitrate nothing (the ClinVar clin_sig cross-check, the declared-licence
disagreement, the non-commercial quote): which genotypes a module annotates is the curator's judgement.
It runs in validate_spec only, and reaches a compile through the warnings it returns — the message
carries a count, and after resolution a one-to-many rsID has become one row per locus, so a second pass
would publish a second, differently numbered copy of the same finding into
manifest.compilation.warnings. Three reference examples fire it, each truthfully:
grch37_build and hfe_hemochromatosis state a carrier and a homozygote and no reference homozygote,
and pathogenic_clinvar states both HBB heterozygotes at 11:5225715 and neither homozygote.
Hints are not a fourth validation class¶
hints.py computes nothing the compiler does not already compute — it reuses validate_bins,
_TABLE_DUPE_KEYS and the models' own validators — and it never fails a build. It has no mode
ladder, because the checks that do have one already exist here and in the enricher, each with the
severity the charter assigns it (including the two deliberate exceptions that warn in both modes). A
hint that could fail a compile would be a third copy of a rule that already lives in two places.
Two findings are the exception. The missing-sentinel warning
(§ 1 above) and, since RM279, a variants.csv conclusion that names genotypes at its own locus and
never its own (just_dna_compiler.conclusion) are computed here and nowhere on the compile path. The
second waits on a warning code, which is minor-class (RM308). Neither can fail anything.
What hints add is when: the same verdict, before the author has written the file, plus the allowed
values and the reason a cell is deliberately left empty. That last part is the load-bearing one — see
hints.REDUNDANCY_BEARING. Class 2 works because two independently-authored things must agree, so a
tool that fills one of them from the source the checker consults does not merely make the check
tautological; for an rsid-only row resolution._verify never runs at all, and the row would go from
honestly unverified to apparently verified. Every looked-up fact is therefore reported with
applied=False and a refusal reason, and the enricher's lookup.py answers in the same shape.
A column name is not a scope, and the reason given now says which table it applies to (RM123).
REDUNDANCY_BEARING maps a bare column to the checker that reads it, and clin_sig is a column on
variants.csv, on all four binning kinds and on diplotypes.csv while verify_clin_sig takes
list[VariantRow]. So the vacuous-check sentence printed on six (column, table) pairs whose checker
cannot see the table — right advice, false reason, and a green run on one of those cells looked like
agreement with an authority that never saw it. REDUNDANCY_BEARING_TABLES narrows the explanation
for the columns whose checker reads fewer tables than carry them; an absent key means unscoped, which
is the honest answer for rsid/chrom/start/ref/alts (resolution reaches the positional kinds,
RM43) and for pmid (a bin and a pharm row are citation sites in their own right, RM47/RM132).
It scopes nothing else: the refusal a
drafting provider obeys stays keyed on the bare column, because whether a provider should start filling
clin_sig on a binning row is a decision nobody has taken.
One consequence of computing nothing new is that the same cell can be seen by two layers, and a hint
must still print it once. An unreplaced TEMPLATE_PLACEHOLDER is the case: _check_placeholders
names the column, and the model then says the same thing less usefully — a row-level ValueError
listing every placeholder path on an authored model, or a vocabulary error quoting the token back on
sources.csv/licensing.csv, whose row model carries no such guard. A freshly drafted 109-row panel
printed two lines per defect. _validate_row drops both restatements, keyed on the cells the
per-column check actually reported, so a placeholder error nothing covered would still be shown. The
guard itself is untouched: it is what makes a generated stub unable to compile, and a hint refuses
nothing.
And what is not the compiler's job at all¶
Static checking has an upper bound here, and the analogue of dynamic analysis lives elsewhere. The
compiler is not a runtime verifier: it never runs a module against a genotype, because a module carries
no sample data and the measurement is supplied by the consumer at query time. The ecosystem's
"valgrind" is the consumer-side verification harness — run a panel against N VCFs and diff a
report-card — which is deliberately not a format feature (USE_CASES.md §3b / RM7).
The format's contribution to it is the properties it already froze: artifact.digest makes the
before/after diff trustworthy, and the mandatory unresolved/callability contract stops a no-call
masquerading as a mismatch.
The compile pipeline¶
A file the compiler has no meaning for is ignored, and that is a contract (S16). A spec directory
may carry a README (every reference_examples/ module does), curation notes, or a publisher's receipt
recording the identity a registry stamps — those keys cannot live in module_spec.yaml, where
extra="forbid" rejects them precisely because the registry owns them. Such a file is not read, not
hashed, and not in artifact.files, so it cannot move artifact.digest; pinned by a test that
compiles the same spec with and without two unknown files and compares digests. The one exception is a
near miss: an unknown .csv within one small edit of a table name (varaints.csv) warns, because
"ignored" is the wrong answer for a typo — every row in that file is being silently dropped from a green
compile. The check is edit-distance-keyed rather than "any unknown csv" on purpose, so it cannot undo the
tolerance above. It reads .json as well as .csv since 0.6, which covers verification.json and
retroactively covers provenance.json — that one had been in the known-name set since 0.4 with no
suffix that could reach it, so a provenence.json was invisible to the very guard written for this.
A registry's published.json is not within an edit of either and stays tolerated, which is the
property the widening had to preserve.
Where the machine-written sidecars may live, and what they may be called (RM49/RM51, 0.6).
resolution.csv, the six fact tables and verification.json are resolved through
just_dna_format.layout, which accepts each of them at the spec root or under a derived/
subdirectory, and accepts the licence table under either sources.csv (deprecated, warn-only, removed
at 1.0) or licensing.csv. Four rules:
- Only the machine-written tables move.
module_spec.yaml,variants.csv,studies.csvand the table kinds have exactly one legal name in exactly one legal place. Two legal homes for an authored table means a module can carry two copies with the ignored one invisible. derived/is tolerated, never canonical.reverse_moduleemits a flat tree and the enricher creates one; a module is split only because somebody split it.reverse_modulewrites the sidecars through the same resolver, so a round trip migrates the name. It regenerates a spec directory rather than editing one, so on a fresh tree there is nothing to follow and the rule yields the preferred spelling — a module carryingsources.csvreverses ontolicensing.csv, and a compile → reverse → compile no longer picks up a deprecation notice its first compile did not have (manifest.compilation.warningsis published, so the two must agree). Reversing over a directory that already carries a copy overwrites that copy instead of leaving a second one beside it.- Either spelling is a key (RM224).
SIDECAR_SPELLINGSis keyed on the table key —sources.csv, the name the parquet and the manifest keep — but everylayouthelper accepts any spelling of the table, so a caller holding the filenamelicensing.csvfollows a module'ssources.csvrather than creating the second copy.layout.sidecar_keyis the filename → key map, derived from the table. - Two copies of one table is an error naming both paths — never a merge, never newest-wins. These tables are fact-hashed and human-overridable, so two copies are two legitimate claims and preferring one silently discards a curator's override. The enricher's rule is the other half: write to the file you read.
- The guard follows into
derived/, and it takes two tests there. Tolerating a second location without extending the guard would put a typo'dderived/varaints.csvexactly where the check written to catch it cannot see — which is also why "search any subdirectory" was refused: one fixed name is the only version the guard can follow. What is legal underderived/is the sidecars alone, but that smaller set is the wrong thing to fuzzy-match against, and matching against it caught neither case: at the 0.8 cutoffvariants.csvis no near miss of any sidecar name, and neither isvaraints.csv. So an authored table name underderived/is reported as misplaced on an exact match — those rows are read from nowhere, and a module that keeps another table compiles green without them — while everything else there is fuzzy-matched against the full known set. The acceptance set stays the smaller one, so a legal sidecar is never reported as a stray, and the mirror case (a sidecar at the spec root) is legal and stays silent.
Neither the name nor the location enters any identity: the fact sidecars are outside _INPUT_FILES
(see the file sets below), so artifact.digest, content_signature, resolution_signature and
source_signature are unchanged by either. manifest.derived records the relative path it found, so
a derived/… entry tells a registry how the tree was laid out.
The compiled outputs are untouched: a module reading licensing.csv still writes sources.parquet
and still publishes manifest.sources. Both of those are renames only a major may make.
compile_module runs in this order:
- Validate (
validate_spec); fail early if invalid. - Load
module_spec.yaml(authority-key pre-strip), thenvariants.csv/studies.csvif present. - Load
resolution.csvif present → group rows byvariant_keyinto a resolution table (a row-parse error fails the compile), then verify every storedvrs_id(below), and stampresolution_signature/resolution_sourceshere — where the table was read, so a module with novariants.csvrecords the identity of the table it carries rather than a null (0.6, RM45). - Resolve (only if
resolve_with_ensembl and variants) — the precedence block below. - Re-validate identity post-resolution (
_cross_validate_variants) — resolution can change identity (fill a coord, expand a one-to-many rsid), so a post-resolution duplicate/inconsistency fails the compile ("post-resolution: …"). - Compute
fully_resolved= every variant haschrom+start(vacuously true for a variant-less module), andresolution_subjects= how many rows that quantified over, from the same list, so the flag cannot be published without its denominator (RM44). - Strict gate — if
strictand any variant still lacks(chrom, start), fail before any parquet is written (refuse a non-reproducible partial artifact). - Write parquets — SNP core (
weights/annotations/studies.parquet, only when the relevant rows exist) + one parquet per present table kind + the derived-fact sidecars (frequencies.parquet,gene_metrics.parquet,literature.parquet, and since 0.6gene_validity.parquet,clinical_assertions.parquet) when their CSVs are present, each cross-checked against what the module actually contains (a frequency coordinate no variant sits at, or a gene the module never mentions, is a warning — an over-broad sidecar is harmless, and failing the compile over it would punish the author for the enricher's generosity). - Collect logs /
provenance.json/ logo / readme (a malformed one fails the compile, not raises). The readme is discovered frommanifest.README_CANDIDATESand hashed intomanifest.readme, outsideartifact.files— so it is attested without being content (S25). - Read the verification attestation (
verification.json, RM45): recompute the binding over the authored inputs — with\r\nread as\nsince 0.6 (RM82), so an editor's line endings are not an edit, whilemanifest.inputs[]keeps listing the raw bytes and raw sizes because it answers are these the exact bytes — re-check the proof-of-work, and either carry the block into the manifest or warn and drop it. A stale attestation never fails a compile — the goal is that it never becomes a published claim, not that it be impossible to write while editing. Nothing here is trusted; see SCHEMAS.md. The same read decides the closure (RM73, 0.6): if the block that survives carries none, the compile says so — a warning in both modes, carrying no count so the identical sentence fromvalidate_specde-duplicates against it. Absence is the only thing said here, because the reason (never closed, closed then edited, no document at all) is already carried by whichever other warning applies. - Build the manifest (
content_signaturere-read from raw disk, the resolution fields, thefrequency/gene_metrics/literature/verificationblocks, andderived[]— byte hashes of the sidecars where they live beside the spec, transport-only and never their identity) and writemanifest.json.
The VRS verify pass (0.5)¶
A GA4GH concept in a no-network tier — deliberately, and asserted. VRS is normally met alongside sequence services and a client library, so importing the idea into the compiler is exactly the kind of change that quietly drags a network dependency behind it. It does not: allele identification is
sha512t24uover canonical JSON — arithmetic — while only normalization (indels) needs sequence access, and that half lives solely in the enricher.just_dna_format.vrsimportsbase64/hashlib/json/reand nothing else;ga4gh.vrsappears nowhere outsidejust_dna_enricher.compiler/tests/test_tier_purity.pypins this in a fresh interpreter (the test suite has the enricher loaded, so an in-process check would prove nothing): the compile path imports no network client,just_dna_format.vrspulls no non-stdlib module, and a full compile — minting the VA, keying on it, and rejecting a tampered id — succeeds withsocket.socketbooby- trapped to raise.The residual risk is not what is there, it is the gradient: the verifier is partial (it can recompute a substitution and not an indel), and the tempting completion is to give the compiler sequence access. That is the line not to cross — the asymmetry is the design, not a gap. Likewise
refget_accessionraising for a non-GRCh38 build is not an invitation to fetch the accession; it is an invitation to add a second committed table (RM15).
A ga4gh:VA.… is content-addressed, so it is the one column in the whole artifact that can be checked
against itself — no reference, no network, and no new dependency, since derive_vrs_allele_id is
stdlib (Goal 2). _verify_vrs_ids runs before anything is written, so a bad id never reaches an
artifact. It belongs here rather than only in the enricher because the compiler is the last gate
before an artifact exists: a spec can be hand-edited and compiled directly, never touching the
enricher, so an enricher-only check would be bypassable. Checking injected data by pure computation is
precisely the compiler's job — the same thing it already does for every hash and digest it writes.
Three outcomes, and why "unverifiable" is not "mismatch"¶
Every row with a vrs_id lands in exactly one of three outcomes. The distinction between the last two
is the point of the whole design, and conflating them would be a lie about what was actually checked:
| Outcome | Meaning | best_effort |
strict |
|---|---|---|---|
| verified | recomputed, and equal | silent | silent |
| mismatch | recomputed, and different | error | error |
| unverifiable, the tier's limit | could not be recomputed here, and no edit would change that | warning | warning |
| unverifiable, the row's contradiction | an id recorded against nothing to check it with | error | error |
Note what the mode column does here: nothing. This pass is not a mode ladder. Severity comes from whose limit the finding is, and both answers are the same on both rungs.
A mismatch is always fatal, in both modes. A substitution's id is fully deterministic here — same
inputs, same 20 lines of hashlib, same answer — so a disagreement cannot be a difference of opinion
between implementations. It is corruption, and there is no mode in which carrying it is right.
An indel is never reported as a mismatch, because it is never compared. This tier cannot recompute
an indel's id (justification needs the reference sequence), so it can only report that it did not
check. Saying "mismatch" would assert a verdict that was never reached. Warnings land in
manifest.compilation.warnings, so an unconfirmed identity is visible to a consumer rather than only
to whoever ran the compile.
Why the tier's own limits do not escalate under strict. They did, for one release cycle, on the
reasoning that unchecked and correct are different things and strict's contract is a reproducible
artifact. The first half is true and is why the outcome exists at all; the second half does not follow.
An enricher-minted indel VA is reproducible — the bytes are injected, the compile is deterministic,
and recompiling yields the same digest. What is out of reach is the verification, not the
reproduction, and escalating on that conflates "I could not check this" with "this cannot be rebuilt".
The cost was concrete rather than theoretical. Minting indel ids online is exactly what
just-dna-enricher exists to do, so every ClinVar-derived module acquired identities that
compile --strict then refused, and the two remedies the error offered were lower your guarantee or
delete a correct identity. Two reference examples — pathogenic_clinvar (185 alleles) and
shox_par1 (2) — stopped compiling in the mode their own READMEs document, and the authoring skill's
step 6 tells every author to run exactly that mode. The rule now matches the one
_vrs_coverage_warnings and frequencies' not_covered already followed: a finding no authored
edit could clear is not a strict matter — strict is orthogonal, and P5 says orthogonal axes stay
orthogonal.
What still errors, and in both modes, is the row contradicting itself: a vrs_id recorded against
no coordinate, against no ALT, or — since R2-5 — against a symbolic allele. That is not a limit of
this tier. The first two assert an identity while withholding the very thing that identity is a digest
of; the third asserts one for an allele that has no sequence to digest, so the id necessarily names a
different allele. Nothing anywhere could check any of them. Same class as the inconsistent reference
allele error, and catchable offline.
Every flow path¶
_recompute_vrs_id returns either the recomputed id or the reason there is none, for one allele.
vrs_id is a comma-joined parallel array of alts, so the pass walks the two together and each ALT
gets its own verdict; an empty member is a hole and reads exactly like an empty cell. Most of the
reasons are limits of a no-network tier rather than defects in the row — the last three are not:
| Row | Path | Both modes |
|---|---|---|
no vrs_id, or a hole in one |
nothing to check — not the same as "could not check" | silent |
| substitution, id agrees | verified | silent |
| substitution, id differs | mismatch | error |
| multi-allelic, every member agrees | verified allele by allele | silent |
| multi-allelic, members swapped | mismatch — the desync a length check cannot see | error |
indel / MNV (C>CA) |
needs the reference sequence — minted upstream, not recomputable here | warning |
| off-assembly contig, or a position past the contig end | no refget accession to address the sequence by | warning |
non-GRCh38 genome_build |
no refget table for that build (RM15) | warning |
*, the unobservable-allele marker (RM59) |
the callability axis — it names no allele, so nothing to digest | warning |
symbolic allele (<DEL:4977>, RM5) carrying an id |
no tier mints one, so a recorded id names a different allele | error |
position-only (no alts) |
an id against no ALT — the row's contradiction, not the tier's | error |
| no coordinate | an id against no place (an rsid row carrying an external id) | error |
A symbolic allele with no id is a coverage warning; one carrying an id is an error, and the
asymmetry is the rule rather than an exception to it. Absence is this tier's limit — no authored edit
clears it, which is what keeps every structural module compilable — while a recorded id is a claim
about an allele that has no content to address, so it can only name something else. Absence is a
limit; a claim is a claim. The escalation waited on the grammar (R2-5): it follows only once vrs_id
is known to hold allele ids alone, which vrs.validate_vrs_allele_id now enforces (ga4gh:VA. only,
probed at 844 corpus ids with no other type). The remedy is one edit — delete the cell; variant_key
still carries the allele's identity.
* sits beside the symbolic row and not inside it (R2-6). The two are different axes, which is why
_vrs_gap_reason and _recompute_vrs_id test is_unobservable_allele separately and above the
substitution fall-through: parse_symbolic_allele asks which variant is this, unspelled, while
is_unobservable_allele asks whether the sample's call could see an allele at all. Reported as an
indel it would carry a remedy — re-run the enricher online — that can never apply.
Multi-allelic used to be one row of this table, blanket-unverifiable, "a VA names exactly one allele;
picking one would invent data". The premise is derive_variant_key's and it is right there — a
variant_key names one thing, so a plural cell falls through to the coordinate key. It was wrong here,
where nothing is picked because every ALT is named. It cost the id on 909 of 1,613 rows in one real
module while every input needed to mint all 2,110 of them sat in the same row.
A symbolic allele is its own row of that table, and it was an indel for the whole of 0.6 until a
module carried one. RM5 shipped the structural grammar and this pass was never told, so <DEL:4977>
fell through to is_substitution and was reported as "an indel or MNV: justification needs the
reference sequence, so only the enricher can mint it (re-run it online)" — false on every clause.
Symbolic notation exists precisely because the sequence is not spelled: there is no sequence to
justify against, nothing for a content-addressed id to be a digest of, and no tier that mints one, so
the id is permanently absent rather than one online run away. (The enricher's minting pass says the
same wrong thing in its own words, and the online run this message recommended crashed on the same
allele — both of those are its side of the item, not this one's.) Two
consequences worth keeping. The branch sits ahead of both the substitution test and the accession
lookup, in _vrs_gap_reason and _recompute_vrs_id alike: a symbolic allele on a GRCh37 module is
also a build with no refget table, and between two true statements the one to print is the one no
release can answer. And the predicate is the lenient is_symbolic_allele — a malformed <FOO> or
an unterminated <DEL names no sequence either, so filing it under "indel" would be the same false
claim about a different mistake.
Coverage — what a VA does not name¶
Verification only ever looks at ids that are present, so a table where nothing was minted verifies
flawlessly. _vrs_coverage_warnings reports the other half: allele slots seen, how many carry an id,
and the remainder grouped by reason class (not by row, and not by _recompute_vrs_id's per-row
prose — grouping on that produced forty lines each naming a different indel). The counts are recorded
in manifest.compilation.vrs_alleles / vrs_alleles_identified, so a consumer can read the
reliability of the identity scheme instead of inferring it; "complete" is identified == alleles,
derived rather than stored twice. A shortfall is a warning in both modes — the usual causes (an
indel with no sequence proxy, a build with no refget table) are fixable by no authored edit, and
strict means "reproducible artifact", which an incompletely-named table still is.
A symbolic allele still counts in the denominator, so a structural module reads as permanently
short of complete coverage — reference_examples/mt_common_deletion compiles at 2/3 and always will.
That is the honest number: the denominator is allele slots in resolution.csv, one definition, and
excluding the slots that can never be named would publish a second, undisclosed one — "coverage over
the alleles an id was possible for" — which is exactly the flag-with-a-hidden-subset shape RM44 exists
to stop. What tells the two situations apart is the reason line beside the count, which now says the
gap is permanent instead of naming a remedy.
The last row is a fixed bug worth naming: refget_accession raises UnsupportedBuildError rather
than returning None — deliberately, so a caller asking for GRCh37 hears "not built yet" instead of
receiving a GRCh38-flavoured answer. That exception used to escape the verify pass and abort the whole
compile over a single unverifiable row. It is now caught and turned into a reason, which is the correct
severity: one row this tier cannot check should not fail a build in either mode.
The inconsistent-reference-allele check (0.5)¶
A VA addresses the place and the alt; the reference base at a position is a fact of the genome and is
not part of the allele's name. Correct VRS semantics — but it drops a guarantee the old
chrom:start:ref:alts key gave for free, since two rows at one position claiming different reference
bases used to be two keys and are now one. At most one can be right, so _cross_validate_variants now
fails a compile where two positioned rows share a key and disagree on ref.
Symbolic alleles: the one check that discards an authored row (RM5, 0.6)¶
The grammar holds <DEL:1500> / <CNV:TR:30> (see SCHEMAS.md § The allele grammar).
What the compiler owns is the other half: a module is a declarative rulebook, and a rule nothing
can apply is worse than an absent one. A <DEL> with no length cannot be sized, matched against a
call, or told apart from any other deletion at that position, so _check_symbolic_alleles reports it —
and this is the first check in the tier that discards an authored row.
| Reason | What it is |
|---|---|
no_length |
a real structural type with no usable length: <DEL>, or <DEL:0> |
unknown_type |
angle-bracketed but outside the closed five — <FOO>, and VCF's <*>, which makes an observability claim rather than naming a variant |
reference_allele |
a symbolic allele in a ref column: REF is always a sequence, so a locus whose own reference is unspelled anchors nothing |
Severity depends on whether the row stands alone as a rule. On variants.csv and
pharm_variants.csv — one self-contained rule per row — best_effort drops the row with a warning
that says so, and strict refuses. On haplotypes.csv and heteroplasmy.csv it is fatal in both
modes: those rows are parts of a composite, so dropping one silently redefines a haplotype or punches a
hole in a bin tiling — not a smaller module but a different one.
A drop that would empty a table outright is an error in both modes as well, and on its own reason:
the drop exists so a module can lose one unusable rule and still say the rest, while a table that loses
every row says nothing at all and says it only in a warning. (Not, as a first cut claimed, because the
compiler already refuses a present-but-empty table — that is true of the _TABLE_KINDS loop and
false of variants.csv, which validates and compiles header-only. Measured, and pinned by a test.)
The warning must say DROPPED. This does not break P7 — the round-trip fixed point is claimed under
strict, where this case refuses — but reverse cannot re-emit a row that never reached the parquet,
so a warning that merely flagged it would leave an author believing their module still carries it.
Three mechanics worth copying. The check reads AuthoredModel.ALLELE_COLUMNS, declared on each model,
so the compiler holds no second copy of a model's column names. It runs in validate_spec too, by the
standing rule (pure computation over authored bytes, no output_dir) — with the identical message, both
because the pre-flight must predict what the compile will do and because compile_module's
de-duplication is on the message; every refusal it can reach is computed in the shared check rather
than at the point of application, so validate cannot go green on a module compile then rejects in
the same mode. And manifest.stats is re-derived over the surviving rows: weights_rows counts the
parquet, so leaving variant_count as validate_spec computed it would publish a count higher than the
artifact holds — the RM44 class, a manifest number a catalog keys on and cannot check.
Findings name a row by its identity (variant_key, else haplotype_name), never by a file
position: load_csv_rows prints a header-inclusive line number and hints.Finding.row is a 0-based
data index, so a third convention would be one too many — and an index computed over the rows that
survived model validation shifts silently behind any earlier load error.
One asymmetry to expect: <FOO> fails at load in genotype / effect_allele /
HaplotypeRow.allele, which have a grammar, and reaches this check only through ref/alts, which
deliberately have none.
Resolution precedence (additive; Principle 3)¶
Inside step 4, gated on resolve_with_ensembl and variants, with resolution_mode = "strict" if strict
else "best_effort":
resolution.csvpresent →resolve_from_table(the preferred, source-independent path — noduckdb, no network, no Ensembl convention). Setsresolution_signature(fact-hash of the rows) andresolution_sources(sorted union of rowsources).- else
ensembl_cachegiven → DEPRECATED DuckDB path. Emits aDeprecationWarning("… removed at 1.0. Produce a resolution.csv …") and routes to the enricher via a guarded lazy import (from just_dna_enricher.resolver import resolve_variants); if the enricher isn't installed, the compile fails with a message pointing at it or at precomputingresolution.csv. The compiler declares no dependency on the enricher. - else (nothing injected) → skip.
Noneno longer auto-discovers a cache (the 0.5 Principle-2 tightening); variants lacking a position are left unresolved with a warning pointing at the enricher.
Digest parity between paths 1 and 2 is the load-bearing guarantee: given the same facts, both emit
byte-identical weights.parquet (hence artifact.digest). The one order-sensitive spot — a one-to-many
expansion — is pinned by sorting on (locus_index, chrom, start, ref).
Resolution¶
Resolution is where authored data meets injected facts, and it is the one place in the compiler where a row can change shape. Everything below follows from one rule:
Resolution must be reversible.
compile → reverse → compilehas to reproduce the module it started from — the same bytes, the same authored content, and the same resolved facts. Where it cannot,strictrefuses rather than emitting an artifact nobody can re-derive.
Scope: the SNP core plus the three positional tables (RM43, 0.6)¶
Resolution applies to variants.csv and to the three positional 0.4 kinds —
pharm_variants.csv, haplotypes.csv, heteroplasmy.csv, derived from the models rather than listed
(a table is positional exactly when it declares both chrom and start). Until 0.6 it applied to
variants.csv alone: every other table went through _build_table, which is the model straight to
parquet, so a row kept exactly the coordinates its author typed — for an rsid-authored module, none —
and a consumer matching a patient VCF by position matched nothing, silently, as an empty result rather
than an error. That is now filled at compile time from the injected table; see The positional fill
below for the four rules it follows and the three columns it needed.
The rest of this section is the 0.5.3 report (S9) that surfaced it, kept because two of its three manifest observations still hold:
fully_resolvedisall(...)overVariantRow, so it is vacuouslytrueon a module with novariants.csv. Its own field comment gives the trust rule "a consumer trusts a module whenresolution_mode == "strict" or fully_resolved" — which is not sufficient for a 0.4-family-led module, where it can betruewhile every row lacks a coordinate. Since 0.6 the denominator is published beside it (resolution_subjects, RM44), so the sufficient rule isresolution_subjects > 0 and (resolution_mode == "strict" or fully_resolved)and the vacuous case is legible from the manifest alone. The counter makes the vacuity visible; RM43 makes the tables joinable, but the flag still quantifies overvariants.csvonly.resolution_modeand the--strictunresolved gate are the same scope.strictrefuses on an unresolvedVariantRow; it says nothing about a table row, which is why such a module compiles under--stricteven where the fill could not place a row. That is deliberate — see the warning's severity below.resolution_signature/resolution_sourcesstay unset for a module whose only subjects are table rows, so its injectedresolution.csvleaves no trace in the manifest. This was blocked on reverse, which rebuiltresolution.csvfromweights.parquetalone; RM43 removed that blocker (it now rebuilds from the positional parquets too), and stamping the two fields is RM45's half of the same round.
The warning's wording is a contract, because the manifest carries it and nothing else.
compile_module copies its warnings into manifest.compilation.warnings, which ships inside
manifest.json, and a catalog reindexing from a published manifest has no spec directory left to
re-derive anything from — so for a table-only module the sentence is the only surviving record that
its rows join to nothing, fully_resolved being vacuously true. A downstream registry substring-
matches "have no chrom+start" to decide a trust badge; compiler.UNJOINABLE_PHRASE names that
fragment and a test pins it, so a reword breaks this build rather than a catalog. Improve the rest of
the sentence freely; move that fragment deliberately.
RM44 shipped in 0.6 and it retired half of this; S31 shipped the other half.
resolution_subjects gives a consumer a structured field for "was the flag about anything", and
positional_rows / positional_rows_placed now give the unjoinable-row count RM44 recorded as
belonging with RM43 — the two counts, parts not a ratio, so "this table joins by position" is
positional_rows_placed == positional_rows and needs no prose. The fragment and its test still
stay, and not out of caution: every artifact published under 0.5 carries neither field, so for those
manifests the sentence remains the only record. Retiring it is a decision to take once the published
corpus has been recompiled.
Reading it from the consumer side, which is what these fields are for. A module's rows join by
position when positional_rows_placed == positional_rows (with positional_rows == 0 meaning the
module carries no such table, so the question does not arise) — and variants.csv is the separate
question fully_resolved over resolution_subjects answers. None on either field means the
artifact was compiled before 0.6, which did not fill those tables at all, so the honest reading is
unknown, and probably unfilled rather than zero: a 1,482-row pharm_variants module compiled under
0.5 has every coordinate null and no manifest field saying so. That is the state the reporting
consumer had to discover by opening the parquet, and it does not change when a new compiler is
installed — a published artifact only moves when it is recompiled and republished, which is the
maintainer's action rather than the consumer's.
The joinability warning now reports the residue. Every positional table is still checked for rows
with no chrom+start, after the fill has run, and the finding is one aggregated line per table
carrying how many rows cannot be joined by position plus why. Three readings: the injected table
names the key at more than one locus (or at one the row's own allele contradicts), so the compiler
leaves it rather than picking; nothing in the table names the key at all, which an enrich run fixes;
or the fill never ran — --no-resolve, or a non-GRCh38 module — in which case the coordinates may
be right there and untried. The third branch is why the check is told whether the join happened
rather than inferring it: this sentence ships inside manifest.compilation.warnings beside
UNJOINABLE_PHRASE, so asserting "the compiler looked and would not pick" about a lookup that never
happened puts a fabricated diagnosis into a document a catalog reads. A half-coordinate (start with
no chrom, the shape a CPIC-drafted haplotypes.csv carries) is counted apart, because it reads as a
position and joins to nothing.
It is a warning in both modes and never a strict error, for two independent reasons: rsid-only
identity is legal by these models' own rule, so escalating would have the format tighten a field it
deliberately left open; and what survives the fill is by construction something no authored edit to
that table clears — the same class as VRS coverage and not_covered, where refusing makes a correct
module uncompilable for something its author cannot fix. It runs in validate as well as compile,
and is de-duplicated between them.
The positional fill (RM43, 0.6)¶
_apply_positional_resolution joins the injected resolution.csv onto each positional table before
_build_table materializes it, and runs in both validate_spec and compile_module — the
joinability line is computed from these rows in both, so filling on one side only would leave the
pre-flight naming a gap the compile had already closed. validate_spec therefore takes
resolve_with_ensembl too, and compile_module passes its own value down: the flag is the master
switch for resolution of every kind, so a pre-flight that ignored it would be the more optimistic of
the two commands. The fill is skipped, with a warning, for a non-GRCh38 module, exactly as
resolve_from_table is (RM15).
Four rules, and the last two are what separate it from the naive repair:
- Fill only what the author left empty. A cell the author wrote is never overwritten. That is the inject-only doctrine (report, never repair), and it also makes the fill idempotent.
- Fill from exactly one locus, or from none. One usable locus fills. Several are filtered by
hosting_verdictagainst whatever allele the row states — agenotypeon a pharm row, the definingalleleon a haplotype junction, nothing at all on a heteroplasmy band. If that leaves one, it fills; otherwise the row stays unplaced and the joinability line says so. There is deliberately no expansion: multiplying a pharm annotation's(variant_key, drug, genotype, …)key across loci the author never named is not the same operation as expanding avariants.csvrow. - A row whose own coordinate contradicts the table is left exactly as authored, and the
disagreement is reported. Completing a half-coordinate from a locus whose
startdisagrees would build a coordinate no source ever stated. The comparison runs even where there is nothing left to fill — a fully-populatedheteroplasmy.csvrow can still contradict the table it is keyed into, and the promise is that such a row is reported, which the SNP core gets from_verify.altsis deliberately outside the comparison: a locus lists every ALT recorded there while a row names the one it is about, soA,GagainstGis agreement, and whether the allele can sit there ishosting_verdict's three-valued question, already asked one step earlier. - Rows are mutated in place and their identity is frozen first. Each positional model stamps
variant_keyandauthored_identat load, from the authored columns only, so filling cannot re-key a row.
Three columns made it possible, all parquet-only. variant_key (materialized so a consumer can
join a PGx row to weights.parquet without re-implementing the precedence rule), authored_ident
(which identity columns the author supplied — the same mechanism VariantRow has had since 0.5), and
alts on PharmVariantRow/HaplotypeRow, filled as data, not identity: the key is still derived
without it, so a pharm annotation keeps matching a variant at chrom:start:ref regardless of allele,
and what the column buys is a direct VCF join. None of the three is authored, offered by a drafting
template, or re-emitted by reverse; none is in content_signature (they are exclude=True, so a
stamped value — a pure function of the authored cells — cannot move a content identity, and no
already-published module's signature changes). VariantRow's own two stamped fields are inside
content_signature; that asymmetry is grandfathered rather than a precedent, since changing it in
either direction moves published signatures.
Reverse rebuilds resolution.csv from the positional parquets, and that is forced rather than
chosen. Once a coordinate is filled, a reverse that dropped the lookup table would emit a spec whose
recompile leaves those parquets unfilled — compile → reverse → compile would stop reproducing the
artifact (Principle 7), and a PGx module carries no weights.parquet for the old writer to read at
all. Weights are written first and own any shared key (variants.csv is the only table carrying
alts as an authored fact), and the positional side contributes at most one row per key, or the
next compile would read two rows as a one-to-many rsID. Provenance is discarded exactly as before
(source="reversed", status="resolved", blank fetched_at). A module that resolved nothing anywhere
and has no weights.parquet gets no file, rather than a header-only sidecar invented out of an
absence.
resolution.csv still gets no parquet, and that is the repair this item deliberately did not
make: it is a build-time derived artifact whose consumers are the compiler and the enricher, and
publishing it would turn it into a consumer contract it was never designed to be. See SCHEMAS.md.
resolve_from_table (compiler/resolution.py)¶
Pure, and mirrors the DuckDB resolver's semantics from the injected table:
- fill (1:1) — a
variant_keywith one usable locus fills the missing coordinate or rsid; the frozen key is kept. A coordinate-authored row also has itsaltsfilled when the author left it out, so the resolved allele reaches the artifact and survives being written back. - expand (1:N) — an rsid with N usable loci becomes N coord-keyed rows, each re-keyed by
derive_variant_key. A locus whose{ref} ∪ altscannot host the authored genotype is not expanded onto (hosting_verdict): one authored genotype is copied to every locus, so a locus that lacks those alleles would be emitted as a row asserting an allele it does not have. - verify — a row carrying both rsid and coordinate is checked against the table; a disagreement
warns in
best_effortand refuses instrict.
GRCh38-bound (a non-GRCh38 module is skipped with a warning; not_found/wrong-build rows are ignored).
Hosting is a three-valued question (RM31)¶
hosting_verdict(genotype, ref, alts) answers can this locus host that genotype with True / False /
None, because an indel has several valid spellings and a string comparison reporting "does not fit" was
asserting a verdict it had not reached. ClinVar publishes a SHOX deletion as X:634689 CAG>C and Ensembl
publishes the same event as X:634690 AGAG>AG.
| Situation | Verdict | What the compile does |
|---|---|---|
No ref/alts recorded |
True |
keeps the locus (lack of evidence never rejects) |
A * among the alleles, on either side (RM59, 0.6) |
— | the member is dropped and the rest is judged normally; nothing observable left is None |
| The raw allele strings match | True |
keeps it — checked first, so normalization can only ever add acceptances |
| Either side names a symbolic allele (RM5, 0.6) | None |
keeps it, reports that it did not decide — no sequence, so no flank and nothing to compare |
The reduced allele sets match (alleles.parsimony_reduce strips the shared flank) |
True |
keeps it; this is what reconciles the two spellings |
| The locus is a substitution or MNV and the alleles differ | False |
drops it — no flank, so no spelling freedom; a strand-flipped genotype stays a hard finding |
| The genotype names fewer than two distinct alleles at an indel locus | None |
keeps it, reports that it did not decide (a homozygous call carries no frame) |
| The event sizes differ | False |
drops it — re-anchoring never changes how many bases an event adds or removes |
| Same sizes, different content | None |
keeps it, reports that it did not decide (a rotation inside a repeat, or two variants) |
The * row sits above the raw comparison, and the symbolic row below it — the two are on different
axes and the placements are not interchangeable. A symbolic allele makes a claim that cannot be
compared, so the whole verdict withholds; * makes no claim at all, so only the member goes and the
observable half must still be matched. That is why it has to run before the subset test rather than
after: {'*','T'} is not a subset of a real A>T locus, so below the comparison it fell through to the
substitution row and returned a confident False — dropping the locus and leaving the row unresolved,
which would have made a * authorable and uncompilable in the same release, refusing under strict
for a reason no authored edit could clear. No source spells * in an ALT list, so this is the ordinary
path rather than a corner. */T at an A>T locus is True; */G there is still False, because the
G is a real contradiction and abstention drops a member, never a verdict; */* is None. It costs
the stability property nothing — the called side is stripped first, so * is never on the left of the
subset test and dropping it from the right cannot change that answer.
Both sides, and the locus side is the half that is easy to miss. parsimony_reduce strips the
flank a collection shares, and * has none, so a * left in the locus stops the whole set reducing
and RM31's reconciliation collapses: hosting_verdict('C/CAG', 'AGAG', 'AG') is True while the same
call against alts="AG,*" came back a confident False — a correctly transcribed indel refused under
--strict, advised to "replace it with the alleles the locus actually has". ALT=AG,* is exactly what
a joint caller emits at an overlapped indel, so this is common data, and an allele the algebra must
ignore cannot be one it ignores in only one direction.
What that costs, measured rather than asserted. Swept over every pre-RM59-reachable
(genotype, ref, alts) triple built from a spread of substitution, MNV, insertion and deletion
alleles: for a locus spelled in nucleotides — every reference example, and every module in practice,
since a * could not be written in a genotype before RM59 — no verdict changes at all. For a locus
that does spell *, some do, and they are corrections in both directions: * was blocking
parsimony_reduce's flank strip and lending its own single character to _indel_shaped's length
set, so AG>AT (really G>T) read as indel-shaped and withheld on calls that were decidable all
along. The verdicts that newly refuse are ones the tier should always have refused; a module relying on
one had a genotype that did not fit its locus, and the * was suppressing the finding.
Because None now has four causes rather than one, resolution.undecided_reason supplies the clause
both reporting sites append (this tier's expansion warning and the enricher's twin). They used to
assert step 9's cause — "the same size but different content … needs the reference sequence" — for
every withheld verdict, which for an all-* call sends the reader to check a reference against a
position nothing observed.
The symbolic row sits above the reductions on purpose. Below the raw comparison every remaining step is
arithmetic over characters, and a <DEL:1500> has none to offer — parsimony_reduce would read it as a
nine-character sequence and the event-size rule would then return a confident False computed from a
token's bracket count. Two stated lengths that differ are undecided for the same reason in the other
direction: symbolic notation exists for imprecision, so a summary length is not the kind of fact an
event size is. Note that it only ever adds acceptances, so no already-compiled module moves.
None is the residual only a reference sequence can settle, which this tier does not have (P2) — the
enricher does, and reports it the same way. genotype_fits remains as the boolean face
(hosting_verdict(...) is not False): it is public and shared with the deprecated DuckDB path in
just-dna-enricher, because digest parity between the two is a documented guarantee and a filter applied
on one side only would break it silently. _check_allele_membership asks the same predicate rather than
comparing strings itself — while it did, the two halves of the compiler disagreed the moment a spelling was
reconciled, and strict refused a module resolution had just accepted.
One thing the reconciliation does not do: the authored genotype keeps the frame its source published
it in, so a compiled row can legitimately carry genotype=C/CAG beside ref=AGAG. A consumer joining the
two applies the same reduction (just_dna_format.alleles is public and dependency-free); rewriting the
authored cell is the parked enricher-co-authoring item.
The authored shape is recorded, not inferred¶
VariantRow.authored_ident lists which of {rsid, chrom, start, ref, alts} the author actually
supplied. Like variant_key it is stamped once at load and never re-derived, so filling or expanding
cannot disturb it, and it is materialized to weights.parquet for reverse to read.
It exists because the alternative — inferring the authored shape from variant_key — cannot work. That
key answers "which variant is this", not "what did the author write": it is identical for an rsid-only
row and an rsid+coordinate pair, and after an expansion it is the per-locus allele id with no trace of
the rsid the author wrote. Reverse therefore used to materialize resolved coordinates into
variants.csv and emit one row per expanded locus, which cost two things: content_signature moved on
every round-trip of an rsid-authored module, and each locus received a copy of the single authored
genotype — writing out, as authored fact, annotations for loci the genotype cannot describe.
Why this is only now fixable: dropping the resolved coordinate from variants.csv is safe because
the key is canonical. A VRS allele id names the variant without the coordinate having to be re-authored,
so the coordinate can live in resolution.csv where it belongs. Under the older coordinate-first keying
the coordinate was load-bearing in variants.csv and could not be dropped.
The one recorded finding strict acts on (RM143, 0.7)¶
strict means reproducible, never right, and that is unchanged — the compiler has no reference, so
it cannot check a coordinate. But when the enricher has already checked one and recorded the
answer, throwing that away at the tier boundary is a separate failure: a GRCh37 coordinate pasted into
a GRCh38 module was refused by enrich --strict, written by enrich best-effort, and then built
silently by compile --strict into an artifact that is internally consistent and about a locus
5.6 Mb away.
build_disagreement_error refuses a strict compile when verification.json records a finding on
genome_build_agreement, in validate_spec and compile_module alike, with the identical error and
ahead of output_dir.mkdir() so a refusal writes nothing.
Why that check and no other. Its findings say the module's rows are on a different assembly than
the genome_build it declares — one authored file contradicting another, which is internal
consistency. Every other recorded finding is the module disagreeing with an outside archive, where
the archive is the stale side often enough that failing a build would have the format arbitrate someone
else's dispute; that is the same reasoning that keeps the ClinVar cross-check a warning under strict,
and it now has a parametrized test over four checks. reference_allele is the one worth naming: it
produces this diagnosis's own input and still does not refuse alone, because a ref mismatch has three
causes and only one of them is an assembly.
Three things it does not do, each with a test, because each would be a worse defect than the one it fixes:
| state of the record | what happens |
|---|---|
no verification.json at all |
silent — an unverified module is the ordinary case, and refusing on absent evidence reads unknown as wrong |
genome_build_agreement with findings=0 |
silent — a clean bill; the gate keys on findings, never on the record's presence |
genome_build_agreement skipped (what --offline writes) |
silent — nobody asked, which is not nothing-wrong |
A stale attestation is dropped by the reader before the gate sees it, which is the right order: bytes that moved since the check ran are bytes the check did not judge.
Coverage is reported by the pre-flight too (RM141, 0.7)¶
compile --strict refuses a module whose variants still have no position after resolution — a partial
artifact is not byte-reproducible, which is the failure behind "my local hash differs from the
published one". Until 0.7 validate --strict said nothing about it, so a spec whose resolution.csv
covered only some of its variants passed the pre-flight clean and was refused by the compile
immediately after.
That is the parity rule's own failure shape, and it hid behind the rule's exemption. What stays compile-only is a check reading resolved rows; whether the injected table can place an authored row is arithmetic over bytes the pre-flight has already loaded, and needs no resolution to have run. The exemption is about resolved rows, not about the word "resolution".
resolution.unresolved_subjects is the predicate resolve_from_table applies, shared by both callers
rather than restated — a second implementation of _usable_loci's three exclusions (a not_found
sentinel, a row recorded under another build, a row with no chrom) is exactly the drift this avoids.
Under strict the pre-flight appends the compile's error verbatim; a pre-flight that refused for
its own differently-worded reason would still send an author hunting.
Three behaviours to know:
| the module's table | what is reported |
|---|---|
| present, covers every row needing a position | nothing |
| present, covers some | rsid_unresolved per uncovered row, then the strict refusal — the row is absent from a file that was consulted |
| absent entirely | resolution_not_injected, once, plus the strict refusal — nothing was consulted, so nothing is named row by row |
--no-resolve silences the check the way it silences the fill: resolution is off by request, and
resolution_disabled already says so once with its row count.
One finding, though two passes produce it. compile_module runs the pre-flight in best_effort
whatever its own mode, so both reach this finding for the same subject; the compile de-duplicates on
the message (the _check_contig_ploidy idiom). That is safe here for the reason the rule requires —
neither message embeds a count resolution could change, so two passes over one subject produce the
identical sentence rather than two differing by a number.
The mishap matrix¶
Five identity columns the author may or may not supply, crossed with what the table says about them, is
a finite set. Every combination is either a round-trip fixed point on all three signatures, or it
fails in strict — enumerated and enforced in compiler/tests/test_resolution_matrix.py.
| Authored shape / mishap | best_effort |
strict |
Round-trip |
|---|---|---|---|
| rsid only, 1:1 fill | ✅ | ✅ | stable |
| rsid only, one-to-many (every locus can host the genotype) | ✅ | ✅ | stable |
| rsid only, pseudoautosomal pair (X + Y, same place) | ✅ | ✅ | stable — the compiler names it as one place; the enricher decides whether both reach the table |
| coordinate + alt, rsid resolved | ✅ | ✅ | stable |
| coordinate only, rsid and alt resolved | ✅ | ✅ | stable |
| pair (rsid + coordinate), table agrees | ✅ | ✅ | stable |
| withdrawn — the table records the rsid as retracted by dbSNP | ❌ refuses | ❌ refuses | — |
| ambiguous — several rsIDs for one allele | ⚠️ warning | ❌ refuses | stable |
| expansion drops a locus that cannot host the genotype | ⚠️ warning | ❌ refuses | unstable |
| every candidate locus contradicts the genotype → unresolved | ⚠️ warning | ❌ refuses | unstable |
not_found — the table records the rsid as genuinely absent |
⚠️ warning | ❌ refuses | unstable |
| no resolution row at all | ⚠️ warning | ❌ refuses | stable (nothing to lose) |
authored ref contradicts the table |
⚠️ warning | ❌ refuses | unstable |
| authored coordinate contradicts the table | ⚠️ warning | ❌ refuses | unstable |
Four things the table encodes that are worth saying out loud:
withdrawn is the only row that refuses in best_effort, and both sides ask it. A merged or
absent rsID leaves the annotation intact — the module is dated, or the label is unserved. A withdrawn
one is dbSNP repudiating the variant, so the annotation may be describing something that does not
exist, and carrying it under best_effort would publish a claim its own source has retracted. It is
never produced by the automated check (a retraction is indistinguishable from a never-assigned id
through the live API), so it fires only where a curator recorded it deliberately. validate refuses
it too, in both modes — it reads the injected table's own column and no resolved row, so the
compile-only exemption does not cover it (RM207); the same is true of ambiguous under strict.
Instability always means the table cannot be reproduced, never the bytes. artifact.digest is a
fixed point in every row above, including the unstable ones — a module that compiles at all compiles to
the same bytes twice. What is lost is resolution_signature: a not_found sentinel has no coordinate
so reverse writes no row for it; a dropped locus is simply not in the artifact; and where an authored
value contradicts the table the authored value wins, so the table's version is gone. Each is a real
reduction in what the injected table said, which is exactly why strict — whose contract is a
reproducible artifact — will not build on it.
ambiguous is the one exception in the other direction. It is round-trip stable, because the
enricher writes a single row carrying the deterministic pick while the candidate list rides in
rsid_alternates, which is provenance and outside the fact set. Strict still refuses, not because
anything is lost but because the label is a pick among equals rather than a finding, and an
all-or-nothing artifact should not rest on one. (An earlier draft of this analysis concluded ambiguity
could not be stable — from a hand-written two-row fixture the enricher would never produce. The real
shape is one row.)
A contradiction is an instability, not a difference of opinion. An authored coordinate that disagrees with the table used to be a warning and nothing more. It has to be stronger: the artifact keeps what the author wrote, so the table's position does not survive a reverse, and the next compile resolves from a table that no longer says what it said.
Reverse¶
reverse_module reads the parquet artifact only (never manifest.json) and emits into output_dir:
module_spec.yaml (always), variants.csv + resolution.csv (when weights.parquet exists;
resolution.csv gated on write_resolution=True), studies.csv (when present), one CSV per present
table kind, and one CSV per derived-fact sidecar whose parquet is present (frequencies.csv,
gene_metrics.csv, literature.csv, gene_validity.csv, clinical_assertions.csv, sources.csv),
plus overrides.csv when overrides.parquet is present (0.7, RM124).
- Preserved (round-trip-critical, Principle 7): every authored
VariantRow/StudyRow/table value; genotype phase (thephasedbit re-emitsA|Gvs sortedA/G); tri-state bools;priorityverbatim; poly-effect annotations keyed on(variant_key, genotype, conclusion, negatives)(genotypesince 0.6, RM80 / S29). - The authored shape is restored, not guessed:
authored_identsays which identity columns the author wrote, and reverse emits exactly those — an rsid-only row comes back rsid-only, a position-only row comes back position-only, a pair comes back as a pair. An expanded one-to-many rsid collapses back to the single row it was authored as, rather than one row per locus. See § Resolution for why the stored key alone cannot answer this and what it cost when reverse tried. resolution.csvemission carries the resolved facts back: oneResolutionRowper distinct positioned fact, keyed on the authored key (so an expansion's N loci are joinable to the one row they came from) withlocus_indexcounting within that key, andsource="reversed",status="resolved". Soreverse → compilereproduces the identicalartifact.digestwith no reference and no network — hardening Principle 7's round-trip from reference-dependent to self-contained.- The derived sidecars come back POST-overlay, and
overrides.csvcomes back beside them (0.7, RM124). The overlay is applied at compile, so what reverse reads out of the parquets is already the corrected table — and it re-emits the overlay too, which means the overlay applies twice and the fixed point is checked by test rather than assumed. It holds because all three operations are idempotent set operations: an update to a value already present, an insert of a row already keyed(subject, member), and a suppress of a row already absent are each a no-op. The alternative — emitting the pre-overlay table so the apply happens exactly once — would need the overlay to record the value it replaced, which is a derived cell inside an authored table and rots the moment the source moves.overrides.csvis written at the spec root under its one legal name, not throughsidecar_write_path: it is authored, likevariants.csv. - Provenance is deliberately dropped on the way back:
source/status/fetched_atare reset andrsid_alternates/rsid_current/rsid_statusare not written, because those columns are kept out of the fact set precisely so they never enter the artifact — there is nothing for reverse to read. Recover them by re-running the enricher. - Derived-fact sidecars round-trip through the same generic writer, minus the columns that are
recomputed rather than stored:
allele_frequencyis derived on write and is not aFrequencyRowfield, so it falls away by construction rather than by a special case, and re-deriving it on the next compile reproduces the identical parquet. - Normalized: title/description/report_title fall back to name-derived defaults; icon/color from args; curator/method from the most-common column value.
- Recovered from
manifest.json, not normalized:genome_build. It reaches the artifact through the manifest and no parquet column, and reverse used to re-emit the constantGRCh38— listed here, for a release, as a harmless normalization "out of the digest". It is not out of the digest, because the build decides the identity key: a GRCh37 module reversed as GRCh38 recompiled withga4gh:VA.…keys minted for GRCh37 coordinates, soartifact.digestmoved and the new key asserted an allele at a base the module never named._genome_build_from_artifactreads it;genome_build=(CLI--genome-build) overrides; a bare parquet directory with no manifest falls back toGRCh38, the format's own default. Seereference_examples/grch37_build/. - Lost (manifest-only, out of
artifact.digest):authorship,panel(deprecated in 0.6, removed at 1.0 — RM4; see below),provenance,logo,readme, and the verification attestation (RM45) —verification.jsonis not in the artifact, so there is nothing for reverse to read and inventing one would mint a claim nobody put; a reversed module carries nomanifest.verification, which is the honest says nothing, and re-attesting means re-running the checks. The RM73 closure rides that document and is lost with it, so a reversed spec is open and warns until a human closes it — deliberate, since reverse holds no key and no standing to declare someone else's authoring finished. That asymmetry costs nothing while the finding is a warning (warnings feed no digest and no signature, so the fixed point is untouched), and it is precisely what blocks promoting the finding to a refusal at 1.0: under a gate, step 3 ofcompile → reverse → compilewould refuse on every module. The three candidate answers are recorded in ROADMAP_1_0.md § RM73 (gate half). A consumer needing these readsmanifest.json(preserved verbatim by the forward compile). The test of whether something belongs on this list is whether losing it can change a parquet byte — which is whygenome_buildmoved off it.readmejoinslogohere for the same reason and with the same consequence:reversewrites no readme into the re-emitted spec, so a recompile of a reversed module attests none — the three signatures are still a fixed point, because prose was never in any of them.
Output artifact & hashing¶
ARTIFACT_PARQUETS is the artifact.files listing order — the tuple's own order, which
build_artifact walks, skipping absent files so a module lists exactly the parquets it has:
weights annotations studies ← the SNP core
activity_phenotype copynumbers repeat_alleles heteroplasmy
haplotypes allele_function diplotypes pgs pharm_variants ← the nine table kinds
frequencies gene_metrics literature gene_validity
clinical_assertions gwas_effects expression_effects
clin_sig_concordance clin_sig_authority_calls ← the derived-fact sidecars (the last two: RM130, 0.7)
sources ← the licence table
overrides ← the authored overlay (0.7)
It is not digest order, and this section said it was. artifact_digest sorts the listing by name
before hashing (integrity.py), so the tuple's position is invisible to the digest and a name is what
places an entry — overrides.parquet hashes between literature.parquet and pgs.parquet however
late it sits in the tuple. What the position does govern is the order a consumer iterating
artifact.files sees. Appending is still the right move for a new parquet, but the reason is the
listing, not the hash; what keeps an already-published digest still is that the module has no such
file at all.
Twenty-three names. LEAD_PARQUETS is the ten carrying a module's own annotation rows — weights plus the
nine table kinds — which is the publisher's is this a module rule and what discovery probes.
The per-parquet column lists are deliberately not reproduced here. They are derivable from the
models, and a hand-kept column list is precisely how SOURCES_FIELDNAMES lost a column
(@fieldnames-from-model); use just-dna-compiler describe or read the model. A point-in-time listing,
including which columns the compiler stamps rather than reads, is in
audit/COMPILER_FROM_CODE.md § 5 — dated evidence, not a maintained list.
ARTIFACT_PARQUETS(feedartifact.digest;_OUTPUT_FILESuntil 0.6):weights/annotations/studies.parquet+ the 9 table-kind parquets +frequencies.parquet/gene_metrics.parquet/literature.parquet/gene_validity.parquet/clinical_assertions.parquet/gwas_effects.parquet/expression_effects.parquet/clin_sig_concordance.parquet/clin_sig_authority_calls.parquet/sources.parquet/overrides.parquetwhen present — the diagram above is the tuple, and this sentence is not a second inventory of it. The sidecars enter the digest because a module carrying frequency data genuinely is different content — but adding one leaves the SNP core's bytes untouched (an explicit test). It is public because the publisher tier has to agree with it:just_dna_enricher.uploadderives its allow-patterns from this tuple, after a hand-kept copy covering three of the sixteen names silently dropped the rest at upload (S35/RM89).LEAD_PARQUETSbeside it names the ten that carry a module's own annotation rows —weightsplus the nine 0.4 families — which is the publisher's "is this a module" rule and what discovery probes._INPUT_FILES(feedmanifest.inputs, raw-bytes hashed):module_spec.yaml+variants.csv+studies.csv+ the 9 table-kind CSVs +overrides.csv(0.7 — every row of it is written by a human, which is what puts it here) — the authored surface, and the reason only the other files gained a second legal name and location in 0.6.resolution.csvis deliberately NOT here (nor inARTIFACT_PARQUETS) — it is a multi-producer artifact hashed only by the normalizedresolution_signature(a raw-bytes hash would be unstable across enricher/human/reverse producers).frequencies.csv,gene_metrics.csv,literature.csv, the 0.6 pairgene_validity.csv/clinical_assertions.csv,gwas_effects.csv,expression_effects.csvand the 0.7 pairclin_sig_concordance.csv/clin_sig_authority_calls.csvare out for exactly the same reason, each hashed by its own*_signature(the derived roster is_DERIVED_FILES; read it rather than this sentence).provenance.jsonis likewise out of the digest.- The derived-fact sidecars are deliberately NOT
_TABLE_KINDS. Those are authored DSL tables withAuthoredModelsemantics, the reserved-namespace guard, duplicate-key checks and raw-byte input hashing. A machine-produced reference-fact table is a third category — injected, fact-hashed, human-overridable — and folding it in would blur the line the 0.5 rework drew. - Manifest
Compilationfields the compiler populates:compile_success,compiled_by,compiler_version,ensembl_reference,compiled_at,warnings, and the 0.5 resolution provenance —resolution_mode(policy),fully_resolved(outcome — orthogonal axis, P5),resolution_subjects(0.6 — the denominator that flag covers),resolution_signature,resolution_sources. All out ofartifact.digest. The trust rule isresolution_subjects > 0 and (resolution_mode == "strict" or fully_resolved); without the first clause it is vacuously satisfied by a module that resolves nothing. - Manifest
frequency/gene_metricsblocks (0.5):signature,sources,datasets,row_count, pluspopulations/variant_counton the former andgeneson the latter. Separate blocks rather than extra fields onCompilation/Resolution, which are about rsID↔coordinate resolution only. Out ofartifact.digest.datasetsis the field a consumer reproducing an ACMG BA1/BS1 filter reads to know which release it is filtering against.
The three hashes and how they compose into (content_signature, resolution_signature, compiler_version)
⟹ artifact.digest are documented in SCHEMAS.md § identity & integrity.
Deterministic ordering — what is preserved, what is normalized¶
Parquet bytes depend on row order, so ordering is part of artifact.digest rather than a nicety. The
asymmetry below is intended: rows are preserved, presentation is normalized.
Preserved.
- Authored row order, through compile → reverse → recompile.
- Expansion order within a one-to-many rsID.
_sorted_locisorts on(locus_index, chrom or "", start or 0, ref or ""), which matches the deprecated DuckDB path'sORDER BY id, chrom, start, ref— the two produce byte-identical parquet, which is what let that path be retired without moving a digest. _symbolic_findingssorts on(table, reason, index, column), so the messages built from it are byte-stable. They reachmanifest.compilation.warnings, which is artifact-visible.table_citations(and its narrower siblingbinning_citations) returns first-occurrence order rather than sorted order, because it feeds emission order.Frequency.populationsis in canonical order (population_sort_key,globalfirst). Every other manifest facet list is sorted — this one is the exception, deliberately._write_resolution_csvemits weights first and never re-emits a key from the positional pass, sovariants.csv'saltsalways wins.
Normalized, not preserved. Column order and cell formatting in every reversed CSV; the
curator/method blank-vs-explicit split; unphased genotype allele order.
Deterministic tie-breaks where the library gives none. _module_name_from_parquets uses min()
over unique(), and _most_common uses min() over mode(), because polars orders neither. This is
the concrete form of the standing rule: never derive an emitted row or a manifest field from
set/dict iteration or from mode()/unique() without an explicit sort or tie-break.
Warning texts a consumer keys on¶
Since 0.7 the channel also carries codes and an actionability split (RM131)¶
manifest.compilation.warnings still carries every finding, with its exact text — nothing that greps
a phrase broke, and nothing here supersedes the fragments below. Two derived fields ship beside it:
| Field | What it is |
|---|---|
compilation.warnings |
the complete list, unchanged; the same sentences in the same order |
compilation.carried |
the subset no edit to the spec directory can clear — a limit of this tier or a fact of a source. Subtract it from warnings for the set an author still owes work on |
compilation.warnings_summary |
{code: count}, keys from vocab.VALID_WARNING_CODES, values summing to len(warnings) so the digest accounts for the whole channel |
The same three fields are on ValidationResult, CompilationResult and ClosureResult, filled on
every path including a failed compile. manifest.json sits outside artifact.digest (a Merkle root
over the parquet FileEntry list), so adding them moved no hash — and both are listed in
release_records.EXCLUDED_MANIFEST_FIELDS, routed to the warnings axis, because they move exactly
when warnings moves.
A code names the finding, never the function that builds it, and one code carries one remediation: two sentences cleared by the same edit share a code and say which cell (the weight-sign pair, the five orphan fact tables), two cleared differently do not. The set is published, so it is permanent within the major — additions are minor-legal, re-spellings are not.
Carried findings — the eleven an author cannot clear, and the reason each is on this side:
| Code | Why no authored edit clears it |
|---|---|
non_grch38_variant_keys |
VRS allele identity is GRCh38-only, so a non-GRCh38 module is coordinate-keyed |
resolution_skipped_cross_build |
resolution and the positional fill are GRCh38-bound (RM15) |
contig_ploidy_undecidable |
the build carries no pseudoautosomal table here, so ploidy cannot be decided |
rsid_expanded_to_multiple_loci |
the source maps one rsID onto several loci; that is a fact about dbSNP |
locus_hosting_undecidable |
deciding needs the reference sequence, which this tier never fetches (P2) |
vrs_id_unverifiable |
the recomputation is beyond this tier — _carried_vrs_warnings is the shape the split generalises |
vrs_coverage_incomplete |
the alleles a VA does not reach; minting more is not an authored edit |
measurement_spans_bins |
the format has no reading for an interval that straddles a boundary (RM56) |
verification_findings_recorded |
a disagreement with an archive, where the archive is the stale side often enough that nothing is owed |
gene_validity_superseded |
a curating body re-curated its own claim; the only edit available is deleting a true record |
gene_validity_currency_undecidable |
the source published several curations of one claim and not enough to order them |
Every published code, by the surface it comes from. Everything not in the table above is
actionable, which is vocab.ACTIONABLE_WARNING_CODES, derived by subtraction:
- spec directory /
module_spec.yaml—table_file_misplaced,table_file_near_miss,sidecar_spelling_deprecated,module_version_coerced,panel_block_deprecated - coordinates and the build —
non_grch38_variant_keys,contig_ploidy_undecidable,contig_ploidy_mismatch - resolution —
resolution_disabled,resolution_not_injected,resolution_skipped_cross_build,positional_identity_contradicted,positional_rows_unjoinable,rsid_unresolved,rsid_without_resolution_label,rsid_expanded_to_multiple_loci,rsid_ambiguous,rsid_coordinate_disagrees,locus_hosting_undecidable,locus_cannot_host_genotype,rsid_no_hosting_locus - VRS —
vrs_id_unverifiable,vrs_coverage_incomplete variants.csvcoherence —weight_sign_disagrees_with_effect,genotype_allele_not_at_locus,effect_allele_not_at_locus,genotype_coverage_gap,quality_floor_inverted,missing_allele_marker_in_alts,vcf_pointer_key_collision,vcf_pointer_unselected_element,composite_gene_cell,symbolic_allele_unusable- binning tables —
bin_tiling_inferred,bin_tiling_contradicted,bin_coverage_gap,bins_ungrounded,measure_field_fractional,measurement_spans_bins,deprecated_bin_modifier - PGx tables —
star_allele_undefined,diplotype_definitions_identical,diplotype_phase_ambiguous - studies and literature —
study_variant_orphan,duplicate_study_citation,p_value_encodings_disagree,study_effect_allele_not_at_locus,citation_not_in_pubmed,literature_row_uncited,quote_counter_stale,quoted_article_license_restrictive - sources and licensing —
source_row_unused,source_terms_unrecorded,declared_license_disagrees - the injected fact tables —
derived_row_orphan,faf95_exceeds_frequency,oe_lof_outside_interval,oe_lof_disagrees_with_counts,clin_sig_contradicts_frequency,clin_sig_concordance_contested,gene_validity_superseded,gene_validity_currency_undecidable - the overlay —
overlay_update_unmatched,overlay_update_target_unreachable,overlay_answer_vindicated,overlay_targets_missing_table,overlay_rows_suppressed - verification and closure —
verification_two_copies,verification_unreadable,verification_stale,verification_findings_recorded,module_not_closed,closure_discarded_unreadable_record
overlay_answer_vindicated (RM117) is the one finding in this catalogue that reports good news,
and it exists because the alternative was actively misleading. clin_sig_concordance.csv holds
contested subjects only and is rewritten whole, so a subject leaving it means the authorities stopped
disagreeing — which is exactly how an author learns the archive caught up with them. An overrides.csv
row answering that conflict then reaches nothing, and without this code it drew the generic the
subject may be mistyped line, put to an author whose judgement had just been confirmed. It says
nothing about who was right about the biology: the authorities agreed, and the overlay row is now
unnecessary.
Which tables can say it is a field on each table, decided once for all nine (RM290).
OverlayTarget.vindication is sole, withheld or none, with its reason beside it, and
overrides.VINDICATING_OVERLAY_TABLES is read off it. Only the concordance record is sole.
resolution.csv is withheld, because a re-derivation renumbers locus_index and the author's locus
can return under another index. The seven merged tables are none: their keys are stable, so a source
adopting the author's value leaves the row in place and the update keeps matching. Absence there is
never agreement. Every other table keeps the generic finding it had.
The two gene_validity_* codes are new in 0.7 (RM108) and are the first fact-table findings the
pre-flight also computes, so validate reports them exactly as compile does. They stay apart on
purpose: gene_validity_superseded says a later curation replaced an earlier one and the manifest now
publishes the later verdict, while gene_validity_currency_undecidable says several curations exist
and nothing orders them — a tie on classification_date, or a row stating none — so every
classification in that group is still published. One number meaning both would tell a reader the
archive moved on when it had simply not said enough.
overlay_rows_suppressed is new in 0.7 and is the one finding here that reports a decision rather
than a defect: a suppress removes a row and leaves no trace of the removal in the build product, so
the overlay says so — one line per reason, with a count, and counted over the overlay's own rows
rather than over the rows removed. That is what keeps it stable across compile → reverse → compile,
where by the second lap the derived table is already post-overlay and the suppress matches nothing.
overrides.csv: {count} suppress override(s) remove {table} row(s) from the compiled artifact,
where nothing else records the removal: {reason}
test_warning_codes.py walks the emission sites and asserts an equality against the vocabulary in
both directions, so a code with no emitter and an emitter with no code both fail; it also asserts this
document lists every member, because a catalogue missing one sends a reader hunting for it.
Two consequences worth knowing before you build against this, both recorded rather than left to be discovered:
- A summary is either empty or complete, never partial.
classifywithholds — an emptycarriedand an emptywarnings_summary— when a channel carries no classified findings at all, which is what a caller passing plain prose intoCompilationResult(warnings=[...])gets; that call has been legal since 0.6 and Principle 3 keeps it legal. It refuses only a part-classified channel, which no legitimate caller can produce. So read an empty summary as this compile did not classify the channel, never as there is nothing to report, and read a non-empty one as accounting for the whole ofwarnings. - The code vocabulary is closed, so an older reader refuses a newer manifest.
warnings_summaryvalidates its keys, andread_manifestraises on a code added after thejust-dna-formatyou have pinned. That is the standing cost of every closed vocabulary on a published field here (VerificationRecord.checkis the shipped precedent) and Principle 6 takes it deliberately, but "additive" describes the writer: a consumer reading manifests from newer compilers upgrades the schema package alongside them.
carried holds full message text, which nearly doubles the channel — measured at 1.84× across the
reference corpus, 1.96× on pathogenic_clinvar. That is the shape the item decided (a list beside,
so a consumer subtracts) rather than an oversight, and the cheaper encodings are weighed in
ROADMAP.md § RM138.
The phrases¶
A warning's text is an API. The manifest carries the prose and, for anything published before the structured field existed, no other handle — so a downstream consumer greps it. Four fragments are named constants for exactly that reason:
UNJOINABLE_PHRASE = "have no chrom+start"
QUAL_INVERSION_PHRASE = "QUAL means the opposite thing on the record this row is read from"
MISSING_ALLELE_PHRASE = "is VCF's MISSING marker, not an allele"
UNCLOSED_PHRASE = "records no closure"
UNJOINABLE_PHRASE has a named external consumer: just-dna-registry 0.11.3 pins
UNJOINABLE_MARKER = "have no chrom+start" in its facet builder. The structured replacement
(manifest.compilation.positional_rows / positional_rows_placed) shipped in 0.6 and the phrase is
still not retired, because artifacts published under 0.5 carry neither field. Retiring it is a
decision about the published corpus, not about this code.
The positional-joinability sentence, as its format string — the counts and both trailing clauses are computed per table, so no example is invented here:
{csv_name}: {unplaced} of {rows} row(s) have no chrom+start, so this table joins by rsID only —
a VCF whose ID column is empty matches none of them. {detail}.{partial_note}
{detail} is one of exactly three sentences, and the distinction between them is the whole point —
not consulted, consulted and found nothing, and consulted and refused to guess are three
different situations a consumer must not collapse:
the resolution table was not consulted for this table — see the skip reported above(orresolution.csv names N of them and was not consulted for this table — …);no resolution.csv row places them — run `just-dna-enricher enrich` first;resolution.csv names N of them, but at more than one locus or at one the row's own allele contradicts, so the compiler leaves them unplaced rather than picking.
The VRS coverage headline, as its format string, followed by one indented {count} allele(s):
{reason} line per gap reason, sorted by descending count then reason:
VRS allele identity covers {identified}/{alleles} allele(s) in resolution.csv ({pct}) —
{missing} carry no ga4gh:VA. id. Anything keying on the VA sees only the covered fraction.
Its twin, for ids that are present and that this tier cannot recompute, is grouped the same way
since S67 — one line per reason, descending count then reason, three variant_keys named
and the rest counted:
{count} allele(s): vrs_id could not be verified — {reason}; carried unverified ({a}, {b}, {c}, and
{n} more).
The two halves used to disagree about shape, and which half an allele landed in was decided by
whether the enricher happened to mint an id for it — nothing else. So _vrs_coverage aggregated the
alleles with no id while _verify_vrs_ids emitted one line per id present, and warning noise
ran inversely to how well-resolved a module was: 80 of one 101-row module's 85 warnings came from
here, with the three findings its author could act on at positions 83, 84 and 85, while a 57,595-row
module with nothing minted was quiet. _BLAME_ROW findings are not grouped — they are errors,
they are rare, and each names a row that contradicts itself, which is the one thing a per-reason line
would take away.
Three more sentences worth quoting exactly, because each states a consequence rather than a status: the
closure reminder (This module records no closure: … Compiling without one is a warning today;
requiring it is filed for 1.0 (RM73).), the symbolic-allele drop (Those row(s) are DROPPED from the
compiled artifact — reverse will not re-emit them — and --strict refuses instead.), and --no-resolve
with an injected table present, which spells out that the flag names Ensembl but is the master switch
and that there is no "do not reach the network" flag because the compiler never does.
Four phrases added since 2026-08-28, each carrying its code:
derived_row_orphan (the concordance arm): {table} records {n} subject(s) no variant in this module
carries: {orphans}. The record is rebuilt whole on every run, so this means variants.csv was narrowed
since the comparison last ran — re-run it rather than editing the table.
declared_license_disagrees: module declares license {declared} and {standing}. Not adjudicated here — a
compatible pair is legitimate, an incompatible one is a real problem, and only a human can tell which.
A declaration matching some but not all of them is the ordinary mixed-licence case, where the most
restrictive term binds the whole artifact.
gene_validity_superseded: gene_validity.csv carries a later curation for {n} gene-disease claim(s), so
an earlier row is superseded and kept: {groups}. Nothing is deleted and nothing is wrong — the newest
classification_date is read as current, both rows stay so the drift is visible, and
manifest.gene_validity.classifications publishes the current one. A curating body re-curating is not
an error in your module.
gene_validity_currency_undecidable: gene_validity.csv carries several curations for {n} gene-disease
claim(s) and nothing orders them: {groups}. Either two rows share a classification_date or one states
none, so no row is called current and none superseded — every classification in those groups is
published, which is the honest answer rather than a winner picked from an identifier. Withheld
deliberately, not skipped.
The overlay's three warnings (0.7, RM124; the third split out on 2026-08-31), and there are only three on purpose. overrides.csv
applies before any check reads a derived row, so what a check reports is what the module asserts. No
operation reports its own no-op — reverse_module emits the post-overlay table plus the overlay, so
on a recompile update-already-equal, insert-already-present and suppress-already-absent are all three
true of a healthy module, and reporting any of them would make a module and its own round trip
disagree on manifest.compilation.warnings. What is left is the pair an overlay operation cannot
manufacture for itself. The unmatched-update case splits in two for the tables whose rows the compiler
can drop before the parquet (literature.csv loses its uncited rows, resolution.csv has no parquet at
all): an update whose subject is still cited or positioned names a table that is short and asks for
the enrichment pass to be re-run, while one whose subject no artifact of this module can carry is
unreachable — and unreachable fires matched-or-not, because reachability is a property of the
module and answers the same on both laps, where "did it match" is exactly the quantity a reverse moves
(@lap-stable-means-a-property-of-the-module; an earlier cut classified only the unmatched set and
reported on lap 2 and not on lap 1). Every other table keeps the three-readings message.
overrides.csv: {n} update override(s) name a row {table} does not carry: {subjects}. Three readings and
nothing here separates them — the subject/member may be mistyped, the source may have stopped
publishing the row the correction was about, or the compiler dropped the row before the parquet so a
reversed module cannot carry it. Neither an insert nor a suppress reports this: an insert creates the
row and a suppress is satisfied by its absence.
overrides.csv: {n} update override(s) name a row {table} does not carry, though this module could carry
it: {subjects}. The subject is cited or positioned, so the table is short rather than the correction
wrong — re-run the enrichment pass that writes {table}. Neither an insert nor a suppress reports this:
an insert creates the row and a suppress is satisfied by its absence.
overrides.csv: {n} update override(s) name a {table} row no artifact of this module can carry:
{subjects}. Two readings and nothing here separates them — the subject may be mistyped, or the
correction may be aimed at a row the compiler drops before the parquet (an uncited citation, an
unresolved locus), in which case the correction is fine and simply has nothing to reach. Reported the
same way whether or not the row is present today, so a module and its own round trip agree.
overrides.csv corrects {tables}, which this module does not carry. An overlay lies on top of a derived
table and never creates one, so those rows change nothing. Run the pass that writes the table, or drop
the override rows.
Both warn in both modes and both are aggregated into one sentence rather than one per row. The
consequence a consumer should know is the one no message can carry: a suppress with a typo'd
subject does nothing, forever, and cannot warn — check a suppression by reading the compiled table.
The uncited-literature sentence names its sites, so it has been reworded twice as sites were added —
no study or bin in 0.6, no study, bin or pharm row in 0.7. The code literature_row_uncited is
the stable handle; the phrase is pinned by the suite, which is what makes each rewording deliberate.
The concordance record's one warning (0.7, RM130). clin_sig_concordance.csv names the subjects
where the module's clinical call and an annotation authority's disagree, or where two authorities
disagree with each other, and this is the sentence that says so at validate and at compile:
clin_sig_concordance.csv records {n} contested subject(s): {opposed} of them opposed calls
(pathogenic-class against benign-class)[, {k} with an authority that could not be consulted]. A
contested subject is a question, not a defect — half the time the archive is the stale side, which is
why this never fails a build in either mode. Answer one by adding a row to overrides.csv naming table
'clin_sig_concordance.csv', the subject's variant_key and its genotype, with the reason you stand by
the module's call.
Three things about it are decisions rather than wording. It never escalates under --strict, for
the reason the check that produced the rows does not: escalating would have this format arbitrate a
clinical dispute. It is actionable rather than carried, unlike verification_findings_recorded
one section down, and the difference is real — a number sitting in verification.json is not
something an author can move, while a contested row is answered by writing an overlay row, and the
count here is taken over the post-overlay table so answering one clears it. And it names
overrides.csv and never provenance.json's outranks: the two are the same idea one table
apart, 0.7 settled the overlap as a succession in the overlay's favour, and an author meeting this
warning for the first time should be sent to the mechanism that survives 1.0.
The count is embedded although both passes emit the sentence. That is normally the trap where a message carrying a number is rebuilt from inputs resolution changed in between; it is not one here, because no compile step between the two passes touches this table or the overlay above it, so both passes reach a byte-identical sentence and the existing de-duplication collapses them.
Substrings the suite pins as contract, so changing one is a deliberate act rather than a reword:
test_validate_agrees_with_compile.py holds "forbid sale", "does not match the id recomputed",
"could not be verified", "p_value_num says", "not among the", "IUPAC ambiguity code",
"not valid YAML", "must be a mapping"; test_strict_compile.py holds "unresolved genomic
positions"; test_roundtrip_regressions.py holds "pointer, not an expression".
CLI, and what it maps onto¶
just-dna-compiler (Typer) is a thin shell over the Python API. Exit 0/1, CI/registry-gateable.
| Command | Python API | Notes |
|---|---|---|
validate <spec> |
compiler.validate_spec |
--strip-identity / --authority-key. Read-only — it never stamps a closure |
close <spec> |
compiler.close_module |
--by, --private-key. Writes the RM73 closure into verification.json, bound to the authored bytes. Refuses an invalid spec; a warning does not refuse |
compile <spec> <out> |
compiler.compile_module |
--strict/--no-strict, --resolve/--no-resolve, --compression, --compiled-by, and the deprecated --ensembl-cache (routes to the enricher; removed at 1.0). Prints digest, content_signature, resolution_mode/fully_resolved/resolution_signature |
signature <spec> |
compiler.content_signature |
no compile, no reference |
Two compile_module parameters the CLI does not expose, stated rather than left to be
rediscovered (RM218). ensembl_reference and ba1_threshold are Python-API-only, and the first has a
visible consequence: manifest.compilation.ensembl_reference cannot be stamped by the shipped
command at all, so a module compiled through the CLI carries no record of which reference the
compile was told about. ba1_threshold tunes the ACMG BA1 common-allele cutoff, which the parameter's
own docstring frames as something a module curating a common recessive carrier allele would want to
move — also unreachable without importing the API.
Neither is a defect and neither is a decision anyone took: test_cli_parity.py guards the surfaces
the schema tier cannot expose and does not assert compile-flag parity, so the two drifted in without
being weighed. Whether the CLI should grow them is RM278 (open, a minor) —
what is settled is that the gap is written down instead of being a surprise.
| reverse <parquet_dir> <out> | compiler.reverse_module | --resolution/--no-resolution (default on) + display overrides |
| verify <module_dir> | format.integrity.verify_manifest | --public-key, --check-inputs/-logs/-provenance/-logo/-readme/-derived |
| keygen | format.signing.generate_private_key_pem + public_key_b64_from_pem | --out (refuses to overwrite) |
| sign <module_dir> | format.signing.sign_digest | --private-key |
| reference | format.reference.authoring_reference / json_schemas | --summary, --schemas |
| template <kind> | draft.blank_template + authoring_requirements | the SNP core, every optional table kind, and sources.csv |
| stub <kind> | draft.stub_template | --rows |
| requirements <kind> | draft.authoring_requirements | --json |
| scaffold <spec> | scaffold.scaffold_module | --kind, --rows, --dry-run |
| describe <kind> | hints.describe_table | one table's columns + pick-lists, each column's category and its vocabulary's per-member notes |
| hint <kind> | hints.inspect_rows | --file/--row, --json |
| sweep <before> <after> | sweep.compare_outputs + gate_findings | --spec-root (compile AFTER with the installed compiler), --release (run the release gate; exit 1 on a finding), --json. A release-sequence command, not an ordinary test — it needs the previous release installed |
--no-resolve is the master switch, not an Ensembl switch (S14). With an injected resolution.csv
beside the spec it used to compile successfully with chrom=None on every weight row — a silent
success, the worst shape a mistake takes. It now warns and names the size of what it discarded
(N row(s), covering K variant key(s) — rows, not keys, since a one-to-many rsid contributes several),
for the same reason vrs_alleles ships beside vrs_alleles_identified: a warning that quantifies over
a table should publish the denominator. The message also states that there is no flag for "do not
reach the network", because the compiler never does (Principle 2, verified branch by branch including
the deprecated ensembl_cache path, which reads an injected local cache) — omitting this flag is that
request. Renaming the parameter is a 1.0 conversation; it is part of a published signature.
Four rows are bold because they belong to just-dna-format, which ships no CLI of its own —
Typer would breach its pydantic-plus-cryptography dependency floor (Goal 2). So anything the schema
tier owns that a user needs has to surface here, and three of the four did not until 0.5:
sign --private-keydemanded a key file the toolchain could not produce, andverify --public-keydemanded a string onlypublic_key_b64_from_pemcould derive. Signing was therefore CLI-complete only for someone willing to write Python — the exact gapverifyhad been added to close, left open one step upstream.keygencloses it; the key is unencrypted PKCS#8 (whatsign_digestreads), which is a deliberate limit rather than an oversight: this bootstraps a key, it is not key management, and a passphrase prompt would imply custody guarantees nothing here provides. It refuses to overwrite an existing key, because every signature made with the old one would stop verifying and a published artifact's bytes are never mutated.authoring_reference()had no route at all, which hurt most for the consumer that most needs it: an MCP surface offering an author the valid values had to importjust_dna_format.referenceand write Python.describeanswers that for one table;referenceanswers it for all of them plus the vocabularies, the open-vs-closed flag,REQUIRED_ANY_OFand the palette.
template sources.csv used to deny the table existed (S21, 0.5.4). DRAFTABLE held the SNP core
and the optional table kinds, so blank_template("sources.csv") answered "is not an authored table of
this format" — false, and said by the surface an author reaches for instead of reading the models,
which is what the consumer who reported it then had to do. The other three fact sidecars stay out
because an enricher pass writes them and nobody starts one by hand; this one the schema instructs a
human to write (a source read by hand leaves no source cell for the coverage check to find) and the
compile licence gate reads it and nothing else. Its natural key is (source, layer) — the same one
licensing.merge_sources_csv merges on, borrowed for the reason every _CORE_DUPE_KEYS entry is
borrowed: a draft must not append a row the other writer treats as already present, and one source
legitimately appears at two layers.
A drift the audit found in the same place. authoring_reference() reported requiredness with
pydantic's two-way is_required(), while just_dna_compiler.draft had already been fixed to the
three-way required / defaulted / optional split — the middle one being the trap where
MeasureBinRow.measure_kind is not required and not safely left blank either. Two surfaces
answering one question, and the drift-proof one was the stale one. The split now lives in
format.base.field_category, the only tier both can import from; the reference emits it as
category beside the existing required key (kept, since removing a published key breaks consumers
and required is insufficient rather than wrong).
Not levelled, deliberately: just-dna-enricher mirrors template but not stub, requirements,
describe, hint or scaffold. The offline authoring surface belongs to the compiler; the one
mirror exists so a PGx author working through the enricher does not have to switch binaries for a
header. Adding the other four would duplicate a surface that has an owner, and removing the mirror
would break scripts for no gain.
The release-record sweep (RM126)¶
just_dna_compiler.sweep. The instrument that measures what a release changed about compiled output,
and the gate that refuses a release whose measurement carries no declaration. The record it feeds and
the pure needs_recompile a consumer reads live in the format tier — see
SCHEMAS § The release record
for the shape, the vocabularies and the interval algebra. This section is the operation.
Why a measurement and not a map. A hand-kept per-release map was the first repair anyone proposed
and it is the defect wearing a public name (@registry-completeness): five of the six RM104–RM111
fixes were a derived value restated by hand. So the sweep measures, and the gate makes the measurement
force the declaration rather than leaving the author to remember writing one.
What it compares, and the one discipline that makes it mean anything¶
Two trees of compiled output — one produced by the previous release, one by this one — from the same
spec root. Feeding each side its own tree's reference_examples/ measures spec drift as compiler
drift: between v0.6.1 and v0.6.6 one example's README moved, which is enough to shift
manifest.readme and inputs. One spec root, two compilers.
Five axes, and each is computed apart because a consumer acts on each differently:
| axis | how it is measured |
|---|---|
parquet_schema |
every parquet's {column: dtype}, read with scan_parquet(...).collect_schema() |
parquet_bytes |
artifact.digest |
content_signature |
the manifest field |
manifest_fields |
every dotted path in the manifest, minus EXCLUDED_MANIFEST_FIELDS |
warnings |
the compilation.warnings set, reported apart and never folded into the above |
EXCLUDED_MANIFEST_FIELDS is a published registry with a reason per member, and
compilation.compiler_version is the one that matters: it moves on every release by construction,
so counting it would make every record fire on every module in every release and rebuild, inside the
instrument, exactly the false-positive class the interval shape exists to avoid.
compilation.compiled_at and compiled_by are environment; content_signature, artifact.digest and
artifact.files are routed to their own axes rather than dropped.
A list is a leaf. Indexing into one would make an inserted element rename every path after it, and
stats.genes is what a consumer keys on anyway. A missing key and a present null are distinguished
by a sentinel, not by dict.get — otherwise literature: null growing the RM119 counters reads as
unchanged on the literature path itself.
Backfilled, and what the numbers say¶
The two records shipped in 0.7 were measured on 2026-08-28 over all sixteen reference_examples/,
compiled from one spec root under each published release in turn with just-dna-compiler and
just-dna-format pinned together (the workspace cuts all three packages at one number). 0.6.0 →
0.6.1 moved nothing — a measured zero with its denominator in the record's evidence, never
silence. 0.6.1 → 0.6.6 moved parquet_schema and parquet_bytes on 10 of 16 (RM120's authored
curator column growing studies.parquet), manifest_fields on 9 (stats.genes/stats.gene_count
on seven, RM121; literature.quotes_unchecked on three, RM119) and content_signature on none.
The 0.6 line published three compiler releases — 0.6.0, 0.6.1, 0.6.6. The tags in between never
reached PyPI, so no stored artifact's compiler_version can name one, and the chain is complete over
what a consumer can actually be holding. Older intervals stay honestly unknown.
Where the gate runs, and what fails¶
In the bump → uv sync → tag sequence, not as an ordinary test, because it needs the previous
release actually installed. A release whose sweep shows a changed axis or field that no ReleaseRecord
declares fails.
# Fresh trees every time. `compile_module` only mkdir -p's its output, so reusing a fixed path
# inherits the last cut's modules: a renamed one lingers as a stale directory the sweep still reads,
# and a parquet a module no longer emits still gets globbed into its schema.
BEFORE=$(mktemp -d) && AFTER=$(mktemp -d)
# 1. the BEFORE side, under the previous published release, against THIS tree's specs
for d in reference_examples/*/; do
uv run --isolated --no-project \
--with just-dna-compiler==<previous> --with just-dna-format==<previous> \
just-dna-compiler compile "$d" "$BEFORE/$(basename "$d")"
done
# 2. bump the versions, `uv sync`, THEN measure and gate — in that order, see below
uv run just-dna-compiler sweep "$BEFORE" "$AFTER" --spec-root reference_examples/ --release <new>
uv sync before the sweep, not after. --spec-root builds the AFTER tree with whatever compiler
is installed, so running the sweep on a stale environment measures the previous release against
itself: every axis reads False and the whole thing looks like a clean release that changed nothing.
The gate refuses it — it checks the interval's upper end against --release and rejects a
degenerate interval outright — but the fastest way not to meet that message is to sync first.
Step 2 exits 1 until release_records.RELEASE_RECORDS gains an entry for <new> covering what moved.
The measured half is what SweepMeasurement.as_record(...) produces — with an empty declared
list on purpose, so the gate keeps refusing until somebody says whether each movement was a
correction (the value we published was wrong) or an addition (it was absent). That is the whole
mechanism.
The two halves are written at different times, and that is where a record drifts (RM161). An item
landing after the measurement adds its DeclaredChange and leaves manifest_fields behind it, which
turns this gate red at the one moment nobody wants to be debugging a record — 0.7.0 reached its own
pre-build run missing gene_validity.superseded_count and identity.version_coerced_from, both
declared the morning they landed. Re-measure whenever a declaration is added, not only when the
release is cut. A test catches the addition half offline: a declared addition must appear in
manifest_fields, since a field that did not exist before moves wherever its block appears. A
declared correction is deliberately exempt — it can be true of the release and unmeasurable on this
corpus, which is what the OVERDECLARED_NOTE_PHRASE note is for.
The gate's finding phrases¶
Named constants, for the same reason a warning's text is: a release script greps them.
NO_RECORD_PHRASE = "has no release record"
UNDECLARED_AXIS_PHRASE = "moved and the release record does not record it moving"
UNDECLARED_FIELD_PHRASE = "moved and the release record does not list it"
UNDECLARED_KIND_PHRASE = "moved and nothing declares it a correction or an addition"
WRONG_PREVIOUS_PHRASE = "was measured against a release the record does not name"
WRONG_VERSION_PHRASE = "did not measure the release being gated"
DEGENERATE_INTERVAL_PHRASE = "measured one release against itself, so it measured nothing"
UNMEASURED_MODULE_PHRASE = "could not be measured on both sides, so the sweep says nothing about it"
NO_MODULES_PHRASE = "measured no module at all"
OVERDECLARED_NOTE_PHRASE = "is declared and this sweep did not see it move"
REGRESSED_MODULE_PHRASE = "compiled under the previous release and does not compile under this one"
UNDECLARED_UNMEASURED_PHRASE = "has no output from the previous release and the record does not list it"
OVERDECLARED_UNMEASURED_NOTE_PHRASE = "is declared unmeasured and this sweep measured it on both sides"
The middle four are about the sweep rather than about the record, and they exist because a gate
that only asked is this movement declared? passed every sweep that measured nothing. A stale
environment, a module whose compile broke under the new release, and two trees sharing no module at
all all produce an all-False measurement, and none of them is a release that changed nothing.
--release accepts the stamped just-dna-compiler X.Y.Z spelling as well as the bare one, for the
same reason needs_recompile does.
OVERDECLARED_NOTE_PHRASE is a note, not a finding, and does not fail the release: the reference
corpus is sixteen modules and a real correction can land on a shape none of them has. It is still
printed, so over-declaring is visible rather than invisible.
The last three are RM139's split of one side only, below.
Which side a module is missing from (RM139)¶
UNMEASURED_MODULE_PHRASE stays in every message here, so a script grepping it catches all of them.
What changed is the clause after it, because the two directions are facts about different
releases:
| the module is | what it means | the gate |
|---|---|---|
in BEFORE, not in AFTER (only_before) |
this release cannot compile a spec the previous one could — a regression, or a stale reused BEFORE directory holding a module the spec root no longer has | fails, unconditionally, with the compiler's own errors beside it where --spec-root built the AFTER side. as_record refuses to mint over one |
in AFTER, not in BEFORE (only_after) |
the previous release produced no output for it: its spec uses a column that release refuses under extra="forbid", or the example is newer than that release |
fails until ReleaseRecord.unmeasured names it. Then it is measured-nothing rather than measured-zero |
The first real use of the gate hit the second row, which the rule did not model: RM70 put the optional
requires_callable column on pharm_variants.csv, reference_examples/cyp2c9_warfarin_grch37/ uses
it, and 0.6.6 refuses that spec. Nothing failed — the module has no before state at all, so no
like-for-like comparison exists — and it recurs in every minor that adds an authored column and
exercises it in the corpus. The 0.7.0 cut stated the exclusion in evidence prose the gate cannot
read and the tag was waved through by hand.
unmeasured is a denominator, not an exemption, and the difference is the equality: a module the
sweep measured on both sides cannot be excused by listing it (that is reported as a note), a
regression cannot be excused by listing it, and a movement on a measured module still gates however
the list reads. What it buys is that the exclusion is forced into the published record by the
measurement — as_record fills it — instead of living in a sentence nothing checks.
Under --json, stdout is the JSON document and nothing else — the notes and the success line go
to stderr there. The caller of that flag is a release script piping to jq, and a note printed after
the blob breaks exactly the consumer the flag exists for. Without --json they read on stdout as
usual.
The warnings axis, and the seam RM131 fills¶
compilation.warnings is a published manifest field, so the naive reading folds it into
manifest_fields — and then RM131's restructuring and RM134's new checks report a manifest field
changed on essentially every module in 0.7, and a registry acting on that mints an immutable PATCH
across a whole catalogue for a reworded message. It cannot simply join compiled_at in the excluded
set either, because a new warning can be a real signal.
So it is its own declared axis, outside RECOMPILE_DRIVING_AXES. RM131's carried split landed in
0.7 and is that discriminator: compare_module now reports carried_added beside
actionable_added, read off the after manifest's compilation.carried rather than re-derived from
prose. A carried finding appearing is usually this repository saying more about a limit it always had;
an actionable one appearing is work arriving at somebody's door.
axes["warnings"] deliberately still fires on any movement of the set. Narrowing it would make a
published axis mean something different from what every record already written claims about it, and
the axis drives no rebuild anyway — the split buys the reading, not a new gate. A manifest with no
carried field (anything compiled before 0.7) reports every addition as actionable, which is the safe
direction: it never tells a reader that a finding they could fix is unfixable.
compilation.carried and compilation.warnings_summary both join compilation.warnings in
EXCLUDED_MANIFEST_FIELDS, for the same reason and with the same consequence — they are derived from
it, so they move exactly when it does.
What this does not touch: SpecRow.needs_upgrade is self.upgraded() != self, computed over
authored row content and a hard filter in the marketplace. A warning never touches an authored row, so
no warning change can flip it. Verified rather than assumed, because the failure mode would have been
modules silently vanishing from a catalogue.
Coverage table (0.3 / 0.4 features)¶
| 0.3 / 0.4 feature | Validated | Materialized (→ parquet) | Computed / derived | Status |
|---|---|---|---|---|
direction (VariantRow) |
✅ full vocab | ✅ weights.parquet — the authored value only, never a derivation |
✅ Python read-time only: effective_direction / upgraded() from state(+weight) |
complete — but read the two cells apart: a state-only module ships an empty column |
stat_significance (VariantRow, StudyRow) |
✅ full vocab | ✅ authored value only | ✅ Python read-time, derived from state (not inferred from p_value) |
complete, same split as direction |
effect_size (VariantRow, StudyRow) |
✅ float | ✅ | — | complete |
effect_measure (VariantRow, StudyRow) |
✅ permissive (open) | ✅ | — | complete (intentionally open) |
effect_allele (VariantRow) |
✅ nucleotides | ✅ | ✅ membership in {ref} ∪ alts (0.5) → warning / error in strict; ⛔ still no strand reconciliation |
validate + membership check |
genotype (VariantRow) |
✅ grammar (phased/hemizygous/sorted) | ✅ alleles + phased |
✅ membership in {ref} ∪ alts (0.5), unioned across a one-to-many rsid's loci |
complete |
flags (VariantRow) |
✅ open; split; reserved set | ✅ List[str] |
✅ unknown-tag INFO (ValidationResult.info) |
complete |
trait_efo_id (VariantRow, StudyRow) |
✅ CURIE(s) | ✅ | — | complete |
doi (StudyRow, RM11) |
✅ DOI grammar, verbatim | ✅ studies.parquet |
— | complete |
provenance_quote / provenance_regex (StudyRow, RM12) |
✅ free-text / author-time re.compile |
✅ studies.parquet |
— | complete (P1 pattern grammar; matched consumer-side) |
authorship (ModuleSpecConfig/ModuleManifest, RM14) |
✅ Contribution (role closed, kind open, extra=forbid) |
✅ manifest (out of digest) | — | complete (metadata; not reversed) |
clin_sig (VariantRow) |
✅ full vocab | ✅ | ✅ ↔ pathogenic/benign aliases |
complete |
module.version (ModuleInfo, 0.4.1) |
✅ freeform advisory (legacy v2/3) |
✅ manifest Identity.version iff valid SemVer; reverse_module(version=) re-emits |
✅ normalize_version preview (RM17 enforces) |
complete (advisory) |
| authority-key strip (0.4.1) | ✅ inject-only pre-strip; dropped → .info; typo'd still extra=forbid |
— | — | complete (DI) |
| strict compile (0.4.1) | ✅ strict=True fails (pre-write) on an unresolved (chrom, start) |
— (refuses a partial) | — | complete (opt-in) |
content_signature (0.4.1) |
✅ over raw authored rows, normalized+sorted, name-/Ensembl-independent | ✅ manifest (out of digest); signature CLI computes it without recompiling |
— | complete (canonical dedup identity) |
resolution.csv path (0.5) |
✅ resolve_from_table consumes injected facts; digest-parity with the DuckDB path proven; provisional shape (§ note) |
✅ drives weights.parquet coords; resolution_signature/resolution_mode/fully_resolved/resolution_sources → manifest (out of digest) |
✅ fill / expand / verify (pure, no duckdb) | complete (preferred path) |
| VRS allele identity (0.5) | ✅ stdlib derive_vrs_allele_id; every member of vrs_id recomputed and verified per ALT, plus a coverage report (mode-dependent severity) |
✅ variant_key is the VA for a resolved substitution → weights/annotations.parquet |
✅ minted per ALT from (chrom, start, ref, alt); a multi-allelic site names each allele, and the key still falls through to the coordinate for indels/MNVs/multi-allelic |
complete (GRCh38-only; multi-build is RM15) |
frequencies.csv path (0.5) |
✅ FrequencyRow; coordinate cross-check → warning; provisional shape |
✅ frequencies.parquet (in artifact.digest); frequency_signature/sources/datasets/populations → manifest (out of digest) |
✅ allele_frequency = AC/AN materialized as Float64 (never stored in the CSV) |
complete (injected; enricher produces it) |
gene_metrics.csv path (0.5) |
✅ GeneMetricsRow; gene cross-check → warning; provisional shape |
✅ gene_metrics.parquet (in digest); gene_metrics_signature/genes/datasets → manifest |
— | complete (injected; offline-capable upstream) |
literature.csv path (0.5) |
✅ LiteratureRow; citation cross-check + nonexistent-PMID warning; provisional shape |
✅ literature.parquet (in digest); literature_signature/sources/coverage counters → manifest |
— | complete (injected; enricher produces it) |
gene_validity.csv path (0.6, RM24) |
✅ GeneValidityRow; gene cross-check → warning; provisional shape |
✅ gene_validity.parquet (in digest); gene_validity_signature/genes/diseases/classifications/submitters/datasets → manifest |
— | complete (injected; ClinGen + GenCC routes in the enricher) |
clin_sig_concordance.csv / clin_sig_authority_calls.csv (0.7, RM130) |
✅ two derived fact tables — the per-variant three-way ClinVar concordance verdict, and every authority's call as it stood at the time (the only derived table recording a value at the time, which is what answered-call currency compares against); orphan rows warn under derived_row_orphan; provisional shape |
✅ clin_sig_concordance.parquet / clin_sig_authority_calls.parquet (in digest, fact-hashed); _TABLE_DUPE_KEYS registered; a cell key carries the value because two rows may state two claims |
✅ clin_sig_concordance_contested (see the phrases) |
complete (written by enrich; the compiler cross-checks and never adjudicates) |
statistical_test (StudyRow, 0.7, RM140) |
✅ free text — which analysis produced this row's p_value/effect_size; in the dedup key by value |
✅ studies.parquet; reversed |
— | complete (passthrough; nothing derives or checks it) |
confidence / confidence_unit (StudyRow, 0.7, RM160) |
✅ the citing source's own review state, unconverted, and the instrument it is on; a magnitude with no unit is refused at the model, a unit alone allowed | ✅ studies.parquet; reversed through the hand-kept fieldnames and the row dict (the fourth touch point) |
— (the enricher's evidence_status_currency re-asks it) |
complete (passthrough) |
clinical_assertions.csv path (0.6, RM25) |
✅ ClinicalAssertionRow; coordinate cross-check → warning; provisional shape. Records the archive's call and review tier; it does not adjudicate against the author's clin_sig — that stays the enricher's warn-in-both-modes cross-check |
✅ clinical_assertions.parquet (in digest); clinical_assertion_signature/clin_sigs/star range/unrated_count/not_found_count → manifest |
— | complete (injected; offline-capable upstream from the ClinVar snapshot) |
| CLI (0.4.1, extended 0.5) | ✅ Typer validate/compile/signature/reverse/verify/sign; --strict, --strip-identity/--authority-key, deprecated --ensembl-cache, --resolution |
— | — | complete (compiler-only dep; tiers intact) |
| queryable p-value (0.5) | ✅ p_value_num in (0, 1]; cross-checked against the verbatim p_value string (relative, 1%) |
✅ studies.parquet; neg_log10_p derived on write, absent from the reversed CSV |
✅ -log10(p_value_num) |
complete |
callable_from (0.5, RM6) |
✅ VCF field-name pointer, namespace-qualifiable and \|-alternatable (shared AuthoredModel validator); bare colliding key → warning both modes (0.6) |
✅ weights.parquet |
— | complete (retired from the reserved namespace) |
recommendation_strength (0.5) |
✅ closed CPIC vocabulary, distinct axis from evidence_level |
✅ diplotypes.parquet |
— | complete |
| dosage sensitivity (0.5) | ✅ haploinsufficiency/triplosensitivity against VALID_DOSAGE_SENSITIVITY |
✅ gene_metrics.parquet (in digest, fact-hashed) |
— | complete (ClinGen route in the enricher) |
redistribution (0.5, settled 0.6) |
✅ tri-state; None ≠ False |
✅ sources.parquet; per-layer facet + module-wide verdict → manifest |
✅ most-restrictive-wins | complete — recorded here, enforced downstream (RM27; the ask is in SCHEMAS.md) |
| GWAS effect sizes (0.6, RM90) | ✅ effect_direction closed; effect_measure open; effect_unit free text and inside the fact hash |
✅ gwas_effects.parquet (in digest, fact-hashed) |
✅ orphan rows warn, never fail | complete (Catalog route in the enricher; fills no weight — see MODULE_LIFECYCLE § Stage 3) |
| Expression effects (0.7, RM194+RM200) | ✅ effect_direction closed (shared with GwasEffectRow — same axis); effect_measure open; effect_unit null on every AlphaGenome row and stated rather than invented; gene and gene_id both inside the fact hash |
✅ expression_effects.parquet (in digest, fact-hashed) |
✅ no orphan check, deliberately — the rows are locus-wide by construction, so gwas_effects' check would fire on nearly every row |
complete (alphagenome expression; non-commercial, gated at compile via sources.csv) |
| weighting declaration (0.6, RM92) | ✅ three free-text strings, extra="forbid" |
✅ manifest.weighting; moves neither identity half |
— | complete; dropped by reverse_module, like license/panel/authorship |
| verification attestation (0.6, RM45) | ✅ binding recomputed from the authored inputs, proof-of-work re-checked; stale ⇒ warn + drop, never fatal | ✅ manifest.verification (out of artifact.digest); nothing reaches a parquet |
— (the enricher puts the checks) | complete (verification.json; nothing in the block is trusted) |
| authoring closure (0.6, RM73) | ✅ published when the attestation holds and carries one; absent ⇒ warn in both modes; a signed closure that does not verify ⇒ drop the whole block | ✅ manifest.verification.closure; moves no digest and no signature (measured on all sixteen reference examples) |
✅ close writes it; validate never does |
mechanism complete (compiler.close_module); the refusal is 1.0 and blocked — see ROADMAP_1_0 § RM73 |
| drafting (0.5) | ✅ appended rows are validated rows; keys reuse _TABLE_DUPE_KEYS |
— (writes authored CSVs, not parquet) | ✅ append / already-present / differs report | complete (draft.append_rows, blank_template); DRAFTABLE covers the SNP core, the table kinds and sources.csv (0.5.4) |
| templating (0.5) | ✅ a stub carries TEMPLATE_PLACEHOLDER, which no mode compiles |
— (writes authored CSVs, not parquet) | ✅ created / kept-untouched plan | complete (draft.stub_template, scaffold.scaffold_module) |
| hints (0.5) | ✅ per-cell validation, bin coverage, duplicate keys, a conclusion naming another genotype (RM279) — all offline | — (writes nothing at all) | ✅ alterations + findings + options | complete (hints.inspect_rows, hints.describe_table) |
| delegated insertion (0.5) | ✅ placed rows are validated rows; shifted rows keep their cells | — (writes authored CSVs, not parquet) | ✅ DraftReport.shifted names every moved row |
complete (draft.place_rows, append_rows(group_by=…)) |
| partial rows (0.5) | ✅ stubbed columns validated by omission; the stub itself never compiles | — (writes authored CSVs, not parquet) | ✅ added / already-present / invalid; DraftReport.header_extended names the columns the header grew by — exactly the ones the written rows fill, never one only a rejected or already-present row filled, and a raw blank is not a fill (2026-09-09); settled before the existing rows are re-rendered (2026-09-02: an append into a header that predates a column used to raise a bare ValueError). Both writers rewrite the author's file atomically, and since 2026-09-12 both take before_commit — a callback run after the bytes are down and before the rename, so a drafter's licence row lands inside the commit of the table it licenses rather than after it (RM232). Optional and defaulted, so a caller that passes nothing is unaffected; it fires only when the call actually writes |
complete (draft.PartialRow, append_partial_rows) |
| genotype widening: hemizygous single allele | ✅ | ✅ (1-element list) | — | complete |
genotype widening: phased A\|G |
✅ (order kept) | ✅ phased bit → lossless round-trip |
✅ | complete |
state (legacy) |
✅ (stays required — P8) | ✅ | ✅ read alias via effective_direction; trimmed to {protective,risk,neutral} on upgraded() |
complete |
| MT / non-diploid genotype | ✅ warning on a two-allele MT or Y genotype | — | — | complete |
| direction/weight sign consistency | ✅ warning | — | — | complete |
0.4 compiler coverage (materialized)¶
| 0.4 kind (model) | Validated | Materialized (→ parquet, round-trip) | Status |
|---|---|---|---|
binning primitive MeasureBinRow + Activity/CopyNumber/RepeatAllele/Heteroplasmy rows |
✅ shared vocab, inclusive [min,max], mandatory unresolved, extra=forbid, source_field pointer + source_element rule (0.6), heteroplasmy tissue + legacy-ref guard |
✅ *.parquet via generic materializer |
materialized |
table-level validate_bins(rows) |
✅ per (key…, trait_efo_id) group |
overlap → error, gap → warning, >1 unresolved/group → error |
enforced |
duplicate-row detection (diplotype pair, pgs_id, (pharm variant, drug, genotype, category, annotation_id), allele-function allele, haplotype-defining variant, (source, layer) in the licensing table since RM107) |
✅ per-kind natural key | error (0.4 analog of duplicate-(variant, genotype)) | enforced |
PGx HaplotypeRow / AlleleFunctionRow / DiplotypeRow (+ drug/response/evidence_level) |
✅ | ✅ | materialized |
PharmGKB PharmVariantRow (single-variant drug response, evidence_level 1A…4, per-genotype) |
✅ | ✅ | materialized |
sources.csv licensing path (0.5) |
✅ SourceRow; tri-state permissions; duplicate (source, layer) → error, both modes (RM107); orphan/undeclared + declared-licence warnings (never escalate) |
✅ sources.parquet (in digest); source_signature/licences/attributions/per-layer facets/derived commercial_use → manifest |
✅ refuses (both modes) when annotation-layer terms forbid sale and no declaration is recorded |
VariantRow general axes: requires_callable / acmg_sf / actionability |
✅ (actionability vs ACTIONABILITY_SEED; acmg_sf vs the ACMG SF list in the enricher, 0.5) |
✅ into weights.parquet (tri-state bool round-trip) |
materialized |
RM70 callability on the PGx locus tables: HaplotypeRow.requires_callable / PharmVariantRow.requires_callable (0.7) |
✅ optional tri-state bool, no vocabulary and no validator — the same column VariantRow carries, on the two PGx tables that name a locus. DiplotypeRow refuses it via extra=forbid; callable_from does not travel |
✅ generic table materializer, no compiler change — _polars_type maps it to a nullable pl.Boolean and _scalar_cell reverses None/False as ""/"false" |
materialized — outside every _KEY_FIELDS; unset it is dropped by exclude_none, so no published module's content_signature moves |
RM132 the third citation site: PharmVariantRow.pmid (0.7) |
✅ optional, free-form, validated by spec.validate_pmid_cell — the one grammar every citation pointer routes through, so the PMCID refusal and the [PMID: N] spelling come with it. Read by _cross_check_literature and by the enricher's literature pass, through the derived _CITING_TABLE_KINDS rather than a third hand-kept list |
✅ generic table materializer and generic reverse writer, no compiler change — both derive their column lists from the model | materialized — outside _KEY_FIELDS; unset it is dropped by exclude_none, so no published module's content_signature moves |
RM29a call-confidence cofactor: quality_from + min_quality (0.5) |
✅ shared pointer grammar (source_field/callable_from/quality_from, one validator, namespace-qualifiable since 0.6); finite floor; both-or-neither model rule |
✅ weights.parquet (Utf8 + Float64); absent floor is null, never 0.0 |
materialized |
RM29b clinical cofactor: DiplotypeRow.clinical_context (0.5) |
✅ whitespace-stripped, open (no vocabulary — guideline bodies scope differently) | ✅ generic table materializer, no compiler change | in _TABLE_DUPE_KEYS — disagreeing CPIC contexts coexist as distinct rows |
PGS PgsRow (declared interface; ancestry-validity fields) |
✅ PGS<digits>, ancestry/tier vocab, match_rate_floor∈[0,1] |
✅ | materialized |
reserved namespace (reference_db / callable_element / quality_element) |
✅ specific diagnosis via reject_reserved on top of extra=forbid |
— | reserved |
authoring reference + palette (reference.authoring_reference()/json_schemas()) |
✅ generated from live models (drift-proof) | n/a | shipped (RM8/RM9) |
frozen variant_key identity (base.derive_variant_key) |
✅ stamped once, never re-keyed by resolution (P7); excluded from authoring_reference() |
✅ weights.parquet (compiler-managed) |
shipped |
| rsid↔coord resolution: one-to-many expansion, deterministic order, inject-only consistency check | ✅ ORDER BY; disagreement → warning; non-GRCh38 skipped |
✅ N coord-keyed rows per one-to-many rsid; idempotent | shipped (the DuckDB engine now lives in just-dna-enricher; GRCh38-only; multi-build RM15) |
expansion marker: VariantRow.locus_index + locus_count (0.6, RM87) |
✅ stamped at the expansion loop, authored cells overwritten by _freeze_identity; exclude=True, so no content_signature moves |
✅ weights.parquet (UInt32 ×2, hand-listed in _build_weights); reverse prefers the stored index over its recompute and never re-emits either column into variants.csv |
shipped — locus_count > 1 is the row-level predicate; the positional pass's hard-coded 0 is honest only while those tables never expand (a line on RM65) |
authored overlay overrides.csv (0.7, RM124) |
✅ OverrideRow (AuthoredModel); table and operation are closed vocabularies; reason required; a field outside the named table's columns, or naming its own subject/member column, is refused; a wildcard member is refused for insert and suppress; duplicate (table, subject, member, field) and a key group carrying two operations are errors — all of it in both validate_spec and compile_module |
✅ overrides.parquet, written only when the module carries an overlay — its absence, not its slot, is why no published digest moves (artifact_digest name-sorts); applied to the nine covered derived tables (OVERRIDABLE_TABLES; clin_sig_concordance.csv and expression_effects.csv joined in 0.7) before any check reads a row, so the fact signatures and resolution_signature are post-overlay |
shipped — the derived files are pure build products, reverse emits post-overlay tables plus the overlay, and no operation reports its own no-op |
| VCF pointer namespace + cardinality (0.6, RM53/RM54/RM61) | ✅ INFO//FORMAT/ qualifier and the spec's key charset accepted (widening only); _check_vcf_pointers warns in both modes on a bare colliding key and on a spec-multi-valued target with no element rule, aggregated by reason |
✅ source_element → the binning parquets via the generic materializer; round-trips through reverse unchanged |
shipped (source_element on MeasureBinRow; callable_element/quality_element reserved, not built) |
Upgrade derivation (state/booleans → 0.3 axes)¶
state and the ClinVar booleans stay required/authoritative for 0.2 backward-compat (P8). The new
axes are optional, and just_dna_format.derive supplies fallbacks:
- Read-time (non-mutating):
VariantRow.effective_direction/effective_stat_significance/effective_clin_sig/effective_pathogenic/effective_benignreturn the set column, else the derivation — so a legacy 0.1/0.2 row exposes all three axes with no re-publish. - Materializing:
VariantRow.upgraded()fills those axes and trimsstateto{protective, risk, neutral}(kept as a derived mirror ofdirection).needs_upgradeis the signal the marketplacerevalidate/needs_upgradeflow consumes. Both idempotent (P7).
The parquet column is the authored value, and the derivation is not in the artifact¶
The direction column in weights.parquet is a materialized passthrough: whatever the author
wrote, or empty. The compiler never fills it from state, and should not — state='significant'
carries no direction at all, so the derivation refines one from the weight sign, which is a sound
fallback for a reader and a fabricated fact in a published table. Every module authored against 0.2
therefore ships an empty direction, correctly.
The consequence to plan for, if you read the artifact rather than the models: the fallback lives in
Python and does not travel with the parquet. A consumer querying weights.parquet with SQL or
polars sees the empty column and nothing else, so a migration from state to direction reads every
legacy module as directionless. Apply the derivation yourself —
just_dna_format.derive.direction_from_state(state, weight) is a pure leaf function published for
exactly this (it imports nothing from spec, so the marketplace revalidate flow already uses it
that way) — or go through VariantRow.effective_direction, which returns the authored value when
there is one and the derivation when there is not.
Whether an artifact should ever carry the derived axes is open, and it is a design question rather than a patch: the objection is that filling a blank asserts what no curator wrote, not that the bytes move.
Intentionally unimplemented — and why¶
- New computed manifest stats.
Statscarries the 0.2 counts only; no new distributions (bydirection/clin_sig) — no consumer needs them yet. effect_allelestrand reconciliation. Since 0.5 the compiler does check thateffect_allele(and every genotype allele) is one of the alleles the locus actually has — so a strand-flippedA/Gat aC>Tlocus is now caught, because neither allele is present. What is still not done is reconciling it: nothing complements the allele and rewrites it, and the+strand /genome_buildassumption remains documentation rather than an enforced computation. Reporting a flip is a redundancy check; silently flipping it back would be repairing authored data.- Single build — GRCh38-bound.
genome_buildis recorded but only GRCh38 is honored: coordinates are GRCh38, the resolution table'sgenome_buildis checked against the module's, andartifact.digestis GRCh38-relative. A GRCh37/T2T module compiles but is not re-resolved for that build. Legacy-from-implementation, not a principle — build-aware identity is RM15. A no-coord rsid mapping to several loci is expanded to one row per locus (data-agnostic), shipping GRCh38-now. reverse_modulereconstructs the compilable core, not manifest-only metadata. It reads the parquets for everything materialized into them, andmanifest.jsonfor the one authored value that is digest-relevant and lives nowhere else:genome_build. That single read replaced a "parquet only, nevermanifest.json" rule which sounded principled and was how the build came to be hardcoded — the rule's real content is nothing in the digest may be invented, and for a non-GRCh38 module the invented build changed the identity key, hence the digest. Absence is handled rather than assumed: no manifest means the format's default, and an explicitgenome_build=always wins.authorship/panel/provenance/logogenuinely are not restored and genuinely cannot move a parquet byte. What is round-trip-critical — every authored value, including a poly-effect variant's per-effectgene/phenotype/category— is restored.- Gene-panel materialization, and now the
panel:block itself (RM4). Compiling aGenePanelSpecintoweights.parquetis not deferred any more, it is dropped. The compiler must not create rows no curator wrote — the same objection that bars fillingdirectionfromstate, and it does not depend on the digest. Expansion at compile would also make a module's content depend on an external file, and leavereversechoosing between re-emitting the declaration (rows lost) and the rows (declaration lost); neither is a fixed point (P7). The want is served instead by enricher draft-scaffolding, which already ships:draft-panelwrites the rows, and the author's no-op over the drafted subset is still an authorial act. The rows are authored bytes before the compiler ever sees them.
With that decided, panel: had no reader left — its last one was the enricher's ClinVar
clin_sig cross-check, which now reads the drafted-from release out of the licence row's dataset
column, written by the drafting pass. So it is deprecated in 0.6 and removed at 1.0, with
validate_spec emitting the warning (and compile_module carrying it into
manifest.compilation.warnings, once — compile seeds its warnings from validate's). The block
still loads, still reaches manifest.panel, and still changes nothing else, which is what makes the
deprecation warn-only and the cadence legal.
Deleting it moves no identity, measured rather than argued: reference_examples/apoe_epsilon
with a panel: block appended compiles to the same artifact.digest and the same
content_signature as without it. Auto-removing it on reverse was considered and refused for the
opposite reason — reverse writes module_spec.yaml, so dropping the block there would change that
file's bytes and break the round-trip fixed point for any module carrying it. A warning the author
acts on is the route.
The warning has two branches since S69, and it stopped claiming nothing else is lost.
Both halves were wrong in a way gating alone would not have fixed. GenePanelSpec carries five
fields and SourceRow.dataset is one release label: it cannot hold genes — the denominator,
and the only thing separating this gene is not in the panel from it is in the panel and had
nothing to report (reported as 425 declared genes against gene_count: 298) — it cannot hold
significance, the predicate that makes a panel module's row set reproducible, and it is a name
rather than a digest so it cannot hold reference_sha256. All three now say so in the message.
And the replacement is legitimately absent on a module drafted before the drafter filled
dataset: merge_sources_file is never-clobber, so re-running does not backfill it, and there is
no path from such a module to the state the old sentence assumed. So the check moved behind the
licence rows — a deprecation is legal in a minor only where its audience can act on it (P3), and
whether they can is a fact about a value the pass had not read when the warning fired. With no
filled clinvar/annotation dataset it now says do not delete the block yet, names what to
fill first, and warns that a re-draft will not do it.
Consequences worth knowing¶
weights.parquet/studies.parquetcarry the 0.3 columns + aphasedbit, so a re-compile under this compiler changesartifact.digestfor every module; reproducibility is pinned bycompiler_version, and published versions keep their digest until re-published. A moving digest is therefore not itself a version gate: a new optional column or table is additive and minor-legal, while removal, promotion to required, retyping, or changing what an identity key means is major-only — see ROADMAP § 0.6.- Round-trip is lossless and idempotent (P7):
reverse_module→ recompile preserves every column including phase, and the same spec compiles twice to the same digest. In 0.5 the round-trip is additionally offline — reverse emitsresolution.csv, so recompile needs no reference and no network (regression-tested: DuckDB compile → reverse → no-cache recompile → identical digest). - The
ValidationResult.infochannel carries non-reservedflagsnotes via stdlib logging — the format packages do not depend on Eliot.
Tests: compiler/tests/test_v03*.py (validator, genotype widening, warnings/INFO, materialization,
round-trip/idempotency); test_v04_compile.py (the nine table kinds); test_resolution_table.py (the
0.5 resolution-table path, digest parity, offline round-trip, strict/best-effort, the deprecation).