just-dna-compiler — the transform tier¶
The package reference for just-dna-compiler: the reference compiler that turns a validated spec
directory into a multi-parquet artifact + manifest.json, and reverses it back. Since 0.5 it is
pure-Python and duckdb-free — its runtime deps are just-dna-format, polars, pyyaml, typer.
It never fetches (Principle 2): resolution is consumed from an injected, source-independent
resolution.csv; the pre-0.5 DuckDB reference path moved to just-dna-enricher and is
reachable only through a deprecated, guarded shim (removed at 1.0).
The compiler adopts the schema with a C++-standard-style feature-coverage stance — not all-or-nothing conformance but a per-feature table (below). As of 0.4 the validator is complete, the upgrade derivation ships, the artifact round-trips losslessly including phase, and all nine 0.4 table kinds materialize with enforced table-level coherence.
Companion docs: SCHEMAS.md (the models it compiles), ENRICHER.md (what produces
resolution.csv), CONSTITUTION.md (the invariants).
Read beside this: the 2026-09-11 code-first re-derivation¶
A second reading of this tier, written from the code alone on 2026-09-11, is in audit/COMPILER_FROM_CODE.md — the pipeline in execution order, per-parquet column lists, and a validation table with validate/compile/severity columns. It is the instrument that found RM93 and RM94 in 0.6.1 and RM207 in 0.7 — the last of which is why the parity this document asserts is a property of the binaries rather than of its own prose. Evidence, not contract: this document is the maintained one. The method is BLIND_REDERIVATION.md.
Public API¶
Import from just_dna_compiler.compiler.
That path was one compiler.py until RM260 made it a package. Its __init__.py is a re-export shell
over every name the single file bound, so an import or attribute written against the old path still
reaches the same object; the shell is removed at 1.0. The code lives in the submodules beside it:
tables (the table registry and the row → polars/CSV helpers), load (reading a spec directory,
spec_tables, content_signature), the check modules variant_checks, positional, binning_checks,
vcf_checks, allele_checks, vrs_checks, table_checks and fact_checks, validate
(validate_spec), pipeline (compile_module, close_module), manifest (manifest assembly, the
sources/licence gate and the verification blocks), parquets (the SNP-core and frequency parquets) and
reverse (reverse_module). A test that monkeypatches a private helper patches the submodule that
looks the name up, not the package.
validate_spec(spec_dir, authority_keys=None, *, strict=False) -> ValidationResult— validate a spec dir without producing output; strips inject-only authority keys pre-validation (dropped keys →.info), runsvalidate_binsand the duplicate/identity checks, populates.stats.strictgrades the mode-ladder findings at the severity astrictcompile would, so a caller can report what strict would refuse without building anything — note it cannot see the unresolved-position gate, which lives incompile_moduleand needs a resolution table.content_signature(spec_dir) -> str— the stable, name-/Ensembl-independent content identity over the raw authored data CSVs (no compile, no resolution); raisesValueErrorif a present data CSV is invalid. See SCHEMAS.md § identity & integrity. Which CSVs, exactly:variants.csv,studies.csv, and any present table kind — the authored roster. The licensing table (sources.csv/licensing.csv) is outside it: it is hashed byintegrity.source_signatureinstead, so neither its presence nor an edit to anoticecell in it moves this digest. That is correct — the licence layer carries its own identity — and it is stated because it is the one authored, hand-editable table a licence audit sends an author looking for (S53).load_spec(path, *, authority_keys=None) -> ModuleSpecConfig— the public route to a parsedmodule_spec.yaml, raisingSpecError(aValueError) on anything wrong. Sibling ofjust_dna_format.read_manifest/read_verification: one loader per file a module carries, same raise-don't-return contract. It exists because the model was public from the start and the only thing producing one was_load_yaml, so a consumer wantingweighting:,authorship:orlicensehand-parsed the file, lost the authority-key handling and every diagnosis, and carried PyYAML purely to work around a private symbol — the enricher itself was doing the same thing across the tier boundary (S74)._load_yamlkeeps its(config, errors, dropped)tuple forvalidate_spec, which accumulates diagnoses from a dozen sources; that difference is why both exist. It is in this tier rather than the format one because parsing YAML needspyyaml, and the format tier ispydantic+cryptographyby charter. It does not folddefaults:— that is per row and isspec_tablesbelow; the pairing isload_specfor the yaml's blocks,spec_tablesfor the rows.spec_tables(spec_dir) -> tuple[dict[str, list[BaseModel]], str]— the parsed, defaults-folded rowscontent_signaturehashes, plus the declared build.content_signatureis exactly this plus the hash, so anything finer than a whole-module digest — per-table, per-row, what moved between two versions — builds on these rows instead of restating the roster and the fold (S53). SameValueError-on-invalid-CSV contract. The fold is why this returns the finished mapping rather than exporting the pieces: a caller hashingload_csv_rowsoutput directly gets a different digest for the same module, because a value written once underdefaults:and the same value written on every row are one content (RM37) and only this path folds them — measured onhfe_hemochromatosis, the raw build reports twelve changed rows where there are none.compile_module(spec_dir, output_dir, compression="zstd", resolve_with_ensembl=True, ensembl_cache=None, compiled_by=None, ensembl_reference=None, log_files=None, provenance_file=None, logo_file=None, readme_file=None, authority_keys=None, strict=False, ba1_threshold=0.05) -> CompilationResult— compile to parquet +manifest.json.ensembl_cacheis deprecated (removed at 1.0) — see the precedence block.resolve_with_ensemblis not deprecated and is misnamed: since 0.5 it is the master switch for consumingresolution.csvat all, so turning it off ignores an injected table and has nothing to do with Ensembl.ba1_thresholdis the ACMG BA1 allele-frequency cutoff — a parameter rather than a constant, because the right value is disease-specific.-
reverse_module(parquet_dir, output_dir, module_name=None, title=None, description=None, report_title=None, icon=None, color=None, version=None, write_resolution=True, genome_build=None) -> Path— reverse a compiled artifact back to the authored DSL. Every metadata argument leftNoneis read from the artifact's ownmanifest.json, and falls back to the bare-directory default only when there is no manifest (RM241, § Reverse). -
load_csv_rows(path, row_model, file_label, genome_build=DEFAULT_GENOME_BUILD) -> (rows, errors, warnings)— the authored-CSV loader, public since 0.5.1 (RM41). It was_load_csv_rows, and it was public in practice:just-dna-enricherconsumes it across a package boundary in a dozen places, and a consumer wiring the pipeline server-side had the choice of a private symbol or a re-implementation. Re-implementing is a trap, not a chore — it is notcsv.DictReaderplusModel(**row), because an empty cell becomesNonewith the key kept (so a defaulted-but-not-Optionalfield likeMeasureBinRow.measure_kindreceivesNonerather than its default and fails on type) andgenome_buildis told to each row rather than read from it._load_csv_rowsremains as an alias, so nothing that imported it breaks. -
load_spec_variants(spec_dir) -> (variants, errors, warnings)— a spec directory'svariants.csv, loaded with the build the module declares and re-stamped for it. Three steps, not one: readgenome_buildout ofmodule_spec.yaml, inject it into every row, then_restamp_for_build— becauseVariantRow._freeze_identityruns at construction, where the yaml is not in scope, so a loader that skips either step mints GRCh38 identities for a GRCh37 module. Missing or unreadable yaml falls back toDEFAULT_GENOME_BUILD, matching what compiling that directory would assume; this is a read-only check helper, unlike the enrichment path, which refuses rather than choose a build for a module whose declaration cannot be read (it writes facts back). Added for the two enricher checks that take rows rather than aspec_dir—verify_acmg_sfandcheck_identifiers, which now accept both. -
load_citing_rows(spec_dir) -> dict[str, list]andtable_citations(rows_by_csv) -> list[str]— every table kind beside a spec whose rows cite (a model declaring apmid: the binning kinds since RM47,pharm_variants.csvsince RM132), and the digit-only PMIDs they name, de-duplicated in first-occurrence order. Public for the RM40/RM41 reason: the enricher's literature pass has to check these pointers alongsidestudies.csv, and its alternatives were a private import or a second roster that goes stale the next time a model declares the column. The set is derived from the models, so a new citing kind is read by both tiers with no edit to either. Row errors raise — a pass reading citations out of a table it could not parse would silently under-report; a caller wanting the per-row diagnosis hasvalidate_spec.load_binning_rows/binning_citationsare the narrower pair, unchanged: they answer over the binning kinds only, because a caller asking for those is asking about thresholds rather than about the citations a module makes.
models.py: ValidationResult (valid, errors, warnings, info, stats) — the .stats key
contract is variant_count/unique_rsids/gene_count/genes/categories/study_count/
clinvar_count/pathogenic_count/benign_count/clin_sig_counts/clin_sig_variant_counts/module_name. CompilationResult (success,
output_dir, errors, warnings, stats, manifest — the emitted ModuleManifest, None on failure).
What the compiler can and cannot validate¶
The compiler sits to the enricher roughly as an assembler-plus-linker sits to a source tree: it consumes an authored schema plus injected resolution facts, and its guarantee is that the result is well-formed and self-consistent, not that it is true. A C++ compiler type-checks a program and still ships your off-by-one; a linker proves every symbol resolves and says nothing about what the functions do. The same boundary applies here, and it is worth stating plainly rather than leaving a reader to infer that "it compiled" means "it is right".
There is also a trust boundary. resolution.csv and the derived-fact sidecars — frequencies.csv,
gene_metrics.csv, literature.csv, gene_validity.csv, clinical_assertions.csv — are consumed as
fact. The compiler can re-derive the parts that are self-verifying and cross-examine the
parts that are redundant, but everything sourced — which coordinate, which rsID, which allele
frequency — is taken on trust from whoever produced it. That trust is the price of Principle 2: a tier
that never fetches cannot independently confirm a fetched fact.
Three things it can check, in increasing strength¶
1. Formal conformance — complete within its domain, like type-checking. Columns, types, closed
vocabularies (extra="forbid" plus the reserved-namespace diagnosis), identifier grammars (rsid, DOI,
PMID, CURIE, ga4gh:VA., CA\d+), value bounds, finite floats (no NaN), required-ness, at most one
unresolved bin sentinel per bin-group key, bin overlap/gap, duplicate natural keys, duplicate
(variant_key, genotype). If a module violates one of these it is malformed, full stop.
Note the one-sidedness of the sentinel rule, because it reads like a completeness check and is not one:
_validate_table_kind counts sentinels per group and refuses a second one, and nothing on the
compile path refuses zero. A binning table carrying no sentinel at all compiles green under
--strict — measured. The missing-sentinel finding lives on the authoring surface only
(hints._check_bins warns no unresolved sentinel row), and it is table-level, so a table whose rows
fragment into several bin groups can satisfy it while leaving most groups with no sentinel.
2. Validate-by-redundancy — where two independently-authored things must agree, disagreement is detectable without any reference. This is where most real authoring bugs are caught:
| Check | Redundancy exploited | Severity |
|---|---|---|
| rsid ↔ coordinate | the pair co-identifies one variant | warning / error in strict |
| inconsistent position for one key | one key, one place | error |
inconsistent ref for one key |
the reference base is a single fact | error |
genotype alleles ⊆ {ref} ∪ alts |
a genotype names alleles the locus has | warning / error in strict |
effect_allele ∈ {ref} ∪ alts |
the effect allele is one of them too | warning / error in strict |
clin_sig pathogenic + high AF |
ACMG BA1: common alleles are not pathogenic | warning |
| a citation PubMed has no record of | the enricher already wrote the verdict down | warning |
allele_count ≤ allele_number |
a count cannot exceed its denominator | error |
2 × homozygote_count ≤ allele_count |
each homozygote contributes two alleles | error |
faf95 ≤ the group's own AF |
a CI lower bound sits below its point estimate | warning |
oe_lof_lower ≤ oe_lof ≤ loeuf |
an estimate lies inside its own interval | warning |
obs_lof / exp_lof == oe_lof |
the same quantity, stored three ways | warning |
| direction ↔ weight sign | two encodings of one claim | warning |
p_value string ↔ the mantissa/exponent pair |
two encodings of one number | warning / error in strict |
| MT/Y two-allele genotype | ploidy contradicts the contig | warning |
| study / frequency / gene-metrics orphans | the sidecar describes something the module lacks | warning |
| literature orphans | a citation no study and no citing table row makes — reported, and left out of the artifact (the row stays in literature.csv) |
warning |
sources.csv orphans / undeclared sources |
every source a fact table cites has terms recorded, and vice versa | warning |
| star allele used but not defined | allele_function/diplotypes name it; haplotypes defines it — matched on (gene, allele), a haplotypes row with no gene defining its name for every gene, genes compared case-blind, and an HLA gene that haplotypes defines nothing for exempt, since its alleles are typed (RM351) |
warning |
| phase-ambiguous diplotypes (0.5) | two different haplotype pairs of one gene whose unphased genotype is identical while their conclusions differ; each haplotype's defining variants are that gene's own (RM351) | warning |
"warning / error in strict" names one channel per mode, never both (RM359). Under strict such a
finding is in errors and is not also in warnings: compile_module runs its pre-flight in its own
mode, and a resolution finding whose refusal is a longer sentence publishes the refusal instead of
the warning. Before RM359 a strict compile carried both, so a consumer counting findings counted one
twice. compiler/tests/test_one_channel_under_strict.py asserts the rule over every mode-ladder code
on a real compile.
Four of these deserve their reasoning rather than just their row.
The phase-ambiguity check is RM28's cis/trans motivation, closed by computation rather than by a
grammar. Compound heterozygosity is the case that most justified a predicate language, and
reference_examples/hfe_compound_het/ shows it needs none: a diplotype is already a statement about
two homologs, so HFE C282Y/H63D (in trans, at-risk) and C282Y-H63D/wt (in cis, a carrier) are
simply two rows. What no table can say is that a consumer without phase cannot tell them apart —
they present the identical unphased genotype (rs1800562 G/A, rs1799945 C/G) and disagree about what it
means, and nearly all consumer data is unphased.
That is derivable from tables the compiler already holds, which is what makes it a check rather than a
requires_phase column: a column would restate what the data determines and go stale the moment a
haplotype is edited. The signature is, per variant, the sorted pair of alleles the two haplotypes
contribute — sorted because that is precisely what losing phase does. An unmentioned variant reads as
the implied reference, and an allele explicitly equal to the row's own ref normalizes to the same
sentinel; neither needs the reference sequence, only that "unmentioned" means one thing. So it runs
unchanged on a CPIC-drafted table where haplotypes are sparse and ref is absent — and correctly finds
nothing there.
Two boundaries. It compares distinct haplotype pairs, never distinct rows: the dedup key admits a
row per drug and per clinical_context for one pair, and grouping on rows reported 595 ambiguities in
a CYP2C19 module that has none. And it is closed-world — it compares the rows a module states, not
the ones it omits, which is why APOE stays quiet despite ε2/ε4 vs ε1/ε3 being the textbook collision:
that module carries no ε1. The neighbouring "used but not defined" warning covers that side.
The allele-membership checks never escalate to an unconditional error, and the tempting version is
wrong. An authored ref+alts contradicting the row's own genotype looks decidable here — but
ref/alts in variants.csv are not necessarily human-authored. reverse_module writes them, and a
one-to-many rsid reverses into N rows that each carry their own locus's alleles beside the one
genotype the author wrote; exactly one of those rows can match. An unconditional error would mean any
module with a one-to-many rsid compiles once and never again — Principle 7's fixed point, broken by a
lint. (Not hypothetical: three variants in reference_examples/pathogenic_clinvar/ have this shape.)
For the resolved case there is a second reason to stay at a warning: alts came from a source, and
ClinVar carries only its submitted alleles while Ensembl carries every allele dbSNP knows, so a short
alt list is a gap in the source at least as often as a defect in the module. Since 0.8 the resolved sentence names
a third reading too (RM374): the row was resolved against an earlier snapshot release than the module is
enriched against now, because an enrich never re-asks a subject it has a row for. It names the release the
row records (ResolutionRow.source_release) or says release not recorded, and gives enrich --rederive
as the remedy. Which release is older is not decidable offline, so it is a reading, never a verdict, and it
is given only where an external source answered the row (authority set): a hand-filled row came from no
snapshot, and a reversed row keeps the authority and release of the row it came from (RM380). A row whose source is ensembl-rest gets a fourth reading beside it (RM447): it was answered by a live link that can
serve an older dbSNP build than the snapshot (REST at Ensembl 116 serves dbSNP 156, the snapshot of the same release
157), and the remedy names a re-derivation where the snapshot is provisioned, since a plain --rederive without it
asks REST again. The comparison is also made
against the union of every locus a key resolves to, never per-expanded-row, for the same reason.
A mirrored insertion and deletion are settled by the witness, not by the strings (RM449, 0.8.9). At one
position, C>CT and CT>C both hold the strings C and CT, so an rsID row's C/CT is hosted by both,
yet it means an inserted T on one and a deleted T on the other. The fill reads ResolutionRow.placement_verdict
there: the spellings a second source placed as same_event are filled, and the others are not. If both are
witnessed, the rsID names both events, so both stay, and a warning says so once
per rsID in both modes. If none is (unplaced on every spelling, RM453, 0.8.10), that is no evidence for
either: both stay, under a warning saying the source asked "placed neither as the same event". 0.8.9 left that
row unpositioned instead. If no second source was asked at all (placement_verdict empty at the site), both
stay as before 0.8.9, and the warning says so. A site whose only recorded verdict is undecided (the witness
could not judge, as a run with no reference sequence writes) reads the same way: both stay, under a sentence
saying the source asked "could not decide whether either is the same event". Whatever the witness, an expansion member re-keyed onto an
authored coordinate row's exact (variant_key, genotype) is not filled: the coordinate row picks its event, and
the warning names the key. Each outcome is a warning and none is escalated in 0.8. In 0.8.9 each sentence
borrows the code of what the row became, since a code added in a patch is refused by an older format reader:
rsid_no_hosting_locus where no position is written, rsid_expanded_to_multiple_loci where one or more loci are
filled (so a site narrowed to one locus counts as an expansion). 0.9 adds rsid_genotype_reads_as_two_events.
A positional row was already left unplaced where two loci host it. It is a corrected derivation:
an artifact compiled before 0.8.9 may hold the claim on both spellings. S158 is the report.
"Exactly one of those rows can match" is a statement about matching, and a reader does more than
match (S33). The sentence above is right and the expansion is staying, but it was written in the
authoring/validation frame, where an unmatchable row is inert. It is not inert to a consumer reading
the artifact: TA/TA beside ref=TA is a well-formed reference homozygote carrying the module's
conclusion, and a reporting consumer classified 2,579 of them as pathogenic genotypes a subject
carried. Two things came out of that, neither of them a filter. The read-side contract is now stated
where a consumer will meet it — SCHEMAS § the consumer join contract
— and manifest.compilation.expanded_keys/expanded_rows publish whether an artifact contains such
rows at all.
And since RM87 an expanded row says so about itself. weights.parquet carries locus_index and
locus_count, stamped at the expansion loop, which is the only site that knows a row is a member of
anything — after it the row is ordinary, with a real coordinate, a real reference allele and the
module's own conclusion. locus_count > 1 is the predicate a consumer applies while holding a single
row; locus_index is the 0-based ordinal that lines that row up with its resolution.csv row. Four
things about the pair:
- The ordinal counts within
usable, i.e. after_hostable_locidropped any locus that positively contradicts the genotype (asked throughsite_verdictssince RM381: a genotype authored in a site's padded frame,GTT/GTTT, is read in it, so only the piece it names is usable, and a compound het no single piece holds is usable on the piece each allele names; anything else ishosting_verdictper locus) — not within the injected table's ownlocus_index, which_sorted_locionly orders by. Understricta dropped locus is a refusal, so the two coincide there. locus_countdefaults to1, not0. A row that was never expanded genuinely resolves to one locus. A zero default would make the predicate readlocus_count > 1 or locus_count == 0, which is the same under-determinationlocus_indexalone has and which the columns exist to remove.- Reverse prefers the stored column and keeps its encounter-order recompute as the fallback. An
artifact compiled before 0.6 has no column and Principle 3 requires it to keep reversing, so
_write_resolution_csvreadslocus_indexwhere it is present and counts where it is not. A test reverses each expanding reference example twice — once with the column, once with it stripped — and asserts the tworesolution.csvfiles are byte-identical; that comparison is what pins the sort dependency the recompute silently relies on. The counter itself is maintained either way, because the positional pass below uses membership in it to enforce weights-first. - Neither column moves a
content_signature, and neither is re-emitted intovariants.csv. They areexclude=Truecompiler-managed columns recording what resolution did.
The positional tables' pass still writes a hard-coded 0, and that is honest only while those
tables never expand — true today, since RM43's fill is one locus per row. It stops being true if RM65
ever puts coordinates on the repeat and copy-number tables, and the RM65 entry carries that line.
The expansion warning is one sentence per rsID, over every authored row at it. It used to be
emitted inside the per-row loop, so a site with two authored genotypes published the identical
sentence twice and each copy said "expanded to 2 rows" of an artifact that had gained four. The union
of loci is what "maps to N loci" counts and the sum over authored rows is what "expanded to M rows"
counts — the two come apart exactly when an author writes more than one genotype at a key, since
_hostable_loci judges hostability per genotype and two rows at one key can legitimately reach
different loci. The deprecated ensembl_cache path keeps the old per-row shape deliberately; it is
removed at 1.0 and its modules report the counts as None.
BA1 is a warning in both modes and its threshold is a parameter (compile_module(ba1_threshold=…),
default 5%). The right cutoff is disease-specific — sickle-cell's rs334 sits at a filtering AF of
~0.048 in African-ancestry groups, just under the line, and a common recessive carrier allele
legitimately sits above it. Failing a compile over that would be the format arbitrating a clinical
judgement.
The sources.csv orphan half exempts the layers with no source column to join, and there are two
of them. "No table used it" is decided by reading the generated fact tables' source columns
(resolution.csv contributes its authority, not its link — RM33). The annotation layer is
variants.csv/diplotypes.csv/…, which carry none by design, so an annotation-layer row can never be
corroborated and used to be reported as stale on every drafted module — the exact row the licence gate
keys on. literature joined that exemption in 0.5.4 (S23) whenever the module carries studies.csv
rows, by the identical argument: studies.csv is the hand-curated literature table and has no source
column either, so a module citing a PMID through it can only be corroborated by the enricher-written
literature.csv, and a module with none has nothing to join. The old behaviour inverted the incentive
where it matters most — vocab.MISPLACED_COLUMN_REASONS['source'] tells an author to declare a
hand-read source by adding a row here, doing so earned a warning that the row was unused, and deleting
it (shipping with the provenance unrecorded) was silent. Compliance warned, omission quiet. It stays
narrow: frequency still warns, because frequencies.csv is machine-written with a source column,
so a frequency declaration in a module carrying no frequencies really is stale. The undeclared half —
a source a fact table cites with no row — is unaffected and warns in every case, and neither half ever
escalates: over-declaring terms is the cheap error, and an author talked out of recording theirs is not.
A coordinate that cannot exist is refused in both modes (RM48, 0.6). _check_build_coordinates
asks one arithmetic question of every row carrying a chrom and a start — could this position exist
on this contig in the build it is recorded under? — and two shapes answer no provably, with no
sequence, no network and no provisioned asset:
- a position past the end of its contig. GRCh38's chromosome 1 ends at 248,956,422 and GRCh37's
runs 294 kb further, so an un-lifted hg19 coordinate in that tail names a base that does not exist.
When the position is inside another build's contig of the same name, the message says which, and
points at
just-dna-enricher hint recover— that is the whole diagnosis, and it costs a dict lookup. - a contig only one build names. The 25 primary contigs are spelled identically in both builds, so
this is entirely about unplaced scaffolds:
GL000209.1is GRCh37's andKI270728.1is GRCh38's.variants.csvrefuses either at the model (itschromvocabulary is 1-22/X/Y/MT and always was — what 0.6 added there is that the rejection names the build); this reachesstudies.csv, the PGx tables,heteroplasmy.csvand the injectedresolution.csv, none of which validate the contig.
It is an error in both modes — the inconsistent-reference-allele class, not a mode ladder. strict
means reproducible artifact, and these rows are not unreproducible, they are false. Nothing
downstream catches them either: a VRS id minted at an impossible position is a correct digest of the
wrong input, which is how a 3,038-row off-by-one once passed every gate including --strict.
Three things it deliberately does not do. It says nothing about a low position — VCF writes POS 0
for a telomeric variant, so only the upper bound is consulted. It withholds on every contig the
tables do not settle: a shared scaffold (GL000194.1), an unversioned accession (GL000205, where the
suffix is what separates the builds), a patch or an alt locus. And it judges each resolution.csv row
against that row's own genome_build column rather than the module's, because that column exists to
say which frame the numbers are in. Findings are grouped by reason — a whole panel authored on hg19 is
one line, not one line per variant. The tables live in just_dna_format.vrs
(PRIMARY_CONTIG_LENGTHS, CONTIGS_ONLY_IN) beside PAR_GRCh38, for the same reason: assembly
constants the compiler needs offline. They carry no refget accessions — a second build's identity
is RM15, and refget_accession still raises for GRCh37.
3. Content-addressed self-verification — the strongest class, because the stored value is a pure
function of other stored values, so a disagreement is provable corruption rather than a difference of
opinion. artifact.digest, content_signature, the three fact-signatures, the Ed25519 signature —
and, since 0.5, vrs_id. Moving allele identity into this class is what the VRS work bought: a
ga4gh:VA.… used to be an opaque cross-reference that had to be believed, and is now a checksum the
compiler recomputes from the coordinate with no dependency and no network.
The inescapable blind spots¶
These are not gaps this tier can close. Each follows from what the compiler is — a transform over injected data — and pretending otherwise would be worse than saying so.
That is a claim about the compiler, not about the ecosystem, and the distinction matters now that two
of the rows below have moved. The enricher holds references the compiler never will, so it can
answer questions this table calls unanswerable — an authored clin_sig against ClinVar's, a cited PMID
against PubMed, an rsID against dbSNP. What stays true is the division of labour: the compiler's own
blind spots are permanent, what it cannot validate it makes legible, and a check that needs a
reference lives one tier up. The "What the format does instead" column below records which tier now
covers what.
| Blind spot | Why it is inescapable | What the format does instead |
|---|---|---|
Is a single-sourced number right? An AC/AN, a pLI, a clin_sig — one source, no redundancy to exploit. A transcription error is indistinguishable from a correct value. |
Nothing to check it against without fetching (Principle 2). | Records dataset (which release) and source (which link) so the number is attributable, and fact-hashes it so it cannot change unnoticed. |
Is the reference base right? A wrong single-base ref mints the correct VA, so the artifact is self-consistent and wrong. |
The compiler holds no sequence. | The enricher checks it (sequences.verify_reference_alleles); the compiler catches only two rows contradicting each other. |
Is an indel's vrs_id right? Cannot be recomputed without justification against the sequence. |
Same. | Reported as unverifiable (never as verified), and carried with that said out loud — a warning in both modes, since no authored edit could clear it. |
| Is the coordinate the variant the author meant? A perfectly valid VA for the wrong locus is indistinguishable from the right one. | Requires knowing intent. | provenance.json, authorship, and the studies table make the claim auditable by a human. Narrowed in 0.6 (RM48): a coordinate that could not exist in the declared build is now refused offline, and one that reads as the old assembly is diagnosed by the enricher against the live GRCh37 service. Neither reaches intent — a well-formed GRCh38 coordinate for the wrong GRCh38 locus is still invisible here. |
Is the annotation medically correct? Whether A/T at HBB → sickle-cell carrier is true. |
Out of scope by charter — the format supplies annotation tables and never a gene–disease inference. | authorship.kind lets a consumer route scrutiny (AI vs human-certified); curator/method record who decided. |
Does the cited study support the row? pmid is grammar-checked; nobody reads the paper. |
Requires the literature. | Partly closed by the enricher (0.5). Its literature pass confirms the PMID resolves, cross-fills the DOI/PMCID, and matches provenance_quote/provenance_regex against fulltext — for the open-access subset only, with coverage reported as a fraction so an unread paper is never mistaken for a failed quote. The compiler still reads nothing; it surfaces the recorded verdict from literature.csv. |
| Is the source stale? A v2.1.1 constraint number is well-formed and current-looking. | The compiler cannot see the world move. | dataset names the release; the gene-metrics pass labels its two routes differently, the snapshot by its release and the live API by a route label that claims none (RM417), and warns on the API. Generalized to identifiers in 0.5: the enricher checks rsIDs against dbSNP (live/merged/absent), trait CURIEs against OLS4 (obsolete + replacement) and gene symbols against HGNC (approved/retired). All report; none rewrite. Extended in 0.5.4 to the relationship between two identifiers (S24) — a row's gene against the chromosome its variant sits on — because both halves can be individually valid while the pairing is fabricated. |
Is acmg_sf right? A gene-list-membership flag the compiler holds no list for. |
Same shape as clin_sig: the list is not in the module and cannot be, since a gene list inside the compiler is an un-injected reference (RM21). |
Closed by the enricher (0.5). acmg.check_acmg_sf compares the flag against ACMG SF v3.2 as NCBI publishes it. Warns in best_effort, refuses in strict — list membership is a published fact, not a clinical judgement, so unlike clin_sig it does escalate. A blank cell is a note, never a defect. |
Is the annotation medically correct? — the clinical half. Whether the module's clin_sig is the right call. |
Out of scope by charter (below), and ClinVar is not truth either. | Surfaced, never adjudicated (0.5). The enricher compares each authored clin_sig against the ClinVar snapshot's, allele-exactly, and reports opposed calls with ClinVar's review-star count. It is the one check whose severity does not escalate in strict: failing there would make the format decide a clinical dispute. |
Did the author declare every source they copied from? A copied annotation with no sources.csv row is indistinguishable from an original one. |
Provenance of a text is not a property of the text. | The enricher writes the row when it fetches (it is the only tier that knows); sources.csv + manifest.sources make the declaration legible and hashed. The compiler warns on a source a fact table cites with no row, but cannot see what was copied by hand. |
| Did the enricher get it right? The resolution table is consumed as fact. | The trust boundary itself. | source, status, resolution_mode, fully_resolved and resolution_signature make the provenance and the policy legible. |
The through-line: what the compiler cannot validate, the format makes legible. It records who produced a fact, from which release, under which policy, and hashes it so it cannot drift silently — then leaves the judgement to a consumer. That is the data-agnostic north star applied to trust.
One gap that was not inescapable — where a bin boundary came from (S19/RM47, closed in 0.6)¶
Everything above is a limit of the tier. This one was a limit of the schema, and it is kept here because the distinction is the point: a tier limit is permanent, a schema limit is a release away.
studies.csv is required iff variants.csv is present, so grounding was enforced exactly where
citations usually arrive already attached (a ClinVar-drafted variants.csv) and absent where a human
made the judgement: reference_examples/htt_repeat_expansion compiled green under --strict asserting
where Huntington disease becomes fully penetrant, with no citation anywhere. A StudyRow named a
variant — rsid, or a bare chrom — and a repeat_alleles.csv row is keyed (gene, repeat_unit), so
nothing could point at it.
0.6 closed it with a second citation site: MeasureBinRow.pmid, one optional column on the binning
base reaching every binning kind, plus a relaxation of StudyRow's subject requirement so the paper behind
a threshold can be described without inventing a variant for it. The rule for reading the pair is the
bin row cites, the citation table describes — the pointer sits on the row that states the number, and
everything about the paper stays in studies.csv. heteroplasmy.csv had a second route from the
start: its optional rsid/chrom/start columns (0.5.1) give a row a variant identity a study row
can name, which reference_examples/mt_heteroplasmy does — but that route is the study row, so it
grounds a bin only in a module that carries one.
_check_binning_grounding still warns in both modes, over the bins that carry no pmid, in a
module with no study rows at all — and the remedy it names is the same for every kind, since every kind
can now cite its boundary; a kind whose rows can be pointed at is offered the study-row route as a
second one. It used to exempt a bin for merely naming a variant, inside a branch that has already
established the module records no study rows, so the citation clearing the bin was one that does not
exist: a heteroplasmy.csv module stating four thresholds and citing nothing was green and silent while
the identical thresholds on repeat_alleles.csv were reported (D1-3, fixed in 0.6). The same-release
obligation was the reason the
item was filed rather than fixed: _cross_check_literature reads the bin pointers alongside
studies.csv (otherwise every threshold-grounding citation would read as a stale orphan), and so does
the enricher's literature pass, so a bin-grounded citation is checked for existence and identifiers
exactly like a study-grounded one. reference_examples/htt_repeat_expansion is deliberately left
uncited: the example exists to show what the warning looks like.
0.7 added a third site the same way (RM132), and stopped listing them. PharmVariantRow.pmid
grounds one drug/genotype/category claim, which studies.csv cannot: a study row keys on
(variant_key, pmid) and would attach to every claim recorded for that variant at once. The lesson
above is now enforced structurally rather than remembered — _CITING_TABLE_KINDS is every
_TABLE_KINDS model declaring a pmid, and table_citations/load_citing_rows walk it, so the next
kind to declare the column is read by both tiers with no edit to either. _check_binning_grounding is
untouched by this and stays about bins: there is deliberately no equivalent grounding warning for an
uncited pharm row, because a drug-response table is not the interpretive-threshold case that check
exists for.
One paper, several analyses, and the dedup key (RM140)¶
The three sites above are about which row cites a paper. This one is about what a citing row's
numbers mean once it has. A studies.csv row carries p_value and effect_size side by side and
asserts they belong together; before 0.7 nothing recorded, and so nothing could check, whether they
came from the same analysis. study_design names the study — case-control, GWAS, meta-analysis — and a
single study routinely reports several analyses of one association: the motivating module cited a paper
giving OR 1.4, p 0.36 from an allelic Fisher's exact test and OR 1.42, p 0.75 from a univariate
logistic regression of the same variant, and two agents building from it produced rows that differed
only in p_value, one of them pairing the magnitude of one analysis with the p-value of the other.
Everything was green — including quote verification, because the quote grounds the significance
verdict and contains no statistic to witness.
StudyRow.statistical_test (0.7) is the column, and it is deliberately not a gate. Free text like
study_design, no vocabulary, no new check. What it changes is one existing one:
duplicate_study_citation fires on a repeated (variant_key, pmid) because the check's own reading of
that pair is the same claim written twice — which two rows naming two analyses are not.
A pmid=NONE row is keyed on its DOI (0.8, RM336). Every DOI-only citation reads NONE, so two
different preprints on one variant would otherwise share (variant_key, NONE). The check keys such a
row on its DOI token, lower-cased (spec.doi_key: a bare DOI and its doi.org URL are one DOI), and the
message names the cell as written (Duplicate (variant, pmid): (rs…, NONE doi 10.1101/…), through
spec.cited_as), so the same preprint written twice, in either spelling, still warns. Every row with a
real pmid is keyed and worded as before. The two other study findings that print a row's pmid
(study_effect_allele_not_at_locus, p_value_encodings_disagree) name a NONE row the same way.
A study's outcome class outside the recommended set warns once (0.8, RM326).
phenotype_class_unrecognized names every phenotype_class token outside
RECOMMENDED_PHENOTYPE_CLASSES with its row count, in one finding, in both modes. The set is open, so
the rows stand; the warning exists for the second spelling, since v2 writes FAMILIAL_LONGEVITY beside
FAMILIAL and MR_MEDIATOR beside MR. Text: studies.csv phenotype_class outside the recommended
set: 'FAMILIAL_LONGEVITY' (2 rows), … — the set is open, so the rows stand; ….
A published column stated without its companion warns once per table and column (0.8, RM366).
paired_field_half_stated reports the refuse=False entries of a model's _PAIRED_FIELDS — today
effect_size with effect_measure empty on variants.csv and studies.csv — with the row count and one
example value, in both modes, from the pre-flight validate runs (so compile carries it unchanged).
The pairs that refuse never reach it: the row failed to load. Text: studies.csv: 11 row(s) state
effect_size with effect_measure empty (e.g. effect_size=…) — effect_measure is what the number means …
Reported rather than refused: a module published before this rule may hold the half.
A proxied study is checked against the proxy (0.8, RM331). study_effect_allele_not_at_locus reads
the resolved alleles of proxy_rsid when a row states one, never the lead's: filed under the lead, a
proxy's allele used to be refused when the two loci differ and passed with the wrong meaning when they
happen to match. The text names both (rs646776 via proxy rs4970836 (PMID …): effect_allele 'T' is not
among the resolved alleles at the proxy (A/C/G)); a row with no proxy reads exactly as before. A proxy
with no resolution.csv row withholds, as an unresolved lead does — the enricher resolves proxies as
subjects so the row exists.
Only both stated and different suppresses the warning. The rule is Kleene rather than a != b:
an absent statistical_test is unknown, and unknown against a stated value cannot establish that
two rows describe separate work. So a pair where neither row names an analysis, where both name the
same one, or where one names one and the other does not, warns exactly as it did before, with the same
code and the byte-identical message — every module published before 0.7 behaves as it did. A row
repeating an analysis already stated for its key still warns however many distinct siblings sit beside
it.
Since 0.8 the same rule reads more columns (PT2 §3 item 16): phenotype_class (one paper measuring
several outcomes, RM326), the age band (age_min, age_max, age_of, read cell by cell, RM329) and
the stratum pair (stratum/stratum_level, RM330). Two rows under one (variant_key, pmid) are two
claims when any of these, or statistical_test, is both stated and different; a row warns when it
is not distinct from some earlier row under its key. The band is read per cell because each bound is
optional on its own: 80– against 80–90 share the one bound both state, and an empty bound is
unknown. proxy_rsid is deliberately not read: two proxies of one lead in one paper are the same
association measured twice. Every case that still warns keeps the byte-identical message, and a module
writing none of the new columns warns exactly as in 0.7.
StudyRow._KEY_FIELDS is not widened, and the divergence is contained in this check. That tuple
drives hints.key_fields and the key.columns an authoring surface publishes; re-keying a shipped
authored table changes what an identity key means, which is major-only under Principle 3. The check
restates (variant_key, pmid) rather than reading _KEY_FIELDS, so splitting it here splits nothing
else.
haplotypes.csv's gene splits its duplicate rule the same way (0.8, RM356), and there the split is
declared on the model because three surfaces must agree on it. The key stays
(haplotype_name, variant_key, allele); HaplotypeRow._KEY_DISTINCT_WHEN_STATED = ("gene",) says two
rows under it are still two rows when both state gene and the symbols differ (compared case- and
space-insensitively, as RM351's cross-table checks compare a gene). The UGT1A genes share exons 2–5, so
UGT1A1's *2 and UGT1A6's *2 can be defined by one variant and allele, and before 0.8 the second was
refused as haplotypes.csv: duplicate row for key …. A row naming no gene still collides with any row
under its key, a gene stated twice still collides, and the refusal keeps its byte-identical text. The
relation is not transitive (a gene-less *2 collides with both genes' rows, which do not collide with
each other), so no hashable key can carry it: the compile/validate refusal, draft.append_rows and the
hints lint each bucket by the key and ask one predicate, compiler.tables.same_row, within the bucket.
hints.key_fields publishes the column as TableKey.distinct_when_stated, beside columns rather than
inside it. The change only narrows what is refused, and nothing it reads feeds the parquet row order or
content_signature: the reference examples with this table compile to the same artifact.digest with
and without the declaration (compiler/tests/test_haplotype_key_gene.py).
Locus-level columns weights.parquet derives from the study rows (0.8)¶
VariantRow names no study, so a fact that lives on studies.csv reaches a variant only by a rule.
_build_weights collects each variant's supporting study rows — any shared identifier, the same rsid
or the same chrom:start:ref, which is the rule the orphan check uses — and derives each column below
from them, repeating it on every genotype row of the locus (a parquet column is approximately free, P9).
None is authored on a variant, and reverse reads none back. All are appended after every 0.7 column.
The orphan check matches by that rule against a wider set since 0.8 (RM28's study-orphan widening): a
study's variant is carried when it is a variants.csv row's, a defining variant in haplotypes.csv, a
pharm_variants.csv row's (whether or not a meta member reaches it), or the variant of a row a
meta.csv member reached (read off the meta expansion, so a selector covers exactly the rows it
matched; a member reaches only variants.csv and pharm_variants.csv rows today, so this arm adds
nothing the others miss). A module scoring APOE only as diplotypes therefore draws no
study_variant_orphan for its studies on rs429358 and rs7412, and those studies feed no
weights.parquet row, since there is none. The check runs whenever the module has a studies.csv, with
or without a variants.csv: a diplotype-only or PGx-only module is checked against the arms it has, and
duplicate_study_citation, computed in the same pass, now runs there too. When an arm's file is present
but no row of it could be read, the orphan half withholds: what the module carries is unknown, and the
load errors are the report. The message keeps its pinned prefix, Studies reference variants not in
variants.csv, and names the wider search after the keys.
effect_unit(RM332) — the unit of the variant's effect, when every supporting study row that states a unit states the same one. Null when none states one, and null when two disagree: never the first, never the most common (PT2 §3 item 17's default, accepted by the pm).eur_only_evidence(RM328) — Kleene over the supporting rows'ancestry: false when any row states a code other thanEUR(EAS|EURincluded), whatever else is unstated; otherwise null when any row states none, or when the locus has no supporting row; otherwise true. ABooleancolumn, null meaning unknown.- The outcome strip (RM326) —
outcome_healthspan,outcome_lifespan,outcome_exceptional. A cell is true when a supporting row'sphenotype_classmaps to it invocab.OUTCOME_STRIP_CELLS(HEALTHSPAN→ healthspan;PARENTAL_LIFESPAN,OWN_SURVIVAL→ lifespan;EXTREME_SURVIVAL_CC,FAMILIAL→ exceptional) and null otherwise, never false: a locus with no study of an outcome is untested, not "no effect". True says the outcome was studied at this locus, not that an effect was found; the study rows carry the finding. Every other class,MULTIVARIATE_AGEINGincluded, maps to no cell. v2's requirements §5.1 name the three cells and state no class map, so the map is PT2's.
Three more schema limits, made legible the same way (0.6, the VCF 4.4 audit)¶
Same class as the bin-boundary gap above — limits of the schema, not of the tier — and they are here for the same reason: so they are not mistaken for the other kind, and so the warning a reader meets on a real module has somewhere to point. All three warn in both modes and none changes a verdict.
The two integer measure kinds are not integral (RM55) — warned, and then fixed in the same line.
VCF 4.4 §7.2 redefined CN to support non-integer copy numbers and §3 types RUC as a Float, so the
premise repeat_count and copy_number were placed in binning._INTEGER_KINDS on has been withdrawn
for both. The consequence was RM35's unsatisfiable triangle re-instantiated on the kinds RM35 exempted,
and worse: under a grid a hole of exactly one is not reported at all, so [0,0] [1,1] [2,2] [3,∞) is a
legal, gapless, green tiling under --strict that answers nothing for a CN of 2.4 — and the schema also
refused the tiling that would fix it, since a shared endpoint on those kinds was an overlap error.
The usable fix landed in 0.6 rather than 0.7: an optional measure_tiling column ({quantised,
continuous}) that the shared-endpoint and gap rules read instead of the kind, with absent meaning
the kind's default so no published table is re-read; modifier_copy_number beside the one genuine
int, modifier_cn deprecated; and a fractional value switching a would-be-quantised group to the
continuous rules by itself, saying so in a warning. See SCHEMAS.md for the per-kind
default table and the resolution order. binning.measurement_shape_warnings still says the RM55
sentence once per table, but only where it is still true — a kind VCF 4.4 types as fractional that
still has a group reading as a grid. A table declaring itself continuous, or carrying a fractional
bound, answers its own boundaries and is silent. Only the removal of modifier_cn stays at 1.0, so
1.0 inherits a removal rather than the retype the original route named.
A measurement can span several bins (RM56). The same two fields carry confidence intervals (CIRUC,
CICN) whose missing bound means unbounded, so a real measurement is an interval; htt_repeat_
expansion states three thresholds inside a 14-count window for one to cross, and the consumer contract
has no state for it. 0.6 warns and states the placeholder — withhold — rather than leaving it
silent. The policy vocabulary (withhold / worst bin / point estimate) and its grain wait for a real
caller VCF. Widening the measurement itself into an interval is not on the table: that puts a
measurement in the module, which the data-agnostic north star forbids.
. in an alts cell splits identity (RM58). VCF's MISSING marker means there are no alternate
alleles, and no ref/alts column has a nucleotide grammar (deliberately — adding one would tighten
the field RM5 exists to widen), so the cell loads and derive_variant_key folds it in as though it named
an allele: 1:1:A:. where the same site with an empty cell is 1:1:A, two content_signatures and no
dedup between them. alleles.non_nucleotide_reason now answers a third reason, "missing", distinct
from "ambiguity" (a permanent uncertainty) and "notation" (a grammar gap a release may widen) — .
is neither, and there is nothing to widen to hold it. A diagnosis, not a grammar: the value is still
accepted, and the compiler warns per table with the two keys side by side. It is the only finding of the
VCF round that reaches identity, and it reaches only the key string — is_substitution refuses a
non-nucleotide alt, so no VA is minted and no content-addressed claim is false.
One finding from the same round is not a schema limit and is listed with the pointer columns in
SCHEMAS.md instead: a min_quality floor stated against QUAL on a requires_callable
row cannot be read on the evidence the row is decided from, because a reference block usually leaves
QUAL missing (RM313). The compiler warns and deliberately does not refuse — what a caller writes on its
reference records is something this tier never sees. Until 0.8 this paragraph said QUAL inverts on a
reference record, which is the spec read backwards; see the retired phrase below.
And a fourth — a pointer that does not identify a VCF field (RM53/RM54, 0.6)¶
Same class as the three above: a limit of the schema, closed in 0.6, and worth keeping in the same place because the check that survives it is again a legibility warning rather than a verdict.
Three authored columns point into a VCF (source_field, callable_from, quality_from) and all
three took a bare token. A VCF field is identified by namespace — INFO and FORMAT are two
reserved-key tables that collide on DP, AD, ADF, ADR, MQ, AF, CICN and, since 4.4, CN — and
described by cardinality (Number, which decides how many values come back and what each is of).
Where both readings are type-compatible, and for DP, AF and CN they are, nothing detects the
confusion: the consumer reads a well-formed number of the wrong kind and bins it without error. Both
shipped reference examples that used these columns were wrong this way, under --strict, with every
offline gate passing — the same failure geometry as the 3,038-row coordinate incident.
The schema half is in SCHEMAS.md: the pointer grammar accepts INFO/DP/FORMAT/DP (bare
still legal, still meaning unqualified), the spec's own key charset is accepted (1000G, a dotted
key — RM61), and MeasureBinRow.source_element names which element of a multi-valued field the bin is
measured against, from a closed set of named rules rather than an index (P1 refuses AD[1]).
_check_vcf_pointers is the compiler half, and it has two findings, both warnings in both modes:
a bare key that is one of the known collisions, and a pointer at a spec-defined multi-valued field with
no element rule. Neither escalates under strict — the grammar was widened rather than replaced, so
refusing there would break P3, and strict means reproducible artifact, which an unqualified pointer
is (P5). Both are aggregated by reason, since a panel pointing every bin at one field would otherwise
print the same sentence hundreds of times.
Two things it declines to say, both for the reason this section exists. Cardinality is read from a
transcription of the spec's reserved keys (VCF 4.5; a key added later is unknown) and nothing else — REPCN is ExpansionHunter's
key, not the spec's, so this tier is not entitled to assert its Number, and a bare CN disagrees
across the two namespaces. Unknown withholds. And the cardinality finding is scoped to pointers that
have a companion column: callable_from/quality_from did not get one in 0.6, and telling an author
to fill a column the schema does not have would be a finding no edit could clear.
And a fifth — a site annotated for some of its genotypes and not the rest (S32, 0.6)¶
Not a schema limit at all: this one is entirely visible in the authored rows, and nothing but a check
was ever missing. A consumer matches a subject on (variant, genotype), so a genotype with no row is a
subject with no answer — and the consumer who reported it found a curated 520-site module authoring no
homozygous-alternate genotype at 208 of them, with their subject homozygous at 74. Every one was
silently unreported.
_check_genotype_coverage fires only at a site the module already annotates for two or more
genotypes, and that scope is the design rather than a tuning choice. One genotype at a site is the
ordinary shape of a drafted-then-curated module — pathogenic_clinvar authors exactly one at 326 of its
327 sites — and it is a legitimate rule, not a gap; reporting those would put a line on almost every
module in existence, which is how a warning stops being read. Two or more is the author demonstrating
that the genotype space at that site is what they are describing, and the missing member is then a
hole in something they started. It expects the reference homozygote, one heterozygote per alternate and
one homozygote per alternate, and never an alternate/alternate pair — RM35's jointly-satisfiable
lesson, since requiring A/T would make a two-alternate site unreachable. The reference allele comes
from the row or from the injected table, never from a guess: with neither, a two-allele site is still
enumerable and is reported by spelling, and a site with three or more alleles is skipped. Sites whose
genotypes are not diploid nucleotide pairs drop out on their own, which is what keeps MT and non-PAR Y
out without a contig list.
It says nothing about any callset, and that boundary is the item's own. Whether a hom-ref row can
ever match is a property of the data a consumer brings — a variant-only VCF emits no record where the
sample matches the reference, a gVCF and an array both do — so that call belongs to the annotator. The
presence of a hom-ref row is therefore never reported: those rows are correct, and on array data they
are the ones that carry the answer. Warning in both modes and never a strict error, joining the small
set of checks that arbitrate nothing (the ClinVar clin_sig cross-check, the declared-licence
disagreement, the non-commercial quote): which genotypes a module annotates is the curator's judgement.
It runs in validate_spec only, and reaches a compile through the warnings it returns — the message
carries a count, and after resolution a one-to-many rsID has become one row per locus, so a second pass
would publish a second, differently numbered copy of the same finding into
manifest.compilation.warnings. Three reference examples fire it, each truthfully:
grch37_build and hfe_hemochromatosis state a carrier and a homozygote and no reference homozygote,
and pathogenic_clinvar states both HBB heterozygotes at 11:5225715 and neither homozygote.
Hints are not a fourth validation class¶
hints.py computes nothing the compiler does not already compute — it reuses validate_bins,
_TABLE_DUPE_KEYS and the models' own validators — and it never fails a build. It has no mode
ladder, because the checks that do have one already exist here and in the enricher, each with the
severity the charter assigns it (including the two deliberate exceptions that warn in both modes). A
hint that could fail a compile would be a third copy of a rule that already lives in two places.
Two findings are the exception. The missing-sentinel warning
(§ 1 above) and, since RM279, a variants.csv conclusion that names genotypes at its own locus and
never its own (just_dna_compiler.conclusion) are computed here and nowhere on the compile path. The
second waits on a warning code, which is minor-class (RM308). Neither can fail anything.
What hints add is when: the same verdict, before the author has written the file, plus the allowed
values and the reason a cell is deliberately left empty. That last part is the load-bearing one — see
hints.REDUNDANCY_BEARING. Class 2 works because two independently-authored things must agree, so a
tool that fills one of them from the source the checker consults does not merely make the check
tautological; for an rsid-only row resolution._verify never runs at all, and the row would go from
honestly unverified to apparently verified. Every looked-up fact is therefore reported with
applied=False and a refusal reason, and the enricher's lookup.py answers in the same shape.
A column name is not a scope, and the reason given now says which table it applies to (RM123).
REDUNDANCY_BEARING maps a bare column to the checker that reads it, and clin_sig is a column on
variants.csv, on every binning kind and on diplotypes.csv while verify_clin_sig takes
list[VariantRow]. So the vacuous-check sentence printed on six (column, table) pairs whose checker
cannot see the table — right advice, false reason, and a green run on one of those cells looked like
agreement with an authority that never saw it. REDUNDANCY_BEARING_TABLES narrows the explanation
for the columns whose checker reads fewer tables than carry them; an absent key means unscoped, which
is the honest answer for rsid/chrom/start/ref/alts (resolution reaches the positional kinds,
RM43) and for pmid (a bin and a pharm row are citation sites in their own right, RM47/RM132).
It scopes nothing else: the refusal a
drafting provider obeys stays keyed on the bare column, because whether a provider should start filling
clin_sig on a binning row is a decision nobody has taken.
One consequence of computing nothing new is that the same cell can be seen by two layers, and a hint
must still print it once. An unreplaced TEMPLATE_PLACEHOLDER is the case: _check_placeholders
names the column, and the model then says the same thing less usefully — a row-level ValueError
listing every placeholder path on an authored model, or a vocabulary error quoting the token back on
sources.csv/licensing.csv, whose row model carries no such guard. A freshly drafted 109-row panel
printed two lines per defect. _validate_row drops both restatements, keyed on the cells the
per-column check actually reported, so a placeholder error nothing covered would still be shown. The
guard itself is untouched: it is what makes a generated stub unable to compile, and a hint refuses
nothing.
And what is not the compiler's job at all¶
Static checking has an upper bound here, and the analogue of dynamic analysis lives elsewhere. The
compiler is not a runtime verifier: it never runs a module against a genotype, because a module carries
no sample data and the measurement is supplied by the consumer at query time. The ecosystem's
"valgrind" is the consumer-side verification harness — run a panel against N VCFs and diff a
report-card — which is deliberately not a format feature (USE_CASES.md §3b / RM7).
The format's contribution to it is the properties it already froze: artifact.digest makes the
before/after diff trustworthy, and the mandatory unresolved/callability contract stops a no-call
masquerading as a mismatch.
The compile pipeline¶
A file the compiler has no meaning for is ignored, and that is a contract (S16). A spec directory
may carry a README (every reference_examples/ module does), curation notes, or a publisher's receipt
recording the identity a registry stamps — those keys cannot live in module_spec.yaml, where
extra="forbid" rejects them precisely because the registry owns them. Such a file is not read, not
hashed, and not in artifact.files, so it cannot move artifact.digest; pinned by a test that
compiles the same spec with and without two unknown files and compares digests. The one exception is a
near miss: an unknown .csv within one small edit of a table name (varaints.csv) warns, because
"ignored" is the wrong answer for a typo — every row in that file is being silently dropped from a green
compile. The check is edit-distance-keyed rather than "any unknown csv" on purpose, so it cannot undo the
tolerance above. It reads .json as well as .csv since 0.6, which covers verification.json and
retroactively covers provenance.json — that one had been in the known-name set since 0.4 with no
suffix that could reach it, so a provenence.json was invisible to the very guard written for this.
A registry's published.json is not within an edit of either and stays tolerated, which is the
property the widening had to preserve.
Where the machine-written sidecars may live, and what they may be called (RM49/RM51, 0.6).
resolution.csv, the fact tables and verification.json are resolved through
just_dna_format.layout, which accepts each of them at the spec root or under a derived/
subdirectory, and accepts the licence table under either sources.csv (deprecated, warn-only, removed
at 1.0) or licensing.csv. Four rules:
- Only the machine-written tables move.
module_spec.yaml,variants.csv,studies.csvand the table kinds have exactly one legal name in exactly one legal place. Two legal homes for an authored table means a module can carry two copies with the ignored one invisible. derived/is tolerated, never canonical.reverse_moduleemits a flat tree and the enricher creates one; a module is split only because somebody split it.reverse_modulewrites the sidecars through the same resolver, so a round trip migrates the name. It regenerates a spec directory rather than editing one, so on a fresh tree there is nothing to follow and the rule yields the preferred spelling — a module carryingsources.csvreverses ontolicensing.csv, and a compile → reverse → compile no longer picks up a deprecation notice its first compile did not have (manifest.compilation.warningsis published, so the two must agree). Reversing over a directory that already carries a copy overwrites that copy instead of leaving a second one beside it.- Either spelling is a key (RM224).
SIDECAR_SPELLINGSis keyed on the table key —sources.csv, the name the parquet and the manifest keep — but everylayouthelper accepts any spelling of the table, so a caller holding the filenamelicensing.csvfollows a module'ssources.csvrather than creating the second copy.layout.sidecar_keyis the filename → key map, derived from the table. - Two copies of one table is an error naming both paths — never a merge, never newest-wins. These tables are fact-hashed and human-overridable, so two copies are two legitimate claims and preferring one silently discards a curator's override. The enricher's rule is the other half: write to the file you read.
- The guard follows into
derived/, and it takes two tests there. Tolerating a second location without extending the guard would put a typo'dderived/varaints.csvexactly where the check written to catch it cannot see — which is also why "search any subdirectory" was refused: one fixed name is the only version the guard can follow. What is legal underderived/is the sidecars alone, but that smaller set is the wrong thing to fuzzy-match against, and matching against it caught neither case: at the 0.8 cutoffvariants.csvis no near miss of any sidecar name, and neither isvaraints.csv. So an authored table name underderived/is reported as misplaced on an exact match — those rows are read from nowhere, and a module that keeps another table compiles green without them — while everything else there is fuzzy-matched against the full known set. The acceptance set stays the smaller one, so a legal sidecar is never reported as a stray, and the mirror case (a sidecar at the spec root) is legal and stays silent.
Neither the name nor the location enters any identity: the fact sidecars are outside _INPUT_FILES
(see the file sets below), so artifact.digest, content_signature, resolution_signature and
source_signature are unchanged by either. manifest.derived records the relative path it found, so
a derived/… entry tells a registry how the tree was laid out.
The compiled outputs are untouched: a module reading licensing.csv still writes sources.parquet
and still publishes manifest.sources. Both of those are renames only a major may make.
compile_module runs in this order:
- Validate (
validate_spec), in this compile's mode and with its options; fail early if invalid. Since RM367 the pre-flight runs the compile's own resolution step and every check the compile asks (compiler/check_paths.pynames the exceptions, asserted against both function bodies), so the compile's own refusals below are defensive except on the deprecatedensembl_cachepath. - Load
module_spec.yaml(authority-key pre-strip), thenvariants.csv/studies.csvif present. - Load
resolution.csvif present, throughresolution_step.read_resolution— the readervalidate_specuses — into the active view: post-overlay rows, empty under--no-resolve(a row-parse error fails the compile either way). Then verify every storedvrs_id(below), and stampresolution_signature/resolution_sourceshere — where the table was read, so a module with novariants.csvrecords the identity of the table it carries rather than a null (0.6, RM45). - Resolve (only if
resolve_with_ensembl and variants) — the precedence block below. - Re-validate identity post-resolution (
identity_after_resolution:_cross_validate_variantsplus the build-coordinate check) — resolution can change identity (fill a coord, expand a one-to-many rsid), so a post-resolution duplicate/inconsistency fails the compile ("post-resolution: …"). - Compute
fully_resolved= every variant haschrom+start(vacuously true for a variant-less module), andresolution_subjects= how many rows that quantified over, from the same list, so the flag cannot be published without its denominator (RM44). - Strict gate — if
strictand any variant still lacks(chrom, start), fail before any parquet is written (refuse a non-reproducible partial artifact). - Load, overlay and check the derived-fact sidecars, and judge the ancillary files
(
ancillary_errors:provenance.jsonparses, a logo's or readme's extension is one the manifest carries) — still beforeoutput_dir.mkdir(). EveryCompilationResult(success=False, …)incompile_modulesits above that call, held by an AST walk (test_check_paths.py, RM367): a refusal after the first write leaves a half-written artifact, and seven did until 0.8. - Write parquets — SNP core (
weights/annotations/studies.parquet, only when the relevant rows exist) + one parquet per present table kind + the derived-fact sidecars (frequencies.parquet,gene_metrics.parquet,literature.parquet, and since 0.6gene_validity.parquet,clinical_assertions.parquet) when their CSVs are present, each cross-checked against what the module actually contains (a frequency coordinate no variant sits at, or a gene the module never mentions, is a warning — an over-broad sidecar is harmless, and failing the compile over it would punish the author for the enricher's generosity). "What the module contains" is one answer,subjects.module_subjects(RM362): a gene any ofvariants.csvand the gene-bearing table kinds names (the setmanifest.stats.genespublishes and the enricher's gene scope asks about), and a locus or rsID any ofvariants.csvand the positional table kinds names. A module withoutvariants.csvis checked the same way. A locus is also every place the injectedresolution.csvputs an identity those rows carry (subjects.identity_placements: byvariant_keyorrsid, the fill'susable_locipredicate, GRCh38 and resolution on only), so an indel authored in one spelling and resolved in another is one place; BA1 and the study join behindweights.parquetask the same per row (subjects.row_positions). - Collect logs /
provenance.json/ logo / readme — copy and hash only; a malformed one was refused at step 8, and byvalidate_specbefore that. Log discovery takes*.logbeside the spec and, underlogs/at any depth, every*.logunchecked for content, plus each file whose content matches its name (RM338):.txt/.mddecode as UTF-8 (a byte-order mark allowed),.jsonparses, every non-blank\n-terminated.jsonlline parses, and a.png/.jpg/.jpegstarts with its format's magic. Only an exact.logsuffix skips the content check; any other extension is compared lowercased, so a.LOGis checked as text. A hidden path (any part starting with.) never takes the checked route. Discovery does not trust the filesystem (RM348), for*.logtoo, beside the spec or underlogs/: a path must resolve to exactly where it lies inside the spec directory, so a symlink at any level stays behind, and alogs/that is itself a link is not walked (one finding nameslogs); every file is held toLOG_MAX_BYTES(5 MiB), read fromstatbefore any byte; and a file the compile cannot stat or open stays behind rather than raising, while a directory it cannot list and search (logs/included) is named in place of the files it hides, which a walk would otherwise skip without a word. Each file that ships and is not*.logis named by an info line onjust_dna_compiler.compilerand, injust-dna-compiler compile, by alog ships: <name>line. Every other path discovery considers drawslog_file_not_shipped(RM337), a warning in both modes and never a refusal, one line per path:<path> does not ship with the module: <reason>.The reason is a fixed sentence fromLOG_NOT_SHIPPED_REASONS, one per arm, decided by the same function that decides the verdict (_log_left_behind_reason), so the two cannot drift: a hidden path, a symlink or a path under one (to a file, a directory, or nothing), a file over the ceiling (both sizes named), a file the compile could not read ("the compile could not read it", RM348), an image whose first bytes are not its format's signature, an SVG ("XML that can carry script"), any other extension, text that is not UTF-8 (the offset of the first bad byte named), a.jsonthat does not parse, and a.jsonlline that does not parse (the first such line numbered, blanks counted). The reason carries no interpreter error message, since it reachesmanifest.json. There is no cap and no_examplestruncation: the decision was to name each file, andwarnings_summarycounts them. An explicitlog_fileslist skips discovery, ships what it names and draws no warning.validate_specruns the same discovery (discover_logs), so the pre-flight names what will stay behind (RM367). A file beside the spec that is not*.logis outside this: the top level discovers*.logonly, and an unknown file there is the spec directory's ordinary tolerance. The readme is discovered frommanifest.README_CANDIDATESand hashed intomanifest.readme, outsideartifact.files— so it is attested without being content (S25). - Read the verification attestation (
verification.json, RM45): recompute the binding over the authored inputs — with\r\nread as\nsince 0.6 (RM82), so an editor's line endings are not an edit, whilemanifest.inputs[]keeps listing the raw bytes and raw sizes because it answers are these the exact bytes — re-check the proof-of-work, and either carry the block into the manifest or warn and drop it. A stale attestation never fails a compile — the goal is that it never becomes a published claim, not that it be impossible to write while editing. Nothing here is trusted; see SCHEMAS.md. The same read decides the closure (RM73, 0.6): if the block that survives carries none, the compile says so — a warning in both modes, carrying no count so the identical sentence fromvalidate_specde-duplicates against it. Absence is the only thing said here, because the reason (never closed, closed then edited, no document at all) is already carried by whichever other warning applies. - Build the manifest (
content_signaturere-read from raw disk, the resolution fields, thefrequency/gene_metrics/literature/verificationblocks, andderived[]— byte hashes of the sidecars where they live beside the spec, transport-only and never their identity) and writemanifest.json.
The VRS verify pass (0.5)¶
A GA4GH concept in a no-network tier — deliberately, and asserted. VRS is normally met alongside sequence services and a client library, so importing the idea into the compiler is exactly the kind of change that quietly drags a network dependency behind it. It does not: allele identification is
sha512t24uover canonical JSON — arithmetic — while only normalization (indels) needs sequence access, and that half lives solely in the enricher.just_dna_format.vrsimportsbase64/hashlib/json/reand nothing else;ga4gh.vrsappears nowhere outsidejust_dna_enricher.compiler/tests/test_tier_purity.pypins this in a fresh interpreter (the test suite has the enricher loaded, so an in-process check would prove nothing): the compile path imports no network client,just_dna_format.vrspulls no non-stdlib module, and a full compile — minting the VA, keying on it, and rejecting a tampered id — succeeds withsocket.socketbooby- trapped to raise.The residual risk is not what is there, it is the gradient: the verifier is partial (it can recompute a substitution and not an indel), and the tempting completion is to give the compiler sequence access. That is the line not to cross — the asymmetry is the design, not a gap. Likewise
refget_accessionraising for a non-GRCh38 build is not an invitation to fetch the accession; it is an invitation to add a second committed table (RM15).
A ga4gh:VA.… is content-addressed, so it is the one column in the whole artifact that can be checked
against itself — no reference, no network, and no new dependency, since derive_vrs_allele_id is
stdlib (Goal 2). _verify_vrs_ids runs before anything is written, so a bad id never reaches an
artifact. It belongs here rather than only in the enricher because the compiler is the last gate
before an artifact exists: a spec can be hand-edited and compiled directly, never touching the
enricher, so an enricher-only check would be bypassable. Checking injected data by pure computation is
precisely the compiler's job — the same thing it already does for every hash and digest it writes.
Three outcomes, and why "unverifiable" is not "mismatch"¶
Every row with a vrs_id lands in exactly one of three outcomes. The distinction between the last two
is the point of the whole design, and conflating them would be a lie about what was actually checked:
| Outcome | Meaning | best_effort |
strict |
|---|---|---|---|
| verified | recomputed, and equal | silent | silent |
| mismatch | recomputed, and different | error | error |
| unverifiable, the tier's limit | could not be recomputed here, and no edit would change that | warning | warning |
| unverifiable, the row's contradiction | an id recorded against nothing to check it with | error | error |
| minted under another VRS version (0.8, RM304) | the row's vrs_spec states a version other than the pin; reported beside each allele's own outcome |
warning vrs_version_differs |
warning |
Note what the mode column does here: nothing. This pass is not a mode ladder. Severity comes from whose limit the finding is, and both answers are the same on both rungs.
A mismatch is always fatal, in both modes. A substitution's id is fully deterministic here — same
inputs, same 20 lines of hashlib, same answer — so a disagreement cannot be a difference of opinion
between implementations. It is corruption, and there is no mode in which carrying it is right.
An id minted under another VRS version is reported, and changes one verdict (0.8, RM304). The id
rule can change at a VRS minor: VRS 2.1 picks the smallest repeat-unit factor for an ambiguous insertion
where 2.0 picked the greatest, which moves that allele's digest. So a row whose vrs_spec names a
version other than the one this tier implements (VRS_SPEC_VERSION, published as the module's pin
manifest.compilation.vrs_version) is reported as vrs_version_differs, one line per stated version.
The line sits beside each allele's own outcome, never instead of it: an indel under 2.1 is still
counted in vrs_id_unverifiable, and a substitution is still recomputed. A substitution needs no
normalization and this tier never recomputes an insertion, so inside the pin's major a substitution that
recomputes differently is still the corruption error, whatever the cell says; a stated version is not a
way to silence one. The one verdict it changes is a substitution under another major ("1.3"),
whose digest is taken over a different serialization: that difference is expected and is counted in the
line rather than refused. A patch spelling ("2.0.1", the library's own) reads as its minor through
vrs.vrs_spec_release, the one normalizer the enricher's stamp and this comparison share. An empty
vrs_spec is unstated, the unknown arm, and an unreadable one is quoted in the line; both rows are
verified exactly as before. Re-minting clears the warning, so it is actionable rather than carried. The
pin is None when no allele in resolution.csv carries an id.
An indel is never reported as a mismatch, because it is never compared. This tier cannot recompute
an indel's id (justification needs the reference sequence), so it can only report that it did not
check. Saying "mismatch" would assert a verdict that was never reached. Warnings land in
manifest.compilation.warnings, so an unconfirmed identity is visible to a consumer rather than only
to whoever ran the compile.
Why the tier's own limits do not escalate under strict. They did, for one release cycle, on the
reasoning that unchecked and correct are different things and strict's contract is a reproducible
artifact. The first half is true and is why the outcome exists at all; the second half does not follow.
An enricher-minted indel VA is reproducible — the bytes are injected, the compile is deterministic,
and recompiling yields the same digest. What is out of reach is the verification, not the
reproduction, and escalating on that conflates "I could not check this" with "this cannot be rebuilt".
The cost was concrete rather than theoretical. Minting indel ids online is exactly what
just-dna-enricher exists to do, so every ClinVar-derived module acquired identities that
compile --strict then refused, and the two remedies the error offered were lower your guarantee or
delete a correct identity. Two reference examples — pathogenic_clinvar (185 alleles) and
shox_par1 (2) — stopped compiling in the mode their own READMEs document, and the authoring skill's
step 6 tells every author to run exactly that mode. The rule now matches the one
_vrs_coverage_warnings and frequencies' not_covered already followed: a finding no authored
edit could clear is not a strict matter — strict is orthogonal, and P5 says orthogonal axes stay
orthogonal.
What still errors, and in both modes, is the row contradicting itself: a vrs_id recorded against
no coordinate, against no ALT, or — since R2-5 — against a symbolic allele. That is not a limit of
this tier. The first two assert an identity while withholding the very thing that identity is a digest
of; the third asserts one for an allele that has no sequence to digest, so the id necessarily names a
different allele. Nothing anywhere could check any of them. Same class as the inconsistent reference
allele error, and catchable offline.
Every flow path¶
_recompute_vrs_id returns either the recomputed id or the reason there is none, for one allele.
vrs_id is a comma-joined parallel array of alts, so the pass walks the two together and each ALT
gets its own verdict; an empty member is a hole and reads exactly like an empty cell. Most of the
reasons are limits of a no-network tier rather than defects in the row — the last three are not:
| Row | Path | Both modes |
|---|---|---|
no vrs_id, or a hole in one |
nothing to check — not the same as "could not check" | silent |
| substitution, id agrees | verified | silent |
| substitution, id differs | mismatch | error |
| multi-allelic, every member agrees | verified allele by allele | silent |
| multi-allelic, members swapped | mismatch — the desync a length check cannot see | error |
indel / MNV (C>CA) |
needs the reference sequence — minted upstream, not recomputable here | warning |
| off-assembly contig, or a position past the contig end | no refget accession to address the sequence by | warning |
non-GRCh38 genome_build |
no refget table for that build (RM15) | warning |
*, the unobservable-allele marker (RM59) |
the callability axis — it names no allele, so nothing to digest | warning |
symbolic allele (<DEL:4977>, RM5) carrying an id |
no tier mints one, so a recorded id names a different allele | error |
position-only (no alts) |
an id against no ALT — the row's contradiction, not the tier's | error |
| no coordinate | an id against no place (an rsid row carrying an external id) | error |
A symbolic allele with no id is a coverage warning; one carrying an id is an error, and the
asymmetry is the rule rather than an exception to it. Absence is this tier's limit — no authored edit
clears it, which is what keeps every structural module compilable — while a recorded id is a claim
about an allele that has no content to address, so it can only name something else. Absence is a
limit; a claim is a claim. The escalation waited on the grammar (R2-5): it follows only once vrs_id
is known to hold allele ids alone, which vrs.validate_vrs_allele_id now enforces (ga4gh:VA. only,
probed at 844 corpus ids with no other type). The remedy is one edit — delete the cell; variant_key
still carries the allele's identity.
* sits beside the symbolic row and not inside it (R2-6). The two are different axes, which is why
_vrs_gap_reason and _recompute_vrs_id test is_unobservable_allele separately and above the
substitution fall-through: parse_symbolic_allele asks which variant is this, unspelled, while
is_unobservable_allele asks whether the sample's call could see an allele at all. Reported as an
indel it would carry a remedy — re-run the enricher online — that can never apply.
Multi-allelic used to be one row of this table, blanket-unverifiable, "a VA names exactly one allele;
picking one would invent data". The premise is derive_variant_key's and it is right there — a
variant_key names one thing, so a plural cell falls through to the coordinate key. It was wrong here,
where nothing is picked because every ALT is named. It cost the id on 909 of 1,613 rows in one real
module while every input needed to mint all 2,110 of them sat in the same row.
A symbolic allele is its own row of that table, and it was an indel for the whole of 0.6 until a
module carried one. RM5 shipped the structural grammar and this pass was never told, so <DEL:4977>
fell through to is_substitution and was reported as "an indel or MNV: justification needs the
reference sequence, so only the enricher can mint it (re-run it online)" — false on every clause.
Symbolic notation exists precisely because the sequence is not spelled: there is no sequence to
justify against, nothing for a content-addressed id to be a digest of, and no tier that mints one, so
the id is permanently absent rather than one online run away. (The enricher's minting pass says the
same wrong thing in its own words, and the online run this message recommended crashed on the same
allele — both of those are its side of the item, not this one's.) Two
consequences worth keeping. The branch sits ahead of both the substitution test and the accession
lookup, in _vrs_gap_reason and _recompute_vrs_id alike: a symbolic allele on a GRCh37 module is
also a build with no refget table, and between two true statements the one to print is the one no
release can answer. And the predicate is the lenient is_symbolic_allele — a malformed <FOO> or
an unterminated <DEL names no sequence either, so filing it under "indel" would be the same false
claim about a different mistake.
Coverage — what a VA does not name¶
Verification only ever looks at ids that are present, so a table where nothing was minted verifies
flawlessly. _vrs_coverage_warnings reports the other half: allele slots seen, how many carry an id,
and the remainder grouped by reason class (not by row, and not by _recompute_vrs_id's per-row
prose — grouping on that produced forty lines each naming a different indel). The counts are recorded
in manifest.compilation.vrs_alleles / vrs_alleles_identified, so a consumer can read the
reliability of the identity scheme instead of inferring it; "complete" is identified == alleles,
derived rather than stored twice. A shortfall is a warning in both modes — the usual causes (an
indel with no sequence proxy, a build with no refget table) are fixable by no authored edit, and
strict means "reproducible artifact", which an incompletely-named table still is.
A symbolic allele still counts in the denominator, so a structural module reads as permanently
short of complete coverage — reference_examples/mt_common_deletion compiles at 2/3 and always will.
That is the honest number: the denominator is allele slots in resolution.csv, one definition, and
excluding the slots that can never be named would publish a second, undisclosed one — "coverage over
the alleles an id was possible for" — which is exactly the flag-with-a-hidden-subset shape RM44 exists
to stop. What tells the two situations apart is the reason line beside the count, which now says the
gap is permanent instead of naming a remedy.
The last row is a fixed bug worth naming: refget_accession raises UnsupportedBuildError rather
than returning None — deliberately, so a caller asking for GRCh37 hears "not built yet" instead of
receiving a GRCh38-flavoured answer. That exception used to escape the verify pass and abort the whole
compile over a single unverifiable row. It is now caught and turned into a reason, which is the correct
severity: one row this tier cannot check should not fail a build in either mode.
The inconsistent-reference-allele check (0.5)¶
A VA addresses the place and the alt; the reference base at a position is a fact of the genome and is
not part of the allele's name. Correct VRS semantics — but it drops a guarantee the old
chrom:start:ref:alts key gave for free, since two rows at one position claiming different reference
bases used to be two keys and are now one. At most one can be right, so _cross_validate_variants now
fails a compile where two positioned rows share a key and disagree on ref.
Symbolic alleles: the one check that discards an authored row (RM5, 0.6)¶
The grammar holds <DEL:1500> / <CNV:TR:30> (see SCHEMAS.md § The allele grammar).
What the compiler owns is the other half: a module is a declarative rulebook, and a rule nothing
can apply is worse than an absent one. A <DEL> with no length cannot be sized, matched against a
call, or told apart from any other deletion at that position, so _check_symbolic_alleles reports it —
and this is the first check in the tier that discards an authored row.
| Reason | What it is |
|---|---|
no_length |
a real structural type with no usable length: <DEL>, or <DEL:0> |
unknown_type |
angle-bracketed but outside the closed five — <FOO>, and VCF's <*>, which makes an observability claim rather than naming a variant |
reference_allele |
a symbolic allele in a ref column: REF is always a sequence, so a locus whose own reference is unspelled anchors nothing |
Severity depends on whether the row stands alone as a rule. On variants.csv and
pharm_variants.csv — one self-contained rule per row — best_effort drops the row with a warning
that says so, and strict refuses. On haplotypes.csv and heteroplasmy.csv it is fatal in both
modes: those rows are parts of a composite, so dropping one silently redefines a haplotype or punches a
hole in a bin tiling — not a smaller module but a different one.
A drop that would empty a table outright is an error in both modes as well, and on its own reason:
the drop exists so a module can lose one unusable rule and still say the rest, while a table that loses
every row says nothing at all and says it only in a warning. (Not, as a first cut claimed, because the
compiler already refuses a present-but-empty table — that is true of the _TABLE_KINDS loop and
false of variants.csv, which validates and compiles header-only. Measured, and pinned by a test.)
The warning must say DROPPED. This does not break P7 — the round-trip fixed point is claimed under
strict, where this case refuses — but reverse cannot re-emit a row that never reached the parquet,
so a warning that merely flagged it would leave an author believing their module still carries it.
Three mechanics worth copying. The check reads AuthoredModel.ALLELE_COLUMNS, declared on each model,
so the compiler holds no second copy of a model's column names. It runs in validate_spec too, by the
standing rule (pure computation over authored bytes, no output_dir) — with the identical message, both
because the pre-flight must predict what the compile will do and because compile_module's
de-duplication is on the message; every refusal it can reach is computed in the shared check rather
than at the point of application, so validate cannot go green on a module compile then rejects in
the same mode. And manifest.stats is re-derived over the surviving rows: weights_rows counts the
parquet, so leaving variant_count as validate_spec computed it would publish a count higher than the
artifact holds — the RM44 class, a manifest number a catalog keys on and cannot check.
Findings name a row by its identity (variant_key, else haplotype_name), never by a file
position: load_csv_rows prints a header-inclusive line number and hints.Finding.row is a 0-based
data index, so a third convention would be one too many — and an index computed over the rows that
survived model validation shifts silently behind any earlier load error.
One asymmetry to expect: <FOO> fails at load in genotype / effect_allele /
HaplotypeRow.allele, which have a grammar, and reaches this check only through ref/alts, which
deliberately have none.
Resolution precedence (additive; Principle 3)¶
Inside step 4, gated on resolve_with_ensembl and variants, with resolution_mode = "strict" if strict
else "best_effort":
resolution.csvpresent →resolve_from_table(the preferred, source-independent path — noduckdb, no network, no Ensembl convention). Setsresolution_signature(fact-hash of the rows) andresolution_sources(sorted union of rowsources).- else
ensembl_cachegiven → DEPRECATED DuckDB path. Emits aDeprecationWarning("… removed at 1.0. Produce a resolution.csv …") and routes to the enricher via a guarded lazy import (from just_dna_enricher.resolver import resolve_variants); if the enricher isn't installed, the compile fails with a message pointing at it or at precomputingresolution.csv. The compiler declares no dependency on the enricher. - else (nothing injected) → skip.
Noneno longer auto-discovers a cache (the 0.5 Principle-2 tightening); variants lacking a position are left unresolved with a warning pointing at the enricher.
Digest parity between paths 1 and 2 is the load-bearing guarantee: given the same facts, both emit
byte-identical weights.parquet (hence artifact.digest). The one order-sensitive spot — a one-to-many
expansion — is pinned by sorting on (locus_index, chrom, start, ref). Since 0.8 it holds when the
table records no vrs_id: path 1 carries the ids resolution.csv records onto the parquets (RM270),
and path 2 has no table to read them from, so with ids recorded the two weights.parquet differ in
vrs_id/vrs_spec and nowhere else. Since RM380 it holds for the facts, not the bytes: a key whose
locus holds a value no row cell rebuilds keeps its resolution_loci, and every enricher-written row records
authority, so path 1 carries loci path 2 has no table to fill. The two artifacts then agree on every file
but weights.parquet, and that file differs in resolution_loci (and, with ids recorded,
vrs_id/vrs_spec) alone (test_offline_enrich_then_compile_matches_duckdb_digest).
Resolution¶
Resolution is where authored data meets injected facts, and it is the one place in the compiler where a row can change shape. Everything below follows from one rule:
Resolution must be reversible.
compile → reverse → compilehas to reproduce the module it started from — the same bytes, the same authored content, and the same resolved facts. Where it cannot,strictrefuses rather than emitting an artifact nobody can re-derive.
Scope: the SNP core plus the three positional tables (RM43, 0.6)¶
Resolution applies to variants.csv and to the three positional 0.4 kinds —
pharm_variants.csv, haplotypes.csv, heteroplasmy.csv, derived from the models rather than listed
(a table is positional exactly when it declares both chrom and start). Until 0.6 it applied to
variants.csv alone: every other table went through _build_table, which is the model straight to
parquet, so a row kept exactly the coordinates its author typed — for an rsid-authored module, none —
and a consumer matching a patient VCF by position matched nothing, silently, as an empty result rather
than an error. That is now filled at compile time from the injected table; see The positional fill
below for the four rules it follows and the three columns it needed.
The rest of this section is the 0.5.3 report (S9) that surfaced it, kept because two of its three manifest observations still hold:
fully_resolvedisall(...)overVariantRow, so it is vacuouslytrueon a module with novariants.csv. Its own field comment gives the trust rule "a consumer trusts a module whenresolution_mode == "strict" or fully_resolved" — which is not sufficient for a 0.4-family-led module, where it can betruewhile every row lacks a coordinate. Since 0.6 the denominator is published beside it (resolution_subjects, RM44), so the sufficient rule isresolution_subjects > 0 and (resolution_mode == "strict" or fully_resolved)and the vacuous case is legible from the manifest alone. The counter makes the vacuity visible; RM43 makes the tables joinable, but the flag still quantifies overvariants.csvonly.resolution_modeand the--strictunresolved gate are the same scope.strictrefuses on an unresolvedVariantRow; it says nothing about a table row, which is why such a module compiles under--stricteven where the fill could not place a row. That is deliberate — see the warning's severity below.resolution_signature/resolution_sourcesstay unset for a module whose only subjects are table rows, so its injectedresolution.csvleaves no trace in the manifest. This was blocked on reverse, which rebuiltresolution.csvfromweights.parquetalone; RM43 removed that blocker (it now rebuilds from the positional parquets too), and stamping the two fields is RM45's half of the same round.
The warning's wording is a contract, because the manifest carries it and nothing else.
compile_module copies its warnings into manifest.compilation.warnings, which ships inside
manifest.json, and a catalog reindexing from a published manifest has no spec directory left to
re-derive anything from — so for a table-only module the sentence is the only surviving record that
its rows join to nothing, fully_resolved being vacuously true. A downstream registry substring-
matches "have no chrom+start" to decide a trust badge; compiler.UNJOINABLE_PHRASE names that
fragment and a test pins it, so a reword breaks this build rather than a catalog. Improve the rest of
the sentence freely; move that fragment deliberately.
RM44 shipped in 0.6 and it retired half of this; S31 shipped the other half.
resolution_subjects gives a consumer a structured field for "was the flag about anything", and
positional_rows / positional_rows_placed now give the unjoinable-row count RM44 recorded as
belonging with RM43 — the two counts, parts not a ratio, so "this table joins by position" is
positional_rows_placed == positional_rows and needs no prose. The fragment and its test still
stay, and not out of caution: every artifact published under 0.5 carries neither field, so for those
manifests the sentence remains the only record. Retiring it is a decision to take once the published
corpus has been recompiled.
Reading it from the consumer side, which is what these fields are for. A module's rows join by
position when positional_rows_placed == positional_rows (with positional_rows == 0 meaning the
module carries no such table, so the question does not arise) — and variants.csv is the separate
question fully_resolved over resolution_subjects answers. None on either field means the
artifact was compiled before 0.6, which did not fill those tables at all, so the honest reading is
unknown, and probably unfilled rather than zero: a 1,482-row pharm_variants module compiled under
0.5 has every coordinate null and no manifest field saying so. That is the state the reporting
consumer had to discover by opening the parquet, and it does not change when a new compiler is
installed — a published artifact only moves when it is recompiled and republished, which is the
maintainer's action rather than the consumer's.
The joinability warning now reports the residue. Every positional table is still checked for rows
with no chrom+start, after the fill has run, and the finding is one aggregated line per table
carrying how many rows cannot be joined by position plus why. Three readings: the injected table
names the key at more than one locus (or at one the row's own allele contradicts), so the compiler
leaves it rather than picking; nothing in the table names the key at all, which an enrich run fixes;
or the fill never ran — --no-resolve, or a non-GRCh38 module — in which case the coordinates may
be right there and untried. The third branch is why the check is told whether the join happened
rather than inferring it: this sentence ships inside manifest.compilation.warnings beside
UNJOINABLE_PHRASE, so asserting "the compiler looked and would not pick" about a lookup that never
happened puts a fabricated diagnosis into a document a catalog reads. A half-coordinate (start with
no chrom, the shape a CPIC-drafted haplotypes.csv carries) is counted apart, because it reads as a
position and joins to nothing.
It is a warning in both modes and never a strict error, for two independent reasons: rsid-only
identity is legal by these models' own rule, so escalating would have the format tighten a field it
deliberately left open; and what survives the fill is by construction something no authored edit to
that table clears — the same class as VRS coverage and not_covered, where refusing makes a correct
module uncompilable for something its author cannot fix. It runs in validate as well as compile,
and is de-duplicated between them.
The positional fill (RM43, 0.6)¶
_apply_positional_resolution joins the injected resolution.csv onto each positional table before
_build_table materializes it, and runs in both validate_spec and compile_module — the
joinability line is computed from these rows in both, so filling on one side only would leave the
pre-flight naming a gap the compile had already closed. validate_spec therefore takes
resolve_with_ensembl too, and compile_module passes its own value down: the flag is the master
switch for resolution of every kind, so a pre-flight that ignored it would be the more optimistic of
the two commands. The fill is skipped, with a warning, for a non-GRCh38 module, exactly as
resolve_from_table is (RM15).
Four rules, and the last two are what separate it from the naive repair:
- Fill only what the author left empty. A cell the author wrote is never overwritten. That is the inject-only doctrine (report, never repair), and it also makes the fill idempotent.
- Fill from exactly one locus, or from none. One usable locus fills. Several are filtered by
hosting_verdictagainst whatever allele the row states — agenotypeon a pharm row, the definingalleleon a haplotype junction, nothing at all on a heteroplasmy band. If that leaves one, it fills; otherwise the row stays unplaced and the joinability line says so. There is deliberately no expansion: multiplying a pharm annotation's(variant_key, drug, genotype, …)key across loci the author never named is not the same operation as expanding avariants.csvrow. - A row whose own coordinate contradicts the table is left exactly as authored, and the
disagreement is reported. Completing a half-coordinate from a locus whose
startdisagrees would build a coordinate no source ever stated. The comparison runs even where there is nothing left to fill — a fully-populatedheteroplasmy.csvrow can still contradict the table it is keyed into, and the promise is that such a row is reported, which the SNP core gets from_verify.altsis deliberately outside the comparison: a locus lists every ALT recorded there while a row names the one it is about, soA,GagainstGis agreement, and whether the allele can sit there ishosting_verdict's three-valued question, already asked one step earlier. startandrefare a spelling, and the enricher's record decides them (RM375, 0.8). An indel anchored elsewhere in its repeat differs in both and is one event, which only a reference read can tell; the compiler has none, so it readsresolution.csvauthored_event(recorded_verdicts, one predicate shared with_verify), and only where itsjudged_coordinateis the row's own coordinate (a verdict about another coordinate, stale or another table's, reads as never judged).same_event: agreement, no warning, and the fill takes only the cells no spelling changes (rsid,chrom— a CPIC half-coordinate is completed), never areforaltsthat would be worn at the author's anchor.undecided: left as authored, under the same code, in words that accuse neither side (a wrongrsidor contig is still accused).different_eventor an empty cell: today's contradiction.- Rows are mutated in place and their identity is frozen first. Each positional model stamps
variant_keyandauthored_identat load, from the authored columns only, so filling cannot re-key a row.
Three columns made it possible, all parquet-only. variant_key (materialized so a consumer can
join a PGx row to weights.parquet without re-implementing the precedence rule), authored_ident
(which identity columns the author supplied — the same mechanism VariantRow has had since 0.5), and
alts on PharmVariantRow/HaplotypeRow, filled as data, not identity: the key is still derived
without it, so a pharm annotation keeps matching a variant at chrom:start:ref regardless of allele,
and what the column buys is a direct VCF join. None of the three is authored, offered by a drafting
template, or re-emitted by reverse; none is in content_signature (they are exclude=True, so a
stamped value — a pure function of the authored cells — cannot move a content identity, and no
already-published module's signature changes). VariantRow's own two stamped fields are inside
content_signature; that asymmetry is grandfathered rather than a precedent, since changing it in
either direction moves published signatures.
Reverse rebuilds resolution.csv from the positional parquets, and that is forced rather than
chosen. Once a coordinate is filled, a reverse that dropped the lookup table would emit a spec whose
recompile leaves those parquets unfilled — compile → reverse → compile would stop reproducing the
artifact (Principle 7), and a PGx module carries no weights.parquet for the old writer to read at
all. Weights are written first and own any shared key (variants.csv is the only table carrying
alts as an authored fact). Since RM364 the rows are the table's own: weights.parquet and every
positional parquet carry resolution_loci, the loci the table filed under the row's authored key, and
reverse writes those back as stored, several under one key where the table had several, so the next
compile reads the one-to-many the first read. Since RM380 every column a locus carries comes back —
every ResolutionRow column but the ones parquets.RESOLUTION_LOCUS_EXCLUDED names, so authority,
status, the placement witness and verdict, left_aligned, source_release and caid survive the
round trip. Reverse stamps the key, the module's genome_build, source="reversed" and a blank
fetched_at itself, and leaves out rsid_alternates/rsid_current/rsid_status. A row whose
resolution_loci is null (its own cells rebuild the key's loci exactly, or a parquet compiled before
the column) is rebuilt from its cells, and there a positional table still contributes one row per key,
since a second row rebuilt from cells would invent a one-to-many the table never stated. A module that
resolved nothing anywhere and has no weights.parquet gets no file, rather than a header-only sidecar
invented out of an absence, unless the compiled spec carried a table: then a header-only table comes
back header-only (RM364).
A key the table files nothing under says so (RM451, 0.8.9). resolution_loci is null where a row's own
cells answer, and that read the same as a key the table never filed. A coordinate row authored after the last
enrich was then rebuilt from its cells into a table row the injected table never had, and the second lap read
it (resolution_signature moved). A module that read a table (InjectedResolution.consulted) now writes []
for a key with no positioned locus on its build, and reverse writes nothing for it. Lap 2 re-reads the header-only
table it is handed and writes [] again. A module that read no table keeps null.
resolution.csv still gets no parquet, and that is the repair this item deliberately did not
make: it is a build-time derived artifact whose consumers are the compiler and the enricher, and
publishing it would turn it into a consumer contract it was never designed to be. See SCHEMAS.md.
The VRS ids on the variant parquets (RM270, 0.8)¶
An indel's variant_key and coordinate are its source's spelling, and a caller may spell the same event
another legal way, so an exact (chrom, pos, ref, alt) join misses it. The VRS allele id is minted over
the fully justified allele, so both spellings of one event carry one id (rs72613567 at 4:87310240
T>TA and 4:87310241 A>AA) and two events carry two (S117's rs8176719). Since 0.8 the id the
enricher recorded reaches the artifact:
| Parquet | vrs_id |
vrs_spec |
|---|---|---|
weights.parquet |
List(Utf8), parallel to alts, null where no id is recorded |
Utf8 |
pharm_variants, haplotypes, heteroplasmy (positional._VRS_BEARING_TABLES) |
Utf8, comma-joined like the table's alts and resolution.csv's cell, holes kept |
Utf8 |
studies.parquet proxy_loci |
a struct field, the proxy row's cell | a struct field |
the positional parquets' resolution_loci (RM364) |
a struct field, the table row's cell | a struct field |
Not on annotations.parquet, which carries variant_key but no allele to be parallel to (it joins to
weights.parquet on the key), nor on frequencies.parquet, whose vrs_id is gnomAD's.
Copied, never minted (compiler/vrs_columns.py): the compiler has no sequence to justify an indel
(P2), and an id minted here would reach resolution.csv on reverse and make the second lap count more
identified alleles than the first. From the row's own resolution.csv row, by allele, not by position
in a cell: a parquet row reads the rows filed under the key reverse files it under
(vrs_columns.resolution_key, the authored key, so an expansion member finds its rsID's rows) at its own
(chrom, start, ref), and its list follows its own alts. Never another subject's id for the same
spelling: true of the allele, but reverse would write it under this row's key and the round trip would
gain ids the table never held. Two rows of one key recording one allele with two different ids withhold
it (null), since one of them is wrong. vrs_spec is the cell as recorded, stated only when every id on
the row came from rows stating the same one. Under --no-resolve both columns are null.
resolve_from_table (compiler/resolution.py)¶
Pure, and mirrors the DuckDB resolver's semantics from the injected table:
- fill (1:1) — a
variant_keywith one usable locus fills the missing coordinate or rsid; the frozen key is kept. A coordinate-authored row also has itsaltsfilled when the author left it out, so the resolved allele reaches the artifact and survives being written back. - expand (1:N) — an rsid with N usable loci becomes N coord-keyed rows, each re-keyed by
derive_variant_key. A locus whose{ref} ∪ altscannot host the authored genotype is not expanded onto (hosting_verdict): one authored genotype is copied to every locus, so a locus that lacks those alleles would be emitted as a row asserting an allele it does not have. - verify — a row carrying both rsid and coordinate is checked against the table; a disagreement
warns in
best_effortand refuses instrict. A locus whose recordedauthored_eventissame_eventis agreement (RM375: the enricher read the reference, the compiler cannot);undecidedkeeps the ladder in words that accuse neither side.
GRCh38-bound (a non-GRCh38 module is skipped with a warning; not_found/wrong-build rows are ignored).
Hosting is a three-valued question (RM31)¶
hosting_verdict(genotype, ref, alts) answers can this locus host that genotype with True / False /
None, because an indel has several valid spellings and a string comparison reporting "does not fit" was
asserting a verdict it had not reached. ClinVar publishes a SHOX deletion as X:634689 CAG>C and Ensembl
publishes the same event as X:634690 AGAG>AG.
| Situation | Verdict | What the compile does |
|---|---|---|
No ref/alts recorded |
True |
keeps the locus (lack of evidence never rejects) |
A * among the alleles, on either side (RM59, 0.6) |
— | the member is dropped and the rest is judged normally; nothing observable left is None |
| The raw allele strings match | True |
keeps it — checked first, so normalization can only ever add acceptances |
| Either side names a symbolic allele (RM5, 0.6) | None |
keeps it, reports that it did not decide — no sequence, so no flank and nothing to compare |
The reduced allele sets match (alleles.parsimony_reduce strips the shared flank) |
True |
keeps it; this is what reconciles the two spellings |
| The locus is a substitution or MNV and the alleles differ | False |
drops it — no flank, so no spelling freedom; a strand-flipped genotype stays a hard finding |
| The genotype names fewer than two distinct alleles at an indel locus | None |
keeps it, reports that it did not decide (a homozygous call carries no frame) |
| The event sizes differ | False |
drops it — re-anchoring never changes how many bases an event adds or removes |
| Same sizes, different content | None |
keeps it, reports that it did not decide (a rotation inside a repeat, or two variants) |
The * row sits above the raw comparison, and the symbolic row below it — the two are on different
axes and the placements are not interchangeable. A symbolic allele makes a claim that cannot be
compared, so the whole verdict withholds; * makes no claim at all, so only the member goes and the
observable half must still be matched. That is why it has to run before the subset test rather than
after: {'*','T'} is not a subset of a real A>T locus, so below the comparison it fell through to the
substitution row and returned a confident False — dropping the locus and leaving the row unresolved,
which would have made a * authorable and uncompilable in the same release, refusing under strict
for a reason no authored edit could clear. No source spells * in an ALT list, so this is the ordinary
path rather than a corner. */T at an A>T locus is True; */G there is still False, because the
G is a real contradiction and abstention drops a member, never a verdict; */* is None. It costs
the stability property nothing — the called side is stripped first, so * is never on the left of the
subset test and dropping it from the right cannot change that answer.
Both sides, and the locus side is the half that is easy to miss. parsimony_reduce strips the
flank a collection shares, and * has none, so a * left in the locus stops the whole set reducing
and RM31's reconciliation collapses: hosting_verdict('C/CAG', 'AGAG', 'AG') is True while the same
call against alts="AG,*" came back a confident False — a correctly transcribed indel refused under
--strict, advised to "replace it with the alleles the locus actually has". ALT=AG,* is exactly what
a joint caller emits at an overlapped indel, so this is common data, and an allele the algebra must
ignore cannot be one it ignores in only one direction.
What that costs, measured rather than asserted. Swept over every pre-RM59-reachable
(genotype, ref, alts) triple built from a spread of substitution, MNV, insertion and deletion
alleles: for a locus spelled in nucleotides — every reference example, and every module in practice,
since a * could not be written in a genotype before RM59 — no verdict changes at all. For a locus
that does spell *, some do, and they are corrections in both directions: * was blocking
parsimony_reduce's flank strip and lending its own single character to _indel_shaped's length
set, so AG>AT (really G>T) read as indel-shaped and withheld on calls that were decidable all
along. The verdicts that newly refuse are ones the tier should always have refused; a module relying on
one had a genotype that did not fit its locus, and the * was suppressing the finding.
Because None now has four causes rather than one, resolution.undecided_reason supplies the clause
both reporting sites append (this tier's expansion warning and the enricher's twin). They used to
assert step 9's cause — "the same size but different content … needs the reference sequence" — for
every withheld verdict, which for an all-* call sends the reader to check a reference against a
position nothing observed.
The symbolic row sits above the reductions on purpose. Below the raw comparison every remaining step is
arithmetic over characters, and a <DEL:1500> has none to offer — parsimony_reduce would read it as a
nine-character sequence and the event-size rule would then return a confident False computed from a
token's bracket count. Two stated lengths that differ are undecided for the same reason in the other
direction: symbolic notation exists for imprecision, so a summary length is not the kind of fact an
event size is. Note that it only ever adds acceptances, so no already-compiled module moves.
None is the residual only a reference sequence can settle, which this tier does not have (P2) — the
enricher does, and reports it the same way. genotype_fits remains as the boolean face
(hosting_verdict(...) is not False): it is public and shared with the deprecated DuckDB path in
just-dna-enricher, because digest parity between the two is a documented guarantee and a filter applied
on one side only would break it silently. _check_allele_membership asks the same predicate rather than
comparing strings itself — while it did, the two halves of the compiler disagreed the moment a spelling was
reconciled, and strict refused a module resolution had just accepted.
One thing the reconciliation does not do: the authored genotype keeps the frame its source published
it in, so a compiled row can legitimately carry genotype=C/CAG beside ref=AGAG. A consumer joining the
two applies the same reduction (just_dna_format.alleles is public and dependency-free); rewriting the
authored cell is the parked enricher-co-authoring item.
One rsID, one placement set (RM374, 0.8)¶
resolution.csv is merge-not-clobber: enrich fills the subjects it has no row for and never re-asks the
rest. So a module enriched before a snapshot moved and again after it holds both releases' answers,
and nothing compared them until 0.8. The 0.8 migration probe's instance is rs761191455, which one table
placed at 13:48041104 G>GG under its own key (the old Ensembl release's right-shifted spelling) and at
13:48041103 T>TG under the positional key a later row was authored at (the new release's left-aligned
one). A consumer joining a left-aligned VCF by position matched one and missed the other; validate,
compile and strict were silent.
read_resolution now asks it, over the active (post-overlay) rows, for both commands
(resolution.placement_disagreements, code rsid_placements_disagree). It is a property of the table, so
it is asked where the table is read, and a module with no variants.csv (a PGx-only one) is asked too. A
warning in best_effort, a refusal under strict and under validate --strict, GRCh38-bound like the rest
of resolution.
What one answer is. A variant_key is one subject, and a re-run replaces a subject's rows whole
(ResolutionRow._KEY_RULE = "subject"), so the rows under one key are one answer and locus_index only
orders its loci. A one-to-many rsID, a pseudoautosomal pair, and one multi-ALT locus that RM270 respelled
into pieces at one start with two refs (shox_par1's rs1569493663 at X:634689, CAG>C and
C>CAG) are each one answer and never this finding (@membership-union). Two answers about one rsID
are two keys whose rows carry it: the rsID's own key, which is the full answer, and a positional key whose
rsID was back-filled, which answers for the one locus its row was authored at. The finding is a positional
answer whose (chrom, start) is not among the full answer's, with a Y pseudoautosomal locus compared as its
X partner. Alleles are not compared: a stale allele set under one key is the membership check's axis
(below).
Where the table cannot tell, the check withholds rather than refuse, and these are its limits:
- No row is keyed by the rsID itself. Two positional rows labelled with one rsID may be two loci of a one-to-many rsID or one event spelled twice, and the table does not say which.
- One key is never reported against itself, even where it repeats a
locus_index. A hand-filled table leaves the column at its default 0 for every locus of a one-to-many rsID, which looks exactly like two answers merged under one key. Such a key is still the full answer the other keys are compared with. - An
ambiguousrow, whose rsID is a deterministic pick rather than a fact, and a row with nostart. - Two answers whose spellings the enricher judged apart, one
left_aligned=trueand the otherfalse. Within 0.8 a filled coordinate is written left-aligned, so a stale answer re-filled now agrees with the fresh one; a true/false pair is one event spelled two ways in one release (an old snapshot's spelling beside the respelled fill), which--rederivewould write again. That is the spelling question.
The full answer is the rows plus source_loci (RM387, 2026-11-05). The rows under the rsID's own key
are the loci its authored genotype can sit on, because the enricher writes them after the hosting filter,
a disputed placement (RM267) and the pseudoautosomal choice; a positional
row's rsID is back-filled from the source's unfiltered answer. So for an rsID the source places at two
loci, a positional row at the sibling locus the filter left out read as a second placement, --strict
refused a valid module, and --rederive wrote the same table again. The rsID-keyed rows now record
every place the links offered in source_loci wherever that is wider than what they place, and the check
compares against the union, with a Y pseudoautosomal place read as its X partner as before; the sentence
then names the offered places, (the source offered 1:1000, 1:6000), after the rsID's own answer. A row with no
source_loci is read as the whole answer: the rows place everything offered, or the table predates the
column, and then RM374's stale case still fires and --rederive is what clears a sibling.
The sentence names each answer as under <variant_key> at <chrom:start ref>alts>, then the releases: each
answer's [release …] when rows record one (ResolutionRow.source_release, RM373), else
(releases not recorded). Under strict it adds that the table must place each rsID one way. The remedy
it gives is just-dna-enricher enrich <spec-dir> --rederive, or deleting resolution.csv and enriching
again; either re-asks every subject against one snapshot.
The authored shape is recorded, not inferred¶
VariantRow.authored_ident lists which of {rsid, chrom, start, ref, alts} the author actually
supplied. Like variant_key it is stamped once at load and never re-derived, so filling or expanding
cannot disturb it, and it is materialized to weights.parquet for reverse to read.
It exists because the alternative — inferring the authored shape from variant_key — cannot work. That
key answers "which variant is this", not "what did the author write": it is identical for an rsid-only
row and an rsid+coordinate pair, and after an expansion it is the per-locus allele id with no trace of
the rsid the author wrote. Reverse therefore used to materialize resolved coordinates into
variants.csv and emit one row per expanded locus, which cost two things: content_signature moved on
every round-trip of an rsid-authored module, and each locus received a copy of the single authored
genotype — writing out, as authored fact, annotations for loci the genotype cannot describe.
Why this is only now fixable: dropping the resolved coordinate from variants.csv is safe because
the key is canonical. A VRS allele id names the variant without the coordinate having to be re-authored,
so the coordinate can live in resolution.csv where it belongs. Under the older coordinate-first keying
the coordinate was load-bearing in variants.csv and could not be dropped.
The one recorded finding strict acts on (RM143, 0.7)¶
strict means reproducible, never right, and that is unchanged — the compiler has no reference, so
it cannot check a coordinate. But when the enricher has already checked one and recorded the
answer, throwing that away at the tier boundary is a separate failure: a GRCh37 coordinate pasted into
a GRCh38 module was refused by enrich --strict, written by enrich best-effort, and then built
silently by compile --strict into an artifact that is internally consistent and about a locus
5.6 Mb away.
build_disagreement_error refuses a strict compile when verification.json records a finding on
genome_build_agreement, in validate_spec and compile_module alike, with the identical error and
ahead of output_dir.mkdir() so a refusal writes nothing.
Why that check and no other. Its findings say the module's rows are on a different assembly than
the genome_build it declares — one authored file contradicting another, which is internal
consistency. Every other recorded finding is the module disagreeing with an outside archive, where
the archive is the stale side often enough that failing a build would have the format arbitrate someone
else's dispute; that is the same reasoning that keeps the ClinVar cross-check a warning under strict,
and it now has a parametrized test over those checks. reference_allele is the one worth naming: it
produces this diagnosis's own input and still does not refuse alone, because a ref mismatch has three
causes and only one of them is an assembly.
Three things it does not do, each with a test, because each would be a worse defect than the one it fixes:
| state of the record | what happens |
|---|---|
no verification.json at all |
silent — an unverified module is the ordinary case, and refusing on absent evidence reads unknown as wrong |
genome_build_agreement with findings=0 |
silent — a clean bill; the gate keys on findings, never on the record's presence |
genome_build_agreement skipped (what --offline writes) |
silent — nobody asked, which is not nothing-wrong |
A stale attestation is dropped by the reader before the gate sees it, which is the right order: bytes that moved since the check ran are bytes the check did not judge.
Coverage is reported by the pre-flight too (RM141, 0.7)¶
compile --strict refuses a module whose variants still have no position after resolution — a partial
artifact is not byte-reproducible, which is the failure behind "my local hash differs from the
published one". Until 0.7 validate --strict said nothing about it, so a spec whose resolution.csv
covered only some of its variants passed the pre-flight clean and was refused by the compile
immediately after.
That is the parity rule's own failure shape, and it hid behind the rule's exemption. What stays compile-only is a check reading resolved rows; whether the injected table can place an authored row is arithmetic over bytes the pre-flight has already loaded, and needs no resolution to have run. The exemption is about resolved rows, not about the word "resolution".
Since RM367 the pre-flight runs the compile's resolution step itself (compiler/resolution_step.py:
resolve_variants, identity_after_resolution, unplaced_refusal), over the rows the compile keeps,
rather than a predicate shared with it. The predicate (resolution.unresolved_subjects) agreed with the
compile only on GRCh38 with resolution switched on: on a GRCh37 rsID-only module, or under
--no-resolve, validate --strict said valid and compile --strict refused. And the exemption itself
did not hold for an injected table — resolving it is computation over bytes the pre-flight has
already loaded — so locus_cannot_host_genotype, rsid_coordinate_disagrees and the post-resolution
duplicate are refused by both commands now. What stays the compile's alone is the deprecated
ensembl_cache reference, which the pre-flight is not handed. Under strict the pre-flight's error is
the compile's, verbatim.
Three behaviours to know:
| the module's table | what is reported |
|---|---|
| present, covers every row needing a position | nothing |
| present, covers some | rsid_unresolved per uncovered row, then the strict refusal — the row is absent from a file that was consulted |
| absent entirely | resolution_not_injected, once, plus the strict refusal — nothing was consulted, so nothing is named row by row |
--no-resolve silences the per-row findings the way it silences the fill: resolution is off by
request, and resolution_disabled already says so once with its row count. The strict aggregate is
not silenced, on either side.
One finding, though two passes produce it. compile_module runs the pre-flight, which emits the
per-subject warning in both modes, so both reach this finding for the same subject; the compile de-duplicates on
the message (the _check_contig_ploidy idiom). That is safe here for the reason the rule requires —
neither message embeds a count resolution could change, so two passes over one subject produce the
identical sentence rather than two differing by a number.
The mishap matrix¶
Five identity columns the author may or may not supply, crossed with what the table says about them, is
a finite set. Every combination is either a round-trip fixed point on all three signatures, or it
fails in strict — enumerated and enforced in compiler/tests/test_resolution_matrix.py.
| Authored shape / mishap | best_effort |
strict |
Round-trip |
|---|---|---|---|
| rsid only, 1:1 fill | ✅ | ✅ | stable |
| rsid only, one-to-many (every locus can host the genotype) | ✅ | ✅ | stable |
| rsid only, pseudoautosomal pair (X + Y, same place) | ✅ | ✅ | stable — the compiler names it as one place; the enricher decides whether both reach the table |
| coordinate + alt, rsid resolved | ✅ | ✅ | stable |
| coordinate only, rsid and alt resolved | ✅ | ✅ | stable |
| pair (rsid + coordinate), table agrees | ✅ | ✅ | stable |
| withdrawn — the table records the rsid as retracted by dbSNP | ❌ refuses | ❌ refuses | — |
| ambiguous — several rsIDs for one allele | ⚠️ warning | ❌ refuses | stable |
| expansion drops a locus that cannot host the genotype | ⚠️ warning | ❌ refuses | stable since RM364 |
| every candidate locus contradicts the genotype → unresolved | ⚠️ warning | ❌ refuses | stable since RM364 |
not_found — the table records the rsid as genuinely absent |
⚠️ warning | ❌ refuses | unstable |
| no resolution row at all | ⚠️ warning | ❌ refuses | stable (nothing to lose) |
authored ref contradicts the table |
⚠️ warning | ❌ refuses | stable since RM364 |
| authored coordinate contradicts the table | ⚠️ warning | ❌ refuses | stable since RM364 |
| one rsID placed two ways by two subjects of the table (RM374) | ⚠️ warning | ❌ refuses | stable — and that is the problem: the disagreement survives every lap |
Four things the table encodes that are worth saying out loud:
withdrawn is the only row that refuses in best_effort, and both sides ask it. A merged or
absent rsID leaves the annotation intact — the module is dated, or the label is unserved. A withdrawn
one is dbSNP repudiating the variant, so the annotation may be describing something that does not
exist, and carrying it under best_effort would publish a claim its own source has retracted. It is
never produced by the automated check (a retraction is indistinguishable from a never-assigned id
through the live API), so it fires only where a curator recorded it deliberately. validate refuses
it too, in both modes — it reads the injected table's own column and no resolved row, so the
compile-only exemption does not cover it (RM207); the same is true of ambiguous under strict, and,
since RM367, of every row of this table, because the pre-flight runs the compile's resolution step.
Instability always means the table cannot be reproduced, never the bytes. artifact.digest is a
fixed point in every row above, including the unstable one — a module that compiles at all compiles to
the same bytes twice. What is lost is resolution_signature: a not_found sentinel has no coordinate,
so no parquet carries a row for it. Until RM364 three more rows lost it too: a dropped locus was not in
the artifact, and where an authored value contradicted the table the authored value won, so the table's
version was gone. Since RM364 every variant parquet carries the table's rows for its key
(resolution_loci), so those come back and the second lap reaches the same finding; strict still
refuses each, for the finding itself.
ambiguous is the one exception in the other direction. It is round-trip stable, because the
enricher writes a single row carrying the deterministic pick while the candidate list rides in
rsid_alternates, which is provenance and outside the fact set. Strict still refuses, not because
anything is lost but because the label is a pick among equals rather than a finding, and an
all-or-nothing artifact should not rest on one. (An earlier draft of this analysis concluded ambiguity
could not be stable — from a hand-written two-row fixture the enricher would never produce. The real
shape is one row.)
A contradiction is an instability, not a difference of opinion. An authored coordinate that
disagrees with the table used to be a warning and nothing more. It has to be stronger: the artifact
keeps what the author wrote, and an all-or-nothing artifact should not rest on a pair its own table
contradicts. Until RM364 the table's position was also lost on a reverse; it now rides resolution_loci
beside the authored value, so the next compile reaches the same finding, and strict refuses for the
contradiction alone (RM380 corrected the refusal sentence, which still gave the old reason).
The meta-table (0.8) — expansion, precedence and the two parquets¶
meta.csv (RM28, RM323, RM324, RM327's rows, RM330's sex, and drug) maps a member set to a finding and/or the
per-locus facts; the grammar and the model are in SCHEMAS.md § The meta-table.
The compiler's part is to expand every member into the rows it matches, so a consumer evaluates no
rule (P1), and to resolve which fact wins where two rows reach one row.
Which rows a member is matched against. The authored rows, after the symbolic-allele drop and
before resolution, on both sides: validate_spec and compile_module hand build_meta the same view,
so a member matches the same rows in the pre-flight and in the compile, and each finding is one
sentence the compile de-duplicates (@parity-by-check). Conditions are AND-ed and compared against the
value the model stores (a diplotype pair is canonicalized haplotype_a <= haplotype_b, a genotype
sorted); on a list column such as flags, = means contains. Inside a combination a member is
existential: any row it matches.
The one residual is an rsID resolution expands. A variants.csv match is recorded by its authored
key (variant_key, genotype), which is weights.parquet's key for every row resolution did not
expand. An rsID that resolves onto several loci is written to weights.parquet coordinate-keyed
(locus_count > 1), so a consumer reaches those rows from a meta key through their rsid column.
Precedence, per row reached and per column. Only one-member rows naming no trait contribute
facts: a combination's facts are about the combination and a per-trait row's are about its trait (a
separate statement by the key), so both stay on their own meta.parquet row. A per-trait row still
reaches the row (it is listed in meta_rows). A member naming both halves of a diplotype pair is matched
as the model stores the pair (haplotype_a <= haplotype_b); the authored cell is kept. Among the rows that state
a column (unstated is not different), the maximal ones under this order win:
- a row whose scope refines another's, on the same member set and trait, outranks it: a sex-scoped
row outranks the row with the same key and no
sex, for that sex; a drug-scoped row the row with nodrug, for that drug; and a row scoped on both outranks either. Amalerow and anazathioprinerow refine neither each other, so where both apply (a male on azathioprine) a disagreeing column is withheld; - a
diplotypesmember outranks anallele_functionmember reaching the same diplotype through one of its two haplotypes (an allele reaches every diplotype of its gene that contains it); - otherwise
member_rankdecides: a concrete row key (the conditions name every key column the row fills), then a locus (variant_keyalone), then a selector with more conditions, then fewer.
Equal ranks are incomparable. One distinct value among the winners fills the column; two or more
withhold it and name it in withheld.
What the compile writes. Both parquets whenever the module carries meta.csv, after the overlay in
ARTIFACT_PARQUETS; neither is a lead table, and meta.csv alone does not satisfy "a module must carry
at least one recognized table".
meta.parquet: one row per authored row, in authored order. Every authored cell verbatim (whatreversewrites back), the stampedmembers_key,match_count,matches(a list of{member, table, key}structs,keyin the table's_KEY_FIELDSorder),replaces(true on a row of two or more members stating a finding: a matching combination replaces the findings of exactly the rows inmatches, within the row's ownsexanddrugscope), andwithheld(direction, when a bare-locus member on a direction row fixes no allele).meta_resolved.parquet: one row per (row reached,sexscope,drugscope), withtable,key,sex,drug, the four fact columns after precedence,tradeoff,withheldandmeta_rows(the 0-basedmeta.parquetrows that apply in that scope, an outranked one included). The scopes of one row reached are every pair({none} ∪ the sexes stated) × ({none} ∪ the drugs stated)by the rows reaching it, skipping a pair no row applies in. The product rather than only the pairs some row states, so a consumer's lookup has one answer: per axis it takes its own value where the target has rows for it andnoneotherwise, and with amalerow and anazathioprinerow the(male, azathioprine)row exists and holds both. No row for the pair means no row applies.
tradeoff (RM327), computed and never authored. A nullable boolean, true or null and never
false. A per-trait judgement is a one-member row with trait_efo_id and a direction; the verdict is
taken per member (its members_key) and is true when that member has a protective row on one
trait and a risk row on another. It then lands on every row the member reaches: each genotype row of a
bare locus (the reference homozygote included, since the verdict is about the locus's shared
effect_allele), and each diplotype carrying a named allele_function allele. A row two members reach
is true when either is; two alleles' judgements are never pooled, so ε2 protective on one trait and ε4
risk on another leaves the ε2/ε4 row null. On one trait at one row, the precedence above runs first:
a judgement on that trait outranked by another (a sex- or drug-scoped row over the same key unscoped, a
diplotypes member over an allele it contains, a genotype over its locus) is dropped there, so a
diplotype row stating ε3/ε4 neutral on the trait ε4 is risk on clears ε4's trade-off at ε3/ε4
only. A mediator row counts (its direction is still a benefit or a harm);
neutral, unknown and contested are not signs, and a direction the compile withheld
(meta_member_allele_ambiguous) is unknown and adds none. The false arm is unreachable: RM327 makes
false mean every trait's judgement is stated and none opposes another, and a module states no closed
list of the traits an allele touches, so one sign alone is null. The name is refused on every authored
table with its own reason (vocab.MISPLACED_COLUMN_REASONS).
The manifest records every member's match count as meta_selectors ({row, selector, matches},
in authored order) and the module's tier_scale. reverse writes meta.csv back from meta.parquet
(the derived columns fall away) and tier_scale back from the manifest.
Refusals and findings. The grammar's refusals are the model's (SCHEMAS). Three need the module and
run in validate and compile alike: a tier in a module with no tier_scale (refused);
meta_selector_matches_nothing (a warning: almost always a typo); meta_member_allele_ambiguous (a
warning: a bare-locus member on a direction row whose variants.csv rows do not share one
effect_allele, so the direction is withheld); and meta_member_drug_disagrees (a warning, RM363: a
row scoped to a drug whose member reaches rows stating another drug; nothing is withheld). A member's
conditions, the (sex, drug) scopes and an allele's reach into its gene's diplotypes compare gene,
drug, doi and chrom through spelling.comparison_key, so a value written in another case is the
same value; meta_resolved.parquet names a scope by its first spelling in meta.csv row order. A duplicate key (A & B against B & A) is refused
through _TABLE_DUPE_KEYS like every keyed table's. The enricher's rsID currency pass asks dbSNP about
every rsID a member names, so one variant written under two keys is caught where NCBI is the oracle
(ENRICHER.md).
The model-organism table (0.8, RM335) — verbatim, and never a lead table¶
model_organisms.csv is annotation only (the model is in
SCHEMAS.md § The model-organism table).
The compiler's part is small and deliberately so:
- One loader on both sides.
validate_specandcompile_modulecallload_authored_table(the next section): the rows parse, a present file with no rows is refused, and a duplicate key (every column butlifespan_effectandconcordance) is refused through_TABLE_DUPE_KEYS. Nothing else reads the rows, so nothing else can refuse them. - Verbatim to
model_organisms.parquet, through the generic materializer, in authored order;reversewritesmodel_organisms.csvback from it through the generic writer. The rows are insidecontent_signatureand the CSV is inmanifest.inputs. - A citation site. The table is in
_CITING_TABLE_KINDS, so a cited row'spmidis read by the literature cross-check and keeps itsliterature.csvrow, and the enricher's literature pass looks it up. An uncited row draws nothing. - Not a table kind. It is registered beside the meta-table rather than in
_TABLE_KINDS, so it does not satisfy "a module must carry at least one recognized table", is not inLEAD_PARQUETS(the publish gate and discovery's is this a module rule), and its genes do not entermanifest.stats.genes. The reason is attables.MODEL_ORGANISMS_CSV: a mouse lifespan finding is nothing a genome is matched against.
The authored tables that are not table kinds (0.8, RM357) — one registry, one loader¶
overrides.csv, meta.csv and model_organisms.csv share a category: a human writes every row, so
the CSV is input (in manifest.inputs, inside content_signature); the rows are keyed; and none is a
lead table, so a directory carrying only one of them is not a module. tables.AUTHORED_NON_KIND_TABLES
registers them, each entry carrying its CSV name, its parquet(s) (the first holds every authored cell,
and is what reverse reads back), its row model and its own table-level row_checks (the overlay's
coherence check). Whether an entry cites is read off its model (pmid), never declared; that none is a
lead table, and that no gene in one is a gene the module's findings are about (manifest.stats.genes
reads variants.csv and the kinds), is membership itself.
Every list that names these tables is derived from the registry: _INPUT_FILES, ARTIFACT_PARQUETS
(appended last, in registry order, which is the manifest.artifact.files listing order),
table_bindings(), _TABLE_DUPE_KEYS, spec_tables' roster, _CITING_TABLE_KINDS,
_KNOWN_SPEC_FILES, the citing view validate_spec and compile_module hand the literature checks,
reverse, draft.DRAFTABLE (and through it describe/template/requirements/stub and the
enricher's identifier rosters), and the enricher's gene_metrics._NOT_THE_MODULES_GENES. The format
tier cannot import the compiler, so reference._AUTHORED_NON_KIND_MODELS is declared there and a test
asserts it equals the registry's models.
One loader, load_authored_table(spec_dir, csv_name), in this order: absent is ([], [], []); the
rows parse or they do not; a present file with no rows is refused; the duplicate key; the entry's
row_checks. Both entry points load all three once, through this loader. A refused table keeps its
rows for the other checks. Both entry points refuse before a parquet is built, but the pre-flight's
citing view still reads them, so a literature.csv row cited only by a refused meta.csv is not also
reported as uncited (before 0.8's RM357, meta.csv dropped every row on a duplicate key, and the author
was sent to literature.csv). On a duplicate key every row parsed and all are kept. On a parse error
the set is incomplete, and the rows that parsed are kept only for a table that cites: the malformed
row's own citation can still read as uncited, and the overlay is never applied half-parsed. A refused
meta.csv is not expanded: its findings are the refusal. load_overlay stays public (the enricher calls
it) and is this loader on overrides.csv.
Reverse¶
reverse_module reads the parquets for every row, and the artifact's manifest.json for every
module_spec.yaml field the manifest carries (RM241). It emits into output_dir: module_spec.yaml
(always), the README, logo, provenance.json and logs the compile shipped (when the manifest records
them), variants.csv + resolution.csv (when weights.parquet exists; resolution.csv gated on
write_resolution=True, and not written for a module whose compile read none, RM364), studies.csv (when present), one CSV per present
table kind, and one CSV per derived-fact sidecar whose parquet is present (frequencies.csv,
gene_metrics.csv, literature.csv, gene_validity.csv, clinical_assertions.csv, sources.csv),
plus each authored table that is not a kind (overrides.csv since 0.7, RM124; meta.csv and
model_organisms.csv since 0.8) when the first of its parquets is present (AUTHORED_NON_KIND_TABLES).
- Preserved (round-trip-critical, Principle 7): every authored
VariantRow/StudyRow/table value; genotype phase (thephasedbit re-emitsA|Gvs sortedA/G); tri-state bools;priorityverbatim; poly-effect annotations keyed on(variant_key, genotype, conclusion, negatives)(genotypesince 0.6, RM80 / S29). - The authored shape is restored, not guessed:
authored_identsays which identity columns the author wrote, and reverse emits exactly those — an rsid-only row comes back rsid-only, a position-only row comes back position-only, a pair comes back as a pair. An expanded one-to-many rsid collapses back to the single row it was authored as, rather than one row per locus. See § Resolution for why the stored key alone cannot answer this and what it cost when reverse tried. resolution.csvemission carries the resolved facts back: oneResolutionRowper distinct positioned fact, keyed on the authored key (so an expansion's N loci are joinable to the one row they came from) withlocus_indexcounting within that key, andsource="reversed",status="resolved". Soreverse → compilereproduces the identicalartifact.digestwith no reference and no network — hardening Principle 7's round-trip from reference-dependent to self-contained. It rebuilds from every table the enricher asks a coordinate for (reverse.RESOLUTION_SOURCE_TABLES, asserted equal toenrich.subjects.SUBJECT_TABLES):weights.parquet, the three positional parquets, and since 0.8studies.parquet.proxy_loci, the compiler-filled list of a study's LD proxy's resolved loci (RM331). A proxy is usually not a module variant, so before that column the reversed spec lost the proxy's row,resolution_signaturemoved, and the proxy allele check went silent on the second lap. Proxy rows come last and never replace a row another table emitted under the same key.proxy_lociis read from the post-overlayresolution.csv, and not at all under--no-resolve, sostudies.parquet(likeweights.parquet's filled coordinates) moves with the injected table whilecontent_signaturedoes not.- A variant row is rebuilt from what the table said, not from its own cells (RM364, 0.8).
weights.parquetand every positional parquet carryresolution_loci, the same struct list asproxy_loci: the loci the post-overlay table files under the row's authored key, on the module's build, kept wherever the row's own cells would not rebuild them exactly and null on a cleanly resolved row (parquets.withhold_reproduced_loci), so a clean module's parquets match the deprecatedensembl_cachepath's byte for byte. "Exactly" is on every column the cell rebuild writes,parquets.CELL_FACT_FIELDS, derived from_CELL_REBUILTand so includingvrs_id/vrs_spec: a locus stating avrs_specwith no id, or an id on a row with noref(which the cell lookup skips), is kept, since withholding it wrote the value back empty (RM389). Reverse writes the loci back, and rebuilds a null row from its cells as before. A row the fill leaves alone is the reason: one whose authored coordinate the table contradicts, a one-to-many none of whose loci was taken, a locus dropped because it cannot host the genotype, and every row of a module on a build the fill skips. Before the column, reverse rebuilt such a row's table rows from its own cells, which lost the rows the fill did not take and wrote the authored coordinate back as a resolution fact; the second lap's contradiction warning went silent andresolution_signaturemoved, andreference_examples/cpic_gene_pairshad dropped its NUDT15rs1457579126row to stay green. Five cases of the round-trip matrix thatstrictrefuses are now stable too. A parquet compiled before the column falls back to the cell rebuild. Still lost: a row about a variant the module does not carry (no parquet row to ride on,table_says_more) and anot_foundsentinel (no locus;strictrefuses). - No table comes back for a module that read none (RM364). When the manifest records no
resolution_signatureand no parquet row carries an identity cell itsauthored_identdoes not name, reverse writes noresolution.csv. It used to rebuild one from the authored coordinates, and the second lap then published a signature, VRS coverage and expansion counts the first did not (grch37_build,mt_heteroplasmy,cyp2d6_structural). A filled cell with no signature (an older compile, the deprecatedensembl_cachepath) still gets the table, since the recompile needs it. - The derived sidecars come back POST-overlay, and
overrides.csvcomes back beside them (0.7, RM124). The overlay is applied at compile, so what reverse reads out of the parquets is already the corrected table — and it re-emits the overlay too, which means the overlay applies twice and the fixed point is checked by test rather than assumed. It holds because all three operations are idempotent set operations: an update to a value already present, an insert of a row already keyed(subject, member), and a suppress of a row already absent are each a no-op. The alternative — emitting the pre-overlay table so the apply happens exactly once — would need the overlay to record the value it replaced, which is a derived cell inside an authored table and rots the moment the source moves.overrides.csvis written at the spec root under its one legal name, not throughsidecar_write_path: it is authored, likevariants.csv. - Provenance is deliberately dropped on the way back:
source/status/fetched_atare reset andrsid_alternates/rsid_current/rsid_statusare not written, because those columns are kept out of the fact set precisely so they never enter the artifact — there is nothing for reverse to read. Recover them by re-running the enricher. vrs_idandvrs_speccome back (RM270, 0.8). Every variant parquet carries them — see The VRS ids on the variant parquets under § Resolution — so reverse writes each positioned row's ids intoresolution.csv, holes kept, and the recompile carries the same ids. Before 0.8 no parquet held one, so a round trip dropped every id and the second lap'svrs_alleles_identifiedread 0 (RM241's residual).- Derived-fact sidecars round-trip through the same generic writer, minus the columns that are
recomputed rather than stored:
allele_frequencyis derived on write and is not aFrequencyRowfield, so it falls away by construction rather than by a special case, and re-deriving it on the next compile reproduces the identical parquet. - Recovered from
manifest.json(RM241): every spec field the artifact carries.REVERSE_RECONSTRUCTEDincompiler/reverse.pylists them — themodule:block's display fields (manifest.display),version(identity, pre-coercion spelling first, RM103),genome_build, the defaults'curator/method,license,weighting,authorship,authority_precedenceandpanel— andREVERSE_EXCLUDEDnames the rest with a reason, under a test that walks the spec models and asserts the two sets cover every field. The README, logo,provenance.jsonand every shipped log (reverse.shipped_files, RM364 for the last two) are copied byte for byte after a hash check against the manifest's record; a file that disagrees, or a recorded name that leaves the module directory, is not copied, and one the compiler would not discover on its own is named. Every value is taken in one order: an explicit argument, then the manifest, then the bare-directory fallback.genome_buildis the value that made this list necessary rather than cosmetic: it decides the identity key, so a GRCh37 module reversed as GRCh38 recompiled withga4gh:VA.…keys minted for GRCh37 coordinates, movingartifact.digestand asserting an allele at a base the module never named (reference_examples/grch37_build/). The rest could not move a digest, and that is why they went unnoticed: the round-trip gate comparedartifact.digestandcontent_signature, both of which leave the manifest out by design, so it could not fail on any of them. It now also compares the manifest across laps, field by field:REVERSE_MANIFEST_RECONSTRUCTED/REVERSE_MANIFEST_EXCLUDED(RM364) put everyModuleManifestandCompilationfield on one side, under a test that walks both models, and every round-trip gate comparesreconstructed_projection(the reconstructed fields, as JSON) lap against lap —compilation.resolution_signatureamong them, from the first lap, on every reference example. The exclusions are the attestation, what a registry stamps, the byte hashes of re-emitted files,compiled_at, the warnings (the first lap's include the attestation and the time-varying resolution provenance reverse leaves to the enricher),dropped_rowsandresolution_sources. - Normalized, only for a bare parquet directory with no manifest: title/report_title fall back to
name-derived defaults, description to
Annotation module: <name>, icon/color todatabase/#6435c9, curator/method to the most-common column value,genome_buildtoGRCh38.defaults.priorityis never written: the manifest carries no default priority, andpriorityis re-emitted verbatim per row instead. - Lost (not in the artifact to be read): the verification attestation
(RM45) —
verification.jsonis not in the artifact, so there is nothing for reverse to read and inventing one would mint a claim nobody put; a reversed module carries nomanifest.verification, which is the honest says nothing, and re-attesting means re-running the checks. The RM73 closure rides that document and is lost with it, so a reversed spec is open and warns until a human closes it — deliberate, since reverse holds no key and no standing to declare someone else's authoring finished. That asymmetry costs nothing while the finding is a warning (warnings feed no digest and no signature, so the fixed point is untouched), and it is precisely what blocks promoting the finding to a refusal at 1.0: under a gate, step 3 ofcompile → reverse → compilewould refuse on every module. The three candidate answers are recorded in ROADMAP_1_0.md § RM73 (gate half). A consumer needing these readsmanifest.json(preserved verbatim by the forward compile). This list used to holdauthorship,panel,logoandreadmetoo, on the test of whether losing a value could change a parquet byte. That test was the wrong one: P7 asks whether an authored value survives, and those four were in the artifact all along (RM241).provenanceand the logs left it in 0.8 (RM364): both were copied into the artifact and hashed into the manifest by every compile.
Output artifact & hashing¶
ARTIFACT_PARQUETS is the artifact.files listing order — the tuple's own order, which
build_artifact walks, skipping absent files so a module lists exactly the parquets it has:
weights annotations studies ← the SNP core
activity_phenotype copynumbers repeat_alleles heteroplasmy
haplotypes allele_function diplotypes pgs pharm_variants ← the table kinds
frequencies gene_metrics literature gene_validity
clinical_assertions gwas_effects
clin_sig_concordance clin_sig_authority_calls expression_effects
locus_to_gene colocalisations consequences ld_pairs ← the derived-fact sidecars (the Open Targets three and ld_pairs: 0.8)
sources ← the licence table
overrides ← the authored overlay (0.7)
meta meta_resolved ← the meta-table (0.8); not lead tables
model_organisms ← the model-organism table (0.8, RM335); not a lead table
It is not digest order, and this section said it was. artifact_digest sorts the listing by name
before hashing (integrity.py), so the tuple's position is invisible to the digest and a name is what
places an entry — overrides.parquet hashes between literature.parquet and pgs.parquet however
late it sits in the tuple. What the position does govern is the order a consumer iterating
artifact.files sees. Appending is still the right move for a new parquet, but the reason is the
listing, not the hash; what keeps an already-published digest still is that the module has no such
file at all.
Thirty names. LEAD_PARQUETS is the ones carrying a module's own annotation rows — weights plus one
per table kind — which is the publisher's is this a module rule and what discovery probes.
The per-parquet column lists are deliberately not reproduced here. They are derivable from the
models, and a hand-kept column list is precisely how SOURCES_FIELDNAMES lost a column
(@fieldnames-from-model); use just-dna-compiler describe or read the model. A point-in-time listing,
including which columns the compiler stamps rather than reads, is in
audit/COMPILER_FROM_CODE.md § 5 — dated evidence, not a maintained list.
ARTIFACT_PARQUETS(feedartifact.digest;_OUTPUT_FILESuntil 0.6): the SNP core, one parquet per table kind, one per derived-fact sidecar, the licence table, the overlay and the 0.8 tables, each when present — the diagram above is the tuple, in its order, undertest_counted_prose.py, and this sentence is not a second inventory of it. The sidecars enter the digest because a module carrying frequency data genuinely is different content — but adding one leaves the SNP core's bytes untouched (an explicit test). It is public because the publisher tier has to agree with it:just_dna_enricher.uploadderives its allow-patterns from this tuple, after a hand-kept copy covering three of the sixteen names silently dropped the rest at upload (S35/RM89).LEAD_PARQUETSbeside it names the ten that carry a module's own annotation rows —weightsplus the nine 0.4 families — which is the publisher's "is this a module" rule and what discovery probes._INPUT_FILES(feedmanifest.inputs, raw-bytes hashed):module_spec.yaml+variants.csv+studies.csv+ the 9 table-kind CSVs +overrides.csv(0.7 — every row of it is written by a human, which is what puts it here) +meta.csvandmodel_organisms.csv(0.8) — the authored surface, and the reason only the other files gained a second legal name and location in 0.6.resolution.csvis deliberately NOT here (nor inARTIFACT_PARQUETS) — it is a multi-producer artifact hashed only by the normalizedresolution_signature(a raw-bytes hash would be unstable across enricher/human/reverse producers). Every derived-fact sidecar is out for exactly the same reason, each hashed by its own*_signature(the derived roster is_DERIVED_FILES; read it rather than a sentence here).provenance.jsonis likewise out of the digest.- The derived-fact sidecars are deliberately NOT
_TABLE_KINDS. Those are authored DSL tables withAuthoredModelsemantics, the reserved-namespace guard, duplicate-key checks and raw-byte input hashing. A machine-produced reference-fact table is a third category — injected, fact-hashed, human-overridable — and folding it in would blur the line the 0.5 rework drew. - Manifest
Compilationfields the compiler populates:compile_success,compiled_by,compiler_version,toolchain(0.8, RM342:{"polars": pl.__version__, "python": platform.python_version()}of the running process),ensembl_reference,compiled_at,warnings, and the 0.5 resolution provenance —resolution_mode(policy),fully_resolved(outcome — orthogonal axis, P5),resolution_subjects(0.6 — the denominator that flag covers),resolution_signature,resolution_sources. All out ofartifact.digest. The trust rule isresolution_subjects > 0 and (resolution_mode == "strict" or fully_resolved); without the first clause it is vacuously satisfied by a module that resolves nothing. - Manifest
frequency/gene_metricsblocks (0.5):signature,sources,datasets,row_count, pluspopulations/variant_counton the former andgeneson the latter. Separate blocks rather than extra fields onCompilation/Resolution, which are about rsID↔coordinate resolution only. Out ofartifact.digest.datasetsis the field a consumer reproducing an ACMG BA1/BS1 filter reads to know which release it is filtering against.
The three hashes and how they compose into (content_signature, resolution_signature, compiler_version)
⟹ artifact.digest are documented in SCHEMAS.md § identity & integrity.
Deterministic ordering — what is preserved, what is normalized¶
Parquet bytes depend on row order, so ordering is part of artifact.digest rather than a nicety. The
asymmetry below is intended: rows are preserved, presentation is normalized.
Preserved.
- Authored row order, through compile → reverse → recompile.
- Expansion order within a one-to-many rsID.
_sorted_locisorts on(locus_index, chrom or "", start or 0, ref or ""), which matches the deprecated DuckDB path'sORDER BY id, chrom, start, ref— the two produce byte-identical parquet, which is what let that path be retired without moving a digest. _symbolic_findingssorts on(table, reason, index, column), so the messages built from it are byte-stable. They reachmanifest.compilation.warnings, which is artifact-visible.table_citations(and its narrower siblingbinning_citations) returns first-occurrence order rather than sorted order, because it feeds emission order.Frequency.populationsis in canonical order (population_sort_key,globalfirst). Every other manifest facet list is sorted — this one is the exception, deliberately._write_resolution_csvemits weights first and never re-emits a key from the positional pass, sovariants.csv'saltsalways wins.- The meta-table's two parquets (0.8).
meta.parquetis in authored row order, and each row'smatcheslist is in member order, then in the matched table's authored row order.meta_resolved.parquetis sorted by member table (META_MEMBER_TABLESorder), then the matched row's authored position, then thesexscope (none,female,male), then thedrugscope (none, then by name); itsmeta_rowsare ascending. No set or dict iteration reaches either file, andtest_meta_compile.pycompiles twice and compares the bytes. model_organisms.parquet(0.8, RM335) is in authored row order: the generic materializer writes the CSV's rows as they stand, andtest_model_organisms_compile.pycompiles twice and compares the bytes.
Normalized, not preserved. Column order and cell formatting in every reversed CSV; the
curator/method blank-vs-explicit split; unphased genotype allele order.
Deterministic tie-breaks where the library gives none. _module_name_from_parquets uses min()
over unique(), and _most_common uses min() over mode(), because polars orders neither. This is
the concrete form of the standing rule: never derive an emitted row or a manifest field from
set/dict iteration or from mode()/unique() without an explicit sort or tie-break.
Warning texts a consumer keys on¶
Since 0.7 the channel also carries codes and an actionability split (RM131)¶
manifest.compilation.warnings still carries every finding, with its exact text — nothing that greps
a phrase broke, and nothing here supersedes the fragments below. Two derived fields ship beside it:
| Field | What it is |
|---|---|
compilation.warnings |
the complete list, unchanged; the same sentences in the same order |
compilation.carried |
the subset no edit to the spec directory can clear — a limit of this tier or a fact of a source. Subtract it from warnings for the set an author still owes work on |
compilation.warnings_summary |
{code: count}, keys from vocab.VALID_WARNING_CODES, values summing to len(warnings) so the digest accounts for the whole channel |
The same three fields are on ValidationResult, CompilationResult and ClosureResult, filled on
every path including a failed compile. manifest.json sits outside artifact.digest (a Merkle root
over the parquet FileEntry list), so adding them moved no hash — and both are listed in
release_records.EXCLUDED_MANIFEST_FIELDS, routed to the warnings axis, because they move exactly
when warnings moves.
A code names the finding, never the function that builds it, and one code carries one remediation: two sentences cleared by the same edit share a code and say which cell (the weight-sign pair, the orphan fact tables), two cleared differently do not. The set is published, so it is permanent within the major — additions are minor-legal, re-spellings are not.
Carried findings — the ones an author cannot clear (vocab.CARRIED_WARNING_CODES), and the reason
each is on this side:
| Code | Why no authored edit clears it |
|---|---|
non_grch38_variant_keys |
VRS allele identity is GRCh38-only, so a non-GRCh38 module is coordinate-keyed |
resolution_skipped_cross_build |
resolution and the positional fill are GRCh38-bound (RM15) |
contig_ploidy_undecidable |
the build carries no pseudoautosomal table here, so ploidy cannot be decided |
rsid_expanded_to_multiple_loci |
the source maps one rsID onto several loci; that is a fact about dbSNP |
locus_hosting_undecidable |
deciding needs the reference sequence, which this tier never fetches (P2) |
vrs_id_unverifiable |
the recomputation is beyond this tier — _carried_vrs_warnings is the shape the split generalises |
vrs_coverage_incomplete |
the alleles a VA does not reach; minting more is not an authored edit |
measurement_spans_bins |
the format has no reading for an interval that straddles a boundary (RM56) |
verification_findings_recorded |
a disagreement with an archive, where the archive is the stale side often enough that nothing is owed |
gene_validity_superseded |
a curating body re-curated its own claim; the only edit available is deleting a true record |
gene_validity_currency_undecidable |
the source published several curations of one claim and not enough to order them |
Every published code, by the surface it comes from. Everything not in the table above is
actionable, which is vocab.ACTIONABLE_WARNING_CODES, derived by subtraction:
- spec directory /
module_spec.yaml—table_file_misplaced,table_file_near_miss,sidecar_spelling_deprecated,module_version_coerced,panel_block_deprecated,log_file_not_shipped - coordinates and the build —
non_grch38_variant_keys,contig_ploidy_undecidable,contig_ploidy_mismatch - resolution —
resolution_disabled,resolution_not_injected,resolution_skipped_cross_build,positional_identity_contradicted,positional_rows_unjoinable,rsid_unresolved,rsid_without_resolution_label,rsid_expanded_to_multiple_loci,rsid_ambiguous,rsid_coordinate_disagrees,rsid_placements_disagree,locus_hosting_undecidable,locus_cannot_host_genotype,rsid_no_hosting_locus - VRS —
vrs_id_unverifiable,vrs_coverage_incomplete,vrs_version_differs variants.csvcoherence —weight_sign_disagrees_with_effect,genotype_allele_not_at_locus,effect_allele_not_at_locus,genotype_coverage_gap,conclusion_names_other_genotype,qual_floor_on_reference_evidence,missing_allele_marker_in_alts,vcf_pointer_key_collision,vcf_pointer_unselected_element,composite_gene_cell,symbolic_allele_unusable- binning tables —
bin_tiling_inferred,bin_tiling_contradicted,bin_coverage_gap,bins_ungrounded,measure_field_fractional,measurement_spans_bins,deprecated_bin_modifier - PGx tables —
star_allele_undefined,diplotype_definitions_identical,diplotype_phase_ambiguous - studies and literature —
study_variant_orphan,duplicate_study_citation,p_value_encodings_disagree,study_effect_allele_not_at_locus,phenotype_class_unrecognized,paired_field_half_stated,citation_not_in_pubmed,literature_row_uncited,quote_counter_stale,quoted_article_license_restrictive - sources and licensing —
source_row_unused,source_terms_unrecorded,declared_license_disagrees - the injected fact tables —
derived_row_orphan,fact_rows_contradict,faf95_exceeds_frequency,oe_lof_outside_interval,oe_lof_disagrees_with_counts,clin_sig_contradicts_frequency,clin_sig_concordance_contested,gene_validity_superseded,gene_validity_currency_undecidable - the overlay —
overlay_update_unmatched,overlay_update_target_unreachable,overlay_answer_vindicated,overlay_targets_missing_table,overlay_rows_suppressed - the meta-table (0.8) —
meta_selector_matches_nothing,meta_member_allele_ambiguous,meta_member_drug_disagrees - verification and closure —
verification_two_copies,verification_unreadable,verification_stale,verification_findings_recorded,module_not_closed,closure_discarded_unreadable_record
overlay_answer_vindicated (RM117) is the one finding in this catalogue that reports good news,
and it exists because the alternative was actively misleading. clin_sig_concordance.csv holds
contested subjects only and is rewritten whole, so a subject leaving it means the authorities stopped
disagreeing — which is exactly how an author learns the archive caught up with them. An overrides.csv
row answering that conflict then reaches nothing, and without this code it drew the generic the
subject may be mistyped line, put to an author whose judgement had just been confirmed. It says
nothing about who was right about the biology: the authorities agreed, and the overlay row is now
unnecessary.
Which tables can say it is a field on each table, decided once for every overridable table (RM290).
OverlayTarget.vindication is sole, withheld or none, with its reason beside it, and
overrides.VINDICATING_OVERLAY_TABLES is read off it. Only the concordance record is sole.
resolution.csv is withheld, because a re-derivation renumbers locus_index and the author's locus
can return under another index. The merged tables are none: their keys are stable, so a source
adopting the author's value leaves the row in place and the update keeps matching. Absence there is
never agreement. Every other table keeps the generic finding it had.
The two gene_validity_* codes are new in 0.7 (RM108) and are the first fact-table findings the
pre-flight also computes, so validate reports them exactly as compile does. They stay apart on
purpose: gene_validity_superseded says a later curation replaced an earlier one and the manifest now
publishes the later verdict, while gene_validity_currency_undecidable says several curations exist
and nothing orders them — a tie on classification_date, or a row stating none — so every
classification in that group is still published. One number meaning both would tell a reader the
archive moved on when it had simply not said enough.
overlay_rows_suppressed is new in 0.7 and is the one finding here that reports a decision rather
than a defect: a suppress removes a row and leaves no trace of the removal in the build product, so
the overlay says so — one line per reason, with a count, and counted over the overlay's own rows
rather than over the rows removed. That is what keeps it stable across compile → reverse → compile,
where by the second lap the derived table is already post-overlay and the suppress matches nothing.
overrides.csv: {count} suppress override(s) remove {table} row(s) from the compiled artifact,
where nothing else records the removal: {reason}
test_warning_codes.py walks the emission sites and asserts an equality against the vocabulary in
both directions, so a code with no emitter and an emitter with no code both fail; it also asserts this
document lists every member, because a catalogue missing one sends a reader hunting for it.
Two consequences worth knowing before you build against this, both recorded rather than left to be discovered:
- A summary is either empty or complete, never partial.
classifywithholds — an emptycarriedand an emptywarnings_summary— when a channel carries no classified findings at all, which is what a caller passing plain prose intoCompilationResult(warnings=[...])gets; that call has been legal since 0.6 and Principle 3 keeps it legal. It refuses only a part-classified channel, which no legitimate caller can produce. So read an empty summary as this compile did not classify the channel, never as there is nothing to report, and read a non-empty one as accounting for the whole ofwarnings. - The code vocabulary is closed, so an older reader refuses a newer manifest.
warnings_summaryvalidates its keys, andread_manifestraises on a code added after thejust-dna-formatyou have pinned. That is the standing cost of every closed vocabulary on a published field here (VerificationRecord.checkis the shipped precedent) and Principle 6 takes it deliberately, but "additive" describes the writer: a consumer reading manifests from newer compilers upgrades the schema package alongside them.
carried holds full message text, which nearly doubles the channel — measured at 1.84× across the
reference corpus, 1.96× on pathogenic_clinvar. That is the shape the item decided (a list beside,
so a consumer subtracts) rather than an oversight, and the cheaper encodings are weighed in
ROADMAP.md § RM138.
The phrases¶
A warning's text is an API. The manifest carries the prose and, for anything published before the structured field existed, no other handle — so a downstream consumer greps it. These fragments are named constants for exactly that reason:
UNJOINABLE_PHRASE = "have no chrom+start"
QUAL_REFERENCE_EVIDENCE_PHRASE = "a QUAL floor cannot be read on the evidence this row is decided from"
MISSING_ALLELE_PHRASE = "is VCF's MISSING marker, not an allele"
UNCLOSED_PHRASE = "records no closure"
LOG_NOT_SHIPPED_PHRASE = "does not ship with the module"
PLACEMENTS_DISAGREE_PHRASE = "two placement sets for this one rsID" # resolution.py, RM374
UNDECIDED_PHRASE = "could not decide whether either is the same event" # resolution.py, RM449
Retired phrases. A consumer may still grep manifests compiled before the retirement, so the text stays listed here after nothing emits it.
QUAL means the opposite thing on the record this row is read from— codequality_floor_inverted(RM57), retired in 0.8 by RM313 because it stated VCF §1.6.1 item 6 backwards: QUAL is the confidence in whatever ALT asserts, so a high QUAL where ALT is.is a confident reference call, never "probably variant". Its remedy (GQ) was ungrounded too, since GQ is conditioned on the site being variant. The code stays inVALID_WARNING_CODES(removing a member is major) and is declared invocab.RETIRED_WARNING_CODES. Its replacement isqual_floor_on_reference_evidence.
UNJOINABLE_PHRASE has a named external consumer: just-dna-registry 0.11.3 pins
UNJOINABLE_MARKER = "have no chrom+start" in its facet builder. The structured replacement
(manifest.compilation.positional_rows / positional_rows_placed) shipped in 0.6 and the phrase is
still not retired, because artifacts published under 0.5 carry neither field. Retiring it is a
decision about the published corpus, not about this code.
The positional-joinability sentence, as its format string — the counts and both trailing clauses are computed per table, so no example is invented here:
{csv_name}: {unplaced} of {rows} row(s) have no chrom+start, so this table joins by rsID only —
a VCF whose ID column is empty matches none of them. {detail}.{partial_note}
{detail} is one of exactly three sentences, and the distinction between them is the whole point —
not consulted, consulted and found nothing, and consulted and refused to guess are three
different situations a consumer must not collapse:
the resolution table was not consulted for this table — see the skip reported above(orresolution.csv names N of them and was not consulted for this table — …);no resolution.csv row places them — run `just-dna-enricher enrich` first;resolution.csv names N of them, but at more than one locus or at one the row's own allele contradicts, so the compiler leaves them unplaced rather than picking.
The VRS coverage headline, as its format string, followed by one indented {count} allele(s):
{reason} line per gap reason, sorted by descending count then reason:
VRS allele identity covers {identified}/{alleles} allele(s) in resolution.csv ({pct}) —
{missing} carry no ga4gh:VA. id. Anything keying on the VA sees only the covered fraction.
The no-ALT reason line (no ALT recorded, and a VRS allele id names exactly one allele) gains a pointer
when the spec's attested verification.json carries an rsid_coordinate_agreement record saying the
injected Ensembl snapshot has no record for some authored rsIDs (RM340). Those rows keep an empty
ref/alts under source="authored" (RM251), and a newer snapshot may fill them. The pointer is appended,
so the phrase above still leads the line: — verification.json's `rsid_coordinate_agreement` record names
authored rsIDs the injected Ensembl snapshot carries no record for, and a newer snapshot may fill their
ALT. The compiler recognises the record by compiler.SNAPSHOT_UNPLACED_PHRASE, the sentence the enricher
writes into the record's detail, so the two cannot drift apart.
Its twin, for ids that are present and that this tier cannot recompute, is grouped the same way
since S67 — one line per reason, descending count then reason, three variant_keys named
and the rest counted:
{count} allele(s): vrs_id could not be verified — {reason}; carried unverified ({a}, {b}, {c}, and
{n} more).
The two halves used to disagree about shape, and which half an allele landed in was decided by
whether the enricher happened to mint an id for it — nothing else. So _vrs_coverage aggregated the
alleles with no id while _verify_vrs_ids emitted one line per id present, and warning noise
ran inversely to how well-resolved a module was: 80 of one 101-row module's 85 warnings came from
here, with the three findings its author could act on at positions 83, 84 and 85, while a 57,595-row
module with nothing minted was quiet. _BLAME_ROW findings are not grouped — they are errors,
they are rare, and each names a row that contradicts itself, which is the one thing a per-reason line
would take away.
Three more sentences worth quoting exactly, because each states a consequence rather than a status: the
closure reminder (This module records no closure: … Compiling without one is a warning today;
requiring it is filed for 1.0 (RM73).), the symbolic-allele drop (Those row(s) are DROPPED from the
compiled artifact — reverse will not re-emit them — and --strict refuses instead.), and --no-resolve
with an injected table present, which spells out that the flag names Ensembl but is the master switch
and that there is no "do not reach the network" flag because the compiler never does.
Phrases added since 2026-08-28, each carrying its code:
derived_row_orphan (the concordance arm): {table} records {n} subject(s) no variant in this module
carries: {orphans}. The record is rebuilt whole on every run, so this means variants.csv was narrowed
since the comparison last ran — re-run it rather than editing the table.
declared_license_disagrees: module declares license {declared} and {standing}. Not adjudicated here — a
compatible pair is legitimate, an incompatible one is a real problem, and only a human can tell which.
A declaration matching some but not all of them is the ordinary mixed-licence case, where the most
restrictive term binds the whole artifact.
gene_validity_superseded: gene_validity.csv carries a later curation for {n} gene-disease claim(s), so
an earlier row is superseded and kept: {groups}. Nothing is deleted and nothing is wrong — the newest
classification_date is read as current, both rows stay so the drift is visible, and
manifest.gene_validity.classifications publishes the current one. A curating body re-curating is not
an error in your module.
conclusion_names_other_genotype (RM308, 0.8), one sentence per reason — rows naming each other, and a
row naming a neighbour that does not name it back:
{n} variants.csv row(s) carry a conclusion that names another genotype at the locus and never the
row's own, and the rows name each other, so two conclusions look swapped: e.g. {rsid} {genotype} names
{named}; …. Check which text belongs to which genotype; nothing here is changed.
{n} variants.csv row(s) carry a conclusion that names another genotype at the locus and never the
row's own, as if copied from that row: e.g. …. Check that the text belongs to this genotype; nothing
here is changed.
fact_rows_contradict (RM275, 0.8): {table} carries {n} key(s) under which two rows state different
values: {key} ({fields}); … (+{m} more). The enricher writes this table one row per key, so a consumer
reading it gets whichever row comes first. Delete the row that is wrong, or delete the table and re-run
the pass that writes it. Refused from 1.0.
gene_validity_currency_undecidable: gene_validity.csv carries several curations for {n} gene-disease
claim(s) and nothing orders them: {groups}. Either two rows share a classification_date or one states
none, so no row is called current and none superseded — every classification in those groups is
published, which is the honest answer rather than a winner picked from an identifier. Withheld
deliberately, not skipped.
derived_row_orphan (the resolution arm, RM422, 0.8.3): resolution.csv places {n} identity(ies) no table
in this module names: {orphans}. Delete those rows: `enrich` keeps a row it is not asked about, so no
run removes them.
The overlay's three warnings (0.7, RM124; the third split out on 2026-08-31), and there are only three on purpose. overrides.csv
applies before any check reads a derived row, so what a check reports is what the module asserts. No
operation reports its own no-op — reverse_module emits the post-overlay table plus the overlay, so
on a recompile update-already-equal, insert-already-present and suppress-already-absent are all three
true of a healthy module, and reporting any of them would make a module and its own round trip
disagree on manifest.compilation.warnings. What is left is the pair an overlay operation cannot
manufacture for itself. The unmatched-update case splits in two for the tables whose rows the compiler
can drop before the parquet (literature.csv loses its uncited rows, resolution.csv has no parquet at
all): an update whose subject is still cited or positioned names a table that is short and asks for
the enrichment pass to be re-run, while one whose subject no artifact of this module can carry is
unreachable — and unreachable fires matched-or-not, because reachability is a property of the
module and answers the same on both laps, where "did it match" is exactly the quantity a reverse moves
(@lap-stable-means-a-property-of-the-module; an earlier cut classified only the unmatched set and
reported on lap 2 and not on lap 1). Every other table keeps the three-readings message.
overrides.csv: {n} update override(s) name a row {table} does not carry: {subjects}. Three readings and
nothing here separates them — the subject/member may be mistyped, the source may have stopped
publishing the row the correction was about, or the compiler dropped the row before the parquet so a
reversed module cannot carry it. Neither an insert nor a suppress reports this: an insert creates the
row and a suppress is satisfied by its absence.
overrides.csv: {n} update override(s) name a row {table} does not carry, though this module could carry
it: {subjects}. The subject is cited or positioned, so the table is short rather than the correction
wrong — re-run the enrichment pass that writes {table}. Neither an insert nor a suppress reports this:
an insert creates the row and a suppress is satisfied by its absence.
overrides.csv: {n} update override(s) name a {table} row no artifact of this module can carry:
{subjects}. Two readings and nothing here separates them — the subject may be mistyped, or the
correction may be aimed at a row the compiler drops before the parquet (an uncited citation, an
unresolved locus), in which case the correction is fine and simply has nothing to reach. Reported the
same way whether or not the row is present today, so a module and its own round trip agree.
overrides.csv corrects {tables}, which this module does not carry. An overlay lies on top of a derived
table and never creates one, so those rows change nothing. Run the pass that writes the table, or drop
the override rows.
Both warn in both modes and both are aggregated into one sentence rather than one per row. The
consequence a consumer should know is the one no message can carry: a suppress with a typo'd
subject does nothing, forever, and cannot warn — check a suppression by reading the compiled table.
The uncited-literature sentence names its sites, so it has been reworded twice as sites were added —
no study or bin in 0.6, no study, bin or pharm row in 0.7. The code literature_row_uncited is
the stable handle; the phrase is pinned by the suite, which is what makes each rewording deliberate.
The concordance record's one warning (0.7, RM130). clin_sig_concordance.csv names the subjects
where the module's clinical call and an annotation authority's disagree, or where two authorities
disagree with each other, and this is the sentence that says so at validate and at compile:
clin_sig_concordance.csv records {n} contested subject(s): {opposed} of them opposed calls
(pathogenic-class against benign-class)[, {k} with an authority that could not be consulted]. A
contested subject is a question, not a defect — half the time the archive is the stale side, which is
why this never fails a build in either mode. Answer one by adding a row to overrides.csv naming table
'clin_sig_concordance.csv', the subject's variant_key and its genotype, with the reason you stand by
the module's call.
Three things about it are decisions rather than wording. It never escalates under --strict, for
the reason the check that produced the rows does not: escalating would have this format arbitrate a
clinical dispute. It is actionable rather than carried, unlike verification_findings_recorded
one section down, and the difference is real — a number sitting in verification.json is not
something an author can move, while a contested row is answered by writing an overlay row, and the
count here is taken over the post-overlay table so answering one clears it. And it names
overrides.csv and never provenance.json's outranks: the two are the same idea one table
apart, 0.7 settled the overlap as a succession in the overlay's favour, and an author meeting this
warning for the first time should be sent to the mechanism that survives 1.0.
The count is embedded although both passes emit the sentence. That is normally the trap where a message carrying a number is rebuilt from inputs resolution changed in between; it is not one here, because no compile step between the two passes touches this table or the overlay above it, so both passes reach a byte-identical sentence and the existing de-duplication collapses them.
Substrings the suite pins as contract, so changing one is a deliberate act rather than a reword:
test_validate_agrees_with_compile.py holds "forbid sale", "does not match the id recomputed",
"could not be verified", "p_value_num says", "not among the", "IUPAC ambiguity code",
"not valid YAML", "must be a mapping"; test_strict_compile.py holds "unresolved genomic
positions"; test_roundtrip_regressions.py holds "pointer, not an expression".
The refusals only strict adds¶
errors is list[str] on every result type and carries no code: RM131 classified the warnings
channel and left errors as prose. Most strict errors are a mode-laddered warning raised to an error, in
the same sentence. Two are not: under strict they are added rather than promoted (RM218), and a
consumer grading a dry run needs to tell them apart. Both open with strict compile:, and no other
message in any list opens with it, so that prefix alone says a refusal strict added. Which one it is
is told by the phrase below, which validate_spec(strict=True) and compile_module(strict=True) both
return:
| Finding | Phrase | What it says |
|---|---|---|
unresolved_positions |
have unresolved genomic positions after resolution |
resolution left rows without a coordinate, so the parquet bytes would not be reproducible. Cleared by a complete resolution, never by an authored edit to the rows |
genome_build_disagreement |
whose coordinates the enricher diagnosed as another assembly's |
verification.json records the enricher's finding that rows sit on a different assembly than the module's genome_build: two authored files contradict each other. Cleared by fixing the coordinates and re-running the checks |
Both phrases are contract, as the four constants above are, and the table is the registry:
test_strict_refusal_phrases.py reads the phrases from it, walks the compiler's source for every
strict compile: literal and refuses one that carries no published phrase, so a third strict-only
refusal fails the suite until it has a row here. The counts in both sentences vary; the phrases do not.
Coded errors, with an errors_summary beside warnings_summary, are RM378, a minor.
CLI, and what it maps onto¶
just-dna-compiler (Typer) is a thin shell over the Python API. Exit 0/1, CI/registry-gateable.
| Command | Python API | Notes |
|---|---|---|
validate <spec> |
compiler.validate_spec |
--strip-identity / --authority-key, --strict/--best-effort, --resolve/--no-resolve and --ba1-threshold (RM367: pass the switches the compile will get). Read-only — it never stamps a closure. test_check_paths.py asserts the command forwards every validate_spec parameter |
close <spec> |
compiler.close_module |
--by, --private-key. Writes the RM73 closure into verification.json, bound to the authored bytes. Refuses an invalid spec; a warning does not refuse |
compile <spec> <out> |
compiler.compile_module |
--strict/--no-strict, --resolve/--no-resolve, --compression, --compiled-by, --ensembl-reference, --ba1-threshold (RM278), and the deprecated --ensembl-cache (routes to the enricher; removed at 1.0). Prints digest, content_signature, resolution_mode/fully_resolved/resolution_signature |
signature <spec> |
compiler.content_signature |
no compile, no reference |
| reverse <parquet_dir> <out> | compiler.reverse_module | --resolution/--no-resolution (default on) + display overrides |
| verify <module_dir> | format.integrity.verify_manifest | --public-key, --check-inputs/-logs/-provenance/-logo/-readme/-derived |
| keygen | format.signing.generate_private_key_pem + public_key_b64_from_pem | --out (refuses to overwrite) |
| sign <module_dir> | format.signing.sign_digest | --private-key |
| reference | format.reference.authoring_reference / json_schemas | --summary, --schemas |
| template <kind> | draft.blank_template + authoring_requirements | the SNP core, every optional table kind, and sources.csv |
| stub <kind> | draft.stub_template | --rows |
| requirements <kind> | draft.authoring_requirements | --json |
| scaffold <spec> | scaffold.scaffold_module | --kind, --rows, --dry-run |
| describe <kind> | hints.describe_table | one table's columns + pick-lists, each column's category and its vocabulary's per-member notes |
| hint <kind> | hints.inspect_rows | --file/--row, --json |
| sweep <before> <after> | sweep.compare_outputs + gate_findings | --spec-root (compile AFTER with the installed compiler), --release (run the release gate; exit 1 on a finding), --json. A release-sequence command, not an ordinary test — it needs the previous release installed |
compile exposes every compile_module parameter but four, by decision (RM278). --ensembl-reference
stamps manifest.compilation.ensembl_reference, which a CLI compile could not do before 0.8, and
--ba1-threshold moves the ACMG BA1 cutoff. The four left out are log_files, provenance_file,
logo_file and readme_file: each is auto-discovered beside the spec, which is where a CLI author
keeps them, so an explicit path is an API-only override. test_cli_parity.py asserts the option set
equals the parameters minus that declared set, so the next parameter cannot drift in unweighed, which
is how the first two did (RM218).
--no-resolve is the master switch, not an Ensembl switch (S14). With an injected resolution.csv
beside the spec it used to compile successfully with chrom=None on every weight row — a silent
success, the worst shape a mistake takes. It now warns and names the size of what it discarded
(N row(s), covering K variant key(s) — rows, not keys, since a one-to-many rsid contributes several),
for the same reason vrs_alleles ships beside vrs_alleles_identified: a warning that quantifies over
a table should publish the denominator. The message also states that there is no flag for "do not
reach the network", because the compiler never does (Principle 2, verified branch by branch including
the deprecated ensembl_cache path, which reads an injected local cache) — omitting this flag is that
request. Renaming the parameter is a 1.0 conversation; it is part of a published signature.
And the table really is not read (RM367). Until 0.8 the warning was the only thing the switch
reached: the table was still overlaid, its vrs_ids verified (a tampered one refused the compile),
its coverage counted into manifest.compilation.vrs_alleles_identified while no id reached a parquet,
and allele membership judged against it. Now both commands read it through one reader whose active view
is empty under the switch, and it is parsed only for the row count above. The guard is the probe
@off-switch-needs-a-probe asks for: over every reference example carrying a resolution.csv,
--no-resolve compiles the same parquets and the same resolution facts as the spec with the table
deleted (test_one_verdict.py). strict still refuses an rsID-only row under the switch, on both
sides, because the artifact is not reproducible however the row came to be unplaced.
Four rows are bold because they belong to just-dna-format, which ships no CLI of its own —
Typer would breach its pydantic-plus-cryptography dependency floor (Goal 2). So anything the schema
tier owns that a user needs has to surface here, and three of the four did not until 0.5:
sign --private-keydemanded a key file the toolchain could not produce, andverify --public-keydemanded a string onlypublic_key_b64_from_pemcould derive. Signing was therefore CLI-complete only for someone willing to write Python — the exact gapverifyhad been added to close, left open one step upstream.keygencloses it; the key is unencrypted PKCS#8 (whatsign_digestreads), which is a deliberate limit rather than an oversight: this bootstraps a key, it is not key management, and a passphrase prompt would imply custody guarantees nothing here provides. It refuses to overwrite an existing key, because every signature made with the old one would stop verifying and a published artifact's bytes are never mutated.authoring_reference()had no route at all, which hurt most for the consumer that most needs it: an MCP surface offering an author the valid values had to importjust_dna_format.referenceand write Python.describeanswers that for one table;referenceanswers it for all of them plus the vocabularies, the open-vs-closed flag,REQUIRED_ANY_OFand the palette.
template sources.csv used to deny the table existed (S21, 0.5.4). DRAFTABLE held the SNP core
and the optional table kinds, so blank_template("sources.csv") answered "is not an authored table of
this format" — false, and said by the surface an author reaches for instead of reading the models,
which is what the consumer who reported it then had to do. The other three fact sidecars stay out
because an enricher pass writes them and nobody starts one by hand; this one the schema instructs a
human to write (a source read by hand leaves no source cell for the coverage check to find) and the
compile licence gate reads it and nothing else. Its natural key is (source, layer) — the same one
licensing.merge_sources_csv merges on, borrowed for the reason every _CORE_DUPE_KEYS entry is
borrowed: a draft must not append a row the other writer treats as already present, and one source
legitimately appears at two layers.
A drift the audit found in the same place. authoring_reference() reported requiredness with
pydantic's two-way is_required(), while just_dna_compiler.draft had already been fixed to the
three-way required / defaulted / optional split — the middle one being the trap where
MeasureBinRow.measure_kind is not required and not safely left blank either. Two surfaces
answering one question, and the drift-proof one was the stale one. The split now lives in
format.base.field_category, the only tier both can import from; the reference emits it as
category beside the existing required key (kept, since removing a published key breaks consumers
and required is insufficient rather than wrong).
Not levelled, deliberately: just-dna-enricher mirrors template but not stub, requirements,
describe, hint or scaffold. The offline authoring surface belongs to the compiler; the one
mirror exists so a PGx author working through the enricher does not have to switch binaries for a
header. Adding the other four would duplicate a surface that has an owner, and removing the mirror
would break scripts for no gain.
The release-record sweep (RM126)¶
just_dna_compiler.sweep. The instrument that measures what a release changed about compiled output,
and the gate that refuses a release whose measurement carries no declaration. The record it feeds and
the pure needs_recompile a consumer reads live in the format tier — see
SCHEMAS § The release record
for the shape, the vocabularies and the interval algebra. This section is the operation.
Why a measurement and not a map. A hand-kept per-release map was the first repair anyone proposed
and it is the defect wearing a public name (@registry-completeness): five of the six RM104–RM111
fixes were a derived value restated by hand. So the sweep measures, and the gate makes the measurement
force the declaration rather than leaving the author to remember writing one.
What it compares, and the one discipline that makes it mean anything¶
Two trees of compiled output — one produced by the previous release, one by this one — from the same
spec root. Feeding each side its own tree's reference_examples/ measures spec drift as compiler
drift: between v0.6.1 and v0.6.6 one example's README moved, which is enough to shift
manifest.readme and inputs. One spec root, two compilers.
Five axes, and each is computed apart because a consumer acts on each differently:
| axis | how it is measured |
|---|---|
parquet_schema |
every parquet's {column: dtype}, read with scan_parquet(...).collect_schema() |
parquet_bytes |
artifact.digest |
content_signature |
the manifest field |
manifest_fields |
every dotted path in the manifest, minus EXCLUDED_MANIFEST_FIELDS |
warnings |
the compilation.warnings set, reported apart and never folded into the above |
EXCLUDED_MANIFEST_FIELDS is a published registry with a reason per member, and
compilation.compiler_version is the one that matters: it moves on every release by construction,
so counting it would make every record fire on every module in every release and rebuild, inside the
instrument, exactly the false-positive class the interval shape exists to avoid.
compilation.compiled_at and compiled_by are environment; content_signature, artifact.digest and
artifact.files are routed to their own axes rather than dropped. compilation.toolchain (RM342) is
environment too, and it moves whenever the install does, so it is excluded for compiler_version's
reason; the sweep reads it instead as the reason a parquet_bytes movement carries (below).
A parquet_bytes movement under a different toolchain names it (RM342). compare_module sets
ModuleDelta.toolchain_differs from manifest.toolchains_differ (tri-state, None when either
manifest recorded none), and ModuleDelta.bytes_reason is "toolchain_differs" exactly when the
digest moved and the recorded toolchains differ. SweepMeasurement.moved_under_other_toolchain lists
those modules (named apart from the delta's tri-state, @flag-means-same), the evidence sentence names
them, and the JSON carries both per module and the list at the top. The movement is still a measured
parquet_bytes movement. The reason names a co-occurring difference, not a proven cause: across a
sweep the compiler version differs by construction, so a release that moves polars and changes the
compiler labels every moved module this way. That is why the gate never accepts it as a declaration's
evidence; the toolchain kind below needs a second sweep.
A toolchain-only byte move: the toolchain kind and its fixed-compiler sweep (RM352)¶
VALID_RELEASE_CHANGE_KINDS has a third member, toolchain: the bytes moved, the content did not,
and the recorded toolchain says why. It is legal on parquet_bytes only (DeclaredChange refuses it
elsewhere). The gate does not take it on trust. A compiler bug moves bytes without moving content
too, and at a cut the compiler version changes together with polars, so the release sweep alone cannot
tell the two apart. The only clean separation is a second sweep with the compiler held fixed: this
release's compiler on both sides, only the toolchain varied. sweep --toolchain-before DIR takes the
tree this release compiled under the previous toolchain; the release sweep's AFTER tree is the other
side. Each SweepMeasurement records before_toolchain / after_toolchain, the one toolchain each side
recorded (None when a module recorded none or the side mixes two), and before_digests /
after_digests, each module's artifact.digest per side.
That sweep alone is not enough, and the separation is per module. It shows the toolchain is enough to move a module's bytes, never that the compiler did not move them as well: polars 2 writes its version into every footer, so every module moves under it whatever else this release changed. A module's release move is the toolchain's only when the previous compiler and this one wrote the same bytes under the previous toolchain: the release sweep's BEFORE digest equals the fixed-compiler sweep's BEFORE digest, and the two sweeps share one AFTER tree. The whole move then lies between the two toolchains. So the release sweep's BEFORE tree is built under the previous toolchain too (the sequence below pins it).
When a record declares toolchain on parquet_bytes and the bytes moved, the gate reads those
fields, never a sentence in evidence, and fails the release unless:
| condition | finding when it fails |
|---|---|
| the second sweep was passed | TOOLCHAIN_UNBACKED_PHRASE |
its two sides carry the same compiler_version |
TOOLCHAIN_ACROSS_COMPILERS_PHRASE |
| that version is the release being gated | TOOLCHAIN_WRONG_COMPILER_PHRASE |
| both sides record a toolchain and the two differ (unrecorded is unknown, never a difference) | TOOLCHAIN_NOT_DIFFERENT_PHRASE |
| its AFTER side is the release sweep's AFTER tree, digest for digest | TOOLCHAIN_OTHER_AFTER_PHRASE |
content_signature did not move under the toolchain on a module the toolchain moved, nor across the release on a module the toolchain explains |
TOOLCHAIN_CONTENT_MOVED_PHRASE, per module |
when toolchain is the axis's only kind: every module whose bytes moved is explained (measured by the second sweep, moved under the toolchain, and the same bytes from both compilers under the previous toolchain) |
TOOLCHAIN_UNEXPLAINED_PHRASE, per module, with which of the three failed |
| when another kind is declared beside it: the toolchain explains at least one module | TOOLCHAIN_EXPLAINS_NOTHING_PHRASE |
A module the toolchain does not explain moved for a compiler reason, a correction or an addition,
and is declared as one. Beside such a declaration it is covered, and a note names it. A second sweep
passed while no toolchain declaration reads it is a note too (TOOLCHAIN_SWEEP_UNUSED_NOTE_PHRASE),
so an operator does not believe it was checked.
A list is a leaf. Indexing into one would make an inserted element rename every path after it, and
stats.genes is what a consumer keys on anyway. A missing key and a present null are distinguished
by a sentinel, not by dict.get — otherwise literature: null growing the RM119 counters reads as
unchanged on the literature path itself.
Backfilled, and what the numbers say¶
The two records shipped in 0.7 were measured on 2026-08-28 over all sixteen reference_examples/,
compiled from one spec root under each published release in turn with just-dna-compiler and
just-dna-format pinned together (the workspace cuts all three packages at one number). 0.6.0 →
0.6.1 moved nothing — a measured zero with its denominator in the record's evidence, never
silence. 0.6.1 → 0.6.6 moved parquet_schema and parquet_bytes on 10 of 16 (RM120's authored
curator column growing studies.parquet), manifest_fields on 9 (stats.genes/stats.gene_count
on seven, RM121; literature.quotes_unchecked on three, RM119) and content_signature on none.
The 0.6 line published three compiler releases — 0.6.0, 0.6.1, 0.6.6. The tags in between never
reached PyPI, so no stored artifact's compiler_version can name one, and the chain is complete over
what a consumer can actually be holding. Older intervals stay honestly unknown.
Where the gate runs, and what fails¶
In the bump → uv sync → tag sequence, not as an ordinary test, because it needs the previous
release actually installed. A release whose sweep shows a changed axis or field that no ReleaseRecord
declares fails.
# Fresh trees every time. `compile_module` only mkdir -p's its output, so reusing a fixed path
# inherits the last cut's modules: a renamed one lingers as a stale directory the sweep still reads,
# and a parquet a module no longer emits still gets globbed into its schema.
BEFORE=$(mktemp -d) && AFTER=$(mktemp -d)
# 1. the BEFORE side, under the previous published release, against THIS tree's specs
for d in reference_examples/*/; do
uv run --isolated --no-project \
--with just-dna-compiler==<previous> --with just-dna-format==<previous> \
just-dna-compiler compile "$d" "$BEFORE/$(basename "$d")"
done
# 2. bump the versions, `uv sync`, THEN measure and gate — in that order, see below
uv run just-dna-compiler sweep "$BEFORE" "$AFTER" --spec-root reference_examples/ --release <new>
A release that moves the toolchain too (0.8: polars 1 → 2) builds step 1's BEFORE tree under the
previous toolchain, by adding --with polars==<previous polars> to that uv run, and adds the
fixed-compiler tree before step 2, built by this release's compiler (after the bump and uv sync)
under the same previous toolchain. uv run --with overlays the locked polars; check that the
fixed-compiler tree's manifests record the old version. A previous release records no toolchain, so the
gate cannot read BEFORE's; the per-module digest equality with the fixed-compiler tree is what shows it
was the same toolchain and the same bytes.
Step 1 pins just-dna-format==<previous> only when the format was published at that number too. It
is not always: on 2026-11-04 PyPI held compiler 0.7.2 but format 0.7.1, so the 0.8 cut pins
--with just-dna-compiler==0.7.2 alone and lets the resolver pick its published partner. <previous>
is the newest published compiler, read off PyPI at the cut, never off a local tag (v0.7.3 was
tagged and unpublished), and the record's previous must equal what BEFORE is stamped with. If the
overlay ever stops shadowing the lock (a polars>=2 floor would refuse it), build the fixed-compiler
tree with uv run --isolated --no-project --with ./schema --with ./compiler --with polars==<previous
polars> instead.
TC_BEFORE=$(mktemp -d)
for d in reference_examples/*/; do
uv run --with polars==<previous polars> \
just-dna-compiler compile "$d" "$TC_BEFORE/$(basename "$d")"
done
uv run just-dna-compiler sweep "$BEFORE" "$AFTER" --spec-root reference_examples/ \
--toolchain-before "$TC_BEFORE" --release <new>
uv sync before the sweep, not after. --spec-root builds the AFTER tree with whatever compiler
is installed, so running the sweep on a stale environment measures the previous release against
itself: every axis reads False and the whole thing looks like a clean release that changed nothing.
The gate refuses it — it checks the interval's upper end against --release and rejects a
degenerate interval outright — but the fastest way not to meet that message is to sync first.
Step 2 exits 1 until release_records.RELEASE_RECORDS gains an entry for <new> covering what moved.
The measured half is what SweepMeasurement.as_record(...) produces — with an empty declared
list on purpose, so the gate keeps refusing until somebody says whether each movement was a
correction (the value we published was wrong), an addition (it was absent) or, on parquet_bytes
with the fixed-compiler sweep behind it, a toolchain move (above). That is the whole mechanism.
The two halves are written at different times, and that is where a record drifts (RM161). An item
landing after the measurement adds its DeclaredChange and leaves manifest_fields behind it, which
turns this gate red at the one moment nobody wants to be debugging a record — 0.7.0 reached its own
pre-build run missing gene_validity.superseded_count and identity.version_coerced_from, both
declared the morning they landed. Re-measure whenever a declaration is added, not only when the
release is cut. A test catches the addition half offline: a declared addition must appear in
manifest_fields, since a field that did not exist before moves wherever its block appears. A
declared correction is deliberately exempt — it can be true of the release and unmeasurable on this
corpus, which is what the OVERDECLARED_NOTE_PHRASE note is for.
The gate's finding phrases¶
Named constants, for the same reason a warning's text is: a release script greps them.
NO_RECORD_PHRASE = "has no release record"
UNDECLARED_AXIS_PHRASE = "moved and the release record does not record it moving"
UNDECLARED_FIELD_PHRASE = "moved and the release record does not list it"
UNDECLARED_KIND_PHRASE = "moved and nothing declares it a correction or an addition"
WRONG_PREVIOUS_PHRASE = "was measured against a release the record does not name"
WRONG_VERSION_PHRASE = "did not measure the release being gated"
DEGENERATE_INTERVAL_PHRASE = "measured one release against itself, so it measured nothing"
UNMEASURED_MODULE_PHRASE = "could not be measured on both sides, so the sweep says nothing about it"
NO_MODULES_PHRASE = "measured no module at all"
OVERDECLARED_NOTE_PHRASE = "is declared and this sweep did not see it move"
REGRESSED_MODULE_PHRASE = "compiled under the previous release and does not compile under this one"
UNDECLARED_UNMEASURED_PHRASE = "has no output from the previous release and the record does not list it"
OVERDECLARED_UNMEASURED_NOTE_PHRASE = "is declared unmeasured and this sweep measured it on both sides"
TOOLCHAIN_UNBACKED_PHRASE = "is declared a toolchain move and no fixed-compiler sweep backs it"
TOOLCHAIN_ACROSS_COMPILERS_PHRASE = "compares two compiler versions, so it cannot separate the toolchain from the compiler"
TOOLCHAIN_WRONG_COMPILER_PHRASE = "held fixed a compiler other than the release being gated"
TOOLCHAIN_NOT_DIFFERENT_PHRASE = "does not record two different toolchains, one per side"
TOOLCHAIN_CONTENT_MOVED_PHRASE = "moved its content_signature, so a toolchain move cannot cover it"
TOOLCHAIN_UNEXPLAINED_PHRASE = "moved its bytes across the release and the toolchain alone does not explain it, so a toolchain move does not cover it"
TOOLCHAIN_OTHER_AFTER_PHRASE = "does not share the release sweep's AFTER tree, so it says nothing about the bytes being released"
TOOLCHAIN_EXPLAINS_NOTHING_PHRASE = "is declared a toolchain move and the toolchain alone explains no module's bytes"
TOOLCHAIN_SWEEP_UNUSED_NOTE_PHRASE = "was passed and no toolchain declaration on a moved parquet_bytes reads it"
UNDECLARED_KIND_PHRASE keeps its wording, since scripts grep it; on parquet_bytes the message
appends (or, on this axis, a toolchain move). The TOOLCHAIN_* nine are RM352's, tabled above; the last is a note.
The middle four are about the sweep rather than about the record, and they exist because a gate
that only asked is this movement declared? passed every sweep that measured nothing. A stale
environment, a module whose compile broke under the new release, and two trees sharing no module at
all all produce an all-False measurement, and none of them is a release that changed nothing.
--release accepts the stamped just-dna-compiler X.Y.Z spelling as well as the bare one, for the
same reason needs_recompile does.
OVERDECLARED_NOTE_PHRASE is a note, not a finding, and does not fail the release: the reference
corpus is sixteen modules and a real correction can land on a shape none of them has. It is still
printed, so over-declaring is visible rather than invisible.
The last three are RM139's split of one side only, below.
Which side a module is missing from (RM139)¶
UNMEASURED_MODULE_PHRASE stays in every message here, so a script grepping it catches all of them.
What changed is the clause after it, because the two directions are facts about different
releases:
| the module is | what it means | the gate |
|---|---|---|
in BEFORE, not in AFTER (only_before) |
this release cannot compile a spec the previous one could — a regression, or a stale reused BEFORE directory holding a module the spec root no longer has | fails, unconditionally, with the compiler's own errors beside it where --spec-root built the AFTER side. as_record refuses to mint over one |
in AFTER, not in BEFORE (only_after) |
the previous release produced no output for it: its spec uses a column that release refuses under extra="forbid", or the example is newer than that release |
fails until ReleaseRecord.unmeasured names it. Then it is measured-nothing rather than measured-zero |
The first real use of the gate hit the second row, which the rule did not model: RM70 put the optional
requires_callable column on pharm_variants.csv, reference_examples/cyp2c9_warfarin_grch37/ uses
it, and 0.6.6 refuses that spec. Nothing failed — the module has no before state at all, so no
like-for-like comparison exists — and it recurs in every minor that adds an authored column and
exercises it in the corpus. The 0.7.0 cut stated the exclusion in evidence prose the gate cannot
read and the tag was waved through by hand.
unmeasured is a denominator, not an exemption, and the difference is the equality: a module the
sweep measured on both sides cannot be excused by listing it (that is reported as a note), a
regression cannot be excused by listing it, and a movement on a measured module still gates however
the list reads. What it buys is that the exclusion is forced into the published record by the
measurement — as_record fills it — instead of living in a sentence nothing checks.
Under --json, stdout is the JSON document and nothing else — the notes and the success line go
to stderr there. The caller of that flag is a release script piping to jq, and a note printed after
the blob breaks exactly the consumer the flag exists for. Without --json they read on stdout as
usual.
The warnings axis, and the seam RM131 fills¶
compilation.warnings is a published manifest field, so the naive reading folds it into
manifest_fields — and then RM131's restructuring and RM134's new checks report a manifest field
changed on essentially every module in 0.7, and a registry acting on that mints an immutable PATCH
across a whole catalogue for a reworded message. It cannot simply join compiled_at in the excluded
set either, because a new warning can be a real signal.
So it is its own declared axis, outside RECOMPILE_DRIVING_AXES. RM131's carried split landed in
0.7 and is that discriminator: compare_module now reports carried_added beside
actionable_added, read off the after manifest's compilation.carried rather than re-derived from
prose. A carried finding appearing is usually this repository saying more about a limit it always had;
an actionable one appearing is work arriving at somebody's door.
axes["warnings"] deliberately still fires on any movement of the set. Narrowing it would make a
published axis mean something different from what every record already written claims about it, and
the axis drives no rebuild anyway — the split buys the reading, not a new gate. A manifest with no
carried field (anything compiled before 0.7) reports every addition as actionable, which is the safe
direction: it never tells a reader that a finding they could fix is unfixable.
compilation.carried and compilation.warnings_summary both join compilation.warnings in
EXCLUDED_MANIFEST_FIELDS, for the same reason and with the same consequence — they are derived from
it, so they move exactly when it does.
What this does not touch: SpecRow.needs_upgrade is self.upgraded() != self, computed over
authored row content and a hard filter in the marketplace. A warning never touches an authored row, so
no warning change can flip it. Verified rather than assumed, because the failure mode would have been
modules silently vanishing from a catalogue.
Coverage table (0.3 / 0.4 features)¶
| 0.3 / 0.4 feature | Validated | Materialized (→ parquet) | Computed / derived | Status |
|---|---|---|---|---|
direction (VariantRow) |
✅ full vocab | ✅ weights.parquet — the authored value only, never a derivation |
✅ Python read-time only: effective_direction / upgraded() from state(+weight) |
complete — but read the two cells apart: a state-only module ships an empty column |
stat_significance (VariantRow, StudyRow) |
✅ full vocab | ✅ authored value only | ✅ Python read-time, derived from state (not inferred from p_value) |
complete, same split as direction |
effect_size (VariantRow, StudyRow) |
✅ float | ✅ | — | complete |
effect_measure (VariantRow, StudyRow) |
✅ permissive (open) | ✅ | — | complete (intentionally open) |
effect_allele (VariantRow) |
✅ nucleotides | ✅ | ✅ membership in {ref} ∪ alts (0.5) → warning / error in strict; ⛔ still no strand reconciliation |
validate + membership check |
genotype (VariantRow) |
✅ grammar (phased/hemizygous/sorted) | ✅ alleles + phased |
✅ membership in {ref} ∪ alts (0.5), unioned across a one-to-many rsid's loci |
complete |
flags (VariantRow) |
✅ open; split; reserved set | ✅ List[str] |
✅ unknown-tag INFO (ValidationResult.info) |
complete |
trait_efo_id (VariantRow, StudyRow) |
✅ CURIE(s) | ✅ | — | complete |
doi (StudyRow, RM11) |
✅ DOI grammar, verbatim | ✅ studies.parquet |
— | complete |
provenance_quote / provenance_regex (StudyRow, RM12) |
✅ free-text / author-time re.compile |
✅ studies.parquet |
— | complete (P1 pattern grammar; matched consumer-side) |
authorship (ModuleSpecConfig/ModuleManifest, RM14) |
✅ Contribution (role closed, kind open, extra=forbid) |
✅ manifest (out of digest) | — | complete (metadata; reversed from the manifest since RM241) |
clin_sig (VariantRow) |
✅ full vocab | ✅ | ✅ ↔ pathogenic/benign aliases |
complete |
module.version (ModuleInfo, 0.4.1) |
✅ freeform advisory (legacy v2/3) |
✅ manifest Identity.version iff valid SemVer; reverse_module(version=) re-emits |
✅ normalize_version preview (RM17 enforces) |
complete (advisory) |
| authority-key strip (0.4.1) | ✅ inject-only pre-strip; dropped → .info; typo'd still extra=forbid |
— | — | complete (DI) |
| strict compile (0.4.1) | ✅ strict=True fails (pre-write) on an unresolved (chrom, start) |
— (refuses a partial) | — | complete (opt-in) |
content_signature (0.4.1) |
✅ over raw authored rows, normalized+sorted, name-/Ensembl-independent | ✅ manifest (out of digest); signature CLI computes it without recompiling |
— | complete (canonical dedup identity) |
resolution.csv path (0.5) |
✅ resolve_from_table consumes injected facts; digest-parity with the DuckDB path proven; provisional shape (§ note) |
✅ drives weights.parquet coords; resolution_signature/resolution_mode/fully_resolved/resolution_sources → manifest (out of digest) |
✅ fill / expand / verify (pure, no duckdb) | complete (preferred path) |
| VRS allele identity (0.5) | ✅ stdlib derive_vrs_allele_id; every member of vrs_id recomputed and verified per ALT, plus a coverage report (mode-dependent severity) |
✅ variant_key is the VA for a resolved substitution → weights/annotations.parquet |
✅ minted per ALT from (chrom, start, ref, alt); a multi-allelic site names each allele, and the key still falls through to the coordinate for indels/MNVs/multi-allelic |
complete (GRCh38-only; multi-build is RM15) |
frequencies.csv path (0.5) |
✅ FrequencyRow; coordinate cross-check → warning; provisional shape |
✅ frequencies.parquet (in artifact.digest); frequency_signature/sources/datasets/populations → manifest (out of digest) |
✅ allele_frequency = AC/AN materialized as Float64 (never stored in the CSV) |
complete (injected; enricher produces it) |
gene_metrics.csv path (0.5) |
✅ GeneMetricsRow; gene cross-check → warning; provisional shape |
✅ gene_metrics.parquet (in digest); gene_metrics_signature/genes/datasets → manifest |
— | complete (injected; offline-capable upstream) |
literature.csv path (0.5) |
✅ LiteratureRow; citation cross-check + nonexistent-PMID warning; provisional shape |
✅ literature.parquet (in digest); literature_signature/sources/coverage counters → manifest |
— | complete (injected; enricher produces it) |
gene_validity.csv path (0.6, RM24) |
✅ GeneValidityRow; gene cross-check → warning; provisional shape |
✅ gene_validity.parquet (in digest); gene_validity_signature/genes/diseases/classifications/submitters/datasets → manifest |
— | complete (injected; ClinGen + GenCC routes in the enricher) |
clin_sig_concordance.csv / clin_sig_authority_calls.csv (0.7, RM130) |
✅ two derived fact tables — the per-variant three-way ClinVar concordance verdict, and every authority's call as it stood at the time (the only derived table recording a value at the time, which is what answered-call currency compares against); orphan rows warn under derived_row_orphan; provisional shape |
✅ clin_sig_concordance.parquet / clin_sig_authority_calls.parquet (in digest, fact-hashed); _TABLE_DUPE_KEYS registered; a cell key carries the value because two rows may state two claims |
✅ clin_sig_concordance_contested (see the phrases) |
complete (written by enrich; the compiler cross-checks and never adjudicates) |
statistical_test (StudyRow, 0.7, RM140) |
✅ free text — which analysis produced this row's p_value/effect_size; in the dedup key by value |
✅ studies.parquet; reversed |
— | complete (passthrough; nothing derives or checks it) |
confidence / confidence_unit (StudyRow, 0.7, RM160) |
✅ the citing source's own review state, unconverted, and the instrument it is on; a magnitude with no unit is refused at the model, a unit alone allowed | ✅ studies.parquet; reversed through the hand-kept fieldnames and the row dict (the fourth touch point) |
— (the enricher's evidence_status_currency re-asks it) |
complete (passthrough) |
clinical_assertions.csv path (0.6, RM25) |
✅ ClinicalAssertionRow; coordinate cross-check → warning; provisional shape. Records the archive's call and review tier; it does not adjudicate against the author's clin_sig — that stays the enricher's warn-in-both-modes cross-check |
✅ clinical_assertions.parquet (in digest); clinical_assertion_signature/clin_sigs/star range/unrated_count/not_found_count → manifest |
— | complete (injected; offline-capable upstream from the ClinVar snapshot) |
| CLI (0.4.1, extended 0.5) | ✅ Typer validate/compile/signature/reverse/verify/sign; --strict, --strip-identity/--authority-key, deprecated --ensembl-cache, --resolution |
— | — | complete (compiler-only dep; tiers intact) |
| queryable p-value (0.5) | ✅ p_value_num in (0, 1]; cross-checked against the verbatim p_value string (relative, 1%) |
✅ studies.parquet; neg_log10_p derived on write, absent from the reversed CSV |
✅ -log10(p_value_num) |
complete |
callable_from (0.5, RM6) |
✅ VCF field-name pointer, namespace-qualifiable and \|-alternatable (shared AuthoredModel validator); bare colliding key → warning both modes (0.6) |
✅ weights.parquet |
— | complete (retired from the reserved namespace) |
recommendation_strength (0.5) |
✅ closed CPIC vocabulary, distinct axis from evidence_level |
✅ diplotypes.parquet |
— | complete |
| dosage sensitivity (0.5) | ✅ haploinsufficiency/triplosensitivity against VALID_DOSAGE_SENSITIVITY |
✅ gene_metrics.parquet (in digest, fact-hashed) |
— | complete (ClinGen route in the enricher) |
redistribution (0.5, settled 0.6) |
✅ tri-state; None ≠ False |
✅ sources.parquet; per-layer facet + module-wide verdict → manifest |
✅ most-restrictive-wins | complete — recorded here, enforced downstream (RM27; the ask is in SCHEMAS.md) |
| GWAS effect sizes (0.6, RM90) | ✅ effect_direction closed; effect_measure open; effect_unit free text and inside the fact hash |
✅ gwas_effects.parquet (in digest, fact-hashed) |
✅ orphan rows warn, never fail | complete (Catalog route in the enricher; fills no weight — see MODULE_LIFECYCLE § Stage 3) |
| Expression effects (0.7, RM194+RM200) | ✅ effect_direction closed (shared with GwasEffectRow — same axis); effect_measure open; effect_unit null on every AlphaGenome row and stated rather than invented; gene and gene_id both inside the fact hash |
✅ expression_effects.parquet (in digest, fact-hashed) |
✅ no orphan check, deliberately — the rows are locus-wide by construction, so gwas_effects' check would fire on nearly every row |
complete (alphagenome expression; non-commercial, gated at compile via sources.csv) |
| weighting declaration (0.6, RM92) | ✅ three free-text strings, extra="forbid" |
✅ manifest.weighting; moves neither identity half |
— | complete; reversed from the manifest since RM241, like license/panel/authorship |
| verification attestation (0.6, RM45) | ✅ binding recomputed from the authored inputs, proof-of-work re-checked; stale ⇒ warn + drop, never fatal | ✅ manifest.verification (out of artifact.digest); nothing reaches a parquet |
— (the enricher puts the checks) | complete (verification.json; nothing in the block is trusted) |
| authoring closure (0.6, RM73) | ✅ published when the attestation holds and carries one; absent ⇒ warn in both modes; a signed closure that does not verify ⇒ drop the whole block | ✅ manifest.verification.closure; moves no digest and no signature (measured on all sixteen reference examples) |
✅ close writes it; validate never does |
mechanism complete (compiler.close_module); the refusal is 1.0 and blocked — see ROADMAP_1_0 § RM73 |
| drafting (0.5) | ✅ appended rows are validated rows; keys reuse _TABLE_DUPE_KEYS |
— (writes authored CSVs, not parquet) | ✅ append / already-present / differs report | complete (draft.append_rows, blank_template); DRAFTABLE covers the SNP core, the table kinds and sources.csv (0.5.4) |
| templating (0.5) | ✅ a stub carries TEMPLATE_PLACEHOLDER, which no mode compiles |
— (writes authored CSVs, not parquet) | ✅ created / kept-untouched plan | complete (draft.stub_template, scaffold.scaffold_module) |
| hints (0.5) | ✅ per-cell validation, bin coverage, duplicate keys, a conclusion naming another genotype (RM279) — all offline | — (writes nothing at all) | ✅ alterations + findings + options | complete (hints.inspect_rows, hints.describe_table) |
| delegated insertion (0.5) | ✅ placed rows are validated rows; shifted rows keep their cells | — (writes authored CSVs, not parquet) | ✅ DraftReport.shifted names every moved row |
complete (draft.place_rows, append_rows(group_by=…)) |
| partial rows (0.5) | ✅ stubbed columns validated by omission; the stub itself never compiles | — (writes authored CSVs, not parquet) | ✅ added / already-present / invalid; DraftReport.header_extended names the columns the header grew by — exactly the ones the written rows fill, never one only a rejected or already-present row filled, and a raw blank is not a fill (2026-09-09); settled before the existing rows are re-rendered (2026-09-02: an append into a header that predates a column used to raise a bare ValueError). Both writers rewrite the author's file atomically, and since 2026-09-12 both take before_commit — a callback run after the bytes are down and before the rename, so a drafter's licence row lands inside the commit of the table it licenses rather than after it (RM232). Optional and defaulted, so a caller that passes nothing is unaffected; it fires only when the call actually writes |
complete (draft.PartialRow, append_partial_rows) |
| genotype widening: hemizygous single allele | ✅ | ✅ (1-element list) | — | complete |
genotype widening: phased A\|G |
✅ (order kept) | ✅ phased bit → lossless round-trip |
✅ | complete |
state (legacy) |
✅ (stays required — P8) | ✅ | ✅ read alias via effective_direction; trimmed to {protective,risk,neutral} on upgraded() |
complete |
| MT / non-diploid genotype | ✅ warning on a two-allele MT or Y genotype | — | — | complete |
| direction/weight sign consistency | ✅ warning | — | — | complete |
0.4 compiler coverage (materialized)¶
| 0.4 kind (model) | Validated | Materialized (→ parquet, round-trip) | Status |
|---|---|---|---|
binning primitive MeasureBinRow + Activity/CopyNumber/RepeatAllele/Heteroplasmy rows |
✅ shared vocab, inclusive [min,max], mandatory unresolved, extra=forbid, source_field pointer + source_element rule (0.6), heteroplasmy tissue + legacy-ref guard |
✅ *.parquet via generic materializer |
materialized |
table-level validate_bins(rows) |
✅ per (key…, trait_efo_id) group |
overlap → error, gap → warning, >1 unresolved/group → error |
enforced |
duplicate-row detection (diplotype pair, pgs_id, (pharm variant, drug, genotype, category, annotation_id), allele-function allele, haplotype-defining variant, (source, layer) in the licensing table since RM107) |
✅ per-kind natural key | error (0.4 analog of duplicate-(variant, genotype)) | enforced |
PGx HaplotypeRow / AlleleFunctionRow / DiplotypeRow (+ drug/response/evidence_level) |
✅ | ✅ | materialized |
PharmGKB PharmVariantRow (single-variant drug response, evidence_level 1A…4, per-genotype) |
✅ | ✅ | materialized |
sources.csv licensing path (0.5) |
✅ SourceRow; tri-state permissions; duplicate (source, layer) → error, both modes (RM107); orphan/undeclared + declared-licence warnings (never escalate) |
✅ sources.parquet (in digest); source_signature/licences/attributions/per-layer facets/derived commercial_use → manifest |
✅ refuses (both modes) when annotation-layer terms forbid sale and no declaration is recorded |
VariantRow general axes: requires_callable / acmg_sf / actionability |
✅ (actionability vs ACTIONABILITY_SEED; acmg_sf vs the ACMG SF list in the enricher, 0.5) |
✅ into weights.parquet (tri-state bool round-trip) |
materialized |
RM70 callability on the PGx locus tables: HaplotypeRow.requires_callable / PharmVariantRow.requires_callable (0.7) |
✅ optional tri-state bool, no vocabulary and no validator — the same column VariantRow carries, on the two PGx tables that name a locus. DiplotypeRow refuses it via extra=forbid; callable_from does not travel |
✅ generic table materializer, no compiler change — _polars_type maps it to a nullable pl.Boolean and _scalar_cell reverses None/False as ""/"false" |
materialized — outside every _KEY_FIELDS; unset it is dropped by exclude_none, so no published module's content_signature moves |
RM132 the third citation site: PharmVariantRow.pmid (0.7) |
✅ optional, free-form, validated by spec.validate_pmid_cell — the one grammar every citation pointer routes through, so the PMCID refusal and the [PMID: N] spelling come with it. Read by _cross_check_literature and by the enricher's literature pass, through the derived _CITING_TABLE_KINDS rather than a third hand-kept list |
✅ generic table materializer and generic reverse writer, no compiler change — both derive their column lists from the model | materialized — outside _KEY_FIELDS; unset it is dropped by exclude_none, so no published module's content_signature moves |
RM29a call-confidence cofactor: quality_from + min_quality (0.5) |
✅ shared pointer grammar (source_field/callable_from/quality_from, one validator, namespace-qualifiable since 0.6); finite floor; both-or-neither model rule |
✅ weights.parquet (Utf8 + Float64); absent floor is null, never 0.0 |
materialized |
RM29b clinical cofactor: DiplotypeRow.clinical_context (0.5) |
✅ whitespace-stripped, open (no vocabulary — guideline bodies scope differently) | ✅ generic table materializer, no compiler change | in _TABLE_DUPE_KEYS — disagreeing CPIC contexts coexist as distinct rows |
PGS PgsRow (declared interface; ancestry-validity fields) |
✅ PGS<digits>, ancestry/tier vocab, match_rate_floor∈[0,1] |
✅ | materialized |
reserved namespace (reference_db / callable_element / quality_element) |
✅ specific diagnosis via reject_reserved on top of extra=forbid |
— | reserved |
RM327 tradeoff (0.8) |
✅ refused on every authored table via reject_misplaced (MISPLACED_COLUMN_REASONS), never authored |
✅ meta_resolved.parquet (Boolean, true or null, never false) |
derived |
authoring reference + palette (reference.authoring_reference()/json_schemas()) |
✅ generated from live models (drift-proof) | n/a | shipped (RM8/RM9) |
frozen variant_key identity (base.derive_variant_key) |
✅ stamped once, never re-keyed by resolution (P7); excluded from authoring_reference() |
✅ weights.parquet (compiler-managed) |
shipped |
| rsid↔coord resolution: one-to-many expansion, deterministic order, inject-only consistency check | ✅ ORDER BY; disagreement → warning; non-GRCh38 skipped |
✅ N coord-keyed rows per one-to-many rsid; idempotent | shipped (the DuckDB engine now lives in just-dna-enricher; GRCh38-only; multi-build RM15) |
expansion marker: VariantRow.locus_index + locus_count (0.6, RM87) |
✅ stamped at the expansion loop, authored cells overwritten by _freeze_identity; exclude=True, so no content_signature moves |
✅ weights.parquet (UInt32 ×2, hand-listed in _build_weights); reverse prefers the stored index over its recompute and never re-emits either column into variants.csv |
shipped — locus_count > 1 is the row-level predicate; the positional pass's hard-coded 0 is honest only while those tables never expand (a line on RM65) |
authored overlay overrides.csv (0.7, RM124) |
✅ OverrideRow (AuthoredModel); table and operation are closed vocabularies; reason required; a field outside the named table's columns, or naming its own subject/member column, is refused; a wildcard member is refused for insert and suppress; duplicate (table, subject, member, field) and a key group carrying two operations are errors — all of it in both validate_spec and compile_module |
✅ overrides.parquet, written only when the module carries an overlay — its absence, not its slot, is why no published digest moves (artifact_digest name-sorts); applied to the twelve covered derived tables (OVERRIDABLE_TABLES; clin_sig_concordance.csv and expression_effects.csv joined in 0.7, the Open Targets lane's three in 0.8) before any check reads a row, so the fact signatures and resolution_signature are post-overlay |
shipped — the derived files are pure build products, reverse emits post-overlay tables plus the overlay, and no operation reports its own no-op |
| VCF pointer namespace + cardinality (0.6, RM53/RM54/RM61) | ✅ INFO//FORMAT/ qualifier and the spec's key charset accepted (widening only); _check_vcf_pointers warns in both modes on a bare colliding key and on a spec-multi-valued target with no element rule, aggregated by reason |
✅ source_element → the binning parquets via the generic materializer; round-trips through reverse unchanged |
shipped (source_element on MeasureBinRow; callable_element/quality_element reserved, not built) |
Upgrade derivation (state/booleans → 0.3 axes)¶
state and the ClinVar booleans stay required/authoritative for 0.2 backward-compat (P8). The new
axes are optional, and just_dna_format.derive supplies fallbacks:
- Read-time (non-mutating):
VariantRow.effective_direction/effective_stat_significance/effective_clin_sig/effective_pathogenic/effective_benignreturn the set column, else the derivation — so a legacy 0.1/0.2 row exposes all three axes with no re-publish. - Materializing:
VariantRow.upgraded()fills those axes and trimsstateto{protective, risk, neutral}(kept as a derived mirror ofdirection).needs_upgradeis the signal the marketplacerevalidate/needs_upgradeflow consumes. Both idempotent (P7).
The parquet column is the authored value, and the derivation is not in the artifact¶
The direction column in weights.parquet is a materialized passthrough: whatever the author
wrote, or empty. The compiler never fills it from state, and should not — state='significant'
carries no direction at all, so the derivation refines one from the weight sign, which is a sound
fallback for a reader and a fabricated fact in a published table. Every module authored against 0.2
therefore ships an empty direction, correctly.
The consequence to plan for, if you read the artifact rather than the models: the fallback lives in
Python and does not travel with the parquet. A consumer querying weights.parquet with SQL or
polars sees the empty column and nothing else, so a migration from state to direction reads every
legacy module as directionless. Apply the derivation yourself —
just_dna_format.derive.direction_from_state(state, weight) is a pure leaf function published for
exactly this (it imports nothing from spec, so the marketplace revalidate flow already uses it
that way) — or go through VariantRow.effective_direction, which returns the authored value when
there is one and the derivation when there is not.
Whether an artifact should ever carry the derived axes is open, and it is a design question rather than a patch: the objection is that filling a blank asserts what no curator wrote, not that the bytes move.
Intentionally unimplemented — and why¶
- New computed manifest stats.
Statscarries the 0.2 counts only; no new distributions (bydirection/clin_sig) — no consumer needs them yet. effect_allelestrand reconciliation. Since 0.5 the compiler does check thateffect_allele(and every genotype allele) is one of the alleles the locus actually has — so a strand-flippedA/Gat aC>Tlocus is now caught, because neither allele is present. What is still not done is reconciling it: nothing complements the allele and rewrites it, and the+strand /genome_buildassumption remains documentation rather than an enforced computation. Reporting a flip is a redundancy check; silently flipping it back would be repairing authored data.- Single build — GRCh38-bound.
genome_buildis recorded but only GRCh38 is honored: coordinates are GRCh38, the resolution table'sgenome_buildis checked against the module's, andartifact.digestis GRCh38-relative. A GRCh37/T2T module compiles but is not re-resolved for that build. Legacy-from-implementation, not a principle — build-aware identity is RM15. A no-coord rsid mapping to several loci is expanded to one row per locus (data-agnostic), shipping GRCh38-now. reverse_modulerestores every authored value the artifact carries, and nothing else. It reads the parquets for everything materialized into them, andmanifest.jsonfor every spec field that reaches the artifact only there (RM241; the list isREVERSE_RECONSTRUCTED). This replaced a "parquet only, nevermanifest.json" rule which sounded principled and was how the build came to be hardcoded, and then a narrower rule that read the manifest only for values that could move a digest, under which the display block,weighting,license,authorship,authority_precedence,paneland the README were silently dropped. Absence is handled rather than assumed: no manifest means the old fallbacks, and an explicit argument always wins. What reverse cannot restore is what the artifact does not hold:provenanceand the verification attestation.- Gene-panel materialization, and now the
panel:block itself (RM4). Compiling aGenePanelSpecintoweights.parquetis not deferred any more, it is dropped. The compiler must not create rows no curator wrote — the same objection that bars fillingdirectionfromstate, and it does not depend on the digest. Expansion at compile would also make a module's content depend on an external file, and leavereversechoosing between re-emitting the declaration (rows lost) and the rows (declaration lost); neither is a fixed point (P7). The want is served instead by enricher draft-scaffolding, which already ships:draft-panelwrites the rows, and the author's no-op over the drafted subset is still an authorial act. The rows are authored bytes before the compiler ever sees them.
With that decided, panel: had no reader left — its last one was the enricher's ClinVar
clin_sig cross-check, which now reads the drafted-from release out of the licence row's dataset
column, written by the drafting pass. So it is deprecated in 0.6 and removed at 1.0, with
validate_spec emitting the warning (and compile_module carrying it into
manifest.compilation.warnings, once — compile seeds its warnings from validate's). The block
still loads, still reaches manifest.panel, and still changes nothing else, which is what makes the
deprecation warn-only and the cadence legal.
Deleting it moves no identity, measured rather than argued: reference_examples/apoe_epsilon
with a panel: block appended compiles to the same artifact.digest and the same
content_signature as without it. Auto-removing it on reverse was considered and refused for the
opposite reason — reverse writes module_spec.yaml, so dropping the block there would change that
file's bytes and break the round-trip fixed point for any module carrying it. A warning the author
acts on is the route.
The warning has two branches since S69, and it stopped claiming nothing else is lost.
Both halves were wrong in a way gating alone would not have fixed. GenePanelSpec carries five
fields and SourceRow.dataset is one release label: it cannot hold genes — the denominator,
and the only thing separating this gene is not in the panel from it is in the panel and had
nothing to report (reported as 425 declared genes against gene_count: 298) — it cannot hold
significance, the predicate that makes a panel module's row set reproducible, and it is a name
rather than a digest so it cannot hold reference_sha256. All three now say so in the message.
And the replacement is legitimately absent on a module drafted before the drafter filled
dataset: merge_sources_file is never-clobber, so re-running does not backfill it, and there is
no path from such a module to the state the old sentence assumed. So the check moved behind the
licence rows — a deprecation is legal in a minor only where its audience can act on it (P3), and
whether they can is a fact about a value the pass had not read when the warning fired. With no
filled clinvar/annotation dataset it now says do not delete the block yet, names what to
fill first, and warns that a re-draft will not do it.
Consequences worth knowing¶
weights.parquet/studies.parquetcarry the 0.3 columns + aphasedbit, so a re-compile under this compiler changesartifact.digestfor every module; reproducibility is pinned bycompiler_version, and published versions keep their digest until re-published. A moving digest is therefore not itself a version gate: a new optional column or table is additive and minor-legal, while removal, promotion to required, retyping, or changing what an identity key means is major-only — see ROADMAP § 0.6.- Round-trip is lossless and idempotent (P7):
reverse_module→ recompile preserves every column including phase, and the same spec compiles twice to the same digest. In 0.5 the round-trip is additionally offline — reverse emitsresolution.csv, so recompile needs no reference and no network (regression-tested: DuckDB compile → reverse → no-cache recompile → identical digest). - The
ValidationResult.infochannel carries non-reservedflagsnotes via stdlib logging — the format packages do not depend on Eliot.
Tests: compiler/tests/test_v03*.py (validator, genotype widening, warnings/INFO, materialization,
round-trip/idempotency); test_v04_compile.py (the table kinds); test_resolution_table.py (the
0.5 resolution-table path, digest parity, offline round-trip, strict/best-effort, the deprecation).