Use a compiled module¶
You have a module directory and want to report on somebody's genotype. This page is the consumer's side of the seam: what is in the artifact, what you must check before trusting it, and the two obligations the format puts on you rather than on the module.
The authoritative field-by-field reference is SCHEMAS.md; this is the working order.
What you received¶
apoe_epsilon/
├── haplotypes.parquet one parquet per table kind the module uses
├── diplotypes.parquet
├── manifest.json identity, build, licence, stats, digests, warnings
└── README.md optional, authored
A module carries only the kinds it uses. There is no fixed file list and no empty parquet standing
in for a domain the module has nothing to say about, so read manifest.json first and let it tell you
what is there.
Three parquets form the SNP core (variants, weights, studies) when a module is variant-shaped;
a PGx module like this one is haplotype-shaped instead. manifest.artifact.files[] is the complete
list, and it is the same list verify hashes.
1. Verify before you install¶
Exit 0 or exit 1, nothing to parse. It re-hashes every file in artifact.files[], recomputes
artifact.digest over the set, and checks the Ed25519 signature against the key you pin. Omit
--public-key and it will report signature: absent rather than fail — pinning is how you demand one.
Add --no-require-marketplace for anything you compiled yourself; the default demands
compiled_by=marketplace-server.
If you would rather not shell out, just-dna-format is the dependency-light way to do the same thing
in-process — that is the whole reason the tier exists, and it costs you pydantic plus cryptography.
The two digests answer different questions¶
| Field | Covers | Moves when |
|---|---|---|
artifact.digest |
the emitted bytes | a recompile, a codec change, a compiler upgrade |
content_signature |
the claims | an authored value changes — and not otherwise |
Cache on artifact.digest. Decide whether the module actually says something new with
content_signature. Confusing the two is the most common integration bug here, and
the FAQ leads with it.
2. Read the tables¶
Plain parquet, no custom container. Anything that reads parquet reads a module.
import json
from pathlib import Path
import polars as pl
module = Path("apoe_epsilon")
manifest = json.loads((module / "manifest.json").read_text())
build = manifest["genome_build"] # 'GRCh38' — it lives HERE, in no parquet column
haplotypes = pl.read_parquet(module / "haplotypes.parquet")
genome_build is a manifest field and not a parquet column
Every coordinate in every table is in the build the manifest names, and nothing in the parquet repeats it. A pipeline that joins two modules must read both manifests — mixing a GRCh37 module into a GRCh38 join produces coordinates that look fine and are wrong by thousands of bases.
start is the 1-based VCF position, not a 0-based offset. Do not subtract one.
Identity¶
variant_key is the join key, minted as rsID → VRS allele id → chrom:start:ref[:alts], in that
order. It is a string, it is stable across recompiles, and it is what the derived tables key on.
An indel's coordinates are in its source's spelling, not left-aligned
The format names no normalization convention for indels, and the compiler cannot apply one:
left-alignment reads the reference sequence around the event, and the compile path has none. So an
insertion or deletion keeps the spelling its source used, and VCF allows several spellings of one
event. The Ensembl snapshot the enricher resolves rsIDs from often writes an indel one base right
of the left-aligned form your caller emits. CCR5-Δ32 (rs333) is one example:
3:46373453 ACAGT…CCAGA>A in the module, 3:46373452 TACAGT…CCAG>T in a GATK or DeepVariant VCF.
A position join that requires POS and REF to agree, as it should for indels, then misses the
row with no error. Normalize both sides before joining indels (left-align and trim, the way
bcftools norm -f <reference> does), or join on the rsID where the row has one. This is
RM270.
Substitutions are unaffected, since a single-base change has only one spelling.
The genotype cell can be in a third spelling. A row's genotype keeps its author's frame
while ref/alts take the resolver's, so they can differ on one row: reference_examples/shox_par1
carries rs1569493663 as genotype C/CAG (ClinVar's frame) beside ref=AGAG,
alts=[AG, AGAGAG] (Ensembl's). Comparing the genotype's alleles with your VCF's by string
equality misses it. Reduce both sides first with just_dna_format.alleles.parsimony_reduce, which
is pure Python with no dependencies: it strips the flank a set of alleles shares, so {C, CAG} and
{AGAG, AG} both become {'', 'AG'}. It cannot reconcile a homozygous indel call, because one
allele has no flank to strip (see alleles.event_profile). This is RM31's consumer half, which
until 2026-09-27 lived only in its closing entry.
3. The join contract — two obligations that are yours¶
A module supplies the annotation; you supply the measurement. Two things the module cannot do for you:
Absence is not reference¶
A conforming consumer MUST distinguish a covered reference call from a no-call before asserting any reference or absence interpretation, and MUST NOT read "absent from a variant-only callset" as hom-reference.
Absence from a variant-only callset means the site matched the reference or was never callable. Collapsing those two fabricates a confident reference genotype, and it fails in the dangerous direction: for a carrier row, or for the reassurance that a pathogenic variant is absent, it is the difference between screened negative and not screened.
The module tells you where this bites:
| Column | Says |
|---|---|
requires_callable |
the absence of this variant is the informative call. Without callability data, withhold the conclusion rather than assert the reference one. Blank is not false |
callable_from |
which VCF field carries the proof (FORMAT/DP, FORMAT/GQ, FORMAT/FT) — a pointer, never an expression |
quality_from + min_quality |
the floor below which what was seen is not good enough to act on. A floor you cannot evaluate is unknown, never satisfied |
unresolved (binning tables) |
the no-call sentinel. A missing measurement selects it, and never the lowest or reference bin |
Every one of those columns has three states, not two: true, false, and unknown — and a blank cell is
unknown, never false. When the answer is unknown, withhold it. Do not report it and do not report
its opposite. The same rule runs through the rest of the schema, so a consumer that collapses blanks
to false anywhere will be wrong in the same direction everywhere.
A row is about a (locus, genotype) pair, and some pairs were never authored¶
An rsID that resolves onto several loci is paired with each of them, so K authored genotypes at one
rsID become K×N rows. ClinVar's reciprocal duplication/deletion pairs are the usual instance: a
genotype written for the duplication lands beside the deletion's ref as a well-formed reference
homozygote that the author never claimed anything about.
Joining by position is unaffected — nothing matches those rows, which is why this went unnoticed for a release. Classifying rows, counting them, or asking "what does this module say about someone who is reference here" reads them as statements, and they are false ones. A reporting consumer found this with 2,579 such rows queued into a genome's pathogenic section, caught before rendering.
manifest.compilation.expanded_keys and expanded_rows tell you whether a module contains expansion
rows and how many. Both are None where resolution did not run — never 0, which would claim someone
looked and found none.
4. Check what you are allowed to do with it¶
Licensing is data, carried in the module's own licence table and summarised in the manifest — it is never a compiler flag and never a mode. Three axes, and they are independent:
declared_use— what the module was compiled for. Three states, not a boolean.redistribution— recorded, and deliberately not gated yet.- unknown terms — a source may publish no licence at all. That warns; it does not gate.
A module built from ClinPGx, CPIC or PharmVar is research-only and no-sale. If you are building a commercial product, read the per-source rows rather than the headline licence: a host's terms are not its contents' terms, so the licence at the top is a floor each record may override.
In a compiled module those rows are sources.parquet and manifest.sources — the authored file is
named licensing.csv (older spelling sources.csv), but the artifact keeps the old name for the whole
0.x line, because both are inside artifact.digest and renaming them waits for the major.
Where to go next¶
- SCHEMAS.md — every model, every column, the vocabularies and the hash family.
- MODULE_LIFECYCLE.md — where a module comes from and what a second version moves.
- FAQ.md — settled questions, keyed by the question you would actually ask.
- Upgrading to 0.7 — the surface delta, including the one break for an old reader.