Use a compiled module¶
You have a module directory and want to report on somebody's genotype. This page is the consumer's side of the seam: what is in the artifact, what you must check before trusting it, and the two obligations the format puts on you rather than on the module.
The authoritative field-by-field reference is SCHEMAS.md; this is the working order.
What you received¶
apoe_epsilon/
├── haplotypes.parquet one parquet per table kind the module uses
├── diplotypes.parquet
├── sources.parquet the licence table (authored as licensing.csv)
├── manifest.json identity, build, licence, stats, digests, warnings
└── README.md optional, authored
A module carries only the kinds it uses. There is no fixed file list and no empty parquet standing
in for a domain the module has nothing to say about, so read manifest.json first and let it tell you
what is there.
Three parquets form the SNP core (variants, weights, studies) when a module is variant-shaped;
a PGx module like this one is haplotype-shaped instead. manifest.artifact.files[] is the complete
list, and it is the same list verify hashes.
1. Verify before you install¶
Exit 0 or exit 1, nothing to parse. It re-hashes every file in artifact.files[], recomputes
artifact.digest over the set, and checks the Ed25519 signature against the key you pin. Omit
--public-key and it will report signature: absent rather than fail — pinning is how you demand one.
Add --no-require-marketplace for anything you compiled yourself; the default demands
compiled_by=marketplace-server.
If you would rather not shell out, just-dna-format is the dependency-light way to do the same thing
in-process — that is the whole reason the tier exists, and it costs you pydantic plus cryptography.
The two digests answer different questions¶
| Field | Covers | Moves when |
|---|---|---|
artifact.digest |
the emitted bytes | a recompile, a codec change, a compiler upgrade |
content_signature |
the claims | an authored value changes — and not otherwise |
Cache on artifact.digest. Decide whether the module actually says something new with
content_signature. Confusing the two is the most common integration bug here, and
the FAQ leads with it.
2. Read the tables¶
Plain parquet, no custom container. Anything that reads parquet reads a module.
import json
from pathlib import Path
import polars as pl
module = Path("apoe_epsilon")
manifest = json.loads((module / "manifest.json").read_text())
build = manifest["genome_build"] # 'GRCh38' — it lives HERE, in no parquet column
haplotypes = pl.read_parquet(module / "haplotypes.parquet")
genome_build is a manifest field and not a parquet column
Every coordinate in every table is in the build the manifest names, and nothing in the parquet repeats it. A pipeline that joins two modules must read both manifests — mixing a GRCh37 module into a GRCh38 join produces coordinates that look fine and are wrong by thousands of bases.
start is the 1-based VCF position, not a 0-based offset. Do not subtract one.
Identity¶
variant_key is the join key, minted as rsID → VRS allele id → chrom:start:ref[:alts], in that
order. It is a string, it is stable across recompiles, and it is what the derived tables key on.
An indel's coordinates are left-aligned only where the enricher filled them
The compiler cannot normalize an indel: left-alignment reads the reference sequence around the
event, and the compile path has none. Since 0.8 the enricher writes a coordinate it fills from
an rsID in its leftmost parsimonious spelling, ALT by ALT — the spelling bcftools norm -m -any -f
<reference> writes — so those rows join a normalized VCF exactly. Offline the enricher cannot
left-align, but it still trims each ALT against its ref (RM381), so a split-snapshot site is written
per ALT, never in the snapshot's padded multi-ALT frame. A coordinate the module's
author wrote keeps its author's spelling, and VCF allows several spellings of one event: the
Ensembl snapshot often writes an indel one base right of the left-aligned form your caller emits
(CCR5-Δ32, rs333: 3:46373453 ACAGT…CCAGA>A against 3:46373452 TACAGT…CCAG>T in a GATK or
DeepVariant VCF). resolution.csv's left_aligned says which rows were checked, and a module
built before 0.8 carries no such guarantee at all.
Join indels on vrs_id where you can mint VRS, or normalize both sides before joining. Every
variant parquet carries the GA4GH VRS allele id per ALT (weights.parquet vrs_id, a list
parallel to alts), minted over the fully justified allele, so both spellings of one event carry
one id and a different event carries another. Otherwise left-align and trim both sides the way
bcftools norm -f <reference> does, or join on the rsID where the row has one. Substitutions are
unaffected, since a single-base change has only one spelling. This was
RM270.
The genotype cell can be in a third spelling. A row's genotype keeps its author's frame
while ref/alts take the resolver's, so they can differ on one row: before RM270
reference_examples/shox_par1 carried rs1569493663 as genotype C/CAG (ClinVar's frame) beside
ref=AGAG, alts=[AG, AGAGAG] (Ensembl's). Comparing the genotype's alleles with your VCF's by
string equality misses such a row. Reduce both sides first with
just_dna_format.alleles.parsimony_reduce, which is pure Python with no dependencies: it strips the
flank a set of alleles shares, so {C, CAG} and {AGAG, AG} both become {'', 'AG'}. It cannot
reconcile a homozygous indel call, because one allele has no flank to strip (see
alleles.event_profile). This is RM31's consumer half, which until 2026-09-27 lived only in its
closing entry.
3. The join contract — two obligations that are yours¶
A module supplies the annotation; you supply the measurement. Two things the module cannot do for you:
Absence is not reference¶
A conforming consumer MUST distinguish a covered reference call from a no-call before asserting any reference or absence interpretation, and MUST NOT read "absent from a variant-only callset" as hom-reference.
Absence from a variant-only callset means the site matched the reference or was never callable. Collapsing those two fabricates a confident reference genotype, and it fails in the dangerous direction: for a carrier row, or for the reassurance that a pathogenic variant is absent, it is the difference between screened negative and not screened.
The module tells you where this bites:
| Column | Says |
|---|---|
requires_callable |
the absence of this variant is the informative call. Without callability data, withhold the conclusion rather than assert the reference one. Blank is not false |
callable_from |
which VCF field carries the proof (FORMAT/DP, FORMAT/GQ, FORMAT/FT) — a pointer, never an expression |
quality_from + min_quality |
the floor below which what was seen is not good enough to act on. A floor you cannot evaluate is unknown, never satisfied |
unresolved (binning tables) |
the no-call sentinel. A missing measurement selects it, and never the lowest or reference bin |
Every one of those columns has three states, not two: true, false, and unknown — and a blank cell is
unknown, never false. When the answer is unknown, withhold it. Do not report it and do not report
its opposite. The same rule runs through the rest of the schema, so a consumer that collapses blanks
to false anywhere will be wrong in the same direction everywhere.
A row is about a (locus, genotype) pair, and some pairs were never authored¶
An rsID that resolves onto several loci is paired with each of them, so K authored genotypes at one
rsID become K×N rows. ClinVar's reciprocal duplication/deletion pairs are the usual instance: a
genotype written for the duplication lands beside the deletion's ref as a well-formed reference
homozygote that the author never claimed anything about.
Joining by position is unaffected — nothing matches those rows, which is why this went unnoticed for a release. Classifying rows, counting them, or asking "what does this module say about someone who is reference here" reads them as statements, and they are false ones. A reporting consumer found this with 2,579 such rows queued into a genome's pathogenic section, caught before rendering.
manifest.compilation.expanded_keys and expanded_rows tell you whether a module contains expansion
rows and how many. Both are None where resolution did not run — never 0, which would claim someone
looked and found none.
4. Check what you are allowed to do with it¶
Licensing is data, carried in the module's own licence table and summarised in the manifest — it is never a compiler flag and never a mode. Three axes, and they are independent:
declared_use— what the module was compiled for. Three states, not a boolean.redistribution— recorded, and deliberately not gated yet.- unknown terms — a source may publish no licence at all. That warns; it does not gate.
A module built from ClinPGx, CPIC or PharmVar is research-only and no-sale. If you are building a commercial product, read the per-source rows rather than the headline licence: a host's terms are not its contents' terms, so the licence at the top is a floor each record may override.
In a compiled module those rows are sources.parquet and manifest.sources — the authored file is
named licensing.csv (older spelling sources.csv), but the artifact keeps the old name for the whole
0.x line, because both are inside artifact.digest and renaming them waits for the major.
5. What 0.8 added to the artifact¶
Most additions are a new parquet, a new column or a new manifest key, which a 0.7 reader ignores. The
exceptions are the old-reader breaks in INTEGRATION_0_8.md § 1 (a 0.7
just-dna-format refuses a manifest carrying a warning code it does not know), so upgrade the format
tier first. The additions a reporting consumer meets:
meta.parquetandmeta_resolved.parquet, from an authoredmeta.csv. Ameta.parquetrow whosereplacesis true is a finding about a combination: when every member matches the sample, it replaces the findings of the rows in itsmatches.meta_resolved.parquetgives each row'stier,age_of_actionand flags after precedence, looked up by(table, key, sex, drug). The manifest'stier_scalenames the scaletiergrades on.model_organisms.parquet: model-organism lifespan findings about a gene. Annotation only, never a weight.studies.parquetgains the study's size, ancestry, age band, stratum, LD proxy and effect unit. ApmidofNONEmeans the work has no PubMed record, anddoithen names it: skipNONEwhen you parse PubMed ids.weights.parquetgains derived cells:effect_unit,eur_only_evidence, and threeoutcome_*cells that are true or null and never false.resolution_loci, on the variant and positional parquets, keeps the lookup's whole answer per locus, includingsource_release(the snapshot or service release that answered) andsource_loci. Read its fields by name. It is not a join key: join on the row's ownchrom/start/ref/alts, or onvrs_id.- Open Targets (
locus_to_gene.parquet,colocalisations.parquet,consequences.parquet) andld_pairs.parquetappear only in a module the enricher's passes filled.
Run against three reference examples compiled into the working directory
(just-dna-compiler compile reference_examples/<name> ./<name>):
import polars as pl
# A combination row, and what it replaces
meta = pl.read_parquet("apoe_klvs_meta/meta.parquet")
print(meta.filter(pl.col("replaces")).select("members", "tier", "match_count", "matches"))
# The per-row statements after precedence: look a row up by (table, key, sex, drug)
resolved = pl.read_parquet("apoe_klvs_meta/meta_resolved.parquet")
print(resolved.select("table", "key", "sex", "drug", "tier", "age_of_action").head(3))
# A study with no PubMed record is cited by its DOI
studies = pl.read_parquet("longevity_studies/studies.parquet")
print(studies.filter(pl.col("pmid") == "NONE").select("rsid", "pmid", "doi", "ancestry"))
# Which release answered each locus, kept inside resolution_loci
weights = pl.read_parquet("longevity_studies/weights.parquet")
loci = weights.select("variant_key", "resolution_loci").explode("resolution_loci").unnest("resolution_loci")
print(loci.select("variant_key", "chrom", "start", "source_release").unique(maintain_order=True).head(3))
┌─────────────────────────────────┬──────┬─────────────┬─────────────────────────────────┐
│ members ┆ tier ┆ match_count ┆ matches │
╞═════════════════════════════════╪══════╪═════════════╪═════════════════════════════════╡
│ diplotypes[gene=KL, haplotype_… ┆ C ┆ 4 ┆ [{0,"diplotypes",["KL", "KL-VS… │
└─────────────────────────────────┴──────┴─────────────┴─────────────────────────────────┘
┌──────────┬───────────────────────┬──────┬──────┬──────┬───────────────┐
│ table ┆ key ┆ sex ┆ drug ┆ tier ┆ age_of_action │
╞══════════╪═══════════════════════╪══════╪══════╪══════╪═══════════════╡
│ variants ┆ ["rs10455872", "A/A"] ┆ null ┆ null ┆ A ┆ MIDLIFE │
│ variants ┆ ["rs10455872", "A/G"] ┆ null ┆ null ┆ A ┆ MIDLIFE │
│ variants ┆ ["rs10455872", "G/G"] ┆ null ┆ null ┆ A ┆ MIDLIFE │
└──────────┴───────────────────────┴──────┴──────┴──────┴───────────────┘
┌────────────┬──────┬─────────────────────────────┬───────────┐
│ rsid ┆ pmid ┆ doi ┆ ancestry │
╞════════════╪══════╪═════════════════════════════╪═══════════╡
│ rs75571981 ┆ NONE ┆ 10.21203/rs.3.rs-6869504/v1 ┆ ["EAS"] │
└────────────┴──────┴─────────────────────────────┴───────────┘
┌─────────────┬───────┬───────────┬─────────────────────────────────┐
│ variant_key ┆ chrom ┆ start ┆ source_release │
╞═════════════╪═══════╪═══════════╪═════════════════════════════════╡
│ rs10455872 ┆ 6 ┆ 160589086 ┆ ensembl-variation-2026_07_13-v… │
│ rs429358 ┆ 19 ┆ 44908684 ┆ ensembl-variation-2026_07_13-v… │
│ rs646776 ┆ 1 ┆ 109275908 ┆ ensembl-variation-2026_07_13-v… │
└─────────────┴───────┴───────────┴─────────────────────────────────┘
(Shapes and dtypes trimmed from polars' output.) INTEGRATION_0_8.md § 2 lists every change by surface, and SCHEMAS.md holds the columns.
Where to go next¶
- SCHEMAS.md — every model, every column, the vocabularies and the hash family.
- MODULE_LIFECYCLE.md — where a module comes from and what a second version moves.
- FAQ.md — settled questions, keyed by the question you would actually ask.
- Upgrading to 0.8 — the 0.7 → 0.8 surface delta, the upgrade order, and the old-reader breaks.
- Upgrading to 0.7 (archived) — the 0.6.6 → 0.7 surface delta, including the one break for an old reader. Read it first if you are still on 0.6.