Compile a module¶
A spec directory in, a parquet artifact plus manifest.json out. This page is the operator's order of
operations; COMPILER.md is the reference behind it — the validation ceiling, the full
pipeline, the resolution matrix and the warning-text catalogue a consumer greps.
The spec directory¶
my_module/
├── module_spec.yaml the header: title, name, genome_build, defaults
├── variants.csv one CSV per table kind — only the kinds you use
├── weights.csv
├── meta.csv optional (since 0.8): statements about rows of the other tables
├── resolution.csv injected by the enricher, not authored by hand
└── licensing.csv who the data came from and what you may do with it
licensing.csv is the current spelling; sources.csv is the old one
Both are accepted and read identically, but sources.csv is deprecated and is removed at 1.0,
so write licensing.csv in anything new. A module carrying both is refused. The old name
collided with the source column that means "which link answered" in other tables, which is
what the rename fixes.
The compiled side keeps the old name for the whole 0.x line: licensing.csv becomes
sources.parquet and manifest.sources, because those are inside artifact.digest and published
keys, so renaming them is a removal and waits for the major. A module therefore reads
licensing.csv → sources.parquet → manifest.sources, and that inconsistency is deliberate.
One CSV = one concern. A module includes only the table kinds it needs; there is no schema where
a PGx module carries GWAS columns on every row. just-dna-compiler template <kind> prints a
header-only CSV generated from the live models, and requirements <kind> says what you must supply:
just-dna-compiler template repeat_alleles.csv > repeat_alleles.csv # requirements go to stderr
just-dna-compiler requirements variants.csv
just-dna-compiler scaffold ./my_module --kind variants.csv --kind studies.csv # never overwrites
scaffold writes module_spec.yaml plus one stub CSV per --kind; with no --kind it writes the
header file alone. Run it again with another --kind to add a table. The stubs carry a <<REPLACE>>
placeholder wherever a person has to decide, so the directory cannot compile until they have:
error: variants.csv line 2 []: Value error, unreplaced template placeholder '<<REPLACE>>' in VariantRow row: conclusion, genotype, rsid, state. Replace the value, or delete the row if you do not need it.
describe <kind> emits the full machine description — columns, options, requirements — and
reference prints the authoring reference for every model at once. Those four commands are the
schema's own voice; a hand-written column table beside them goes stale on the next release.
For the same material as a page rather than as stdout, the site carries one table reference per kind — what the table is grained on, who writes which cell, what it becomes in the artifact, and the symptom when it goes wrong. Every one is generated from the live models, so a new table kind gets a page by construction; they are built rather than committed, so these are absolute links:
| Table | Page |
|---|---|
variants.csv |
tables/variants/ |
studies.csv |
tables/studies/ |
licensing.csv |
tables/licensing/ |
overrides.csv |
tables/overrides/ |
resolution.csv |
tables/resolution/ |
meta.csv (since 0.8) |
tables/meta/ |
model_organisms.csv (since 0.8) |
tables/model_organisms/ |
Each carries a #columns anchor for the column list alone. The pages are named for the current
spelling, so sources.csv's page is tables/licensing/.
Validate first¶
validate refuses everything compile refuses, except the checks that need resolved coordinates, and
it writes nothing. Use it as the fast loop while authoring.
Since 0.8 it takes the compile's switches, --strict, --resolve/--no-resolve and
--ba1-threshold, so pass it what the compile will get. A green validate --strict then means
compile --strict will not refuse:
hint is the other read-only tool and the more interesting one: it inspects your authored rows and
reports what is wrong, what the model will rewrite, and what is deliberately left to you. It writes
nothing — the corrected text goes to stdout for you to use or ignore.
hint reads one table at a time: name its kind, then pass the file with --file or a row inline with
--row. On reference_examples/longevity_studies' variants.csv it rewrites nothing and lists the
cells it leaves to the author, each with the check that would be made vacuous if a tool filled it:
info: [chrom] chrom is left to the author on purpose: compiler.resolution._verify (rsid vs coordinate) compares it against a source, and filling it from that same source would make the check vacuous
…
variants.csv: 5 row(s), 0 normalized, 0 advisory, 6 finding(s)
A hint never fills a cell that a cross-examining check reads. That is not a limitation to work around; a tool that filled it would be manufacturing the agreement the check exists to test.
Compile¶
| Flag | What it does |
|---|---|
--strict |
all-or-nothing: refuse rather than emit an artifact with unresolved positions |
--no-resolve |
do not use the injected resolution.csv at all |
--compression |
parquet codec, default zstd. Changes artifact.digest, never content_signature |
--compiled-by |
the provenance tag verify checks by default (e.g. marketplace-server) |
--strip-identity / --authority-key |
drop authority-owned identity keys on the way out |
--ensembl-reference |
record which reference the resolution table came from, in manifest.compilation.ensembl_reference (since 0.8) |
--ba1-threshold |
the allele frequency above which a pathogenic row draws the ACMG BA1 warning, default 0.05 (since 0.8) |
The compiler never reaches the network. Every coordinate it did not get from you
came out of the injected resolution.csv; with nothing injected it skips resolution and says so in a
warning rather than going to look.
What --strict actually decides¶
--strict is about unresolved rows, not about warnings in general. Most checks report in both
modes, because a report and a refusal answer different questions — and a few are documented as
never escalating under strict, the ClinVar clin_sig cross-check among them: a source re-curating
its own call is not grounds to refuse someone else's build.
Where the compiler cannot establish something, it withholds. It does not guess, and it does not write the opposite value.
The refusals new in 0.8¶
A spec 0.7 accepted can be refused by 0.8 in these cases. Every one below was produced by editing a
scratch copy of a reference example and running validate.
Under --strict only, a resolution.csv that places one rsID two ways, under its own key and
under a positional key at another coordinate (rsid_placements_disagree). It is what a table enriched
before the Ensembl snapshot switch and again after it, without --rederive, looks like:
error: rs10455872: resolution.csv holds two placement sets for this one rsID, under rs10455872 at 6:160589086 A>G [release ensembl-variation-2026_07_13-vep116.0]; and under 6:160589099:A:G at 6:160589099 A>G [release ensembl-variation-2026_07_13-vep116.0]. … Re-derive the table (`just-dna-enricher enrich <spec-dir> --rederive`) or delete resolution.csv and enrich again. Strict refuses until the table places each rsID one way: an artifact compiled from it can publish one identity at two places.
The same text is a warning without --strict. An rsID left unresolved because two authorities place it
as two different events is refused under --strict through the existing unresolved gate.
In both modes, a pmid cell holding only a DOI, a half-stated pair among the 0.8 columns, and an
HLA allele off IPD-IMGT/HLA's grammar on allele_function.csv:
error: studies.csv line 2 [pmid]: Value error, pmid names DOI(s) ['10.1101/2025.03.30.25324899'] and no PubMed ID. … Write the DOI in `doi`; if the work has a PubMed record, write its PMID here (`just-dna-enricher hint citation --doi 10.1101/2025.03.30.25324899` looks it up), and if it has none, write 'NONE' here. Got: '10.1101/2025.03.30.25324899'
error: studies.csv line 3 []: Value error, stratum='sex' is set and stratum_level is empty — a stratum is written as both or neither …
error: allele_function.csv line 2 [allele]: Value error, HLA-DRB1 is an HLA gene, so its allele must follow IPD-IMGT/HLA nomenclature: … Got: '*1501'
pmid = NONE with the work's DOI in doi is the accepted way to cite a work with no PubMed record,
such as a preprint. reference_examples/longevity_studies cites one that way and validates.
INTEGRATION_0_8.md § 2 step 13 is the complete list.
New tables in 0.8¶
Two authored tables are new, and studies.csv gained columns. All of them are optional. A module
carrying neither new table keeps those two out of its digest, but any module with a studies.csv
gets a new artifact.digest on a 0.8 recompile, because the parquet gained the columns.
meta.csv: one row per statement about rows of the other tables, picked by a selector such asvariants[variant_key=rs10455872]. A one-member row grades its rows (tier, on the scalemodule_spec.yamlnames intier_scale:, andage_of_action); a row joining members with&is a finding about the combination, which can replace its members' own findings when all of them match (replacesin the parquet). It compiles tometa.parquetandmeta_resolved.parquet.model_organisms.csv: model-organism lifespan findings about a human gene, compiled verbatim tomodel_organisms.parquet. Annotation only: it reaches no weight, tier or meta row.studies.csvgainedsample_size,sample_unit,n_cases,phenotype_class,ancestry,age_min/age_max/age_of,stratum/stratum_level,proxy_rsidandeffect_unit.requirements studies.csvlists them.
Each has a reference example that compiles as it stands:
compiled: apoe_klvs_meta
digest: sha256:db38e8e209f6624a139a59009ab4600c6560246027321ed7fc2884313ab16847
content_signature: sha256:79305ea43a8773e6325f73441b4c0858c3a7669565f8cdaf9f3e13a840a32663
resolution_mode: best_effort fully_resolved: True (over 6 variant row(s))
resolution_signature: sha256:576b454c5050039c6c36f70cbfb1e5c2e91ffb38af5097167662a5636c283c00
Its output directory holds meta.parquet and meta_resolved.parquet beside the per-table parquets.
reference_examples/model_organisms compiles to a model_organisms.parquet, and
reference_examples/longevity_studies exercises the new studies.csv columns.
Use a compiled module shows what to read out of them.
Read the manifest¶
{
"artifact": {
"digest": "sha256:c86809…",
"files": [{"name": "haplotypes.parquet", "sha256": "sha256:2beb75…", "size": 12302}, "…"]
},
"content_signature": "sha256:343333…",
"compilation": {
"compile_success": true,
"compiled_by": null,
"compiler_version": "just-dna-compiler 0.8.2",
"toolchain": {"polars": "2.1.0", "python": "3.14.8"},
"warnings": ["…"]
}
}
Every warning the compile emitted is carried into compilation.warnings, so the consumer reads the
same sentence you did. Those texts are an API — they are catalogued in
COMPILER.md § Warning texts precisely because integrations grep them.
Sign it¶
keygen prints the public key alongside the private one it writes, so you never have to derive it:
The key is unencrypted PKCS#8 and keygen refuses to overwrite. It bootstraps a key; it is not a
key-management system, and a publishing key belongs in whatever secret store you already run.
Sign the compiled artifact, then check it the way a consumer will:
just-dna-compiler sign ./out --private-key ./keys/private.pem
just-dna-compiler verify ./out --public-key 'GGorZF2yf8gKfK1XlPoh1c4toqVZmsxoNm3iIasrhrQ=' --no-require-marketplace
verified: out
digest: sha256:c868099845720b90b62336b8edd4348b72da24c2a80e97be49e1bf1b63d1feec files: 3
signature: verified against the pinned key
The signature covers artifact.digest, which is already a hash over every file's hash — so one
signature covers every byte, and re-signing after an edit is impossible to forget: the digest moves
and the old signature stops verifying. Sign last, because anything that rewrites a parquet invalidates
it.
Close the authoring phase¶
Authoring has an end. close writes a closure block into the module's verification.json, naming
the hash of module_spec.yaml and the authored CSVs as they stand right now, with newlines
normalized. Edit any of them afterwards and the hash moves, the compiler drops the closure, and the
module is open again. That is the point.
Three things about it worth knowing before you run it:
validatewill never do it for you, however cleanly it passes. A record stamped by whatever happened to run says only that something ran.--private-keyis what turns someone closed this into this party closed this, using the same keysignuses on a compiled artifact.- It refuses on a spec that does not validate, and does not refuse on warnings. An unresolved rsID or an ungrounded threshold is a legitimate state to call finished.
reverse deliberately refuses to reproduce somebody else's closure into a spec it reconstructed — it
holds no authority to declare another party's work finished, and it says so at the time.
Reverse and recompile¶
just-dna-compiler reverse ./out ./spec-again
just-dna-compiler compile ./spec-again ./out-again # reproduces ./out
compile → reverse → compile reproduces the module, or --strict refuses. The round trip is lossless
and idempotent, and tests pin it — which is what makes an artifact a safe thing to hold: the authored
form is always recoverable from it.
Row order is preserved through the round trip because parquet bytes depend on it. Column order and cell formatting are normalized rather than preserved — that asymmetry is intended.
Where to go next¶
- COMPILER.md — the reference: validation ceiling, pipeline, resolution matrix, deterministic ordering, every warning text.
- Resolve, draft and enrich — producing the
resolution.csvthis page injects. - CLI reference — every command and flag, generated from the tools themselves. It is written at build time from the live command tree, so it exists on the site and not in this repository.