just_dna_compiler.compiler.load¶
just_dna_compiler.compiler.load ¶
Reading a spec directory: module_spec.yaml, the per-table CSV loaders, the citing and binning
row sets, and spec_tables/content_signature over what was read.
SpecError ¶
Bases: ValueError
A module_spec.yaml that could not be loaded — see load_spec.
A ValueError subclass so a caller that already brackets a load with except (OSError,
ValueError), the way read_verification's callers do, keeps working without knowing this type
exists.
load_spec ¶
Load and validate a module_spec.yaml, raising on anything wrong (S74).
The public route to a ModuleSpecConfig. The model has always been exported from
just_dna_format.spec and the only thing that produced one was _load_yaml, underscored — so a
consumer wanting weighting: or authorship: had to yaml.safe_load the file and read a raw
dict, losing the authority-key handling and every diagnosis below, and carrying PyYAML for no
reason except that ours was unreachable. Sibling of just_dna_format.read_manifest and
read_verification: same shape, same contract, one per file a module carries.
It lives here rather than in the format tier because loading it needs pyyaml, and the format
tier is pydantic + cryptography by charter. A consumer already depending on the compiler —
which every caller of validate_spec is — can drop their own PyYAML with this.
authority_keys is inject-only and unchanged: pass
just_dna_format.normalize.IDENTITY_AUTHORITY_KEYS so a registry-stamped
namespace:/owner:/canonical_id: is stripped before validation rather than tripping
extra="forbid". The format applies none by default, and a key outside the injected set still
trips. Which keys were dropped is not reported here — that is validate_spec's .info, and a
caller who needs it wants the validator rather than the loader.
Raises SpecError with every diagnosis joined, where _load_yaml returns them for accumulation.
That difference is the whole reason both exist: validate_spec collects errors from a dozen
sources and reports them together, which is right for a validator and wrong for a loader — a
caller who just wants the object should not have to check a tuple's second element to find out
it got None.
Source code in compiler/src/just_dna_compiler/compiler/load.py
load_csv_rows ¶
load_csv_rows(
path: Path,
row_model: type,
file_label: str,
genome_build: str = DEFAULT_GENOME_BUILD,
) -> tuple[list[Any], list[str], list[str]]
Load a CSV and validate each row against a Pydantic model. Returns (rows, errors, warnings).
Public as of 0.5.1 (RM41), and it was public in practice long before. This is the only correct
way to turn an authored CSV into row models, just-dna-enricher consumes it across a package
boundary in a dozen places, and a downstream consumer wiring the pipeline server-side had the
choice of reaching for a private symbol or re-implementing it. Re-implementing is a trap rather
than a chore, because it is not csv.DictReader plus Model(**row) — it carries the two rules
below, each of which this workspace has already had to fix once:
- an empty cell becomes
None, and the key is kept.MeasureBinRow.measure_kindhas a default, sois_required()isFalse, but the model then receivesNonerather than its default and fails on type. A""where this would have putNoneis a different failure again. genome_buildis told to each row (below), so a loader that omits it mints GRCh38 identities for a GRCh37 module.
_load_csv_rows remains as an alias so no caller breaks.
genome_build is told to each row, not read from it. A coordinate is not absolute, so a row
deriving an identity from one needs the module's assembly — and a pydantic model built from a CSV
dict has no module_spec.yaml in scope. Injecting it here keeps the build declared exactly once
(the yaml) while reaching every row: it is a private attribute, so it is not a column, reaches no
parquet, and moves no digest. See AuthoredModel._genome_build for why per-row or per-CSV
declaration was rejected. Callers that load a build-independent table — the resolution and fact
sidecars, which carry their own genome_build column — leave it at the default and are unaffected.
Source code in compiler/src/just_dna_compiler/compiler/load.py
load_authored_table ¶
Read one authored table that is not a table kind and answer (rows, errors, warnings) (RM357).
The one loader body for every AUTHORED_NON_KIND_TABLES entry, called by validate_spec and
compile_module alike, so the two cannot come to disagree about a row (@validate-refuses-all).
([], [], []) when the file is absent. Otherwise, in this order: the rows parse or they do not; a
present file with no rows is an error, the answer every authored table gives (a header somebody
meant to fill); the duplicate key goes through _validate_table_kind, so it reads as every other
keyed table's does; and the entry's own row_checks run last.
A refused table keeps its rows for the other checks. Both callers refuse on any error before a
parquet could be built from them, but the pre-flight's other checks still read them: a
literature.csv row cited only by a refused meta.csv would otherwise be reported as uncited in
the same run, sending the author to the wrong file. On a duplicate key or a row_checks error every
row parsed, so all of them are kept. On a parse error the set is incomplete, and the rows that
parsed are kept only for a table that cites (NonKindTable.cites): the citing view is the one reader
for which a partial set is better than none (only the malformed row's own citation can then read as
uncited), while the overlay applied half-parsed would raise findings against the tables it corrects.
A parse error returns before the key and the row checks, which read only a whole table.
Not for the derived sidecars: those live under derived/ or beside the spec through
_locate_sidecar, and an authored table has exactly one legal name in exactly one legal place.
Source code in compiler/src/just_dna_compiler/compiler/load.py
load_spec_variants ¶
A spec directory's variants.csv, loaded and re-stamped for the build the module declares.
The other half of RM41. Two enricher checks take rows rather than a spec_dir —
acmg.verify_acmg_sf and identifiers.check_identifiers — unlike every other pass, so a caller
has to do this itself, and doing it right means three steps rather than one: read the declared
build out of module_spec.yaml, inject it into every row, and then re-stamp the identities, since
VariantRow._freeze_identity runs at construction where the yaml is not in scope.
Missing or unreadable yaml falls back to DEFAULT_GENOME_BUILD, matching what compiling that
directory would assume — this is a read-only check helper, not the enrichment path, which refuses
rather than choose a build for a module whose declaration cannot be read (it writes facts back).
Returns (variants, errors, warnings); an absent variants.csv is an empty list and one error,
exactly as load_csv_rows reports it.
Source code in compiler/src/just_dna_compiler/compiler/load.py
load_citing_rows ¶
Every citing table present beside a spec, keyed by CSV name — the annotation rows that
ground their own claim with a pmid (MeasureBinRow.pmid RM47, PharmVariantRow.pmid RM132).
Public because a second tier needs it: the enricher's literature pass has to check these pointers
alongside studies.csv, and its two alternatives were importing a private symbol or hand-keeping a
parallel list of the citing kinds — the RM40/RM41 shape exactly, and the list would go stale on the
next kind that declares the column.
Supersedes load_binning_rows, which stays and still means what it always did: it reads the
binning kinds only, so a caller wanting the citations a module makes wants this one.
Source code in compiler/src/just_dna_compiler/compiler/load.py
load_binning_rows ¶
Every binning table present beside a spec, keyed by CSV name (RM47).
Narrower than load_citing_rows since 0.7 and deliberately kept: a caller asking for the binning
kinds is asking about thresholds, not about citations, and quietly widening what it returns would
hand such a caller PharmVariantRows where it expects MeasureBinRows.
Source code in compiler/src/just_dna_compiler/compiler/load.py
table_citations ¶
Digit-only PMIDs the module's annotation tables cite, de-duplicated — every citing kind
(MeasureBinRow.pmid RM47, PharmVariantRow.pmid RM132).
Takes the whole table-kind map a caller already holds and reads only the citing kinds out of it, so
a caller cannot accidentally hand over a haplotypes.csv (no pmid column) and get an attribute
error. The kind set is derived from the models, never hand-listed, so a kind that gains the column
is read here without an edit.
First-occurrence order rather than sorted, because it feeds emission order downstream (P7), and
normalization goes through extract_pmids so a table pointer and studies.csv cannot drift into
two spellings of one citation.
Source code in compiler/src/just_dna_compiler/compiler/load.py
binning_citations ¶
Digit-only PMIDs the binning tables cite (MeasureBinRow.pmid), de-duplicated.
Narrowed by table_citations since 0.7 the same way load_binning_rows is by load_citing_rows,
and kept for the same reason. Nothing inside the compiler calls it any more: the literature
cross-check must read every citation site, and this one answers a question about thresholds.
Source code in compiler/src/just_dna_compiler/compiler/load.py
spec_tables ¶
The parsed, defaults-folded authored rows content_signature hashes, and the declared build.
PUBLIC, and the reason is that everything finer than a whole-module hash needs these rows and had
no way to get them (S53). content_signature returned only the digest, so a tool answering what
moved between two versions of this module — per table, per row — had to rebuild the mapping
outside, and rebuilding it meant restating two private things: the table roster (_TABLE_KINDS)
and the defaults: fold (_resolve_spec_defaults, _DEFAULTED_VARIANT_FIELDS).
The fold is the part that silently produces a wrong answer, which is why this returns the
finished mapping rather than exporting the pieces. A caller hashing load_csv_rows output directly
gets a different digest from content_signature for the same module: measured on
reference_examples/hfe_hemochromatosis, writing one curator value on every variant row in one
copy and the identical value under defaults: in another, content_signature agrees across the
pair (correct — RM37) while the raw-rows build disagrees, so a per-table comparison built the
obvious way reports twelve changed rows where there are none. Exporting _TABLE_KINDS and
_resolve_spec_defaults separately would hand out three pieces that must be assembled in one
order — load with the declared build injected, fold, then hash — and the order is the easy half to
get wrong. One function that returns the finished mapping cannot be assembled wrongly.
The roster is authored tables only, so the licensing table is outside it: sources.csv /
licensing.csv is hashed by integrity.source_signature instead, and neither renaming it nor
editing a cell in it moves content_signature. Both verified on the same example. That is correct
— the licence layer is its own identity — and it is stated here because it is the one authored,
hand-editable table a licence audit sends an author looking for.
Raises ValueError if a present data CSV fails validation, exactly as content_signature does:
the contract carries over unchanged, because that function is now this one plus the hash.
Source code in compiler/src/just_dna_compiler/compiler/load.py
content_signature ¶
Stable content identity over the raw authored data CSVs — name- and Ensembl-independent.
Reads variants.csv, studies.csv, and any present 0.4 table CSVs, validates each row, and
hashes the normalized + deterministically-sorted rows via
just_dna_format.integrity.content_signature. The data is read as authored (no Ensembl
resolution, no parquet build), so this is cheap and reference-independent — a client can compute
it without recompiling and dedup against a registry, surviving both metadata-strip and a recompile
against a different reference. Raises ValueError if a present data CSV fails validation.
"As authored" means the rows, not the spelling: module_spec.yaml's defaults: block is
folded into each variant row first (_resolve_spec_defaults, RM37), because a value written once
under defaults: and the same value written on every row are the same content.
This is spec_tables plus the hash and nothing else, so a consumer wanting the rows behind the
digest — per-table or per-row work — calls that instead of restating the roster and the fold (S53).