Skip to content

just_dna_compiler.compiler.tables

just_dna_compiler.compiler.tables

The table registry: which CSV kinds a module may author, their dedup keys, the parquet roster, the authored/derived input files, and the model-driven helpers that turn rows into polars frames and back into CSV cells.

NonKindTable dataclass

NonKindTable(
    csv: str,
    parquets: tuple[str, ...],
    model: type[AuthoredModel],
    row_checks: Callable[[Sequence[Any]], list[str]]
    | None = None,
)

One authored table that is deliberately not a _TABLE_KINDS entry (RM357).

What the three have in common is the category the constants above argue one by one: a human writes every row, so the CSV is input (raw-byte hashed into manifest.inputs, inside content_signature); the rows are keyed, so a duplicate is refused through _TABLE_DUPE_KEYS; and the table is never a lead table, so a directory carrying only it is not a module and none of its parquets is in LEAD_PARQUETS. Membership here is that last statement, so there is no lead flag to set, and it is also why no gene here is a gene the module's findings are about: manifest.stats.genes reads variants.csv and the table kinds, and the enricher's gene scope follows that published set by excluding every table registered here. Whether an entry cites is read off its model (cites), and every entry is draftable (draft.DRAFTABLE), so neither is a flag either.

Every list that names these tables is derived from AUTHORED_NON_KIND_TABLES rather than kept by hand, and one loader body reads them all (load.load_authored_table). Before RM357 each had its own copy of the loader and about ten lists named them, and the copy that dropped its rows on a duplicate key made a literature.csv row cited only by meta.csv read as uncited in the same run.

cites property

cites: bool

A citation site exactly when the model declares pmid (RM132's rule, _CITING_TABLE_KINDS).

study_coordinate_key

study_coordinate_key(
    chrom: str | None, start: int | None, ref: str | None
) -> str | None

The chrom:start:ref half of "a study names a variant by ANY shared identifier", or None.

One expression for the two places that ask the question — _cross_validate_studies' orphan check and the study facts weights.parquet derives (0.8) — so the two match a study to a variant by the same rule. The other half is plain rsid equality. They match against different SETS, by design: the orphan check counts every row of the module's locus roster (subjects.module_subjects, RM362: variants.csv and every positional table kind, which hold every row a meta.csv member reaches), while weights.parquet has a row only for a variants.csv variant. So a study on an ε-defining SNP of a diplotype-scored APOE is not an orphan and feeds no weights.parquet row.

Source code in compiler/src/just_dna_compiler/compiler/tables.py
def study_coordinate_key(chrom: str | None, start: int | None, ref: str | None) -> str | None:
    """The `chrom:start:ref` half of "a study names a variant by ANY shared identifier", or `None`.

    One expression for the two places that ask the question — `_cross_validate_studies`' orphan check
    and the study facts `weights.parquet` derives (0.8) — so the two match a study to a variant by the
    same rule. The other half is plain `rsid` equality. They match against different SETS, by design:
    the orphan check counts every row of the module's locus roster (`subjects.module_subjects`, RM362:
    `variants.csv` and every positional table kind, which hold every row a `meta.csv` member reaches),
    while `weights.parquet` has a row only for a `variants.csv` variant. So a study on an ε-defining SNP of a diplotype-scored APOE is not an orphan
    and feeds no `weights.parquet` row.
    """
    return derive_variant_key(None, chrom, start, ref) if chrom is not None else None

same_row

same_row(row: Any, earlier: list[Any]) -> Any | None

The first of earlier — rows already sharing row's key — that row is the same row as, or None.

The one predicate every "same row" site asks: the compile/validate refusal, the drafting merge and the hints lint. Bucketing by the key tuple first and comparing within the bucket is forced, not chosen: since RM356 the relation is not transitive (a gene-less *2 collides with UGT1A1's and with UGT1A6's, which do not collide with each other), so no hashable value can stand for it.

Source code in compiler/src/just_dna_compiler/compiler/tables.py
def same_row(row: Any, earlier: list[Any]) -> Any | None:
    """The first of `earlier` — rows already sharing `row`'s key — that `row` is the same row as, or
    `None`.

    The one predicate every "same row" site asks: the compile/validate refusal, the drafting merge and
    the hints lint. Bucketing by the key tuple first and comparing within the bucket is forced, not
    chosen: since RM356 the relation is not transitive (a gene-less `*2` collides with UGT1A1's and with
    UGT1A6's, which do not collide with each other), so no hashable value can stand for it."""
    return next((candidate for candidate in earlier if not _distinct_when_stated(row, candidate)), None)

table_bindings

table_bindings() -> dict[str, tuple[str, ...]]

csv -> the parquet(s) it becomes, assembled from the registries and never named by hand.

Public because two surfaces outside this module need it and both would otherwise keep a copy: the docs site's generated per-table reference, and test_artifact_parquet_bindings.py, which asserts the union of the values is exactly ARTIFACT_PARQUETS. That equality is what makes the map total — a new table kind whose parquet is written by a literal at its own write site fails the test instead of quietly missing a reference page.

Four registries, each owning its own slice: SNP_CORE_PARQUETS (the core, where variants.csv fans out to two), _TABLE_KINDS (the optional authored kinds), _FACT_TABLES (the derived facts) and AUTHORED_NON_KIND_TABLES (the authored tables that are not kinds, where meta.csv fans out to two like variants.csv).

Source code in compiler/src/just_dna_compiler/compiler/tables.py
def table_bindings() -> dict[str, tuple[str, ...]]:
    """`csv -> the parquet(s) it becomes`, assembled from the registries and never named by hand.

    Public because two surfaces outside this module need it and both would otherwise keep a copy: the
    docs site's generated per-table reference, and `test_artifact_parquet_bindings.py`, which asserts the
    union of the values is exactly `ARTIFACT_PARQUETS`. That equality is what makes the map **total** — a
    new table kind whose parquet is written by a literal at its own write site fails the test instead of
    quietly missing a reference page.

    Four registries, each owning its own slice: `SNP_CORE_PARQUETS` (the core, where `variants.csv`
    fans out to two), `_TABLE_KINDS` (the optional authored kinds), `_FACT_TABLES` (the derived facts)
    and `AUTHORED_NON_KIND_TABLES` (the authored tables that are not kinds, where `meta.csv` fans out to
    two like `variants.csv`).
    """
    bound: dict[str, tuple[str, ...]] = dict(SNP_CORE_PARQUETS)
    for csv_name, parquet, _model in _TABLE_KINDS:
        bound[csv_name] = (parquet,)
    for csv_name, parquet, _model in _FACT_TABLES:
        bound[csv_name] = (parquet,)
    for table in AUTHORED_NON_KIND_TABLES:
        bound[table.csv] = table.parquets
    return bound

authored_input_entries

authored_input_entries(spec_dir: Path) -> list[FileEntry]

The authored files a module is made of, hashed for the verification binding.

Public because two tiers must agree on it byte for byte (the RM41 lesson): the compiler recomputes the binding from this set when it decides whether to publish manifest.verification, and the enricher hashes the identical set into the attestation's module_hash so a later compile can tell whether the spec has been edited since the checks ran. A private symbol here would leave the enricher choosing between reaching into a private name and re-implementing the list, and a re-implementation is a place for the two to drift — at which point every attestation this workspace writes reads as stale to its own compiler.

Authored files only, which is the boundary just_dna_format.verification argues at length: the derived sidecars carry per-run noise (fetched_at) that would invalidate an attestation on a re-enrichment that changed nothing anyone claimed.

The bytes are newline-normalized, and manifest.inputs[] is deliberately not (RM82). This docstring said the compiler hashes these into manifest.inputs; it never did — that field is filled independently, by file_entries(spec_dir, _INPUT_FILES) over the raw bytes, and the two were only ever equal by coincidence of computing the same thing. Since 0.6 they are two different questions asked of one file set. manifest.inputs[] and artifact.digest answer are these the exact bytes, so they follow every byte, line endings included. The binding answers is this still the module those checks were put against, and an editor rewriting \r\n as \n changes no value, so it must not un-close a module. The asymmetry is the decision, not an inconsistency to tidy: see integrity.newline_normalized_file_entry for the transform and where it stops.

Source code in compiler/src/just_dna_compiler/compiler/tables.py
def authored_input_entries(spec_dir: Path) -> list[FileEntry]:
    """The authored files a module is made of, hashed for the verification binding.

    Public because **two tiers must agree on it byte for byte** (the RM41 lesson): the compiler
    recomputes the binding from this set when it decides whether to publish `manifest.verification`,
    and the enricher hashes the identical set into the attestation's `module_hash` so a later compile
    can tell whether the spec has been edited since the checks ran. A private symbol here would leave
    the enricher choosing between reaching into a private name and re-implementing the list, and a
    re-implementation is a place for the two to drift — at which point every attestation this
    workspace writes reads as stale to its own compiler.

    **Authored files only**, which is the boundary `just_dna_format.verification` argues at length:
    the derived sidecars carry per-run noise (`fetched_at`) that would invalidate an attestation on a
    re-enrichment that changed nothing anyone claimed.

    **The bytes are newline-normalized, and `manifest.inputs[]` is deliberately not (RM82).** This
    docstring said the compiler hashes *these* into `manifest.inputs`; it never did — that field is
    filled independently, by `file_entries(spec_dir, _INPUT_FILES)` over the raw bytes, and the two
    were only ever equal by coincidence of computing the same thing. Since 0.6 they are two different
    questions asked of one file set. `manifest.inputs[]` and `artifact.digest` answer *are these the
    exact bytes*, so they follow every byte, line endings included. The binding answers *is this still
    the module those checks were put against*, and an editor rewriting `\\r\\n` as `\\n` changes no
    value, so it must not un-close a module. The asymmetry is the decision, not an inconsistency to
    tidy: see `integrity.newline_normalized_file_entry` for the transform and where it stops.
    """
    return newline_normalized_file_entries(Path(spec_dir), list(_INPUT_FILES))