Skip to content

just_dna_compiler.compiler.manifest

just_dna_compiler.compiler.manifest

Manifest assembly: the files collected beside the artifact, the stats, the per-table fact blocks, the sources and licence gate, the verification and closure blocks, and _build_manifest.

discover_logs

discover_logs(
    spec_dir: Path, explicit: list[Path] | None
) -> tuple[list[tuple[str, Path]], list[str]]

Which logs ship, as (name in the module dir, source file), and a warning for each left behind.

Read-only, so the pre-flight asks it too (RM367): log_file_not_shipped used to reach an author only from the compile, which also copied as it judged. _collect_logs copies what this returns.

Source code in compiler/src/just_dna_compiler/compiler/manifest.py
def discover_logs(spec_dir: Path, explicit: list[Path] | None) -> tuple[list[tuple[str, Path]], list[str]]:
    """Which logs ship, as `(name in the module dir, source file)`, and a warning for each left behind.

    Read-only, so the pre-flight asks it too (RM367): `log_file_not_shipped` used to reach an author only
    from the compile, which also copied as it judged. `_collect_logs` copies what this returns.
    """
    pairs: list[tuple[str, Path]] = []  # (relative name in module dir, source file)
    warnings: list[str] = []
    if explicit is not None:
        for path in map(Path, explicit):
            try:
                rel = path.relative_to(spec_dir).as_posix()
            except ValueError:
                rel = path.name
            pairs.append((rel, path))
    else:
        candidates = sorted(spec_dir.glob("*.log"))
        logs_dir = spec_dir / "logs"
        # `logs/` is a candidate like any path under it (RM348): a link is judged and never walked,
        # since listing it would already be reading outside the module, and one finding names it.
        if logs_dir.is_symlink() or logs_dir.is_dir():
            candidates.append(logs_dir)
            if not _reached_through_a_link(logs_dir, spec_dir):
                candidates.extend(sorted(logs_dir.rglob("*")))
        for path in candidates:
            # A symlink is judged whatever it points at (a file, a directory, nothing), since discovery
            # never follows one. A real directory is walked, not judged, unless the compile cannot list
            # it, when everything under it stays behind unseen and the directory is what is named. A
            # directory that resolves elsewhere without being a symlink (a Windows junction) is judged
            # as a link. A top-level `*.log` that is a directory is not a log, and stays silent.
            if path.is_symlink() or path.is_file():
                reason = _log_left_behind_reason(path, logs_dir, spec_dir)
            elif path.is_dir() and path.is_relative_to(logs_dir):
                if _reached_through_a_link(path, spec_dir):
                    reason = LOG_NOT_SHIPPED_REASONS["symlink"]
                elif _listable(path):
                    continue
                else:
                    reason = LOG_NOT_SHIPPED_REASONS["unreadable"]
            else:
                continue
            rel = path.relative_to(spec_dir).as_posix()
            if reason is not None:
                warnings.append(
                    CodedWarning("log_file_not_shipped", f"{rel} {LOG_NOT_SHIPPED_PHRASE}: {reason}.")
                )
                continue
            pairs.append((rel, path))

    return pairs, warnings

ancillary_errors

ancillary_errors(
    spec_dir: Path,
    *,
    provenance_file: Path | None = None,
    logo_file: Path | None = None,
    readme_file: Path | None = None,
) -> list[str]

Every refusal the ancillary files can draw, read-only, before anything is written (RM367).

The compile shipped provenance.json, the logo and the readme after the parquets were written, and judged them while copying, so a malformed document refused a compile that had already filled its output directory, and validate_spec never read them at all. A provenance.json that is not UTF-8 escaped as an exception. Both commands ask this now, the compile before output_dir.mkdir(), with the same discovery the collectors use and the compile's own wording.

Source code in compiler/src/just_dna_compiler/compiler/manifest.py
def ancillary_errors(
    spec_dir: Path,
    *,
    provenance_file: Path | None = None,
    logo_file: Path | None = None,
    readme_file: Path | None = None,
) -> list[str]:
    """Every refusal the ancillary files can draw, read-only, before anything is written (RM367).

    The compile shipped `provenance.json`, the logo and the readme **after** the parquets were written,
    and judged them while copying, so a malformed document refused a compile that had already filled its
    output directory, and `validate_spec` never read them at all. A `provenance.json` that is not UTF-8
    escaped as an exception. Both commands ask this now, the compile before `output_dir.mkdir()`, with
    the same discovery the collectors use and the compile's own wording.
    """
    errors: list[str] = []
    provenance = _provenance_source(spec_dir, provenance_file)
    if provenance is not None:
        try:
            ProvenanceDoc.model_validate_json(provenance.read_bytes())
        except ValidationError as exc:
            errors.append(f"{_PROVENANCE_FILE} is invalid: {exc}")
        except OSError as exc:
            errors.append(f"{_PROVENANCE_FILE} could not be read: {exc}")
    logo = _logo_source(spec_dir, logo_file)
    if logo is not None:
        errors.extend(e for e in [_extension_error("logo", logo, LOGO_EXTENSIONS)] if e is not None)
    readme = _readme_source(spec_dir, readme_file)
    if readme is not None:
        errors.extend(e for e in [_extension_error("readme", readme, README_EXTENSIONS)] if e is not None)
    return errors

variant_stats

variant_stats(variants: list[VariantRow]) -> dict[str, Any]

The variants.csv-derived facets of ValidationResult.stats / manifest.stats.

Its own function because it now has two callers, and the second is why: compile_module may discard a row for carrying an unusable symbolic allele (RM5), and the stats were computed by validate_spec before that happened. weights_rows counts the parquet and so is post-drop, so a published manifest claimed a variant_count one higher than the artifact contained — the RM44 class of defect exactly, a manifest number a catalog keys on and cannot check.

Source code in compiler/src/just_dna_compiler/compiler/manifest.py
def variant_stats(variants: list[VariantRow]) -> dict[str, Any]:
    """The `variants.csv`-derived facets of `ValidationResult.stats` / `manifest.stats`.

    Its own function because it now has **two** callers, and the second is why: `compile_module` may
    discard a row for carrying an unusable symbolic allele (RM5), and the stats were computed by
    `validate_spec` before that happened. `weights_rows` counts the parquet and so is post-drop, so a
    published manifest claimed a `variant_count` one higher than the artifact contained — the RM44
    class of defect exactly, a manifest number a catalog keys on and cannot check.
    """
    genes = distinct_spellings((v.gene for v in variants), "gene")
    return {
        "variant_count": len({v.variant_key for v in variants}),
        "unique_rsids": len({v.rsid for v in variants if v.rsid is not None}),
        "gene_count": len(genes),
        "genes": genes,
        "categories": sorted({v.category for v in variants if v.category}),
        # ClinVar/quality flag counts over variant rows (ROADMAP item 5).
        "clinvar_count": sum(1 for v in variants if v.clinvar),
        "pathogenic_count": sum(1 for v in variants if v.pathogenic),
        "benign_count": sum(1 for v in variants if v.benign),
        **clin_sig_tier_counts(variants),
    }

clin_sig_tier_counts

clin_sig_tier_counts(
    variants: list[VariantRow],
) -> dict[str, dict[str, int]]

manifest.stats.clin_sig_counts and clin_sig_variant_counts (RM300).

Every VALID_CLIN_SIG member is a key, zeros included, in sorted order: the vocabulary is closed and fixed per compiler_version, so an absent key would be ambiguous between zero and not counted. The tier is the row's effective clin_sig, so a boolean-only row counts where clin_sig_from_booleans can name one; a row with no derivable tier counts nowhere, and its number is readable as a total minus the sum rather than stored (@derived-not-stored).

Source code in compiler/src/just_dna_compiler/compiler/manifest.py
def clin_sig_tier_counts(variants: list[VariantRow]) -> dict[str, dict[str, int]]:
    """`manifest.stats.clin_sig_counts` and `clin_sig_variant_counts` (RM300).

    Every `VALID_CLIN_SIG` member is a key, zeros included, in sorted order: the vocabulary is closed
    and fixed per `compiler_version`, so an absent key would be ambiguous between zero and not counted.
    The tier is the row's effective `clin_sig`, so a boolean-only row counts where
    `clin_sig_from_booleans` can name one; a row with no derivable tier counts nowhere, and its number
    is readable as a total minus the sum rather than stored (`@derived-not-stored`).
    """
    rows = dict.fromkeys(sorted(VALID_CLIN_SIG), 0)
    keys: dict[str, set[str]] = {tier: set() for tier in rows}
    for v in variants:
        tier = v.effective_clin_sig
        if tier is None:
            continue
        rows[tier] += 1
        keys[tier].add(v.variant_key)
    return {
        "clin_sig_counts": rows,
        "clin_sig_variant_counts": {tier: len(members) for tier, members in keys.items()},
    }

module_stats

module_stats(
    variants: list[VariantRow],
    kind_rows: dict[str, list[Any]] | None = None,
) -> dict[str, Any]

variant_stats plus the gene facets taken over every authored table, not just variants.

PUBLIC, and it exists rather than a second parameter on variant_stats because that function's name is a promise about which table it reads and renaming it would be a major (S14's rule). What the two return differs in exactly two keys.

stats describes the module, and Stats has always said so — "card/detail stats derived from the spec", not from one table of it. variant_stats nevertheless derived genes from variants.csv alone, so a module whose lead table is diplotypes.csv, allele_function.csv, copynumbers.csv or any other gene-bearing kind published gene_count: 0, genes: [] however many of its rows named a gene — and a registry's gene index is fed from that field, so the module was unreachable by a gene search (S57). Measured on reference_examples/cyp2c19_star_alleles/: 1,332 rows carrying gene=CYP2C19 across three tables, and genes: [].

The honest workaround an author was left with was prose in the README, and the dishonest one — inventing an empty variants.csv to be discoverable — is what makes this ours to fix rather than a documentation note.

Only authored kinds count. The genes are subjects.module_subjects' (RM362), whose roster derives from _TABLE_KINDS, which deliberately excludes the derived fact sidecars, so a gene reaching gene_metrics.csv because a pass looked it up never becomes a gene the module claims to be about; the orphan checks and the enricher's gene scope ask the same function, so the three cannot differ.

Source code in compiler/src/just_dna_compiler/compiler/manifest.py
def module_stats(variants: list[VariantRow], kind_rows: dict[str, list[Any]] | None = None) -> dict[str, Any]:
    """`variant_stats` plus the gene facets taken over **every** authored table, not just variants.

    PUBLIC, and it exists rather than a second parameter on `variant_stats` because that function's
    name is a promise about which table it reads and renaming it would be a major (S14's rule). What
    the two return differs in exactly two keys.

    **`stats` describes the module, and `Stats` has always said so** — *"card/detail stats derived from
    the spec"*, not from one table of it. `variant_stats` nevertheless derived `genes` from
    `variants.csv` alone, so a module whose lead table is `diplotypes.csv`, `allele_function.csv`,
    `copynumbers.csv` or any other gene-bearing kind published `gene_count: 0, genes: []` however many
    of its rows named a gene — and a registry's gene index is fed from that field, so the module was
    unreachable by a gene search (S57). Measured on `reference_examples/cyp2c19_star_alleles/`: 1,332
    rows carrying `gene=CYP2C19` across three tables, and `genes: []`.

    The honest workaround an author was left with was prose in the README, and the *dishonest* one —
    inventing an empty `variants.csv` to be discoverable — is what makes this ours to fix rather than a
    documentation note.

    Only authored kinds count. The genes are `subjects.module_subjects`' (RM362), whose roster derives
    from `_TABLE_KINDS`, which deliberately excludes the derived fact sidecars, so a gene reaching
    `gene_metrics.csv` because a pass looked it up never becomes a gene the module claims to be about;
    the orphan checks and the enricher's gene scope ask the same function, so the three cannot differ.
    """
    stats = variant_stats(variants)
    # One gene however it is written (RM363), from the one answer to "which genes is this module about"
    # (RM362): `APOE` and `Apoe` are one entry, in its first spelling.
    ordered = distinct_spellings(
        module_subjects({"variants.csv": variants, **(kind_rows or {})}).genes, "gene"
    )
    stats["gene_count"] = len(ordered)
    stats["genes"] = ordered
    return stats

sources_in_use

sources_in_use(
    resolution_rows: list[ResolutionRow],
    fact_rows: dict[type, list],
) -> set[str]

The sources the module's derived tables name, which _source_checks reads sources.csv against.

One function for both commands (RM367): the compile asked it in a closure the pre-flight had no copy of, so source_row_unused reached an author only at compile. resolution_rows are every row the file holds, active or not: a source row is about the module's files, and --no-resolve does not make the resolution layer's terms stale. Resolution contributes its authority (which link answered, RM33) and each placement witness (RM267), never its source; sources.csv cannot vouch for itself, and a literature row's source names the registry that answered, whose terms are per article (RM46).

Source code in compiler/src/just_dna_compiler/compiler/manifest.py
def sources_in_use(resolution_rows: list[ResolutionRow], fact_rows: dict[type, list]) -> set[str]:
    """The sources the module's derived tables name, which `_source_checks` reads `sources.csv` against.

    One function for both commands (RM367): the compile asked it in a closure the pre-flight had no copy
    of, so `source_row_unused` reached an author only at compile. `resolution_rows` are every row the file
    holds, active or not: a source row is about the module's files, and `--no-resolve` does not make the
    resolution layer's terms stale. Resolution contributes its **authority** (which link answered, RM33)
    and each placement witness (RM267), never its `source`; `sources.csv` cannot vouch for itself, and a
    literature row's `source` names the registry that answered, whose terms are per article (RM46).
    """
    used = {row.authority for row in resolution_rows if row.authority}
    used |= {
        w for row in resolution_rows if row.placement_witness for w in row.placement_witness.split("+") if w
    }
    for model, parsed in fact_rows.items():
        if model in (SourceRow, LiteratureRow):
            continue
        used |= {getattr(r, "source", None) for r in parsed if getattr(r, "source", None)}
    return {s for s in used if s}

build_disagreement_error

build_disagreement_error(
    block: Verification | None,
) -> str | None

The one recorded finding strict refuses on, or None (S78, RM143).

This does not move the strict line, and the distinction is the whole item. strict means reproducible, never right — the compiler has no reference, so it cannot check a coordinate, and a whole file shifted by one base still passes. genome_build_agreement is the exception on internal-consistency grounds rather than correctness ones: a recorded finding there says the module's rows are on a different assembly than the genome_build it declares, which is one authored file contradicting another. Every other recorded finding is a disagreement between the module and an outside archive, where the archive is the stale side often enough that failing a build would have the format arbitrate someone else's dispute — that reasoning is unchanged and covers clinical_significance, reference_allele and the rest.

A fact the toolchain already established, not a check re-run here. The judgement is the enricher's, made against the GRCh37 service the compiler may never call (Principle 2); what changed is that it stops being discarded at the boundary. So the gate keys on a record the enricher wrote — findings > 0 on that one check — and the compiler adds no reference, no network and no opinion of its own.

Silent when no attestation exists, deliberately, and that is not a hole this leaves open: an unverified module is the ordinary case, _read_verification_block says nothing about it on purpose, and refusing there would fail every module that has never been enriched. What this closes is the case where the answer was obtained and thrown away.

A stale attestation is dropped before this sees it, which is the correct order: bytes that moved since the check ran are bytes the check did not judge.

Source code in compiler/src/just_dna_compiler/compiler/manifest.py
def build_disagreement_error(block: Verification | None) -> str | None:
    """The one recorded finding `strict` refuses on, or `None` (S78, RM143).

    **This does not move the strict line, and the distinction is the whole item.** `strict` means
    *reproducible*, never *right* — the compiler has no reference, so it cannot check a coordinate, and
    a whole file shifted by one base still passes. `genome_build_agreement` is the exception on
    internal-consistency grounds rather than correctness ones: a recorded finding there says the
    module's rows are **on a different assembly than the `genome_build` it declares**, which is one
    authored file contradicting another. Every other recorded finding is a disagreement between the
    module and an outside archive, where the archive is the stale side often enough that failing a
    build would have the format arbitrate someone else's dispute — that reasoning is unchanged and
    covers `clinical_significance`, `reference_allele` and the rest.

    **A fact the toolchain already established, not a check re-run here.** The judgement is the
    enricher's, made against the GRCh37 service the compiler may never call (Principle 2); what changed
    is that it stops being discarded at the boundary. So the gate keys on a *record* the enricher
    wrote — `findings > 0` on that one check — and the compiler adds no reference, no network and no
    opinion of its own.

    **Silent when no attestation exists, deliberately**, and that is not a hole this leaves open: an
    unverified module is the ordinary case, `_read_verification_block` says nothing about it on purpose,
    and refusing there would fail every module that has never been enriched. What this closes is the
    case where the answer *was* obtained and thrown away.

    A stale attestation is dropped before this sees it, which is the correct order: bytes that moved
    since the check ran are bytes the check did not judge.
    """
    if block is None:
        return None
    found = [r for r in block.checks if r.check == BUILD_AGREEMENT_CHECK and r.findings]
    if not found:
        return None
    total = sum(r.findings for r in found)
    subjects = sum(r.subjects for r in found)
    return (
        f"strict compile: verification.json records {total} row(s) of {subjects} whose coordinates "
        f"the enricher diagnosed as another assembly's ({BUILD_AGREEMENT_CHECK}). The module declares "
        f"a genome_build its own rows contradict, so the artifact would be internally consistent and "
        f"about the wrong locus. Read the record's `detail` for which rows and the rs-numbers to "
        f"author instead, fix the coordinates and re-run the checks — or compile without strict, "
        f"which builds it and says so."
    )