Skip to content

just_dna_compiler.compiler.parquets

just_dna_compiler.compiler.parquets

Building the SNP-core and frequency parquets: weights, annotations, studies, frequencies.

resolution_loci

resolution_loci(
    key: str | None,
    resolution: dict[str, list[ResolutionRow]],
    genome_build: str,
    *,
    table_read: bool = False,
) -> list[dict[str, Any]] | None

The loci resolution.csv files under key on this build, in locus_index order, or None.

Why a parquet carries them. reverse_module rebuilds resolution.csv from parquets, so a fact no parquet holds does not come back, resolution_signature moves, and whatever the compile said about it goes silent on the second lap. Two kinds of row are keyed into the table without holding its answer in their own columns: a study's LD proxy, which is usually not a module variant (RM331), and a variant row the fill left alone (RM364) — one whose authored coordinate the table contradicts, one whose key names several loci none of which was taken, a locus dropped as unable to host the genotype, and every row of a module on a build the fill skips. Rebuilding those from the row's own cells wrote the author's coordinate back as a resolution fact.

Positioned loci only: a not_found row carries no fact for reverse to restore.

An empty list says the table filed nothing under this key (RM451). None already means "the row's own cells answer" (withhold_reproduced_loci, or a parquet compiled before the column, or a module that read no table), and reverse rebuilds from the cells there. A coordinate row a module's table never filed (authored after the last enrich) was read the same way, so reverse invented a table row for it and the second lap read a row the first did not. A module that read a table therefore says [] for a key under which it holds no positioned locus on this build, and reverse writes nothing for it. table_read is whether a table was read at all (InjectedResolution.consulted), never whether it has rows: reverse hands an empty table back as a header-only file, and the second lap must say [] where the first did.

Source code in compiler/src/just_dna_compiler/compiler/parquets.py
def resolution_loci(
    key: str | None,
    resolution: dict[str, list[ResolutionRow]],
    genome_build: str,
    *,
    table_read: bool = False,
) -> list[dict[str, Any]] | None:
    """The loci `resolution.csv` files under `key` on this build, in `locus_index` order, or `None`.

    **Why a parquet carries them.** `reverse_module` rebuilds `resolution.csv` from parquets, so a fact
    no parquet holds does not come back, `resolution_signature` moves, and whatever the compile said
    about it goes silent on the second lap. Two kinds of row are keyed into the table without holding
    its answer in their own columns: a study's LD proxy, which is usually not a module variant (RM331),
    and a variant row the fill left alone (RM364) — one whose authored coordinate the table contradicts,
    one whose key names several loci none of which was taken, a locus dropped as unable to host the
    genotype, and every row of a module on a build the fill skips. Rebuilding those from the row's own
    cells wrote the author's coordinate back as a resolution fact.

    Positioned loci only: a `not_found` row carries no fact for reverse to restore.

    **An empty list says the table filed nothing under this key (RM451).** `None` already means "the
    row's own cells answer" (`withhold_reproduced_loci`, or a parquet compiled before the column, or a
    module that read no table), and reverse rebuilds from the cells there. A coordinate row a module's
    table never filed (authored after the last enrich) was read the same way, so reverse invented a
    table row for it and the second lap read a row the first did not. A module that read a table
    therefore says `[]` for a key under which it holds no positioned locus on this build, and reverse
    writes nothing for it. `table_read` is
    whether a table was read at all (`InjectedResolution.consulted`), never whether it has rows: reverse
    hands an empty table back as a header-only file, and the second lap must say `[]` where the first did.
    """
    if key is None:
        return None
    loci = sorted(
        (
            r
            for r in resolution.get(key, [])
            if r.genome_build == genome_build and r.chrom is not None and r.start is not None
        ),
        key=lambda r: r.locus_index,
    )
    if not loci:
        # Nothing positioned under the key on this build, a `not_found` row included: reverse writes
        # none of it back, so the next lap reads the key as unfiled too, and both say `[]` (RM451).
        return [] if table_read else None
    return [{name: getattr(r, name) for name in RESOLUTION_LOCUS_FIELDS} for r in loci]

cell_fact

cell_fact(cells: Mapping[str, Any]) -> tuple[Any, ...]

What reverse's cell rebuild writes for one row, over every CELL_FACT_FIELDS column, in order.

cells holds the row's value for each of those columns, a resolution_loci locus holds them as stored, and the two compare equal exactly when the cell rebuild would write the locus back. Indexed: every caller states each column, the positional one by row.get because _resolution_record reads a kind's missing column as empty too. The defect this replaced (RM389) was a column not compared at all; one compared as empty against a locus holding a value keeps that locus, the safe direction. locus_index absent is 0, the ordinal reverse gives a row that names none.

Source code in compiler/src/just_dna_compiler/compiler/parquets.py
def cell_fact(cells: Mapping[str, Any]) -> tuple[Any, ...]:
    """What `reverse`'s cell rebuild writes for one row, over every `CELL_FACT_FIELDS` column, in order.

    `cells` holds the row's value for each of those columns, a `resolution_loci` locus holds them as
    stored, and the two compare equal exactly when the cell rebuild would write the locus back. Indexed:
    every caller states each column, the positional one by `row.get` because `_resolution_record` reads
    a kind's missing column as empty too. The defect this replaced (RM389) was a column not compared at
    all; one compared as empty against a locus holding a value keeps that locus, the safe direction.
    `locus_index` absent is 0, the ordinal reverse gives a row that names none.
    """
    return tuple(
        (cells[name] or 0) if name == "locus_index" else _as_cell(cells[name]) for name in CELL_FACT_FIELDS
    )

withhold_reproduced_loci

withhold_reproduced_loci(
    cells: list[list[dict[str, Any]] | None],
    keys: list[str | None],
    facts: list[tuple[Any, ...] | None],
) -> list[list[dict[str, Any]] | None]

cells with a key's loci nulled wherever its rows' own cells already rebuild them exactly (RM364).

facts[i] is what reverse's cell rebuild writes for row i (None where it writes nothing, a row with no coordinate). Where the set of those facts over a key equals the facts of the loci the table files under it, the cells answer and the column stays null, so a module the fill resolved cleanly carries nothing new, and its artifact.digest matches the deprecated ensembl_cache path's, which reads no table. Everywhere else, which is every row the fill left alone, the loci are kept.

Since RM380 a locus holding provenance is kept too — authority, a placement witness, left_aligned, source_release, an ambiguous status, RM375's verdict — because no cell holds those and withholding the locus would drop them on the round trip. So the "carries nothing new" case is now a table with no provenance (a hand-filled or reversed-from-old table, or the deprecated path's), and an enriched module carries its loci on every keyed row: a parquet column, priced near zero (P9).

Source code in compiler/src/just_dna_compiler/compiler/parquets.py
def withhold_reproduced_loci(
    cells: list[list[dict[str, Any]] | None],
    keys: list[str | None],
    facts: list[tuple[Any, ...] | None],
) -> list[list[dict[str, Any]] | None]:
    """`cells` with a key's loci nulled wherever its rows' own cells already rebuild them exactly (RM364).

    `facts[i]` is what `reverse`'s cell rebuild writes for row `i` (`None` where it writes nothing, a row
    with no coordinate). Where the set of those facts over a key equals the facts of the loci the table
    files under it, the cells answer and the column stays null, so a module the fill resolved cleanly
    carries nothing new, and its `artifact.digest` matches the deprecated `ensembl_cache` path's, which
    reads no table. Everywhere else, which is every row the fill left alone, the loci are kept.

    **Since RM380 a locus holding provenance is kept too** — `authority`, a placement witness,
    `left_aligned`, `source_release`, an `ambiguous` status, RM375's verdict — because no cell holds those
    and withholding the locus would drop them on the round trip. So the "carries nothing new" case is
    now a table with no provenance (a hand-filled or reversed-from-old table, or the deprecated path's),
    and an enriched module carries its loci on every keyed row: a parquet column, priced near zero (P9).
    """
    by_key: dict[str, set[tuple[Any, ...]]] = {}
    for key, fact in zip(keys, facts, strict=True):
        if key is not None:
            facts_of_key = by_key.setdefault(key, set())
            if fact is not None:
                facts_of_key.add(fact)
    reproduced = {
        key
        for key, cell in zip(keys, cells, strict=True)
        if key is not None
        and cell
        # A row's cells hold no provenance (RM375's verdict, RM267's witness, RM373's release, RM387's
        # `source_loci`, ...), so a locus holding any is never rebuilt from the cells (RM380). `False` is
        # a value: `is not None`.
        and not any(_locus_holds_more_than_cells(locus) for locus in cell)
        and by_key[key] == {cell_fact(locus) for locus in cell}
    }
    return [None if key in reproduced else cell for key, cell in zip(keys, cells, strict=True)]

with_resolution_loci

with_resolution_loci(
    table: DataFrame,
    resolution: dict[str, list[ResolutionRow]],
    genome_build: str,
    *,
    table_read: bool = False,
) -> pl.DataFrame

A positional table's parquet with resolution_loci: the loci filed under each row's key (RM364).

Keyed on the row's stored variant_key, the key the positional fill looked the row up by and the key reverse files it under. Kept wherever the row's own cells would not rebuild them (withhold_reproduced_loci): a row the fill left alone still carries what the table said about it. An empty resolution (no table, or --no-resolve) leaves the column null.

Source code in compiler/src/just_dna_compiler/compiler/parquets.py
def with_resolution_loci(
    table: pl.DataFrame,
    resolution: dict[str, list[ResolutionRow]],
    genome_build: str,
    *,
    table_read: bool = False,
) -> pl.DataFrame:
    """A positional table's parquet with `resolution_loci`: the loci filed under each row's key (RM364).

    Keyed on the row's stored `variant_key`, the key the positional fill looked the row up by and the
    key `reverse` files it under. Kept wherever the row's own cells would not rebuild them
    (`withhold_reproduced_loci`): a row the fill left alone still carries what the table said about it.
    An empty `resolution` (no table, or `--no-resolve`) leaves the column null.
    """
    keys = table["variant_key"].to_list()
    cells = [resolution_loci(key, resolution, genome_build, table_read=table_read) for key in keys]
    # `row.get` where reverse's cell rebuild reads `row.get` too (`_resolution_record`): a kind with no
    # `rsid`, `ref` or `vrs_id` column writes those empty, and a positional row's ordinal is always 0.
    facts = [
        None
        if row["chrom"] is None or row["start"] is None
        else cell_fact({**{name: row.get(name) for name in CELL_FACT_FIELDS}, "locus_index": 0})
        for row in table.iter_rows(named=True)
    ]
    cells = withhold_reproduced_loci(cells, keys, facts)
    return table.with_columns(pl.Series("resolution_loci", cells, dtype=pl.List(RESOLUTION_LOCUS_DTYPE)))