Skip to content

just_dna_enricher.enrich.orchestration

just_dna_enricher.enrich.orchestration

The run itself: enrich() takes the spec lock and _run_enrichment walks the resolver chain, the verification passes and the commit (RM260 split this out of the former single-file enrich.py).

_run_enrichment is one function, so this module is larger than its siblings; splitting it would be a logic edit, not a move.

enrich

enrich(
    spec_dir: Path,
    *,
    mode: str = "best_effort",
    offline: bool = False,
    ensembl_cache: Path | None = None,
    clinvar_cache: Path | None = None,
    pubmind_cache: Path | None = None,
    civic_cache: Path | None = None,
    clinvar_gkm_cache: Path | None = None,
    use_clinvar: bool = True,
    use_gnomad: bool = True,
    download: bool = True,
    genome_build: str | None = None,
    write: bool = True,
    mint_vrs: bool = True,
    verify_ref: bool = True,
    verify_clinsig: bool = True,
    verify_rsids: bool = True,
    verify_datasets: bool = True,
    keep_par_twin: bool = False,
    rederive: bool = False,
    keep_staging: bool = False,
    progress: Callable[[int, int], None] | None = None,
    resolver: RsidResolver | None = None,
    gnomad_client: Optional[GnomadClient] = None,
    grch37_client: Grch37Client | None = None,
    release_probes: Mapping[str, ReleaseProbe]
    | None = None,
) -> EnrichmentResult

Resolve a spec's variants into resolution.csv. See the module docstring for the chain/modes.

The chain is: existing rows → Ensembl cache → ClinVar cache (use_clinvar, stamps source="clinvar") → live Ensembl → live gnomAD (use_gnomad, stamps source="gnomad"). Each later link fills only what the earlier ones missed, so whichever link first knows a variant decides its alts — and since alts is a fact column, that decides the compiled bytes. The ordering is therefore chosen so no already-compiled module's artifact.digest can move when a new link is added. --offline clamps the chain to the two local caches (zero egress).

mint_vrs stamps a ga4gh:VA.… allele id onto every resolved row (see vrs.mint_resolution_rows). Substitutions mint offline with no dependency; indels need the sequence, so they mint only when the run is online.

verify_ref checks each authored/resolved ref against the actual reference sequence and reports disagreements — enrichment is partly validation of authored data, and this tier is the only one that can perform it (see sequences.verify_reference_alleles). It never repairs, and severity follows the mode, mirroring the compiler's VRS verify pass: strict treats a mismatch as fatal (its contract is a reproducible artifact, and a wrong ref can silently mint a different allele id), while best_effort warns and carries on. Needs sequence access, so it is skipped offline.

A mismatch is then asked one further question, over those rows only: does the coordinate read as GRCh37 rather than as a wrong cell? That needs the live GRCh37 service, so it is skipped offline, it is bounded (grch37.DEFAULT_DIAGNOSIS_LIMIT — a systematic wrong build answers the same on every row), and grch37_client injects the client the way resolver/gnomad_client do. It adds no flag of its own: verify_ref gates the family and --offline is the egress switch.

verify_clinsig compares each authored clin_sig against the ClinVar snapshot's own. It is offline-capable (the snapshot is local) and is the one check whose severity does not follow the mode: it warns in strict too, because failing a compile would make the format arbitrate a clinical disagreement. See clinical.verify_clin_sig for the full argument.

verify_datasets compares every release this module records having been drafted from (SourceRow.dataset) against the one that source publishes now, and reports the gap (RM85). It is rederive's cheap neighbour: both ask has the world moved, one about the rows and one about the release label, so this is the question you put first — it costs one request per source and tells you whether the full re-derivation is worth running. It reads and never writes; repairing a stale label is a re-draft, which is an author's decision and a different command. Severity follows the mode. --offline makes it unchecked, never up to date, and an unreachable source is never escalated by strict — nothing an author can edit clears a failed request. release_probes injects the registry the way resolver/gnomad_client inject theirs.

Every placement is compared with a second authority before it is written (RM267). Whichever link placed an rsID, the ClinVar snapshot and the ClinVar-GKM snapshot (clinvar_gkm_cache, local only, never downloaded here) are asked where they place it, and a disagreement withholds the disputed locus — see placement.compare_placements. Each written row records the witness asked and its answer (placement_witness, placement_verdict). Subjects already recorded in resolution.csv are not re-asked (merge-not-clobber); rederive is how a recorded placement meets the comparison.

keep_par_twin keeps both spellings of a pseudoautosomal locus. By default only the X one is recorded, because every annotation source uses X and a standard GRCh38 analysis set hard-masks the Y PAR — see select_par_representative for the probe behind that. Set it for a consumer whose reference is unmasked. The switch belongs here and could not live on the compiler: resolution.csv is injected data that travels with the module, so the choice is recorded and compile → reverse → compile stays a fixed point either way, whereas a compiler flag would not survive reverse_module rebuilding the spec from parquet alone (Principle 7).

The run is a transaction. What the sources answer is staged to disk as the run proceeds, in the target's own directory (transaction.ResolutionJournal), and the table itself is written once — at the gate, through a writer that renames into place. Two properties follow, and the second is the one the item was really about. A kill at minute 29 leaves the staged answers, so the next run resumes instead of paying the thirty minutes again. And a refused strict run commits nothing: every refusal below raises before the write block, so the module is left exactly as it was — a written promise rather than an accident of statement order. keep_staging leaves the staged answers behind after a successful commit, for debugging; the default removes them.

The whole run is one read-modify-write window over resolution.csv, so it is held under an advisory flock on the spec directory (transaction.spec_lock) — two concurrent runs were last-writer-wins over a merge with neither able to see the other, and a shorter table is indistinguishable from a module whose author resolved less. A second run refuses rather than waiting; a platform or filesystem that will not take the lock degrades with a warning that says so. write=False takes no lock and stages nothing: with nothing written there is no window to exclude, which keeps the flag meaning one thing everywhere.

rederive re-asks the sources about every subject, including the ones already recorded, and reports which of them changed value (EnrichmentResult.rederived). That is the drift canary: an ordinary run gap-fills and never re-asks, so a source that silently revised an answer moves nothing an author could notice. A recorded subject this run could not ask about keeps its recorded rows — re-deriving must never be a way to shorten the table, which is the very incident this transaction exists for.

progress is called (done, total) over subjects, total known before the first call. It exists for a caller with an idle timeout: what such a caller needs is a keepalive with monotonic progress, which a phase counter cannot give (a twenty-nine-minute phase emits nothing) and a link counter cannot promise (its total is not known until resolution finds the links). Exceptions from the callback are not swallowed; under the transaction that costs nothing, since the staged answers survive an abort exactly as they survive a kill.

Source code in enricher/src/just_dna_enricher/enrich/orchestration.py
def enrich(
    spec_dir: Path,
    *,
    mode: str = "best_effort",
    offline: bool = False,
    ensembl_cache: Path | None = None,
    clinvar_cache: Path | None = None,
    pubmind_cache: Path | None = None,
    civic_cache: Path | None = None,
    clinvar_gkm_cache: Path | None = None,
    use_clinvar: bool = True,
    use_gnomad: bool = True,
    download: bool = True,
    genome_build: str | None = None,
    write: bool = True,
    mint_vrs: bool = True,
    verify_ref: bool = True,
    verify_clinsig: bool = True,
    verify_rsids: bool = True,
    verify_datasets: bool = True,
    keep_par_twin: bool = False,
    rederive: bool = False,
    keep_staging: bool = False,
    progress: Callable[[int, int], None] | None = None,
    resolver: RsidResolver | None = None,
    gnomad_client: Optional["GnomadClient"] = None,
    grch37_client: Grch37Client | None = None,
    release_probes: Mapping[str, ReleaseProbe] | None = None,
) -> EnrichmentResult:
    """Resolve a spec's variants into `resolution.csv`. See the module docstring for the chain/modes.

    The chain is: existing rows → Ensembl cache → ClinVar cache (`use_clinvar`, stamps
    `source="clinvar"`) → live Ensembl → live gnomAD (`use_gnomad`, stamps `source="gnomad"`). Each
    later link fills only what the earlier ones missed, so whichever link *first* knows a variant
    decides its `alts` — and since `alts` is a fact column, that decides the compiled bytes. The
    ordering is therefore chosen so no already-compiled module's `artifact.digest` can move when a new
    link is added. `--offline` clamps the chain to the two local caches (zero egress).

    `mint_vrs` stamps a `ga4gh:VA.…` allele id onto every resolved row (see `vrs.mint_resolution_rows`).
    Substitutions mint offline with no dependency; indels need the sequence, so they mint only when the
    run is online.

    `verify_ref` checks each authored/resolved `ref` against the actual reference sequence and reports
    disagreements — enrichment is partly *validation* of authored data, and this tier is the only one
    that can perform it (see `sequences.verify_reference_alleles`). It never repairs, and severity
    follows the mode, mirroring the compiler's VRS verify pass: `strict` treats a mismatch as fatal
    (its contract is a reproducible artifact, and a wrong `ref` can silently mint a *different* allele
    id), while `best_effort` warns and carries on. Needs sequence access, so it is skipped offline.

    A mismatch is then asked one further question, over those rows **only**: does the coordinate read
    as GRCh37 rather than as a wrong cell? That needs the live GRCh37 service, so it is skipped
    offline, it is bounded (`grch37.DEFAULT_DIAGNOSIS_LIMIT` — a systematic wrong build answers the
    same on every row), and `grch37_client` injects the client the way `resolver`/`gnomad_client` do.
    It adds no flag of its own: `verify_ref` gates the family and `--offline` is the egress switch.

    `verify_clinsig` compares each authored `clin_sig` against the ClinVar snapshot's own. It is
    offline-capable (the snapshot is local) and is the **one check whose severity does not follow the
    mode**: it warns in `strict` too, because failing a compile would make the format arbitrate a
    clinical disagreement. See `clinical.verify_clin_sig` for the full argument.

    `verify_datasets` compares every release this module records having been drafted from
    (`SourceRow.dataset`) against the one that source publishes now, and reports the gap (RM85). It is
    `rederive`'s cheap neighbour: both ask *has the world moved*, one about the rows and one about the
    release **label**, so this is the question you put first — it costs one request per source and
    tells you whether the full re-derivation is worth running. It reads and never writes; repairing a
    stale label is a re-draft, which is an author's decision and a different command. Severity follows
    the mode. `--offline` makes it `unchecked`, never *up to date*, and an unreachable source is never
    escalated by `strict` — nothing an author can edit clears a failed request. `release_probes`
    injects the registry the way `resolver`/`gnomad_client` inject theirs.

    **Every placement is compared with a second authority before it is written (RM267).** Whichever
    link placed an rsID, the ClinVar snapshot and the ClinVar-GKM snapshot (`clinvar_gkm_cache`, local
    only, never downloaded here) are asked where they place it, and a disagreement withholds the
    disputed locus — see `placement.compare_placements`. Each written row records the witness
    asked and its answer (`placement_witness`, `placement_verdict`). Subjects already recorded in
    `resolution.csv` are not re-asked (merge-not-clobber); `rederive` is how a recorded placement meets
    the comparison.

    `keep_par_twin` keeps both spellings of a pseudoautosomal locus. By default only the X one is
    recorded, because every annotation source uses X and a standard GRCh38 analysis set hard-masks the
    Y PAR — see `select_par_representative` for the probe behind that. Set it for a consumer whose
    reference is unmasked. The switch belongs here and could not live on the compiler: `resolution.csv`
    is injected data that travels with the module, so the choice is *recorded* and
    `compile → reverse → compile` stays a fixed point either way, whereas a compiler flag would not
    survive `reverse_module` rebuilding the spec from parquet alone (Principle 7).

    **The run is a transaction.** What the sources answer is staged to disk as the run proceeds, in the
    target's own directory (`transaction.ResolutionJournal`), and the table itself is written once — at
    the gate, through a writer that renames into place. Two properties follow, and the second is the
    one the item was really about. A kill at minute 29 leaves the staged answers, so the next run
    resumes instead of paying the thirty minutes again. And **a refused `strict` run commits nothing**:
    every refusal below raises before the write block, so the module is left exactly as it was — a
    written promise rather than an accident of statement order. `keep_staging` leaves the staged
    answers behind after a successful commit, for debugging; the default removes them.

    The whole run is one read-modify-write window over `resolution.csv`, so it is held under an
    advisory `flock` on the spec directory (`transaction.spec_lock`) — two concurrent runs were
    last-writer-wins over a merge with neither able to see the other, and a shorter table is
    indistinguishable from a module whose author resolved less. A second run refuses rather than
    waiting; a platform or filesystem that will not take the lock degrades with a warning that says so.
    `write=False` takes no lock and stages nothing: with nothing written there is no window to exclude,
    which keeps the flag meaning one thing everywhere.

    `rederive` re-asks the sources about **every** subject, including the ones already recorded, and
    reports which of them changed value (`EnrichmentResult.rederived`). That is the drift canary: an
    ordinary run gap-fills and never re-asks, so a source that silently revised an answer moves
    nothing an author could notice. A recorded subject this run could not ask about keeps its recorded
    rows — re-deriving must never be a way to shorten the table, which is the very incident this
    transaction exists for.

    `progress` is called `(done, total)` over **subjects**, `total` known before the first call. It
    exists for a caller with an idle timeout: what such a caller needs is a keepalive with monotonic
    progress, which a phase counter cannot give (a twenty-nine-minute phase emits nothing) and a link
    counter cannot promise (its total is not known until resolution finds the links). Exceptions from
    the callback are not swallowed; under the transaction that costs nothing, since the staged answers
    survive an abort exactly as they survive a kill.
    """
    spec_dir = Path(spec_dir)
    # The lock spans the whole window — from the read of the recorded table to the commit — so it is
    # taken here and not inside the run. A `write=False` caller touches no file and takes none.
    with spec_lock(spec_dir, enabled=write, error=EnrichmentError):
        return _run_enrichment(
            spec_dir,
            mode=mode,
            offline=offline,
            ensembl_cache=ensembl_cache,
            clinvar_cache=clinvar_cache,
            pubmind_cache=pubmind_cache,
            civic_cache=civic_cache,
            clinvar_gkm_cache=clinvar_gkm_cache,
            use_clinvar=use_clinvar,
            use_gnomad=use_gnomad,
            download=download,
            genome_build=genome_build,
            write=write,
            mint_vrs=mint_vrs,
            verify_ref=verify_ref,
            verify_clinsig=verify_clinsig,
            verify_rsids=verify_rsids,
            verify_datasets=verify_datasets,
            keep_par_twin=keep_par_twin,
            rederive=rederive,
            keep_staging=keep_staging,
            progress=progress,
            resolver=resolver,
            gnomad_client=gnomad_client,
            grch37_client=grch37_client,
            release_probes=release_probes,
        )