just_dna_enricher.enrich.orchestration¶
just_dna_enricher.enrich.orchestration ¶
The run itself: enrich() takes the spec lock and _run_enrichment walks the resolver chain,
the verification passes and the commit (RM260 split this out of the former single-file enrich.py).
_run_enrichment is one function, so this module is larger than its siblings; splitting it would be a
logic edit, not a move.
enrich ¶
enrich(
spec_dir: Path,
*,
mode: str = "best_effort",
offline: bool = False,
ensembl_cache: Path | None = None,
clinvar_cache: Path | None = None,
pubmind_cache: Path | None = None,
civic_cache: Path | None = None,
clinvar_gkm_cache: Path | None = None,
use_clinvar: bool = True,
use_gnomad: bool = True,
download: bool = True,
genome_build: str | None = None,
write: bool = True,
mint_vrs: bool = True,
verify_ref: bool = True,
verify_clinsig: bool = True,
verify_rsids: bool = True,
verify_datasets: bool = True,
keep_par_twin: bool = False,
rederive: bool = False,
keep_staging: bool = False,
progress: Callable[[int, int], None] | None = None,
resolver: RsidResolver | None = None,
gnomad_client: Optional[GnomadClient] = None,
grch37_client: Grch37Client | None = None,
release_probes: Mapping[str, ReleaseProbe]
| None = None,
) -> EnrichmentResult
Resolve a spec's variants into resolution.csv. See the module docstring for the chain/modes.
The chain is: existing rows → Ensembl cache → ClinVar cache (use_clinvar, stamps
source="clinvar") → live Ensembl → live gnomAD (use_gnomad, stamps source="gnomad"). Each
later link fills only what the earlier ones missed, so whichever link first knows a variant
decides its alts — and since alts is a fact column, that decides the compiled bytes. The
ordering is therefore chosen so no already-compiled module's artifact.digest can move when a new
link is added. --offline clamps the chain to the two local caches (zero egress).
mint_vrs stamps a ga4gh:VA.… allele id onto every resolved row (see vrs.mint_resolution_rows).
Substitutions mint offline with no dependency; indels need the sequence, so they mint only when the
run is online.
verify_ref checks each authored/resolved ref against the actual reference sequence and reports
disagreements — enrichment is partly validation of authored data, and this tier is the only one
that can perform it (see sequences.verify_reference_alleles). It never repairs, and severity
follows the mode, mirroring the compiler's VRS verify pass: strict treats a mismatch as fatal
(its contract is a reproducible artifact, and a wrong ref can silently mint a different allele
id), while best_effort warns and carries on. Needs sequence access, so it is skipped offline.
A mismatch is then asked one further question, over those rows only: does the coordinate read
as GRCh37 rather than as a wrong cell? That needs the live GRCh37 service, so it is skipped
offline, it is bounded (grch37.DEFAULT_DIAGNOSIS_LIMIT — a systematic wrong build answers the
same on every row), and grch37_client injects the client the way resolver/gnomad_client do.
It adds no flag of its own: verify_ref gates the family and --offline is the egress switch.
verify_clinsig compares each authored clin_sig against the ClinVar snapshot's own. It is
offline-capable (the snapshot is local) and is the one check whose severity does not follow the
mode: it warns in strict too, because failing a compile would make the format arbitrate a
clinical disagreement. See clinical.verify_clin_sig for the full argument.
verify_datasets compares every release this module records having been drafted from
(SourceRow.dataset) against the one that source publishes now, and reports the gap (RM85). It is
rederive's cheap neighbour: both ask has the world moved, one about the rows and one about the
release label, so this is the question you put first — it costs one request per source and
tells you whether the full re-derivation is worth running. It reads and never writes; repairing a
stale label is a re-draft, which is an author's decision and a different command. Severity follows
the mode. --offline makes it unchecked, never up to date, and an unreachable source is never
escalated by strict — nothing an author can edit clears a failed request. release_probes
injects the registry the way resolver/gnomad_client inject theirs.
Every placement is compared with a second authority before it is written (RM267). Whichever
link placed an rsID, the ClinVar snapshot and the ClinVar-GKM snapshot (clinvar_gkm_cache, local
only, never downloaded here) are asked where they place it, and a disagreement withholds the
disputed locus — see placement.compare_placements. Each written row records the witness
asked and its answer (placement_witness, placement_verdict). Subjects already recorded in
resolution.csv are not re-asked (merge-not-clobber); rederive is how a recorded placement meets
the comparison.
keep_par_twin keeps both spellings of a pseudoautosomal locus. By default only the X one is
recorded, because every annotation source uses X and a standard GRCh38 analysis set hard-masks the
Y PAR — see select_par_representative for the probe behind that. Set it for a consumer whose
reference is unmasked. The switch belongs here and could not live on the compiler: resolution.csv
is injected data that travels with the module, so the choice is recorded and
compile → reverse → compile stays a fixed point either way, whereas a compiler flag would not
survive reverse_module rebuilding the spec from parquet alone (Principle 7).
The run is a transaction. What the sources answer is staged to disk as the run proceeds, in the
target's own directory (transaction.ResolutionJournal), and the table itself is written once — at
the gate, through a writer that renames into place. Two properties follow, and the second is the
one the item was really about. A kill at minute 29 leaves the staged answers, so the next run
resumes instead of paying the thirty minutes again. And a refused strict run commits nothing:
every refusal below raises before the write block, so the module is left exactly as it was — a
written promise rather than an accident of statement order. keep_staging leaves the staged
answers behind after a successful commit, for debugging; the default removes them.
The whole run is one read-modify-write window over resolution.csv, so it is held under an
advisory flock on the spec directory (transaction.spec_lock) — two concurrent runs were
last-writer-wins over a merge with neither able to see the other, and a shorter table is
indistinguishable from a module whose author resolved less. A second run refuses rather than
waiting; a platform or filesystem that will not take the lock degrades with a warning that says so.
write=False takes no lock and stages nothing: with nothing written there is no window to exclude,
which keeps the flag meaning one thing everywhere.
rederive re-asks the sources about every subject, including the ones already recorded, and
reports which of them changed value (EnrichmentResult.rederived). That is the drift canary: an
ordinary run gap-fills and never re-asks, so a source that silently revised an answer moves
nothing an author could notice. A recorded subject this run could not ask about keeps its recorded
rows — re-deriving must never be a way to shorten the table, which is the very incident this
transaction exists for.
progress is called (done, total) over subjects, total known before the first call. It
exists for a caller with an idle timeout: what such a caller needs is a keepalive with monotonic
progress, which a phase counter cannot give (a twenty-nine-minute phase emits nothing) and a link
counter cannot promise (its total is not known until resolution finds the links). Exceptions from
the callback are not swallowed; under the transaction that costs nothing, since the staged answers
survive an abort exactly as they survive a kill.
Source code in enricher/src/just_dna_enricher/enrich/orchestration.py
230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 | |