just_dna_enricher.opentargets_build¶
just_dna_enricher.opentargets_build ¶
Open Targets Platform snapshot builder (0.8, RM333): one pinned release, trimmed to what a module joins.
The source is the Platform's bulk output on EBI's FTP, served over https
(https://ftp.ebi.ac.uk/pub/databases/opentargets/platform/<release>/output/<dataset>/part-*.parquet).
Five datasets are read, all Spark output split into parts (probes/LANES_0_8.md § 1 has their schemas,
row counts and sizes as measured on 2026-11-04):
| dataset | what the snapshot keeps |
|---|---|
variant |
the allele table (variants.parquet) and every per-transcript consequence (consequences-*.parquet) |
credible_set |
GWAS credible sets (credible_sets.parquet) and their members' posteriors (credible_set_members-*.parquet) |
l2g_prediction |
the locus-to-gene score per (study locus, gene) (l2g.parquet) |
colocalisation |
the GWAS-left × molecular-QTL-right pairs, with the right side's gene and tissue (colocalisations-*.parquet) |
study |
read only to name a coloc row's right-hand gene and tissue |
Flattened at build, read by duckdb at run time (@duckdb-vs-polars). Every list column the runtime
pass would otherwise unnest per query (locus, transcriptConsequences) is exploded here, once. The
one list kept as a list is an allele's rsIds (variants.parquet's rs_ids): the pass asks only
whether it holds one rsID, for a module row that states no ALT, and never unnests it.
Alleles come from columns, never from the id. variantId is <chr>_<pos>_<ref>_<alt> with
underscores (the croissant says hyphens) and, for 2,114 long alleles, an OTVAR_… hash; the snapshot
keeps the id only as an internal join key and the alleles from referenceAllele/alternateAllele.
What is trimmed, and why each cut is not a judgement. Credible sets and members are kept for GWAS
studies only, because RM239's posterior and RM333's L2G are about a GWAS locus. Coloc rows are kept
when the left side is a GWAS credible set and the right side is not (rightStudyType != 'gwas'): a
GWAS-GWAS pair says two traits share a locus and names no gene, which is ~72% of the dataset. No row is
dropped for its value: no score, posterior or h4 threshold is applied anywhere.
A partial build says so. parts limits how many parts of each dataset are read, for a test-sized
or bandwidth-limited build; release.json then records parts_read beside parts_total, and the
runtime pass warns that an absent coloc or L2G row may mean "not read" rather than "not published".
Builder-only (polars is a guarded import), as every builder here.
OpenTargetsBuildError ¶
Bases: RuntimeError
The Open Targets lane failed; the builder's own type, so the lane adapter can catch it as this lane.
OpenTargetsUnavailable ¶
Bases: OpenTargetsBuildError
The FTP did not answer: the fetch failed, not the data.
OpenTargetsRetired ¶
Bases: OpenTargetsUnavailable, EndpointRetired
The source answered HTTP 410: it retired an Open Targets release file (RM361, audit round 2).
Its lane's OpenTargetsUnavailable, so every handler still catches it; marked EndpointRetired, so
the reason is retired and the message says the file moved and to upgrade, never that the
host was down.
OpenTargetsBuildResult
dataclass
¶
OpenTargetsBuildResult(
out_dir: Path,
release: str,
row_counts: dict[str, int] = dict(),
parts_read: dict[str, int] = dict(),
parts_total: dict[str, int] = dict(),
)
What a build wrote, counted per table.
list_parts ¶
list_parts(
dataset: str,
listing_dir: Path,
*,
release: str = OPENTARGETS_RELEASE,
base_url: str = DEFAULT_OPENTARGETS_BASE_URL,
) -> list[str]
The part filenames of one dataset, read off the FTP's https directory listing.
The listing is fetched through net.stream_to_file, so it is retried and translated like every
other download here; the parts are sorted so a partial build always reads the same ones.
Source code in enricher/src/just_dna_enricher/opentargets_build.py
download_release ¶
download_release(
dest: Path,
*,
release: str = OPENTARGETS_RELEASE,
base_url: str = DEFAULT_OPENTARGETS_BASE_URL,
parts: int | None = None,
workers: int = 8,
) -> dict[str, int]
Mirror the five datasets' parts into dest/<dataset>/, returning each dataset's total part count.
A part already on disk with a parquet footer is not fetched again, so an interrupted mirror resumes.
parts keeps the first N parts of each dataset (sorted by name), for a test-sized or
bandwidth-limited build; build_snapshot records how many were read against the total. The
croissant (the release's only licence and version marker) is fetched beside them.
Source code in enricher/src/just_dna_enricher/opentargets_build.py
build_snapshot ¶
build_snapshot(
source: Path,
out_dir: Path,
*,
release: str = OPENTARGETS_RELEASE,
base_url: str = DEFAULT_OPENTARGETS_BASE_URL,
) -> OpenTargetsBuildResult
Flatten a mirrored release (source/<dataset>/part-*.parquet) into the snapshot at out_dir.
Every table is sorted on its key before it is written, so a rebuild from the same parts writes the
same parquet bytes (Principle 7); release.json's built_at is the only per-run byte.
Source code in enricher/src/just_dna_enricher/opentargets_build.py
subset_snapshot ¶
subset_snapshot(
snapshot: Path,
out_dir: Path,
alleles: list[tuple[str, int, str, str]],
studies: list[str] | None = None,
) -> dict[str, int]
Cut a built snapshot down to the loci of alleles ((chrom, position, ref, alt)), as a snapshot.
For a test fixture or a regional snapshot: every table keeps exactly the rows the runtime pass would
read for those alleles (their variants and consequences, their credible-set memberships, the sets
they are members of, those sets' L2G and coloc rows), so a pass over the subset answers as it would over the
whole release for those variants. studies keeps every credible-set head of those studies too,
so "the study has credible sets, and the variant is in none" answers on the subset as it does on
the release. Every other allele of the release at a wanted allele's (chrom, position) is kept too,
with all of its rows, so the pass's lookup for a module row with no ALT sees the position's alleles
as the release does. The release.json is the parent's with subset_of added.
Source code in enricher/src/just_dna_enricher/opentargets_build.py
507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 | |