Skip to content

just_dna_enricher.opentargets_build

just_dna_enricher.opentargets_build

Open Targets Platform snapshot builder (0.8, RM333): one pinned release, trimmed to what a module joins.

The source is the Platform's bulk output on EBI's FTP, served over https (https://ftp.ebi.ac.uk/pub/databases/opentargets/platform/<release>/output/<dataset>/part-*.parquet). Five datasets are read, all Spark output split into parts (probes/LANES_0_8.md § 1 has their schemas, row counts and sizes as measured on 2026-11-04):

dataset what the snapshot keeps
variant the allele table (variants.parquet) and every per-transcript consequence (consequences-*.parquet)
credible_set GWAS credible sets (credible_sets.parquet) and their members' posteriors (credible_set_members-*.parquet)
l2g_prediction the locus-to-gene score per (study locus, gene) (l2g.parquet)
colocalisation the GWAS-left × molecular-QTL-right pairs, with the right side's gene and tissue (colocalisations-*.parquet)
study read only to name a coloc row's right-hand gene and tissue

Flattened at build, read by duckdb at run time (@duckdb-vs-polars). Every list column the runtime pass would otherwise unnest per query (locus, transcriptConsequences) is exploded here, once. The one list kept as a list is an allele's rsIds (variants.parquet's rs_ids): the pass asks only whether it holds one rsID, for a module row that states no ALT, and never unnests it.

Alleles come from columns, never from the id. variantId is <chr>_<pos>_<ref>_<alt> with underscores (the croissant says hyphens) and, for 2,114 long alleles, an OTVAR_… hash; the snapshot keeps the id only as an internal join key and the alleles from referenceAllele/alternateAllele.

What is trimmed, and why each cut is not a judgement. Credible sets and members are kept for GWAS studies only, because RM239's posterior and RM333's L2G are about a GWAS locus. Coloc rows are kept when the left side is a GWAS credible set and the right side is not (rightStudyType != 'gwas'): a GWAS-GWAS pair says two traits share a locus and names no gene, which is ~72% of the dataset. No row is dropped for its value: no score, posterior or h4 threshold is applied anywhere.

A partial build says so. parts limits how many parts of each dataset are read, for a test-sized or bandwidth-limited build; release.json then records parts_read beside parts_total, and the runtime pass warns that an absent coloc or L2G row may mean "not read" rather than "not published".

Builder-only (polars is a guarded import), as every builder here.

OpenTargetsBuildError

Bases: RuntimeError

The Open Targets lane failed; the builder's own type, so the lane adapter can catch it as this lane.

OpenTargetsUnavailable

Bases: OpenTargetsBuildError

The FTP did not answer: the fetch failed, not the data.

OpenTargetsRetired

Bases: OpenTargetsUnavailable, EndpointRetired

The source answered HTTP 410: it retired an Open Targets release file (RM361, audit round 2).

Its lane's OpenTargetsUnavailable, so every handler still catches it; marked EndpointRetired, so the reason is retired and the message says the file moved and to upgrade, never that the host was down.

OpenTargetsBuildResult dataclass

OpenTargetsBuildResult(
    out_dir: Path,
    release: str,
    row_counts: dict[str, int] = dict(),
    parts_read: dict[str, int] = dict(),
    parts_total: dict[str, int] = dict(),
)

What a build wrote, counted per table.

list_parts

list_parts(
    dataset: str,
    listing_dir: Path,
    *,
    release: str = OPENTARGETS_RELEASE,
    base_url: str = DEFAULT_OPENTARGETS_BASE_URL,
) -> list[str]

The part filenames of one dataset, read off the FTP's https directory listing.

The listing is fetched through net.stream_to_file, so it is retried and translated like every other download here; the parts are sorted so a partial build always reads the same ones.

Source code in enricher/src/just_dna_enricher/opentargets_build.py
def list_parts(
    dataset: str,
    listing_dir: Path,
    *,
    release: str = OPENTARGETS_RELEASE,
    base_url: str = DEFAULT_OPENTARGETS_BASE_URL,
) -> list[str]:
    """The part filenames of one dataset, read off the FTP's https directory listing.

    The listing is fetched through `net.stream_to_file`, so it is retried and translated like every
    other download here; the parts are sorted so a partial build always reads the same ones.
    """
    listing = stream_to_file(
        listing_dir / f"{dataset}.listing.html",
        dataset_url(dataset, release=release, base_url=base_url),
        error_cls=OpenTargetsUnavailable,
        retired=OpenTargetsRetired,
        what=f"the Open Targets {release} `{dataset}` listing",
    ).path
    names = sorted(set(_PART_PATTERN.findall(listing.read_text(encoding="utf-8", errors="replace"))))
    if not names:
        raise OpenTargetsBuildError(
            f"the {dataset} listing at {dataset_url(dataset, release=release, base_url=base_url)} names no parquet part"
        )
    return names

download_release

download_release(
    dest: Path,
    *,
    release: str = OPENTARGETS_RELEASE,
    base_url: str = DEFAULT_OPENTARGETS_BASE_URL,
    parts: int | None = None,
    workers: int = 8,
) -> dict[str, int]

Mirror the five datasets' parts into dest/<dataset>/, returning each dataset's total part count.

A part already on disk with a parquet footer is not fetched again, so an interrupted mirror resumes. parts keeps the first N parts of each dataset (sorted by name), for a test-sized or bandwidth-limited build; build_snapshot records how many were read against the total. The croissant (the release's only licence and version marker) is fetched beside them.

Source code in enricher/src/just_dna_enricher/opentargets_build.py
def download_release(
    dest: Path,
    *,
    release: str = OPENTARGETS_RELEASE,
    base_url: str = DEFAULT_OPENTARGETS_BASE_URL,
    parts: int | None = None,
    workers: int = 8,
) -> dict[str, int]:
    """Mirror the five datasets' parts into `dest/<dataset>/`, returning each dataset's total part count.

    A part already on disk with a parquet footer is not fetched again, so an interrupted mirror resumes.
    `parts` keeps the first N parts of each dataset (sorted by name), for a test-sized or
    bandwidth-limited build; `build_snapshot` records how many were read against the total. The
    croissant (the release's only licence and version marker) is fetched beside them.
    """
    dest = Path(dest)
    dest.mkdir(parents=True, exist_ok=True)
    stream_to_file(
        dest / "croissant.json",
        f"{base_url.rstrip('/')}/{release}/croissant.json",
        error_cls=OpenTargetsUnavailable,
        retired=OpenTargetsRetired,
        what=f"the Open Targets {release} croissant",
    )
    totals: dict[str, int] = {}
    jobs: list[tuple[Path, str, str]] = []
    for dataset in OPENTARGETS_DATASETS:
        names = list_parts(dataset, dest, release=release, base_url=base_url)
        totals[dataset] = len(names)
        # `study` is one part and is read whole; a limit on it would leave coloc rows unnameable.
        chosen = names if (parts is None or dataset == "study") else names[:parts]
        for name in chosen:
            target = dest / dataset / name
            if target.exists() and _footer_ok(target):
                continue
            jobs.append((target, dataset_url(dataset, release=release, base_url=base_url) + name, dataset))
    atomic_write_text(dest / "parts_total.json", json.dumps(totals, sort_keys=True))

    stop = threading.Event()

    def _fetch(job: tuple[Path, str, str]) -> None:
        target, url, dataset = job
        stream_to_file(
            target,
            url,
            error_cls=OpenTargetsUnavailable,
            retired=OpenTargetsRetired,
            what=f"Open Targets `{dataset}` part {target.name}",
            _stop=stop,
        )

    with ThreadPoolExecutor(max_workers=max(1, workers)) as pool:
        futures = [pool.submit(_fetch, job) for job in jobs]
        try:
            for future in futures:
                future.result()
        except BaseException:
            # The first failure, or the user's Ctrl-C, surfaces now: the queued parts are cancelled and
            # the in-flight ones stop at their next chunk, each removing its `.part` (RM409). Without the
            # flag the shutdown joined every in-flight part streamed to its end, and an interrupted
            # 22 GB mirror ignored two Ctrl-C. A part that finished before the stop stays on disk, and
            # the next run resumes past it.
            stop.set()
            pool.shutdown(wait=True, cancel_futures=True)
            raise
    return totals

build_snapshot

build_snapshot(
    source: Path,
    out_dir: Path,
    *,
    release: str = OPENTARGETS_RELEASE,
    base_url: str = DEFAULT_OPENTARGETS_BASE_URL,
) -> OpenTargetsBuildResult

Flatten a mirrored release (source/<dataset>/part-*.parquet) into the snapshot at out_dir.

Every table is sorted on its key before it is written, so a rebuild from the same parts writes the same parquet bytes (Principle 7); release.json's built_at is the only per-run byte.

Source code in enricher/src/just_dna_enricher/opentargets_build.py
def build_snapshot(
    source: Path,
    out_dir: Path,
    *,
    release: str = OPENTARGETS_RELEASE,
    base_url: str = DEFAULT_OPENTARGETS_BASE_URL,
) -> OpenTargetsBuildResult:
    """Flatten a mirrored release (`source/<dataset>/part-*.parquet`) into the snapshot at `out_dir`.

    Every table is sorted on its key before it is written, so a rebuild from the same parts writes the
    same parquet bytes (Principle 7); `release.json`'s `built_at` is the only per-run byte.
    """
    _require_polars()
    source, out_dir = Path(source), Path(out_dir)
    # Built beside the live payload and swapped in at the end, never in place: a build that fails
    # halfway must leave the previous snapshot whole rather than a short payload under an old
    # `release.json` the resolver would accept.
    data_dir = out_dir / "data.building"
    if data_dir.exists():
        shutil.rmtree(data_dir)
    data_dir.mkdir(parents=True)

    counts: dict[str, int] = {}
    counts["variants"] = _build_variants(source, data_dir)
    counts["consequences"] = _build_consequences(source, data_dir)
    counts["credible_sets"], counts["credible_set_members"], locus_study = _build_credible_sets(
        source, data_dir
    )
    gwas_loci = pl.read_parquet(data_dir / CREDIBLE_SETS_FILE, columns=["study_locus_id"])
    counts["l2g"] = _build_l2g(source, data_dir, gwas_loci)
    counts["colocalisations"] = _build_coloc(source, data_dir, gwas_loci, locus_study)

    totals_path = source / "parts_total.json"
    totals = json.loads(totals_path.read_text(encoding="utf-8")) if totals_path.exists() else {}
    read = {dataset: len(_parts(source, dataset)) for dataset in OPENTARGETS_DATASETS}
    markers = _croissant_markers(source)
    digest = _source_digest(source)
    live = out_dir / "data"
    # The record is withdrawn here and not earlier: until the live payload is removed it still
    # describes it, and the staging above can fail for hours without touching it (RM369).
    snapshot = begin_snapshot_write(out_dir)
    if live.exists():
        shutil.rmtree(live)
    data_dir.rename(live)
    snapshot.commit(
        SnapshotRelease(
            dataset=release_label(release),
            source_url=f"{base_url.rstrip('/')}/{release}/output/",
            source_sha256=digest,
            license="CC0-1.0",
            genome_build="GRCh38",
            built_at=now_utc_iso(),
            builder_version=_builder_version(),
            row_counts=counts,
            parts_read=read,
            parts_total=totals or None,
            croissant=markers,
        ),
    )
    logger.info("Built Open Targets %s snapshot at %s: %s", release, out_dir, counts)
    return OpenTargetsBuildResult(
        out_dir=out_dir, release=release, row_counts=counts, parts_read=read, parts_total=totals
    )

subset_snapshot

subset_snapshot(
    snapshot: Path,
    out_dir: Path,
    alleles: list[tuple[str, int, str, str]],
    studies: list[str] | None = None,
) -> dict[str, int]

Cut a built snapshot down to the loci of alleles ((chrom, position, ref, alt)), as a snapshot.

For a test fixture or a regional snapshot: every table keeps exactly the rows the runtime pass would read for those alleles (their variants and consequences, their credible-set memberships, the sets they are members of, those sets' L2G and coloc rows), so a pass over the subset answers as it would over the whole release for those variants. studies keeps every credible-set head of those studies too, so "the study has credible sets, and the variant is in none" answers on the subset as it does on the release. Every other allele of the release at a wanted allele's (chrom, position) is kept too, with all of its rows, so the pass's lookup for a module row with no ALT sees the position's alleles as the release does. The release.json is the parent's with subset_of added.

Source code in enricher/src/just_dna_enricher/opentargets_build.py
def subset_snapshot(
    snapshot: Path,
    out_dir: Path,
    alleles: list[tuple[str, int, str, str]],
    studies: list[str] | None = None,
) -> dict[str, int]:
    """Cut a built snapshot down to the loci of `alleles` (`(chrom, position, ref, alt)`), as a snapshot.

    For a test fixture or a regional snapshot: every table keeps exactly the rows the runtime pass would
    read for those alleles (their variants and consequences, their credible-set memberships, the sets
    they are members of, those sets' L2G and coloc rows), so a pass over the subset answers as it would over the
    whole release for those variants. `studies` keeps every credible-set head of those studies too,
    so "the study has credible sets, and the variant is in none" answers on the subset as it does on
    the release. Every other allele of the release at a wanted allele's `(chrom, position)` is kept too,
    with all of its rows, so the pass's lookup for a module row with no ALT sees the position's alleles
    as the release does. The `release.json` is the parent's with `subset_of` added.
    """
    _require_polars()
    snapshot, out_dir = Path(snapshot), Path(out_dir)
    src, data_dir = snapshot / "data", out_dir / "data"
    wanted = pl.DataFrame(
        alleles, schema={"chrom": pl.Utf8, "position": pl.Int64, "ref": pl.Utf8, "alt": pl.Utf8}, orient="row"
    )
    # Every allele of the release at a wanted allele's `(chrom, position)` is kept whole, beside the wanted
    # one: a module row naming an rsID and no ALT is matched against every allele at its position, so a
    # slice without them would answer "one allele here" where the release answers "several", and a
    # sibling the pass supplies must answer as it does on the release.
    variants = (
        pl.read_parquet(src / VARIANTS_FILE)
        .join(wanted.select("chrom", "position").unique(), on=["chrom", "position"], how="semi")
        .sort("chrom", "position", "ref", "alt", "variant_id")
    )
    ids = variants.select("variant_id")
    members = pl.scan_parquet(str(src / MEMBERS_GLOB)).join(ids.lazy(), on="variant_id", how="semi").collect()
    sets = pl.read_parquet(src / CREDIBLE_SETS_FILE)
    of_studies = sets.filter(pl.col("study_id").is_in(studies or [])).select("study_locus_id")
    kept_sets = sets.join(
        pl.concat([members.select("study_locus_id"), of_studies]).unique(),
        on="study_locus_id",
        how="semi",
    ).sort("study_locus_id")
    # Every set a wanted allele is a MEMBER of (RM333, members not lead-only). A lead is a member of its
    # own set: over the whole 26.09 release no GWAS set's lead is missing from its member list, so the
    # member loci already hold the led ones.
    loci = members.select("study_locus_id").unique()
    tables = {
        VARIANTS_FILE: variants,
        "consequences-00000.parquet": pl.scan_parquet(str(src / CONSEQUENCES_GLOB))
        .join(ids.lazy(), on="variant_id", how="semi")
        .sort("variant_id", "transcript")
        .collect(),
        CREDIBLE_SETS_FILE: kept_sets,
        "credible_set_members-00000.parquet": members.sort("variant_id", "study_locus_id"),
        L2G_FILE: pl.read_parquet(src / L2G_FILE)
        .join(loci, on="study_locus_id", how="semi")
        .sort("study_locus_id", "gene_id"),
        "colocalisations-00000.parquet": pl.scan_parquet(str(src / COLOC_GLOB))
        .join(loci.lazy(), on="study_locus_id", how="semi")
        .sort("study_locus_id", "right_study_locus_id")
        .collect(),
    }
    counts: dict[str, int] = {}
    parent = read_release(snapshot)
    write = begin_snapshot_write(out_dir)
    data_dir.mkdir(parents=True, exist_ok=True)
    for name, frame in tables.items():
        frame.write_parquet(data_dir / name)
        counts[name] = frame.height
    record = parent.model_dump(exclude_none=True) if parent is not None else {}
    record.update(
        {
            "built_at": now_utc_iso(),
            "row_counts": counts,
            "subset_of": [f"{c}_{p}_{r}_{a}" for c, p, r, a in alleles],
        }
    )
    write.commit(SnapshotRelease(**record))
    return counts