Skip to content

just_dna_enricher.ld_pairs

just_dna_enricher.ld_pairs

ld-pairs — the computed LD pair behind a study measured on a proxy (0.8, RM331).

An author writes only StudyRow.proxy_rsid. r² has no meaning without its population, so the pass asks a reference for r² and D′ between proxy and lead in the study's own ancestry, and writes ld_pairs.csv, keyed (proxy_rsid, variant_key, ld_population). The reference is Ensembl REST's pairwise LD endpoint over the 1000 Genomes phase 3 superpopulations (PT2 §4 question 4, accepted 2026-11-04), probed by hand in docs/probes/LANES_0_8.md §2 before any of this was written. What the probe found, and what each finding costs here:

  • The five superpopulations are not in Ensembl's own LD population listing, yet the endpoint answers for them. So this pass spells 1000GENOMES:phase_3:<CODE> itself from the closed ancestry set, and a 400 … population object is a defect of that spelling (LdError), never a withhold.
  • 200 [] is not r² = 0. It comes back for a pair on two chromosomes and for the APOE pair in AFR, and lowering the r2 floor does not bring a row back. It is written as not_reported, r² empty.
  • A 400 … variation object is an answer: the reference does not know one of the two rsIDs. It is written as rsid_unknown, and is neither retried nor read as an outage.
  • The response renames a merged rsID (asking rs1624712 answers as rs646776). The row stays keyed on what the author wrote and answered_rsid records the reference's name. The proxy goes first in the URL, so variation1 is the proxy; the probe's two orderings both kept the order asked.
  • r2 and d_prime are JSON strings, six decimals. Parsed to floats; str(float) writes the same six digits back, so the CSV round trip is exact.
  • The service flakes: HTML 500 pages and 37-second answers. 5xx, 429 and transport errors are retried, then arrive as LdUnavailable (@client-exception-contract); the timeout is long enough for the slow answer. 55,000 calls per hour; the pacing gate stays well under it.

A study naming several superpopulations, or none, gets its row with r² withheld and the reason (the pm's setting: the lookup never picks a panel). A lead with no rsID (coordinate-authored, and no back-filled rsID in resolution.csv) gets its row with lead_has_no_rsid: the endpoint takes rsIDs. Neither consults the reference, so neither writes a licence row (@write-the-sourcerow).

Merge, never clobber (@sidecar-authoritative): a pair already in the file is not asked again; delete the file to re-ask after a new release. --offline asks nothing and writes nothing: there is no LD snapshot, and a row for a pair nobody asked about would claim an answer (@unreachable-not-absent). Could-not-ask writes no row: when Ensembl stays down past the retries, the pairs already answered this run are committed (each one cost a request, and the gap-fill resumes from them) and LdUnavailable is raised naming the rest.

LdError

Bases: RuntimeError

The LD pass could not do its work: a local table it cannot read, or an answer it cannot use.

LdUnavailable

Bases: LdError

Ensembl could not be asked: a transport failure, or a 5xx/429 that outlasted the retries.

A subclass so an except LdError keeps catching it, and a caller that must tell a dead source from bad data can, by type (@client-exception-contract).

LdRetired

Bases: LdUnavailable, EndpointRetired

Ensembl answered HTTP 410: it retired the endpoint asked (RM354).

A fact about the source, not an outage, and not retried. An LdUnavailable so every existing handler still catches it; the message says the endpoint moved rather than that the service was down.

LdAnswer dataclass

LdAnswer(
    rows: tuple[dict, ...] = (), unknown: str | None = None
)

What the reference said about one pair: its rows, or that it does not know an rsID.

rows is the JSON array as served (possibly empty); unknown is the reference's own message when it answered 400 for a variation it cannot fetch.

LdClient dataclass

LdClient(
    endpoint: str = DEFAULT_REST_ENDPOINT,
    species: str = "human",
    timeout: float = LD_TIMEOUT,
    client: Client | None = None,
    gate: PacingGate | None = None,
)

Ensembl REST pairwise LD, with the four legs of the client contract.

A 200 is an answer (its JSON array); a 400 naming a variation is an answer too (LdAnswer.unknown); every other 4xx (a 404 is a moved path or a wrong species, which a re-run repeats) and a 200 that is not a JSON array of pairs is LdError; 5xx (EBI serves its 500 as an HTML page), 429 and transport failures are retried and then LdUnavailable.

pairwise

pairwise(
    proxy: str, lead: str, population: str
) -> LdAnswer

r² and D′ between proxy and lead in population, as the reference serves them.

Source code in enricher/src/just_dna_enricher/ld_pairs.py
def pairwise(self, proxy: str, lead: str, population: str) -> LdAnswer:
    """r² and D′ between `proxy` and `lead` in `population`, as the reference serves them."""
    path = f"ld/{self.species}/pairwise/{proxy}/{lead}"
    response = self._request(path, {"population_name": population})
    if response.status_code == 400:
        try:
            payload = response.json()
        except ValueError as exc:
            raise LdError(
                f"Ensembl refused {path} for {population} with a body that is not JSON"
            ) from exc
        message = payload.get("error") if isinstance(payload, dict) else None
        if isinstance(message, str) and _UNKNOWN_VARIATION.search(message):
            return LdAnswer(unknown=message)
        raise LdError(f"Ensembl refused {path} for {population}: {message or payload!r}")
    payload = _json(response, path)
    if not isinstance(payload, list) or not all(isinstance(row, dict) for row in payload):
        raise LdError(f"Ensembl answered {path} with {type(payload).__name__}, not a list of pairs")
    return LdAnswer(rows=tuple(payload))

release

release() -> int

The Ensembl release answering, from /info/data ({"releases": [116]}).

Source code in enricher/src/just_dna_enricher/ld_pairs.py
def release(self) -> int:
    """The Ensembl release answering, from `/info/data` (`{"releases": [116]}`)."""
    payload = _json(self._request("info/data"), "info/data")
    release = ensembl_data_release(payload)
    if release is None:
        raise LdError(f"Ensembl's /info/data states no single release: {payload!r}")
    return release

LdPairsResult dataclass

LdPairsResult(
    rows: list[LdPairRow] = list(),
    written: list[LdPairRow] = list(),
    unasked: list[PairKey] = list(),
    unusable: list[tuple[PairKey, str]] = list(),
    dropped: list[PairKey] = list(),
    requests: int = 0,
    dataset: str | None = None,
    skipped_offline: bool = False,
)

What one ld-pairs run did.

enrich_ld_pairs

enrich_ld_pairs(
    spec_dir: Path,
    *,
    offline: bool = False,
    write: bool = True,
    client: LdClient | None = None,
) -> LdPairsResult

Fill ld_pairs.csv for every study row naming a proxy_rsid.

Source code in enricher/src/just_dna_enricher/ld_pairs.py
def enrich_ld_pairs(
    spec_dir: Path,
    *,
    offline: bool = False,
    write: bool = True,
    client: LdClient | None = None,
) -> LdPairsResult:
    """Fill `ld_pairs.csv` for every study row naming a `proxy_rsid`."""
    spec_dir = Path(spec_dir)
    target = sidecar_path(spec_dir, LD_PAIRS_CSV, error=LdError)
    if write:
        # A placeholder or half-edited licence table fails now, before any request (RM231).
        require_sources_file(spec_dir, error=LdError)

    existing: dict[tuple, LdPairRow] = {}
    if target.exists():
        rows, errors, _ = load_csv_rows(target, LdPairRow, LD_PAIRS_CSV)
        if errors:
            raise LdError(f"existing {LD_PAIRS_CSV} is invalid: {errors[0]}")
        existing = {merge_key(row): row for row in rows}

    named = _wanted(_read_studies(spec_dir), _backfilled_rsids(spec_dir))
    named_keys = {w.key for w in named}
    wanted = [w for w in named if w.key not in existing]
    to_ask = [w for w in wanted if w.status is None]
    result = LdPairsResult()

    if offline:
        if to_ask:
            logger.warning(
                "LD pairs skipped: --offline and there is no LD snapshot. %d pair(s) were not asked about "
                "and get no row (unknown, not absent): %s",
                len(to_ask),
                [w.key for w in to_ask][:5],
            )
        result.rows = sorted(existing.values(), key=_sort_key)
        result.unasked = [w.key for w in to_ask]
        result.skipped_offline = True
        return result

    fetched_at = now_utc_iso()
    fresh: list[LdPairRow] = [_withheld_row(w, fetched_at) for w in wanted if w.status is not None]
    outage: LdUnavailable | None = None
    if to_ask:
        owned = client is None
        ensembl = client or LdClient()
        try:
            outage = _ask(ensembl, to_ask, result, fresh, fetched_at)
        finally:
            if owned:
                ensembl.close()

    result.written = sorted(fresh, key=_sort_key)
    # A recorded pair no study row names any more (its ancestry or proxy was edited) is an orphan: its
    # study is gone, so it is dropped and counted rather than left to be counted in the manifest.
    result.dropped = sorted(
        (k for k in existing if k not in named_keys), key=lambda k: (k[0], k[1], k[2] or "")
    )
    if result.dropped:
        logger.warning(
            "Dropped %d ld_pairs.csv row(s) whose pair no studies.csv row names any more: %s",
            len(result.dropped),
            result.dropped[:5],
        )
    merged = {k: row for k, row in existing.items() if k in named_keys}
    for row in fresh:
        merged[merge_key(row)] = row
    result.rows = sorted(merged.values(), key=_sort_key)

    if write and (fresh or result.dropped):
        answered = result.requests > 0
        _write(
            result.rows,
            target,
            before_commit=(
                # Inside the table's commit (`@licence-row-inside-the-commit`), and only when Ensembl answered
                # this run: a run that wrote withholds alone consulted no source (`@write-the-sourcerow`).
                # Labelled by the release every committed pair states, not by this run's alone (RM373).
                (
                    lambda: record_table_sources(
                        table_datasets(result.rows, release=lambda row: row.dataset),
                        LD_LAYER,
                        spec_dir,
                        error=LdError,
                        # A routine merge declares nothing, so it retires no licence row (RM383).
                        answered_every_row=False,
                    )
                )
                if answered
                else None
            ),
        )
    if result.unusable:
        raise LdError(
            f"{len(result.unusable)} pair(s) got an answer this pass cannot use and have no row "
            f"(first: {result.unusable[0][0]}: {result.unusable[0][1]}); the {len(fresh)} other row(s) of "
            f"this run {'were committed' if write else 'are in the result'}"
            + (f", and {len(result.unasked)} pair(s) were not asked: {outage}" if outage is not None else "")
        )
    if outage is not None:
        raise LdUnavailable(
            f"{outage} — {len(result.unasked)} pair(s) were not asked and have no row; "
            f"{result.requests} answered this run {'were committed' if write else 'are in the result'}, and "
            f"a re-run asks only the rest."
        ) from outage
    return result