Skip to content

just_dna_enricher.gwas

just_dna_enricher.gwas

The NHGRI-EBI GWAS Catalog pass — published effect sizes as a derived fact table (0.6, RM90).

Fills gwas_effects.csv for the variants variants.csv names, from the Catalog's REST API v2 (RM354). What it does not do is fill weight: a consumer asked for exactly that and it is barred — MODULE_LIFECYCLE § Stage 3 names weight/direction/effect_size in the cells no tool fills, and every check in this tier reports rather than repairs. The effect lands in its own table beside the authored column, and a consumer chooses between them wholesale.

The legacy API is gone. /gwas/rest/api/singleNucleotidePolymorphisms/<rsid>/associations, which this pass read from 0.6 until 2026-11-04, answers HTTP 410 on every path with "This legacy GWAS Catalog REST API is no longer available". A 410 now raises GwasRetired, never the plain could-not-ask GwasError, because a source that moved and a source that is down are cleared by different things.

Everything below was established by probing v2 on 2026-11-04 (/gwas/rest/api/v2/associations? rs_id=), not from its migration guide, and these findings shaped the pass:

  • The association is fat now, and the request budget collapsed with it. pubmed_id, accession_id and efo_traits sit on the association itself, so the legacy 1 + 2N per variant (a study and a trait link per association, measured at 382 requests for one real module) is gone. Ancestry is the one fact still behind a request — /studies/<accession>/ancestries — so the cost is one request per page plus one per distinct study, and _LinkCache keyed on that URL now pays.
  • It pages, 20 per page by default. rs4149056 carries 386 associations. The pass asks for large pages and follows _links.next until there is none, and every page is counted in the request total. A server may refuse a large page, not only cap it (RM428): size=1000 answered whole on 2026-11-04 and a bare 400 six days later, when the ceiling had dropped to 500. A refused first request is asked once more with no size, so the Catalog's own default page decides, and the run warns once naming the refused size.
  • The empty answer is a 200 with no _embedded key at all, page.totalElements: 0 — for an rsID the Catalog has never heard of as much as for a rare clinical one. The legacy API 404'd there instead, so the meaning of a 404 flipped: on the v2 search path it is a wrong path, never an answer.
  • The effect is one composite string, "0.05 umol/l increase" (or "-" when the association is an odds ratio, which sits in or_per_copy_num). The legacy betaNum / betaUnit / betaDirection come out of it by one grammar; a string outside it is withheld whole and counted. The unit is kept verbatim between the number and the direction — one study writes U+2009 in it.
  • snp_effect_allele is rs4149056-C, or rs4149056-? when the study never established which allele carries the effect — the legacy riskAlleleName grammar, unchanged. -? becomes None, never a guess, and the row is still written: an effect relative to an unknown allele is real evidence that cannot be used as a weight.
  • The unit is free text and frequently uninterpretable — umol/l on one association and unit on many others for the same variant. It is stored verbatim including the useless values, because "these betas are on unknown and possibly different scales" is the fact a consumer must have.
  • v2 publishes no standard error. standard_error stays null on every row this pass writes.
  • The p-value has its curated pair back. p_value is 0.0 past float64's range, as before, but pvalue_mantissa/pvalue_exponent beside it still say 3e-493; the verbatim string is built from the pair, and the queryable number is read from that same string.

The published budget is generous, and the gate is not it. The v2 docs page (https://www.ebi.ac.uk/gwas/rest/api/v2/docs, "Throttling", read 2026-11-11) says "Users of the API are limited to 15 queries per second ... If the limit is exceeded, the API will slow down subsequent calls." An excess slows rather than refuses, unlike gnomad.py, whose 10-requests-per-60-seconds is load-bearing. The gate here (one request a second) stays a conservative default well under that budget, not a transcription of it, so nobody "corrects" it up to 15 per second against a server that never refused. The page states no page-size ceiling; RM428's 500 is measured only.

GwasError

Bases: RuntimeError

A GWAS Catalog fetch or parse failed in a way the caller must see.

Every transport and parse failure is translated into this before it leaves the module. A client that leaks httpx.HTTPError has no contract — the caller would have to import the transport library to catch it, and a swap of transport would silently become a breaking change.

GwasNotFound

Bases: GwasError

The Catalog answered, and it has no record of this variant at all (HTTP 404).

A subclass rather than a flag, because this is an answer and not a failure. Found by running the pass against a real module: the Catalog holds only variants with a published genome-wide association, so rs111033563 — a rare clinical HFE variant — 404s rather than returning an empty list. The first version of this pass treated that as an outage and died on the first rare variant in the module, which is every clinically-authored module.

It stays inside GwasError so a caller that does not care about the distinction still catches one type. Since v2 (RM354) only follow treats it as an answer — one study's ancestry withheld. The v2 search answers a variant it holds nothing for with a 200 and zero elements, and its only measured 404 was a path that does not exist, so association_pages lets this propagate: reading it as "no associations" would write a confident negative about a variant nobody could ask about.

GwasRetired

Bases: GwasError, EndpointRetired

The Catalog answered HTTP 410 Gone: it retired the endpoint this client asked (RM354).

A fact about the source, not an outage. The legacy REST API began answering 410 on every path with "This legacy GWAS Catalog REST API is no longer available", and the generic status arm turned that into a plain GwasError — the same could-not-ask an outage raises — so a source that had moved read as one that was down, and nothing said to upgrade. The message carries the URL and the source's own text, which names where it went.

Not retried: a 410 is an HTTPStatusError, which is outside the retry predicate by construction, and asking again changes nothing. A subclass of GwasError so every existing except GwasError still catches it; nothing in the tier catches GwasNotFound before GwasError on a path that could see a 410, and follow's except GwasNotFound does not match it, so it propagates.

GwasResult dataclass

GwasResult(
    rows: list[GwasEffectRow] = list(),
    covered: list[str] = list(),
    missing: list[str] = list(),
    requests_made: int = 0,
    requests_saved: int = 0,
    p_value_underflows: int = 0,
    unparsed_effects: int = 0,
    unusable: int = 0,
    skipped_offline: bool = False,
)

What one pass did, including what it spent.

GwasCatalogClient dataclass

GwasCatalogClient(
    endpoint: str = DEFAULT_GWAS_ENDPOINT,
    stats_url: str = DEFAULT_GWAS_STATS_URL,
    timeout: float = 30.0,
    gate: PacingGate = (
        lambda: PacingGate(DEFAULT_REQUEST_INTERVAL)
    )(),
    size_refusals: int = 0,
    _client: Client | None = None,
)

Thin REST client for the Catalog, paced and retried.

endpoint and gate are injectable so a test can drive the real parsing code against recorded payloads without a network or a real sleep — the same shape EnsemblResolver and the gnomAD client use.

release

release() -> str | None

The release date the Catalog states, or None when it could not be read or states none (RM373).

Asked once per run. A failure is the unknown arm, logged, never an error: the associations are the run's work, and a release that could not be read is withheld from their label, never replaced by the day the run happened.

Source code in enricher/src/just_dna_enricher/gwas.py
def release(self) -> str | None:
    """The release date the Catalog states, or `None` when it could not be read or states none (RM373).

    Asked once per run. A failure is the unknown arm, logged, never an error: the associations are
    the run's work, and a release that could not be read is withheld from their label, never
    replaced by the day the run happened.
    """
    try:
        payload = self._get(self.stats_url)
    except GwasError as exc:
        logger.warning("GWAS Catalog did not state its release (%s); the rows name none.", exc)
        return None
    date = payload.get("date") if isinstance(payload, dict) else None
    return date.strip() if isinstance(date, str) and date.strip() else None

association_pages

association_pages(rsid: str) -> list[list[dict]]

Every page of the Catalog's associations for one rsID, one list per request made.

Pages rather than a flat list so the pass can count what it spent (@dont-discard-computed). Three outcomes, not two — the S20 shape. Associations are an answer; a single empty page is also an answer (the Catalog was reached and holds nothing for this variant, which the caller records as not_found); and a raised GwasError means it could not be asked, which is unchecked rather than empty. Fusing the last two would turn a failed request into a definite negative — which is why a 404 here raises (see GwasNotFound).

Source code in enricher/src/just_dna_enricher/gwas.py
def association_pages(self, rsid: str) -> list[list[dict]]:
    """Every page of the Catalog's associations for one rsID, one list per request made.

    Pages rather than a flat list so the pass can count what it spent (`@dont-discard-computed`).
    **Three outcomes, not two** — the S20 shape. Associations are an answer; a single empty page
    is *also* an answer (the Catalog was reached and holds nothing for this variant, which the
    caller records as `not_found`); and a raised `GwasError` means it could not be asked, which is
    unchecked rather than empty. Fusing the last two would turn a failed request into a definite
    negative — which is why a 404 here raises (see `GwasNotFound`).
    """
    url, payload = self._first_page(rsid)
    pages: list[list[dict]] = []
    asked: set[str] = {url}
    while True:
        associations = (payload.get("_embedded") or {}).get("associations")
        if associations is None:
            # v2's empty answer is a 200 with no `_embedded` key and a page block saying zero.
            # Anything else without associations is a shape this pass does not model, and
            # returning `[]` for it would record a definite "no associations".
            if (payload.get("page") or {}).get("totalElements") == 0:
                pages.append([])
                break
            raise GwasError(
                f"GWAS Catalog response for {rsid} carries no `_embedded.associations` and does "
                f"not say it is empty — the API shape has changed and this pass would otherwise "
                f"record a false negative"
            )
        pages.append(list(associations))
        nxt = (((payload.get("_links") or {}).get("next")) or {}).get("href")
        if not nxt:
            break
        url = str(nxt).split("{")[0]  # a HAL link may be a URI template
        if url in asked:
            raise GwasError(f"GWAS Catalog paging for {rsid} links back to {url}; refusing to loop")
        asked.add(url)
        # A later page is never re-asked on a 400: by now the size has been accepted (RM428).
        payload = self._get(url)
    return pages

study_ancestries_url

study_ancestries_url(accession: str) -> str

Where v2 keeps one study's ancestries. Built from the accession: v2 associations carry no _links.study, only self, loci and snp.

Source code in enricher/src/just_dna_enricher/gwas.py
def study_ancestries_url(self, accession: str) -> str:
    """Where v2 keeps one study's ancestries. Built from the accession: v2 associations carry no
    `_links.study`, only `self`, `loci` and `snp`."""
    return f"{self.endpoint}/studies/{accession}/ancestries"

follow

follow(url: str) -> dict

A linked sub-resource, or {} when the Catalog has none.

A 404 here withholds the ancestry for one association rather than sinking the pass: the association itself is still a real published effect, and dropping it because its study record moved would lose evidence over metadata. A 410 is not caught: GwasRetired is the source moving, and swallowing it here would null every row's ancestry and say nothing.

Source code in enricher/src/just_dna_enricher/gwas.py
def follow(self, url: str) -> dict:
    """A linked sub-resource, or `{}` when the Catalog has none.

    A 404 here withholds the ancestry for one association rather than sinking the pass: the
    association itself is still a real published effect, and dropping it because its study record
    moved would lose evidence over metadata. A 410 is not caught: `GwasRetired` is the source
    moving, and swallowing it here would null every row's ancestry and say nothing.
    """
    try:
        return self._get(url)
    except GwasNotFound:
        logger.warning("GWAS Catalog has no record at %s; that association keeps a null ancestry", url)
        return {}

enrich_gwas

enrich_gwas(
    spec_dir: Path,
    *,
    mode: str = "best_effort",
    offline: bool = False,
    write: bool = True,
    client: GwasCatalogClient | None = None,
    dataset: str | None = None,
    declared_use: str = "unstated",
    study_facts: bool = True,
) -> GwasResult

Record the Catalog's published effect sizes for this module's variants.

Existing rows are authoritative and merged, never clobbered — the standing rule for every pass, with the standing consequence: to regenerate after a machinery change, delete the file first.

offline makes the pass a no-op with a warning rather than a failure: the Catalog publishes a bulk download, but this pass reads the REST API, and there is no snapshot for it to fall back on. An injected client still wins, because handing over a transport you already hold is not egress.

A variant the Catalog holds nothing for gets a not_found row, unlike enrich_gene_validity's silence, and the difference is real rather than stylistic: a curating body's silence means nobody has assessed the gene, whereas the Catalog's empty answer means no genome-wide association has been published for this variant — which is a fact about it, and one a consumer weighing an authored weight against the literature wants to see.

mode is the severity ladder the CLI's --strict sets, and it is deliberately not about missing: see the block above the raise at the end of this function for why the Catalog's empty answer is a fact rather than a shortfall, and what strict reads instead.

Source code in enricher/src/just_dna_enricher/gwas.py
def enrich_gwas(
    spec_dir: Path,
    *,
    mode: str = "best_effort",
    offline: bool = False,
    write: bool = True,
    client: GwasCatalogClient | None = None,
    dataset: str | None = None,
    declared_use: str = "unstated",
    study_facts: bool = True,
) -> GwasResult:
    """Record the Catalog's published effect sizes for this module's variants.

    Existing rows are authoritative and merged, never clobbered — the standing rule for every pass,
    with the standing consequence: to regenerate after a machinery change, delete the file first.

    `offline` makes the pass a no-op with a warning rather than a failure: the Catalog publishes a
    bulk download, but this pass reads the REST API, and there is no snapshot for it to fall back on.
    An injected `client` still wins, because handing over a transport you already hold is not egress.

    A variant the Catalog holds nothing for gets a **`not_found` row**, unlike
    `enrich_gene_validity`'s silence, and the difference is real rather than stylistic: a curating
    body's silence means nobody has assessed the gene, whereas the Catalog's empty answer means no
    genome-wide association has been published for this variant — which *is* a fact about it, and one
    a consumer weighing an authored weight against the literature wants to see.

    `mode` is the severity ladder the CLI's `--strict` sets, and it is deliberately **not** about
    `missing`: see the block above the raise at the end of this function for why the Catalog's empty
    answer is a fact rather than a shortfall, and what `strict` reads instead.
    """
    spec_dir = Path(spec_dir)
    output_path = sidecar_path(spec_dir, "gwas_effects.csv", error=GwasError)
    if write:
        # Fail on a placeholder or half-edited licence table now, before the fetch (S98, RM231).
        require_sources_file(spec_dir, error=GwasError)

    if offline and client is None:
        logger.warning(
            "GWAS pass skipped: --offline. This pass reads the Catalog's REST API and has no "
            "snapshot to fall back on, so it is a no-op offline rather than a failure."
        )
        return GwasResult(skipped_offline=True)

    existing_rows: list[GwasEffectRow] = []
    if output_path.exists():
        parsed, errors, _ = load_csv_rows(output_path, GwasEffectRow, output_path.name)
        if errors:
            raise GwasError(f"existing {output_path.name} is invalid: {errors[0]}")
        existing_rows = parsed

    subjects = _module_subjects(spec_dir)
    if not subjects:
        logger.warning(
            "GWAS pass has no subjects: no row in variants.csv carries an rsID, and the Catalog is "
            "queried by rsID. Nothing was fetched and nothing was written."
        )
        return GwasResult(rows=existing_rows)

    owned = client is None
    # `try/finally`, like every sibling pass (RM100). The close used to be a bare
    # `if client is None: catalog.close()` after ~80 lines of fetching and writing, so any
    # exception in between -- a `GwasError`, a sidecar collision, a write failure -- leaked the
    # httpx client. `frequencies`, `gene_metrics` and `enrich` all had this shape already.
    try:
        catalog = client or GwasCatalogClient()
        cache = _LinkCache(fetch=catalog.follow)
        # The Catalog's own release (RM373), never the day this run happened to ask.
        release = dataset or release_dataset(GWAS_LANE, catalog.release())
        fetched_at = now_utc_iso()

        seen = {_merge_key(row) for row in existing_rows}
        out: list[GwasEffectRow] = list(existing_rows)
        covered: list[str] = []
        missing: list[str] = []
        unusable = 0
        direct_requests = 0
        refusals_before = catalog.size_refusals
        underflows: list[str] = []

        unparsed: list[str] = []

        for rsid, variant_key in subjects:
            pages = catalog.association_pages(rsid)
            direct_requests += len(pages)
            associations = [association for page in pages for association in page]
            if not associations:
                missing.append(rsid)
                row = GwasEffectRow(
                    association_id=f"{rsid}:not_found",
                    variant_key=variant_key,
                    rsid=rsid,
                    dataset=release,
                    source=GWAS_SOURCE,
                    status="not_found",
                    fetched_at=fetched_at,
                )
                if _merge_key(row) not in seen:
                    seen.add(_merge_key(row))
                    out.append(row)
                continue
            covered.append(rsid)
            for association in associations:
                built = _build_row(
                    association,
                    rsid=rsid,
                    variant_key=variant_key,
                    cache=cache,
                    study_url=catalog.study_ancestries_url,
                    release=release,
                    fetched_at=fetched_at,
                    underflows=underflows,
                    unparsed=unparsed,
                    study_facts=study_facts,
                )
                if built is None:
                    unusable += 1
                    continue
                key = _merge_key(built)
                if key in seen:
                    continue
                seen.add(key)
                out.append(built)

        size_refusals = catalog.size_refusals - refusals_before
        direct_requests += size_refusals
        if size_refusals:
            # Once per run (RM428), so the next drop of the Catalog's page ceiling is seen rather than
            # paid for silently, one extra request per rsID.
            logger.warning(
                "GWAS Catalog refused a page of %d association(s) with a 400 for %d rsID(s); each was "
                "asked again without a size, at the Catalog's default page. Its ceiling has moved: "
                "lower `_PAGE_SIZE` in gwas.py to what it accepts.",
                _PAGE_SIZE,
                size_refusals,
            )
        if underflows:
            # Aggregated by reason, once. On a well-studied variant this fires dozens of times, and the
            # rows are all still there — only the queryable number is withheld.
            logger.warning(
                "GWAS pass withheld p_value_num on %d association(s) whose p-value the Catalog reports "
                "below float64's range (it publishes 0.0); the verbatim p_value string is kept and the "
                "associations are recorded in full.",
                len(underflows),
            )
        if unusable:
            # Aggregated by reason, once — a per-row warning over a well-studied variant's dozens of
            # associations is a wall nobody reads.
            logger.warning(
                "GWAS pass skipped %d association(s) the Catalog published without an id this pass "
                "could key on; every usable association was still recorded.",
                unusable,
            )

        if unparsed:
            # Aggregated by reason, once, like the two above.
            logger.warning(
                "GWAS pass withheld the effect columns of %d association(s) whose `beta` string is "
                "outside the grammar it reads (`<number> [<unit>] <increase|decrease>`); the rows are "
                "recorded with every other column.",
                len(unparsed),
            )

        out.sort(key=_sort_key)
        result = GwasResult(
            rows=out,
            covered=covered,
            missing=missing,
            requests_made=direct_requests + cache.misses,
            requests_saved=cache.hits,
            p_value_underflows=len(underflows),
            unusable=unusable,
            unparsed_effects=len(unparsed),
        )

        if write:
            # The licence row lands inside the table's commit (S98, RM231).
            _write_gwas_csv(
                out,
                output_path,
                # Labelled by the release every committed row states, not by this run's alone (RM373).
                before_commit=lambda: record_table_sources(
                    table_datasets(out, release=lambda row: stated_release(row.dataset, GWAS_LANE)),
                    "gwas_effect",
                    spec_dir,
                    error=GwasError,
                    # A routine merge declares nothing, so it retires no licence row (RM383).
                    answered_every_row=False,
                    declared_use=declared_use,
                ),
            )
    finally:
        if owned:
            catalog.close()

    # **What `mode` means here, and why it is not `missing`** (RM100). The parameter was
    # accepted and never read while the CLI advertised `--strict` as a severity ladder, so the
    # flag was inert. Every sibling pass escalates on `result.missing`, and that would be wrong
    # here for the reason this pass's own docstring gives: the Catalog holding nothing for a
    # variant is a FACT about the variant, recorded as a `not_found` row, and true of most
    # variants. Escalating it would refuse nearly every module and say nothing.
    #
    # What `strict` does escalate is the pass failing to hold what the source DID say: an
    # association served without an id to key on, and a p-value below float64 whose queryable
    # number is therefore withheld. Both are already warned about in `best_effort`, aggregated
    # by reason; `strict` means a reproducible artifact, and both of these are places where the
    # artifact is missing something the Catalog published. RM354 adds the third of the kind: an
    # effect string this pass could not read, whose columns were withheld.
    if mode == "strict" and (result.unusable or result.p_value_underflows or result.unparsed_effects):
        raise GwasError(
            f"strict GWAS enrichment: {result.unusable} association(s) were served without an "
            f"id this pass can key on, {result.p_value_underflows} carried a p-value below "
            f"float64's range and this pass could not read the effect of "
            f"{result.unparsed_effects}, so the artifact does not hold everything the Catalog published. "
            f"All three are the Catalog's shape rather than an authoring mistake -- use "
            f"mode='best_effort' to record what is holdable and read the warnings."
        )
    return result

gwas_source_row

gwas_source_row(
    *,
    declared_use: str = "unstated",
    dataset: str | None = None,
) -> SourceRow

The licence row this pass writes. Exposed so a caller can record the terms without fetching.

Source code in enricher/src/just_dna_enricher/gwas.py
def gwas_source_row(*, declared_use: str = "unstated", dataset: str | None = None) -> SourceRow:
    """The licence row this pass writes. Exposed so a caller can record the terms without fetching."""
    return GWAS_CATALOG_TERMS.row("gwas_effect", declared_use=declared_use, dataset=dataset)