Skip to content

just_dna_enricher.literature

just_dna_enricher.literature

enrich-literature — the fourth pass: citations in, literature.csv out.

Closes the reachable part of the compiler's "does the cited study support the row?" blind spot. Three questions, in increasing ambition and decreasing coverage:

  1. Does the citation exist? PubMed esummary, batched. A nonexistent PMID comes back as a record carrying an error key, so this is a clean yes/no.
  2. Do the identifiers agree? The DOI and PMCID arrive free in the same response. An absent authored DOI is filled here in the sidecar; an authored DOI that contradicts the registry is a finding. Neither is ever written back into studies.csv — the enricher does not edit authored files, because content_signature is defined as reference-independent, and a network fetch that could change it would make that documented property false.
  3. Does the quoted passage appear in the article? Only for the open-access subset, and honestly labelled as such.

All three are attested (RM45): the pass writes citation_existence, citation_identifier and provenance_quote into verification.json on its way out, including on the offline return, so a module can say which of the three was put and over how many citations. Until 0.6 the answers reached a log line and the result object and died there, which left a module whose citations had been checked indistinguishable from one where the command was never run.

Coverage is partial by nature, and saying so is part of the check. A pass that reported "0 quotes found" for an article it could not read would be describing its own reach as if it were a property of the module. So the result separates checked and not found from never retrievable — quotes_found is null rather than zero when no fulltext could be read — and, since dogfooding caught the conflation, also from nothing to check: a citation with no authored quote asked no question and is not counted against coverage. The same distinctions are carried into manifest.literature.

Corrections to the drafted plan, made under probing rather than assumed:

  • The PMC ID converter is not used, though the plan budgeted for it. esummary already returns both doi and pmc in articleids, and Europe PMC's search returns doi/pmcid too — so the converter is a third request for data already in hand. Worse, it answers a different question: for PMID 12345678 (a real, indexed PubMed record) it replies status: error, "Identifier not found in PMC", which is about PMC membership, not existence. Wiring it in as an existence check would report every paywalled article as a broken citation.
  • Europe PMC is not an existence oracle either. Asked for three ids where one does not exist, it returns two results and simply omits the third — no error, no marker. Absence there is indistinguishable from "not indexed", so PubMed decides existence and Europe PMC only decides retrievability.

On running provenance_regex here. The charter requires a linear-time / ReDoS-safe engine for pattern matching, written when the match was specified as consumer-side — arbitrary patterns meeting arbitrary documents. Here the pattern comes from the module being enriched and the document from a public archive, on the author's own machine, so the threat model is a curator writing a slow pattern by accident rather than an attacker. That is worth a bound rather than a compiled dependency — so the match runs under a wall-clock timeout, and a timeout is recorded as not checked.

That bound is enforced with a child process, not a thread, and the reason is worth stating because the thread version looks correct: re cannot be interrupted, threads cannot be killed, and the interpreter joins pool threads at exit — so a thread-based timeout returns on schedule and then hangs the process on the way out. See regex_matches.

LiteratureEnrichmentError

Bases: RuntimeError

Raised in strict mode when a citation does not resolve, or contradicts its own identifiers.

LiteratureUnavailable

Bases: LiteratureEnrichmentError

PubMed could not be reached, so no citation question was put at all (RM101).

A subclass rather than a second exception, so every existing except LiteratureEnrichmentError still catches it (P3 — additive within a major). It separates the source was asked and never answered from a citation that genuinely does not resolve under strict. Only this one means nothing was established either way.

Before RM101 an EutilsError travelled straight out of enrich_literature through a try/finally with no except, so a caller's except LiteratureEnrichmentError was silent for exactly the failure it was written for.

Scoped to the eutils leg on purpose. EuropePmcClient.fulltext and CrossrefClient.exists already answer a transport failure with None rather than an exception — the tri-state withhold this codebase uses for "could not be retrieved" — and turning either into an error here would convert a withheld answer into a failed run.

FulltextUnavailable

Bases: LiteratureUnavailable

A fulltext request never got an answer: a 5xx or a transport failure, after retries (RM258).

Raised by EuropePmcClient.fetch_fulltext and PmcBiocClient.fetch_fulltext, the three-way form of the question. It is the unreachable arm, kept apart from a None return, which is absent (the service holds no copy). The two used to be one None, and the pass pinned an outage as an abstract-only row that merge-not-clobber never asked about again. A subclass, so an existing except LiteratureUnavailable still catches it.

LiteratureRetired

Bases: LiteratureUnavailable, EndpointRetired

Europe PMC answered HTTP 410: it retired the endpoint asked (RM354).

A fact about the source, not an outage, and not retried; a LiteratureUnavailable so every existing handler still catches it. Raised by EuropePmcClient.lookup only: the fulltext routes raise FulltextRetired, which their callers already catch as FulltextUnavailable.

FulltextRetired

Bases: FulltextUnavailable, EndpointRetired

A fulltext service answered HTTP 410 for the route asked (RM361).

Before RM361 every status below 500 read as absent, so a retired route wrote a permanent fulltext_status="absent" about every article it was asked for. A FulltextUnavailable, so the pass's handler still catches it; the pass records retired and asks again on the next run.

DoiConflict dataclass

DoiConflict(pmid: str, authored: str, registry: str)

An authored DOI that disagrees with the one the registry reports for the same PMID.

PmcidConflict dataclass

PmcidConflict(pmid: str, authored: str, registry: str)

An authored PMC id that disagrees with the one PubMed reports for the same PMID (RM50).

The DoiConflict shape, for the other cross-registry identifier. It costs no request — the PMC id is already in the esummary articleids block that answered existence — and it catches the case the schema guard cannot see: a cell like 21551363 (PMC3110567) carries a real PubMed id, so nothing refuses it, while the two halves name different articles.

DoiCitation dataclass

DoiCitation(
    doi: str,
    exists: bool | None = None,
    crossref_asked: bool = False,
    title: str | None = None,
    agency: str | None = None,
    quotes_authored: int = 0,
    quotes_found: int | None = None,
    quote_source: str | None = None,
    fulltext_status: str | None = None,
    europepmc_id: str | None = None,
    license: str | None = None,
)

What one run learned about a work cited by DOI alone, through pmid=NONE (RM349).

One per distinct DOI (spelling.doi_key), however many rows cite it and in whatever spelling. Not written to literature.csv, whose rows are keyed on a PubMed id and feed a published parquet the compiler reads by PMID; so it is not pinned either, and every online run asks again. What it records goes into verification.json, which is the durable half.

  • exists is Crossref's tri-state (None: not asked under --no-doi, or never answered — crossref_asked tells the two apart), and title the work Crossref names for it (@existence-not-identity). No citing table carries a stated title to compare it with, so it is recorded and reported, never compared. agency is who registered the DOI when Crossref did not (CrossrefWork.agency).
  • The quote half has LiteratureRow's columns and meanings: fulltext_status is retrieved, absent or unreachable (None when no quote asked the question), quote_source says which text was searched, and quotes_checked() is the same rule (settled_quotes).

LiteratureResult dataclass

LiteratureResult(
    rows: list[LiteratureRow],
    missing: list[str] = list(),
    crossref_unanswered: str = UNREACHABLE,
    doi_conflicts: list[DoiConflict] = list(),
    pmcid_conflicts: list[PmcidConflict] = list(),
    cited: list[str] = list(),
    existence_checked: int = 0,
    unresolved_citations: int = 0,
    doi_verdicts_stale: int = 0,
    doi_never_checked: int = 0,
    noncommercial_quoted: list[str] = list(),
    titles_as_quotes: list[str] = list(),
    fulltext_requested: bool = True,
    quotes_authored: int = 0,
    quotes_found: int = 0,
    quotes_checked: int = 0,
    quotes_unchecked: int = 0,
    quotes_unexamined: int = 0,
    quotes_without_pmid: int = 0,
    citations_without_pmid: int = 0,
    doi_citations: list[DoiCitation] = list(),
    fulltext_checked: list[str] = list(),
    abstract_checked: list[str] = list(),
    doi_missing: list[str] = list(),
    identifiers_authored: int = 0,
    identifiers_compared: int = 0,
    identifiers_conflicting: int = 0,
    identifiers_unmatched: int = 0,
    identifiers_foreign: int = 0,
    sources: list[str] = list(),
    mode: str = "best_effort",
    skipped_offline: bool = False,
)

What the pass found, and — through subject_rows — what it found it about.

One subject set, read by the report, by the strict gates and by the attestation. The set is the citations the module makes now, answered by the rows literature.csv holds now, and it is the single rule the whole pass turns on. Three separate defects came from parts of this file reading three different sets:

  • the strict gates read lists appended inside the fetch loop, so a module enriched once with --best-effort and then with --strict was blessed on a citation PubMed has no record of — the refusal depended on the order the two runs happened in, and the attestation written by the same run said findings=1 while the gate beside it said nothing was wrong;
  • the identifier cross-check ran inside that loop too, so an existing literature.csv hid every DOI/PMC id disagreement;
  • the counts read rows, which is everything the sidecar carries and never shrinks, so a citation deleted from studies.csv went on being counted and no authored edit could clear the finding it produced.

Merge-not-clobber is what makes the distinction real: the sidecar keeps rows for citations the module has since dropped, and those rows are still written back (deleting a curator's row is not this pass's call) — they are simply not what any of this run's questions were about.

subject_rows property

subject_rows: list[LiteratureRow]

The rows this run's questions were about: a row for a citation the module still makes.

Not rows, which is the whole sidecar — see the class docstring. Not this run's fetch list either: literature.csv is the pin, so a row merged from an earlier run carries a verdict that still stands, and counting only what this run looked up would let a no-op re-run replace a true attestation with subjects=0, which reads as "the check ran and had nothing in scope".

coverage property

coverage: str

The sentence this pass exists to be able to say honestly.

The denominator is what had something to check — a citation with no authored quote was not skipped for lack of a fulltext, it simply asked no question. Counting it as unretrievable was a real bug found by running this against reference_examples/pathogenic_clinvar/, whose single citation is open access and carries no quote: the old wording claimed its fulltext could not be retrieved, which was the opposite of true.

Counted in quotes, off the tally, and not in citations off the run's own fetch lists. The earlier wording read fulltext_checked, which only ever holds PMIDs this run retrieved, so a no-op re-run over a pinned open-access citation printed "0 of 1 … 1 with nothing retrievable" one line above "1/1 found" — the same false retrievability claim the paragraph above records as a bug, arrived at from the other direction. Quotes also partition cleanly where citations do not: one citation can carry a settled quote and a never-examined one at once, so a citation-granular sentence has to put it in one bucket and be wrong about the other.

EuropePmcClient dataclass

EuropePmcClient(
    base_url: str = DEFAULT_EUROPEPMC_BASE,
    repo_url: str = DEFAULT_EUROPEPMC_REPO_URL,
    batch_size: int = 25,
    min_request_interval: float = 0.5,
    timeout: float = 30.0,
    gate: PacingGate | None = None,
    _client: Client | None = None,
)

Open-access lookup and fulltext retrieval. Paced like every other client in this tier.

lookup

lookup(pmids: list[str]) -> dict[str, dict]

pmid -> {pmcid, doi, is_open_access, license, abstract} for the ids Europe PMC knows.

Ids it does not know are simply absent from the result, with no error marker — which is why this is not an existence check. A caller must read a miss here as "not retrievable", never as "does not exist".

The abstract comes back for paywalled records too, in this same response, and that is worth more than it looks: probed across a mix of non-open-access papers, four of five carried one (only a 1994 non-research document did not). It is the difference between checking a quote for the open-access minority and checking it for nearly everything.

So does the article's license (RM46), which is why per-article terms cost no extra request. Probed 2026-08-13 over 100 records: the values are lowercase CC spellings — cc by (64), cc by-nc (28), cc by-nc-nd (8). It is independent of isOpenAccess and must not be derived from it: PMID 28546431 comes back isOpenAccess: N with license: cc by, since the flag describes Europe PMC's OA subset while the licence describes the article. Stored verbatim; licensing.article_terms maps it to rights at read time.

Source code in enricher/src/just_dna_enricher/literature.py
def lookup(self, pmids: list[str]) -> dict[str, dict]:
    """`pmid -> {pmcid, doi, is_open_access, license, abstract}` for the ids Europe PMC knows.

    Ids it does not know are simply **absent from the result**, with no error marker — which is why
    this is not an existence check. A caller must read a miss here as "not retrievable", never as
    "does not exist".

    **The abstract comes back for paywalled records too**, in this same response, and that is worth
    more than it looks: probed across a mix of non-open-access papers, four of five carried one
    (only a 1994 non-research document did not). It is the difference between checking a quote for
    the open-access minority and checking it for nearly everything.

    **So does the article's `license`** (RM46), which is why per-article terms cost no extra
    request. Probed 2026-08-13 over 100 records: the values are lowercase CC spellings — `cc by`
    (64), `cc by-nc` (28), `cc by-nc-nd` (8). It is **independent of `isOpenAccess`** and must not
    be derived from it: PMID 28546431 comes back `isOpenAccess: N` with `license: cc by`, since
    the flag describes Europe PMC's OA subset while the licence describes the article. Stored
    verbatim; `licensing.article_terms` maps it to rights at read time.
    """
    self._check()  # a field may have been set since the client was built (RM455)
    out: dict[str, dict] = {}
    for batch in batched(dedupe(pmids), self.batch_size):
        query = " OR ".join(f"EXT_ID:{pmid}" for pmid in batch)
        # **Translate, do not leak** (`@client-exception-contract`, RM230). `_get` retries the
        # transport legs and re-raises what survives; this method used to call it bare and then
        # `.json()` the result, so all three failure legs escaped as their httpx/json types:
        # a persistent 503 as `httpx.HTTPStatusError`, a refused connection as
        # `httpx.ConnectError`, and a 200 that is not JSON as `json.JSONDecodeError`. The last is
        # the exact fourth leg `test_client_exception_contract.py` was written for, and every
        # sibling client in this tier closes it in its own module.
        #
        # It mattered where it landed: `enrich_literature` calls this inside a `try:` whose only
        # companion is `finally:` — no `except` — which is verbatim the shape
        # `test_pass_exception_contract.py` exists to refuse. The pass-level suite never drove
        # this leg because its stub raises earlier, on the eutils call.
        #
        # `fulltext` below is deliberately NOT changed: it catches httpx and returns `None`, the
        # tri-state withhold this tier uses for "could not be retrieved". The two methods answer
        # different questions, which is why the class's contract-suite exemption named `fulltext`
        # and silently covered `lookup` too.
        try:
            response = self._get("search", {"query": query, "resultType": "core", "format": "json"})
            payload = response.json()
        except httpx.HTTPError as exc:
            raise_if_retired(exc, LiteratureRetired, "Europe PMC")
            raise LiteratureUnavailable(f"Europe PMC could not be asked: {exc}") from exc
        except ValueError as exc:
            raise LiteratureUnavailable(
                f"Europe PMC answered {response.status_code} with a body that is not JSON: {exc}"
            ) from exc
        for record in (payload.get("resultList") or {}).get("result") or []:
            pmid = str(record.get("pmid") or "")
            if not pmid:
                continue
            out[pmid] = {
                "pmcid": record.get("pmcid"),
                "doi": record.get("doi"),
                # The API answers with the strings 'Y'/'N', not booleans.
                "is_open_access": _yes_no(record.get("isOpenAccess")),
                # `inEPMC` is NOT a fulltext signal: a record can be in Europe PMC with
                # `isOpenAccess=N`, and `fullTextXML` then answers 404. Probed on PMID 23788249.
                "in_epmc": _yes_no(record.get("inEPMC")),
                # Verbatim, and `None` when the field is absent — an article whose terms Europe
                # PMC does not state is unknown, not unlicensed.
                "license": (record.get("license") or None),
                "abstract": record.get("abstractText"),
            }
    return out

pmids_for_doi

pmids_for_doi(doi: str) -> list[str]

The PubMed ids Europe PMC holds for one DOI, sorted, from its MEDLINE source (RM263).

Europe PMC's search covers all of PubMed, where NCBI's id converter answers only for articles in PMC, so this reaches the paywalled case the converter misses (10.1038/ng826 is 11788828, Enattah 2002, not in PMC). An empty list is Europe PMC's answer that it holds no MEDLINE record for the DOI, never a statement about PubMed. More than one id is returned as it came, so a caller can report the multiplicity rather than pick. Failures translate to LiteratureUnavailable on every leg, as lookup does (@client-exception-contract).

Source code in enricher/src/just_dna_enricher/literature.py
def pmids_for_doi(self, doi: str) -> list[str]:
    """The PubMed ids Europe PMC holds for one DOI, sorted, from its MEDLINE source (RM263).

    Europe PMC's search covers all of PubMed, where NCBI's id converter answers only for articles
    in PMC, so this reaches the paywalled case the converter misses (`10.1038/ng826` is
    `11788828`, Enattah 2002, not in PMC). An empty list is **Europe PMC's** answer that it holds
    no MEDLINE record for the DOI, never a statement about PubMed. More than one id is returned as
    it came, so a caller can report the multiplicity rather than pick. Failures translate to
    `LiteratureUnavailable` on every leg, as `lookup` does (`@client-exception-contract`).
    """
    try:
        response = self._get(
            "search", {"query": f'DOI:"{doi}" AND SRC:MED', "resultType": "lite", "format": "json"}
        )
        payload = response.json()
    except httpx.HTTPError as exc:
        raise LiteratureUnavailable(f"Europe PMC could not be asked about DOI {doi}: {exc}") from exc
    except ValueError as exc:
        raise LiteratureUnavailable(
            f"Europe PMC answered {response.status_code} with a body that is not JSON: {exc}"
        ) from exc
    records = (payload.get("resultList") or {}).get("result") or []
    return sorted({str(r["pmid"]) for r in records if r.get("pmid")}, key=int)

fetch_fulltext

fetch_fulltext(pmcid: str) -> str | None

Whitespace-normalized article text; None when Europe PMC holds no copy; raises FulltextUnavailable when the request never got an answer (RM258, RM361).

A 404 is an answer: an embargoed or author-manuscript-only record answers it, which is absent. A 410 raises FulltextRetired. A 429, any other 4xx, a 5xx, a transport failure and a 200 whose body does not parse as XML are unreachable, and the caller asks again on a later run.

Source code in enricher/src/just_dna_enricher/literature.py
def fetch_fulltext(self, pmcid: str) -> str | None:
    """Whitespace-normalized article text; `None` when Europe PMC holds no copy; raises
    `FulltextUnavailable` when the request never got an answer (RM258, RM361).

    A 404 is an answer: an embargoed or author-manuscript-only record answers it, which is
    *absent*. A 410 raises `FulltextRetired`. A 429, any other 4xx, a 5xx, a transport failure
    and a 200 whose body does not parse as XML are *unreachable*, and the caller asks again on a
    later run.
    """
    try:
        response = self._get(f"{pmcid}/fullTextXML")
    except httpx.HTTPStatusError as exc:
        _fulltext_absent_or_raise(exc, "Europe PMC", pmcid)
        return None
    except httpx.HTTPError as exc:
        raise FulltextUnavailable(
            f"Europe PMC could not be reached for {pmcid}'s fulltext: {exc}"
        ) from exc
    text = extract_text(response.text)
    if text is None:
        # A 200 that is not JATS (a maintenance page) is the fourth leg: withheld, never `absent`.
        raise FulltextUnavailable(
            f"Europe PMC answered 200 for {pmcid}'s fulltext with a body that does not parse"
        )
    return text

fulltext

fulltext(pmcid: str) -> str | None

fetch_fulltext with its two negative answers folded into one None, as before 0.8.

Kept for callers that already treat None as "no text either way" (the authoring plugin's fetch_fulltext tool relies on it for Europe PMC's 500s). A caller that needs to tell an outage from an absent copy uses fetch_fulltext.

Source code in enricher/src/just_dna_enricher/literature.py
def fulltext(self, pmcid: str) -> str | None:
    """`fetch_fulltext` with its two negative answers folded into one `None`, as before 0.8.

    Kept for callers that already treat `None` as "no text either way" (the authoring plugin's
    `fetch_fulltext` tool relies on it for Europe PMC's 500s). A caller that needs to tell an
    outage from an absent copy uses `fetch_fulltext`.
    """
    try:
        return self.fetch_fulltext(pmcid)
    except FulltextUnavailable as exc:
        logger.warning("Fulltext fetch failed for %s (%s)", pmcid, exc)
        return None

record_for_doi

record_for_doi(doi: str) -> dict | None

Europe PMC's record for one DOI, from any of its sources, or None when it holds none (RM349).

The route to a pmid=NONE citation's text. Europe PMC indexes bioRxiv and medRxiv preprints (source PPR) by DOI, so the same search that pmids_for_doi restricts to MEDLINE reaches them here unrestricted. Only a record whose own doi is the one asked about is taken: the search is a query, and a query's first hit need not be the work (@existence-not-identity).

Returns {id, source, pmcid, is_open_access, license, abstract, title, preprint_id}. preprint_id is the PPR id when the record says Europe PMC holds the preprint's full text (its fullTextIdList), and None otherwise, so a caller asks the repository only where there is something to fetch: probed 2026-11-04, the repository's answer for a preprint with no text (PPR1146237) is a 500, the same status as an outage. Failures translate to LiteratureUnavailable on every leg, as pmids_for_doi does (@client-exception-contract).

Source code in enricher/src/just_dna_enricher/literature.py
def record_for_doi(self, doi: str) -> dict | None:
    """Europe PMC's record for one DOI, from any of its sources, or `None` when it holds none (RM349).

    The route to a `pmid=NONE` citation's text. Europe PMC indexes bioRxiv and medRxiv preprints
    (source `PPR`) by DOI, so the same `search` that `pmids_for_doi` restricts to MEDLINE reaches
    them here unrestricted. Only a record whose own `doi` is the one asked about is taken: the
    search is a query, and a query's first hit need not be the work (`@existence-not-identity`).

    Returns `{id, source, pmcid, is_open_access, license, abstract, title, preprint_id}`.
    `preprint_id` is the `PPR` id when the record says Europe PMC holds the preprint's full text
    (its `fullTextIdList`), and `None` otherwise, so a caller asks the repository only where there
    is something to fetch: probed 2026-11-04, the repository's answer for a preprint with no text
    (`PPR1146237`) is a 500, the same status as an outage. Failures translate to
    `LiteratureUnavailable` on every leg, as `pmids_for_doi` does (`@client-exception-contract`).
    """
    try:
        response = self._get("search", {"query": f'DOI:"{doi}"', "resultType": "core", "format": "json"})
        payload = response.json()
    except httpx.HTTPError as exc:
        raise LiteratureUnavailable(f"Europe PMC could not be asked about DOI {doi}: {exc}") from exc
    except ValueError as exc:
        raise LiteratureUnavailable(
            f"Europe PMC answered {response.status_code} with a body that is not JSON: {exc}"
        ) from exc
    wanted = doi_key(doi)
    matching: list[dict] = []
    for record in (payload.get("resultList") or {}).get("result") or []:
        if not record.get("doi") or doi_key(str(record["doi"])) != wanted:
            continue
        record_id = str(record.get("id") or "")
        fulltext_ids = (record.get("fullTextIdList") or {}).get("fullTextId") or []
        preprint = record.get("source") == "PPR" and record_id in fulltext_ids
        matching.append(
            {
                "id": record_id or None,
                "source": record.get("source"),
                "pmcid": record.get("pmcid"),
                "is_open_access": _yes_no(record.get("isOpenAccess")),
                # Verbatim, and `None` when absent: unknown terms are unknown, never unlicensed.
                "license": record.get("license") or None,
                "abstract": record.get("abstractText"),
                "title": record.get("title"),
                "preprint_id": record_id if preprint else None,
            }
        )
    # One DOI can have several records (a MEDLINE or AGRICOLA one beside the preprint's), and only
    # some carry a route to the text: the first one that does wins, else the first match, so an
    # earlier text-less hit never makes the answer `absent` while a later one holds the text.
    return next(
        (r for r in matching if r["pmcid"] or r["preprint_id"]), matching[0] if matching else None
    )

fetch_preprint_fulltext

fetch_preprint_fulltext(preprint_id: str) -> str | None

A preprint's JATS body as normalized text; None when the repository lists no XML file for it; raises FulltextUnavailable when a request never got an answer (RM349, the RM258 shape).

Two requests: the record's file list (type=METADATA), then the JATS file it names (type=FILE). The file name (EMS…nxml) is never guessed, and the PDF the list also names is never used: its link is session-bound (it answered 403 "expired" on 2026-11-04). Ask only for a preprint whose search record lists a full text (record_for_doi's preprint_id), because a 5xx here then means an outage rather than the repository's "no such copy", which shares it. A 200 that is not the list is the fourth leg and withholds too (@client-exception-contract).

Source code in enricher/src/just_dna_enricher/literature.py
def fetch_preprint_fulltext(self, preprint_id: str) -> str | None:
    """A preprint's JATS body as normalized text; `None` when the repository lists no XML file for
    it; raises `FulltextUnavailable` when a request never got an answer (RM349, the RM258 shape).

    Two requests: the record's file list (`type=METADATA`), then the JATS file it names
    (`type=FILE`). The file name (`EMS…nxml`) is never guessed, and the PDF the list also names is
    never used: its link is session-bound (it answered 403 "expired" on 2026-11-04). Ask only for a
    preprint whose search record lists a full text (`record_for_doi`'s `preprint_id`), because a
    5xx here then means an outage rather than the repository's "no such copy", which shares it.
    A 200 that is not the list is the fourth leg and withholds too (`@client-exception-contract`).
    """
    try:
        listing = self._fetch(self.repo_url, {"pprId": preprint_id, "type": "METADATA"})
        files = listing.json().get("files") or []
        xml = next(
            (f for f in files if isinstance(f, dict) and f.get("type") == "xml" and f.get("filename")),
            None,
        )
        if xml is None:
            logger.info("Europe PMC lists no XML full text for %s", preprint_id)
            return None
        response = self._fetch(
            self.repo_url,
            {
                "pprId": preprint_id,
                "type": "FILE",
                "fileName": xml["filename"],
                "mimeType": xml.get("mimeType") or "application/xml",
            },
        )
    except httpx.HTTPStatusError as exc:
        _fulltext_absent_or_raise(exc, "Europe PMC's preprint repository", preprint_id)
        return None
    except httpx.HTTPError as exc:
        raise FulltextUnavailable(
            f"Europe PMC's preprint repository could not be reached for {preprint_id}: {exc}"
        ) from exc
    except (ValueError, AttributeError) as exc:
        raise FulltextUnavailable(
            f"Europe PMC's preprint repository answered for {preprint_id} with no readable file list: {exc}"
        ) from exc
    text = extract_text(response.text)
    if text is None:
        # A 200 that is not JATS (a maintenance page) is the fourth leg: withheld, never `absent`.
        raise FulltextUnavailable(
            f"Europe PMC's preprint repository answered 200 for {preprint_id}'s JATS with a body that "
            f"does not parse"
        )
    return text

CrossrefWork dataclass

CrossrefWork(
    exists: bool | None,
    title: str | None = None,
    journal: str | None = None,
    year: str | None = None,
    first_author: str | None = None,
    agency: str | None = None,
)

What Crossref said about one DOI: whether it exists, and which work it names (RM262, S113).

exists is the tri-state CrossrefClient.exists has always answered. The four bibliographic fields are the same four bibliographic pulls out of a PubMed record, so a DOI-only citation can be checked for identity and not merely existence. Each is None when the record does not carry it, never "", and all four are None whenever exists is not True.

CrossrefClient dataclass

CrossrefClient(
    base_url: str = DEFAULT_CROSSREF_BASE,
    min_request_interval: float = 0.1,
    timeout: float = 30.0,
    contact_email: str | None = None,
    gate: PacingGate | None = None,
    _client: Client | None = None,
)

Bases: Withholding

DOI existence, for the citations PubMed does not index.

PubMed answers "does this article exist" for anything it indexes — including paywalled work; a paywall governs the fulltext, not the record. What it cannot answer for is everything outside its scope: preprints, books, theses, datasets, standards. Those have DOIs and no PMID, and Crossref is the registry that mints and resolves them (a probed bioRxiv preprint, 10.1101/2024.06.17.599351, returns type: posted-content; a fabricated DOI returns a clean 404).

This is also what makes the 1.0 doi-first flip low-risk: once pmid becomes optional and a citation may carry only a DOI, existence checking has to work without PubMed, and it already does.

Crossref asks callers to identify themselves in the User-Agent and gives the "polite pool" in return; the contact address is sent only when one is configured, for the same reason eutils omits it rather than inventing one.

exists

exists(doi: str) -> bool | None

True/False, or None when Crossref could not be asked. work(doi).exists.

Source code in enricher/src/just_dna_enricher/literature.py
def exists(self, doi: str) -> bool | None:
    """`True`/`False`, or `None` when Crossref could not be asked. `work(doi).exists`."""
    return self.work(doi).exists

work

work(doi: str) -> CrossrefWork

Existence and identity, from the one request exists always made (RM262, S113).

exists used to read the status and throw the body away, so a DOI got an existence answer and never the title that says which paper it is — the answer existence is not identity exists to refuse. The body was already in hand; parsing it costs no request.

exists=None rather than False on a transport failure or an unexpected status: "we could not check" and "this DOI does not exist" are different claims, and only the second is a finding against the module. The translation stays here and the retrying stays in _request above — reraise=True means the last failure arrives back at this except after the attempts are spent. A 200 whose body is not a Crossref record withholds too (@client-exception-contract, the fourth leg): a maintenance page or a CDN interstitial is not Crossref saying the DOI exists, and exists used to answer True for one because it read only the status.

Source code in enricher/src/just_dna_enricher/literature.py
def work(self, doi: str) -> CrossrefWork:
    """Existence **and** identity, from the one request `exists` always made (RM262, S113).

    `exists` used to read the status and throw the body away, so a DOI got an existence answer and
    never the title that says which paper it is — the answer `existence is not identity` exists to
    refuse. The body was already in hand; parsing it costs no request.

    `exists=None` rather than `False` on a transport failure or an unexpected status: "we could not
    check" and "this DOI does not exist" are different claims, and only the second is a finding
    against the module. The translation stays here and the retrying stays in `_request` above —
    `reraise=True` means the last failure arrives back at this `except` after the attempts are
    spent. **A 200 whose body is not a Crossref record withholds too** (`@client-exception-contract`,
    the fourth leg): a maintenance page or a CDN interstitial is not Crossref saying the DOI exists,
    and `exists` used to answer `True` for one because it read only the status.
    """
    try:
        response = self._request(doi)
    except httpx.HTTPError as exc:
        logger.warning("Crossref lookup failed for %s (%s); not checked", doi, exc)
        return CrossrefWork(exists=None)
    if response.status_code == 404:
        return self._registered_elsewhere(doi)
    if response.status_code != 200:
        self.note_retired("Crossref", response)
        logger.warning("Crossref answered HTTP %s for %s; not checked", response.status_code, doi)
        return CrossrefWork(exists=None)
    try:
        message = response.json().get("message")
    except (ValueError, AttributeError):
        message = None
    if not isinstance(message, dict):
        logger.warning("Crossref answered 200 for %s with no readable record; not checked", doi)
        return CrossrefWork(exists=None)
    return CrossrefWork(exists=True, agency="crossref", **crossref_bibliographic(message))

PmcIdRecord dataclass

PmcIdRecord(
    pmcid: str,
    pmid: str | None = None,
    in_pmc: bool = True,
    error: str | None = None,
)

One answer from the PMC id converter. in_pmc=False is PMC saying it has no such record; an id the request never reached is absent from the result entirely, never one of these.

PmcIdConverterClient dataclass

PmcIdConverterClient(
    base_url: str = DEFAULT_PMC_IDCONV_URL,
    batch_size: int = 200,
    min_request_interval: float = 0.5,
    timeout: float = 30.0,
    contact_email: str | None = None,
    gate: PacingGate | None = None,
    _client: Client | None = None,
)

Bases: Withholding

PMCID → PMID, for a curator holding the wrong half of the pair (RM50).

It reports; it never fills. The PMID it returns is handed back as an advisory the author types themselves, because filling StudyRow.pmid from NCBI would make literature.exists compare NCBI against NCBI — the hints.REDUNDANCY_BEARING rule, already argued for doi (Crossref is asked about the authored DOI, since a derived one exists by construction).

Why this exists although the pass docstring says the converter is unused. That statement is true of the direction the pass needs — PMID → PMCID arrives free in the esummary articleids block, so calling the converter for it would be a third request for data already in hand, and it answers a different question (asked about a paywalled PMID it replies "Identifier not found in PMC", which is about PMC membership rather than existence). None of that says anything about PMCID → PMID, which is the direction the converter is actually for and the one a curator has no other route to.

Probed 2026-08-13: the www.ncbi.nlm.nih.gov/pmc/utils/idconv/v1.0/ path 301-redirects to the address below, an absent id comes back as a record with status: "error" and errmsg: "Identifier not found in PMC", and pmid arrives as a JSON number.

resolve

resolve(pmcids: list[str]) -> dict[str, PmcIdRecord]

PMCID -> PmcIdRecord for every id the converter answered about.

Four outcomes, and none of them is spelled the same way as another. An id it resolved carries a pmid; an id it knows with no PubMed id carries in_pmc=True, pmid=None (a real answer — some PMC records genuinely have none); an id it has no record of carries in_pmc=False with the service's own errmsg; and an id the request never reached is simply absent from the result. A caller must not read a missing key as "does not exist" — that conflation is exactly what S20 was about, one service over.

Source code in enricher/src/just_dna_enricher/literature.py
def resolve(self, pmcids: list[str]) -> dict[str, PmcIdRecord]:
    """`PMCID -> PmcIdRecord` for every id the converter answered about.

    **Four outcomes, and none of them is spelled the same way as another.** An id it resolved
    carries a `pmid`; an id it knows with no PubMed id carries `in_pmc=True, pmid=None` (a real
    answer — some PMC records genuinely have none); an id it has no record of carries
    `in_pmc=False` with the service's own `errmsg`; and an id the request never reached is simply
    **absent from the result**. A caller must not read a missing key as "does not exist" — that
    conflation is exactly what S20 was about, one service over.
    """
    self._check()  # a field may have been set since the client was built (RM455)
    out: dict[str, PmcIdRecord] = {}
    for batch in batched(dedupe(pmcids), self.batch_size):
        params = {"ids": ",".join(batch), "format": "json", "tool": "just-dna-enricher"}
        if self.contact_email:
            params["email"] = self.contact_email
        try:
            payload = self._get(params).json()
        except httpx.HTTPError as exc:
            self.note_retired("The PMC id converter", exc)
            logger.warning("PMC id converter could not be asked for %s (%s)", batch, exc)
            continue
        for record in payload.get("records") or []:
            requested = str(record.get("requested-id") or record.get("pmcid") or "")
            if not requested:
                continue
            pmid = record.get("pmid")
            out[requested] = PmcIdRecord(
                pmcid=requested,
                pmid=str(pmid) if pmid is not None else None,
                in_pmc=record.get("status") != "error",
                error=record.get("errmsg"),
            )
    return out

PmcBiocClient dataclass

PmcBiocClient(
    base_url: str = DEFAULT_PMC_BIOC_URL,
    min_request_interval: float = 0.5,
    timeout: float = 60.0,
    gate: PacingGate | None = None,
    _client: Client | None = None,
)

Bases: Withholding

PMC's BioC text-mining service: the body text of an article Europe PMC will not serve (RM257, S110).

The records this reaches are the ones the Europe PMC rung cannot. An NIH author manuscript is isOpenAccess: N in Europe PMC, so the pass never asked it for fulltext, and asking anyway answered HTTP 500 for PMC6463297 on 2026-09-24. PMC serves the same manuscript here for text mining — 294 passages, both body tables among them as tab-separated table passages, and for a GWAS paper the tables are where the per-locus rows live. Probed the same day: an article outside the service's set answers 200 with the body [Error] : No result can be found., which is an answer (no copy) rather than an outage, so it reads as absent like a 404 does.

Retrieval here says nothing about the article's licence. The service's own infon reads "This file is available for text mining. It may also be used consistent with the principles of fair use", which names no licence, so the row keeps Europe PMC's is_open_access and license untouched (@no-named-licence). It changes how far the quote search reached, which is what quote_source records, and nothing else.

REF passages are dropped: they are the reference list, most of the document by count (186 of 294 on the probed article), and a quote located in someone else's title is not located in this paper. An NCBI host, so the pass hands it the E-utilities gate rather than giving it a budget of its own (@shared-pacing-gate).

fetch_fulltext

fetch_fulltext(pmcid: str) -> str | None

Body text with the reference list removed; None when the service holds no copy; raises FulltextUnavailable when the request never got an answer (RM258).

Only the recorded "no copy" 200 is an answer (RM361). The service's reply for an article outside its set is a 200 with a plain-text body opening BIOC_NO_RESULT_PREFIX (assets/pmc_bioc_no_result.txt), so it reads as absent, as a 404 does. Any other 200 that is not JSON (a maintenance page) is the fourth leg and unreachable; a 410 raises FulltextRetired; a 429, any other 4xx, a 5xx or a transport failure, after the retries, is unreachable.

Source code in enricher/src/just_dna_enricher/literature.py
def fetch_fulltext(self, pmcid: str) -> str | None:
    """Body text with the reference list removed; `None` when the service holds no copy; raises
    `FulltextUnavailable` when the request never got an answer (RM258).

    **Only the recorded "no copy" 200 is an answer** (RM361). The service's reply for an article
    outside its set is a 200 with a plain-text body opening `BIOC_NO_RESULT_PREFIX`
    (`assets/pmc_bioc_no_result.txt`), so it reads as *absent*, as a 404 does. Any other 200 that
    is not JSON (a maintenance page) is the fourth leg and *unreachable*; a 410 raises
    `FulltextRetired`; a 429, any other 4xx, a 5xx or a transport failure, after the retries, is
    *unreachable*.
    """
    try:
        response = self._get(pmcid)
    except httpx.HTTPStatusError as exc:
        _fulltext_absent_or_raise(exc, "PMC BioC", pmcid)
        return None
    except httpx.HTTPError as exc:
        raise FulltextUnavailable(f"PMC BioC could not be reached for {pmcid}: {exc}") from exc
    try:
        payload = response.json()
    except ValueError as exc:
        if response.text.lstrip().startswith(BIOC_NO_RESULT_PREFIX):
            logger.info("No BioC text for %s (the service's 'no result' answer)", pmcid)
            return None
        raise FulltextUnavailable(
            f"PMC BioC answered 200 for {pmcid} with a body that is neither JSON nor its 'no result' "
            f"answer: {' '.join(response.text.split())[:80]!r}"
        ) from exc
    return extract_bioc_text(payload)

fulltext

fulltext(pmcid: str) -> str | None

fetch_fulltext with its two negative answers folded into one None, as before 0.8.

Source code in enricher/src/just_dna_enricher/literature.py
def fulltext(self, pmcid: str) -> str | None:
    """`fetch_fulltext` with its two negative answers folded into one `None`, as before 0.8."""
    try:
        return self.fetch_fulltext(pmcid)
    except FulltextUnavailable as exc:
        if isinstance(exc, FulltextRetired) and isinstance(exc.__cause__, httpx.HTTPStatusError):
            self.note_retired("PMC BioC", exc.__cause__)
        logger.warning("BioC fetch failed for %s (%s)", pmcid, exc)
        return None

crossref_bibliographic

crossref_bibliographic(
    message: dict,
) -> dict[str, str | None]

The four identity fields out of a Crossref /works/{doi} message, shaped like bibliographic.

The shapes differ from PubMed's and each difference is a place to go wrong. title and container-title are lists, empty for much posted-content. The year is the leading int of issued.date-parts[0], Crossref's earliest-publication date. The first author is the entry marked sequence: "first", which need not be at index 0, and is rendered Family GI so it reads the same as PubMed's sortfirstauthor; an organisational author has only a name. Title markup (Crossref passes JATS tags such as <i> through) is kept verbatim: the field is for a person to compare against the paper they meant, and rewriting it would be a second opinion about the title.

Source code in enricher/src/just_dna_enricher/literature.py
def crossref_bibliographic(message: dict) -> dict[str, str | None]:
    """The four identity fields out of a Crossref `/works/{doi}` `message`, shaped like `bibliographic`.

    The shapes differ from PubMed's and each difference is a place to go wrong. `title` and
    `container-title` are **lists**, empty for much `posted-content`. The year is the leading int of
    `issued.date-parts[0]`, Crossref's earliest-publication date. The first author is the entry marked
    `sequence: "first"`, which need not be at index 0, and is rendered `Family GI` so it reads the same
    as PubMed's `sortfirstauthor`; an organisational author has only a `name`. Title markup (Crossref
    passes JATS tags such as `<i>` through) is kept verbatim: the field is for a person to compare
    against the paper they meant, and rewriting it would be a second opinion about the title.
    """
    out: dict[str, str | None] = {"title": None, "journal": None, "year": None, "first_author": None}
    for key, target in (("title", "title"), ("container-title", "journal")):
        values = message.get(key)
        if isinstance(values, list) and values and isinstance(values[0], str) and values[0].strip():
            out[target] = " ".join(values[0].split())
    parts = (message.get("issued") or {}).get("date-parts")
    if (
        isinstance(parts, list)
        and parts
        and isinstance(parts[0], list)
        and parts[0]
        and isinstance(parts[0][0], int)
    ):
        out["year"] = str(parts[0][0])
    authors = [a for a in message.get("author") or [] if isinstance(a, dict)]
    first = next((a for a in authors if a.get("sequence") == "first"), authors[0] if authors else None)
    if first is not None:
        family = first.get("family")
        if isinstance(family, str) and family.strip():
            given = first.get("given") if isinstance(first.get("given"), str) else ""
            initials = "".join(part[0] for part in given.replace("-", " ").split() if part[:1].isalpha())
            out["first_author"] = f"{family.strip()} {initials}".strip()
        elif isinstance(first.get("name"), str) and first["name"].strip():
            out["first_author"] = first["name"].strip()
    return out

extract_bioc_text

extract_bioc_text(payload: object) -> str | None

BioC JSON → one normalized string of every non-reference passage, or None if it holds none.

The service wraps its collection in a one-element list; a bare collection is accepted too, since the BioC format itself does not require the wrapper.

Source code in enricher/src/just_dna_enricher/literature.py
def extract_bioc_text(payload: object) -> str | None:
    """BioC JSON → one normalized string of every non-reference passage, or `None` if it holds none.

    The service wraps its collection in a one-element list; a bare collection is accepted too, since
    the BioC format itself does not require the wrapper.
    """
    collections = payload if isinstance(payload, list) else [payload]
    texts: list[str] = []
    for collection in collections:
        if not isinstance(collection, dict):
            continue
        for document in collection.get("documents") or []:
            for passage in document.get("passages") or []:
                if (passage.get("infons") or {}).get("section_type") == "REF":
                    continue
                if passage.get("text"):
                    texts.append(passage["text"])
    joined = _WHITESPACE.sub(" ", " ".join(texts)).strip()
    return joined or None

extract_text

extract_text(xml: str) -> str | None

JATS XML → one normalized string, or None if it is not a JATS article.

itertext() rather than a tag-stripping regex: real JATS carries nested inline markup inside sentences (<italic>, <xref>, <sup>), and a regex that removes tags without joining their text would silently glue words together and break matches that should succeed.

A document that parses is not yet an article (RM361): a maintenance page served as XHTML is well-formed XML, and reading its text as the article's would make every quote a miss against a fulltext, which settles it. Only a root element named article is JATS.

Source code in enricher/src/just_dna_enricher/literature.py
def extract_text(xml: str) -> str | None:
    """JATS XML → one normalized string, or `None` if it is not a JATS article.

    `itertext()` rather than a tag-stripping regex: real JATS carries nested inline markup inside
    sentences (`<italic>`, `<xref>`, `<sup>`), and a regex that removes tags without joining their
    text would silently glue words together and break matches that should succeed.

    **A document that parses is not yet an article** (RM361): a maintenance page served as XHTML is
    well-formed XML, and reading its text as the article's would make every quote a miss against a
    fulltext, which settles it. Only a root element named `article` is JATS.
    """
    try:
        root = ET.fromstring(xml)
    except ET.ParseError as exc:
        logger.warning("Fulltext did not parse as XML (%s)", exc)
        return None
    if root.tag.rsplit("}", 1)[-1] != "article":
        logger.warning("Fulltext parsed as XML with a <%s> root, not a JATS <article>", root.tag)
        return None
    return _WHITESPACE.sub(" ", " ".join(root.itertext())).strip()

quote_matches

quote_matches(quote: str, fulltext: str) -> bool

Literal, whitespace- and case-insensitive containment. No engine, so nothing to bound.

Source code in enricher/src/just_dna_enricher/literature.py
def quote_matches(quote: str, fulltext: str) -> bool:
    """Literal, whitespace- and case-insensitive containment. No engine, so nothing to bound."""
    return _normalize(quote) in _normalize(fulltext)

regex_matches

regex_matches(
    pattern: str,
    fulltext: str,
    *,
    timeout: float = DEFAULT_REGEX_TIMEOUT,
) -> bool | None

True/False, or None when the match could not be completed in time.

The three-way return is the point: a pattern that runs long has not failed to match, it has failed to be checked, and reporting it as "not found" would send an author to fix a quote that is probably there.

Why a subprocess and not a thread. The obvious implementation — submit to a ThreadPoolExecutor and take future.result(timeout=...) — does not work, and fails in a way that looks like it works: re never releases the GIL to a cancellation, threads cannot be killed, and the interpreter joins non-daemon pool threads at exit. So a runaway pattern returns None on time and then hangs the process on the way out. Verified by writing it that way first and watching the test suite stop. A child process is the only bound in the standard library that can actually be enforced, and it is cheap here: it runs only for a row that has a provenance_regex and a retrievable fulltext, which is a small subset of a small subset.

Source code in enricher/src/just_dna_enricher/literature.py
def regex_matches(pattern: str, fulltext: str, *, timeout: float = DEFAULT_REGEX_TIMEOUT) -> bool | None:
    """`True`/`False`, or **`None` when the match could not be completed in time**.

    The three-way return is the point: a pattern that runs long has not failed to match, it has failed
    to be *checked*, and reporting it as "not found" would send an author to fix a quote that is
    probably there.

    **Why a subprocess and not a thread.** The obvious implementation — submit to a
    `ThreadPoolExecutor` and take `future.result(timeout=...)` — does not work, and fails in a way that
    looks like it works: `re` never releases the GIL to a cancellation, threads cannot be killed, and
    the interpreter joins non-daemon pool threads at exit. So a runaway pattern returns `None` on time
    and then hangs the process on the way out. Verified by writing it that way first and watching the
    test suite stop. A child process is the only bound in the standard library that can actually be
    enforced, and it is cheap here: it runs only for a row that has a `provenance_regex` *and* a
    retrievable fulltext, which is a small subset of a small subset.
    """
    try:
        re.compile(pattern)
    except re.error as exc:
        # The compiler already grammar-checks `provenance_regex`, so this is a corrupt sidecar rather
        # than an authoring slip; abstain rather than claiming the quote is absent.
        logger.warning("provenance_regex %r did not compile (%s); not checked", pattern, exc)
        return None

    sink: multiprocessing.Queue = multiprocessing.Queue()
    worker = multiprocessing.Process(target=_regex_worker, args=(pattern, fulltext, sink))
    worker.daemon = True
    worker.start()
    try:
        worker.join(timeout)
        if worker.is_alive():
            worker.terminate()
            worker.join(1.0)
            logger.warning(
                "provenance_regex %r exceeded %.1fs against a %d-character fulltext; recorded as "
                "NOT CHECKED (never as not-found)",
                pattern,
                timeout,
                len(fulltext),
            )
            return None
        return sink.get_nowait() if not sink.empty() else None
    except Exception as exc:
        logger.warning("provenance_regex %r could not be evaluated (%s); not checked", pattern, exc)
        return None
    finally:
        sink.close()
        if worker.is_alive():
            worker.kill()

enrich_literature

enrich_literature(
    spec_dir: Path,
    *,
    mode: str = "best_effort",
    offline: bool = False,
    check_fulltext: bool = True,
    check_doi: bool = True,
    regex_timeout: float = DEFAULT_REGEX_TIMEOUT,
    write: bool = True,
    eutils: EutilsClient | None = None,
    europepmc: EuropePmcClient | None = None,
    crossref: CrossrefClient | None = None,
    bioc: PmcBiocClient | None = None,
) -> LiteratureResult

Fill literature.csv from the citations a module makes.

studies.csv is one citation site of several (RM47, RM132): MeasureBinRow.pmid grounds the threshold its row states, and PharmVariantRow.pmid grounds the drug/genotype claim its row makes — a claim studies.csv cannot ground, because a study row attaches to the whole variant. A module whose only citations come from those tables is enriched exactly like one with a studies.csv; reading fewer than all the sites would leave the rest unchecked, which is worse than the honest gap the columns replaced. The kinds come from the compiler's own derived registry through load_citing_rows/table_citations, so a kind that gains the column is read here with no edit to this tier.

Existing rows are authoritative and merged, never clobbered — the same rule enrich() applies to resolution.csv, with the same consequence: to regenerate after a machinery change you must delete the file first. That includes the licence columns (RM46): rows written before 0.6 carry no license, and re-running will not back-fill them, because merge-not-clobber cannot tell an absent value from a curator's deliberate blank. Delete the sidecar to re-derive.

--offline makes this a no-op with a warning. There is no offline literature snapshot and there will not be one; once literature.csv is written it is the pin, and later compiles read it offline and deterministically.

Three checks are attested on the way out (_attest), the offline return included: a run that did not put a question has to say so, or the manifest cannot tell it from a run that put one and found nothing.

Source code in enricher/src/just_dna_enricher/literature.py
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
1513
1514
1515
1516
1517
1518
1519
1520
1521
1522
1523
1524
1525
1526
1527
1528
1529
1530
1531
1532
1533
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
1564
1565
1566
1567
1568
1569
1570
1571
1572
1573
1574
1575
1576
1577
1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606
1607
1608
1609
1610
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620
1621
1622
1623
1624
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
1701
1702
1703
1704
1705
1706
1707
1708
1709
1710
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
1725
1726
1727
1728
1729
1730
1731
1732
1733
1734
1735
1736
1737
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
def enrich_literature(
    spec_dir: Path,
    *,
    mode: str = "best_effort",
    offline: bool = False,
    check_fulltext: bool = True,
    check_doi: bool = True,
    regex_timeout: float = DEFAULT_REGEX_TIMEOUT,
    write: bool = True,
    eutils: EutilsClient | None = None,
    europepmc: EuropePmcClient | None = None,
    crossref: CrossrefClient | None = None,
    bioc: PmcBiocClient | None = None,
) -> LiteratureResult:
    """Fill `literature.csv` from the citations a module makes.

    **`studies.csv` is one citation site of several** (RM47, RM132): `MeasureBinRow.pmid` grounds the
    threshold its row states, and `PharmVariantRow.pmid` grounds the drug/genotype claim its row
    makes — a claim `studies.csv` cannot ground, because a study row attaches to the whole variant. A
    module whose only citations come from those tables is enriched exactly like one with a
    `studies.csv`; reading fewer than all the sites would leave the rest unchecked, which is worse
    than the honest gap the columns replaced. The kinds come from the compiler's own derived registry
    through `load_citing_rows`/`table_citations`, so a kind that gains the column is read here with no
    edit to this tier.

    Existing rows are authoritative and merged, never clobbered — the same rule `enrich()` applies to
    `resolution.csv`, with the same consequence: to regenerate after a machinery change you must delete
    the file first. **That includes the licence columns** (RM46): rows written before 0.6 carry no
    `license`, and re-running will not back-fill them, because merge-not-clobber cannot tell an
    absent value from a curator's deliberate blank. Delete the sidecar to re-derive.

    `--offline` makes this a no-op with a warning. There is no offline literature snapshot and there
    will not be one; once `literature.csv` is written it *is* the pin, and later compiles read it
    offline and deterministically.

    **Three checks are attested on the way out** (`_attest`), the offline return included: a run that
    did not put a question has to say so, or the manifest cannot tell it from a run that put one and
    found nothing.
    """
    spec_dir = Path(spec_dir)
    # `studies.csv` stays a plain join: it is an AUTHORED table and lives in the spec root, and
    # `sidecar_path` is for the machine-written files RM49 allowed under `derived/`.
    studies_path = spec_dir / "studies.csv"
    # Through the resolver, never joined by hand (RM99): a module keeping its sidecars under
    # `derived/` (RM49) has this file there, and a pass with its own literal would read the split copy
    # and write a flat one, leaving the module with both -- the collision RM49 made an error rather
    # than a preference. `@sidecar-name-and-place`: write to the file you read.
    output_path = sidecar_path(spec_dir, "literature.csv", error=LiteratureEnrichmentError)

    studies: list[StudyRow] = []
    if studies_path.exists():
        studies, errors, _ = load_csv_rows(studies_path, StudyRow, "studies.csv")
        if errors:
            raise LiteratureEnrichmentError(f"studies.csv is invalid: {errors[0]}")
    # The table pointers, through the compiler's own loader: importing its private citing-kind tuple
    # or keeping a second list of the kinds here is the RM41 shape, and the copy goes stale the next
    # time a model declares a `pmid`. Its `ValueError` is re-raised as this pass's own error so a bad
    # citing row is diagnosed the way a bad `studies.csv` two lines above already is, rather than
    # tracebacking out of the CLI.
    try:
        citing_rows = load_citing_rows(spec_dir)
    except ValueError as exc:
        raise LiteratureEnrichmentError(f"a citing table is invalid: {exc}") from exc
    table_pmids = table_citations(citing_rows)
    # A `pmid=NONE` row cites by DOI alone (RM336): a citation, so the module is not citation-free, and
    # one no PubMed lookup below can reach, so it is counted apart and said in the records.
    cited_by_doi_only = sum(
        1
        for row in [*studies, *(r for rows in citing_rows.values() for r in rows)]
        if row.pmid and not extract_pmids(row.pmid)
    )
    if not studies and not table_pmids and not cited_by_doi_only:
        raise LiteratureEnrichmentError(
            f"no citations in {spec_dir} — the literature pass checks the citations a module makes, "
            f"and this one has neither studies.csv rows nor a `pmid` on any binning or "
            f"pharm_variants row."
        )

    existing: dict[tuple, LiteratureRow] = {}
    if output_path.exists():
        rows, errors, _ = load_csv_rows(output_path, LiteratureRow, "literature.csv")
        if errors:
            raise LiteratureEnrichmentError(f"existing literature.csv is invalid: {errors[0]}")
        for row in rows:
            existing[merge_key(row)] = row

    citations = _citations(studies, table_pmids)
    authored_total = sum(
        1 for rows in citations.values() for s in rows if s.provenance_quote or s.provenance_regex
    )
    # A `pmid=NONE` row cites by DOI alone (RM336), so `_citations` has no key for it and its quote is
    # in none of the counts above. Counted apart so the record never says "no quote on any citation"
    # about a module carrying one. Online, the DOI is the key it is asked about instead (RM349).
    without_pmid = sum(
        1 for s in studies if not extract_pmids(s.pmid) and (s.provenance_quote or s.provenance_regex)
    )
    doi_cited = _doi_only_citations(studies, citing_rows)

    if offline:
        logger.warning(
            "Literature enrichment skipped: --offline and there is no offline PubMed/Europe PMC "
            "snapshot. Any existing literature.csv is kept as the pin; compiles stay reproducible."
        )
        out = sorted(existing.values(), key=lambda r: int(r.pmid))
        if write and existing:
            _write_literature_csv(out, output_path)
        return _attest(
            LiteratureResult(
                rows=out,
                mode=mode,
                skipped_offline=True,
                sources=sorted({r.source for r in out if r.source}),
                cited=sorted(citations, key=int),
                quotes_authored=authored_total,
                quotes_without_pmid=without_pmid,
                citations_without_pmid=cited_by_doi_only,
                fulltext_requested=check_fulltext,
            ),
            spec_dir,
            write=write,
            check_fulltext=check_fulltext,
            check_doi=check_doi,
        )

    # **A pin whose quote count no longer matches `studies.csv` is re-fetched (RM277).** The row's own
    # input changed, so it describes a different set of quotes: its `quotes_found` cannot be paired
    # with the new count (`_tally_quotes` reads exactly that mismatch as "unexamined"), and rewriting
    # the count alone would attach the old verdict to the new quotes. Re-deriving the row is the
    # corrected derivation, and it is what makes the compiler's `quote_counter_stale` remedy, "re-run
    # the literature pass", true. Offline has returned above and keeps the pin.
    stale_pins = {
        row.pmid
        for row in existing.values()
        if row.pmid in citations
        and (row.quotes_authored or 0)
        != sum(1 for s in citations[row.pmid] if s.provenance_quote or s.provenance_regex)
    }
    if stale_pins:
        logger.info(
            "Literature: re-fetching %d pinned citation(s) whose quote count changed in studies.csv: %s",
            len(stale_pins),
            ", ".join(sorted(stale_pins, key=int)),
        )
    # Against the rows rather than the dict's keys, which are merge-key tuples and not bare PMIDs.
    have = {row.pmid for row in existing.values()} - stale_pins
    wanted = [pmid for pmid in citations if pmid not in have]
    # **Pinned rows whose fulltext question never got an answer are asked that one question again
    # (RM258).** Everything else on the row stays merged, since answered is per field
    # (`@answered-is-not-absent`): existence, identifiers and the licence were answered, and dropping
    # them to get a retry is the refused repair. Only rows with a quote to check, and only when this run
    # checks fulltext at all.
    reask = (
        [
            row
            for row in sorted(existing.values(), key=lambda r: int(r.pmid))
            if row.pmid in have
            and row.pmid in citations
            and _has_quote(citations[row.pmid])
            and _needs_reask(row)
        ]
        if check_fulltext
        else []
    )
    fetched_at = now_utc_iso()
    result = LiteratureResult(
        rows=[row for row in existing.values() if row.pmid not in stale_pins],
        mode=mode,
        cited=sorted(citations, key=int),
        quotes_authored=authored_total,
        # Set here, before the coverage line is logged, so the log carries the DOI-only clause (RM349).
        quotes_without_pmid=without_pmid,
        citations_without_pmid=cited_by_doi_only,
        fulltext_requested=check_fulltext,
    )

    # **Titles are needed for MERGED rows too, which is the correction S54's reporter filed against
    # their own report.** A pinned row is not in `wanted`, so the fetch loop never sees it — and on
    # the four modules that motivated the title check, every row is pinned, so the check would not
    # have fired on a single one of the 3,668 quotes it was written for. `esummary` batches, so
    # adding the already-pinned PMIDs that carry a quote costs no extra round trip in the common
    # case and nothing at all when there are none.
    titled = [pmid for pmid in citations if pmid not in wanted and _has_quote(citations[pmid])]
    # The citations by DOI alone (RM349): Crossref for existence, Europe PMC for the text. Nothing about
    # them is pinned (see `DoiCitation`), so an online run asks again whenever it has a question.
    ask_dois = bool(doi_cited) and (
        check_doi or (check_fulltext and any(_has_quote(rows) for rows in doi_cited.values()))
    )
    result.doi_citations = [
        DoiCitation(doi=key, quotes_authored=_quote_count(doi_cited[key])) for key in sorted(doi_cited)
    ]
    if wanted or titled or reask or ask_dois:
        owned_eutils = eutils is None
        client = eutils or EutilsClient()
        summaries: dict = {}
        try:
            if wanted or titled or reask:
                summaries = client.esummary("pubmed", wanted + titled)
        except EutilsError as exc:
            raise LiteratureUnavailable(f"PubMed could not be reached: {exc}") from exc
        finally:
            if owned_eutils:
                client.close()

        owned_epmc = europepmc is None
        epmc = europepmc or EuropePmcClient()
        owned_crossref = crossref is None
        crossref = crossref or CrossrefClient()
        # A caller's client may carry an earlier run's 410; only one met from here on is this run's (RM446).
        crossref_mark = retired_mark(crossref)
        # On the E-utilities gate, which outlives the esummary client above: one NCBI budget.
        owned_bioc = bioc is None
        bioc = bioc or PmcBiocClient(gate=client.gate)
        try:
            # Guarded: a run whose only questions are about DOI-only citations has no PMID to look up.
            indexed = epmc.lookup(wanted) if wanted else {}
            for pmid in wanted:
                summary = summaries.get(pmid, {})
                exists = not is_missing(summary)
                ids = _identifiers(summary)
                epmc_record = indexed.get(pmid, {})
                doi = ids.get("doi") or epmc_record.get("doi")
                pmcid = ids.get("pmcid") or epmc_record.get("pmcid")
                is_open = epmc_record.get("is_open_access") if epmc_record else None
                # Verbatim from Europe PMC; rights derived at read time so a mapping correction
                # reaches rows already written (`licensing.article_terms`).
                license_name = epmc_record.get("license") if epmc_record else None
                terms = article_terms(license_name)

                # Nothing is tallied in this loop, and that is the rule rather than a preference:
                # every count this pass reports is taken afterwards over `subject_rows`, so the
                # `strict` gates, the CLI report and the attestation cannot end up reading three
                # different sets. A citation already pinned in `literature.csv` is not fetched here,
                # so a count made in this loop is a count of *this run's requests* — which made the
                # existence gate, the identifier cross-check and the record disagree with each other
                # depending on the order two runs happened in.

                # Crossref checks the **authored** DOI in preference to the derived one. Checking the
                # registry's own DOI would be circular — it exists by construction, since the registry
                # just handed it over. The authored cell is the one nobody has verified, and it is the
                # only one that exists at all for a citation PubMed does not index (a preprint, book or
                # dataset), which is the case this whole client is here for.
                target_doi = _doi_to_check(citations[pmid], doi)
                doi_exists: bool | None = None
                if check_doi and target_doi:
                    doi_exists = crossref.exists(target_doi)

                quotes = [s for s in citations[pmid] if s.provenance_quote or s.provenance_regex]
                found: int | None = None
                quote_source: str | None = None
                fulltext_status: str | None = None
                if check_fulltext and quotes:
                    # PMC serves an author manuscript for text mining where Europe PMC calls it
                    # closed or fails on it (RM257, S110). Any PMCID qualifies: an article outside
                    # the service's set costs one paced request answering "no copy". How the question
                    # ended is recorded, so an outage is asked again rather than pinned (RM258).
                    text, fulltext_status = _ask_fulltext(epmc, bioc, pmcid, is_open)
                    if text is not None:
                        result.fulltext_checked.append(pmid)
                        quote_source = "fulltext"
                    else:
                        # Fall back to the abstract, which Europe PMC serves for paywalled records
                        # too. A HIT here is as conclusive as one in the body; a MISS is not, because
                        # the body was never searched — which is what `quote_source` records, and why
                        # this row still counts as unchecked below.
                        text = epmc_record.get("abstract")
                        if text:
                            result.abstract_checked.append(pmid)
                            quote_source = "abstract"
                    if text:
                        found = _quotes_found(quotes, text, regex_timeout=regex_timeout)
                # Costs no request: the title arrived in the same `esummary` response that answered
                # existence. Outside the `check_fulltext` guard deliberately — this compares the quote
                # against metadata already held, so it answers even for a paywalled article whose
                # fulltext was never retrievable, which is where the defect hides.
                if quotes and _quote_is_the_title(quotes, summary):
                    result.titles_as_quotes.append(pmid)
                # What the retrieved text settled is tallied once, after the loop, over every row —
                # see `_tally_quotes`. The per-row facts it reads (`quotes_authored`, `quotes_found`,
                # `quote_source`) are written right here, so the tally is the same arithmetic applied
                # to merged rows as well as fresh ones.
                result.rows.append(
                    LiteratureRow(
                        pmid=pmid,
                        doi=doi,
                        pmcid=pmcid,
                        exists=exists,
                        is_open_access=is_open,
                        license=license_name,
                        share_alike=terms.share_alike,
                        commercial_use=terms.commercial_use,
                        redistribution=terms.redistribution,
                        quotes_authored=len(quotes),
                        quotes_found=found,
                        quote_source=quote_source,
                        fulltext_status=fulltext_status,
                        doi_exists=doi_exists,
                        # Which DOI that verdict is about. Without it the pin cannot say, and a
                        # re-run pairs the stored answer with whatever the author writes next.
                        doi_checked=target_doi if doi_exists is not None else None,
                        # PubMed is the row's source: it decides existence and supplies the
                        # identifiers. Europe PMC contributes `is_open_access` and the fulltext, but
                        # it cannot originate a row (it silently omits ids it does not know), so it
                        # is not a `source` in the sense the other sidecars use the word.
                        source="pubmed",
                        status="resolved" if exists else "not_found",
                        fetched_at=fetched_at,
                    )
                )
            # The pinned rows whose fulltext question is still open (RM258): that one question, and a
            # row rewritten only where the answer changed it. An answer overwrites the quote counts,
            # `quote_source` and the status; a request that fails again leaves the row as it was,
            # marked `unreachable`, so the next run asks again.
            for row in reask:
                text, status = _ask_fulltext(epmc, bioc, row.pmcid, row.is_open_access)
                update: dict[str, object] = {"fulltext_status": status}
                if text is not None:
                    quotes = [s for s in citations[row.pmid] if s.provenance_quote or s.provenance_regex]
                    update |= {
                        "quote_source": "fulltext",
                        "quotes_found": _quotes_found(quotes, text, regex_timeout=regex_timeout),
                    }
                    result.fulltext_checked.append(row.pmid)
                if all(getattr(row, name) == value for name, value in update.items()):
                    continue
                healed = row.model_copy(update={**update, "fetched_at": fetched_at})
                result.rows = [healed if r.pmid == row.pmid else r for r in result.rows]
            # The pinned rows: no row is written for them (the sidecar is authoritative) and the
            # fulltext is never fetched, but the title comparison needs only the summary — so it
            # answers here where `quotes_found` cannot.
            for pmid in titled:
                if _quote_is_the_title(
                    [s for s in citations[pmid] if s.provenance_quote or s.provenance_regex],
                    summaries.get(pmid, {}),
                ):
                    result.titles_as_quotes.append(pmid)
            if ask_dois:
                result.doi_citations = [
                    _check_doi_citation(
                        key,
                        doi_cited[key],
                        crossref=crossref,
                        epmc=epmc,
                        bioc=bioc,
                        check_doi=check_doi,
                        check_fulltext=check_fulltext,
                        regex_timeout=regex_timeout,
                    )
                    for key in sorted(doi_cited)
                ]
            result.crossref_unanswered = withheld_reason(crossref, since=crossref_mark)
        finally:
            if owned_epmc:
                epmc.close()
            if owned_crossref:
                crossref.close()
            if owned_bioc:
                bioc.close()

    result.titles_as_quotes = sorted(set(result.titles_as_quotes), key=int)
    result.rows.sort(key=lambda r: int(r.pmid))
    result.fulltext_checked = sorted(set(result.fulltext_checked), key=int)
    result.sources = sorted({r.source for r in result.rows if r.source})
    # Every tally runs here, once, over the sorted subject rows — so each list is ordered by PMID
    # rather than by whatever order the fetch took, and the gates below refuse on exactly what the
    # attestation records.
    _tally_existence(result, citations)
    # A DOI no registration agency knows is a cited DOI that does not resolve, whichever kind of row
    # cites it (RM349), so the `strict` gate and the report below name it with the others.
    # Both sides by `doi_key` (RM363): the PubMed path records the registry's DOI as the registry spells it.
    missing = {doi_key(doi) for doi in result.doi_missing}
    result.doi_missing += [
        c.doi for c in result.doi_citations if c.exists is False and doi_key(c.doi) not in missing
    ]
    _tally_quotes(result, citations)
    _compare_identifiers(result, citations)
    logger.info("Literature: %s", result.coverage)

    if mode == "strict" and result.missing:
        raise LiteratureEnrichmentError(
            f"strict literature enrichment: PubMed has no record of {len(result.missing)} cited "
            f"PMID(s): {result.missing}. Either the identifier is a typo or the article was pulled "
            f"from the index; both mean the row's grounding evidence does not resolve. Fix the "
            f"citation, or enrich with mode='best_effort' to record it as a warning."
        )
    if mode == "strict" and result.doi_missing:
        raise LiteratureEnrichmentError(
            f"strict literature enrichment: Crossref has no record of {len(result.doi_missing)} "
            f"cited DOI(s): {result.doi_missing}. A DOI that does not resolve is a citation that "
            f"cannot be followed."
        )
    if mode == "strict" and result.doi_conflicts:
        raise LiteratureEnrichmentError(
            f"strict literature enrichment: {len(result.doi_conflicts)} authored DOI(s) disagree with "
            f"the registry: {[str(c) for c in result.doi_conflicts]}. One of the two identifiers "
            f"points at the wrong paper."
        )
    if mode == "strict" and result.pmcid_conflicts:
        raise LiteratureEnrichmentError(
            f"strict literature enrichment: {len(result.pmcid_conflicts)} authored PMC id(s) disagree "
            f"with PubMed's for the same record: {[str(c) for c in result.pmcid_conflicts]}. One of "
            f"the two identifiers points at the wrong paper."
        )
    if write:
        _write_literature_csv(result.rows, output_path)
    return _attest(result, spec_dir, write=write, check_fulltext=check_fulltext, check_doi=check_doi)

bibliographic

bibliographic(summary: dict) -> dict[str, str | None]

Pull the fields that say which paper this is out of an esummary record.

Public, unlike _identifiers, because two tiers need it and the alternative is a consumer re-implementing a parse of a payload we already hold — the RM41 lesson. lookup.CitationHint reads it so "does this PMID exist" can become "does this PMID name the paper you meant": existence alone cannot catch a fabricated citation, because PMIDs are densely allocated and an invented number is usually a real record for a different article.

Every value is None when the field is absent rather than empty-string, so a caller can tell "PubMed did not say" from "PubMed said nothing is there" — the house tri-state, applied to metadata. year is the leading four digits of pubdate (which is free-form: 2017 Nov 20, 2017, 2017 Nov-Dec), and nothing is invented when it does not start with a year.

Source code in enricher/src/just_dna_enricher/literature.py
def bibliographic(summary: dict) -> dict[str, str | None]:
    """Pull the fields that say *which paper this is* out of an esummary record.

    Public, unlike `_identifiers`, because two tiers need it and the alternative is a consumer
    re-implementing a parse of a payload we already hold — the RM41 lesson. `lookup.CitationHint`
    reads it so "does this PMID exist" can become "does this PMID name the paper you meant":
    existence alone cannot catch a fabricated citation, because PMIDs are densely allocated and an
    invented number is usually a real record for a different article.

    Every value is `None` when the field is absent rather than empty-string, so a caller can tell
    "PubMed did not say" from "PubMed said nothing is there" — the house tri-state, applied to
    metadata. `year` is the leading four digits of `pubdate` (which is free-form: `2017 Nov 20`,
    `2017`, `2017 Nov-Dec`), and nothing is invented when it does not start with a year.
    """
    out: dict[str, str | None] = {"title": None, "journal": None, "year": None, "first_author": None}
    for key, target in (
        ("title", "title"),
        ("fulljournalname", "journal"),
        ("sortfirstauthor", "first_author"),
    ):
        value = summary.get(key)
        if isinstance(value, str) and value.strip():
            out[target] = value.strip()
    pubdate = summary.get("pubdate")
    if isinstance(pubdate, str):
        leading = pubdate.strip()[:4]
        if leading.isdigit():
            out["year"] = leading
    return out