just_dna_enricher.literature¶
just_dna_enricher.literature ¶
enrich-literature — the fourth pass: citations in, literature.csv out.
Closes the reachable part of the compiler's "does the cited study support the row?" blind spot. Three questions, in increasing ambition and decreasing coverage:
- Does the citation exist? PubMed
esummary, batched. A nonexistent PMID comes back as a record carrying anerrorkey, so this is a clean yes/no. - Do the identifiers agree? The DOI and PMCID arrive free in the same response. An absent
authored DOI is filled here in the sidecar; an authored DOI that contradicts the registry is a
finding. Neither is ever written back into
studies.csv— the enricher does not edit authored files, becausecontent_signatureis defined as reference-independent, and a network fetch that could change it would make that documented property false. - Does the quoted passage appear in the article? Only for the open-access subset, and honestly labelled as such.
All three are attested (RM45): the pass writes citation_existence, citation_identifier and
provenance_quote into verification.json on its way out, including on the offline return, so a
module can say which of the three was put and over how many citations. Until 0.6 the answers reached
a log line and the result object and died there, which left a module whose citations had been checked
indistinguishable from one where the command was never run.
Coverage is partial by nature, and saying so is part of the check. A pass that reported "0 quotes
found" for an article it could not read would be describing its own reach as if it were a property of
the module. So the result separates checked and not found from never retrievable — quotes_found
is null rather than zero when no fulltext could be read — and, since dogfooding caught the
conflation, also from nothing to check: a citation with no authored quote asked no question and is
not counted against coverage. The same distinctions are carried into manifest.literature.
Corrections to the drafted plan, made under probing rather than assumed:
- The PMC ID converter is not used, though the plan budgeted for it.
esummaryalready returns bothdoiandpmcinarticleids, and Europe PMC'ssearchreturnsdoi/pmcidtoo — so the converter is a third request for data already in hand. Worse, it answers a different question: for PMID 12345678 (a real, indexed PubMed record) it repliesstatus: error, "Identifier not found in PMC", which is about PMC membership, not existence. Wiring it in as an existence check would report every paywalled article as a broken citation. - Europe PMC is not an existence oracle either. Asked for three ids where one does not exist, it returns two results and simply omits the third — no error, no marker. Absence there is indistinguishable from "not indexed", so PubMed decides existence and Europe PMC only decides retrievability.
On running provenance_regex here. The charter requires a linear-time / ReDoS-safe engine for
pattern matching, written when the match was specified as consumer-side — arbitrary patterns meeting
arbitrary documents. Here the pattern comes from the module being enriched and the document from a
public archive, on the author's own machine, so the threat model is a curator writing a slow pattern by
accident rather than an attacker. That is worth a bound rather than a compiled dependency — so the
match runs under a wall-clock timeout, and a timeout is recorded as not checked.
That bound is enforced with a child process, not a thread, and the reason is worth stating because
the thread version looks correct: re cannot be interrupted, threads cannot be killed, and the
interpreter joins pool threads at exit — so a thread-based timeout returns on schedule and then hangs
the process on the way out. See regex_matches.
LiteratureEnrichmentError ¶
Bases: RuntimeError
Raised in strict mode when a citation does not resolve, or contradicts its own identifiers.
LiteratureUnavailable ¶
Bases: LiteratureEnrichmentError
PubMed could not be reached, so no citation question was put at all (RM101).
A subclass rather than a second exception, so every existing except LiteratureEnrichmentError
still catches it (P3 — additive within a major). It separates the source was asked and never
answered from a citation that genuinely does not resolve under strict. Only this one means
nothing was established either way.
Before RM101 an EutilsError travelled straight out of enrich_literature through a
try/finally with no except, so a caller's except LiteratureEnrichmentError was silent for
exactly the failure it was written for.
Scoped to the eutils leg on purpose. EuropePmcClient.fulltext and CrossrefClient.exists
already answer a transport failure with None rather than an exception — the tri-state withhold
this codebase uses for "could not be retrieved" — and turning either into an error here would
convert a withheld answer into a failed run.
FulltextUnavailable ¶
Bases: LiteratureUnavailable
A fulltext request never got an answer: a 5xx or a transport failure, after retries (RM258).
Raised by EuropePmcClient.fetch_fulltext and PmcBiocClient.fetch_fulltext, the three-way form
of the question. It is the unreachable arm, kept apart from a None return, which is absent
(the service holds no copy). The two used to be one None, and the pass pinned an outage as an
abstract-only row that merge-not-clobber never asked about again. A subclass, so an existing
except LiteratureUnavailable still catches it.
LiteratureRetired ¶
Bases: LiteratureUnavailable, EndpointRetired
Europe PMC answered HTTP 410: it retired the endpoint asked (RM354).
A fact about the source, not an outage, and not retried; a LiteratureUnavailable so every
existing handler still catches it. Raised by EuropePmcClient.lookup only: the fulltext routes
raise FulltextRetired, which their callers already catch as FulltextUnavailable.
FulltextRetired ¶
Bases: FulltextUnavailable, EndpointRetired
A fulltext service answered HTTP 410 for the route asked (RM361).
Before RM361 every status below 500 read as absent, so a retired route wrote a permanent
fulltext_status="absent" about every article it was asked for. A FulltextUnavailable, so the
pass's handler still catches it; the pass records retired and asks again on the next run.
DoiConflict
dataclass
¶
An authored DOI that disagrees with the one the registry reports for the same PMID.
PmcidConflict
dataclass
¶
An authored PMC id that disagrees with the one PubMed reports for the same PMID (RM50).
The DoiConflict shape, for the other cross-registry identifier. It costs no request — the PMC id
is already in the esummary articleids block that answered existence — and it catches the case
the schema guard cannot see: a cell like 21551363 (PMC3110567) carries a real PubMed id, so
nothing refuses it, while the two halves name different articles.
DoiCitation
dataclass
¶
DoiCitation(
doi: str,
exists: bool | None = None,
crossref_asked: bool = False,
title: str | None = None,
agency: str | None = None,
quotes_authored: int = 0,
quotes_found: int | None = None,
quote_source: str | None = None,
fulltext_status: str | None = None,
europepmc_id: str | None = None,
license: str | None = None,
)
What one run learned about a work cited by DOI alone, through pmid=NONE (RM349).
One per distinct DOI (spelling.doi_key), however many rows cite it and in whatever spelling.
Not written to literature.csv, whose rows are keyed on a PubMed id and feed a published
parquet the compiler reads by PMID; so it is not pinned either, and every online run asks again.
What it records goes into verification.json, which is the durable half.
existsis Crossref's tri-state (None: not asked under--no-doi, or never answered —crossref_askedtells the two apart), andtitlethe work Crossref names for it (@existence-not-identity). No citing table carries a stated title to compare it with, so it is recorded and reported, never compared.agencyis who registered the DOI when Crossref did not (CrossrefWork.agency).- The quote half has
LiteratureRow's columns and meanings:fulltext_statusisretrieved,absentorunreachable(Nonewhen no quote asked the question),quote_sourcesays which text was searched, andquotes_checked()is the same rule (settled_quotes).
LiteratureResult
dataclass
¶
LiteratureResult(
rows: list[LiteratureRow],
missing: list[str] = list(),
crossref_unanswered: str = UNREACHABLE,
doi_conflicts: list[DoiConflict] = list(),
pmcid_conflicts: list[PmcidConflict] = list(),
cited: list[str] = list(),
existence_checked: int = 0,
unresolved_citations: int = 0,
doi_verdicts_stale: int = 0,
doi_never_checked: int = 0,
noncommercial_quoted: list[str] = list(),
titles_as_quotes: list[str] = list(),
fulltext_requested: bool = True,
quotes_authored: int = 0,
quotes_found: int = 0,
quotes_checked: int = 0,
quotes_unchecked: int = 0,
quotes_unexamined: int = 0,
quotes_without_pmid: int = 0,
citations_without_pmid: int = 0,
doi_citations: list[DoiCitation] = list(),
fulltext_checked: list[str] = list(),
abstract_checked: list[str] = list(),
doi_missing: list[str] = list(),
identifiers_authored: int = 0,
identifiers_compared: int = 0,
identifiers_conflicting: int = 0,
identifiers_unmatched: int = 0,
identifiers_foreign: int = 0,
sources: list[str] = list(),
mode: str = "best_effort",
skipped_offline: bool = False,
)
What the pass found, and — through subject_rows — what it found it about.
One subject set, read by the report, by the strict gates and by the attestation. The set is
the citations the module makes now, answered by the rows literature.csv holds now, and it is
the single rule the whole pass turns on. Three separate defects came from parts of this file
reading three different sets:
- the
strictgates read lists appended inside the fetch loop, so a module enriched once with--best-effortand then with--strictwas blessed on a citation PubMed has no record of — the refusal depended on the order the two runs happened in, and the attestation written by the same run saidfindings=1while the gate beside it said nothing was wrong; - the identifier cross-check ran inside that loop too, so an existing
literature.csvhid every DOI/PMC id disagreement; - the counts read
rows, which is everything the sidecar carries and never shrinks, so a citation deleted fromstudies.csvwent on being counted and no authored edit could clear the finding it produced.
Merge-not-clobber is what makes the distinction real: the sidecar keeps rows for citations the module has since dropped, and those rows are still written back (deleting a curator's row is not this pass's call) — they are simply not what any of this run's questions were about.
subject_rows
property
¶
The rows this run's questions were about: a row for a citation the module still makes.
Not rows, which is the whole sidecar — see the class docstring. Not this run's fetch list
either: literature.csv is the pin, so a row merged from an earlier run carries a verdict
that still stands, and counting only what this run looked up would let a no-op re-run replace
a true attestation with subjects=0, which reads as "the check ran and had nothing in scope".
coverage
property
¶
The sentence this pass exists to be able to say honestly.
The denominator is what had something to check — a citation with no authored quote was
not skipped for lack of a fulltext, it simply asked no question. Counting it as unretrievable
was a real bug found by running this against reference_examples/pathogenic_clinvar/, whose
single citation is open access and carries no quote: the old wording claimed its fulltext
could not be retrieved, which was the opposite of true.
Counted in quotes, off the tally, and not in citations off the run's own fetch lists. The
earlier wording read fulltext_checked, which only ever holds PMIDs this run retrieved, so
a no-op re-run over a pinned open-access citation printed "0 of 1 … 1 with nothing
retrievable" one line above "1/1 found" — the same false retrievability claim the paragraph
above records as a bug, arrived at from the other direction. Quotes also partition cleanly
where citations do not: one citation can carry a settled quote and a never-examined one at
once, so a citation-granular sentence has to put it in one bucket and be wrong about the
other.
EuropePmcClient
dataclass
¶
EuropePmcClient(
base_url: str = DEFAULT_EUROPEPMC_BASE,
repo_url: str = DEFAULT_EUROPEPMC_REPO_URL,
batch_size: int = 25,
min_request_interval: float = 0.5,
timeout: float = 30.0,
gate: PacingGate | None = None,
_client: Client | None = None,
)
Open-access lookup and fulltext retrieval. Paced like every other client in this tier.
lookup ¶
pmid -> {pmcid, doi, is_open_access, license, abstract} for the ids Europe PMC knows.
Ids it does not know are simply absent from the result, with no error marker — which is why this is not an existence check. A caller must read a miss here as "not retrievable", never as "does not exist".
The abstract comes back for paywalled records too, in this same response, and that is worth more than it looks: probed across a mix of non-open-access papers, four of five carried one (only a 1994 non-research document did not). It is the difference between checking a quote for the open-access minority and checking it for nearly everything.
So does the article's license (RM46), which is why per-article terms cost no extra
request. Probed 2026-08-13 over 100 records: the values are lowercase CC spellings — cc by
(64), cc by-nc (28), cc by-nc-nd (8). It is independent of isOpenAccess and must not
be derived from it: PMID 28546431 comes back isOpenAccess: N with license: cc by, since
the flag describes Europe PMC's OA subset while the licence describes the article. Stored
verbatim; licensing.article_terms maps it to rights at read time.
Source code in enricher/src/just_dna_enricher/literature.py
pmids_for_doi ¶
The PubMed ids Europe PMC holds for one DOI, sorted, from its MEDLINE source (RM263).
Europe PMC's search covers all of PubMed, where NCBI's id converter answers only for articles
in PMC, so this reaches the paywalled case the converter misses (10.1038/ng826 is
11788828, Enattah 2002, not in PMC). An empty list is Europe PMC's answer that it holds
no MEDLINE record for the DOI, never a statement about PubMed. More than one id is returned as
it came, so a caller can report the multiplicity rather than pick. Failures translate to
LiteratureUnavailable on every leg, as lookup does (@client-exception-contract).
Source code in enricher/src/just_dna_enricher/literature.py
fetch_fulltext ¶
Whitespace-normalized article text; None when Europe PMC holds no copy; raises
FulltextUnavailable when the request never got an answer (RM258, RM361).
A 404 is an answer: an embargoed or author-manuscript-only record answers it, which is
absent. A 410 raises FulltextRetired. A 429, any other 4xx, a 5xx, a transport failure
and a 200 whose body does not parse as XML are unreachable, and the caller asks again on a
later run.
Source code in enricher/src/just_dna_enricher/literature.py
fulltext ¶
fetch_fulltext with its two negative answers folded into one None, as before 0.8.
Kept for callers that already treat None as "no text either way" (the authoring plugin's
fetch_fulltext tool relies on it for Europe PMC's 500s). A caller that needs to tell an
outage from an absent copy uses fetch_fulltext.
Source code in enricher/src/just_dna_enricher/literature.py
record_for_doi ¶
Europe PMC's record for one DOI, from any of its sources, or None when it holds none (RM349).
The route to a pmid=NONE citation's text. Europe PMC indexes bioRxiv and medRxiv preprints
(source PPR) by DOI, so the same search that pmids_for_doi restricts to MEDLINE reaches
them here unrestricted. Only a record whose own doi is the one asked about is taken: the
search is a query, and a query's first hit need not be the work (@existence-not-identity).
Returns {id, source, pmcid, is_open_access, license, abstract, title, preprint_id}.
preprint_id is the PPR id when the record says Europe PMC holds the preprint's full text
(its fullTextIdList), and None otherwise, so a caller asks the repository only where there
is something to fetch: probed 2026-11-04, the repository's answer for a preprint with no text
(PPR1146237) is a 500, the same status as an outage. Failures translate to
LiteratureUnavailable on every leg, as pmids_for_doi does (@client-exception-contract).
Source code in enricher/src/just_dna_enricher/literature.py
fetch_preprint_fulltext ¶
A preprint's JATS body as normalized text; None when the repository lists no XML file for
it; raises FulltextUnavailable when a request never got an answer (RM349, the RM258 shape).
Two requests: the record's file list (type=METADATA), then the JATS file it names
(type=FILE). The file name (EMS…nxml) is never guessed, and the PDF the list also names is
never used: its link is session-bound (it answered 403 "expired" on 2026-11-04). Ask only for a
preprint whose search record lists a full text (record_for_doi's preprint_id), because a
5xx here then means an outage rather than the repository's "no such copy", which shares it.
A 200 that is not the list is the fourth leg and withholds too (@client-exception-contract).
Source code in enricher/src/just_dna_enricher/literature.py
CrossrefWork
dataclass
¶
CrossrefWork(
exists: bool | None,
title: str | None = None,
journal: str | None = None,
year: str | None = None,
first_author: str | None = None,
agency: str | None = None,
)
What Crossref said about one DOI: whether it exists, and which work it names (RM262, S113).
exists is the tri-state CrossrefClient.exists has always answered. The four bibliographic
fields are the same four bibliographic pulls out of a PubMed record, so a DOI-only citation can
be checked for identity and not merely existence. Each is None when the record does not carry
it, never "", and all four are None whenever exists is not True.
CrossrefClient
dataclass
¶
CrossrefClient(
base_url: str = DEFAULT_CROSSREF_BASE,
min_request_interval: float = 0.1,
timeout: float = 30.0,
contact_email: str | None = None,
gate: PacingGate | None = None,
_client: Client | None = None,
)
Bases: Withholding
DOI existence, for the citations PubMed does not index.
PubMed answers "does this article exist" for anything it indexes — including paywalled work;
a paywall governs the fulltext, not the record. What it cannot answer for is everything outside
its scope: preprints, books, theses, datasets, standards. Those have DOIs and no PMID, and Crossref
is the registry that mints and resolves them (a probed bioRxiv preprint, 10.1101/2024.06.17.599351,
returns type: posted-content; a fabricated DOI returns a clean 404).
This is also what makes the 1.0 doi-first flip low-risk: once pmid becomes optional and a
citation may carry only a DOI, existence checking has to work without PubMed, and it already does.
Crossref asks callers to identify themselves in the User-Agent and gives the "polite pool" in
return; the contact address is sent only when one is configured, for the same reason eutils omits
it rather than inventing one.
exists ¶
True/False, or None when Crossref could not be asked. work(doi).exists.
work ¶
Existence and identity, from the one request exists always made (RM262, S113).
exists used to read the status and throw the body away, so a DOI got an existence answer and
never the title that says which paper it is — the answer existence is not identity exists to
refuse. The body was already in hand; parsing it costs no request.
exists=None rather than False on a transport failure or an unexpected status: "we could not
check" and "this DOI does not exist" are different claims, and only the second is a finding
against the module. The translation stays here and the retrying stays in _request above —
reraise=True means the last failure arrives back at this except after the attempts are
spent. A 200 whose body is not a Crossref record withholds too (@client-exception-contract,
the fourth leg): a maintenance page or a CDN interstitial is not Crossref saying the DOI exists,
and exists used to answer True for one because it read only the status.
Source code in enricher/src/just_dna_enricher/literature.py
PmcIdRecord
dataclass
¶
One answer from the PMC id converter. in_pmc=False is PMC saying it has no such record;
an id the request never reached is absent from the result entirely, never one of these.
PmcIdConverterClient
dataclass
¶
PmcIdConverterClient(
base_url: str = DEFAULT_PMC_IDCONV_URL,
batch_size: int = 200,
min_request_interval: float = 0.5,
timeout: float = 30.0,
contact_email: str | None = None,
gate: PacingGate | None = None,
_client: Client | None = None,
)
Bases: Withholding
PMCID → PMID, for a curator holding the wrong half of the pair (RM50).
It reports; it never fills. The PMID it returns is handed back as an advisory the author types
themselves, because filling StudyRow.pmid from NCBI would make literature.exists compare NCBI
against NCBI — the hints.REDUNDANCY_BEARING rule, already argued for doi (Crossref is asked
about the authored DOI, since a derived one exists by construction).
Why this exists although the pass docstring says the converter is unused. That statement is
true of the direction the pass needs — PMID → PMCID arrives free in the esummary articleids
block, so calling the converter for it would be a third request for data already in hand, and it
answers a different question (asked about a paywalled PMID it replies "Identifier not found in
PMC", which is about PMC membership rather than existence). None of that says anything about
PMCID → PMID, which is the direction the converter is actually for and the one a curator has
no other route to.
Probed 2026-08-13: the www.ncbi.nlm.nih.gov/pmc/utils/idconv/v1.0/ path 301-redirects to the
address below, an absent id comes back as a record with status: "error" and
errmsg: "Identifier not found in PMC", and pmid arrives as a JSON number.
resolve ¶
PMCID -> PmcIdRecord for every id the converter answered about.
Four outcomes, and none of them is spelled the same way as another. An id it resolved
carries a pmid; an id it knows with no PubMed id carries in_pmc=True, pmid=None (a real
answer — some PMC records genuinely have none); an id it has no record of carries
in_pmc=False with the service's own errmsg; and an id the request never reached is simply
absent from the result. A caller must not read a missing key as "does not exist" — that
conflation is exactly what S20 was about, one service over.
Source code in enricher/src/just_dna_enricher/literature.py
PmcBiocClient
dataclass
¶
PmcBiocClient(
base_url: str = DEFAULT_PMC_BIOC_URL,
min_request_interval: float = 0.5,
timeout: float = 60.0,
gate: PacingGate | None = None,
_client: Client | None = None,
)
Bases: Withholding
PMC's BioC text-mining service: the body text of an article Europe PMC will not serve (RM257, S110).
The records this reaches are the ones the Europe PMC rung cannot. An NIH author manuscript is
isOpenAccess: N in Europe PMC, so the pass never asked it for fulltext, and asking anyway
answered HTTP 500 for PMC6463297 on 2026-09-24. PMC serves the same manuscript here for text
mining — 294 passages, both body tables among them as tab-separated table passages, and for a
GWAS paper the tables are where the per-locus rows live. Probed the same day: an article outside
the service's set answers 200 with the body [Error] : No result can be found., which is an
answer (no copy) rather than an outage, so it reads as absent like a 404 does.
Retrieval here says nothing about the article's licence. The service's own infon reads "This
file is available for text mining. It may also be used consistent with the principles of fair
use", which names no licence, so the row keeps Europe PMC's is_open_access and license
untouched (@no-named-licence). It changes how far the quote search reached, which is what
quote_source records, and nothing else.
REF passages are dropped: they are the reference list, most of the document by count (186 of
294 on the probed article), and a quote located in someone else's title is not located in this
paper. An NCBI host, so the pass hands it the E-utilities gate rather than giving it a budget of
its own (@shared-pacing-gate).
fetch_fulltext ¶
Body text with the reference list removed; None when the service holds no copy; raises
FulltextUnavailable when the request never got an answer (RM258).
Only the recorded "no copy" 200 is an answer (RM361). The service's reply for an article
outside its set is a 200 with a plain-text body opening BIOC_NO_RESULT_PREFIX
(assets/pmc_bioc_no_result.txt), so it reads as absent, as a 404 does. Any other 200 that
is not JSON (a maintenance page) is the fourth leg and unreachable; a 410 raises
FulltextRetired; a 429, any other 4xx, a 5xx or a transport failure, after the retries, is
unreachable.
Source code in enricher/src/just_dna_enricher/literature.py
fulltext ¶
fetch_fulltext with its two negative answers folded into one None, as before 0.8.
Source code in enricher/src/just_dna_enricher/literature.py
crossref_bibliographic ¶
The four identity fields out of a Crossref /works/{doi} message, shaped like bibliographic.
The shapes differ from PubMed's and each difference is a place to go wrong. title and
container-title are lists, empty for much posted-content. The year is the leading int of
issued.date-parts[0], Crossref's earliest-publication date. The first author is the entry marked
sequence: "first", which need not be at index 0, and is rendered Family GI so it reads the same
as PubMed's sortfirstauthor; an organisational author has only a name. Title markup (Crossref
passes JATS tags such as <i> through) is kept verbatim: the field is for a person to compare
against the paper they meant, and rewriting it would be a second opinion about the title.
Source code in enricher/src/just_dna_enricher/literature.py
extract_bioc_text ¶
BioC JSON → one normalized string of every non-reference passage, or None if it holds none.
The service wraps its collection in a one-element list; a bare collection is accepted too, since the BioC format itself does not require the wrapper.
Source code in enricher/src/just_dna_enricher/literature.py
extract_text ¶
JATS XML → one normalized string, or None if it is not a JATS article.
itertext() rather than a tag-stripping regex: real JATS carries nested inline markup inside
sentences (<italic>, <xref>, <sup>), and a regex that removes tags without joining their
text would silently glue words together and break matches that should succeed.
A document that parses is not yet an article (RM361): a maintenance page served as XHTML is
well-formed XML, and reading its text as the article's would make every quote a miss against a
fulltext, which settles it. Only a root element named article is JATS.
Source code in enricher/src/just_dna_enricher/literature.py
quote_matches ¶
Literal, whitespace- and case-insensitive containment. No engine, so nothing to bound.
regex_matches ¶
regex_matches(
pattern: str,
fulltext: str,
*,
timeout: float = DEFAULT_REGEX_TIMEOUT,
) -> bool | None
True/False, or None when the match could not be completed in time.
The three-way return is the point: a pattern that runs long has not failed to match, it has failed to be checked, and reporting it as "not found" would send an author to fix a quote that is probably there.
Why a subprocess and not a thread. The obvious implementation — submit to a
ThreadPoolExecutor and take future.result(timeout=...) — does not work, and fails in a way that
looks like it works: re never releases the GIL to a cancellation, threads cannot be killed, and
the interpreter joins non-daemon pool threads at exit. So a runaway pattern returns None on time
and then hangs the process on the way out. Verified by writing it that way first and watching the
test suite stop. A child process is the only bound in the standard library that can actually be
enforced, and it is cheap here: it runs only for a row that has a provenance_regex and a
retrievable fulltext, which is a small subset of a small subset.
Source code in enricher/src/just_dna_enricher/literature.py
enrich_literature ¶
enrich_literature(
spec_dir: Path,
*,
mode: str = "best_effort",
offline: bool = False,
check_fulltext: bool = True,
check_doi: bool = True,
regex_timeout: float = DEFAULT_REGEX_TIMEOUT,
write: bool = True,
eutils: EutilsClient | None = None,
europepmc: EuropePmcClient | None = None,
crossref: CrossrefClient | None = None,
bioc: PmcBiocClient | None = None,
) -> LiteratureResult
Fill literature.csv from the citations a module makes.
studies.csv is one citation site of several (RM47, RM132): MeasureBinRow.pmid grounds the
threshold its row states, and PharmVariantRow.pmid grounds the drug/genotype claim its row
makes — a claim studies.csv cannot ground, because a study row attaches to the whole variant. A
module whose only citations come from those tables is enriched exactly like one with a
studies.csv; reading fewer than all the sites would leave the rest unchecked, which is worse
than the honest gap the columns replaced. The kinds come from the compiler's own derived registry
through load_citing_rows/table_citations, so a kind that gains the column is read here with no
edit to this tier.
Existing rows are authoritative and merged, never clobbered — the same rule enrich() applies to
resolution.csv, with the same consequence: to regenerate after a machinery change you must delete
the file first. That includes the licence columns (RM46): rows written before 0.6 carry no
license, and re-running will not back-fill them, because merge-not-clobber cannot tell an
absent value from a curator's deliberate blank. Delete the sidecar to re-derive.
--offline makes this a no-op with a warning. There is no offline literature snapshot and there
will not be one; once literature.csv is written it is the pin, and later compiles read it
offline and deterministically.
Three checks are attested on the way out (_attest), the offline return included: a run that
did not put a question has to say so, or the manifest cannot tell it from a run that put one and
found nothing.
Source code in enricher/src/just_dna_enricher/literature.py
1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 | |
bibliographic ¶
Pull the fields that say which paper this is out of an esummary record.
Public, unlike _identifiers, because two tiers need it and the alternative is a consumer
re-implementing a parse of a payload we already hold — the RM41 lesson. lookup.CitationHint
reads it so "does this PMID exist" can become "does this PMID name the paper you meant":
existence alone cannot catch a fabricated citation, because PMIDs are densely allocated and an
invented number is usually a real record for a different article.
Every value is None when the field is absent rather than empty-string, so a caller can tell
"PubMed did not say" from "PubMed said nothing is there" — the house tri-state, applied to
metadata. year is the leading four digits of pubdate (which is free-form: 2017 Nov 20,
2017, 2017 Nov-Dec), and nothing is invented when it does not start with a year.