just_dna_enricher.net¶
just_dna_enricher.net ¶
Shared HTTP-politeness primitives for the network tier.
Extracted from gnomad.py when a second and third rate-limited service arrived (NCBI eutils, the PMC
ID converter, Europe PMC, OLS4, HGNC). Nothing here knows about any particular API — it is the pacing,
batching and ordering discipline every client in this package has to obey, in one place so the rule
cannot drift between them.
Two of these look trivial and are not:
PacingGatetakes its clock and its sleep as parameters, so a test can prove a six-second interval is honoured without a suite that really sleeps six seconds per request.dedupepreserves first-occurrence order rather than going through aset, because the order requests are made in decides the order rows are emitted in, and emitted order is part ofartifact.digest(Principle 7).
PacingGate
dataclass
¶
PacingGate(
interval: float,
clock: Callable[[], float] = time.monotonic,
sleeper: Callable[[float], None] = time.sleep,
last: float | None = None,
spent: int = 0,
_lock: Lock = threading.Lock(),
)
Enforce a minimum interval between requests, on an injectable clock.
The clock and the sleep are parameters rather than direct time calls so a test can prove the gate
honours the interval without a suite that really sleeps six seconds per request. Monotonic by
default, so a wall-clock adjustment mid-run cannot collapse the interval to zero.
One gate is safe to share across threads, and it had to become so (S15). LookupClients'
docstring tells callers to hold a client and reuse it — precisely because a fresh one per question
would discard this state — so a server running its blocking work through a thread pool shares one
gate by following our own advice. The unsynchronized version read last, slept, then wrote it, so
two threads could both find the interval elapsed, both skip the sleep, and turn a published 3/s
budget into 6/s. That budget is a courtesy someone else enforces by blocking the operator's IP, so
"single-threaded callers only" was not a contract worth keeping unstated or worth keeping.
The lock covers the bookkeeping, not the sleep. Each caller reserves the next free slot and then
waits for it alone, so N callers get N slots spaced interval apart rather than serializing behind
one lock held across a sleep — same guarantee, and no thread is blocked by another's wait. Behaviour
on a single thread is unchanged.
spent is what the gate admitted, and the one number a host cannot otherwise get (S95). A
proxy metering egress per upstream had to charge by the shape of a request — an upper bound,
because nothing downstream reported the calls actually made. Every egressing client waits on its
gate once per attempt, inside its retry loop (gnomad._post, eutils._request), so one increment
is one upstream attempt: a 429 retried three times counts three, and a snapshot hit that never
reached the gate counts nothing. Monotonic, bumped under the same lock as the slot, and never
reset — a reader that wants a rate takes two readings.
attempt_floor ¶
Bases: stop_base
stop_after_attempt(default), resolved per call so a deployment can raise it (RM42).
Drop-in for the stop_after_attempt(n) it replaces, and deliberately only for a bare one: a
composed policy (stop_after_attempt(3) | stop_after_delay(60)) means both, and raising one term
silently changes something whose author meant the conjunction. None of this tier's policies is
composed today; the rule matters the day one is.
Deliberately no count in this prose. It read "the nine policies" in two places while the tree
carried twelve, and nobody noticed because the guard was a floor (len(found) >= 9) walking seven
of the nine modules that own one -- so three new policies and two whole unwalked modules were both
invisible. @registry-completeness, the same shape as RM96's _ALL_MODELS hole: a number in prose
is a registry nothing iterates. test_gated_snapshots.py now discovers the modules instead of
listing them and asserts an equality over what it walked, which is a claim that cannot go stale.
Source code in enricher/src/just_dna_enricher/net.py
EndpointRetired ¶
Bases: Exception
Mixed into every client's *Retired error: the source answered HTTP 410 (RM361).
A marker, not a catchable family of its own: handlers keep catching their client's could-not-ask
type, and read the reason off the exception with could_not_ask_reason.
Withholding ¶
Mixed into every client whose contract is a withheld value rather than an exception (RM361).
Such a client answers a failed request with None (or its outcome's could-not-ask arm), so a 410
reached its caller as an outage. retired is how it says otherwise without raising: the
retired_message of the first 410 the client met, None until then. Sticky for the client's
life, because a retired route is a fact about the source rather than about one request; a caller
recording why a question went unanswered reads it with withheld_reason.
A run, though, is not the client's life: a caller may reuse one client across runs (RM404's
LookupClients), and run 1's 410 must not tell run 2's timeout to upgrade (RM446). So every 410 is
also counted in retired_met, which a run reads before it asks (retired_mark) and compares after
(withheld_reason(client, since=mark)); the client itself is never reset, since a caller's client is
not the run's to alter.
note_retired ¶
Record a 410 from failure (an exception or a response); True when it was one.
Source code in enricher/src/just_dna_enricher/net.py
StreamedFile
dataclass
¶
What one bulk download established: where it landed, its digest, and the source's own labels.
The digest is returned rather than logged. Four of the eleven copies computed a sha256 while
streaming and then only wrote it to the log, so a caller that needed to record the provenance of
the bytes it had just fetched had to hash the file again (@dont-discard-computed). Two of them
later grew a tuple[Path, str] return for exactly that reason, one lane at a time.
etag and last_modified are the source's, None when it sends neither. They are how a lane
can later ask has this file changed without downloading it again — MANE, CIViC and PubMind
record them today, and the rest get them for free rather than growing their own copy later.
wait_retry_after ¶
Bases: wait_base
A tenacity wait that is at least what the server's Retry-After asked for, else fallback's.
Never shorter than the fallback, so a server asking for less than the backoff does not shorten it;
capped at ceiling, which retry_if_transient_within keeps a retry from ever reaching.
Source code in enricher/src/just_dna_enricher/net.py
check_batch_size ¶
Raise ValueError for a batch_size below 1, naming owner's field (RM455).
batched with a size of 0 yields no batch, so a client asked nothing and answered {}, which every
caller reads as "no record for any id". Each settings class calls this when it is built and again
where its batched read reads the field, since a dataclass field may be set after construction (the
form 0.9 gave EnsemblSettings, review round 1 item 8).
Source code in enricher/src/just_dna_enricher/net.py
batched ¶
Split into batches of at most size, preserving order (so emission stays deterministic).
Source code in enricher/src/just_dna_enricher/net.py
dedupe ¶
First-occurrence-order de-duplication (Principle 7: never set iteration for emitted order).
Source code in enricher/src/just_dna_enricher/net.py
retry_attempts ¶
How many attempts this client may make: its own default, raised to the configured floor.
A floor, never a flat setting. The per-client numbers are deliberate — gnomAD and eutils sit at 4 because their budgets are the tightest — so a single value that set every client would flatten tuning that was chosen on purpose, while one that raises preserves it. Below the default it is a no-op rather than a way to make a client give up sooner; there is no deployment that wants less persistence than an author at a terminal, and allowing it would turn one variable into a footgun.
Why a knob exists at all: three attempts is right for the audience the CLI was written for — a
person who would rather see a failure in ten seconds than wait out a flapping upstream. It is wrong
for the other shape the 0.5 tiering created, a server running enrich() inside an unattended
publish, where giving up on a transient 502 does not cost ten seconds, it costs the publisher a
whole re-upload of a module the server had already accepted and validated. Two callers wanting
opposite things from one constant is the definition of a knob.
Safe to raise because every gated client with a published rate budget paces before it retries: an extra attempt spends a slot of that budget rather than bursting past it. The AlphaGenome Atlas client's gate has a zero interval, because the Atlas publishes no budget and none was found up to 21 calls/s (RM307); it still counts every attempt.
Source code in enricher/src/just_dna_enricher/net.py
is_transient_status ¶
Whether exc is an HTTP status a second attempt could change (a 5xx or a 429).
Source code in enricher/src/just_dna_enricher/net.py
retry_if_transient ¶
Retry a transport failure, a timeout, a 5xx or a 429 — never a permanent 4xx (RM354).
also adds a client's own retried type, such as the rate-limit error gnomad and eutils raise
on a 429 before raise_for_status().
Source code in enricher/src/just_dna_enricher/net.py
retired_message ¶
What a client's *Retired error says: the URL, the 410, and the source's own text.
For a client that reads the status itself rather than raising on it; raise_if_retired is the
same message for one that translates an HTTPStatusError.
Source code in enricher/src/just_dna_enricher/net.py
raise_if_retired ¶
Raise error — a client's *Retired type — when exc is a 410; otherwise return.
Called first in a client's status-translating except arm, so the arm keeps translating every
other status the way it did.
Source code in enricher/src/just_dna_enricher/net.py
could_not_ask_reason ¶
retired when exc says the source retired the endpoint, else unreachable (RM361).
For a handler that caught a client's could-not-ask type and is about to record why the question
went unanswered. A retired endpoint is a fact about the source and is cleared by an upgrade; an
outage is cleared by a re-run; writing the second about the first sends the reader to the wrong
remedy. The exception's causes are read too, so a client that translates a 410 into its plain
could-not-ask type (raise XUnavailable(...) from exc) is still read as retired.
Source code in enricher/src/just_dna_enricher/net.py
aggregate_reason ¶
The reason a group of unanswered questions is recorded under: retired if any one was (RM361).
For a record that aggregates several requests: one retired endpoint means the run needs an upgrade
whatever else happened, which is the remedy the record must lead with. Otherwise unreachable.
Source code in enricher/src/just_dna_enricher/net.py
retired_mark ¶
How many 410s withholder has met so far, read before a run asks (RM446).
None for a client that does not count them (a host's own, written before the counter), which
withheld_reason then reads as RM361 did: retired for the client's life.
Source code in enricher/src/just_dna_enricher/net.py
withheld_reason ¶
retired when a withholding client met a 410, else unreachable (RM361).
With since, the mark retired_mark read before this run asked, only a 410 met after it counts
(RM446): a reused client's earlier 410 says nothing about why this run went unanswered.
Source code in enricher/src/just_dna_enricher/net.py
release_dataset ¶
A dataset label, <lane>_<release>, or the bare lane when the source stated no release (RM361).
The label is a merge-key member and a sources.csv cell; a release the file did not state is
withheld from it, never spelled unknown (a release of that name) or None.
Source code in enricher/src/just_dna_enricher/net.py
stated_release ¶
The release a dataset label states, or None when it states none (RM373).
The inverse of release_dataset for a reader: a blank cell, and the bare lane that label writes when
the source stated no release, are both no release. What a licence row's dataset is computed from,
so a table whose rows say gwas_catalog names no release rather than one called gwas_catalog.
Source code in enricher/src/just_dna_enricher/net.py
stream_to_file ¶
stream_to_file(
dest: Path,
url: str,
*,
error_cls: type[Exception],
retired: type[Exception],
what: str,
timeout: float | None = _BULK_TIMEOUT,
remedy: str = "",
_stop: Event | None = None,
) -> StreamedFile
Stream url to dest atomically, hashing as it goes; retry the transport, translate the rest.
The one body every download_* in this package calls. Four properties, and each is a defect that
reached a user before it was one:
Atomic. The bytes go to <dest>.part and are renamed only once the stream finished, so a
failed fetch leaves the directory as it found it rather than truncating a good file already there
(@a-failed-fetch-is-not-a-no-op). The partial is removed on failure — a .part left behind is
the one residue a re-run would have to reason about.
Retried, but only where retrying is honest. httpx.TransportError covers a connection cut
mid-body — RemoteProtocolError subclasses it, which is the failure that motivated this — and a
second attempt genuinely fixes it. A status error is not retried: a 404 from a mistyped
release tag is the same 404 four times over, and the caller wants it now rather than after three
backoffs. attempt_floor reads $JUST_DNA_HTTP_RETRY_ATTEMPTS at retry time like every other
policy here, so a deployment can raise the floor without touching code (RM42).
Translated. httpx's exceptions do not leave this function. A caller of this package may not
be made to depend on the transport library's exception tree to know that a fetch failed, and the
lane adapters catch their own builder's type — so a leak is not merely untidy, it is a lane that
cannot report built=False (@client-exception-contract).
Restarted from zero on each attempt. The hasher and the output file are created inside the
attempt, not outside it: a retry that appended to a partial body would produce a file whose
digest is real and whose contents are nonsense, which no footer check and no raise_for_status
would catch.
A 410 is the lane's retired, never its error_cls (RM361, audit round 2). A source that
retired the file moved it, and the remedy is an upgrade rather than a re-run; retired is the
lane's *Retired (a subclass of error_cls marked EndpointRetired), required so no lane can
forget it, and the message is retired_message's (the host that answered, the source's own body
quoted), followed by what and the lane's remedy.
Stoppable between chunks, for a caller on a thread pool (RM409). _stop is checked before the
request opens and before each chunk is written, and a retry's backoff waits on it rather than
sleeping; once it is set the download raises, the .part is removed, and nothing lands under dest. A KeyboardInterrupt reaches only the main thread, so a
pool that fetches parts in workers sets the flag on interrupt before it joins them; without it the
join waited for every in-flight part to stream to its end, and a 22 GB mirror ignored two Ctrl-C.
Bounded, by default (RM421). timeout is 120 s unless the caller names its own, and httpx
applies it to connect, read, write and pool alike. The read half bounds the gap between two chunks,
not the whole download, so a slow mirror that keeps sending is never cut off, while a server that
accepts the connection and then sends nothing raises httpx.ReadTimeout, which is retried as a
transport error above and translated to error_cls on giving up. The default was once None, which
httpx reads as no timeout at all: eleven of fourteen callers named none, so such a stall blocked
forever, no retry fired, and a pool worker stuck in the read never reached the _stop check. An
explicit timeout=None still means no timeout; no caller in this package passes it, and a walk over
the call sites keeps it that way.
what names the thing being fetched for the message ("the ClinVar VCF"); remedy is an optional
sentence saying what the caller can do instead, for the lanes that accept a local file.
Source code in enricher/src/just_dna_enricher/net.py
616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 | |
retry_after_seconds ¶
retry_after_seconds(
exc: BaseException | None,
*,
now: Callable[[], float] = time.time,
) -> float | None
How long a throttled answer asked the client to wait, or None when it did not say.
Read only off a status a retry can change (is_transient_status: a 5xx or a 429), in either of
RFC 9110's forms — delta-seconds (Ensembl sends a fraction, 40.0) or an HTTP-date, measured
against now (wall clock, since the date is one). A date already past is 0.0. No header, an
unreadable one, or any other exception is None: unknown, never a wait of zero.
Source code in enricher/src/just_dna_enricher/net.py
asks_past_ceiling ¶
Whether exc asked for a wait longer than ceiling — a request not to retry now.
Source code in enricher/src/just_dna_enricher/net.py
retry_if_transient_within ¶
retry_if_transient_within(
*also: type[BaseException],
ceiling: float = RETRY_AFTER_CEILING,
) -> retry_base
retry_if_transient, except a throttle whose Retry-After exceeds ceiling is not retried.