just_dna_enricher.caches

just_dna_enricher.caches

The cache lanes as a registry, and the one endpoint that rebuilds them — RM176.

Every snapshot this tier knows about, all but one of them built here. Each is supposed to have three stages: acquire (a download, a file the operator supplies, or — since RM171 — its parent lanes being on disk), build (the conversion into the snapshot a check reads), and publish (the upload a deployment then pulls with cache pull). Before this module the three stages existed as eleven independent CLI commands and one hand-kept four-tuple list inside cli.py, and the gaps that arrangement hid were all of the same kind: something true of a lane that no code anywhere asserted.

The count is deliberately not stated here. It was "twelve snapshots, eleven of them built here", which is the counted-prose failure this workspace keeps re-learning: a sentence no test reads, true on the day it was written and quietly wrong the first time the registry grew correctly (@counted-prose-needs-a-fixed-field). test_cache_lanes.py asserts the equality that sentence was gesturing at, over the walked set.

• Three lanes were missing from the roster entirely, so cache status reported nine caches on a machine that has twelve and cache pull could not be asked about them.
• Three had a licence permitting publication and no publish command.
• One had no way to be found at all without a flag on every invocation.

So the registry is the point, not the rebuild command. A list is only as complete as whoever last edited it; a registry can be walked, and test_cache_lanes.py walks it against the *_build modules on disk. That is the shape this repository keeps re-learning (@registry-completeness): assert an equality over a walked set, never a floor and never a count in prose.

Every absence carries its reason as a field, not as a comment. A lane with no ensure states why in unpublished, and the three reasons are genuinely different — PharmVar's and PubMind's are refusals (a personal key; terms nobody publishes), ACMG's and MANE's are unestablished permissions, and Ensembl's is that it is built elsewhere. A comment saying so speaks to whoever opens this file; a field says it in cache status, in cache pull and in cache rebuild, which is where somebody actually asks the question.

The rebuild outcome is three-valued (@tri-state). Built, failed, and could not run unattended — ACMG needs a workbook that is Elsevier supplementary material, PharmVar needs a personal key, CIViC needs a release date to pin. Folding the third into "failed" would have a nightly rebuild report errors for lanes behaving exactly as designed; folding it into "built" would be a lie. It is printed, never silently skipped.

RebuildRequest dataclass

RebuildRequest(
    out_dir: Path,
    declared_use: str = "unstated",
    pin: str | None = None,
    source: Path | None = None,
    parents: dict[str, Path] = dict(),
)

What one lane's rebuild is given. The same fields for every lane, so the loop has no branches.

pin is the release to build — a MANE version, a CIViC release date, a STRchive tag. A lane that has no notion of one ignores it; a lane that needs one and did not get it reports could-not-run rather than quietly building the moving default, because a snapshot whose reference was "whatever was upstream that afternoon" cannot be compared against twice.

source is the acquire stage's input when the operator already holds it — ACMG's workbook, a ClinVar VCF, a ClinPGx archive, a STRchive catalogue. For ACMG it is the only way, because the workbook is Elsevier supplementary material and nothing may fetch it on the author's behalf; for the rest it is the off-switch that lets a rebuild run with no network, which is the shape @off-switch-needs-a-probe asks for. MANE and CIViC take three input files each and accept no source: two of three is not a build for either of them, and a flag that can only ever supply one would be a flag that cannot do its job.

parents is where each parent lane's snapshot is, for the one lane that has any (RM171). Filled by the caller rather than resolved inside the adapter, because a rebuild run has an answer the registry cannot give: cache rebuild writes every lane into out/<lane>/, so the ClinVar the miss lane should join against is the one this run just cut, not whatever is in the live cache. A parent the caller does not name falls back to that lane's own resolver, which is what cache prepare and a bare --only mitomap_miss want.

RebuildOutcome dataclass

RebuildOutcome(
    lane: str,
    built: bool | None,
    detail: str,
    out_dir: Path | None = None,
)

Built / failed / could-not-run, and the reason in every case.

built is the tri-state: True the snapshot is on disk, False something went wrong, None this lane cannot be rebuilt unattended and that is by design. The third is not a failure and must not be counted as one — a nightly rebuild reporting an error for PharmVar would be reporting the licence working as intended.

CacheLane dataclass

CacheLane(
    name: str,
    subdir: str,
    serves: str,
    build_command: str | None,
    resolve: Callable[..., Path | None],
    default_dir: Callable[..., Path],
    env_var: str,
    rebuild: RebuildAdapter | None,
    ensure: Callable[..., Path] | None,
    publish_repo: str | None,
    terms: SourceTerms | None,
    unpublished: str | None = None,
    unbuilt: str | None = None,
    release_label: Callable[
        [Path], str | None
    ] = _dataset_label,
    publish_command: str | None = None,
    parents: tuple[str, ...] = (),
    approx_mb: int | None = None,
)

One snapshot, all three of its stages, and the reason for each stage it does not have.

LaneStatus dataclass

LaneStatus(
    lane: CacheLane,
    state: str,
    looked_in: Path,
    path: Path | None = None,
    release: str | None = None,
    release_unreadable: bool = False,
    size_bytes: int | None = None,
)

One lane as it stands on this machine — the projection cache status renders (S91, RM204).

state is three-valued. present: resolve() found a snapshot at path. absent: nothing at the place the lane looks. occupied: the place the lane looks exists and is non-empty and holds no snapshot — a build that failed after its downloads, a payload deleted beside its release.json, a stray .part, a foreign parquet. It used to render as absent, which tells an operator to run a pull that prepare is going to refuse: provisioning never deletes, so it stops in front of a non-empty target rather than building over it. The two states want different hands — absent wants cache pull or the lane's build command, occupied wants the directory moved aside or cache prune.

looked_in is the directory the verdict is about: the lane's env_var if set, else its default directory. prepare's own refusal is about the default directory specifically, so an override pointing at a junk directory reads occupied here while prepare would build into an empty default that the override then hides — say which directory, and the operator can see it.

release is what the snapshot names, None when it does not say; release_unreadable is the present-and-unreadable case, a release.json that exists and does not parse, which is a provenance failure and not a data failure — the snapshot is still usable and the line says so rather than hiding it. Both are None/False unless present.

PrepareOutcome dataclass

PrepareOutcome(
    lane: str,
    ready: bool | None,
    route: str,
    detail: str,
    path: Path | None = None,
)

One lane's provisioning result, and which route answered — not merely whether it worked.

ready is the tri-state: True the snapshot is on disk and usable, False the attempt failed, None this lane offers no route on this machine and that is by design rather than an error.

route is the half a caller cannot reconstruct from ready, and it is the whole point of the command: present (already there, nothing touched), pulled (downloaded from HuggingFace), built (built locally because nothing publishes it), none. A deployment auditing its own caches has to be able to tell a snapshot it fetched from one it made — they are different artifacts with different provenance, and release.json says which release but not which route.

lane_name

lane_name(spelling: str) -> str | None

A lane name as the registry declares it, or None when nothing answers to that spelling.

Accepts a hyphen where the declared member has an underscore and returns the declared member, never the caller's spelling — the rule every closed vocabulary in this workspace follows, and the reason it is a function is that a caller who merely calls a normalizer and keeps its own string has done nothing (@vocab-separator-slip). drug_labels and mitomap_miss are the two members it matters for: both are commonly written with a hyphen, and mitomap-miss is the spelling the design note and draft-panel --source use.

Source code in enricher/src/just_dna_enricher/caches.py

def lane_name(spelling: str) -> str | None:
    """A lane name as the registry declares it, or `None` when nothing answers to that spelling.

    Accepts a hyphen where the declared member has an underscore and **returns the declared member**,
    never the caller's spelling — the rule every closed vocabulary in this workspace follows, and the
    reason it is a function is that a caller who merely *calls* a normalizer and keeps its own string
    has done nothing (`@vocab-separator-slip`). `drug_labels` and `mitomap_miss` are the two members
    it matters for: both are commonly written with a hyphen, and `mitomap-miss` is the spelling the
    design note and `draft-panel --source` use.
    """
    folded = spelling.strip().lower().replace("-", "_")
    return folded if folded in LANES_BY_NAME else None

parent_snapshots

parent_snapshots(
    lane: CacheLane, request: RebuildRequest
) -> tuple[dict[str, Path], list[str]]

Where each parent's snapshot is, and the parents that are not anywhere.

The caller's request.parents wins, then the parent lane's own resolver. Both halves are needed and they answer different questions: a cache rebuild run has just cut a fresh ClinVar into out/clinvar and the child must join that one, while a lone --only mitomap_miss has no such run behind it and means the caches on this machine.

Source code in enricher/src/just_dna_enricher/caches.py

def parent_snapshots(lane: CacheLane, request: RebuildRequest) -> tuple[dict[str, Path], list[str]]:
    """Where each parent's snapshot is, and the parents that are not anywhere.

    The caller's `request.parents` wins, then the parent lane's own resolver. Both halves are needed
    and they answer different questions: a `cache rebuild` run has just cut a fresh ClinVar into
    `out/clinvar` and the child must join *that* one, while a lone `--only mitomap_miss` has no such
    run behind it and means the caches on this machine.
    """
    found: dict[str, Path] = {}
    missing: list[str] = []
    for name in lane.parents:
        parent = LANES_BY_NAME.get(name)
        supplied = request.parents.get(name)
        if supplied is not None:
            # **A directory is not a snapshot.** Every adapter `mkdir`s its `out_dir` before it
            # downloads anything, so a parent whose fetch was cut mid-body — the NCBI incident RM187
            # was written for — leaves an empty `out/<parent>/` that `is_dir()` accepted as present.
            # The child then ran, its join found no parquet, and it was reported FAILED: the absence
            # of another lane filed as this one failing, the very arm the guard below exists to
            # refuse. Ask the parent lane's own resolver whether the *payload* is there, the same
            # predicate the registry route two lines down uses. A supplied path with no payload is
            # missing and says so; it does not fall back to whatever the machine's cache holds,
            # because a child pinned to an old parent while sitting beside a failed new one is the
            # fork `parents_from_rebuild_dir` exists to prevent.
            payload = _snapshot_at(parent, Path(supplied))
            if payload is not None:
                found[name] = payload
            else:
                missing.append(f"{name} (supplied path {supplied} holds no snapshot)")
            continue
        resolved = parent.resolve() if parent is not None else None
        if resolved is None:
            missing.append(name)
            continue
        found[name] = resolved
    return found, missing

parents_from_rebuild_dir

parents_from_rebuild_dir(
    lane: CacheLane, out: Path
) -> dict[str, Path]

A derived lane's parents as this rebuild run just cut them, under out/<parent>/.

Two callers — rebuild_caches and the cache rebuild command — and one derivation, because the two producing different answers is precisely the fork that would give a child pinned to one ClinVar and sitting beside another. A parent this run did not build is left out and falls back to the registry's resolver inside rebuild_lane, which is the --only <child> case.

Built means the payload is there, not that the directory is. A parent whose download was cut leaves an empty out/<parent>/ behind (every adapter mkdirs first), and is_dir() handed that to the child as a parent — so the child's join failed on a parquet that did not exist and the child was the lane reported FAILED. Judged by the parent lane's resolver now.

Source code in enricher/src/just_dna_enricher/caches.py

def parents_from_rebuild_dir(lane: CacheLane, out: Path) -> dict[str, Path]:
    """A derived lane's parents as **this rebuild run** just cut them, under `out/<parent>/`.

    Two callers — `rebuild_caches` and the `cache rebuild` command — and one derivation, because the
    two producing different answers is precisely the fork that would give a child pinned to one
    ClinVar and sitting beside another. A parent this run did not build is left out and falls back to
    the registry's resolver inside `rebuild_lane`, which is the `--only <child>` case.

    **Built means the payload is there, not that the directory is.** A parent whose download was
    cut leaves an empty `out/<parent>/` behind (every adapter `mkdir`s first), and `is_dir()` handed
    that to the child as a parent — so the child's join failed on a parquet that did not exist and
    the child was the lane reported FAILED. Judged by the parent lane's resolver now.
    """
    return {
        name: out / name
        for name in lane.parents
        if _snapshot_at(LANES_BY_NAME.get(name), out / name) is not None
    }

rebuild_lane

rebuild_lane(
    lane: CacheLane, request: RebuildRequest
) -> RebuildOutcome

Run one lane's three stages, or say why it did not run.

A lane with no adapter is a None outcome carrying unbuilt — Ensembl is the only one, and its snapshot is provisioned rather than rebuilt, so the honest answer to "rebuild everything" is that this one is somebody else's build.

A derived lane whose parents are not on disk is the third state too, and naming the parent is the whole point (RM171). The two wrong answers are both available and both silent: an empty miss set reads as "MITOMAP publishes nothing ClinVar lacks", which is a claim about the world derived from a comparison that never ran, and a False would file the absence as this lane failing when what is absent belongs to another one. The guard is here rather than in the adapter because it is registry-driven — it is lane.parents that decides, so a second derived lane inherits it.

Source code in enricher/src/just_dna_enricher/caches.py

def rebuild_lane(lane: CacheLane, request: RebuildRequest) -> RebuildOutcome:
    """Run one lane's three stages, or say why it did not run.

    A lane with no adapter is a `None` outcome carrying `unbuilt` — Ensembl is the only one, and its
    snapshot is provisioned rather than rebuilt, so the honest answer to "rebuild everything" is that
    this one is somebody else's build.

    **A derived lane whose parents are not on disk is the third state too, and naming the parent is
    the whole point** (RM171). The two wrong answers are both available and both silent: an empty miss
    set reads as "MITOMAP publishes nothing ClinVar lacks", which is a claim about the world derived
    from a comparison that never ran, and a `False` would file the absence as this lane failing when
    what is absent belongs to another one. The guard is here rather than in the adapter because it is
    registry-driven — it is `lane.parents` that decides, so a second derived lane inherits it.
    """
    if lane.rebuild is None:
        return RebuildOutcome(lane.name, None, lane.unbuilt or "no builder in this tier")
    resolved: dict[str, Path] = {}
    if lane.parents:
        resolved, missing = parent_snapshots(lane, request)
        if missing:
            how = "; ".join(
                f"{entry} (`{LANES_BY_NAME[name].build_command}`, or `cache pull --only {name}`)"
                if LANES_BY_NAME.get(name) is not None and LANES_BY_NAME[name].build_command
                else entry
                for entry in missing
                for name in (entry.split(" ", 1)[0],)
            )
            return RebuildOutcome(
                lane.name,
                None,
                f"derived from {' and '.join(lane.parents)}; not on disk: {how}. A miss set computed "
                f"without a parent would be an increment measured against a comparison that never "
                f"ran, not an empty one",
            )
        request = RebuildRequest(
            out_dir=request.out_dir,
            declared_use=request.declared_use,
            pin=request.pin,
            source=request.source,
            parents=resolved,
        )
    logger.info("Rebuilding the %s snapshot into %s ...", lane.name, request.out_dir)
    return lane.rebuild(request)

snapshot_bytes

snapshot_bytes(path: Path) -> int

Bytes on disk under a provisioned snapshot — every regular file, or the one file itself.

Source code in enricher/src/just_dna_enricher/caches.py

def snapshot_bytes(path: Path) -> int:
    """Bytes on disk under a provisioned snapshot — every regular file, or the one file itself."""
    if path.is_file():
        return path.stat().st_size
    return sum(entry.stat().st_size for entry in path.rglob("*") if entry.is_file())

provisioning_closure

provisioning_closure(lane: CacheLane) -> list[CacheLane]

The lanes a blank box has to hold for lane to build: its transitive parents, then itself.

In registry order, each lane once, parents before children — the order prepare already walks. A derived lane's approx_mb prices the increment it stores; the closure prices what provisioning it actually costs (S97): mitomap_miss is under a megabyte and its closure is a ClinVar download.

Source code in enricher/src/just_dna_enricher/caches.py

def provisioning_closure(lane: CacheLane) -> list[CacheLane]:
    """The lanes a blank box has to hold for `lane` to build: its transitive parents, then itself.

    In registry order, each lane once, parents before children — the order `prepare` already walks.
    A derived lane's `approx_mb` prices the increment it stores; the closure prices what provisioning
    it actually costs (S97): `mitomap_miss` is under a megabyte and its closure is a ClinVar download.
    """
    wanted: set[str] = set()
    frontier = [lane.name]
    while frontier:
        name = frontier.pop()
        if name in wanted:
            continue
        wanted.add(name)
        frontier.extend(LANES_BY_NAME[name].parents)
    return [member for member in CACHE_LANES if member.name in wanted]

lane_status

lane_status(
    lanes: list[CacheLane] | None = None,
) -> list[LaneStatus]

Where every lane stands, read-only — nothing is downloaded and nothing is written.

The registry's status half, beside its provisioning half (prepare_caches). It existed only as the loop inside cache status until a consumer serving the same answer over HTTP wrote the loop a second time (S91), which is two projections of one registry — the shape RM176 exists to end. In registry order, one entry per lane, so a renderer that iterates it renders the whole registry.

Source code in enricher/src/just_dna_enricher/caches.py

def lane_status(lanes: list[CacheLane] | None = None) -> list[LaneStatus]:
    """Where every lane stands, read-only — nothing is downloaded and nothing is written.

    The registry's status half, beside its provisioning half (`prepare_caches`). It existed only as
    the loop inside `cache status` until a consumer serving the same answer over HTTP wrote the loop a
    second time (S91), which is two projections of one registry — the shape RM176 exists to end. In
    registry order, one entry per lane, so a renderer that iterates it renders the whole registry.
    """
    out: list[LaneStatus] = []
    for lane in CACHE_LANES if lanes is None else lanes:
        path = lane.resolve()
        # `resolve()` has loaded `.env` by now, so the override is read the way the resolver read it.
        override = os.getenv(lane.env_var)
        looked_in = Path(override).expanduser() if override else lane.default_dir()
        if path is not None:
            release = lane.release_label(path)
            unreadable = release is None and (path / RELEASE_FILENAME).exists() and read_release(path) is None
            out.append(
                LaneStatus(lane, "present", looked_in, path, release, unreadable, snapshot_bytes(path))
            )
            continue
        occupied = looked_in.is_file() or (looked_in.is_dir() and any(looked_in.iterdir()))
        out.append(LaneStatus(lane, "occupied" if occupied else "absent", looked_in))
    return out

prepare_lane

prepare_lane(
    lane: CacheLane, request: RebuildRequest
) -> PrepareOutcome

Provision one lane by whichever route it has: pull it, or build it, or say why neither.

The route is a property of the lane, never a flag. A lane with an ensure is published, so pulling is right and building would spend an operator's bandwidth re-deriving bytes somebody already made. A lane without one is unpublished for a recorded reason — PharmVar's personal key, PubMind's absent terms, NCBI's policy, ACMG's supplementary material — and building locally is the only route there will ever be. Asking the caller to choose would be asking them to restate the licensing story as a flag.

A present cache is left alone, exactly as cache pull leaves one alone: provisioning is idempotent and cheap to re-run, and re-deriving a snapshot somebody is reading is how a resolver comes to see a half-written table. Re-cutting one is cache rebuild, which writes somewhere else on purpose.

Source code in enricher/src/just_dna_enricher/caches.py

def prepare_lane(lane: CacheLane, request: RebuildRequest) -> PrepareOutcome:
    """Provision one lane by whichever route it has: pull it, or build it, or say why neither.

    **The route is a property of the lane, never a flag.** A lane with an `ensure` is published, so
    pulling is right and building would spend an operator's bandwidth re-deriving bytes somebody
    already made. A lane without one is unpublished *for a recorded reason* — PharmVar's personal
    key, PubMind's absent terms, NCBI's policy, ACMG's supplementary material — and building locally
    is the only route there will ever be. Asking the caller to choose would be asking them to restate
    the licensing story as a flag.

    **A present cache is left alone**, exactly as `cache pull` leaves one alone: provisioning is
    idempotent and cheap to re-run, and re-deriving a snapshot somebody is reading is how a resolver
    comes to see a half-written table. Re-cutting one is `cache rebuild`, which writes somewhere else
    on purpose.
    """
    existing = lane.resolve()
    if existing is not None:
        return PrepareOutcome(lane.name, True, "present", f"already provisioned at {existing}", existing)

    if lane.ensure is not None:
        if lane.terms is not None:
            # The terms are accepted when the data is TAKEN, and a download is taking it.
            try:
                reason = check_declared_use(lane.terms, request.declared_use)
            except LicenseRefusal as exc:
                return PrepareOutcome(lane.name, False, "pulled", f"refused: {exc}")
            if reason is not None:
                return PrepareOutcome(lane.name, None, "none", f"skipped: {reason}")
        try:
            path = lane.ensure()
        except Exception as exc:
            # Not an error here either, and for a sharper reason than in `cache pull`: this command's
            # job is to leave the machine with a usable cache, and a repo nobody has created yet is a
            # fact about the world that no amount of retrying changes. Caught by TYPE — the name is
            # not the contract, and a string compare here would survive a rename that broke it
            # (`@client-exception-contract`).
            if isinstance(exc, SnapshotNotPublished):
                return PrepareOutcome(lane.name, None, "none", f"nothing published yet: {exc}")
            return PrepareOutcome(lane.name, False, "pulled", str(exc))
        return PrepareOutcome(lane.name, True, "pulled", f"downloaded to {path}", path)

    if lane.rebuild is None:
        return PrepareOutcome(lane.name, None, "none", lane.unbuilt or "no route in this tier")

    # Built locally, because nothing publishes it. Into a staging directory beside the target and
    # moved across only once the build has finished: the resolvers read the target by globbing, so a
    # build writing straight into it would be visible half-done, and a short parquet still has a
    # footer. `resolve()` answered `None` above, which means the target holds no PAYLOAD — not that
    # it is absent. A directory that exists with no snapshot in it (a build that failed after its
    # downloads, a payload deleted by hand beside its `release.json`, a stray `.part`) made
    # `staging.replace(target)` raise `Directory not empty` out of the whole command, and every lane
    # after it was never attempted. Provisioning never deletes (deletion is by declaration or by
    # `cache prune`, never a side effect), so it is refused here, before a build is spent on it.
    target = lane.default_dir()
    if target.exists() and any(target.iterdir()):
        return PrepareOutcome(
            lane.name,
            False,
            "built",
            f"{target} exists and holds no {lane.name} snapshot; prepare never deletes, so move it "
            f"aside (or `cache prune --only {lane.name}` if it is a retired file) and re-run",
        )
    staging = target.parent / f"{target.name}.incoming"
    if staging.exists():
        shutil.rmtree(staging)
    outcome = rebuild_lane(
        lane,
        RebuildRequest(
            out_dir=staging,
            declared_use=request.declared_use,
            pin=request.pin,
            source=request.source,
            parents=request.parents,
        ),
    )
    if outcome.built is not True:
        shutil.rmtree(staging, ignore_errors=True)
        return PrepareOutcome(
            lane.name,
            outcome.built,
            "built" if outcome.built is False else "none",
            outcome.detail,
        )
    if not staging.is_dir():
        # A builder that reports success and writes nothing. Not reachable through today's adapters,
        # and guarded anyway because the alternative is a raw `FileNotFoundError` out of `replace()`
        # naming a staging path the operator has never heard of — a generic rejection where a
        # specific one is a fix (`@specific-rejection`). It also keeps the contract on the visible
        # side: `built is True` means a snapshot exists, and this is where that is established.
        return PrepareOutcome(
            lane.name,
            False,
            "built",
            f"the {lane.name} builder reported success but wrote nothing to {staging}",
        )
    target.parent.mkdir(parents=True, exist_ok=True)
    staging.replace(target)
    return PrepareOutcome(lane.name, True, "built", f"{outcome.detail} → {target}", target)

prepare_caches

prepare_caches(
    lanes: list[CacheLane] | None = None,
    *,
    declared_use: str = "unstated",
    pins: dict[str, str] | None = None,
    sources: dict[str, Path] | None = None,
) -> list[PrepareOutcome]

Provision every cache by its own route — the Python half of cache prepare.

The counterpart to rebuild_caches, and a real API rather than a loop the CLI happens to own: a deployment's own provisioning step is Python far more often than it is a shell script, and the alternative is every caller re-deriving which lane pulls and which builds from the licensing story. That derivation is the command.

Source code in enricher/src/just_dna_enricher/caches.py

def prepare_caches(
    lanes: list[CacheLane] | None = None,
    *,
    declared_use: str = "unstated",
    pins: dict[str, str] | None = None,
    sources: dict[str, Path] | None = None,
) -> list[PrepareOutcome]:
    """Provision every cache by its own route — the Python half of `cache prepare`.

    The counterpart to `rebuild_caches`, and a real API rather than a loop the CLI happens to own: a
    deployment's own provisioning step is Python far more often than it is a shell script, and the
    alternative is every caller re-deriving *which lane pulls and which builds* from the licensing
    story. That derivation is the command.
    """
    pins = pins or {}
    sources = sources or {}
    outcomes = []
    for lane in lanes if lanes is not None else CACHE_LANES:
        request = RebuildRequest(
            out_dir=lane.default_dir(),
            declared_use=declared_use,
            pin=pins.get(lane.name),
            source=sources.get(lane.name),
        )
        try:
            outcomes.append(prepare_lane(lane, request))
        except Exception as exc:  # one lane's crash may not sink the others' report
            # The per-lane isolation `cache pull` has had all along. `prepare_lane` reports every
            # failure it can foresee as an outcome; this is for the one it cannot, and the
            # alternative is a traceback that hides which lanes DID provision.
            logger.exception("preparing the %s cache raised", lane.name)
            outcomes.append(
                PrepareOutcome(
                    lane.name,
                    False,
                    "pulled" if lane.ensure is not None else "built",
                    str(exc),
                )
            )
    return outcomes

rebuild_caches

rebuild_caches(
    lanes: list[CacheLane] | None = None,
    *,
    out: Path,
    declared_use: str = "unstated",
    pins: dict[str, str] | None = None,
    sources: dict[str, Path] | None = None,
) -> list[RebuildOutcome]

Rebuild every lane into out/<lane>/ — the Python half of cache rebuild.

Separate from prepare_caches because the two answer different questions and a flag joining them would hide that. Rebuild re-derives, into a directory nothing is reading, so a deployment can cut a new set and adopt it deliberately. Prepare provisions, into the live cache locations, and leaves a lane that already has a snapshot alone. A caller wanting "make sure every cache exists" wants prepare; one wanting "cut a fresh set from today's sources" wants rebuild.

Source code in enricher/src/just_dna_enricher/caches.py

def rebuild_caches(
    lanes: list[CacheLane] | None = None,
    *,
    out: Path,
    declared_use: str = "unstated",
    pins: dict[str, str] | None = None,
    sources: dict[str, Path] | None = None,
) -> list[RebuildOutcome]:
    """Rebuild every lane into `out/<lane>/` — the Python half of `cache rebuild`.

    Separate from `prepare_caches` because the two answer different questions and a flag joining them
    would hide that. **Rebuild re-derives**, into a directory nothing is reading, so a deployment can
    cut a new set and adopt it deliberately. **Prepare provisions**, into the live cache locations,
    and leaves a lane that already has a snapshot alone. A caller wanting "make sure every cache
    exists" wants prepare; one wanting "cut a fresh set from today's sources" wants rebuild.
    """
    pins = pins or {}
    sources = sources or {}
    # **A derived lane joins the snapshots THIS run cut, where this run cut them** (RM171). Every lane
    # is written into `out/<lane>/`, so by the time the child's turn comes its parents are siblings on
    # disk — and joining the live cache instead would produce a child whose `release.json` pins two
    # parents that are not the ones beside it.
    return [
        rebuild_lane(
            lane,
            RebuildRequest(
                out_dir=out / lane.name,
                declared_use=declared_use,
                pin=pins.get(lane.name),
                source=sources.get(lane.name),
                parents=parents_from_rebuild_dir(lane, out),
            ),
        )
        for lane in (lanes if lanes is not None else CACHE_LANES)
    ]