just_dna_compiler.sweep¶
just_dna_compiler.sweep ¶
The instrument behind the release record (RM126): measure what a release changed about compiled output, and fail a release whose measurement carries no declaration.
just_dna_format.release_records holds the table and the pure function a consumer reads. This module
holds the half that cannot live in the format tier, because producing a record means compiling.
Why a measurement and not a hand-kept map. The map was the first thing everybody proposed and the first thing rejected: it is the defect wearing a public name. Five of the six RM104–RM111 fixes were a derived value restated by hand, and a per-release map of what a release changed is that shape exactly. So the sweep measures, the gate refuses a measured change nobody declared, and the declaration is forced by the measurement rather than remembered by the author.
What it compares. Two trees of compiled output — one produced by the previous release, one by
this one, from the same spec inputs. Feeding each side its own tree's reference_examples/ would
measure spec drift as compiler drift; the discipline is one spec root, two compilers. The comparison
itself needs only this tier: a manifest is JSON and a parquet schema is a polars read.
Scope, and it is narrower than "what a release changed". Compiler-derived outputs only. The
enricher's sidecars are unmeasured, which is not the same claim as unchanged, and
release_records says so in the record it publishes.
Run it in the release sequence, not as an ordinary test: it needs the previous release actually installed. COMPILER.md § The release-record sweep carries the exact command sequence.
ModuleOutput
dataclass
¶
One compiled module as the sweep reads it: the manifest, and every parquet's column types.
ModuleDelta
dataclass
¶
ModuleDelta(
name: str,
axes: dict[str, bool],
manifest_fields: tuple[str, ...],
warnings_added: tuple[str, ...],
warnings_removed: tuple[str, ...],
carried_added: tuple[str, ...] = (),
actionable_added: tuple[str, ...] = (),
toolchain_differs: bool | None = None,
)
What moved for one module across the interval, per axis.
bytes_reason
property
¶
TOOLCHAIN_DIFFERS when parquet_bytes moved and the recorded toolchains differ, else None.
The reason names a co-occurring difference, not a proven cause: across a sweep the compiler version differs too by construction, so a release that moves polars and changes the compiler labels every moved module this way. With no movement there is no reason, and an unrecorded toolchain explains nothing, so that case is the plain movement it always was.
SweepMeasurement
dataclass
¶
SweepMeasurement(
before: str,
after: str,
axes: dict[str, bool],
manifest_fields: tuple[str, ...],
modules: tuple[str, ...],
only_before: tuple[str, ...],
only_after: tuple[str, ...],
per_module: tuple[ModuleDelta, ...],
build_failures: dict[str, str],
before_toolchain: dict[str, str] | None = None,
after_toolchain: dict[str, str] | None = None,
before_digests: dict[str, str | None] = dict(),
after_digests: dict[str, str | None] = dict(),
)
The union across every module the sweep could measure, plus the ones it could not.
A module present on one side only is a module this sweep says nothing about, and rolling it
silently into an all-False result would be the silence the record exists to replace. Which
side it is missing from is a different fact about a different release (RM139), so the two are
kept apart and unmeasured is their union rather than a third stored field:
only_before— the previous release compiled it and this one does not. That is a regression in the release being cut, and it is fatal however it is declared. A stale reused BEFORE directory holding a module the spec root no longer has lands here too, which is the fail-safe direction; the runbook's answer is a fresh tree every time.only_after— the previous release produced no output for it, so no before state exists to compare against. A module added since the last release, or one whose spec now uses a column the previous release refuses underextra="forbid", which happens in every minor that adds an authored column and exercises it in the corpus. Nothing failed and nothing is measurable, so the record names it inunmeasuredand the gate checks that list for equality.
is_toolchain_sweep
property
¶
One compiler on both sides and two recorded toolchains that differ: RM352's backing sweep.
unmeasured
property
¶
Every module measured on one side only, derived rather than stored beside the two halves.
moved_under_other_toolchain
property
¶
The modules whose bytes_reason is TOOLCHAIN_DIFFERS (RM342): bytes moved, toolchains differ.
Named apart from ModuleDelta.toolchain_differs, which is the tri-state toolchain comparison
alone, whether or not anything moved (@flag-means-same).
moved_counts
property
¶
How many of the measured modules moved on each axis — the denominator is modules.
Computed and published, not computed and discarded: an axis that reads False on the
record is a measured zero, and a zero with no denominator beside it is the shape a consumer
cannot check.
evidence
property
¶
The sentence a ReleaseRecord.evidence carries — what was compiled, and what was not.
as_record ¶
The measured half of a record, ready for the declared half to be written onto it.
Produces a record with an empty declared list on purpose: the gate then refuses it
until somebody says whether each moved value was wrong or merely absent, which is the whole
mechanism that keeps this from becoming a map somebody maintains by memory.
Refuses to stamp an interval this measurement did not take. A record whose version and
previous disagree with its own evidence sentence is the hand-kept map again, one field
smaller.
Source code in compiler/src/just_dna_compiler/sweep.py
read_output ¶
Read one compiled module directory — manifest.json plus every parquet beside it.
Source code in compiler/src/just_dna_compiler/sweep.py
read_outputs ¶
Every compiled module under root, keyed by directory name.
Source code in compiler/src/just_dna_compiler/sweep.py
build_outputs ¶
Compile every spec under spec_root into out_root/<name>/ with THIS compiler, and what broke.
Discovery rather than a list, the same rule test_reference_examples_roundtrip follows: a spec
added without being added here is a spec nobody sweeps, and adding one is precisely when a new
shape arrives.
The second half of the pair is the compiler's own errors per spec that did not compile. Returned
rather than only logged (RM139): the module lands in only_before, which is fatal, and a fatal
finding that names the error is the difference between a release an operator can fix and one they
have to go looking for.
Source code in compiler/src/just_dna_compiler/sweep.py
changed_manifest_fields ¶
The published manifest paths that moved, with EXCLUDED_MANIFEST_FIELDS never among them.
Source code in compiler/src/just_dna_compiler/sweep.py
compare_module ¶
The per-axis movement for one module.
The warnings axis is computed apart and reported apart, and it is not folded into
manifest_fields however published compilation.warnings is. A release that reworks the warning
channel would otherwise report a manifest field changed on every module in a catalogue, and a
registry acting on that mints an immutable PATCH for a message change.
RM131's carried split is now that discriminator, and it is reported rather than acted on.
axes["warnings"] still fires on any change to the set — narrowing it here would make a
published axis mean something different from what every record already written claims about it,
and the axis is outside RECOMPILE_DRIVING_AXES anyway, so nothing keys a rebuild on it. What the
split buys is the reading: carried_added is a finding no author can clear appearing, usually
this repository saying more about a limit it always had; actionable_added is work arriving at
somebody's door. A manifest with no carried field reports every addition as actionable, which
is the safe direction — it never tells a reader that a finding they could fix is unfixable.
Source code in compiler/src/just_dna_compiler/sweep.py
compare_outputs ¶
compare_outputs(
before: dict[str, ModuleOutput],
after: dict[str, ModuleOutput],
build_failures: dict[str, str] | None = None,
) -> SweepMeasurement
Union the per-module deltas into the measurement one release record carries.
build_failures is what build_outputs saw refusing to compile under THIS release, so a module
in only_before can be reported with the compiler's own error rather than as a bare absence.
Source code in compiler/src/just_dna_compiler/sweep.py
gate_findings ¶
gate_findings(
measurement: SweepMeasurement,
version: str,
records: dict[str, ReleaseRecord] | None = None,
toolchain_measurement: SweepMeasurement | None = None,
) -> tuple[list[str], list[str]]
(findings, notes) — the release gate. A non-empty findings fails the release.
A finding is a measured movement no declaration covers, or a sweep that did not measure what
the caller thinks it did. The second half is not decoration: the likeliest operator error in
the documented sequence is running the sweep before uv sync has propagated the version bump,
which builds the AFTER tree with the previous release still installed. Both sides are then one
compiler, every axis reads False, and a gate checking only the lower end of the interval passes
a release it never measured — a false green in the one mechanism the whole item rests on. So the
interval's upper end is checked against the version being gated, a degenerate interval is
refused outright, and a module that compiled on one side only fails rather than vanishing into an
all-False result over its surviving neighbours.
Which side a module is missing from decides which of those it is (RM139). One side only was
read as a compile failed until the first real cut, where it was wrong: RM70 put an optional
column on pharm_variants.csv, one reference example uses it, and 0.6.6 refuses that spec under
extra="forbid". Nothing failed — the previous release simply cannot produce a before state, and
that recurs in every minor that adds an authored column and exercises it in the corpus, as it does
for any example added since the last release. So:
- a module in the BEFORE tree and not the AFTER one is a regression in the release being gated and fails unconditionally, with the compiler's own error beside it where the AFTER side was built here;
- a module in the AFTER tree and not the BEFORE one fails unless the record's
unmeasurednames it, and a record naming one the sweep did measure is reported as a note.
That is the same forcing shape as declared, not an exemption bolted beside it: as_record
mints the measured half and the gate refuses until the author commits it to the published record.
It cannot silence anything, because it reaches neither direction that carries a measurement — a
movement on a measured module still gates however unmeasured reads, and a regression is fatal
however it is listed.
A note is the other direction — a declaration this sweep did not see move — and it is deliberately not a failure: the reference corpus is sixteen modules and a real correction can land on a shape none of them has.
A toolchain declaration on parquet_bytes is backed, never trusted (RM352).
toolchain_measurement is the second sweep that backs it (this release's compiler on both sides,
only the toolchain varied), and _toolchain_findings lists what it must show. The degenerate-
interval check is deliberately not applied to it: one compiler on both sides is that sweep's point.
The gate runs in the bump → uv sync → tag sequence rather than as an ordinary test, because it
needs the previous release actually installed.
Source code in compiler/src/just_dna_compiler/sweep.py
637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 | |
measurement_json ¶
The measurement as plain JSON, so a release script can diff or archive it.