Build cache tiers (Linux cross lane)

Build cache tiers (Linux cross lane)#

STATUS 2026-08-23: § 4 (S3 / T4) is DESIGN ONLY — that code change was written, reviewed and REVERTED. The first implementation reintroduced the literal ${tag}-buildcache cache ref, which verify-critical-fixes.sh has forbidden since fix7 (2026-07) precisely because that ref was self-defeating; the preflight gate failed on it immediately. It was also inert by default and had no regression coverage, so a future “just turn it on” edit would have flipped ~12-13 GB of extra registry traffic per run with nothing asserting the behaviour. § 4 is therefore the design and the constraints any implementation must satisfy — read the flake-recovery interaction below before writing code. § 5 is not covered by this banner: the ccache→sccache switch it describes is shipped and running.

Audience: anyone about to “add a registry cache”, “turn on mode=max”, “wire sccache”, or free disk by deleting a cache directory.

This is the why companion to the one-screen map in linux-build-basics.md § Caching Layers. That table says what is cached where; this document says what each tier does NOT cover, what destroys it, and what it costs to replace it — the three questions that produced an 18-hour cold rebuild (backlog S3, BIT 2026-08-21).

Scope: the Linux cross lane (build-cross-chain.sh → 01-core/cross-stage-build.sh). The Windows lane keeps its own cache story in its own docs; nothing here proposes work there.

1. The tiers#

#	Tier	Wired at	Covers	Stored in	Destroyed by
T0	BuildKit layer/history cache	implicit (the worker)	every vertex, including intermediate `FROM … AS x` stages	the buildkitd store	`nerdctl system prune`; buildkitd GC (`gckeepstorage` in `~/.config/buildkit/buildkitd.toml`)
T1	exec cache mounts (ccache, sccache, apt, cargo, uv, pip, source trees)	`--mount=type=cache,…` in the Dockerfiles	compiler/package output at translation-unit / file granularity	the buildkitd store (`exec.cachemount` records)	`nerdctl system prune` (35 → 1 records observed 2026-08-21). Survives `linux/host-config/prune-safe.sh` and a daemon restart — both proven repeatedly
T2	local stage cache export	`cross-stage-build.sh:193-197` (`--cache-to type=local,…,mode=max`)	every vertex of that stage’s Dockerfile	`~/.cache/kata-buildcache/<tag-slug>/` — outside the buildkit store	our own disk guard (LRU below `CROSS_DISK_GUARD_GB`=40 G free, plus the `CROSS_CACHE_MAX_GB`=250 G total cap, `build-cross-chain.sh:523-614`); the preflight, watchdog and lane-entry trims (§ 3.1, § 3.2); the disk-preflight’s `rm -rf` hint (`build-cross-chain.sh:467`); `CROSS_NO_LOCAL_CACHE_EXPORT=1` stops new writes
T3	inline registry cache	`cross-stage-build.sh:200-205` (`--cache-to type=inline` + `--cache-from type=registry,ref=<tag>`)	only the final image’s own layers (`mode=min`)	inside the pushed image config — no separate blob, so no ghcr 400	a re-push of the tag; `ghcr-cleanup.yml` then deletes the untagged old version (every tagged version is kept; 14-day floor)
T4	per-stage registry cache, `mode=max`	not enabled, and DESIGN ONLY — the pilot knob `CROSS_REGISTRY_CACHE=max` was written and reverted with the rest of S3; no code reads it (`cross-stage-build.sh:178-207` adds the local `mode=max` export and, when pushing, `type=registry` cache-from + `type=inline` cache-to, and nothing else)	every vertex, off-host	a `<tag>-buildcache` ref on ghcr	see § 4 — the cost, not the mechanism, is the blocker

Two properties are worth internalising because they are counter-intuitive:

T2 is not in the buildkit store. nerdctl system prune does not touch ~/.cache/kata-buildcache. The things that empty it are ours: the disk guard, its trims and the preflight hint.
T0/T1 are in the same store, so the one command that wipes layers also wipes the compiler caches. That is why Standing rule 3 exists.

1.1 The cerbero state cachemount (CERB-CACHE)#

03-media/build/gstreamer/android/build-android-from-source.sh keeps its cerbero state in a T1 exec cachemount (linux/Dockerfile.android, the android-gstreamer RUN, target=/var/cache/cerbero) with an unusual job: it makes a failed build resumable. bootstrap + package is ONE RUN, so before this mount existed any failure inside it discarded the whole ~40-60 min bootstrap — wave5k/5l re-paid it on every freedesktop flap, while wave6b restarted WARM (HIT: resuming … 13G / 20G) and finished with zero fetch failures.

What lives there (cerbero’s home_dir): sources/local (downloaded tarballs and the git repos it checks out), sources/<pkg> (extracted + compiled build trees), build-tools/ (the host bootstrap prefix), rust/, dist/android_<arch>/ (the target prefix), and the cache-file.cache / build-tools.cache pickles recording which recipes are already built (build/cookbook.py: _cache_file()).

Four properties worth internalising:

A resume is not a re-verification. Read against the pinned cerbero 1.29.2: Oven._cook_recipe_step returns before it ever calls the step function when that step is already in the pickle (build/oven.py:511), and _cook_recipe skips the whole recipe once needs_build is false (:554). The only caller of Source.verify() — the sha256 against the recipe’s tarball_checksum — is the fetch step itself (build/source.py:366/378/458). Bytes reused out of sources/local are therefore never re-hashed: they buy speed, not trust.
The pickle only invalidates on recipe content. A recipe’s status is thrown away when its built_version changes, or when the content hash of the recipe file plus its patch files changes (build/cookbook.py:426-440; Recipe.get_checksum = files_checksum over the recipe’s own FILES, not over the tarball) — so the sed-patched recipes in that script (PKGCFG-MIRROR, soundtouch, glib libiconv) do invalidate themselves. Nothing in the pickle knows about the toolchain, which is why ANDROID_NDK_VERSION and ANDROID_API_LEVEL are part of the cachemount id: a pin bump must start a NEW store, because cerbero would happily resume a tree built against the old NDK.
The state dir is deliberately NOT /opt/cerbero. That path stays the git CHECKOUT: a mount target already exists when the RUN body starts, so the [ ! -d cerbero ] guard would skip the clone and git clone refuses a non-empty destination anyway. Separating them also un-collides cerbero’s read-only seed dir cached_sources (= <checkout>/sources) from the live tree.
The disk trade-off, with the number, because the host runs near-full. The prune at the end of the script is on the SUCCESS path ONLY: the package exists, so the extracted/compiled trees are dead weight and everything kept lives outside sources/ (the dist prefix, the build-tools prefix at config.py:1065, the two pickles) plus sources/local — that is local_sources (config.py:1146-1151, because home_dir is non-default), the tarballs and git repos that make the next run cheap and network-independent (FD-OUTAGE). A FAILED attempt keeps its whole tree — that is the entire point of the mount, and it costs up to ~10-15 GB per lane, i.e. ~30-45 GB with all three arch lanes sitting on failures. Retries reuse the same cachemount id and the same paths, so a failing lane overwrites in place rather than accumulating a tree per attempt. Pruning on ENTRY instead was considered and REJECTED as unsafe: cerbero records progress per STEP, so a recipe can be mid-flight with extract recorded and compile not (oven.py:511 then skips straight to a configure/compile on a directory the prune would have deleted) — that wedges the lane on every retry instead of bounding disk. CERBERO_CACHE_RESET=1 empties the CURRENT store on entry when a lane must be forced cold.

GENERATIONS are the unbounded axis: the id carries the NDK/API pins, so a pin bump starts a fresh store and ORPHANS the previous one, and nothing inside the build can reach it (a different cachemount id is simply not mounted into the RUN). linux/host-config/prune-safe.sh will not clean it up either — it prunes type==regular only and treats every cachemount as sacred by design (CACHE1: it aborts if the cachemount count drops). Reclaiming an orphan is a deliberate host-side act, e.g. a duration-bounded buildctl prune --filter type==exec.cachemount --keep-duration <t> chosen so the ccache/sccache mounts a live build keeps warm stay above the cutoff — check what it would remove before running it.

The id keys the lane on ${TARGET_ARCH}, never ${TARGETARCH}: the stage’s ARG TARGETARCH rebinds to BuildKit’s default, the --platform linux/amd64 that cross-stage-build.sh passes on every lane, so all lanes would share one store. The NDK/API pins expand from the android-sdk parent’s ENV, where an ARG-only name would expand empty and merge every generation into one store. The tarballs stay per lane rather than in a shared id, because cerbero downloads them non-atomically next to its live per-lane git repos, and the sharing=locked that would need holds its lock for the whole multi-hour RUN. ${ANDROID_LIB} is in the id so the RUN stays byte-identical across the five library stages while the four non-cerbero lanes get their own empty store.

1.2 The web-lane tool binaries (riscv-tools, 2026-09-23)#

Three T1 ids hold the from-source wasm-pack and flutter_rust_bridge_codegen (contract):

id	Stage	Holds
`web-lane-tools-cross-<target>`	android, `web-lane-tools`	cross-built binaries, one entry per key
`cargo-registry-web-lane-tools-<target>`	android, `web-lane-tools`	crate sources for that producer only, `sharing=locked`: cargo’s `.package-cache` lock sits outside the mount, as for the media lanes above
`web-lane-tools-bin-<arch>`	package, the setup RUN	native builds, so a pin reaches QEMU once per key

An entry is <tool>-<version>_rust-<rustc>_<triple>_<key12>/ holding the binary and a manifest whose first eight lines are the key text. A read accepts it only when that text equals this build’s exactly, the sha256 matches, and the binary passes the same gate a fresh build must pass; anything else is deleted with a WARN and rebuilt. Writes go to .tmp.<pid>.<tool> and are renamed into place, and each tool keeps its newest three entries, so a pin bump replaces an entry rather than orphaning one. The ids themselves are fixed.

The trust boundary is the one every cachemount here has: a binary and manifest forged consistently by someone who can write the buildkitd store pass. Since RUNTIME_NO_CACHE=1 does not touch a cachemount, WEB_LANE_TOOLS_CACHE=refresh is the way to rebuild past a good entry; off neither reads nor writes.

1.3 Cache-mount ids and source checkouts#

Dockerfile.media keys its cache ids three ways on purpose. ${TARGET_ARCH} (one store per lane) serves the sharing=locked apt mounts and the arch-specific ffmpeg SDK store; ${TARGETARCH} serves ccache, sccache, uv and pip, which are unlocked and content-addressed, so a lane collision is free while a split would triple the cold compile; and onnxruntime-web carries no arch, because its wasm output is arch-independent and the lanes serializing on that one locked RUN is the price of one emscripten compile instead of three. locked serializes every consumer of an id, concurrent lanes included, and private hands each concurrent consumer a new empty directory, so it never buys sharing. The cargo registry is the odd one: its sources are arch-independent, but the gstreamer RUN patches crates in place and cargo’s .package-cache lock sits at the CARGO_HOME root, outside the mounted subtrees, so concurrent containers on one id are unguarded.

Source checkouts are not cache-mounted, and a mount could not be populated: build-opencv.sh and build-ffmpeg.sh delete their source dir on cleanup, setup-gstreamer.sh builds in a PID-suffixed dir, and ONNX Runtime clones into the image path /opt/onnxruntime. A future source cache must key its id on the pinned version through an ARG declared in the same stage above the RUN (an out-of-scope variable expands to nothing and collapses every version onto one store), and still serves a frozen tip whenever the ref is a branch.

3. Failure modes and the recovery path that already exists#

Symptom	Cause	What the code does by itself	Operator knob
`DeadlineExceeded: failed to compute cache key: … httpReadSeeker … no active session`	ghcr cache import flake (killed 6 attempts across 2 lanes in one afternoon, 2026-08-18)	retry; after 2 such failures drop the registry cache-from + inline cache-to and finish on the local tier (`_cross_build_drop_registry_cache_after_flake`, cross-stage-build.sh)	`NO_CACHE_EXPORT=1` does the same thing up front
`error writing layer blob: 400 Bad Request` on cache push	ghcr rejecting an oversized `mode=max` cache blob (why T4 was removed in `4f27634`; `195285f` replaced it with T2+T3)	nothing — 400 is not in the transient class (`_cross_stage_push_error_is_transient`), so the stage fails	`NO_CACHE_EXPORT=1`
A stage fails after hours of completed work → T2 exports nothing	`--cache-to type=local` only materialises on a successful solve (~8 h of arm64 media work exported ZERO once)	S1 salvage: re-drive the same build per named `--target`, each a pure cache hit, so the export lands anyway (`cross-stage-build.sh:209-247`)	`SALVAGE_CACHE_EXPORT=0`
Disk below 40 G between stages	T2 growth (209 G measured today)	LRU-prune unprotected slugs; stages still to run in this chain are protected; if still short, stop writing new exports	`CROSS_DISK_GUARD_GB`, `CROSS_CACHE_MAX_GB`, `CROSS_NO_LOCAL_CACHE_EXPORT=1`
`could not read …/kata-buildcache/…`	empty slug dir, i.e. a clean miss	suppressed: cache-from is only added when `index.json` is non-empty (`cross-stage-build.sh:186-190`)	—
Chain refuses to start: `Insufficient disk: 19G free, ~180G recommended`	T2 grew 62 G → 110 G in ONE session (D4) and there is no room for the run	preflight trim: reclaim oldest-first from `~/.cache/kata-buildcache` up to the deficit, then re-measure and continue if it now fits (§ 3.1)	`CROSS_PREFLIGHT_TRIM=0`, `FORCE_LOW_DISK=1`
A stage fails while disk is nearly full, and the salvage then writes GBs more	S1 salvage re-drives up to 15 named media targets, exporting cache for stages that get rebuilt anyway (D5)	skip the salvage, with a warning naming the free space and the threshold (§ 3.1)	`SALVAGE_MIN_FREE_GB` (`0` = always salvage), `SALVAGE_CACHE_EXPORT=0`
Sessions die after 1–2 h of parallel load	BKD1 (buildkitd session rot)	—	restart buildkitd between rounds; cachemounts provably survive

The rule any new cache tier has to obey: it must be droppable on the flake path. The auto-recovery works by removing argument pairs from build_cmd before the retry; a tier that is not in that case statement keeps hammering a registry that is already timing out.

3.1 Preflight trim (D4) and the salvage disk gate (D5)#

Both landed 2026-08-31, after the r3 disk emergency forced a controlled chain stop at 19 G free.

Preflight trim. _chain_disk_preflight used to print rm -rf ${bc_dir}/* and abort, leaving the reclaim to a human. It now calls _disk_guard_trim_cache_export (01-core/disk-guard.sh) first:

target = the same need_gb the preflight already computes;
budget = need_gb - free_gb, i.e. the deficit only — the loop stops the moment either the target or the budget is reached, so it trims instead of emptying the directory. Without that bound a second process eating disk would turn the trim into rm -rf of every slug;
victims come from _disk_guard_pick_victim, so removal is strictly oldest-mtime-first and honours a protected list (the preflight passes none — nothing has run yet, and protecting every upcoming stage would protect everything);
every removal is logged with its size, plus a removed N slug(s), freed X GiB summary and the resulting free space. A silent reclaim is how people stop trusting a tool.

Then free space is re-measured. If it now clears need_gb the chain proceeds; otherwise the old hint-and-abort path runs unchanged. CROSS_PREFLIGHT_TRIM=0 skips the trim entirely and restores the pre-2026-08-31 behaviour.

Why this is safe. T2 is a cache export, not the buildkit store. Losing a slug costs export reuse for that stage and nothing else — no ccache, no sccache, no cachemount. The trim only ever rm -rfs directories directly under BUILDKIT_CACHE_DIR; it never invokes any prune. The buildkit store is reached only by the filtered buildctl prune of § 3.2.1, and by linux/host-config/prune-safe.sh by hand. nerdctl builder prune and nerdctl system prune are never run by anything in the chain.

Salvage disk gate. _cross_salvage_disk_ok gates the S1 salvage on free space: below SALVAGE_MIN_FREE_GB (default CROSS_DISK_GUARD_GB, i.e. 40 G) the salvage is skipped and says so, because those stages are rebuilt anyway and the export is the last thing that should run when disk is scarce. Unknown free space, a non-numeric threshold, or SALVAGE_MIN_FREE_GB=0 all keep the old always-salvage behaviour; SALVAGE_CACHE_EXPORT=0 still disables it outright.

Unit coverage: linux/scripts/tests/test-disk-guard.sh (oldest-first order, budget respected, no-op when ample, protected slugs, never aborts under set -euo pipefail) and linux/scripts/tests/test-cache-salvage-gate.sh.

3.2 In-stage disk watchdog and the runtime-lane gate (B2)#

Landed 2026-09-01, after the r2 rebuild died with no image was built on the amd64 wrapper export.

What actually happened. _chain_stage_disk_guard runs between stages only. The runtime lane is ONE stage that contains three per-arch wrapper builds, so nothing sampled disk inside it. The evidence from out/rebuild-20260901-r2/chain.log (483 MB) and its resource CSV:

time	free	what
02:55:18	88 G	android done → between-stage guard runs, 88 G > 40 G, silent no-op
02:55–03:23	88 → 4 G	amd64 wrapper build + export, no sampling at all
03:23:12	4 G	`level=fatal msg="no image was built"` → push fails 4× → `Arch amd64 failed during build`

grep -c disk-guard over the whole log returns 0. The last measured headroom was comfortable; the drain happened entirely inside the stage that followed.

The watchdog. _chain_disk_watch_start backgrounds _disk_guard_watch_loop for the duration of the runtime helper and _chain_disk_watch_stop reaps it on both the success and the failure path. Each tick:

logs [disk-watch] <N>G free on <dir> — an unconditional sample, so the chain log alone reconstructs the disk trajectory of a finished run. This is also what makes an external reclaim visible: the two large mid-run jumps in the r2 CSV (+138 G at 02:03, +271 G at 03:25) were operator prunes in another terminal and left no trace anywhere in the chain’s own log;
below CROSS_DISK_GUARD_GB it calls the same _disk_guard_trim_cache_export as § 3.1 — keep-floor and all — and then _disk_guard_reclaim_record, which emits exactly one [disk-reclaim] line per reclaim including the reclaim that freed nothing. That last case is a WARN: it is the one an operator must act on, and it is what silence used to look like.

It never calls nerdctl system prune / builder prune; those wipe the exec.cachemount records and cost hours (§ 7). When the trim leaves the disk still short, the watchdog falls back to a filtered buildkit prune instead of giving up — § 3.2.1.

The lane-entry gate. _chain_runtime_lane_disk_gate runs immediately before build-runtime-manifest.sh. It requires CROSS_RUNTIME_LANE_GB (default 120) × concurrency free, trims first, and then refuses. FORCE_LOW_DISK=1 or CROSS_RUNTIME_LANE_GB=0 override it.

Why concurrency and not arch count. 120 G is the transient cost of one wrapper build, measured off the r2 CSV. The chain runs the lane’s arches sequentially — it does not forward --parallel-archs to build-runtime-manifest.sh — so the requirement is 120 G once, not 360 G. Sizing it per-arch would have refused essentially every run on this 915 G host (218 G free, 26 G of trimmable cache export) — a gate that refuses everything gets switched off, and then it guards nothing.

The start-of-run preflight gained the same term as an advisory warning on top of its existing need_gb, for the same reason: at chain start the earlier stages’ consumption is still unknown, so the honest enforcement point is lane entry, where free space is actually measured against what is about to be spent. The r2 run would have been refused at 02:55 with 88 G — hours before it died, and after a trim that might have rescued it.

3.2.1 The buildkit-store fallback (DISK1)#

Landed 2026-09-04, after the 2026-09-03 runtime run died at 4 G free while 415 G sat in ~/.local/share/buildkit. The watchdog fired, trimmed ~/.cache/kata-buildcache, freed nothing — the dir held exactly three slugs and CROSS_TRIM_KEEP_SLUGS is 3 — and logged NOTHING was reclaimable. The manual rescue, PRUNE_KEEP_GB=120 linux/host-config/prune-safe.sh, freed 223 G in 36 s with all 97 cache-mount records surviving.

_disk_guard_buildkit_fallback is that rescue, wired after the cache-export trim at all three places the chain reclaims — _disk_guard_watch_once (the in-stage sampler, where 2026-09-03 actually bit), _chain_stage_disk_guard (between stages, just before it gives up and sets CROSS_NO_LOCAL_CACHE_EXPORT=1) and _chain_runtime_lane_disk_gate (lane entry, between the trim and the [disk-reclaim] record). The two chain gates are the ones that REFUSE a run, so a fallback unreachable from them could not save the run it was written for. It re-measures free space and returns immediately unless the trim left the path still under the threshold. Only then does it run

buildctl prune --filter type==regular --keep-storage <keep_gb * 1000>

the same filtered form prune-safe.sh uses. nerdctl builder prune -f is not an alternative and never becomes one: it has no --filter, so it deletes the exec.cachemount records too — ccache, sccache, uv, cargo, apt — and cost 1.5–2 h of cold LLVM on 2026-08-17 (§ 7). prune-safe.sh remains the operator-facing tool with the full before/after inventory; the guard is the in-chain, non-interactive subset.

Why 120 G of keep-storage, when the guard fires at 40 G free. The two numbers measure different things and do not compete. CROSS_DISK_GUARD_GB (40) is free space on the filesystem — the point at which a stage is about to ENOSPC. CROSS_BUILDKIT_KEEP_GB (120) is how much layer cache stays in the store — the reuse the remaining stages still need. A prune that clears the 40 G emergency while keeping 120 G of layers satisfies both, which is exactly what the 2026-09-03 rescue did on a 415 G store. Below ~100 G of retained layers the next stages start recompiling what they should have pulled from cache (the rebuild-disk-management note), so a non-zero CROSS_BUILDKIT_KEEP_GB under 100 is raised to 100 and logged. 0 is the explicit “reclaim everything regenerable” escape hatch and is left alone. When the store is smaller than the keep value the prune is a no-op and the log says it freed 0 G — honest, and the operator still gets the NOTHING was reclaimable warning from the reclaim record.

The keep value is layer cache, so the cache mounts are added to it. BuildKit’s --keep-storage bounds the whole store: cache/manager.go (v0.33.0) sums every unshared record, of every type, before the type filter is applied, and deletes filtered records until that sum fits. The type==regular filter never frees cache mounts or local sources, so a bare 120000 against 196 GB of cache mounts (2026-10-01) deletes every idle layer record. That is how PRUNE_KEEP_GB=100 prune-safe.sh once left 0.17 GB (CON53); the same call “kept” 91 GB later that day only because a running chain held those records. _disk_guard_regular_keep_mb now adds the unprunable records’ size, read from buildctl du --format '{{json .}}', so 120 means 120 GB of layers. A cache mount in use by a running build reports 0 B there, so mid-run the result is a floor. Proof: test-disk-guard.sh and test-prune-safe.sh.

Once per reclaim episode. The watchdog samples every CROSS_DISK_WATCH_SECS (120 s). After the first prune the store is already at --keep-storage, so a repeat would walk the whole store — 36 s of I/O during a running build — and free nothing. The second and later samples log already pruned once here and stop. An episode is one whole stage for the sampler, which is a backgrounded subshell and so latches on its own; for the two chain gates, which run in the orchestrator process and are hours apart, it is one invocation. Those call _disk_guard_reclaim_begin first — without it the store the base stage pruned would still be latched when the runtime lane refuses, which is the same “reclaim unreachable from the gate that refuses” defect one level down.

What it logs. Every arm is greppable as [disk-buildkit]:

line	when
`<N>G free < <T>G after the cache-export trim — pruning type==regular layer cache with --keep-storage <K>G`	the prune is about to run
`reclaimed <N>G of layer cache in <S>s (<before>G -> <after>G free; keep-storage <K>G)`	after it, always — including `reclaimed 0G`
`all <N> cache-mount record(s) survived`	the filter held
`cache-mount records dropped <N> -> <M>` (WARN)	it did not; compile caches are gone
`SKIP: no buildctl on PATH` (WARN)	tool missing
`SKIP: buildkit store unreachable` (WARN)	`buildctl du` will not answer
`already pruned once here`	the latch
`disabled (CROSS_BUILDKIT_PRUNE=0)`	the knob

The reclaimed GB also reaches _disk_guard_reclaim_record, so the single [disk-reclaim] line per reclaim now reads … freed X GiB + 223G of buildkit layer cache, and NOTHING was reclaimable is emitted only when both tiers came up empty. Pinning that give-up line as correct behaviour is what the old suite did.

Degradation. A missing buildctl, an unreachable socket, a non-numeric target or an unknown free-space reading all return 0 after one WARN, leaving exactly the pre-2026-09-04 behaviour. The fallback never aborts the stage it samples — it is a watchdog, and a watchdog that kills the build it watches is worse than the drain.

Unit coverage: the DISK1 section of linux/scripts/tests/test-disk-guard.sh (trim-first ordering, the filter and keep-storage argument, the keep floor, the once-latch, the survival count, both SKIP arms, and a structural assertion that disk-guard.sh does not name nerdctl or builder prune anywhere), the DISK2 cases in test-chain-lifecycle.sh (both gates reach the store before they refuse, and each opens its own episode), plus the disk-guard.* mutations.

What DISK2 left open, measured on the live store 2026-09-05 (buildkit v0.31.2, 219.13 GB total / 187.99 GB reclaimable, no build running):

--keep-storage is documented by buildctl prune --help as MB, so the guard’s keep_gb * 1000 really is ~120 GB. The convention was inherited from what worked in anger; it is now checked against the installed flag.
the readiness probe is not the expensive half: bare buildctl du and du --filter type==exec.cachemount both return in under a second on that store, and the second reports the same 97 records the 2026-09-03 rescue saw. The 36 s belonged to the prune.
BUILDKIT_HOST defaulting to unix:///run/user/$(id -u)/buildkit/buildkitd.sock resolves for the chain’s own uid on this rootless host — buildctl du answers rc 0 — so _disk_guard_buildkit_ready passes rather than logging its SKIP.

Still unanswered, and still needing a real run: whether a prune issued while a build holds records open reclaims anything like the idle number, and whether 120 G of retained layers avoids recompile churn afterwards. The 2026-09-04 --only runtime run entered with 275 G free and never crossed 40 G, so it sampled ([disk-watch] down to 106 G) and reclaimed nothing.

3.2.2 The image-store lever (DISK3)#

Landed 2026-09-07, after the 2026-09-05 rebuild reported this at 28 G free:

[INFO] [disk-trim]     removed 0 slug(s), freed 0.0 GiB; 28G free now
[INFO] [disk-buildkit] already pruned once here -- the store is at keep-storage
[WARN] [disk-reclaim]  in-stage: NOTHING was reclaimable (28G -> 28G free)

It was right that it could not free anything and wrong that nothing was reclaimable. At that moment ~/.local/share/containerd held 295 GB, including three cross-android-* stage images from a PREVIOUS run at 41.5 / 41.8 / 38.0 GB. disk-guard.sh contained no image listing at all: it knew its own cache-export slugs and BuildKit, and BuildKit was already at --keep-storage, so its two levers were spent while its third, larger one was invisible to it. That run needed four manual rescues.

The ordering constraint is the whole design, and it is not a preference. buildctl prune never touches the containerd image store and is safe while a stage runs — that is why the in-stage sampler may pull it. Removing IMAGES is safe only when nothing is in flight: nerdctl image prune -f killed the arm64 runtime lane on 2026-09-06 while the chain was unpacking overlayfs@sha256:…, the OCI stage handoff, and the prune took a blob out from under it (content digest sha256:d694…: not found). The same command had worked several times earlier that day, which is luck, not safety.

So _disk_guard_image_store_fallback takes a stage_in_flight argument and refuses BY NAME when it is 1. _disk_guard_watch_once — the sampler, which runs DURING a stage by definition — passes 1 and therefore never removes anything; the two gates that run between runs (_chain_stage_disk_guard after the arch loop has joined, and _chain_runtime_lane_disk_gate before any wrapper build starts) pass 0. CROSS_IMAGE_PRUNE=0 disables the lever entirely.

Two steps, in risk order. First nerdctl image prune with no -a: dangling images only, zero risk, and 20 GB on its own in the run that found this. Then this chain’s own cross-<stage>-<arch> tags that the rest of the run does not need. Protected are the stages still to build AND the one just completed, whose image is the next stage’s parent under the local OCI handoff; everything else in that shape is re-pullable from ghcr by the digest each stage pins. Nothing else is a candidate — not ubuntu:24.04, not an untagged image (that is the dangling prune’s business), and never -a, which would take the parents this run pinned.

Size is not the metric — unique layers are. Deleting the three cross-sdk-* images (80 GB by nerdctl images) freed zero bytes in that run, because every layer they hold is also held by the cross-android-* images built on top of them; the previous release’s latest-cross* freed 113 GB from a similar nominal size, because its layers were nobody else’s. A guard that picks the biggest tags does nothing, so this one measures free space after EACH removal and stops the moment the target is reached. Each removal logs what it actually freed. A tag still listed after its own rmi joins the protected set rather than staying the head of the list forever — the same spin the cache-export victim loop guards against.

The give-up warning names the store it did not look in. When nothing was reclaimable, [disk-reclaim] now adds how many tagged images the store holds and says stop the lane, then reclaim — because “NOTHING was reclaimable” read like an environment limit when it was a coverage gap. linux/host-config/prune-safe.sh remains the operator-facing tool for the BuildKit half.

3.3 Runtime-failure summary (B3)#

When one arch fails, set -e aborts build-runtime-manifest.sh inside run_parallel_arch_loop, and every gate downstream of that loop is skipped: the wrapper content gate, verify-shipped-wrapper.sh, the runtime-image smoke (and with it assert_pinned_versions), and the manifest coherence / completeness / freshness gates. In r2 the run ended with a bare [ERROR] runtime stage failed; grep -c returns 0 for all of them.

_chain_runtime_failure_report now runs on that failure path. It reads each wrapper tag’s own run-id stamp (ancestry_recorded_run_id, the same provenance the coherence gate uses) and classifies the arch as built-this-run, stale or missing, then:

names the arches with no image from this run;
names the gates that were therefore skipped — but only in that case. If every arch carries this run’s id the failure is at or after the gates, and claiming a skip would be a lie;
writes both into chain-status.json as arch_outcomes and gates_not_run, so a later --manifest-only repair cannot assemble an index believing everything was checked. A green run’s file is unchanged — the keys appear only when a failure recorded them.

Unit coverage for § 3.2–3.3: test-disk-guard.sh (sampling on every tick, the reclaim record, the no-op-reclaim warning, the loop actually looping, the concurrency sizing) and test-chain-lifecycle.sh (the lane gate going red at 88 G, its three overrides, per-arch classification, the “do not claim a skip” branch, the JSON fields, and call-site wiring assertions for all of it).

4. S3 — per-stage registry cache refs (`mode=max`)#

4.1 What it would buy#

Exactly the gap in § 2: intermediate stage layers, off-host, surviving a store wipe. Had it been in place on 2026-08-21 the relaunch would have been a fast-forward instead of ~18 h.

4.2 What it costs — measured, not estimated#

The local mode=max exports are the same content a registry mode=max ref would hold, in the same compressed form, so the disk sizes are a direct proxy for the bytes on the wire (± the export compression setting).

Measured on this host, 2026-08-23 (du -sh ~/.cache/kata-buildcache/*):

slug	size
`cross-media-amd64`	52 G
`cross-media-arm64`	49 G
`cross-media-riscv64`	34 G
`cross-android-amd64`	27 G
`cross-android-arm64`	26 G
`cross-android-riscv64`	24 G
total	209 G

The measured uplink is ~4–5 MB/s (the PUSH1 note at cross-stage-build.sh:161-162; PUSH1 also measured a media image push at ~10 min with zstd, i.e. ~2.7 GB of new layers).

At 4.5 MB/s:

what	bytes	added push time
one media arch	~50 G	~3.1 h
all three media arches	135 G	~8.5 h
all three android arches	77 G	~4.9 h
everything	212 G	~12–13.5 h per chain run

Verdict: a blanket per-stage mode=max export to ghcr is net negative here by roughly an order of magnitude. It would add ~12 h to every chain run to insure against an ~18 h loss that has happened twice in a fortnight. On a 100 Mbit uplink the same design would be obviously correct — the numbers, not the idea, are what disqualify it. Re-run the table above before re-opening S3 if the uplink ever changes.

Secondary costs, for completeness:

Registry storage. When this was costed, ghcr-cleanup.yml kept 3 versions per tag, so 11 stage tags × 3 generations of cache was up to ~600 G of cache blobs retained. Since 2026-09-08 it keeps the tagged version and deletes untagged ones after 14 days (ghcr-prune-package.sh). Its referential-integrity pass (added 2026-08-11) resolves kept tags’ manifests, so it will not orphan a live cache ref — but the package listing doubles in size.
Failure surface. A cache export that 400s fails the stage outright (§ 3), and it does so after the image already pushed.

4.3 Where it does pay: the cheap shared stages#

The value ratio is recompute_time / push_time, and it inverts by stage:

stage	cache size	push at 4.5 MB/s	cold recompute	verdict
`compiler` (GCC + LLVM)	not currently on disk; the archive recorded 12 G-class `sdk` slugs	~45–60 min for a 12–16 G cache	LLVM alone was ~50 min warm and projected 11 h cold (PAR4 note, CHANGELOG 2026-08-21)	plausible — and it is built once, not per arch
`sdk`	~12 G (archive)	~45 min	~1 h class	marginal
`media`, `android`	24–52 G per arch	1.5–3.3 h per arch	1.5–2 h per framework	no

Read that table with the right arithmetic: the push cost is paid on every run, the recompute saving only on the rare run that starts with the local tiers gone. For media the premium is ~3 h per arch per run against a payout of roughly one cold media rebuild (~6 h per arch, from the 18 h / 3 arches figure) — it only breaks even if the local tier is lost more often than every other run. It is not; it has been lost twice in a fortnight.

So if S3 is implemented at all, it is implemented for compiler (and maybe sdk) only, never chain-wide.

4.4 The shape that dodges the cost entirely#

A re-drive of an identical build is a pure cache-hit solve — S1 salvage already depends on that property and it is proven in production. Therefore the cache export does not have to sit on the build’s critical path:

After a green chain, re-drive the stage you want insured with the registry cache export configured. Every vertex hits, nothing recompiles, and the multi-hour upload happens in a window you chose, can interrupt, and whose failure cannot kill a build that already succeeded.

That is the recommended S3 shape: an out-of-band, opt-in “cache snapshot” pass, not a flag on every stage build. It also composes with the flake recovery for free, because it is not on the retry path at all.

Cheaper still, and worth doing first because it costs zero bytes:

Fix the preflight hint. build-cross-chain.sh:467 tells the operator to rm -rf ${bc_dir}/* to reclaim space, describing T2 as “regenerable” — it is, at ~18 h. The hint should point at the LRU path (the disk guard’s own ordering, or prune-safe.sh), not at the wholesale delete of the only tier that holds intermediates. (Filed, not fixed here: that file is outside this change’s scope.)
Never nerdctl system prune — Standing rule 3, already sharpened.

4.5 The pilot knob, and the rules it must obey#

CROSS_REGISTRY_CACHE=max is DESIGN ONLY — read the STATUS banner at the top. It was written so the experiment above could be run without editing code mid-wave, and it was reverted along with the rest of S3; grep -rn CROSS_REGISTRY_CACHE over the tree matches these docs and nothing else, so setting it today is a silent no-op. What follows is the shape a re-implementation has to take. In the reverted version it was off by default and the default path was byte-identical to what shipped before it existed (verified by dry-running the old and new function side by side and diffing the assembled command) — keep that property.

When set to max the stage would additionally get:

--cache-from type=registry,ref=<tag>-buildcache
--cache-to   type=registry,ref=<tag>-buildcache,mode=max,image-manifest=true,oci-mediatypes=true

image-manifest=true,oci-mediatypes=true is there because the cache is then exported as a plain OCI image manifest rather than the index shape that motivated the original ghcr 400. That is a hypothesis this repo has not yet tested — proving or disproving it is the first job of the pilot.

Rules the reverted implementation satisfied, and that any future one must keep:

It composes with the flake auto-recovery. The DeadlineExceeded/ httpReadSeeker handler must strip --cache-to type=registry* alongside the registry cache-from and the inline cache-to, so two import flakes still degrade the build to local-only instead of leaving a multi-GB export pointed at a registry that is timing out. The reverted implementation was verified by driving the real _cross_stage_build_impl through four failing attempts: attempts 1–2 carried all four registry/inline arguments, attempts 3–4 only the local --cache-to. The handler that ships today strips only --cache-from type=registry* and --cache-to type=inline* (cross-stage-build.sh:255-281) — because there is no registry export left to strip. Re-adding one means re-adding its case arm.
NO_CACHE_EXPORT=1 still turns everything registry-facing off, because the knob lived inside that guard.
A missing -buildcache ref on the first run is a miss, not a fatal. In-repo evidence: 195285f’s post-mortem of the never-written -buildcache ref — the importer failed and the stages rebuilt, the solve continued.

Pilot protocol (do not skip a step; this is the item that has burned this repo before):

Pick compiler, and use build-cross-stage.sh — not a full chain. A cache-export 400 is non-transient and will fail the stage.
Run 1 (cold ref): re-implement the knob, then set CROSS_REGISTRY_CACHE=max. Confirm the build accepts the flags (nerdctl/buildkit here must support image-manifest), that the -buildcache ref appears in the ghcr package list, and record the wall time the export added.
Run 2 (warm): same command. Confirm the cache-from actually resolves and the solve reports cache hits for intermediate vertices, not just final layers.
Run 3 (the real test): move the local slug aside (mv, do not delete) and re-run. If the intermediates still fast-forward, T4 works and S3 is proven for that stage. If they do not, the tier bought nothing and the item closes.
Only after 3 green runs, and only for the stages in the § 4.3 “plausible” row, consider making it a default.

In the reverted implementation, revert was CROSS_REGISTRY_CACHE=inline — or unsetting it, or NO_CACHE_EXPORT=1; any value other than the literal max gave exactly the shipped behaviour, and nothing else in the chain read it. That is the property to preserve in a re-implementation, not a knob to reach for today.

5. SCC1 — the ccache/sccache hybrid#

5.1 The division of labour — REVERSED 2026-08-26#

This section previously read “ccache owns C/C++, sccache owns rustc + the GPU compilers … a full switch to sccache was rejected by the owner on 2026-08-17”. The owner reversed that on 2026-08-26 and the C/C++ switch is implemented and built (5d94a37, c42091e, d5bafe8). The old text is kept nowhere but in git history, because a design doc that states a superseded decision reads authoritative and gets followed.

sccache is the C/C++ compiler cache. ccache is the automatic fallback. There is no per-language split any more, and every site resolves through ONE resolver: compiler_cache_launcher() (01-core/common.sh:459-483) direct from the 02-toolchain GCC/LLVM builds and the CMake/onnxruntime/wheelhouse call sites, and via _resolve_compiler_cache_launcher() on the media lane (01-core/compiler-cache.sh:85-106 — the single seam that forwards to the common.sh resolver when 01-core is loaded, 2026-08-30 backlog F2). Either way: the guarded launcher if sccache’s server answers, else ccache, else build uncached.

It is never bare sccache in a stage that mounts 01-core: each site starts at sccache and upgrades to sccache-launcher.sh as soon as one is executable, keeping the bare name only where the helper is absent (common.sh:467-474, compiler-cache.sh:97-99). The gate at verify-critical-fixes.sh:98-119 is what stops the hardcoded bare form coming back — that is how the first cut shipped inert.

All three load-bearing points of the old rationale still hold — the last one only after a detour through the config file:

Still true: ccache cannot wrap rustc, nvcc or hipcc; sccache can.
Still true, and now the operating constraint: sccache HARD-FAILS where ccache simply execs the compiler. That is not limited to unidentifiable compilers, and it bit hard during the switch:

The TryCompile trap (root cause, 2026-08-26). CMake creates a CMakeScratch/TryCompile-XXXX directory, compiles a probe in it, and DELETES it. sccache then spawns the compiler with that directory as the working directory — and spawning a process whose cwd no longer exists fails with ENOENT. Measured directly: spawning /bin/true with cwd set to a removed directory returns errno 2, exactly what sccache reports. The same trap in preprocessor-cache mode surfaces earlier, as while hashing the input file, because that mode re-reads the source after the compile.

It killed the media stage three times, MOVING between OpenCV, onnxruntime and litert depending on which probe lost the race — and it does not reproduce against directories that persist. Nine isolated attempts came back clean before the pattern was read correctly.

The answer is 01-core/sccache-launcher.sh: sccache runs for every compile, and only its OWN fatal errors (“sccache: encountered fatal error”) fall through to running the compiler directly. A real compile error is passed through untouched — retrying blindly would hide genuine failures. In the run that proved it, the guard fired 1451 times with ZERO build aborts.

ccache stays installed and mounted as the deeper fallback, and probe-sccache.sh exists — it asserts, per compiler SHAPE this chain feeds a launcher, that the compile survives AND that cache activity was recorded.
True again, and on purpose: “sccache’s C/C++ path always preprocesses”. The base image does bake use_preprocessor_cache_mode = true plus file_stat_matches / use_ctime_for_stat into /etc/sccache/config.toml (Dockerfile.base:115-142, reached via SCCACHE_CONF, :102) — those knobs have NO environment-variable path in sccache, which is why the config file exists at all. (It is not read while SCCACHE_DIR is set, which is always: § The shipped image’s cache dirs.) And the runtime turns the mode off: both entry points export SCCACHE_DIRECT=false (01-core/common.sh:419-433, 01-core/compiler-cache.sh:132), and the environment variable wins over the config file. That is the TryCompile trap seen from the other end — preprocessor-cache mode re-reads the INPUT FILE after the compile in order to store the entry, so a deleted scratch dir surfaces as while hashing the input file and is fatal. It killed OpenCV’s compiler test, then onnxruntime’s, three times running. Effective state today: preprocessor-cache mode is OFF and every TU is preprocessed. The cost is hit rate, not correctness; set SCCACHE_DIRECT=true to re-test it once the ephemeral-dir interaction is understood. ignore_time_macros stays false on purpose: ccache’s sloppiness list included it, but that trades correctness for hit rate and this chain ships compilers.

Two knobs have no counterpart and must not be looked for: there is no sccache -M (the cap is SCCACHE_CACHE_SIZE), and CCACHE_COMPILERCHECK=content needs none — sccache does not key on mtime+size the way ccache does by default, so the invalidation that setting prevents does not arise.

Rust rejoined this on 2026-08-27; nvcc did not. The hard clear that used to sit in build-gstreamer-monorepo.sh is gone. RUSTC_WRAPPER resolves to the same guarded launcher from two places: setup_sccache exports it (01-core/compiler-cache.sh:172-210), and build_gstreamer_monorepo installs it for any process where the variable was never set at all (build-gstreamer-monorepo.sh:649-659). What made that safe is the UDS fix in § 5.4 (compiler-cache.sh:62-76, common.sh:409-418), not optimism — the deaths at 99 % were the wrong server answering, and the launcher is the second belt that turns a remaining sccache hiccup into lost hits instead of a lost build. To go back to uncached Rust, export RUSTC_WRAPPER=""; that is what Dockerfile.toolchain:66 and Dockerfile.package:262 do, and it holds for every stage that never calls setup_sccache. Where setup_sccache does run it overwrites the empty value (compiler-cache.sh:199), so the off switch on that lane is USE_SCCACHE=0 — which also drops C/C++ back to ccache (compiler-cache.sh:136-148). nvcc and hipcc stay untouched (§ 5.4), and the Windows lane records that released sccache breaks the build around them.

5.2 Does the current setup double-cache or fight ccache? No — verified#

The precedence is explicit and correct:

setup_ccache (01-core/compiler-cache.sh:108-170) sets CMAKE_C/CXX_COMPILER_LAUNCHER to the guarded launcher (01-core/sccache-launcher.sh, resolved via _resolve_compiler_cache_launcher, :85-106) when USE_SCCACHE is not disabled, sccache is on PATH, and its server answers --show-stats; otherwise it falls back to ccache and says why. (Before 2026-08-26 it set ccache unconditionally. Its first cut at the switch hardcoded bare sccache here, which shipped inert; verify-critical-fixes.sh:98-119 now fails any launcher in this file pointed at bare sccache.)
setup_sccache (:172-210) sets those two launchers only if they are empty (:201), and otherwise touches only RUSTC_WRAPPER (:199) — resolved the same way (:193-198): the guarded launcher where it is mounted, bare sccache only where it is not.
media_common_init (03-media/core/common.sh:125-143) always runs setup_ccache first, then setup_sccache only under ENABLE_SCCACHE_RUST=1.

So on the media lane there is exactly one C/C++ launcher, and it is the guarded sccache launcher; ccache is used only when sccache’s server does not answer. The LLVM cross build picks the same way round (02-toolchain/llvm-cross.sh:394-399, commented “preference INVERTED”: it asks compiler_cache_launcher (01-core/common.sh:459-483), i.e. sccache first, ccache only as the fallback).

The cache mounts do not collide either: /var/cache/ccache and /var/cache/sccache are separate ids. Their ${TARGETARCH} (not ${TARGET_ARCH}) suffix is deliberate and documented at Dockerfile.media:26-31 — they are unlocked and content-addressed, so a lane collision is free while splitting them would triple the cold compile. Do not “align” those ids with the apt ones.

5.3 Three things that are not what the comments claim#

Found while reading for this document; all filed, none fixed here (the files are outside this change’s scope):

The ENABLE_SCCACHE_RUST gate is not the single control point it says it is — and since 2026-08-27 it barely controls anything. setup-gstreamer.sh:50 calls setup_sccache unconditionally, and USE_SCCACHE defaults to true (compiler-cache.sh:19, Dockerfile.media ARG USE_SCCACHE=true). So in that process RUSTC_WRAPPER is exported even with ENABLE_SCCACHE_RUST=0, and an sccache server is started. The counterweight used to be an explicit export RUSTC_WRAPPER="" inside build_gstreamer_monorepo; it is gone. That block is now the opposite — it installs the guarded launcher when RUSTC_WRAPPER is unset (build-gstreamer-monorepo.sh:647-659, sourced into the same process at setup-gstreamer.sh:557-562 and only called at :645). The monorepo’s Rust is therefore cached deliberately, and the gate does not gate it. What the gate still decides is one thing: media_common_init runs setup_sccache only under ENABLE_SCCACHE_RUST=1 (03-media/core/common.sh:136-143), and that is the path install-rice-proto.sh’s cargo cinstall takes — a different process (build-gstreamer-stage.sh:112, before setup-gstreamer.sh is invoked at :147), which therefore still inherits the image’s RUSTC_WRAPPER="" (Dockerfile.toolchain:66) and builds uncached. Between :50 and :645 the only cargo touch is cargo --version (:447). So the knob’s name promises the monorepo and delivers rice-proto; read the two together before trusting either.
A dead parameter. source_build_acceleration_helpers takes include_sccache (03-media/build/onnxruntime/build/lib/common.sh:193-213), but all three call sites (30-build-native.sh:7, -amd.sh:11, -nvidia.sh:26) pass nothing, so the true branch has never run.
USE_CCACHE=false does not disable compile caching — it migrates C/C++ to sccache. With ccache off (or simply absent from PATH), setup_ccache returns early leaving the launchers unset, and any later setup_sccache fills them (compiler-cache.sh:172-210). There is no log line that distinguishes “ccache disabled” from “sccache silently took over C/C++”.

5.4 Where sccache is actually worth it#

target	gate	state	recommendation
rustc (gst-plugins-rs, the monorepo’s Rust)	none any more — `ENABLE_SCCACHE_RUST=1` only reaches `media_common_init` (§ 5.3 item 1)	ON by default since 2026-08-27, through the guarded launcher (`compiler-cache.sh:172-210`, `build-gstreamer-monorepo.sh:649-659`)	details
nvcc / hipcc	`ENABLE_SCCACHE_CUDA=1` (one gate, three sites: `build-opencv.sh:647`, `30-build-native-nvidia.sh:244`, `30-build-native-amd.sh:65`)	wiring exists, default OFF	details
C/C++	—	sccache via the guarded launcher, always on; ccache is the automatic fallback	leave it — this is the owner-directed default since 2026-08-26 (§ 5.1), and both launcher resolvers already pick sccache first (§ 5.2).
cross-machine tier (`SCCACHE_MULTILEVEL_CHAIN`, webdav L2)	—	Windows lane only	details

Per-target detail#

The targets whose recommendation needs more than a table cell.

rustc (gst-plugins-rs, the monorepo’s Rust)#

The one genuine win — and it was taken on 2026-08-27. It is also the one that broke: sccache’s server died mid-compile in three separate rounds (“Failed to send/receive data from server”, “No such file or directory” on trivial crates), each time killing an otherwise-green gstreamer build at 99 %. That signature was root-caused on 2026-08-26 and it was never about Rust: the server is located by a fixed TCP port, which is not container-local, so concurrent BuildKit steps reached each other’s server — one that cannot see their files. SCCACHE_SERVER_UDS took the media stage from 2359 sccache faults to zero (it then regressed on 2026-08-30 and was re-fixed on 2026-09-05 — why), so Rust caching came back, pointed at the guarded launcher rather than bare sccache (build-gstreamer-monorepo.sh:647-659); a server hiccup now costs hits, not a build at 99 %. The preconditions this section used to prescribe are already unconditional in code: SCCACHE_IDLE_TIMEOUT=0 (compiler-cache.sh:129, common.sh:390 — the Windows-lane forensics traced all-zero end-of-vertex stats to the server idle-exiting at 600 s), SCCACHE_ERROR_LOG (compiler-cache.sh:135, common.sh:434), and sccache --show-stats printed to stderr, the stream buildkit’s 2 MiB step-log clip never cuts. What is still open is the measurement: two consecutive green cross-arch media runs with a non-zero Rust hit rate. Until those are on the board the re-enable is shipped but unproven — judge it by the stats line, not by the flag (§ 7).

nvcc / hipcc#

Do not flip on the strength of the hit-rate argument alone. The theoretical win is large (CUDA_ARCHITECTURES compiles each kernel 4×) but a sibling lane produced an evidence-backed verdict that the pinned sccache’s nvcc decomposition silently drops device-conditional code — a byte-identical link failure across two runs, falsified as cache poisoning, i.e. silent wrong code, disqualifying regardless of hit rate (CHANGELOG 2026-08-10/11). Any re-enable needs the three-canary bar recorded there: verify probe + a real kernel compile + a full link canary, on the sccache version actually installed.

cross-machine tier (`SCCACHE_MULTILEVEL_CHAIN`, webdav L2)#

Out of scope for this document. Note only that the mechanism is real and version-gated: on an older sccache the chain variable is ignored silently, which is exactly the failure shape this repo keeps eliminating — so it would need a pinned sccache and a hit-rate assertion before it could be considered on the Linux lane.

5.5 The one-line answer to “does sccache fight ccache?”#

No — but the gates still lie about who is on. The honest summary is: C/C++ runs through the guarded sccache launcher everywhere, with ccache as the automatic fallback for a server that will not answer; Rust runs through that same guarded launcher since 2026-08-27, which ENABLE_SCCACHE_RUST=0 does not stop (§ 5.3 item 1); only the GPU compilers are uncached, and that is a correctness verdict, not an oversight. Any change to that should be a measurement, not a default flip.

The rules as AGENTS.md carried them, with their reasons#

Moved out of AGENTS.md on 2026-09-15. Apart from this heading, only its code pointers and counts have changed since. The RULES stay there, in four lines each; what is below is the evidence for them – which file lists, which commits, which measured failure – and it belongs with the tiers rather than in a guardrails file.

Full map: docs/linux-build-basics.md § Caching Layers. Toggles: USE_CCACHE, USE_SCCACHE, USE_LLD accept 0/false/no/off to disable (since 2026-08-08 — previously ONLY the literal false worked and USE_CCACHE=0 was silently ignored); ENABLE_SCCACHE_RUST/ENABLE_SCCACHE_CUDA are strict 0/1. The rules an agent must never violate:

Closure freeze between runs that should cache-hit. Editing ANY file in the base/toolchain closure changes the compiler image digest and forces sdk/media/android to rebuild from scratch on the next run. Since 2026-08-08 (A1 applied, commit 5d7a318) Dockerfile.base mounts a traced per-file closure, not whole directories — 15 bind mounts per block (12× 01-core .sh + versions.env + cmake.sh + packaging-deps.sh; the lists live in Dockerfile.base itself), one block adding cpython-dev-packages.sh and one the linux/vulkan directory. The base/toolchain closure adds python/build_python.sh, the three bundled 06-packaging/smoke-* scripts, Dockerfile.base and Dockerfile.toolchain. Editing 01-core files OUTSIDE those lists no longer busts base — but Dockerfile.toolchain’s verify layer (# 3c. VERIFY TOOLCHAIN CONTRACT, :216) still binds 01-core and 02-toolchain whole, so an edit anywhere in either directory re-runs it — and everything after it, Rust and the source-built CPython included. It sits after the GCC/LLVM compiles, so those still cache-hit: minutes, not hours. A file NEWLY needed by a base RUN must still be ADDED to the per-file mount lists (closure = source edges + exec/bash edges; the A1 validation build caught exactly such a miss). Batch closure edits; apply them in ONE commit at a planned rebuild boundary. Files in a not-yet-started stage’s closure are free to fix until that stage begins (each nerdctl build snapshots its context at stage start).
~/.config/buildkit/buildkitd.toml pins the GC budget (gckeepstorage) so the multi-hour layers survive between runs. Restart buildkitd only BETWEEN runs (systemctl --user restart buildkit), never while a build solves. Do not delete this file.
sccache is the C/C++ compiler cache; ccache is its FALLBACK. Keep both wired. Owner directive 2026-08-26 (c42091e), REVERSING the earlier “full switch rejected (2026-08-17)”. This is no longer a per-language split: every C/C++ launcher resolves at RUNTIME through compiler_cache_launcher() (01-core/common.sh), which returns sccache when its server answers, else ccache, else fails so the caller builds uncached. Call sites: build-gcc.sh (CC/CXX prefix), build-clang.sh and llvm-cross.sh (CMAKE_*_COMPILER_LAUNCHER), cmake-cache-linker.sh, the onnxruntime build lib, and build-app-wheelhouse.sh (IREE). Since 2026-08-30 (backlog F2) compiler-cache.sh routes through the SAME resolver: _resolve_compiler_cache_launcher() calls compiler_cache_launcher() when 01-core is loaded (every media/ORT caller) and inlines the identical decision only for the android preamble, which sources compiler-cache.sh standalone. Both setup_ccache and setup_sccache consume it; the agreement is pinned by tests/test-compiler-cache.sh. New cache logic belongs in the resolver, not in another duplicate.
- Do NOT hardcode ccache as a launcher anywhere. cmake-cache-linker.sh is SHARED; a literal there silently overrides the decision for every consumer.
- --ccache on build-gcc.sh/build-clang.sh is a historical FLAG NAME. It means “use the compiler cache”, not “use ccache”. llvm.sh passes it.
- Both mounts stay on every heavy RUN (ccache AND sccache), because the fallback needs somewhere to persist.
- Rust IS cached again (2026-08-27, 4200f7b + 54fc1df) — through the guarded launcher. The 2026-08-20 “Rust stays UNCACHED” rule was earned by the sccache SERVER dying mid-compile in three media rounds, killing green builds at 99% — but that signature was the wrong-server-by-fixed-TCP-port bug, cured by SCCACHE_SERVER_UDS (2359 media-stage sccache faults → 0). Two places set the wrapper, in this order: setup_sccache (compiler-cache.sh:172-210), which setup-gstreamer.sh:50 runs unconditionally for the Rust-heavy gstreamer lane, and build-gstreamer-monorepo.sh’s launcher-preferring block, which only fires when RUSTC_WRAPPER is still UNSET. Both PREFER 01-core/sccache-launcher.sh, so an sccache hiccup costs cache hits, not a build at 99%. Since 26a30740 (2026-08-27, owner decision “immer sccache”) their FALLBACKS AGREE: with no executable launcher on disk both ship BARE sccache — build-gstreamer-monorepo.sh’s for _rw in …sccache-launcher.sh loop (export RUSTC_WRAPPER="${RUSTC_WRAPPER:-sccache}") and setup_sccache’s _sc_launcher="sccache" default (compiler-cache.sh:193). Never uncached; the launcher is an upgrade, not a precondition. The launcher is only reachable because 01-core is bind-mounted at /opt/scripts/core on every heavy media RUN; keep it on those mount lists. Exporting RUSTC_WRAPPER="" is the opt-out — Dockerfile.toolchain:66 and Dockerfile.package:262 do exactly that. nvcc stays untouched — the Windows lane records that released sccache breaks around it.
- sccache-specific knobs live in /etc/sccache/config.toml (baked in Dockerfile.base, reached via SCCACHE_CONF), because CCACHE_SLOPPINESS and preprocessor/direct mode have NO env-var path in sccache. They take no effect while SCCACHE_DIR is set, which the chain always does (§ The shipped image’s cache dirs). The size cap is SCCACHE_CACHE_SIZE; there is no sccache -M to call.
- PREFER 01-core/sccache-launcher.sh; fall back to bare sccache rather than to nothing. Superseded 2026-08-27 (26a30740): this rule used to read “NEVER point a launcher at bare sccache”, and taken literally it tells you to delete the default at compiler-cache.sh:193 (_sc_launcher="sccache", upgraded to the launcher when one is on disk) — which would turn verify-critical-fixes.sh RED, because that gate checks the DECISION (never UNCACHED), not the spelling. Always cache; use the launcher when it is available. The launcher still matters: sccache ABORTS the compile on its own internal errors where ccache would just exec the compiler, and that is not theoretical: it killed the media stage three times. The root cause — CMake creates a TryCompile scratch dir, compiles in it, DELETES it, and sccache then spawns the compiler with that dir as cwd (ENOENT) — is written up ONCE, in build-cache-tiers.md § 5.1, with the measurements. Read it there. The launcher runs sccache for every compile and only bypasses on “sccache: encountered fatal error”; a REAL compile error is passed through untouched, because blindly retrying would hide genuine failures.
- Preprocessor cache mode stays OFF (SCCACHE_DIRECT=false, set in ensure_sccache_env and compiler-cache.sh; it overrides the use_preprocessor_cache_mode = true that Dockerfile.base’s config.toml still bakes, § 5.1). We turned it on to recover ccache’s direct-mode hit rate; it re-reads the input file AFTER the compile to store the entry and therefore dies on the same deleted scratch dirs. It is off by default upstream.
- Resolve the launcher through compiler_cache_launcher(). If you add a new cache call site, route it through the helper. The duplicated resolution in compiler-cache.sh is why this class shipped INERT twice: setup_ccache hardcoded the string sccache, so the guard had no effect on the media lane for three runs (fixed c5b17ce); then setup_sccache exported RUSTC_WRAPPER="sccache" unconditionally, and because setup-gstreamer.sh:50 runs it BEFORE build-gstreamer-monorepo.sh’s [ -z "${RUSTC_WRAPPER+x}" ] test, every gst-plugins-rs crate went through bare sccache (measured on the live media lane 2026-08-27; fixed 54fc1df). Because the class shipped inert twice, verify-critical-fixes.sh now GATES compiler-cache.sh against export RUSTC_WRAPPER|CMAKE_C{,XX}_COMPILER_LAUNCHER="sccache". The failure mode this replaced (mount without wiring, wiring without mount) was invisible — builds stayed green, just slow. Before committing a multi-hour run to a change here, run bash linux/scripts/02-toolchain/probe-sccache.sh INSIDE the compiler image: it costs seconds and asserts, per compiler shape this chain actually feeds a launcher, both that the compile survives AND that sccache recorded cache activity. sccache HARD-FAILS on a compiler it cannot identify where ccache would simply exec it, so “it compiles” is not the whole question.
Never edit a running orchestrator’s main script (build-cross-chain.sh while a chain runs): bash reads it incrementally by byte offset; an edit can corrupt the in-flight process. Sourced library files are safe to edit for FUTURE runs (the running process holds them in memory) but see rule 1.

6. Knob reference#

Knob	Default	Effect
`NO_CACHE=1`	unset	`--no-cache`; also disables T2 and T3 entirely
`RUNTIME_NO_CACHE=1`	unset	`--no-cache` on just the runtime package + wrapper builds (and the `export` build below)
`RUNTIME_WHEELS_SOURCE`	`auto` (= `image`)	`export`: the wrapper mounts a small directory staged from the android image before the package build, so it no longer needs the ~12.7 GB android snapshot in the BuildKit store; a prune after the package build then costs nothing. Only the torch RUN’s key changes; the lib is outside every closure (both paths)
`NO_CACHE_EXPORT=1`	unset	drops everything registry-facing (T3, and T4 when enabled); keeps T2
`CROSS_NO_LOCAL_CACHE_EXPORT=1`	unset (set by the disk guard)	stop writing T2; still read it
`SALVAGE_CACHE_EXPORT=0`	`1`	disable the per-`--target` salvage re-drive after a failed stage
`SALVAGE_TARGET_TIMEOUT`	`600`	per-target timeout for that salvage pass
`CROSS_DISK_GUARD_GB`	`40`	free-space floor that triggers T2 LRU pruning (`0` disables)
`CROSS_CACHE_MAX_GB`	`250`	total T2 size cap (`0` disables)
`CROSS_PREFLIGHT_TRIM=0`	unset (trim on)	skip the disk-preflight trim of T2 (§ 3.1)
`CROSS_RUNTIME_LANE_GB`	`120`	free GB the runtime lane needs per concurrent wrapper build; `0` disables the lane-entry gate (§ 3.2)
`CROSS_DISK_WATCH=0`	unset (watch on)	disable the in-stage disk watchdog (§ 3.2)
`CROSS_DISK_WATCH_SECS`	`120`	in-stage sampling interval
`CROSS_TRIM_KEEP_SLUGS`	`3`	newest T2 slugs the trim will never remove
`CROSS_BUILDKIT_PRUNE=0`	unset (fallback on)	disable the guard’s filtered buildkit-store prune (§ 3.2.1)
`CROSS_BUILDKIT_KEEP_GB`	`120`	GB of `type==regular` layer cache that prune keeps; a non-zero value under 100 is raised to 100, `0` reclaims all of it (§ 3.2.1)
`CROSS_IMAGE_PRUNE=0`	unset (lever on)	disable the between-stage image-store lever (§ 3.2.2)
`SALVAGE_MIN_FREE_GB`	`CROSS_DISK_GUARD_GB` (40)	free-space floor below which the S1 salvage is skipped (`0` = always salvage)
`BUILDKIT_CACHE_DIR`	`~/.cache/kata-buildcache`	where T2 lives
`PUSH_MAX_ATTEMPTS` / `PUSH_RETRY_BASE_SECS`	`4` / `15`	transient-push retry budget
`ENABLE_SCCACHE_RUST`	`0`	not the monorepo’s Rust switch any more — it only adds `setup_sccache` to `media_common_init`, i.e. it caches `install-rice-proto.sh`’s `cargo cinstall` (§ 5.3 item 1)
`ENABLE_SCCACHE_CUDA`	`0`	sccache as the CUDA/HIP compiler launcher (§ 5.4)
`USE_CCACHE` / `USE_SCCACHE` / `USE_LLD`	`true`	per-tool switches in `compiler-cache.sh`; note § 5.3 item 3
`KATAGLYPHIS_LINKER`	unset (`default`)	`lld` or `mold` for a consumer build: `LDFLAGS` plus the host Rust target, fails rather than falls back (`linker-select.sh`)
`WEB_LANE_TOOLS_CACHE`	`on`	`refresh` rebuilds past the web-lane tool cache and re-stores; `off` bypasses it (§ 1.2)
`HAILO_NESTED_CACHE`	`carry`	`off` builds the protobuf that HailoRT’s configure compiles under `env -i` without the cache, as before 2026-09-24 (two switches)

CROSS_REGISTRY_CACHE is deliberately not in this table. The T4 pilot knob was written and reverted, nothing in cross-stage-build.sh reads it, and listing it beside live knobs is how an operator ends up setting a no-op. § 4.5 keeps the design.

7. Do-not-repeat list#

Do not add a --cache-from whose matching --cache-to is gated off. That self-defeating pair made every stage rebuild from scratch until 195285f.
Do not add a cache tier that the DeadlineExceeded auto-recovery cannot strip.
Do not “align” the ccache/sccache cache-mount ids with the apt ones (Dockerfile.media:26-31).
Do not reclaim buildkit-store disk with nerdctl builder prune -f or nerdctl system prune, from a script or by hand. Neither has a --filter, so both delete the exec.cachemount records with the layer cache: 4.9 GB of compile cache thrown away to reclaim 207 GB of regenerable layers on 2026-08-17, and both target-LLVM builds then ran cold. The filtered form is buildctl prune --filter type==regular (§ 3.2.1, prune-safe.sh).
Do not size the runtime-lane disk gate per arch. It is per concurrent build (§ 3.2); the per-arch number refuses every run on this host.
Do not judge a cache change by whether the flags appear in the command. Judge it by bytes that did not move: cache hits on intermediate vertices, or a measured wall-clock delta. Two changes were reverted on 2026-08-23 for looking right and doing nothing.

sccache failure classes the launcher must bypass#

01-core/sccache-launcher.sh decides whether an sccache failure costs cache hits or the whole build. Two classes must both bypass to a direct compiler run:

sccache: encountered fatal error — the CMake TryCompile ENOENT class.
sccache: error: failed to execute compile with caused by: Failed to send data to or receive data from server — the sccache server died mid-build. Observed live on the first F2 media validation build (2026-08-30); the launcher did not classify it as sccache-internal, handed it to ninja as a real compiler failure, and killed the TVM step.

A real compiler error must pass through untouched (the compiler is never re-run), and a clean compile passes through. test-sccache-launcher.sh pins every case.

A consumer’s own CMake gets it too (2026-10-07). cmake/Cache.cmake’s myproject_enable_cache used to set both compiler launchers to the bare sccache it found. BeschleunigerBallett run 37509585546 then failed every compile of one job with Connection to server timed out ... Connection refused. Off Windows, with COMPILER_CACHE=sccache, it now sets the guarded launcher beside it in the hub (linux/scripts/01-core/sccache-launcher.sh), which runs sccache from PATH. ccache and Windows keep their bare binary, and test-cmake-cache-launcher.sh pins all three.

What was measured about the ENOENT class#

From the 2026-09-01 run, 3062 bypasses:

2952x sccache: encountered fatal error + failed to spawn <compiler> … No such file or directory (os error 2), on sccache’s OWN -E preprocessor pass. It is intermittent (~10–40% of compiles) and appears in the heavily parallel steps only. The compiler path is absolute, the cwd is a live build dir, and the direct fallback of that same argv succeeds right after — so neither a missing compiler nor sccache’s scrubbed env explains it.
110x sccache: error: failed to execute compile + Failed to send data to or receive data from server — the sccache server died mid-build.

Root cause FOUND 2026-09-05, and it is not inside sccache’s spawn at all: the client was addressing another container’s server. See the server address section.

The single retry (2026-09-03)#

Because the direct fallback of the same argv succeeds immediately afterwards, the launcher retries once before giving up the cache entry. A bypass was never a correctness problem — it compiles fine, it just discards the cache, and that is the entire cost of this bug.

The retry is bounded: a second sccache-internal failure falls through to the direct compile exactly as before, and a real compiler error surfacing on the retry is handed back untouched rather than re-run.

The retry’s own measurement — ANSWERED, and the answer is negative#

The retry doubles as the experiment: retry succeeded (cache kept) versus failed twice in a build log says directly whether the class is transient. That count was taken on the 2026-09-03 media-arm64 build and the question is answered — do not re-run it:

outcome	count
`retry succeeded (cache kept)`	27
`failed twice`	514

The retry stays — it costs nothing and 27 recovered entries are 27 recovered entries — but it is a mitigation, not the fix.

Read that table by STEP, not as one rate (2026-09-05). Splitting the same log by BuildKit vertex dissolves the “5%”: every one of the 514 sits in #24 (litert, 364), #34 (ORT genai, 100) and #45 (pyav, 50), and all 27 recoveries sit in #29 (TVM) — which has no failures of the other kind at all. Four containers, each all-or-nothing, not one population recovering 5% of the time. A per-request property inside one server cannot produce that segregation; a per-CONTAINER one does, which is what the next section shows it was.

The server address must be exported where the compiles run#

This is the root cause of both classes above, and it is a regression of a bug this document already carried as fixed (§ 5.4). SCCACHE_SERVER_UDS makes the server address container-local. 4aa92fb6 (2026-08-27) exported it from the body of setup_ccache, i.e. in the shell that then ran cmake and ninja, and the media stage went from 2359 faults to zero. The F2 one-resolver refactor (8c97cdd8, 2026-08-30) moved that block, unchanged, into _resolve_compiler_cache_launcher — a function every caller invokes as $(_resolve_compiler_cache_launcher). A subshell cannot export to its parent. From that commit on, the launcher name reached the build and the server address did not, so every sccache client fell back to the default TCP port 4226, which under rootless buildkitd is reachable from every concurrently running step.

The same shape predates F2 on the other resolver: compiler_cache_launcher() (common.sh) is called as CC="$(compiler_cache_launcher) gcc" at every one of its ~15 sites, so ensure_sccache_env’s exports have never reached a compile there either.

Reproduced statically, no build needed: source compiler-cache.sh, run setup_ccache with a fake sccache on PATH, and read the caller’s environment — CMAKE_C_COMPILER_LAUNCHER is set, SCCACHE_SERVER_UDS is not.

The evidence in the run logs is arithmetic, not inference. sccache --show-stats reports the counters of whichever server answered:

log	step	at	Compile requests reported
`sccache-retry-20260903-150440`	`#71` libcamera	0.357 s into the step	1264 (1230 hits)
`sccache-retry-20260903-150440`	`#34` ORT genai	0.382 s / 0.634 s	90 → 131 … then 97 at 199.5 s
`rvv-media-20260902-025642`	`#73`	0.226 s / 0.372 s	65 → 544 (+479 in 0.15 s)
`qnn-media-arm64-20260903-125457`	`#34`	0.305 s	423

A step cannot have made 1264 compile requests in the third of a second it has been alive, and a single server’s request counter cannot go 131 → 97. Both are only possible if the client is talking to a server in another container — and in #34’s case to two different ones inside one step. Three independent runs, one signature.

The cost is larger than the bypass counts suggest, because a compile the foreign server does serve is stored in the foreign container’s cache mount: in that same log #24 (litert, id=sccache-arm64) and #29 (TVM, id=sccache-amd64) were running concurrently, mounting different cache directories at /var/cache/sccache while addressing one port.

That also explains the retry’s per-step segregation: the foreign server spawns the compiler by the path the request carries, that path exists only in the caller’s container, and it stays missing for as long as that server owns the port — so an immediate retry changes nothing, while the direct fallback runs in the caller’s own container and always works. It explains the second class too — “Failed to send data to or receive data from server” is what a client sees when the container that owned the server finishes its step.

The fix puts the address back where it has to be: sccache_export_server_address in 01-core/compiler-cache.sh is the one owner of the address, and setup_ccache and setup_sccache call it before the substitution, so cmake, ninja, every launcher process and --show-stats all address the same container-local socket. Pinned by test-compiler-cache.sh; the resolver’s bootstrap arm calls the same function instead of repeating it.

The other resolver has its own owner, for the same reason. compiler_cache_launcher_env in common.sh runs ensure_sccache_env >&2 || true and is called by all 12 $(compiler_cache_launcher) sites — 02-toolchain’s GCC/clang/llvm-cross, opencv, media_compiler_launcher (ffmpeg and pyav), onnxruntime (lib + AMD + NVIDIA), tvm, cmake-cache-linker and the wheelhouse’s two. It is a one-line wrapper on purpose: it is where the RULE is named, it is what a mutation can pin, and it keeps the >&2 discipline that the 2026-08-26 CC="[INFO] …sccache gcc" disaster taught in one place rather than twelve. It is TOTAL (|| true) because it runs in the caller’s shell ahead of the substitution, where a host with no usable sccache must not kill a build that would compile uncached perfectly well, and it honours USE_SCCACHE=0 before anything starts a server. test-compiler-cache-launcher.sh pins the parent-shell propagation, the totality, the switch, the empty stdout, AND that every resolution site in the closure is preceded by the call — that last one is a drift pin, because one owner is worth nothing if a site forgets it.

Dockerfile.toolchain now mounts sccache-launcher.sh beside common.sh in both per-file 01-core blocks. Before that the GCC/LLVM stages resolved to bare sccache — no guarded launcher — so an sccache fault there was a build abort rather than a lost cache entry. The suite asserts the two mount counts are equal, so a future block that mounts one without the other fails.

One duplication is left and it is blocked on a Dockerfile, not on judgement. ensure_sccache_env still carries its own copy of the address block. It is not dead — it is what makes the server exist for the subshell’s own --start-server — but it is a second owner, and collapsing it means common.sh sourcing compiler-cache.sh, which Dockerfile.base’s six per-file blocks do not mount. Measured cost of leaving it: 28 shingles, inside its recorded budget.

How the next chain proves it. The launcher now prints the address it used on both of its lines ([server=/tmp/sccache-0.sock] versus [server=tcp:4226]). tcp:4226 in any bypass line is this bug, still live, in whichever step printed it; a socket path in every line is the fix holding. Count the two message classes per step, not over the whole log.

Reading these counts out of a log needs one caution: a cached BuildKit step replays its old output verbatim, so a pre-retry launcher’s messages can reappear in a post-retry build (496 of them from one cached step, #30, on the first read of that same build) and look like the new launcher failing to take effect. Check which step the lines came from before believing them. A second sample needs a compile-heavy lane — --only runtime produces no sccache-launcher lines at all.

The shipped image’s cache dirs#

:latest-cross shipped CCACHE_DIR=/workspace/.ccache and SCCACHE_DIR=/workspace/.sccache until 2026-09-04. /workspace is the consumer’s bind-mounted checkout (Dockerfile.torch makes it the WORKDIR and a VOLUME), so every compile inside the image wrote a cache into someone else’s repository: it pollutes their working tree, can be swept into CI artifacts, and on a non-ext4 host mount flatpak-builder aborts outright with Can't initialize ccache use: Failed to set permissions of /workspace/.ccache/disabled/ccache.conf: Operation not permitted. Reported by the OmniAccelerANT lane and reproduced as uid 1001.

Where the value came from. eff8deac (2026-06-01) added CCACHE_DIR next to CCACHE_SECONDARY_STORAGE=true, i.e. as one half of a “cache travels with the checkout” idea. The other half was wrong and was deleted in 025bf68c — CCACHE_SECONDARY_STORAGE is a URL and set to true it failed every ccache compile — but the path it was invented for stayed. d5bafe86 (2026-08-26) then added SCCACHE_DIR deliberately alongside it so a consumer would not have a ccache dir and no sccache dir; the /workspace prefix was inherited from the line above, not chosen.

Does any stage need a per-workspace cache? No. Nothing else in the Linux tree names /workspace/.ccache; the build stages get their cache dirs from Dockerfile.base’s ENV, from llvm-cross.sh’s explicit exports, and from --mount=type=cache,target=/var/cache/ccache — none of which reads the package stage’s runtime ENV. The only instructions after that ENV block are symlink wiring, the apt mirror rewrite, setup-package-image.sh and validate-compilers.sh, and none of them runs a compiler through a cache launcher. So the value was runtime-only, and repointing it cannot change what a build does. A consumer who wants the cache next to their checkout still sets -e CCACHE_DIR=... themselves; a consumer who wants it to persist across runs mounts a volume at /var/cache/ccache, which is what that directory is for.

Why the ENV cannot simply be deleted. Dockerfile.base already declares the same defaults, but its image config does not reach the shipped image: the runtime package stage is built on a rootfs/OCI-layout export of its parent (context-management.sh), which carries the filesystem and drops the parent’s Config.Env. Inspecting the shipped latest-cross-amd64 shows exactly Dockerfile.package’s own ENV blocks plus Dockerfile.torch’s — base’s CCACHE_MAXSIZE, SCCACHE_CACHE_SIZE, SCCACHE_CONF, SCCACHE_IDLE_TIMEOUT and SCCACHE_ERROR_LOG are all absent from it, while base’s filesystem effects (the 1777 cache dirs) are present. The runtime image must therefore declare the paths itself.

One owner. 01-core/compiler-cache.sh owns the two paths as its := defaults; Dockerfile.package’s ENV repeats them because a Dockerfile cannot source shell, and test-compiler-cache.sh sources the library with both vars unset and asserts the ENV equals what it produces — plus a second assertion that neither value sits under /workspace. Drift on either side fails the suite.

The caps and server knobs (2026-09-28). Dockerfile.package now also repeats base’s CCACHE_MAXSIZE=30G, SCCACHE_CACHE_SIZE=30G, SCCACHE_IDLE_TIMEOUT=0 and SCCACHE_ERROR_LOG=/tmp/sccache.log. Until then the published :latest (v0.0.28) carried only the two dirs, so every consumer’s sccache ran with its 10 GiB default. test-compiler-cache.sh asserts the four equal base’s. SCCACHE_CONF stays out on purpose: it changes nothing while SCCACHE_DIR is set (see below).

What the shipped image already provides. /var/cache/ccache and /var/cache/sccache are drwxrwxrwt (Dockerfile.base, 1777 since 35f83d62) and uid 1001 writes to both — verified by running the shipped image. A missing cache directory is not an error either: ccache creates CCACHE_DIR on first use, which is how a consumer’s -e CCACHE_DIR=/some/mount keeps working.

Layer cost: none. This is an ENV change, so nothing is copied up and no file metadata is rewritten.

One build in this image’s lineage does compile through the launcher (since the Hailo fold-in, 2026-09-20): the Hailo RUN in Dockerfile.torch builds HailoRT, pyhailort, libzmq and TAPPAS on the ccache-<arch> and sccache-<arch> cache mounts. The paragraph above says nothing after the package ENV does; that stopped being true there. Base’s caps do not reach that RUN either, for the same rootfs-handoff reason, so compiler-cache.sh’s 10G defaults applied. An sccache server trims its directory to its own cap on its first compile (the design probe of 2026-09-23 saw an 8 MiB cap cut a 41 MiB directory to 8 MiB), and ccache -M 10G rewrote the shared ccache.conf. The RUN mounts sccache-${TARGETARCH} and ccache-${TARGETARCH}, the ids every cross stage mounts for the arch CROSS_BUILD_PLATFORM names. On that arch the Hailo RUN shares the chain’s own cache and could shrink it: amd64 on the cross host, arm64 on the Jetson (CROSS_BUILD_PLATFORM=linux/arm64). Only an arch the runtime lane builds under QEMU, such as arm64 on the cross host, has mounts no cross stage uses. riscv64 skips the Hailo build and never writes its mounts.

SCCACHE_CONF does not set the cap, nor any other knob in the file: as soon as SCCACHE_DIR, SCCACHE_CACHE_SIZE, SCCACHE_DIRECT or SCCACHE_LOCAL_RW_MODE is set, sccache builds its disk config from the environment, and that replaces the file’s whole [cache.disk] section (sccache src/config.rs, config_from_env and CacheConfigs::merge). Only SCCACHE_CACHE_SIZE sets the cap. Measured on the published :latest on 2026-09-28 with sccache --show-stats:

Environment	Max cache size
`SCCACHE_DIR`	10 GiB
`SCCACHE_DIR` + `SCCACHE_CONF`	10 GiB
`SCCACHE_CONF` alone	30 GiB (the file)
`SCCACHE_DIR` + `SCCACHE_CACHE_SIZE=30G`	30 GiB

Base’s ENV and compiler-cache.sh always set SCCACHE_DIR, so the baked file_stat_matches = true and hash_working_directory = false have never applied. What applies is sccache’s env default: preprocessor-cache mode on, with file_stat_matches = false and hash_working_directory = true. The chain’s entry points turn the mode off with SCCACHE_DIRECT=false; the shipped image sets no SCCACHE_DIRECT, so a consumer runs with the mode on. Since 2026-09-24 the Hailo RUN passes SCCACHE_CACHE_SIZE=30G and CCACHE_MAXSIZE=30G, and tests/test-hailo-build.sh pins them to Dockerfile.base; its [CACHE] hailo/<phase> lines print cap=30 GiB. compiler-cache.sh keeps its 10G defaults: changing them is an 01-core edit, which re-keys the chain from the compiler stage.

Keeping the compiler cache across CI runs#

A consumer lane starts every step in a fresh --rm container, so the image’s /var/cache/{sccache,ccache} die with it and every CI run compiles cold. The compiler-cache-restore / compiler-cache-save actions mount a runner directory at those paths and carry it between runs through actions/cache.

Measured in AccelerANTgine’s linux-x64 lane on 2026-09-29, each warm run beside a control run of the same commit at the same time (3 control, 2 warm, 1 cold):

clang job	no cache	warm cache
whole job	25.1 min (20.3-29.4)	14.4 min (14.1-14.7)
the four `cmake --build`	14.7 min	3.9 min

The cache was 405 MB and restored in about 7 s. A cold run, which is what the first run after an eviction gets, took 32.6 min. What a warm run cannot shorten is the host prologue: the image pull stayed at about 8 min.

What caches. clang’s C++20 modules cache with the stock sccache on Linux: the interface units store their BMI as an output and the importers hash every BMI they import, so an interface edit misses exactly its importers (verified on BeschleunigerBallett and AccelerANTgine). GCC’s -fmodules-ts translation units do not cache with sccache 0.18.0 or ccache 4.12.3, so a GCC lane gains only its non-module TUs.

The quota is 10 GB per repository, shared with every other cache (CodeQL, setup-uv). An actions/cache entry cannot be overwritten, so every save gets a per-run key and restore takes the newest. Since 2026-09-29, compiler-cache-save also deletes the key’s entries from older runs on the same ref before it saves, so each key holds one entry.

Before that, the superseded entries stayed in the quota until GitHub evicted them. Its eviction removes the least recently used entry of the whole repository, not of the key. BeschleunigerBallett reached 11.2 GB and OxidANT 12.4 GB. Ballett’s TSan job then restored nothing: its only entry had been evicted, while two stale x64 duplicates, read more recently, survived. A scheduled clean-up job would come too late for this, because GitHub evicts at the moment a save passes the quota.

The deletion needs permissions: actions: write in the saving job. Without it the action warns and saves anyway. Budget a repository so one generation of all its lanes, plus one entry being saved, fits inside the quota:

one key per preset that is worth it;
each capped with the env input (sccache keeps its most recent entries under the cap).

enabled: auto saves on pushes to the default branch only (since 2026-09-29). The deletion works per branch, so a release push to main would otherwise have stored a second generation of every key: about 7.3 GB more in BeschleunigerBallett. Every other branch, and every pull request and dispatch, restores the default branch’s entry, which GitHub lets any branch read.

Windows container lanes take container-ci-windows.yml’s compiler-cache-key input (2026-09-29). An SCCACHE_DIR on a mounted volume stored nothing on the dev host (§ sccache’s cache directory on a Windows container volume), so the cache is not mounted for sccache to write into. actions/cache restores it into the workspace as .ci-cache, the container gets CI_COMPILER_CACHE=C:\ws\.ci-cache, and the build script moves it into container-local directories before the build and back after it (OxidANT’s Invoke-WindowsLane.ps1). Pushes to the default branch save it.

That reusable workflow saves with actions/cache/save and deletes nothing. Its job grants only contents: read, and a reusable workflow cannot ask for more than every caller grants. A caller that sets compiler-cache-key therefore runs compiler-cache-prune from a later job that grants actions: write (OxidANT’s windows-x64.yml).

A clang-cl C++20 module build must not use it: the released sccache serves stale objects for module importers until mozilla/sccache#2876 ships.

A cap below the working set is worse than no cache. BeschleunigerBallett’s release and profile caches filled their first caps (489/500 MB, 981/1000 MB): sccache evicted entries during the cold build, and the warm release build took 782 s against 533 s without a cache, paying the hashing for no hits. With room (1.1 GB each) the same builds took 257 s and 349 s. Size a cap from a cold run’s saved size, with headroom.

The shipped image carries no build-host setting#

A published image’s environment names nothing outside the container (both lanes, 2026-09-23). The Windows image shipped the owner’s LAN WebDAV as SCCACHE_WEBDAV_ENDPOINT, and every consumer’s sccache 0.18 server died on its storage check — the full account, the Windows split of ARG versus ENV and the consumer-side probe are in windows-build-resources.md § What the published image carries.

The Linux lane never had the leak: its cache is BuildKit cache mounts, with no remote tier (§ 5.4, the cross-machine tier), and the mirror URLs (FAST_UBUNTU_MIRROR_URL, …) are ARGs. The runtime defaults above (CCACHE_DIR, CCACHE_MAXSIZE, SCCACHE_DIR, SCCACHE_CACHE_SIZE, SCCACHE_IDLE_TIMEOUT, SCCACHE_ERROR_LOG) are container-local and stay. What keeps a leak from ever reaching :latest:

The static pass, linux/scripts/verify_image_env.py --dockerfile, runs in lint-dockerfiles.sh (preflight slug dockerfile-lint) over both lanes’ Dockerfiles. It refuses an ENV of an sccache remote-backend or layout name (SCCACHE_WEBDAV_*, SCCACHE_REDIS*, SCCACHE_BUCKET, SCCACHE_MULTILEVEL_CHAIN, SCCACHE_FORCE_LOCAL, …), an ENV that expands one under another name, and an RFC1918 or link-local literal in an ENV or an ARG default. On Windows it also refuses a compiling RUN whose stage never declares ARG SCCACHE_WEBDAV_ENDPOINT.
_manifest_image_env_gate, the first step of create_manifest in build-runtime-manifest.sh, reads each wrapper’s config ENV with nerdctl image inspect --platform linux/<arch> --format '{{range .Config.Env}}…' and runs the same script with --env-file. An unreadable config fails it too, and a tag missing locally is pulled first (only then, like the content gate). It has no switch: it runs on every path that creates an index, --manifest-only/--repair included, and neither RUNTIME_IMAGE_SMOKE=0, WRAPPER_CONTENT_GATE=0 nor --force waives it. Until 2026-09-24 it was check 6 of verify-shipped-wrapper.sh, which sits behind RUNTIME_IMAGE_SMOKE and never runs on --manifest-only. It does not cover the per-arch wrapper tags: those are pushed by the build loop before any runtime gate, as they always were.

The address rule, the version exemption and what neither pass covers (hostnames, loopback, files, image history) are the same on both lanes, because one case file, linux/scripts/tests/image-env-cases.json, grades this script and the Windows PowerShell twin alike. Tests: linux/scripts/tests/test-image-env.sh; mutations dockerfile-lint.image-env-* and image-env.*.