# Windows backlog archive — 2026-08-21 Resolved narratives moved out of `windows-builds.md` per the lean-OPEN-only policy (COUNTING NOTE there). This tranche covers everything resolved up to and including 2026-08-21: the post-08-17 numbered entries (#72-#110 with their verification evidence), the CURRENT-SEQUENCE ride log (versions.env bump ride, the sccache deadlock/miscompile verdict), P0-P7 narratives, and the complete P8 refactor-audit cycle (#120-#131, opened AND liquidated 2026-08-21, commits 48f5733f..891394c4). Standing directives extracted from these entries live on in `windows-builds.md` — this file is the record, not the rulebook. Earlier tranches: windows-backlog-archive-2026-08-11.md, windows-backlog-archive-2026-08-17.md. > **Editorial note (2026-08-25).** `windows-builds.md` was split on that date. > The standing directives this file points at now live in > [`windows-refactor-backlog.md`](windows-refactor-backlog.md); the lane > mechanics moved to [`windows-build-lanes.md`](windows-build-lanes.md). The > entries below are left exactly as written -- this is the record, not the > rulebook. > **Editorial note (2026-09-08).** #112 below points at > `scripts/diagnostics/archive/Test-FfmpegProvenance.ps1`. That path no longer > exists: the settled-probe sweep deleted the probe and retired > `diagnostics/archive/` entirely (`Test-SccacheWrite.ps1`, the one live > resident, moved back out to `diagnostics/`). The probe is in git history -- > the entry keeps its name because the name is the record. Same for any other > `diagnostics/archive/` path in this file. ## CURRENT SEQUENCE (the one list — batches A–D/G completed, see archive) > Ordered by what unblocks what; the verification chain is the bottleneck, not > the code. One experiment per build. > > **2026-08-17 late: steps 1+2 are DONE-GREEN** (verify10–15 + final; smoke > gate 190 passed / 1 skipped / 0 failed). That run verified #93/#95/#65/#66/#88 > end-to-end and en passant surfaced+fixed: the #47 TVM LLVM heal (own minimal > LLVM — scoop has none, dev tarball is /MT), the Anubis/`.git` wrap-download > pair, graphene's clang-cl port, and #113 (stall-guard exports). > > 1. **versions.env bump full-chain ride — DONE-GREEN 2026-08-18** (4 h 01, > smoke 190/1/0, image verified; first full-chain #61 manifest: onnx 60.5 > min, litert 43.9, base 31.1, merge 28.9 incl. two snapshotter-mount > retries, tvm 21.8 with the mini-LLVM warm). #112's `chain=''` reproduced > deterministically on this ride too — it is a parse hole, not a flake. > 2. **Deadlock repro — VERDICT IN (2026-08-18, run 2):** > * **Deadlock GONE under WebDAV-only** — all 1891 CUDA objects incl. > every fused_moe launcher compiled through the sccache server, no > stall. The two historical wedges were #99 collateral (the L0 > write-failure storm), not a decomposition hang. > * **Miscompile CONFIRMED, storage-independent, on a COLD-CACHE run** — > link died on dropped instantiations (`QkvToContext<*, __nv_fp8_e4m3>`, > `BiasSoftmaxImpl`, `run_memory_efficient_attention`): the > objects are wrong as they leave the wrapped compile, so the loss is in > sccache's nvcc decomposition itself, not cache-hit replay. **CUDA > stays bare; the launcher-default question is CLOSED (canary 3 is red > before any hits are even consumed).** Addendum draft updated: > `out/upstream-sccache-2808-addendum.md` (owner posts). > * (Run 1 earlier that morning was a false all-clear — `& pwsh -File` > flattened the -BuildArg pair into one mangled string buildctl silently > discarded. Fixed: in-process invocation + driver key validation.) > 3. **After those builds free the mounts/files:** #100 (FFmpeg/PyAV sccache), > #107 (extract sccache session helpers), #112 (opencv-stage provenance > gate empty-read), #68/#69 (FFmpeg fallback + pin drift), #45 (CUDA path > fail-open), #106 rest (mass `#requires`), #49/#51 (media ENV split). > 4. **Base-tier batch — NEVER land alone** (#50 + #81, plus #78/#79 if their > fixes touch setup-vs): one deliberate base rebuild for all of them. > 5. **Owner decisions:** #31 (registry push; #59 branch protection DECLINED 2026-08-17), and the > upstream actions in "Pending" below. > 6. **Latent / needs a repro first:** #73 (ONNX-CUDA infinite retry — needs a > runtime repro), #75/#76 (timeout+heartbeat insurance; re-measured quiet > across 5 chains), #80 (warning-stream noise filter), #52/#54 (toolchain > bind-mounts / merge CUDA-runtime dedup — each wants its own verify build, > #54 carries a load-bearing warning). ## P0 — LIVE DEFECTS (not refactors; the chain is green *and* wrong) > Found by the 2026-08-14 deep audit (static sweep of 151 scripts + 6 > Dockerfiles + 102 build logs). Each was verified against the tree/logs, not > inferred. These ship broken today — do them before any refactor below. ## P0d — RESOLVED 2026-08-16 (stub; full narrative in windows-backlog-archive-2026-08-17.md) sccache lost 100 % of L0 cache writes (`os error 3`). ROOT CAUSE: BuildKit WCOW cache mounts lose writes into a directory an EARLIER RUN populated (158/250 on the mount vs 0/250 in a plain dir; fresh dir on the same mount is clean). RESOLUTION: `SCCACHE_MULTILEVEL_CHAIN` defaults to `""` (WebDAV only) in both media Dockerfiles; genai went 157 write errors → 0 → 157 HITS. Eleven hypotheses were measured and killed on the way; the method lessons live in AGENTS.md (probe-reproduces-environment-not-failure; probe-destroys-own- experiment). Trail items #89–#99 (verbatim, with every measurement): archive. Upstream follow-ups: see "Pending" at the bottom. ## P0b — Confirmed by log forensics (49 runs, 185 MB; 2026-08-14) > The corpus predates the step-log fix, so 28 of these logs are CLIPPED > (the green reference run is 49 % blind in its merge step; historical ONNX > steps are 83 % blind). Findings below survive that caveat — they rest on > end-of-step stat blocks and timestamps, not on clipped body text. Re-run the > forensics once a full chain has been captured with the env now in place. - **72 [M·★★★, none] CLOSED 2026-08-20 - PREMISE DISPROVEN by the 2026-08-16 re-measurement (export ~1.2% of the chain, not 23%; the old figures came from stall/cache-failure-dominated runs). Standing instruction: do NOT collapse the media-core checkpoints on this item's authority - the resume granularity is worth more than ~60 s. Original finding: image export/unpack costs MORE than the build it wraps — 4.33 h across the corpus, 339 operations.** The chain is split into 9+ separate `buildctl` invocations and **each pays a full Windows-image export AND unpack**. Torch: 358.4 s export vs 172.6 s build (**2.08×**). LiteRT: 1401.8 s export vs 1117 s build (1.25×). Today's base build: 605.6 s export/unpack = **33.5 % of the whole stage**. In the green 41:30 run, export/unpack is **23 % of the entire chain** — more than every COPY, every fan-in and the torch build combined. No static reading of the Dockerfiles reveals this. FIX: collapse the four `media-core-built-*` checkpoints into one invocation → removes 3 export/unpack round-trips per run. **CONFLICT — decide before doing:** those four checkpoints are exactly what gives media-core its per-library BuildKit-native resume (the reason litert/tvm, which lack them, re-pay a whole branch on any failure). Collapsing them buys ~3 export round-trips and costs that resume granularity. Measure both before choosing; this is a trade-off, not a free win. **RE-MEASURED 2026-08-16 on five complete media chains — THE PREMISE NO LONGER HOLDS; DOWNGRADE TO [S·★].** Batch A's standing caution said this item rests on the clipped pre-fix corpus and must be re-checked before anyone acts on it. Done, across `bk-run-{webdavonly,reuse,chain-disk,legacy-disk,forcelocal}.log`: | | per chain | |---|---| | `exporting layers`, all vertices summed | **63–69 s** | | longest single build vertex (onnx) | 3277–3517 s | | whole chain | ~5 400 s | Export is **~1.2 % of the chain**, not 23 %, and nowhere near "more than the build it wraps". The old figures were real for the runs they came from, but those runs were dominated by stalls and cache failures that no longer occur. **Do NOT collapse the four `media-core-built-*` checkpoints on this item's authority** — the resume granularity it would cost is now worth far more than the ~60 s it would save. Keep the entry only as the record of a disproven premise. - **73 [S·★★★, none] SOLVED 2026-08-20 (verify: next media rebuild must show ZERO -Winfinite-recursion) - and the culprit was OUR OWN inline patch: the `_udiv128 -> udiv128` substitution (added for clang-cl's missing MSVC intrinsic; probe-udiv128-recursion proved clang 22.1.8 has no _udiv128) rewrote the call INSIDE cutlass's udiv128 into a self-call. Fix: disable CUTLASS's intrinsic guard for __clang__ instead - the portable 128-bit loop compiles, correct by construction. Upstream candidate (owner): NVIDIA/cutlass's guard `#if _MSC_VER >= 1920 && !defined(__CUDA_ARCH__)` should also carry `&& !defined(__clang__)`. Original finding: latent defect in the SHIPPED ONNX CUDA provider: infinite recursion in CUTLASS `udiv128`.** 225 occurrences of `uint128.h(96,90): warning: all paths through this function will call itself [-Winfinite-recursion]`, reached via `flash_api.h:36` while compiling `onnxruntime_providers_cuda` TUs (`attention.cc`, `paged_attention.cc`, `packed_multihead_attention.cc`). CUTLASS selects an MSVC `_udiv128` intrinsic path that clang-cl does not resolve, so the function calls itself unconditionally → stack overflow **if that path is taken at runtime**. It sits in the flash-/paged-attention code of a shipped provider. Static analysis cannot see this — it exists only in the clang-cl port's compiler output. FIX: runtime smoke test of flash-attention, then a clang-cl `udiv128` patch alongside the existing `patches/onnxruntime` set. - **74 [S·★★, none] DONE + VERIFIED 2026-08-20: the batch-verify ride (cold C++ media build, green + smoke 190/0/1) logged ninja -j19 in ALL three formerly -j9 stages (genai, litert, tvm) - the MemGBPerJob=2 change measures out. Original PARTLY DONE 2026-08-15 — the `-j9` half is fixed, the measurement is not.** Backlog #28 lowered `MemGBPerJob` to 2 for onnx and opencv only, so the 2026-08-15 chain still logged `ninja -j9` three times (genai, litert, tvm) against `-j19` three times. Those three are now at 2 as well, justified by the ONNX measurement (9274 samples, peak per-process WorkingSet 998 MB, same nvcc workload) and by build-iree, which compiles LLVM in-tree — the heaviest TUs in the chain — at 2 all along. **UNVERIFIED:** no build has run since the change, and the claim that sccache is "provisioned for 32 jobs" while ninja runs at 9 came from the CLIPPED corpus. Confirm the job count and the wall-clock on the next cold media build before closing. - **75 [S·★★★, none] DONE 2026-08-19 (module edit, next media rebuild): the downgrade is now LOUD - Write-Warning before (naming the cross-run crash signature to watch for) and after (stamping the serial fallback's added minutes on a green finish); the ladder was already bounded to ONE incremental attempt. Original finding: the `-j` downgrade ladder is a SILENT self-heal that converts a failure into an hours-long serial rebuild.** One run (`bk-chain-20260810-nogpu`) burned **11 h 17 m re-running the same ONNX build 11 times**, each cycle ending `ninja -j9 failed (exit 2) - retrying incrementally with -j2...` at 4911.5 s / 4909.2 s — a ±2 s determinism that identifies the sccache-CUDA server crash rather than an env flake. 12 occurrences corpus-wide. FIX: make the downgrade loud and bounded; abort instead of grinding serially. **RE-MEASURED 2026-08-16: zero occurrences across five complete media chains.** Still worth fixing — a self-heal that can silently cost 11 h must be loud and bounded whether or not it fires today — but it is **latent, not active**, so it does not belong ahead of work on live defects. Same status as #76. - **76 [S·★★, none] DONE 2026-08-20: the make/gawk provisioning region is bounded (10-min ceiling, 60-s heartbeat via Invoke-BoundedProvisionStep) - a recurrence costs minutes and names itself. Original finding: the ~120-min ffmpeg stall (old #35) is CONFIRMED as a one-off and DENIED as recurring — and it is a TIMEOUT, not jitter.** Exact gap: **7200.9 s** (≈ exactly 2 h) of zero output between `WARNING: vswhere returned no installation; using filesystem fallback` and `Replaced MSYS2 awk with gawk`. In 10 other runs that marker lands at t = 11.5-17.9 s. Actual ffmpeg work in that step was only ~162 s. The two-hour boundary reads as a network timeout in the MSYS2/gawk provisioning call. Not recurred in 9 subsequent runs → latent, not active. FIX: bound that step with an explicit timeout + heartbeat. **Supersedes the old #35 observe entry**, which can now be closed. **RE-MEASURED 2026-08-16: not recurred in five more complete chains** (longest vertex anywhere 3 517 s = onnx, no >2 h gap). 14 clean runs since the one-off. Confirms latent-not-active; the timeout+heartbeat is still the right fix and is cheap, but it is insurance, not a repair. - **77 [S·★★, none] STALE 2026-08-17 — the retry has NOT fired since the GStreamer 1.29.2 bump.** Today's full chain compiled GStreamer clean on attempt 1 (0 hits for the retry marker), matching the code comment: in 1.29.2 there is no `_commit` collision and the reactive path is documented DORMANT INSURANCE (the .patch stays git-appliable for a future clang/io.h/gstreamer combination). The "3/3 runs, ~20 min each" evidence predates the bump. No action; re-open only if the marker reappears. Original finding: GStreamer's GES `_commit` conflict is patched REACTIVELY after a failed compile — deterministic, 3/3 runs, ~20 min discarded each time.** `Compile attempt 1 failed; patching _commit conflict in GES and retrying...` at 1236.8 s / 1392.2 s / 1391.3 s in three separate runs. Tight clustering + 100 % reproduction = this belongs in `patches/gstreamer` applied up-front, not as a post-failure repair. - **78 [S·★★, none] DONE 2026-08-20: the filesystem fallback prefers the VISUAL_STUDIO_VERSION major (a VS promotion can no longer float in newest-first), warns loudly on a pin miss, and is memoized per process (was x100 warnings). Original finding: the VS major-version pin is NOT being honoured — the toolchain is pinned by luck.** Today's base build: `WARNING: major-pinned VS alias unavailable — used floating 'stable' channel (currently VS 18…)`. Plus `vswhere returned no installation; using filesystem fallback` ×100 — so the build depends on a literal path string rather than on discovery. The day Microsoft promotes VS 19, the pin floats AND the fallback path breaks simultaneously, re-opening the documented vcpkg/VS-toolset rejection class. - **79 [S·★★, none] DONE 2026-08-20: pinned-alias retry budget 2->3 before the loud stable degrade; the Adoptium half already shipped separately (github-first JDK fetch in build-litert-lm-bazel). The MZ-signature guard remains the HTML defence; no bootstrapper preseed (its hash floats within a channel by design). Original finding: `aka.ms` serves HTML instead of the VS bootstrapper binary — the same failure family as the known nuget trap.** Today's base build: `expected a MZ-signature file but got first bytes 60,33 (likely an HTML error page)` — `60,33` is `nul` (`Build-GstreamerFromSource.ps1` ~:294 and the libffi fetch at ~:313), so the ONE thing that distinguishes a moved wrap revision (404) from a DNS/TLS problem is discarded — and 20 lines earlier the main tarball already uses `Invoke-DownloadWithRetry` with backoff and non-empty verification. Only the four MANDATORY plugins are gated; every other codec silently becomes optional. FIX: route the wrap and libffi fetches through the shared helper, and fail (or at least summarise loudly at the end of the stage) rather than emitting 22 warnings nobody counts. NOTE this was in the 2026-08-14 audit and was dropped when the findings were numbered — the numbers came from the audit's list, and this one fell out; re-verify the P2b set against the audit before assuming it is complete. - **69 [S·★★, none] DONE 2026-08-20: W1c AST scanner covers the if($env:KEY){...}else{''} idiom (pin membership filters behavior defaults; scanner-rot guard pins 9 sites) and caught 3 LIVE drifts, all fixed: build-ffmpeg NV_CODEC_HEADERS_REF n13.0.19.0->n13.1.15.0 (the documented 404/NVENC-skip seed), build-litert-lm-bazel 0.15.0->0.16.1, assemble-torch-app v0.0.22->v0.0.27. Original finding: live pin drift that the parity gate structurally cannot see.** `Build-FfmpegFromSource.ps1:241` hardcodes `else { 'n13.0.19.0' }` against `versions.env:184 NV_CODEC_HEADERS_REF=n13.1.15.0` — verified drift. `SourceBuild.PinParity.Tests.ps1:80` scans only `Get-SourceBuildVersion` call sites, so the `if ($env:X) {…} else {}` idiom is invisible to it (~13 such sites; four more version literals bypass the gate the same way). versions.env:180-183 records that a wrong nv-codec-headers ref once "404'd and NVENC was silently skipped on both lanes" — this is that incident's seed, re-planted. FIX: route the literals through `Get-SourceBuildVersion`; teach the AST scanner the second idiom. - **70 [S·★★, none] DONE 2026-08-20, subsumed by #100: FFmpeg compiles through the make-time sccache launcher (2198/2198, 100.00% on the hit run) and the chain epilogue emits its stats.** Original finding: FFmpeg is the only compile stage with NO sccache wiring at all — verified: 0 `Write-SccacheStats` calls, and the script never sets the sccache endpoint. The precedent is its sibling, which documents that GStreamer "ran completely uncached (~30 min hot)" until 2026-08-04 because "the merge builder simply never wired the endpoint through". A 30-60 min stage recompiles cold every attempt and is not even MEASURABLE. Emitting the stats is unconditionally safe; whether FFmpeg's `configure` tolerates `--cc="sccache clang-cl"` needs a configure-only probe first. ## P3 — Cache tiering (pure rebuild-time cost; no correctness change) - **49 [M·★★★, media-core once] LANDED 2026-08-19, riding the full ride (verify: a later PYAV-only bump must NOT re-run onnx): per-component ARG/ENV blocks in the BK stages, media-core-env is classic-lane-only, TwinParity suite carries the new contract. Original finding: nine version ARGs share ONE ENV layer directly above the ~75-min ONNX compile.** `Dockerfile.media-builder:142-168` declares ONNX/GENAI/OPENCV/FFMPEG/PYAV/NV_CODEC/CUDA_ARCH/PYTHON in a single `media-core-env`, and opencv/ffmpeg/genai chain `FROM` ONNX's output. So a **PyAV bump re-runs the full ONNX build** and cascades through the whole branch (hours). The 2026-08-07 versions.env-COPY removal fixed this at BRANCH granularity and never reached COMPONENT granularity. FIX: move each ARG+ENV into the stage that consumes it. - **50 [M·★★★, base once] DONE 2026-08-18 (riding the #114 base batch): versions.env COPY relocated below scoop/vcpkg/rust; the 9 consumed keys (incl. helper-reads GIT_VERSION/WIX_*/SCOOP_INSTALLER_SHA256 - invisible to a naive $env: grep) ride as ARGs mirrored in both drivers. AFTERMATH FIXED 2026-08-19: the final-stage ARG mirrors sat in the process env during the bake RUN, load-versions' override branch left Machine untouched, and those keys (measured: SCCACHE_GIT_REV, machine=[] in-container) were never baked into post-#50 images - load-versions now persists the winning override to Machine (Dockerfile.load-versions-probe, fail-closed; rides the next base build). Original finding: `versions.env` is COPY'd above scoop + vcpkg + the ~30-min rust/sccache-from-source layer.** `Dockerfile.base:87-89`, then `:114-120`, then `:156`. versions.env is shared by BOTH lanes, so editing a purely *Linux* key (`PANDOC_VERSION`, `ROCM_VERSION`, `UBUNTU_DIGEST`) re-pays GB-scale scoop + vcpkg + the 30-min rust layer on the next base build. The file already proves it knows the pattern — `Install-Vs.ps1` was deliberately hoisted above this COPY for exactly this reason (`:71-76`). Only 8 keys are needed below the COPY; promote those to ARGs and move the COPY down. (The sibling ARG-below-the-expensive-RUN fix for TensorRT shipped 2026-08-14 — same pattern, see the archive addendum.) **SCOPED 2026-08-14 — and the audit's "only 8 keys" was wrong.** Enumerating what the three RUNs below the COPY actually read (both `$env:X` *and* `Resolve-ContainerImageValue -EnvironmentVariable 'X'`, which the first pass missed because it uses no `$env:` syntax): - `Install-ScoopTools.ps1`: CMAKE_VERSION, FLUTTER_VERSION, GIT_INSTALLER_URL, GIT_VERSION, GIT_WINDOWS_INSTALLER_SHA256, LLVM_WINDOWS_VERSION, NASM_WINDOWS_VERSION, NINJA_WINDOWS_VERSION, SCOOP_INSTALLER_SHA256, VULKAN_VERSION, WIX_UI_EXT_VERSION, WIX_VERSION - `Install-Vcpkg.ps1`: VCPKG_REF - `Install-RustToolchain.ps1`: SCCACHE_GIT_REV, RUSTUP_DIST_SERVER, RUSTUP_IO_THREADS Five of those (CMAKE/LLVM/NINJA/NASM/VULKAN) **already have ARGs** and are passed as parameters, so the work is ~11 NEW ARG declarations, each duplicating a versions.env pin into the most expensive Dockerfile in the repo. **TRADE-OFF — decide before doing.** The gain is purely cache cost: editing a Linux-only key (PANDOC_VERSION, ROCM_VERSION, UBUNTU_DIGEST) would stop re-paying scoop + vcpkg + the ~30-min rust/sccache layer on the next base build. The cost is eleven new duplicated pins, i.e. exactly the drift surface that #69 still tracks and that `Pins.CanonicalValues.Tests.ps1` was written to police (closed #60) — that test covers Dockerfile.media-merge-builder today and would have to be extended to Dockerfile.base before landing them. Not obviously worth it; that judgement is the owner's, which is why this was NOT landed with #81 on 2026-08-14 even though the base was rebuilt anyway. - **51 [M·★★★, media once] DONE 2026-08-20: no longer image metadata - the driver publishes the effective budget to the webdav (preseed/memory-limit-gb.txt), Get-BuildJobCount reads env -> webdav (memoized) -> CIM; under -ConcurrentAux every branch now gets the halved budget (the old full+halved+halved asymmetry oversubscribed the host). Original finding: `MEMORY_LIMIT_GB` — a scheduling knob — is an image ENV and therefore a CACHE KEY** (`Dockerfile.media-builder:29,67`). The driver halves it for `-ConcurrentAux` (`build-buildkit.ps1:378`), so merely TOGGLING that flag changes the layer digest and invalidates every litert/tvm compile. Same on any host with different RAM. `Dockerfile.torch:57-60` already states the principle ("Build-time state belongs in the build step, not in the artifact"). FIX: derive in-container, or bind-mount it. - **52 [M·★★, toolchain] LANDED 2026-08-19, riding the full ride: BK 'built' stage bind-mounts script/module/versions.env (sibling versions.env preferred), classic lane gets builder-classic COPY stage (build.ps1 target updated). Original finding: the toolchain builder never got the bind-mount treatment.** `Dockerfile.toolchain-builder:38-43` COPYs the shared module + versions.env + the build script into the stage whose child RUNs the CPython compile — so editing *any* of them (incl. a module ~30 scripts share) re-pays the full CPython build, and toolchain is the parent of every media branch. `Dockerfile.media-builder:243-259` documents the exact solution. - **81 [S·★, base] DONE - STALE ENTRY: already fixed 2026-08-14 (the SHELL line sets PSNativeCommandUseErrorActionPreference correctly; the entry outlived its fix). Original finding: The base SHELL sets a variable that does not exist.** `Dockerfile.base:58` sets `$PSNativeCommandErrorActionPreference = $false`; the real pwsh variable is **`PSNativeCommandUseErrorActionPreference`** (verified against pwsh 7.6.4, exactly `PWSH_VERSION`: `Get-Variable PSNative*` returns only `PSNativeCommandArgumentPassing` and `PSNativeCommandUseErrorActionPreference`). The assignment creates an unrelated variable and does nothing — the base believes it has a guard it does not have. Harmless *today* only because the real variable already defaults to `False`; the day pwsh flips that default (its stated direction), every native non-zero exit inside a base RUN starts throwing. Note the repo spells it correctly elsewhere (`WindowsFormatting.Common.psm1:279`). Also: all six derived Dockerfiles re-declare `SHELL` and drop the clause — `SHELL` IS inherited via image config, so those are redundant layers against the 125-cap. - **54 [S·★★, merge] DONE 2026-08-20, PREMISE DISPROVEN + RE-SCOPED TO A TRIM: the merge lineage (merge-fanin FROM toolchain; final <- torch <- media) never carries the nvidia originals, so the flatten is the ONLY copy, not a duplicate. The real win: the closure probe (probe-cuda-runtime-closure) showed 13/36 staged DLLs statically imported; the stage now trims the closure-verified-unreferenced, non-dynamic-load families (cusparse/cusolver/cusolvermg/nvjpeg/npps, ~436 MB) and KEEPS all cudnn_* + the nvrtc JIT chain (dlopened at runtime; unverifiable on this GPU-less host). Next merge build. Original finding: `cuda-runtime-stage` ships a SECOND, flattened copy of the CUDA + cuDNN runtime DLLs** (`Dockerfile.media-merge-builder:138`); cuDNN's set alone is 0.52 GB uncompressed, plus CUDA 13's cublas/cufft/cusolver/nvrtc. The originals are still in the image (merge descends from the nvidia stage) and `Dockerfile.nvidia:97` already PATHs them. One extra PATH entry for cuDNN's nested layout likely replaces the whole stage. NOTE: verify the actual cuDNN 9 nesting against the installed tree before removing the stage — the flatten fix was load-bearing for OpenCV's `cudnn64_9.dll`. - **100 [M·★★★, media-core] SOLVED 2026-08-20 after 5 probe rounds - make-time launcher ON, 2198/2198 compile requests through sccache (was 0 forever):** the crash trigger was `-options:strict` - cl.exe-only; clang-cl parses the PREFIX as the deprecated `-o` (output). Bare builds survived by ORDER (the later -Fo wins); sccache's generate_compile_commands REORDERS (-Fo first), the hijack wins, and the object lands in an NTFS ALTERNATE DATA STREAM (`ptions:strict.obj`, literally recovered by Test-SccacheOptionsStrict.ps1) at exit 0 -> 'failed to zip up compiler outputs'. Chain fix: the flag is stripped in Remove-MakefileShowIncludes (a bare-build correctness fix too - the silent -o hijack was always there, just overridden). configure stays bare (its own tests still break through sccache). UPSTREAM angles (owner decides, possible PR 3): (a) sccache's -Fo-first reorder is a semantic hazard for ANY unknown flag whose prefix parses as -o; fix = emit the output flag AFTER the forwarded args; (b) sccache never logs the spawned compile line even at trace. Hit-run VERIFIED 2026-08-20: 2198/2198 hits, 100.00%, 0 misses. PyAV remains the open lower-value half. Previous state: (1) configure --cc='sccache clang-cl': configure's own compiler tests produce objects lld-link rejects ("unknown file type"); (2) make-time CC override: dies ~20 files in with sccache "failed to zip up compiler outputs" on the RELATIVE forward-slash -Fo outputs (libavdevice/dshow*.o resolved to C: emp\...\libavdevice/dshow_pin.o, file absent) - and the bare `make install` then silently recompiled everything launcher-less (15 min), so both "green" rides were uncached anyway. Next angle: sccache-side (does it mishandle relative -Fo through a server whose cwd differs? possibly upstream PR 3 material - owner decides); PyAV unchanged. Original finding: FFmpeg and PyAV compile with sccache COMPLETELY BYPASSED — the whole ffmpeg branch is uncached, every build, forever.** Measured 2026-08-15 in the #99 verification run: the `media-core-built-ffmpeg` stage reported `Compile requests 0` — not "0 hits", *zero requests*. sccache never saw a single compile. CAUSE: sccache is wired **only** through CMake, in `WindowsBuild.Common.psm1:642-643` (`CMAKE_C_COMPILER_LAUNCHER` / `CMAKE_CXX_COMPILER_LAUNCHER`). FFmpeg does not use CMake — it configures with `--toolchain=msvc --cc=clang-cl --ld=lld-link` (`Build-FfmpegFromSource.ps1:317`), so every one of its C files goes straight to `clang-cl`. PyAV is the same story from the other direction: it builds through setuptools, which invokes MSVC `cl.exe` directly (visible in the same log right before `Staged wheel: av-18.0.0-…`). WHY IT WENT UNNOTICED: the chain's aggregate hit rate looks excellent (onnx 1498/1498, opencv 1861/1862) precisely BECAUSE the uncached components contribute no requests to the denominator. A component that bypasses sccache entirely is invisible in a hit-rate metric — it can only be seen by reading `Compile requests` per stage. Cf. the AGENTS.md "aggregate evidence" rule. FIX TO TRY: pass the launcher into FFmpeg's own configure — `--cc="sccache clang-cl"` (FFmpeg's configure tolerates a launcher prefix in `--cc`; verify `ffbuild/config.mak` afterwards, and that `--ld` stays bare). For PyAV, setuptools honours `CC`/`CXX` only on non-MSVC; the realistic lever is a compiler shim on PATH, so treat PyAV as a separate, lower-value item. VERIFY BY: `Compile requests` > 0 for the ffmpeg stage — that number, not the hit rate, is the acceptance criterion. Do NOT accept a rerun with a warm cache as evidence: a stage with 0 misses writes nothing and proves nothing (that trap cost two stages' worth of "0 write errors" in this very run). ## P4 — Missing regression tests (each maps to a bug that already cost hours) - **59 [S·★★, none] CLOSED 2026-08-17 — owner decision: no branch protection wanted. Lint/tests are advisory, not gating.** `main` is not branch-protected (`gh api …/protection` → 404); `windows-scripts.yml:50` runs the linter WITHOUT `-FailOnAnalyzer`; `.githooks/pre-commit` runs the Linux preflight but neither `Invoke-Lint.ps1` nor `Invoke-Tests.ps1`. The gate is currently human discipline plus a post-hoc notification. ## P5 — Observability (makes everything above measurable) - **61 [M·★★★, none] DONE 2026-08-17 (first manifest lands with the next driver run) — stage logs now carry a per-run id (bk--