# Windows backlog archive — closed items and history (archived 2026-08-11)

Everything below was the live "Refactor Backlog (Windows container chain)"
section of docs/windows-builds.md until the 2026-08-11 close-out. All items
annotated *(DONE ...)* are completed and verified (gates held after every
sub-batch: Invoke-Lint 141 files 0/0 incl. the AST-trap pass, 457/457 unit
tests, Test-PatchesApplyClean 12/12). Consult for evidence and rationale;
do not resurrect items without re-verifying against the current tree.
The lean OPEN-only backlog lives in docs/windows-builds.md § Refactor Backlog.

---

## Addendum — closed 2026-08-14 (Batch D: nothing verified the artifact)

Four items treated as ONE piece of work, because they all live in
`Test-Container.ps1`. **The gate found 8 real failures in the shipped
image on its first run** — see backlog P0c (#83-#87).

**The gate's own first lesson — 8 reported failures, ZERO image defects.**
Pointed at `bk-winamd64` the gate reported 8 failures, and they were written
into the backlog as P0c "the shipped image fails 8 assertions". **That writeup
was wrong in full, and is retracted.** Final state after diagnosis:
**184 passed, 0 failed, 1 skipped — "All smoke tests passed!"**

- **Six were the GATE's fault.** A bare Dockerfile `RUN` bypasses `ENTRYPOINT`,
  and `entrypoint.cmd` is what loads VsDevCmd (`-arch=amd64`) and prepends
  LLVM's versioned clang_rt ASAN dir to PATH. So msbuild-on-PATH,
  `VCToolsInstallDir`, MSBuild+ClangCL, nvcc (it needs `cl.exe`) and the ASAN
  probe all failed against a perfectly good image, because the gate was testing
  a configuration no user ever runs. Fixed by invoking through
  `C:\temp\scripts\entrypoint.cmd` → 176→182 passed.
- **Two were STALE ASSERTIONS, also not image defects.** Probing the freshly
  built base settled both:
  - *vcpkg zlib* — the port DOES install, as
    `installed\x64-windows\lib\z.lib` (+ `debug\lib\zd.lib`). Upstream vcpkg
    switched zlib's output to the Unix-style name; the check still looked for
    `zlib.lib`. Now accepts either, so the next rename does not re-open it.
  - *SCOOP_GLOBAL_SHIMS* — the `--global` install works
    (`C:\ProgramData\scoop\apps\flutter` exists, `flutter` resolves by name);
    scoop simply never creates a global `shims` dir in this configuration. The
    assertion tested an implementation detail of scoop instead of the outcome.
    Rewritten to assert the GOAL — a `--global` package resolves by NAME — with
    the shims dir as one acceptable way of getting there.

**Generalised rule, the second instance in one day** (after the ANSI-marshalled
`LoadLibraryW` probe): **a new gate reporting a pile of failures is far more
likely mis-configured than the artifact is broken.** Both times the fix was to
reproduce the real usage path — a known-good control, then the actual entrypoint
— rather than to "fix" the thing under test. Publishing the first reading as
fact cost a retraction; probe first, publish second.

- **44 (DONE) The smoke test was never invoked, and SKIP was not fatal.**
  `grep -i smoke windows/build.ps1 windows/build-buildkit.ps1` returned **zero
  hits in both**; the final Dockerfile has no RUN; CI never referenced it. A
  ~5 h chain ended with "Done" and zero evidence the image worked. Worse, the
  verdict read only `$summary.Failed`, so a run where every section skipped
  printed "All smoke tests passed!" and exited 0 — with 24 `Skip-Test` sites and
  seven env-gated sections that is a reachable shape, not a hypothetical.
  Now: `-MinPassed` / `-MaxSkipped` coverage floors plus an `Aborted` check,
  reported as **exit 3 (INSUFFICIENT COVERAGE)** distinct from exit 1
  (failures) — "0 failures with too little executed is indistinguishable from a
  broken harness". Wired into the BK driver after `final` via a new
  `windows/Dockerfile.smoke-gate`, run as a **buildctl solve, not `nerdctl
  run`**, because containerd's pipe is admin-only while the driver is
  deliberately non-admin. `-SkipSmokeGate` exists for chain iteration and says
  so loudly. The gate **bind-mounts the current script and modules** instead of
  running the copy baked into the image: otherwise a fix to the smoke test could
  not be re-verified without first rebuilding the whole image — exactly the
  friction that let it go unrun for a month — and a mount adds no layer, so the
  gate never alters the artifact it verifies.
- **57 (DONE) Bulk DLL-load enumeration.** New `Assert-AllDllsLoad` walks a root
  and LoadLibrary's EVERY DLL, with an explicit `-Allow` list for
  unloadable-by-design plugins and a `-MinimumChecked` rot guard so a moved root
  cannot pass vacuously. Wired into the OpenCV section, where exactly ONE of
  ~25-30 module DLLs (`opencv_core`) had been load-tested and the rest were
  existence checks — the OPENGL32 defect verbatim. **First run: 65 OpenCV DLLs
  loaded, 0 failures.** Note the probing happens OUTSIDE the `Assert-Test`
  condition on purpose: `-FailMessage` is a plain string evaluated at call time,
  so a message built from results computed inside the condition would always be
  empty.
- **46 (DONE) DirectML could vanish with zero red at either end.** The smoke
  test skipped itself when `DirectML.dll` was absent — keyed on the very
  artifact it exists to verify — while the staging helper only `Write-Warning`s
  on a missing sidecar. `USE_DML=ON` is unconditional in
  Build-OnnxFromSource.ps1, so an absent redist is never legitimate, and on
  the AMD reference host DirectML is the ONLY working GPU path. Now a hard
  failure.
- **67 (DONE) LiteRT exports are pinned.** The build gates on
  `tensorflowlite_c.lib` being installed, but the documented failure was an
  import lib that existed while the DLL exported ZERO C-API symbols — invisible
  to a presence check, surfacing one branch later in gst's meson link. Now
  asserts the DLL exists and exports `TfLiteInterpreterCreate`,
  `TfLiteXNNPackDelegateCreate` and `TfLiteXNNPackDelegateOptionsDefault` — the
  three the `/EXPORT:` injection forces.

## Addendum — closed 2026-08-14 (Batch G: the test net)

Five items, tests **486 → 493**, no rebuild required. #59 deliberately NOT
closed — see the end.

- **55 (DONE) `.gitattributes` did not cover COPY-reachable files.** The gap was
  live, not theoretical: `windows/scripts/patches/litert-lm/` held ONE CRLF
  `.cmake` next to five LF siblings with `attr/` unspecified on all six, and
  `core.autocrlf=true` on the host — a fresh clone (this machine, or GitHub
  `windows-latest`) would have written CRLF for all six, flipping bytes on five
  and busting media-litert plus everything downstream. **The new test found
  more than the audit did:** nine files across `windows/scripts` and
  `windows/downloads` had a worktree/index EOL mismatch, including
  `001-allow-msys-builds.patch` — already `-text`-protected, yet mismatched —
  and `downloads/README.md`, which rides `COPY downloads C:\temp\downloads`
  into a layer. All nine normalized to LF (byte-identical to the index, so zero
  diff; verified with `git diff --quiet` + md5, since `git status` reports a
  stale `M` after an attribute change). Rules are now **path-based**
  (`windows/scripts/** -text`, `windows/downloads/** -text`) so a new file type
  needs no maintenance. `Dockerfile.EolAttributes.Tests.ps1` walks the real
  COPY instructions rather than a hand-kept list.
- **56 (DONE) CMake patchers had 12 replace-ops and ZERO no-op guards**, while
  printing "Patched …" unconditionally. New `patches/litert-lm/patch-assert.cmake`
  provides `patch_replace_required` / `patch_regex_replace_required`, which
  `FATAL_ERROR` when the pattern matched nothing. It lives INSIDE `litert-lm/`
  on purpose: the Dockerfile COPYs that directory specifically, so a helper one
  level up would not be in the image. Verified with real cmake: pattern present
  → exit 0; pattern absent → **exit 1**, build aborts. Applied to all four
  source-rewriting patchers; the `string(REPLACE "|" ";")` in
  `rust-lib-stage.cmake` is a local list split, not a source patch, so it
  carries an explicit `patch-assert-exempt` marker — the exemption is on the
  record rather than a silent gap. `Patches.CmakeNoOpGuards.Tests.ps1` fails any
  future unguarded, unmarked replace. This matters most for sentencepiece's
  duplicate `ABSL_FLAG(minloglevel)`: a silent no-op there ships a link-clean
  `litert_lm_main.exe` that aborts on every run.
- **58 (DONE) `CUDA_ARCHITECTURES` is now pinned at the SOURCE OF TRUTH.** The
  only prior assertion checked the code FALLBACK with the env var cleared —
  and the very next test proved the env value overrides it, so trimming
  versions.env kept the whole suite green and `sync_versions.py --write` would
  have propagated the trim into the Dockerfile. Now asserts
  `versions.env == 80;86;89;90` **and** that the Dockerfile ARG default matches.
- **60 (DONE) `Dockerfile.media-merge-builder` ARG parity.** It was opened by no
  test at all (`BuildKit.TwinParity` hardcodes the media-BUILDER path; PinParity
  never reads a Dockerfile) — in exactly the stage where the "~8 versions.env-bump
  breaks" landed. Now every ARG default that names a versions.env key is
  compared, with a rot guard (`>=8` comparable ARGs) so a restructured ARG block
  cannot make it pass vacuously. Independently cross-checked: 9 ARGs compared,
  all in sync.
- **82 (DONE) The lint gate crashed intermittently** — ~2 in 9 runs, PSSA 1.25.0
  throwing `Object reference not set to an instance of an object.`, never the
  same file twice, always clean on re-run. An exception is not a finding, so it
  aborted the gate for a reason unrelated to code quality. Now: retry the file
  once, and on a second failure report an **INFRASTRUCTURE FAILURE** and
  `exit 2` — distinct from `exit 1` (lint failure) so CI can tell a tooling
  fault from a code problem, and so "LINT OK" is never printed over files that
  were never analysed. Files are never silently skipped (that would fail open).
  **Caveat: the retry path is UNEXERCISED** — four consecutive runs after the
  change were clean, so the crash branch has not actually fired yet.

**#59 NOT closed — owner action.** `main` is not branch-protected
(`gh api …/protection` → 404) and `windows-scripts.yml` runs the linter without
`-FailOnAnalyzer`. Both are repo-settings/policy decisions rather than code, so
they are left for the owner: enable branch protection with both jobs required,
and decide whether analyzer findings should be fatal.

## Addendum — closed 2026-08-14 (#48: two host gates that failed OPEN)

Both were host-side only (`WindowsBuildDriver.Common.psm1` is COPY'd into no
image), so they were safe to land while a full chain was mid-flight — the media
scripts and patch dirs are bind-mounted and were deliberately left alone.

- **`Assert-ShimPatch` returned GREEN when it could not find the shim** — a
  `Write-Warning` and `return`. The gate exists because a STOCK shim kills heavy
  RUN layers with `ExportLayer 0x3` *after* the compile is already paid for, so
  "could not check" is the one answer that must not read as "fine". Worse, the
  hardcoded `$env:ProgramFiles\Stevedore\...` default missed a `D:\Stevedore`
  host — a layout the SAME driver explicitly supports when resolving buildctl.
  Now: probe both candidate roots, and **throw** when neither has one, with
  `-SkipHostChecks` as the deliberate override. (`$advice` had to move above the
  branch, since the new throw quotes it.)
- **The per-stage disk gate only ever looked at `C:`.** `[string]$Drive = 'C'`
  and neither caller passed anything, while the LAUNCH gate passes
  `-Drive @($repoRoot)` — it learned the lesson the hard way: the build context
  is the repo checkout, on the reference host a dynamically-expanding VHDX at
  `D:`, which "fell to 11.7 GB free while a C:-only gate reported everything
  fine". The per-stage gate exists *because* the launch gate is not enough, and
  it was watching the wrong disk. Both lanes now derive the drive from
  `$repoRoot` (verified: `D:\GitHub\…` → `D`). Its silent `return` on an
  unreadable drive now warns first — in a multi-MB build log, no output is
  indistinguishable from "plenty of space".

## Addendum — closed 2026-08-14 (Batch A / #71: sccache was NOT broken)

- **71 (CLOSED — DISPROVEN, not fixed).** The log forensics reported the
  single most alarming number in the whole audit: `Cache hits` = **0** in
  **94/94** stat blocks, `Cache misses` = `Cache write errors` = **189,861**
  (exactly equal — every write failing), read errors 0. It was ranked the
  highest-leverage item in the backlog and credited with 18.42 h of repeated
  ONNX compilation.

  **It was stale evidence.** The newest log carrying sccache stats is
  `bk-run-xnnpack.log`, **2026-08-13 19:43**. The dufs WebDAV server was
  migrated to a session-independent SYSTEM ONSTART task that same day —
  specifically to stop mid-run write failures — and every run afterwards had
  media-core CACHED. **No run has exercised sccache since the fix**, so the
  corpus was structurally incapable of showing the improvement. Aggregating
  across it produced a confident conclusion about a state that no longer
  existed.

  **Empirical disproof** (probe: `bk-windows-base`, the real cache mount
  `id=sccache-winamd64-2`, the real WebDAV endpoint, one TU compiled twice with
  `sccache clang-cl`):

  ```text
  Compile requests 2 | Cache hits 1 | Cache misses 1 | hit rate 50.00 %
  Cache read errors 0 | Cache write errors 0 | Cache errors 0
  ```

  Miss → stored → second compile **HIT**. Supporting checks: `HEAD`/`GET`/`PUT`
  all succeed **from inside a container** (the direction `Assert-SccacheEndpoint`
  never tests — it probes from the host, which is why a container-side outage
  could never have been caught by the gate), and dufs auto-creates nested parent
  directories on PUT, so sccache's `/6/e/b/<hash>` layout is fine. The cache
  mount itself was found **EMPTY**, which is also why `SCCACHE_ERROR_LOG` was
  absent from all 102 logs — there was never a log to recover.

  **What this does NOT clear.** The probe used clang-cl on a trivial TU. The
  sccache-**CUDA** server crash (±2 s deterministic, ~4909-4911 s into ONNX)
  and the `-j9 → -j2 → -j1` silent downgrade ladder it triggers (#75) are
  untouched by this result, as is the reported nvcc miscompile behind
  mozilla/sccache#2808. The at-scale hit rate stays **unmeasured** until the
  first real media build after 2026-08-13.

  **Method note worth keeping.** Two probes in a row were wrong before this one
  was right: the first mounted `target=C:\sccache` in a hand-written Dockerfile
  with no `# escape=` directive, so `\` ate the `s` and the mount silently did
  not exist (`MOUNT_EXISTS=False`); the second used `/Fo:C:\probe.obj`, and
  sccache turned that into `C:\:C:\probe.obj` and failed with the misleading
  "failed to zip up compiler outputs". **Use `/Fo<path>` without a colon, and
  never hand-write a Windows Dockerfile without `# escape=\``.** Both failures
  looked like product defects and were tooling defects — the same shape as the
  `LoadLibraryW` ANSI-marshaling mistake in the TensorRT entry below.

## Addendum — closed 2026-08-14 (Batch C: the TensorRT pin defect)

- **53 (DONE) `TENSORRT_VERSION` ARG+ENV moved BELOW the CUDA RUN.** An ARG in
  scope keys every following RUN, and nothing up to and including
  `Install-Cuda.ps1` consumes it — TensorRT arrives via `COPY --from`. With it
  declared above, bumping the TensorRT pin forced a ~3.8 GB CUDA + cuDNN
  re-download, which is precisely what made fixing #38 expensive. It now rides
  the existing `GPU_TYPE`/`CUDA_ROOT` ENV block, so **no extra layer** is spent.
  Landed FIRST, on purpose.
- **38 (DONE) The shipped image had a DEAD TensorRT EP.** `versions.env` pinned
  `11.2.1.2` while the staged archive was `TensorRT-Enterprise-11.1.0.106`.
  Two contradictory resolutions of the same directory existed:
  `Resolve-TensorRtRoot` globbed `TensorRT-*` (pin-INdependent → correct) while
  `Dockerfile.nvidia` built the runtime PATH from the PIN → a directory that
  did not exist. The build stayed green — logs confirm ×24 `TensorRT detected
  at C:\tensorrt\TensorRT-11.1.0.106` **and** `onnxruntime_USE_TENSORRT=ON`, so
  ONNX compiled the EP against the real 11.1 headers — and only the RUNTIME
  lookup broke, so ORT dropped the EP with no error. A REGRESSION of the
  incident `versions.env:238-248` documents verbatim ("Bump this WITH the
  staged zip, never alone"): the pin was bumped alone, because nothing
  ENFORCED the rule.

  **A Machine-PATH write does NOT fix this** (the first attempt, discarded):
  `Dockerfile.base` sets `ENV PATH=...` explicitly, so the image config wins and
  a registry PATH written inside a RUN is invisible to later stages. The fix
  that works *with* the ENV is a stable directory NAME:
  - NEW `windows/scripts/Set-TensorrtTree.ps1`, bind-mounted into the
    `trt-extract` stage: renames the NEWEST `TensorRT-<version>` tree (by
    `[version]` comparison, not a lexical sort — see the correction below) →
    **`current`**, WARNS (never fails) on pin-vs-zip drift so it can no longer
    be silent, folds a FLAT layout into `current` as well, and **fails closed**
    when neither `bin\` nor `lib\` carries runtime DLLs.
  - `Dockerfile.nvidia` PATH is now
    `$TENSORRT_ROOT\current\bin;$TENSORRT_ROOT\current\lib` — `bin` first
    because that is where TensorRT 10+ puts the runtime DLLs; `lib` kept for the
    8.x/9.x layout. Correct for any staged version, forever. The zip-less
    graceful-skip contract is unchanged (the directories simply do not exist and
    the entries are inert).
  - `Resolve-TensorRtRoot` prefers `current`, falling back to the versioned glob
    so pre-normalization images and host-lane trees keep resolving. Two new
    tests pin both behaviours (486 total, was 484).

  **THE BUILD FOUND A SECOND, BIGGER DEFECT — the fail-closed gate earned its
  keep on its first run.** The sdk rebuild failed with
  `TensorRT lib directory 'C:\tensorrt\current\lib' contains no DLLs`.
  Inspecting the staged 11.1.0.106 Enterprise zip:

  | dir | contents |
  |-----|----------|
  | `lib\` | **6 × `.lib`** — link-time import libraries only |
  | `bin\` | **14 × `.dll`** — the runtime DLLs |

  TensorRT 8.x/9.x shipped the runtime DLLs in `lib\`; **10+ moved them to
  `bin\`**, and `Dockerfile.nvidia` never caught up. So the PATH entry was
  broken in TWO independent ways — wrong VERSION *and* wrong DIRECTORY —
  meaning **the TensorRT EP could never have loaded at runtime in this image,
  even with a correctly-pinned zip**. The pin mismatch merely hid the second
  defect behind the first. Neither failed a build, because ONNX resolves its
  BUILD-time root with a glob and only the RUNTIME lookup was wrong.
  Final form: PATH gets `current\bin` (runtime) **and** `current\lib` (kept for
  the 8.x/9.x layout), and the normalizer requires DLLs in at least one of them.

  **CORRECTION (independent review, same day).** The first version of this
  script had two defects the review caught: it sorted the versioned trees
  **lexically ascending and took the FIRST**, i.e. the OLDEST — directly
  contradicting the owner's "always newest" directive one file over; and its
  flat-layout branch printed "consumers use the root directly" while the ENV
  PATH pointed at `current\bin`/`current\lib`, which on a flat tree do not
  exist. Two early `return`s also bypassed the fail-closed DLL gate. All fixed
  and re-verified across five synthetic trees: newest-of-two wins
  (`11.10.0.1` over `11.2.1.2` — a lexical sort ranks these backwards), flat
  layout folds into `current`, and a pre-existing `current` without DLLs now
  exits 1 instead of returning early.

  **VERIFIED, end to end.** Synthetic trees on the host (normalize + drift
  warning, tree-without-DLLs → **exit 1**, tree-without-`lib\` → **exit 1**,
  empty root → exit 0 graceful). Then the real sdk build:
  `TensorRT 11.1.0.106 normalized to 'C:\tensorrt\current' (14 runtime DLLs in:
  bin)`, `[bk] Done in 00:12:00. Stages: sdk (GPU)`. Then a LOAD test against
  the produced image — the only proof that actually matters for this defect
  class:

  ```text
  TRT_PATH_ENTRIES=C:\tensorrt\current\bin,C:\tensorrt\current\lib
  CONTROL LOADED VCRUNTIME140.dll
  TRTSUMMARY loaded=14 failed=0
  ```

  **Methodology note, worth keeping.** The first load probe reported
  `LoadLibraryW` error 126 for all 14 DLLs and briefly looked like a third
  defect. It was the PROBE that was wrong: a hand-rolled
  `[DllImport("kernel32")] LoadLibraryW(string)` marshals `string` as **ANSI**
  by default, so a UTF-16 API received a byte string and answered "module not
  found". The repo's own `Assert-DllLoads`
  (`WindowsSmokeTest.Common.psm1:233`) has always declared it correctly with
  `CharSet=CharSet.Unicode`. Two rules follow: **use the existing helper rather
  than re-declaring P/Invoke**, and **always run a known-good control** (here
  `VCRUNTIME140.dll`) — the control is what exposed the harness, since the MSVC
  runtime obviously loads in an image built with MSVC.
  **DRIFT RESOLVED THE SAME DAY — and it set a standing rule.** The owner
  staged the newer **11.2.1.2** zip rather than accept a downgrade:
  *"always use the newest tensorrt version"*. **Never resolve TensorRT
  pin-vs-zip drift by pinning BACKWARDS** — that directive is now recorded at
  the `TENSORRT_VERSION` definition in versions.env. Three follow-ons landed
  with it:
  - `TENSORRT_ZIP_SHA256` was **EMPTY**, so ~2 GB of EULA-gated payload entered
    the image unverified. Now set (`9cdb433f…`) and verified by the build.
    **Recompute it on every restage** — a stale hash fails loudly, as intended.
  - The superseded 11.1.0.106 zip was removed from `windows/downloads/` (the
    owner's original copy remains in `~\Downloads`).
  - The extract RUN selected the zip with `Select-Object -First 1`, which
    silently picks the OLDER file when two are staged. It now **version-sorts
    descending** via a `[version]` cast — a plain string sort gets this wrong
    (`11.10.0.1` must beat `11.2.1.2`; verified against four real filenames).

  **Final state, verified in the produced image:**

  ```text
  VERSION_ENV=11.2.1.2   TREE=current
  CONTROL=LOADED VCRUNTIME140.dll
  RESULT loaded=14 failed=0
  ```

  The rebuild took **2:24 vs 12:00** because `Install-Cuda.ps1` came back
  **CACHED** — #53 paying for itself on its first real use: a TensorRT restage
  no longer re-downloads ~3.8 GB of CUDA + cuDNN.

## Addendum — closed 2026-08-14 (Batch B: driver/log hardening, zero rebuild)

Eight backlog items landed in one sitting. Selection criterion: **none touches a
Dockerfile**, so nothing invalidated a cache layer and the whole batch was
verifiable with lint + unit tests alone — no container build. Result: lint
152/0, tests **484/484** (up from 480; the 4 new ones are #40's regression
guard). All eight came out of the 2026-08-14 deep audit.

- **41 (DONE) The retry path destroyed the failing attempt's log.**
  `build-buildkit.ps1` teed without `-Append`, so attempt 2 truncated attempt 1
  — a stage that burned its budget kept only the last attempt, and when
  attempt 1 held the real compile error while 2-3 died on infra flakes the
  evidence was gone. Now: log cleared once per RUN, appended per ATTEMPT with a
  `===== attempt N/M =====` banner. Direct "never swallow logs" fix, inside the
  path that exists to survive failures.
- **42 (DONE) The failure tail was computed, then thrown away — both lanes.**
  `$tail` was built purely to CLASSIFY the error and then discarded, so the
  throw carried a bare path (BK) or a bare exit code (classic) and the owner
  opened a deliberately-unbounded log by hand. Both lanes now print the tail
  before throwing, and the classic lane's throw names the log file.
- **43 (DONE) 4 of 5 ninja logs died with the build tree; genai had none.**
  Only ONNX wrote to the persistent `$SCCACHE_DIR\logs`; opencv/iree/tvm/litert
  wrote inside `$buildDir`, which is discarded with the failed solve (IREE is a
  60-120 min build whose only surviving diagnosis was a 50-line tail), and
  `build-onnx-genai` passed no `-LogFile` at all, so not even that existed.
  **Fixed as a SHARED helper, not five copies:** `Get-PersistentBuildLogPath`
  in `WindowsSourceBuild.Common.psm1`, with all six call sites converted —
  including ONNX's original inline block. Copy-paste is exactly how this drift
  happened; there is now one implementation to drift from.
- **39 (DONE) `-MediaBranches <subset>` silently shipped a STALE image.** The
  merge is skipped on a branch subset, but torch/final still resolved
  `BASE_IMAGE` from the `windows-media` tag — the PREVIOUS run's merge. So
  "I fixed LiteRT, re-ship" delivered a `winamd64` without the fix, with a zero
  exit code. Now throws when the merge is skipped while torch/final are
  selected, naming both the subset and the missing branches.
- **40 (DONE) The scripted resume never worked (closure scope).**
  `.GetNewClosure()` snapshots LOCALS only, but
  `$Docker`/`$MediaCoreCpus`/`$MediaMemoryGb`/`$ResumeStage` are script-level
  `param()` vars → empty inside the module invoker, degrading the run to
  `[] run --isolation hyperv --cpu-count  --memory "g"` and dying with a
  PowerShell *parser* error that pointed nowhere. The identical fix had been on
  the sibling `Invoke-RunCommitStage` since 2026-07 *with an explanatory
  comment*; this path never got it. Fixed with four local copies. **Note the
  first draft of this entry overstated it:** the `docker container rm -f` is NOT
  data loss — the state is committed to `$partial` and exit-code-checked first,
  and the removal is required to reuse the container name.
  **NEW TEST — guards the CLASS, not the instance:**
  `tests/Driver.ClosureScope.Tests.ps1` walks the AST of both drivers, finds
  every `.GetNewClosure()` block *inside a function*, and fails if it reads a
  top-level `param()` variable that is not shadowed by a local. It correctly
  ignores top-level closures (there, params ARE locals) and locally-shadowed
  names like `$isolation`; it carries its own negative control (a synthetic
  re-introduction of the defect must be caught) and a false-positive control
  (the sibling's correct pattern must NOT be flagged).
- **62 (DONE) `-ConcurrentAux` dropped six parameters — three guaranteed a
  failure AFTER media-core was paid for.** Children re-run the full preflight
  with defaults, so `-SkipHostChecks`, `-SkipRdna4Gate`, `-SkipStepLogGate`,
  `-NoSccache`, `-MinFreeGb`, `-HostReserveGb` now forward.
  `-ConcurrentAux -NoSccache` previously could NEVER succeed. Same change adds
  **fail-fast** (a child dying at minute 5 was unnoticed until the other
  finished ~40 min later) and a **`finally` that stops surviving children** —
  they were spawned outside any `try`, so killing the parent orphaned two
  pwsh+buildctl trees racing the same store.
- **63 (DONE) Every preflight-gate failure orphaned a hidden sampler.** It
  started ~500 lines before the `try` that owns it, so each rejected launch
  (sccache down, RDNA4 enabled, disk short, dockerd stopped — all common) left
  an invisible `-WindowStyle Hidden` pwsh in a `while ($true)` loop forever, one
  per failed attempt. Moved to immediately before the `try/finally`; the
  preflight costs seconds, so no sample coverage is lost.
- **64 (DONE) The driver prescribed a remedy it could not express.** The
  determinism gate says "the fix is `-NoCache` on this stage alone, NOT a
  retry", but `-NoCache` applied to every solve — for a poisoned
  `media-core-built-opencv` the only lever was `-Stages media -NoCache`, which
  re-does all four media-core sub-stages plus litert plus tvm plus merge. Added
  **`-NoCacheStage <label[]>`**, substring-matched against the same `$Label` the
  stage logs and disk gate already use (`-NoCacheStage opencv` works).

## Addendum — closed 2026-08-13 (`:winamd64` green end-to-end)

- **34 + ELEVATED WINDOW + dufs (DONE 2026-08-13, applied by owner).** Owner ran
  the between-runs elevated bundle: (1) buildkitd service env restored
  (`BUILDKIT_STEP_LOG_MAX_SIZE/-SPEED=-1`) so `-SkipStepLogGate` is no longer
  needed; (2) GC budgets deployed (`Set-BuildkitdGcpolicy.ps1`, buildkitd.toml
  400/450GB) — fixes the cross-run snapshot eviction (#34); (3) poisoned
  probe-build-copy layer chain pruned (`Test-BuildCopy.ps1` no longer FALSE
  RED); (4) diagnostic tag cleanup. Also ran `Install-DufsService.ps1` → dufs is
  now a session-independent SYSTEM ONSTART task (no more mid-run WebDAV-write
  fail-open). Sanity: `buildctl debug workers -v` should show reservedSpace=200GB.
- **Upstream issues (POSTED 2026-08-13):** mozilla/sccache#2808 (nvcc deadlock +
  miscompile) and google-ai-edge/LiteRT-LM#3245 (CMake-lane staleness). opencv/opencv
  (dnn ORT `char*`/`wchar_t`) draft still unposted.
- **28 (LANDED 2026-08-13) Ninja job-count `-MemGBPerJob 2`.** Changed `4`→`2`
  at the Build-OnnxFromSource.ps1 (line ~451) and Build-OpencvFromSource.ps1
  (line ~295) `Invoke-NinjaBuildWithRetry` call sites → ~19 jobs (from ~9) at the
  measured ~1 GB/process, well under the 39 GB budget. Code landed + lint/tests
  green; the throughput win verifies on the next full media-core build.
- **27 (DONE + VERIFIED 2026-08-14) Dockerfile.base 1214-char single-line RUN →
  mounted script.** Extracted the pwsh-7 bootstrap blob into
  `windows/scripts/Initialize-Pwsh.ps1` (WPS-5.1-safe, byte-identical logic:
  3-attempt backoff + in-loop SHA256 + original-exception rethrow) and rewrote
  the base RUN to `RUN --mount=type=bind,source=windows/scripts/Initialize-Pwsh.ps1,target=C:\Initialize-Pwsh.ps1 & 'C:\Initialize-Pwsh.ps1'`.
  Bind-mount, NOT COPY — no layer, nothing COPY'd this early in base by design;
  runs under WPS 5.1 (SHELL not yet switched to pwsh). Despite the item's
  "never alone / batch with a base bump" caveat, verified standalone with a
  scoped `-Stages base` build: **step `#6 … RUN --mount=…Initialize-Pwsh.ps1… DONE
  11.1s`**, build proceeded to VS Build Tools (#9) — the mounted-script bootstrap
  behaves exactly as the inline blob did. Lint 151/0, parse-clean. NOTE: landing
  it busts the base-tier cache (base instruction changed) → the next full chain
  rebuilds downstream of base; that cost is inherent to the change, not the
  verify build.


- **37 (DONE 2026-08-13) TVM `import tvm` → WinError 127.** Root cause was NOT
  an LLVM ABI skew (the archived suspicion): TVM 0.25 vendors `3rdparty/tvm-ffi`
  at an UNRELEASED commit (0.1.13.dev1, not on PyPI), so the PyPI `apache-tvm-ffi`
  wheel pip pulled was ABI-skewed vs our source-built `tvm_runtime.dll`. Fix =
  build tvm_ffi FROM the vendored source + install the tvm wheel `--no-deps`
  (Build-TvmFromSource.ps1). Three sub-fixes gated that source build:
  `Copy-CpythonPyConfigHeader` (CMake-4.4 FindPython can't read in-tree
  pyconfig.h), install `cython` (core.pyx transpile → MSB8066), install
  `typing_extensions` (`--no-deps` starves tvm_ffi's only declared dep). See
  memory `tvm-ffi-source-build-winfix`.
- **36 (DONE 2026-08-13) litert branch → BAZEL — chain integration complete.**
  The canary recipe (`Build-LitertLmBazel.ps1`) was wired into the media-litert
  branch and the full chain built green; media-litert now produces
  litert_lm_main.exe via bazel, CMake path frozen as fallback. Included the
  zlib.net→GitHub-mirror WORKSPACE patch (flaky-download fix).
- **(NEW, not previously numbered) GStreamer merge stage — 8 versions.env-bump
  regressions** that blocked `:winamd64`, all fixed 2026-08-13: /LIBPATH-breaks-
  clang-cl, opencv-5 data-dir + API relocation (xobjdetect/geometry/calib),
  tflite C-API export, cpp_std c++11→c++17 (MSVC 14.51 STL), plugin-LOAD gate
  (recursive dumpbin walker), and CUDA-runtime FLATTEN-deploy for opencv's
  cudnn64_9.dll (`Copy-CudaRuntime.ps1`). Full detail in memory
  `gstreamer-merge-winfix`. Verified: `[bk] Done in 00:41:30` →
  `local/kataglyphis:bk-winamd64` (all stages, GPU).

---

## Refactor Backlog (Windows container chain)

> Cross-lane / Linux-side items live in [docs/refactoring-backlog.md](refactoring-backlog.md);
> this section owns the WINDOWS chain exclusively (pointer there exists too).

Owner-requested backlog from the 2026-08-10 systematic code review (8-angle
sweep over `windows/`). Ordering = suggested attack order: correctness-adjacent
first, then reuse, then cosmetics. **Before touching anything, check the
cache-tier map** (AGENTS.md / windows-refactor notes): edits to base/toolchain
closure files force a full chain rebuild — batch those, and never remove the
deliberate media-merge version-ARG mirrors.

**Contents**: P0 architecture (0a-0c) → P1 correctness-adjacent (1-4,
16-19) → P2 reuse (5-9, 20) → P3 hygiene (10-15, 21-25) → second-pass
build-definition/ops items (26-32) → pending host/upstream actions →
already-fixed protocol. Cross-cutting pairs to do together: #1+#18 (RDNA4
single source), #7+#19 (patch-stanza helper incl. hard-fail rung), #8+#30
(log convention + retention).

### Execution guide (effort·impact legend as in docs/refactoring-backlog.md; tier = what a change cache-busts)

Work the backlog in BATCHES keyed on the cache tier — every media-closure
edit costs one ONNX-vertex rebuild, so land them together:

| Batch | Items | Tier | Effort | Impact |
|---|---|---|---|---|
| **W0 pending actions** | buildkitd env (now ENFORCED by the 0a gate — restore before run 13!), sccache issue, tag cleanup | host | S | ★★★ (unblocks log visibility) |
| **W1 host-only quick wins** | ~~2, 5, 6, 8+30, 9, 12, 13, 15, 21, 22, 24, 25, 29~~ ALL DONE | host scripts/tests only — zero cache impact | S-M | ★★ |
| **W2 preflight architecture** | ~~0a, 0c, 1+18, 26, 32~~ ALL DONE | host (drivers, diagnostics, probe assets) | M | ★★★ |
| **W3 media-closure batch** | ~~3, 4, 7+19, 10, 11, 16+17, 20, 23~~ ALL DONE (2026-08-10 night, landed inside run 12's already-busted closure window) | media closure | M-L | ★★★ |
| **W4 base-tier batch** | 27 (+ anything else touching base closure) | base — FULL chain rebuild; batch with the next planned base bump | M | ★ |
| **W5 process/policy** | 0b (CI half DONE — patch-drift job; human bump-protocol half stands), 28 (measurement RUNNING: per-process fleet sampler captured run 12's ONNX compile → `out\build-logs\onnx-tu-memory-samples-20260810.csv`; analyze, then split job pools), 31 (needs the registry-creds decision) | repo/CI policy + measurements | M | ★★★ |

OPEN as of 2026-08-10 night: **NEW 34 [M·★★★, P0] — the chain has NO
cross-run caching; MECHANISM IDENTIFIED (2026-08-11 ~08:15): snapshot GC
eviction under an undersized tier-2 budget, NOT export nondeterminism.**
Evidence chain: (i) prefix stages rebuild every driver run (73/35 min
instead of seconds, zero manifest-digest overlap between runs 14/15);
(ii) back-to-back canary solves show digests ARE stable on cache-hit
re-export (with AND without SOURCE_DATE_EPOCH) — so the exporter is
deterministic given cached snapshots; (iii) the run-over-run delta is the
hours of vertex churn between runs: tier 2 (buildkitd.toml, `keepDuration
720h, maxUsedSpace 150GB`) is far below one night's vertex fleet
(multiple ONNX/OpenCV generations at tens of GB each), so GC evicts the
OLDEST snapshots = the prefix stages; the re-solved layers carry fresh
internal file timestamps → new blobs → new digests → downstream FROMs
miss, cascade. `BUILD_DATE`/`VCS_REF` stamps are exonerated (they reach
only torch/final). FIX (elevated window, bundle with the W0 env restore):
raise tier-2 maxUsedSpace 150→350GB and tier-3 240→450GB in
windows/buildkitd.toml (930 GB disk, ~790 free — satisfiable; keep the
2026-08-08 invariants), redeploy via Set-BuildkitdGcpolicy.ps1.
SIDE-FINDING from the fix research: `rewrite-timestamp=true` on the
Windows image exporter CRASHES mid-finalize and poisons the layer chain —
never use it here (see the probe-chain cleanup pending-action). Item-34 fix research (2026-08-11 ~08:00): **`rewrite-timestamp=true` on the Windows image exporter is DANGEROUS** — it crashed mid-finalize (`hcsshim::ActivateLayer ... process cannot access the file`) and left a POISONED SNAPSHOT in the probe's layer chain (same snapshot id failed identically on the next two solves; a unique-layer discriminator solve was green, so the damage is chain-local, not a host wedge). ⚠ Until that snapshot is pruned (`buildctl prune` of the probe refs / worst case reboot), **Test-BuildCopy.ps1 will return a FALSE RED on this host** — do not trust a probe verdict before the cleanup. Epoch-only canary (no rewrite-timestamp) running separately. **NEW 35 [observe·★]**: run-15's ffmpeg stage stalled
for ~120 min between `Enter-VsDevCmdEnvironment`'s vswhere-fallback line
(15 s) and the awk-replacement line (7216 s) — a section that runs in
seconds normally (run 14: whole prefix 177 s; no scoop-install messages,
so both install branches were skipped). Smells like a stuck TCP/timeout
(VsDevCmd first-run or scoop probe); transient, self-recovered, build
continued green. If it recurs, wrap the section's candidates in explicit
short timeouts. Also **27** (base-tier window), **28** (analyze the
captured samples → pool split), **31** (owner decision), **0b human half**
(bump protocol), **33** *(DONE 2026-08-11 morning: unmapped patch dirs now FAIL with
"add it to $repoMap"; current tree 12/12 mapped+OK.)* Everything else in this
backlog is closed.
Done-when held throughout: gates green after every sub-batch (lint 141
files 0/0 incl. the new AST-trap pass, 457/457 tests), behavior changes in
AGENTS + this doc + CHANGELOG per repo priority 4.

### Pending host/upstream actions (not refactors — do not let these evaporate)

- **Restore the buildkitd service env** (`BUILDKIT_STEP_LOG_MAX_SIZE=-1`,
  `MAX_SPEED=-1`): found EMPTY on 2026-08-10 (wiped by the Stevedore/repair
  work); the default 2 MiB step-log clip hid verdicts all day. Elevated
  `Install-NewHost.ps1` (idempotent, refuses during builds) or the registry
  Multi-String + `Restart-Service buildkitd` — ONLY between chain runs.
  **Since 2026-08-10 night this is ENFORCED: `Assert-BuildkitdStepLogEnv`
  refuses to launch the BK driver until restored (0a).**
- **Make dufs session-independent** (attribution dossier 2.8, 2026-08-10
  night): today it is an ONLOGON scheduled task bound to the RDP session —
  alive at driver-launch time (so Assert-SccacheEndpoint passes), killable
  by a mid-run logoff/lock, and every WebDAV write then fails OPEN
  (multilevel policy `l0`) with zero L2 entries as the only symptom.
  Convert to a Windows service or ONSTART/SYSTEM task + single-instance
  guard (two dufs.exe were live on 2026-08-10). Evidence:
  `out/sccache-fault-attribution.md`.
- **Post the sccache upstream issue**: `out/upstream-issue-sccache-nvcc.md`
  now carries BOTH failure classes — the deterministic server death on the
  fused_moe launchers (±2 s across two runs) AND the silent miscompilation
  of arch-guarded instantiations (runs 10/11 vs bare-nvcc run 5, fresh-cache
  control) — ready to file against mozilla/sccache. The
  docker/for-win#14977 comment is already posted.
- **Admin cleanup of 2026-08-10 diagnostic tags** (`copyprobe-*`, `sweep-*`,
  `rdna4ab-*`, `flush-*`, `size*`, `pw*`, `mlchain-probe`,
  `verify-cuda-cache`, `postboot-*`, `nano-*`, `gpuab-*`, **plus the
  pre-rename stable probe tags `probe-build-copy` and
  `probe-build-copy-heavy`** — orphaned by the #29 `diag-` prefix adoption,
  so the `findstr diag-` one-liner does NOT find them): admin
  `nerdctl --namespace buildkit rmi` per docs § Store GC (see backlog #29
  for the durable convention).

### P0 — architecture-level (highest leverage; from the same review's deep pass)

0a. *(DONE 2026-08-10 night: `Assert-BuildkitdStepLogEnv` (non-admin registry
    read, throws with the elevated remediation, `-SkipHostChecks` downgrades)
    closed the one missing cheap check — disk/shim/daemon/RDNA4/dufs-endpoint
    gates already ran in both drivers. NOTE: the gate now REFUSES to launch
    until the wiped buildkitd env is restored — do W0 first.)*
    **Host-drift detection as a mandatory driver preflight.** Four of the
    2026-08-10 blockers were pure host drift (dufs ONLOGON task dead after
    reboot, buildkitd service env wiped by the Stevedore repair, Defender
    exclusion uncertainty, dGPU state): the code is reproducible, the host
    is not. `Test-HostSetup.ps1` already exists — carve out its CHEAP
    subset (service-env registry read, dufs HEAD request, RDNA4 state,
    shim hash) and run it at the top of BOTH drivers. Seconds at launch
    instead of minute-80 surprises.
0b. *(HALF DONE 2026-08-10 night: `patch-drift` job added to
    windows-scripts.yml — Test-PatchesApplyClean now runs in CI on every
    windows/**/versions.env trigger (shallow+sparse clones, ~minutes). The
    HUMAN half stands: a versions.env bump still needs one local full-chain
    build before trust — CI builds no containers.)*
    **Version bumps must ride the Windows lane.** The 2026-08-03 bump
    (ONNX 1.28, OpenCV 5.0.0) carried FIVE latent Windows breaks because
    `[build-win]` is opt-in and nobody built the bump. Rule: versions.env
    bump commits require `[build-win]` (or a weekly scheduled canary), and
    `Test-PatchesApplyClean.ps1` runs as a pre-commit gate whenever
    versions.env changes — that alone would have caught the OpenCV patch
    drift before any container built.
0c. *(DONE 2026-08-10 night: `Driver.PreflightParity.Tests.ps1` — shared
    contract table + lane allowlists with reasons + a catch-all that fails on
    any NEW unlisted `Assert-*` in either driver; caught its first real case
    (`Assert-ImageExists`) on the first run.)*
    **Lane parity is unowned.** The RDNA4 gate initially landed only in the
    BK driver; the classic lane got it a day later via review. Either demote
    build.ps1 to bootstrap-only, or add a parity test (BuildKit.TwinParity
    is the in-repo pattern) asserting both drivers wire the same preflight
    set.

### P1 — correctness-adjacent (drift that already bites or will)

1. *(DONE 2026-08-10 night: `Get-Rdna4HazardDevice` + `Set-Rdna4DeviceState`
   exported from WindowsBuildDriver.Common are the single source; the gate,
   toggle (empty `-GpuName` default = resolve all hazard SKUs, `-NoPrompt`
   for automation) and the A/B all consume them. +3 unit tests.)*
   **RDNA4 hazard set exists in THREE divergent copies**: the
   `Assert-NoActiveRdna4Gpu` regex (covers RX 9xxx + AI PRO R9700), the
   hardcoded `FriendlyName -eq 'AMD Radeon RX 9070 XT'` in
   `Set-Rdna4Gpu.ps1`, and the same literal as `-GpuName` default in
   `Test-Rdna4LayerLock.ps1`. Concrete dead end: on an RX 9060 host the
   gate refuses and points at a toggle script that cannot find the device.
   Fix: export ONE `Get-Rdna4HazardDevice` from WindowsBuildDriver.Common and
   let toggle + A/B resolve through it. *(The gate regex additionally
   gained `(TM)`-rename tolerance on 2026-08-10 — the other two copies did
   NOT, widening the drift this item exists to close. Do together with
   #18.)*
2. *(DONE 2026-08-10 — candidate lists landed same-day (scope note: the truly single-candidate set was 6); the night batch finished the consolidation: reset-container-stores + verify-cuda-cache now resolve via `Get-PreferredToolPath` (candidates → PATH; `$null`-safe — the old `Test-Path $bt` threw when buildctl was absent), and reset-container-stores' hardcoded `D:\GitHub` repo path became `$PSScriptRoot`-relative. DELIBERATE keepers: probe-build-copy stays inline (first-script-on-a-new-host must be module-free), collect-host-docker-state's glob is forensics enumeration not tool resolution, deploy-shim-patch/verify-host-setup keep their overridable Program-Files param defaults — the shim deploy target must not silently follow a stale D:\ layout)* **The Stevedore tool path was hardcoded single-candidate in several files**
   (2026-08-10 evening sweep: probe-build-copy, verify-cuda-cache,
   test-rdna4-layer-lock, test-layer-rename, test-process-isolation-commit,
   deploy-shim-patch, reset-container-stores, verify-host-setup, ... — only
   build-buildkit.ps1 and friends carry the candidate list incl.
   `D:\Stevedore\bin`). On a D:\ layout the diagnostics throw exactly where
   they are needed, and the constant now has a dozen copies. Fix:
   `Get-PreferredToolPath` (already exported by WindowsScripts.Shared)
   everywhere; one candidate list, zero copies.
3. *(DONE 2026-08-10 night: `Write-SccacheStatsToStderr` in
   WindowsScripts.Shared (name states the sink; the sink-stays-with-caller
   doctrine holds for other consumers); called end-of-build by ALL SIX
   source-build scripts — onnx, opencv, genai, tvm, iree, litert.)*
   **Consolidate the ONNX stderr stats loop into `Write-SccacheStats`**
   (give the helper a `-Sink Stderr` switch and call it) so formatting and
   the `-RequireRemote` contract live in one place and the next CUDA
   consumer (TVM/OpenCV) inherits the clip-surviving sink for free.
   *(The original finding's acute half — the missing `-RequireRemote`,
   which would have spawned a throwaway server on no-remote builds — was
   fixed same-day; only the consolidation remains.)*
4. *(DONE 2026-08-10 night: onnx ninja logs to
   `$env:SCCACHE_DIR\logs\onnx-ninja.log` — the PERSISTENT cache mount, so
   the stream survives a failed vertex into the next run/debug container;
   one `.prev` rotation generation bounds growth.)*
   **ONNX ninja runs without `-LogFile`** — the full ninja stream exists
   only in the (clip-prone) step log; `[n/1891]` progress is invisible from
   the host. Violates the never-swallow-logs invariant. Fix: pass a LogFile
   under the build tree (or out-mounted), always.

### P2 — reuse / single-source-of-truth

5. *(DONE 2026-08-10 night: the A/B now delegates each GPU-state side to
   `Test-BuildCopy.ps1 -Heavy` — digest-pinned base, lane logs and the
   output-shape lessons live once; the A/B only orchestrates GPU state.)*
   **`Test-Rdna4LayerLock.ps1` re-implements the finalize probe** that
   `Test-BuildCopy.ps1` (exit codes + `-Heavy` + per-lane Tee logs) was
   just upgraded to provide. Fix: call the probe (or extract a shared
   lane-runner) so the load-bearing output-shape/quoting lessons live once.
   *(Its log-swallowing was fixed same-day — the duplication remains.)*
6. *(DONE 2026-08-10 night: toggle lifted into the module as
   `Set-Rdna4DeviceState` (post-state-verified); toggle script + A/B finally
   block both consume it, and the toggle gained `-NoPrompt` + exit 1 on a
   failed state change.)*
   **GPU toggle logic duplicated** between `Set-Rdna4Gpu.ps1` and the
   A/B script's finally-block re-enable (the safety-critical path). Fix:
   parameterize the toggle script (`-GpuName`, `-NoPrompt`) and call it, or
   lift the toggle into the module.
7. *(DONE 2026-08-10 night: `Invoke-SourcePatchWithFallback` in
   WindowsSourceBuild.Patches (fallback scriptblock runs in caller scope,
   `-Fatal` throws on a double miss); all six build-onnx stanzas collapsed;
   4 unit tests incl. the Fatal rung. See #19.)*
   **Patch-apply stanzas ×6 in Build-OnnxFromSource.ps1** (try →
   Invoke-SourcePatch → catch → inline fallback → WarnMessage, near-identical
   each time; 3 added on 2026-08-10 alone). Fix: `Invoke-PatchWithFallback`
   helper in WindowsSourceBuild.Patches.psm1.
8. *(DONE 2026-08-10 night: `Get-DiagnosticLogPath` + `Limit-DiagnosticLogs`
   in WindowsScripts.Shared own path + retention; verify-cuda-cache
   consumes them, the A/B's copy vanished with the #5 delegation, and
   probe-build-copy keeps its inline block BY DESIGN (module-free
   first-script rule).)*
   **Log-persistence convention has no owner.** Pair with #30.
9. *(DONE 2026-08-10 night: script-local `Invoke-ProbeLane` (exe check, Tee
   log, tail echo, exit report, attempted/failed bookkeeping) called for
   all three lanes; args are quoted array elements per the ArgQuoting
   lesson. NOTE: not yet live-smoked — run `Test-BuildCopy.ps1 -Heavy`
   once after run 12's media-core finishes, before trusting a verdict.)*
   **probe-build-copy's three lanes are the same 9-line block ×3.**

### P3 — cosmetics / hygiene

10. *(DONE 2026-08-10 night: log reset once up front, every invocation
    appends; flag deleted.)*
    **`$invokeNinja`'s positional `$true/$false` append flag**
    (WindowsSourceBuild.Common): delete the LogFile once up front and always
    `-Append` — removes a silent-log-truncation failure mode and two
    branches.
11. *(DONE 2026-08-10 night: marker is the single channel; the job stream
    only speaks when a marker WRITE fails — see #17.)*
    **Stall-guard verdicts print twice** (job stream + marker file re-read).
    Marker file is the single source of truth (ladder reads it, survives
    clip) — drop the Write-Output/Receive-Job channel.
12. *(DONE 2026-08-10 W1)* **`Native.ArgQuoting.Tests.ps1` manual case-insensitive loop** —
    `-notcontains` already compares case-insensitively; five lines → one.
13. *(DONE 2026-08-10 W1)* **Fake-ninja 'fail' line evaluates its condition twice**
    (SourceBuild.NinjaRetry.Tests) — fold into one guarded block like the
    FAILONCE line below it.
14. *(DONE 2026-08-10 night: payload lives at
    `windows/diagnostics/verify-cuda-cache/Test-Cache.ps1` (lint scope),
    copied into the solve context and run as `RUN & C:\Test-Cache.ps1`.)*
    **`Test-CudaCache.ps1`'s 21-statement concatenated RUN line.**
15. *(DONE 2026-08-10 night: `git mv verify-defender-exclusions.ps1
    Sync-DefenderExclusions.ps1`, all references updated — the name now
    says what it does.)*
    **`verify-defender-exclusions.ps1` naming**: it verified AND applied.

### P1 addenda from the full sweep

16. *(DONE 2026-08-10 night: guard v2 — CPU delta demoted to pre-filter,
    verdict is a timed `sccache --show-stats` client probe (15 s,
    Start-Process + `.Handle` quirk); an answering server resets to healthy
    idle, a hanging probe = kill on first confirmation.)*
    **Stall-guard trigger is a CPU proxy — replace with a timed sccache
    client probe.**
17. *(DONE 2026-08-10 night: marker truncated inside `$invokeNinja` before
    every invocation, ladder classifies on THIS attempt's kills only and
    prints lines as it consumes them; `Add-Content` is try/catch with a
    loud job-stream fallback on write failure. +1 regression test
    (fail-fail-succeed at full `-j` ×2). The former POISONING paragraph is
    RETRACTED: run 11 failed byte-identically on a FRESH mount — the run-10
    link failure was the sccache nvcc MISCOMPILE (see the CUDA-launcher-OFF
    block in Build-OnnxFromSource.ps1), not truncated cache objects. The
    mount-id bump to `sccache-winamd64-2` stays, harmlessly.)*
    **Marker-based retry classification is not attempt-scoped.**
18. *(DONE 2026-08-10 night, pragmatic form: gate-specific `-SkipRdna4Gate`
    in BOTH drivers (message points at `Test-BuildCopy.ps1 -Heavy` as the
    evidence to earn it); the full probe-as-verdict altitude change is
    deliberately NOT built — the probe costs minutes, the gate seconds, and
    the skip-switch covers the healthy-host-after-driver-fix case.)*
    **RDNA4 gate altitude**: regex-match should be the cheap trigger, the
    finalize probe the verdict (block only on a red probe); add a
    gate-specific `-SkipRdna4Gate` instead of the all-or-nothing
    `-SkipHostChecks` (which also disarms the disk gates); the day the
    driver interaction is fixed upstream, healthy RDNA4 hosts stay blocked
    until module surgery.
19. *(DONE 2026-08-10 night: the 006 stanza passes `-Fatal` to
    `Invoke-SourcePatchWithFallback` — a double miss throws at patch time
    instead of re-arming the sccache crash at re-enable time.)*
    **Patch-fallback last rung is soft**: when both the `.patch` AND the
    inline-regex anchor miss (next ORT bump), `Invoke-InlineRegexPatch`
    warns and returns `$false` piped to `Out-Null` — for patch 006 that
    silently re-arms the deterministic sccache crash ~4900 s in. Hard-fail
    the 006 rung (`-Require` or check the return).

### P2/P3 addenda

20. *(DONE 2026-08-10 night: `Start-SccacheStallGuard` returns `$null`
    without `Test-SccacheRemoteConfigured`, same gate as the launcher
    wiring.)*
    **Guard job spawns even when sccache has no remote configured** (baked
    into the toolchain image, so it exists on PATH in every container):
    gate `Start-SccacheStallGuard` on `Test-SccacheRemoteConfigured` like
    the launcher wiring does.
21. *(DONE 2026-08-10 night: detectors moved to
    `modules/WindowsLint.Common.psm1`; Invoke-Lint runs them inside its
    parse loop (one parse per file, `\archive\` excluded, violations fail
    the gate) over a now fully-recursive windows\ walk that also picked up
    `windows\upstream`; the test suite keeps the 6 positive controls.)*
    **AST sweep double-parses the tree** every gate cycle.
22. *(DONE 2026-08-10 W1)* **`Test-CudaCache.ps1` exports a throwaway image** nobody consumes —
    drop `--output` (solve-only is enough for the hit/write assertions;
    contrast: the finalize probes NEED `type=image,unpack=true`).
23. *(DONE 2026-08-10 night: moved to `C:\sccache\logs\` in both media
    Dockerfiles; `Initialize-SourceBuildEnvironment` creates the dir before
    the server can spawn — sccache does not create missing parents.)*
    **`SCCACHE_ERROR_LOG` sits at the root of a GC-capped cache mount** —
    eviction under pressure can delete the postmortem exactly when needed;
    consider a subdirectory exempted by policy or copying the log out in
    the chain epilogue.
24. *(DONE 2026-08-10 night: prepended to all 40 remaining files, each
    matching its own EOL style.)*
    **`#Requires -Version 7.0` missing across `windows/scripts/tests/`.**
25. *(DONE 2026-08-10 W1)* **`Test-Rdna4LayerLock.ps1` StrictMode fragility**: `$offTiny`/
    `$offHeavy` are only-safe-by-control-flow; initialize them up front so a
    future try/catch edit cannot turn the verdict line into a StrictMode
    error on the exact host being diagnosed.

### Second deep pass (build-definition & operations angles, 2026-08-10 evening)

The 8-finder review read almost only `.ps1`/`.psm1`; this pass covered the
Dockerfiles, resource economics, store hygiene and probe supply-chain.
Checked and deliberately NOT flagged: isolation-probe already parameterizes
its FROM; scrub coverage matches its documented tier map; `.dockerignore`
guards the context; the ENV/ARG mirrors are the documented deliberate ones.

26. *(DONE 2026-08-10 W1: ARG BASE in both probe Dockerfiles; Test-BuildCopy.ps1 pins it to versions.env WINDOWS_BASE_DIGEST via dependency-free parse on all three lanes, smoke-verified; the A/B script inherits the tag default until #5)* **Probe Dockerfiles float their base** (`FROM ...servercore:ltsc2025`
    without a digest) while the chain pins `WINDOWS_BASE_DIGEST` — when MS
    rolls the tag, the probe certifies a DIFFERENT base than the chain
    builds on, and pulls a fresh multi-GB image to do it. Fix: `ARG BASE`
    like Dockerfile.isolation-probe, defaulted by the probe script from
    versions.env's digest.
27. **Dockerfile.base carries a 1214-char single-line RUN** (the pwsh
    bootstrap blob) — unreviewable and undiffable. Move to a mounted script
    like the heavy lanes; BASE-TIER CLOSURE, so batch it with the next
    planned base rebuild, never alone.
28. **~60-70 % of the CPU is idle during the heaviest stage** (26-31 %
    total at ONNX `-j9`, memory-capped by a flat MemGBPerJob=4 that treats
    a 200 MB CXX TU like a 4 GB flash-attention TU). AGENTS.md priority 1
    calls idle cores "the standing wall-clock reserve": measure real
    per-TU-class peaks, then split into heavy/light ninja job pools or
    per-stage MemGBPerJob. Potentially the single largest wall-clock win
    left in the chain.
    *(MEASURED COMPLETE 2026-08-11: run-13 full window, 7821 samples @15 s
    (`run13-tu-memory-samples.csv`) covering toolchain build + the ENTIRE
    ONNX vertex incl. the early flash-attention region + the OpenCV
    attempt; corroborated by run-12's 1453-sample tail window. Peak
    per-process WorkingSet: cicc 998 MB / clang-cl 989 / cudafe++ 944 /
    ptxas 830 — NO process reaches 1 GB; peak concurrent fleet total
    5.5 GB across 21 processes at `-j9`. READY TO LAND: `-MemGBPerJob 2`
    at the build-onnx/build-opencv call sites → 19 jobs (peak assumption
    still ~2× measured; extrapolated fleet ≈ 11-12 GB against the 39 GB
    budget). DELIBERATELY NOT hot-landed mid-run-14: media-closure edit =
    one more ONNX rebuild, so bundle it with the NEXT closure window per
    the standing cache-tier rule.)*
29. *(DONE 2026-08-10 night: the tag-minting diagnostics now share the
    `diag-` prefix (`diag-probe-build-copy[-heavy]`, docker lane
    `local/test:diag-probe-build-copy`; the A/B mints nothing since the #5
    delegation, verify-cuda-cache nothing since #22). Cleanup one-liner:
    admin `nerdctl --namespace buildkit images | findstr diag-` → `rmi`.
    The pre-convention 2026-08-10 tags — incl. the old
    `probe-build-copy[-heavy]` names — stay on the pending-cleanup list.)*
    **Diagnostic image debris pins store layers.**
30. *(DONE 2026-08-10 night: `Limit-DiagnosticLogs` retention — driver keeps
    the newest 80 stage logs in `out\windows-build-logs`, diagnostics the
    newest 60 in `out\build-logs` via `Get-DiagnosticLogPath`. Console-log
    convention: the LAUNCHER names one timestamped file per driver
    invocation — never append across runs.)*
    **Build-log growth is unbounded.**
31. **Green stage images live ONLY in the local containerd store** until a
    manual `-PushRef` — a host loss (this host has form: repair-installs,
    driver surgery) costs every stage. Auto-push green stage tags (or at
    least export-cache) once a chain goes green; the driver params already
    exist, they are just never invoked by default.
32. *(DONE 2026-08-10 night — answered by reading the workflow: the Windows
    CI lane is GITHUB-HOSTED `windows-latest`, builds NO container images
    (scripts gate only, stated in windows-scripts.yml's header) and hosted
    VMs carry no AMD dGPU → **CI is immune to the RDNA4 hazard**; the
    gate/toggle workflow is local-host-only, which is where it lives.)*
    **Document the CI runner's GPU class vs the RDNA4 hazard.**

### Already fixed during the review session (2026-08-10, for the record)

The sweep also surfaced correctness bugs in same-day code; these were fixed
immediately rather than backlogged: probe zero-lane false-green (exit 0 with
no lane run), `Repair-WindowsComponentstore.ps1` still using the retired
`type=local` probe shape, the classic lane (`build.ps1`) missing the RDNA4
gate, `Get-SccacheStatsText` not re-exported (would have thrown AFTER the
multi-hour ONNX build), `Dockerfile.heavy`'s trailing-backslash COPY dest
(Dockerfile escape char), the RDNA4 A/B swallowing lane logs and re-enable
failures, the `(TM)`-rename hole in the hazard regex, the opencv/001 patch
EOL flip, `SCCACHE_ERROR_LOG` parity for the merge builder, and a stale
module comment describing the abandoned launcher-opt-out design.