# Windows container chain — refactor backlog Open work, standing directives and pending host/upstream actions for the Windows container chain. Settled items are archived under `docs/windows-backlog-archive-*.md`. The Linux-side equivalent is [`refactoring-backlog.md`](refactoring-backlog.md). > **COUNTING NOTE:** item numbers are HISTORICAL and never reused — the highest > number is not the item count. Resolved narratives move to the dated archives > (`windows-backlog-archive-*.md`); a bare "#N" that is not in this file > resolves there. Lean-OPEN-only is the owner's standing policy. ## OPEN ### ARM64 parity (opened 2026-08-23) The `:winarm64` cross lane reached runtime parity with `:winamd64` apart from the exclusions listed below, and **2026-08-31 reran the FULL chain with the QNN EP enabled** (#121 build-time path proven; the earlier acceptance gap — HEAD vs the 2026-08-26 tree — was two dead-on-arrival GStreamer cross-lane fixes that are now landed: see #135 follow-up below). Read the table as the fully green run of 2026-08-31 (`bk-winarm64`); a later full arm64 run is #176's `bk-20260920-203631` (smoke 120/0/15, arch gate 1047/0, CUDA on). The 2026-09-02 dual-lane rebuild (post-wave toolchain with sanitizers, rustup 1.29.1) reproduced both SMOKE columns exactly — arm64 **97/0/15**, amd64 **222/0/0** (`bk-20260902-002412`); the other rows below are the 2026-08-31 measurements and were not re-extracted. | | arm64 cross (`bk-winarm64`, 2026-08-31, QNN ON) | amd64 native (`bk-20260826-130136`) | |---|---|---| | arch gate | **1168 binaries, 0 violations** (QNN payload rides in) | 1134, 0 | | import walk | **793 files, 0 unresolved** (85 device-OS: FastRPC + client SKU) | — | | bundle manifest | 7 DLL homes, **6 wheels**, 3 ABSENT markers | — | | GStreamer contract plugins | **6/6** — `libav opencv onnx webrtc nice tflite` | 6/6 | | smoke gate | **97 passed / 0 failed / 15 skipped** (floors 66/20) | **222 / 0 / 0** | | QNN | EP ON in **onnxruntime only** (see #154); QAIRT 2.44.0 `aarch64-windows-msvc` runtime COPIED beside all five installs, but only ORT loads it | off (x64 CPU backend is pointless) | | wall clock | ~40 min (media+final) | 2 h 18 min | **Exactly three components are ABSENT on arm64**, each marked in the bundle by an `ABSENT-ON-ARM64.txt` (LiteRT-LM, the default name) or a `COMPILER-ABSENT-ON-ARM64.txt` (the two compilers, which pass `-FileName` explicitly) — the manifest globs both families. Call sites, each a `Write-AbsentOnCrossMarker` call: `Build-TvmFromSource.ps1`, `Build-IreeFromSource.ps1`, `Build-LitertAll.ps1`: the **TVM compiler**, the **IREE compiler** (both need an LLVM cross-built for aarch64-windows) and **LiteRT-LM** (Bazel + an x86_64 prebuilt `.lib`; a CMake port exists upstream — #133(d)). Their *python packages* DO now ship (`apache_tvm`, `apache_tvm_ffi`, `iree.runtime` — closed by #133). Also excluded by owner decision or construction: the **torch app stage** (`uv sync` must execute the target interpreter). **CUDA is no longer excluded by upstream facts:** NVIDIA now ships Windows-ARM64 CUDA 13.4.2 installers, and the x64 host can cross-compile to ARM64 with them (documented in NVIDIA's porting guide; probe-proven 2026-09-19 — `AA64 machine (ARM64)` — once the arm64 library archives are staged). Wiring that into the lane is **#176**. The **QNN EP is PRESENT and PROVEN 2026-08-31** (build-time path; runtime execution still needs a Snapdragon host — see #121 below). **The honest caveat, smaller now:** until 2026-09-25 nothing the arm64 lane produces had been *executed*. The consumer cross lanes' `windows-11-arm` run jobs run their products since then, and the hub's `windows-arm64.yml` runs the published bundle's tools, Python, GStreamer pipelines and, since 2026-10-12, an ONNX Runtime session on the CPU EP (`BACKLOG.md` CON30). Inside the cross build itself every verdict above is still a static check — PE machine type, import resolution, exported symbols. The 15 skipped smoke assertions are the ones that would have to run the aarch64 payload on an x64 host. **Permanently out of reach — do not re-litigate without new upstream facts:** classic TensorRT (genuinely x64-only — NVIDIA's support matrix has no ARM64 row), and the `torch` app stage (`uv sync` must **execute** the target interpreter — uv can cross-RESOLVE into a directory, but the synced venv is the stage's contract — and PyTorch builds no `win_arm64` wheel for **Python 3.14**, this repo's cp314 pin). --- ### Open items - **#179 — every RUN that started an sccache server idles ~120 s before DONE: 16 to 27 min per chain (opened 2026-10-12, #153's sweep). Cut to ~30 s on 2026-10-14 by the host's `TcpTimedWaitDelay`; the patched sccache below would remove the rest.** Counted from a RUN's last output line to its DONE, the gap is 114 to 124 s in every such RUN: 9 RUNs and 1,077 s in bk-20261011-184301, 14 RUNs and 1,642 s (8.4 % of 19,592 s) in -233025, 8 RUNs and 955 s (13.2 %) in bk-20261010-121545. RUNs without a server close 1 to 9 s after their last line, a multi-GB `Install-Rocm` extract included, and the arm64 VVL RUN, whose launchers are cleared, closes 1.3 s after 513 s of compiling. The size of the RUN makes no difference. **Probed 2026-10-12** with buildctl over `bk-windows-toolchain`, one RUN per variant and its own cache mount id. The tail comes from the sccache server ending *gracefully*: - A RUN with no sccache reaches DONE 1 s after its script ends. - With `sccache --stop-server`, DONE comes 120 to 123 s after the end. That holds for the webdav and the local-disk backend, with or without a compile, and also when the server process is gone 0.1 s after the stop and no process of the RUN is left. A server left running costs the same. - `Stop-Process -Force` on the server instead: DONE 1 s after the end, as without sccache. Killing an unrelated detached process costs nothing either. - **Proven the same day: any TIME_WAIT in the container holds its teardown until it expires** (Windows' default `TcpTimedWaitDelay`, 120 s). One loopback pair without sccache, closed cleanly, costs 123 to 126 s; the same pair closed with a reset leaves no TIME_WAIT and costs 1 s. - **The TIME_WAIT is the client's.** sccache 0.18.0's client always closes first (`commands.rs` drops the connection after reading the response; the server reads to EOF), so every compile and every `--show-stats` leaves one on its ephemeral port. Killing the server therefore cannot help: compile + stats + `Stop-Process -Force` still cost 128 s. - **Nothing in the container removes it.** `SetTcpEntry(DELETE_TCB)` refuses a TIME_WAIT row (317). The registry value inside the container changes nothing (the stack is the host's). `SCCACHE_SERVER_UDS` is ignored on Windows: 0.18.0 still listens on 127.0.0.1:4226. - **The graceful stop protects no cache write in this chain.** With `SCCACHE_MULTILEVEL_CHAIN` unset (webdav only), each compile awaits its PUT before the client gets `CompileFinished`. Under `disk,webdav` the webdav writes are detached tasks that the stop does not wait for either (`multilevel.rs`, `write_remaining_levels_async`). Two fixes, both owner decisions; the second is in place: - **A patched sccache.** The client sets `SO_LINGER {on, 0}` after connecting, and the server takes the resulting reset as a disconnect rather than logging `Failed to bind socket`. The patch is 56 lines on v0.18.0 (`net.rs`, `server.rs`, a windows-sys feature). The hub would build sccache from source again, as before 2026-09-18 (#114). The end step then kills the server instead of `--stop-server`, which also resets its pooled webdav connections. That removes the whole tail. The probe that builds and times it is written but has not run. - **The host's `TcpTimedWaitDelay` at 30 s, adopted 2026-10-14.** `Install-NewHost.ps1` sets it and `Test-HostSetup.ps1` checks it (`windows-host-setup.md` § C6); it takes effect after a reboot. In the from-base chain `bk-20261014-cpu` the sccache RUNs (ONNX Runtime, FFmpeg) reached DONE 29.9 s and 30.2 s after their last line, and the RUNs without a server 0.9 to 3.3 s. That is about 90 s less per sccache RUN, 12 to 21 min per chain at the counts above. The servers that the launchers alone started (hailo, merge 2/9) need the same end step. In the same function, webdav `NotFound (permanent)` 404s are cache misses rather than failures, and the `-Last 60` dump contradicts its "Failures in full" comment. Before `upstream/hcsshim-teardown-timeout/ISSUE.md` is filed, re-measure its "legitimate 117 s teardown": that OpenCV specimen ran sccache as well. Rides the next full chain, since `WindowsSourceBuild.Common` is mounted into every media layer. - **#182 — GStreamer's meson build bypasses sccache on both arches, and its setup fetches ~50 wraps live (opened 2026-10-12, #153). The bypass is FIXED IN SOURCE the same day:** meson's `compilers/detect.py` passes an `sccache` prefix to the MSVC compiler class but drops it for clang-cl. `Build-GstreamerFromSource.ps1` now builds a `clang-cl.exe` shim (an x64 host exe, so the arm64 cross file can use it) that runs `"" "" `. It hands the shim to meson as `CC`/`CXX`, and the stage fails when sccache counted fewer than 100 compile requests. A buildctl probe on `bk-windows-toolchain` showed meson 1.12.1 detecting it as `clang-cl 23.1.3` with lld-link, the linker it had before, and every ninja compile reaching sccache. The compile it saves is 663 s, 515 s and 612 s per merge rebuild in the swept chains. The next full chain proves it on GStreamer. Still open: meson setup takes 422 to 557 s and downloads 48 to 51 wrap sources live (the prefetch covers about 9), ninja then regenerates the build files once more (79 s, 161 s), and gst-plugins-rs (135 to 151 s) runs without `RUSTC_WRAPPER`. Put meson's packagecache on a cache mount or prefetch every wrap, find the regenerate trigger with `ninja -d explain`, and set `RUSTC_WRAPPER`. - **#183 — vendored code compiles under clang-cl's `/Wall`: 1.36 M warning lines in one arm64 merge vertex (opened 2026-10-12, #153; `BACKLOG.md` CON89's VVL item). VVL and x265 are FIXED IN SOURCE the same day:** both builds run with `CL=/w`, which clang-cl reads before its own flags, and `/w` still beats a later `-Weverything` (probed in `:winamd64`: 3 warnings without it, 0 with it). The first fix used `_CL_`, which clang-cl appends after them; CMake calls clang-cl as `... -- `, so `/w` landed after the `--` and read as a file name, failing x265's compiler check in bk-20261012-190400. sccache hashes neither variable, and `/w` changes no code. Abseil in ORT and OpenCV's `matx.hpp` stay open; they warn from inside our own builds, where a blanket `/w` would hide our warnings too. VVL's GCC-style `-Wall` reaches clang-cl as `/Wall`, which is `-Weverything`. In bk-20261010-121545 that vertex is 6,228,195 lines and 752 MB of the 768 MB merge log (97.9 %); the "about 110k" CON89 quoted is one translation unit, `vk_validation_error_messages.cpp`. The same class, smaller: x265's 27,423 `-Wunsafe-buffer-usage` lines in ffmpeg, abseil's 19,001 to 29,957 `-Wdeprecated-attributes` in onnx, 7,200 from OpenCV's `matx.hpp`. Fix: `-w` for vendored code (VVL's `$common`, x265 in `Build-FfmpegCodecs.ps1`, ORT's abseil flags), retry VVL with the sccache launchers back (cleared after the server hung twice on that unit), build it at -j19 rather than -j9, and `$null =` the two `Invoke-GitClone` calls that print a stray `True`. Rides the next full chain with CON89. - **#185 — chain RUNs fetch live on every rebuild, and three fetches are unpinned (opened 2026-10-12, #153).** Against the everything-in-the-image rule, a rebuild fetches the PyTorch clone and submodules (262 s in bk-20261011-233025), the LiteRT sources (88 s, 244 s), the IREE clone (~218 s), the aarch64 rust-std, the crates.io index for gst-plugins-rs, CPython's PCbuild externals (the toolchain layer deletes them, and arm64 builds the target CPython twice per chain) and 32 wheels the uv cache mount still downloaded for torch. **The three unpinned fetches are pinned in source (2026-10-12):** `Build-LitertLmBazel.ps1` took bazelisk from `releases/latest`, fell back to Adoptium's `latest/21/ga` JDK and installed `git-lfs` through scoop, none SHA-checked. `Get-PinnedTool` now fetches bazelisk v1.29.0, Temurin 21.0.5+11 and git-lfs v3.8.0 by their `versions.env` pins (`BAZELISK_WINDOWS_*`, `TEMURIN_JDK_WINDOWS_*`, `GIT_LFS_WINDOWS_*`) into the tool cache, refuses any other bytes and re-fetches a corrupt cache entry. bazelisk itself gave way to the pinned bazel binary (`LITERT_LM_BAZEL_WINDOWS_*`, the clone's `.bazelversion`) on 2026-10-13 (CON92). **Still open:** serve pinned source archives from a cache mount keyed by commit, bake the aarch64 rust-std into the toolchain image, and build the target CPython once and keep its externals. Rides the next full chain. - **#178 — fold `Expand-PinnedArchive` into `Expand-SourceTarball` at the next media rebuild (opened 2026-09-25).** `Expand-PinnedArchive` (`WindowsMigraphx.Common`) is `Expand-SourceTarball`'s two 7-Zip passes, except that it skips the links 7-Zip 25+ refuses as dangerous instead of failing (flatbuffers 25.12.19 carries nine in-tree ones). It lives apart only because `WindowsSourceBuild.Common` is mounted into every media layer, so an edit there re-keys the whole media lane. At the next deliberate media rebuild, give `Expand-SourceTarball` that tolerance, point `Save-PinnedSource` at it, delete `Expand-PinnedArchive`, and delete its row in `docs/scripts/code-dupes.allow` (the `WindowsMigraphx.Common` ↔ `WindowsSourceBuild.Common` pair). Rebuild-coupled like CON34 in `BACKLOG.md`. - **#177 — the image history still names the build host's endpoint (opened 2026-09-23).** Since 2026-09-23 `SCCACHE_WEBDAV_ENDPOINT` reaches the compiling RUNs as an ARG, so no published ENV carries it and the publish gate refuses one that does. But BuildKit writes a RUN's build args into that layer's history `created_by` (`|3 SCCACHE_WEBDAV_ENDPOINT=http://… RUN …`), and the final image inherits every one. Nothing reads history at run time, so consumers are unaffected; it still publishes an RFC1918 address. The fix is to pass the endpoint as a secret (`RUN --mount=type=secret,id=sccache_endpoint,env=SCCACHE_WEBDAV_ENDPOINT`, with `buildctl --secret`), which records nothing. **Verified on WCOW 2026-10-12** (buildkitd v0.33.0, a buildctl probe over `bk-windows-toolchain`): the `env=` form sets the variable in its RUN, the next RUN no longer sees it, and no file is left in `C:\`, `C:\run`, `%TEMP%`, `C:\Windows\Temp` or `C:\ProgramData`. Its value also stays out of the cache key, which an ARG's does not. **Never the file form on WCOW:** a `target=C:\…` secret mount stays in the snapshot, so the next RUN still reads the file and the layer carries it (an upstream defect to report, owner's call). Left: change all ten compiling stages at one planned re-key. Context: [`windows-build-resources.md` § What the published image carries](windows-build-resources.md#what-the-published-image-carries). - **#155 — LiteRT QNN: the real flags, and why wiring them now would be a fourth silent no-op.** RESEARCHED 2026-08-31. DO NOT integrate before reading this. The genuine switches, all verified defined at v2.2.0: `LITERT_ENABLE_QUALCOMM` (`litert/vendors/CMakeLists.txt:330`, default OFF) and `QAIRT_HEADERS_DIR` (`:20`), where setting the headers dir AUTO-FORCES the former ON (`:331-334`). QAIRT is consumed as **headers only, at configure time** — no import libs, no link-time dependency; backends are `LoadLibrary`'d by name at runtime. NOTE `LITERT_ENABLE_NPU` (`litert/CMakeLists.txt:72`) is NOT the QNN switch: its only effect is a `#cmakedefine01` in `build_config.h`, and it does not gate `add_subdirectory(vendors)`. **Why it is not wired yet — five upstream defects, all Windows-specific:** 1. The dispatch DLL **exports nothing** on Windows: the SHARED target sets no export macro, no `WINDOWS_EXPORT_ALL_SYMBOLS`, no `.def`. It would build green and be unloadable — exactly #154's failure mode, one level deeper. 2. `-Wl,--whole-archive` / `--no-undefined` are passed on every non-Apple platform, Windows included; only `if(APPLE)` is carved out. GNU-driver syntax under lld-link. 3. Both `dynamic_loading.cc` and `dynamic_loading_windows.cc` compile on Windows. 4. The Windows branch of the vendor link lines is empty. 5. `litert/vendors/CMakeLists.txt` does **three unconditional configure-time downloads** — NeuroPilot from AWS S3, and QAIRT **2.47.0.260601** from softwarecenter.qualcomm.com with no `EXPECTED_HASH` and no error check — whenever the corresponding `*_HEADERS_DIR` is empty. Two are escapable by pointing them at a dummy dir; the QAIRT one is escaped by setting `QAIRT_HEADERS_DIR` to our staged 2.44 tree, which also stops the version skew. The `litert/` tree is also a SEPARATE top-level project (`project(LiteRT VERSION 1.4.0)`), not a flag on the `tflite/` tree we configure — a second configure, and it pulls tflite in `EXCLUDE_FROM_ALL`, so `tensorflowlite_c` must be named explicitly. Upstream has never configured this tree for Windows: the only CMake CI is linux-x86_64/Android. **And the arch gate cannot prove any of it** — QNN backends are `LoadLibrary`'d by name, so a dispatch DLL adds zero static import edges. A green gate would mean nothing here. - **#163 — `-ConcurrentAux` has never been used: ~24 min idle-capacity per full amd64 chain.** 43 manifests, zero concurrent runs; litert 3535 s + tvm 1425 s always sequential; the 19 GB half-budget path works and no longer re-keys. The three ConcurrentAux fixes it was waiting on landed on 2026-09-17 (archive 2026-09-17), so it is unblocked: default it for full amd64 chains after one owed measurement — litert at 19 GB must not exceed the hidden 1425 s (its bazel half grew 18→59 min since July). - **#176 — CUDA for Windows ARM64. PHASE 1 LANDED 2026-09-20 (built, smoke adapted; running it still needs a device).** Opened 2026-09-19 on new upstream facts that supersede #122; the x64 host **can** cross-compile CUDA to ARM64 (NVIDIA's Windows-on-Arm Porting Guide; probe-proven in `out/probe-cuda-cross2/` the same day), so the device is needed to *run* the result, not to build it. Phase 1, all in tree and built: * `Install-Cuda.ps1 -TargetArch arm64` stages the arm64 redist payload (`cuda_cudart`, `libcublas`, `libcufft`, `libcurand`, `libnvjitlink`; SHA pins in `versions.env` from `redistrib_13.4.2.json`) into the x64 toolkit's root as `lib\arm64` / `bin\arm64`, plus `cudnn-windows-arm64-_cuda13.4-archive.zip` (NVIDIA names the arm64 archive `_cuda13.4`, the x64 one `_cuda13`). * `Get-NvccCudaCmakeArgs` drives `Hostx64\arm64\cl.exe` + `nvcc --use-local-env` on the cross lane; `Get-CudnnLibraryDir` picks `lib\arm64`; `Test-CudaWindowsArm64Payload` is the cross POSITIVE signal — no host GPU probe decides a target flag. * `Build-OnnxFromSource.ps1` builds the ORT CUDA EP for arm64: the 2026-09-20 run `bk-20260920-031232` produced `onnxruntime_providers_cuda.dll` **0xAA64** in the `win_arm64` wheel (2065-object ORT build), merge arch gate 1011/0, smoke 119/1/15 → the one red was the cuDNN run-probe, now a link + PE-machine assert (`Assert-NativeLinkRun -CrossLinkOnly`, same 1:1 substitution as §14). * `-Gpu -TargetArch arm64` is a supported driver combination; `EXPECT_GPU` rides the cross smoke so a lost CUDA env reds instead of skipping. PHASE 2 LANDED 2026-09-20 (`bk-20260920-203631`, 2:45:53): OpenCV, GenAI and TVM build their CUDA paths for arm64 too. * Payload grew `libnpp`/`libcusolver`/`libcusparse` (OpenCV links CUDA::nppial/...; the first phase-2 configure died on the missing NPP targets). * OpenCV: arch-aware host compiler + `--use-local-env` + explicit arm64 `CUDNN_LIBRARY`; its one x86 intrinsic in the CUDA tree (`_mm_popcnt_u64`, cudafilters `wavelet_matrix_2d.cuh`) is patched to a software popcount (`patches/opencv_contrib/002-...`, probe-proven; nvcc's device pass rejects `__builtin_popcountll`, so a guard tweak alone was not enough). * GenAI: `onnxruntime-genai-cuda.dll` **0xAA64** in the `win_arm64` wheel. * TVM: `USE_CUDA/CUBLAS/CUDNN=ON` with the arm64 libs named explicitly (its legacy FindCUDA hardcodes `lib\x64`), `CUDA_HOST_COMPILER=Hostx64\arm64\cl`, `CUDA_NVCC_FLAGS=--use-local-env`. * Smoke 120/0/15; arch gate 1047/0; `opencv_cuda*500.dll` installed under `lib\opencv5\arm64\vc18\bin`. STILL OPEN: classic TensorRT (x64-only; TensorRT-RTX unwired) and the first RUN on an RTX Spark/N1x-class device. Do not re-derive #122's August reasoning — the archive entry records what it rested on. ### CLOSED (pointers — full narratives in the dated archives) - **#181 — the LiteRT layer no longer exports bazel's trees. DONE 2026-10-12.** `Build-LitertLmBazel.ps1` stops the bazel server and removes `C:\bzl`, `C:\llm`, `C:\bzl-tools` and the bazel log after the install's contract check; the merge copies only `lib\litert-lm` out of the layer. Proven on a local media-litert stage (bk-20261012-152007): 8.7 GB removed (6.66 + 1.72 + 0.33 GB), and the layer's export took 52.9 s, against 1,126 s in bk-20261011-184301. Its unpack took 1,047 s, against 396 s there and 1,337 s in bk-20261011-233025, so unpack time follows the host, not this layer, and the stage came to 2,415 s against 2,728 s. bazel's user root (1.21 GB) and bazelisk's cache (0.11 GB) stayed; since CON92 the user root is `C:\bzl-user` and leaves with the rest, and bazelisk is gone. Bazel's progress now reaches the stage log once per 30 s. Not done: a remote action cache for the 3,603 actions every rebuild compiles afresh. - **#180 — the patched clang-cl relinks reproducibly. DONE in source 2026-10-12.** CMake's default compiler for the LLVM build was MSVC `cl`, and LLVM appends `/Brepro` only for a Clang compiler, so each relink stamped `clang-cl.exe` with the link time (0x6ACBBE2C in bk-20261011-184301, 0x6AC6CAC6 in the published `:winamd64`, same 92,396,032 bytes) and sccache, which keys on the compiler's bytes, missed nearly every media compile after it. `Build-LlvmFromSource.ps1` now builds with the base image's pinned clang-cl (`LLVM_USER_BIN`) and fails the stage when the installed `clang-cl.exe` still carries a link-time stamp (`Assert-ReproducibleLink`). Proven on a local toolchain stage (bk-20261012-144845, 18:23): LLVM compiled cold with clang-cl in about 12 min, and `clang-cl.exe` carries the content hash 0x35341D1A. The next full chain rebuilds the media lane once against it; the relink after that is the one that keeps its hits. - **#184 — a stopped BuildKit run keeps its record. DONE 2026-10-12.** `Build-Buildkit.ps1` writes `bk--manifest.txt` with `status=ok` or, from its `finally`, `status=failed:` and every finished stage, in the invariant culture (a German host had written `patched-llvm=778,9` beside `total_s=13058`). `bk--driver.log` records the run, each stage's start, end and failure, every retry verdict with the line that made it transient, and every disk-gate verdict (`Write-DriverRecord`). The resource CSV gains `diskFreeGB` for the build drive and `-Summarize` a `MinDiskFreeGB` column. buildctl's output is decoded as UTF-8. Left out: a `--progress rawjson` copy, since buildctl takes one progress mode; the driver log's stage stamps place the time before a stage's first RUN. - **#153 — the step-time / silent-retry sweep. DONE 2026-10-12.** The 2026-09-01/02 logs it named were pruned (only manifests survive), so it ran over the three complete chains left, 51 logs: bk-20261011-184301 (amd64, 13,058 s, smoke 268/0/1), bk-20261011-233025 with -20261012-051453 (amd64 rocm, 19,592 s to the disk gate, then final and smoke) and bk-20261010-121545 (arm64, 7,211 s, smoke 147/0/17). **The 441.3 s lifecycle floor is gone:** a trivial `RUN cmd /c exit 0` takes 4.9 to 9.0 s. **One silent retry in three chains:** rocm-llama's first RUN over the fresh migraphx image hit `ActivateLayer 0x20`, and attempt 2/3 went green for about 70 s; the merge fan-in self-heal did not recur (every merge log is `attempt 1/5`, with no `ActivateLayer`). **No clipped output** in the 51 logs, the 6.2 M-line VVL vertex included. **The rule for reading step times now:** subtract ~120 s from each RUN that started an sccache server, and nothing from the others (#179). What the sweep found beyond its question is #179 to #185. Before 2026-08-30 the corpus was pruned too, so those forensic conclusions stay unverifiable. - **#158 / #159 / #160 / #164 / #167 / #168-#174 / #175 — the 2026-09-17 batch: the thirteen audited defects, the eight settled-probe deletions, the compiler-rt verify+tar port, LLVM through sccache, smoke section 23, the comment-discipline wave and the three neighbourhood checks. Landed in `2bb0410f` (2026-09-18), unbuilt when archived. Archive: [`windows-backlog-archive-2026-09-17.md`](windows-backlog-archive-2026-09-17.md). - **The 2026-09-03 doc-drift findings and the PascalCase anchors** — all six doc defects fixed, `deps.json`'s sccache attribution corrected, and `verify_doc_links.py` green over the renamed script headings. Archive: same page. - **#152** — the 2026-08-31 wave: PROVEN BY BUILD 2026-09-01/02. The dual-lane rebuild ran it for real, and the gate did its job: the wave's compiler-rt ride-along had set `COMPILER_RT_BUILD_SANITIZERS=OFF`, the amd64 smoke gate failed on exactly the ASAN probe, the fix (`=ON`) re-ran and the lane closed at the best recorded state **222/0/0** (`[PASS] AddressSanitizer …`, `clang_rt.asan_dynamic-x86_64.dll` installs verified in the toolchain log); arm64 closed green at its baseline **97/0/15**. The predicted re-pay was real (full LLVM + media on both lanes). Narrative: [`changelog-archive-2026-09-07.md`](changelog-archive-2026-09-07.md) § 2026-09-01/02. - **#149** — the `c9586c1^` warm/materialize rollback recipe: DEAD, not stale. FOUR independent breakages (every script path, the missing TargetArch + Tvm modules, the swapped media-core order, the QAIRT pin). The restore recipe is now a derivation rule in `windows-build-lanes.md` § Restoring the warm/materialize rollback — derive each RUN from the stage that runs that script TODAY. Archive: `windows-backlog-archive-2026-08-31.md` § Resolved 2026-08-31 (second pass). - **#154** — QNN is integrated in ONE framework, not five: TVM, IREE and LiteRT were passing CMake flags upstream never defined, so CMake dropped them silently while all three logged success. Flags and banners removed on both lanes; the class is gated by `Assert-CmakeArgsConsumed`. The surviving lead is #155. Archive: same page. - **#156** — LiteRT-LM QNN: DECLINED with reasons (also a standing directive below). Archive: same page. - **#147 / #148 / #150 / #151** — the 2026-08-31 wave: classic driver deleted, the `patched-llvm` whole-dir module mount narrowed (and gated), TVM + GStreamer helpers moved to their leaf modules, `Set-StrictMode` plus the four latent bugs it exposed, the deliberate declines, the dead `build.ps1` permission entries, and the CHANGELOG archive split. Seven commits, `7cc2e95f^..a5eac0dd`. Full narratives: [`windows-backlog-archive-2026-08-31.md`](windows-backlog-archive-2026-08-31.md) § Backlog wave #147-#151. Its declines are also a standing directive below. - **#121** — QNN EP: BUILD-TIME PATH PROVEN 2026-08-31 on the full `:winarm64` cross run — SDK qairt-2.44.0.260225 (SHA-pinned), `onnxruntime_USE_QNN=ON` with the `aarch64-windows-msvc` backend set, QNN provider built, QNN runtime copied beside all five installs — but only ORT consumes it (#154); arch gate 1168/0, smoke 97/0/15. Execution verification still needs a Snapdragon host. Archive: this entry. - **`-ResumeStage` BK equivalent** — CLOSED 2026-08-29 (no BK equivalent needed). The classic lane's `-ResumeStage` preserved a stopped container (hours of compile state), injected a fix via `docker cp`, committed to a partial image, and re-ran from a specific stage. BuildKit has no intermediate containers to preserve — but it caches completed RUN vertices, so re-running `buildctl` automatically resumes from the last cached step. Script fixes land via bind mounts (re-read at solve time), so the "inject a fix and re-run" case is covered natively. No BK equivalent is needed. - **#135** — LLVM 23.1.0 AArch64 codegen: DONE 2026-08-29. `BUILD_PATCHED_LLVM=1` is now the DEFAULT (Dockerfile ARG + driver default), workarounds removed from `Build-OpencvFromSource.ps1`. `-StockLlvm` is the opt-out. Items 1+3 closed; item 2 (NINJA_KEEP_GOING census) is now unnecessary (root cause fixed); item 4 filed as llvm#219200. Archive: `windows-backlog-archive-2026-08-26.md` § #135. - **#135 follow-up — patched toolchain lacks the aarch64 compiler-rt** — DONE 2026-08-31. The source-built `C:\llvm-patched` ships `clang_rt.builtins-x86_64.lib` only, so the arm64 GStreamer link died on `__udivti3` (2026-08-30 cross run, merge stage). The merge stage now self-heals: `Build-GstreamerFromSource.ps1` § 5d mines `clang_rt.builtins-aarch64.lib` from the LLVM release archive on the cross lane (same recipe as Install-ScoopTools.ps1). Chosen over adding the lib to the toolchain layer because the media branches derive FROM `bk-windows-toolchain`, so that would re-pay ~2 h of media compiles for one lib. Toolchain-level fix LANDED 2026-08-31 (`bd150ae1`, `Install-TargetCompilerRt` in Build-LlvmFromSource.ps1) and shipped with the 2026-09-01/02 toolchain rebuilds — the merge-stage self-heal is now the never-firing fallback. Nothing left here. Regression: `SourceBuild.GstreamerCompilerRt.Tests.ps1`. Docs: `docs/windows-cross-builds.md` § aarch64 compiler-rt. Second unmasked failure fixed in the same window: the speculative cross-lane opus intrinsics enablement (2026-08-30) is REVERTED to the proven disabled state — under clang-cl aarch64 the RTCD path applies `-mfpu=neon` (ARM32-only flag) and its CPU probe uses MSVC's `__emit` (absent from clang-cl). Working enablement recipe for a future TESTED window: `-Dopus:intrinsics=enabled -Dopus:rtcd=disabled` (presumes NEON+dotprod; needs a real-device smoke). Final gate fix in the same window: the arch-gate import walk gained the Qualcomm FastRPC pair `libcdsprpc.dll`/`libadsprpc.dll` as device-OS allowances (imported by the QAIRT HTP stub DLLs staged by `Copy-QnnRuntime`; they ship in every Windows-on-Snapdragon OS image, never in the SDK zip). - **#133(d)** — LiteRT-LM CMake port: CLOSED 2026-08-29 (owner decision — staying on Bazel). - **#134** — post-#133 cleanup wave: DONE 2026-08-29. amd64 acceptance PASSED (smoke 192/0/1, arch gate 1134/0). Narrative: [`changelog-archive-2026-09-07.md`](changelog-archive-2026-09-07.md) § 2026-08-29 — amd64 acceptance build GREEN, and [`changelog-archive-2026-08-28.md`](changelog-archive-2026-08-28.md) § #134 acceptance run. NOT in `windows-backlog-archive-2026-08-26.md` — that archive (line 490) names #134 among the entries that deliberately stayed live. - **Layer headroom dispute** — SETTLED 2026-08-28. The final image sits at ~75 layers (counted from the inherited chain's instructions: base 16 + nvidia 3 + toolchain 4 + media-merge 15 + torch 3 + final 2 = 43, + 20 ENV + ~12 servercore = ~75). The ~108 figure was the pre-ENV-consolidation count. Updated in `docs/windows-build-invariants.md`. - **#122** — CUDA on arm64: CLOSED 2026-08-28 (owner decision — at the time NVIDIA shipped no Windows-on-ARM CUDA). **REOPENED as #176 on 2026-09-19:** CUDA 13.4.2 ships `windows_arm64` installers, and x64→ARM64 cross-compilation is documented and probe-proven here — the device is needed to run the result, not to build it. Archive: `windows-backlog-archive-2026-08-26.md` § #122. - **#136** — VS RUN caching: SOLVED + DEPLOYED 2026-08-26. Archive: `windows-backlog-archive-2026-08-26.md` § #136. - **#137** — sccache: DONE 2026-08-28, **LANDED**, SUPERSEDED 2026-09-18. The `SCCACHE_GIT_REV=8ab39266` source build landed and the full arm64 chain rebuilt green through it on 2026-08-31; 0.18.0 (2026-09-14) ships everything that pin carried, so the source build retired and the base now installs the released zip (`Install-RustToolchain.ps1`). The next base build re-pays the rust layer once. - **#134 free follow-ups** — ALL DONE 2026-08-28: smoke floor recalibrated (85→66), §19 PROVISIONAL marker removed, pin parity updated for `TVM_COMMIT`, TVM fixtures fixed (single-quoted `` `n `` → real newlines), three merge-stage test suites added (39 tests), resource sampler wired into `Build-Buildkit.ps1`. - **#133 (a)+(b)+(c)** — DONE 2026-08-26. Archive: `windows-backlog-archive-2026-08-26.md`. - **#128, #129** — DONE 2026-08-25/26. GStreamer webrtc/nice, OpenCV NEON dispatch. - **#124, #125, #126, #127** — DONE 2026-08-25. vcruntime140, sitecustomize, staged deps, import walk. - **#31 [owner decision] registry push** — push the verified images to a registry instead of local-only tags. Parked until the owner wants it (#59 branch protection was DECLINED, #31 was not). ## STANDING DIRECTIVES (survive their archived entries — do not re-litigate) - **#157 and #162 are DECLINED by the owner (2026-09-18).** The QNN payload keeps its five copies (`Copy-QnnRuntime` unchanged), and `versions.env` stays the whole-file COPY into base — one file, no generated Windows subset and no bake relocation. The ~4 h re-key a Linux-only pin edit costs is accepted. Do not re-propose either without new facts (a measured cost that changes the trade-off, or an upstream that ships WebNN where it matters). - **NEVER trim CUDA_ARCHITECTURES as a speed lever** (the current set is `86;87;89;120`, owner decision 2026-09-23 — 80/A100 retired, 120/Blackwell added; in ALL builds, incl. dev iterations; pinned by Pins.CanonicalValues). - **CUDA compiles go THROUGH sccache** — `SCCACHE_CUDA_LAUNCHER` is DEFAULT ON since 2026-08-18 (`Dockerfile.media-builder`'s `ARG SCCACHE_CUDA_LAUNCHER="1"`; the OpenCV stage via its own `ARG OPENCV_CUDA_LAUNCHER="1"`). **Do not flip it off silently** — the onnx stage then re-pays ~25 min of CUDA compiles per rebuild. Opt out per run with `-BuildArg SCCACHE_CUDA_LAUNCHER=`. CORRECTED 2026-08-31: this directive read "CUDA compiles stay BARE nvcc" and had been wrong since 2026-08-18. The 2026-08-10 miscompile it rested on was root-caused to sccache's dryrun quote-collapse and fixed upstream (mozilla/sccache#2811, merged 2026-08-19); patch 006 is retired (the "Patch 006" comment in `Build-OnnxFromSource.ps1`). A directive is the worst place for a stale rule — acting on this one would have cost ~25 min per rebuild against an explicit Dockerfile warning. - **Do NOT collapse the media-core checkpoints** (#72: export is ~1.2% of the chain; resume granularity is worth more than ~60 s). - **Do not re-propose** branch protection (#59) or a scheduled nightly/weekly chain run (#111) — DECLINED by owner 2026-08-17; manual launches are the verification cadence. - **Do not re-propose #147's declines** (#148): deleting `Get-LlvmMasmCmakeArg` or narrowing the `WindowsSourceBuild.Common` re-exports (external-consumer API), `Set-StrictMode` on the two consumer modules or the two dot-sourced scripts, or moving the handoff helpers off the facade. - **No logging-idiom sweep** (#110): chain scripts use Write-Host, gstreamer keeps its structured `log`, Write-BuildLog stays host-driver territory; enforcement is review, not a cache-busting mass edit. - **Do not re-propose QNN in LiteRT-LM** (#156, 2026-08-31). It is declined, not deferred. QNN only pays on arm64, where LiteRT-LM does not build (no windows_arm64 Bazel config, no prebuilts, `build:windows --copt=/arch:AVX2` on every Windows compile, cpuinfo's `[restrict static 1]` under clang-cl). On x64 it is worse than pointless: **QAIRT ships no `QnnHtpV*Stub.dll` for `x86_64-windows-msvc`**, so there is no path to a Hexagon DSP at all. `litert_lm_main` also parses `--litert_dispatch_lib_dir` and never reads it, and the NPU models are EAP-gated. Re-open only if upstream ships a windows-arm64 target. - **Do not re-propose #147's declines** (#148, 2026-08-31). Each was considered and refused with a reason: deleting `Get-LlvmMasmCmakeArg` or narrowing the `WindowsSourceBuild.Common` re-exports (the modules are external-consumer API — a zero-references audit is not evidence of zero consumers); `Set-StrictMode` on `WindowsFlutter.Common` / `WindowsContainerLog.Common` (same rule, and a module does not inherit its caller's strict mode), on `Initialize-CiEnvironment.ps1` or `Export-LitertLmBridge.ps1` (dot-sourced — it leaks, or is a no-op); moving `Export-`/`Import-BuildHandoff` off the hot facade (they are the #149 rollback payload). Full reasoning in the 2026-08-31 archive § Backlog wave. - **GES `_commit` retry is DORMANT INSURANCE** (#77): re-open only if its retry marker reappears in a gstreamer build. - **Restore `disk,webdav` only after WCOW cache mounts are PROVEN** (#99 re-verification recipe in the archive; also listed under Pending). ## Pending host/upstream actions (not refactors — do not let these evaporate) > The elevated between-runs window (buildkitd step-log env restore, GC-budget > deploy = #34, poisoned probe-chain prune, diagnostic tag cleanup) and the dufs > SYSTEM-service migration were APPLIED by the owner 2026-08-13 — see the archive > addendum. Sanity-check the GC deploy with `buildctl debug workers -v` — the > output says `Reserved space: 161.0612736GB` for a healthy 150 GiB pin, NOT > "150GB": the toml takes GiB and buildctl prints GB, and the labels differ too. - **UPSTREAM, consolidated 2026-08-17 (was scattered across #99's body and two Open-items entries):** 1. **moby/buildkit — WCOW cache mounts lose writes into an inherited directory.** Cause, A/B measurements and the 2-minute repro are in the archive (#99). Strengthen before filing: reproduce with PLAIN file writes (no sccache). Goes to moby/buildkit, NOT mozilla/sccache. 2. **mozilla/sccache#2808 addendum** — the issue's "WebDAV cache was largely empty" reasoning is now explained by the BuildKit mount defect (writes never reached the remote because L0 failed first under the default write-error-policy=l0). Core findings (nvcc deadlock + miscompile) stand; a two-sentence correction protects the report's credibility. Also note CUDA was launcher-off by default only between 2026-08-10 and 2026-08-18 and has been launcher-ON since (post-#2811); the repro is explicit either way — `Invoke-SccacheCudaLlmDeadlock.ps1:108` passes SCCACHE_CUDA_LAUNCHER=1 itself. 3. **Restore `disk,webdav`** once WCOW cache mounts stop losing inherited writes — owner intent; two-step re-verification recipe in the archive (#99): probe twice (ON-mount row must be clean on the SECOND, inheriting run), then one media build with the chain re-enabled and genai at 0 write errors. - **hcsshim follow-ups still unfiled** (package README status header, re-checked 2026-08-21): the ISSUE.md issue and the Windows-Containers#547 comment for microsoft/hcsshim#2855 (the draft PR itself IS filed; the package's submission recipe is now marked HISTORICAL so nobody files a duplicate). - **Post the upstream issues** — POSTED 2026-08-13: mozilla/sccache → https://github.com/mozilla/sccache/issues/2808 (nvcc deadlock + miscompile), google-ai-edge/LiteRT-LM → https://github.com/google-ai-edge/LiteRT-LM/issues/3245 (CMake-lane staleness, four findings). **POSTED 2026-08-24:** opencv/opencv#29788 (dnn/ORT `char*` vs `wchar_t`, from out/upstream-issue-opencv-ort-wchar.md) — now closed: upstream fixed it on `5.x` (opencv/opencv#29309) after the 5.0.0 tag we pin, so our `004-dnn-ort-profiling-wchar.patch` stays until `OPENCV_VERSION` moves past it ([`upstream-windows-patches.md` § Already filed](upstream-windows-patches.md#already-filed), checked 2026-09-18). **NEW DRAFTS (2026-08-24 evening, not posted — owner's call), both found by #116's first cross runs:** out/upstream-issue-iree-host-bin-dir-exe.md — `IREE_HOST_BIN_DIR` composes host tool paths without `.exe` on a Windows host; out/upstream-issue-iree-elf-arch-arm64-msvc.md — `MSVC_C_ARCHITECTURE_ID MATCHES 64` matches `ARM64`, archiving the x64 MASM object into an ARM64 library. That one is packaged as a ready-to-send PR since 2026-09-02, still not posted: [`windows/upstream/iree-elf-arch-x64-match/`](../windows/upstream/iree-elf-arch-x64-match/PR.md). - **REGISTRY of unfiled drafts under `out/` (added 2026-08-31 — the section title says "do not let these evaporate", and four of these were named nowhere):** `upstream-buildkit-wcow-cache-mount-draft.md` (item 1 above; its own header records the one strengthening step still open), `upstream-sccache-2808-addendum.md` (item 2), `upstream-issue-iree-host-bin-dir-exe.md`, `upstream-issue-iree-elf-arch-arm64-msvc.md`, `upstream-issue-meson-summary-build-subproject.md` (DRAFT, not filed; verified against meson 1.12.0), `upstream-issue-opencv-softfloat-neon.md`, `upstream-comment-for-win-14977.md` (a comment for docker/for-win#14977), and `upstream-llvm-aarch64-seh-instsize.md` — that last one is **not** llvm#219200: two commits are written, built and green on branch `aarch64-instsize-verify`, UNPUSHED. Already posted and needing no entry: `upstream-issue-sccache-nvcc.md` (#2808), `upstream-issue-litert-lm-cmake.md` (#3245), `upstream-issue-opencv-ort-wchar.md` (#29788). - **Post-run diagnostics queue — DEMOTED 2026-08-14: the sccache half is CLOSED.** The forensics had escalated this to "sccache has never worked, on any run" (0 hits / 189,861 failed writes across 94 stat blocks). **That was stale evidence, not a live defect** — the newest sccache stats in the whole corpus are from 2026-08-13 19:43, the dufs SYSTEM-service migration landed the same day, and every run since had media-core CACHED, so nothing could have shown the improvement. A direct probe (real cache mount, real WebDAV endpoint, one TU compiled twice) returned **miss → store → HIT, 0 write errors, 0 read errors**, and confirmed `HEAD`/`GET`/`PUT` succeed **from inside a container** — the direction `Assert-SccacheEndpoint` never tests. `SCCACHE_ERROR_LOG` is no longer "the missing artifact": the cache mount was found EMPTY, so there was never a log to recover. What remains here — RE-AUDITED 2026-08-31, and only item (2) is still open: (1) **CLOSED 2026-08-18/19.** The exact-TU replay exists and ran (`Test-OnnxTuReplay.ps1`, `bias_softmax_impl.cu`): it root-caused the miscompile to sccache's dryrun quote-collapse (verified bare 3189 == wrapped 3189 symbols; mozilla/sccache#2811 merged 2026-08-19), and the nvcc/CUDA crash proved to be #99 collateral, gone under a healthy backend. #75's `-j` ladder is no longer silent either — it warns before and after (archive 2026-08-21, item 75). (2) **RE-SCOPED 2026-09-18, run measured:** one `Test-BuildCopy.ps1 -Heavy` smoke after the poisoned-chain prune. With the RDNA4 dGPU ENABLED (RX 9070 XT, `Status OK`) it is red on both lanes with the known signature — `ActivateLayer 0x20` on the COPY commit, `failed to kill process on cancel` on the heavy RUN — which is the documented RDNA4 host behaviour, not evidence about the prune. Run it inside the build window after `Set-Rdna4Gpu.ps1 -Disable` (the chain's `Assert-NoActiveRdna4Gpu` preflight requires that anyway); a green heavy probe with the dGPU enabled is the only thing that would justify `-SkipRdna4Gate`. (3) **CLOSED.** The at-scale hit rate was measured twice on real media builds: the 2026-08-18 base ride (100.00 % CUDA/PTX/CUBIN, 207/816 hits) and the 2026-08-19 opencv run (99.97 % overall; opencv stage ~13 → ~4.3 min). VERIFIED 2026-08-14: the buildkitd service env now really does carry `BUILDKIT_STEP_LOG_MAX_SIZE=-1` + `..._MAX_SPEED=-1` (checked at the service registry key, and today's base build no longer emits the clip warning) — so the next full chain would be the FIRST fully-captured one. That chain has since run (the 2026-08-18 full-chain green), so a re-run of the forensics against a captured corpus is now possible and has not been done. The 49-run corpus analysed above predates this: 28 of those logs contain real clip events, the green reference run is 49 % blind in its merge step, and historical ONNX steps are 83 % blind. Re-run the forensics against a full captured chain.