Windows container chain — refactor backlog

Windows container chain — refactor backlog#

Open work, standing directives and pending host/upstream actions for the Windows container chain. Settled items are archived under docs/windows-backlog-archive-*.md.

The Linux-side equivalent is refactoring-backlog.md.

COUNTING NOTE: item numbers are HISTORICAL and never reused — the highest number is not the item count. Resolved narratives move to the dated archives (windows-backlog-archive-*.md); a bare “#N” that is not in this file resolves there. Lean-OPEN-only is the owner’s standing policy.

OPEN#

ARM64 parity (opened 2026-08-23)#

The :winarm64 cross lane reached runtime parity with :winamd64 apart from the exclusions listed below, and 2026-08-31 reran the FULL chain with the QNN EP enabled (#121 build-time path proven; the earlier acceptance gap — HEAD vs the 2026-08-26 tree — was two dead-on-arrival GStreamer cross-lane fixes that are now landed: see #135 follow-up below). Read the table as the fully green run of 2026-08-31 (bk-winarm64); a later full arm64 run is #176’s bk-20260920-203631 (smoke 120/0/15, arch gate 1047/0, CUDA on). The 2026-09-02 dual-lane rebuild (post-wave toolchain with sanitizers, rustup 1.29.1) reproduced both SMOKE columns exactly — arm64 97/0/15, amd64 222/0/0 (bk-20260902-002412); the other rows below are the 2026-08-31 measurements and were not re-extracted.

arm64 cross (bk-winarm64, 2026-08-31, QNN ON)

amd64 native (bk-20260826-130136)

arch gate

1168 binaries, 0 violations (QNN payload rides in)

1134, 0

import walk

793 files, 0 unresolved (85 device-OS: FastRPC + client SKU)

—

bundle manifest

7 DLL homes, 6 wheels, 3 ABSENT markers

—

GStreamer contract plugins

6/6 — libav opencv onnx webrtc nice tflite

6/6

smoke gate

97 passed / 0 failed / 15 skipped (floors 66/20)

222 / 0 / 0

QNN

EP ON in onnxruntime only (see #154); QAIRT 2.44.0 aarch64-windows-msvc runtime COPIED beside all five installs, but only ORT loads it

off (x64 CPU backend is pointless)

wall clock

~40 min (media+final)

2 h 18 min

Exactly three components are ABSENT on arm64, each marked in the bundle by an ABSENT-ON-ARM64.txt (LiteRT-LM, the default name) or a COMPILER-ABSENT-ON-ARM64.txt (the two compilers, which pass -FileName explicitly) — the manifest globs both families. Call sites, each a Write-AbsentOnCrossMarker call: Build-TvmFromSource.ps1, Build-IreeFromSource.ps1, Build-LitertAll.ps1: the TVM compiler, the IREE compiler (both need an LLVM cross-built for aarch64-windows) and LiteRT-LM (Bazel + an x86_64 prebuilt .lib; a CMake port exists upstream — #133(d)). Their python packages DO now ship (apache_tvm, apache_tvm_ffi, iree.runtime — closed by #133). Also excluded by owner decision or construction: the torch app stage (uv sync must execute the target interpreter). CUDA is no longer excluded by upstream facts: NVIDIA now ships Windows-ARM64 CUDA 13.4.2 installers, and the x64 host can cross-compile to ARM64 with them (documented in NVIDIA’s porting guide; probe-proven 2026-09-19 — AA64 machine (ARM64) — once the arm64 library archives are staged). Wiring that into the lane is #176. The QNN EP is PRESENT and PROVEN 2026-08-31 (build-time path; runtime execution still needs a Snapdragon host — see #121 below).

The honest caveat, smaller now: until 2026-09-25 nothing the arm64 lane produces had been executed. The consumer cross lanes’ windows-11-arm run jobs run their products since then, and the hub’s windows-arm64.yml runs the published bundle’s tools, Python, GStreamer pipelines and, since 2026-10-12, an ONNX Runtime session on the CPU EP (BACKLOG.md CON30). Inside the cross build itself every verdict above is still a static check — PE machine type, import resolution, exported symbols. The 15 skipped smoke assertions are the ones that would have to run the aarch64 payload on an x64 host.

Permanently out of reach — do not re-litigate without new upstream facts: classic TensorRT (genuinely x64-only — NVIDIA’s support matrix has no ARM64 row), and the torch app stage (uv sync must execute the target interpreter — uv can cross-RESOLVE into a directory, but the synced venv is the stage’s contract — and PyTorch builds no win_arm64 wheel for Python 3.14, this repo’s cp314 pin).

Open items#

  • #179 — every RUN that started an sccache server idles ~120 s before DONE: 16 to 27 min per chain (opened 2026-10-12, #153’s sweep). Cut to ~30 s on 2026-10-14 by the host’s TcpTimedWaitDelay; the patched sccache below would remove the rest. Counted from a RUN’s last output line to its DONE, the gap is 114 to 124 s in every such RUN: 9 RUNs and 1,077 s in bk-20261011-184301, 14 RUNs and 1,642 s (8.4 % of 19,592 s) in -233025, 8 RUNs and 955 s (13.2 %) in bk-20261010-121545. RUNs without a server close 1 to 9 s after their last line, a multi-GB Install-Rocm extract included, and the arm64 VVL RUN, whose launchers are cleared, closes 1.3 s after 513 s of compiling. The size of the RUN makes no difference. Probed 2026-10-12 with buildctl over bk-windows-toolchain, one RUN per variant and its own cache mount id. The tail comes from the sccache server ending gracefully:

    • A RUN with no sccache reaches DONE 1 s after its script ends.

    • With sccache --stop-server, DONE comes 120 to 123 s after the end. That holds for the webdav and the local-disk backend, with or without a compile, and also when the server process is gone 0.1 s after the stop and no process of the RUN is left. A server left running costs the same.

    • Stop-Process -Force on the server instead: DONE 1 s after the end, as without sccache. Killing an unrelated detached process costs nothing either.

    • Proven the same day: any TIME_WAIT in the container holds its teardown until it expires (Windows’ default TcpTimedWaitDelay, 120 s). One loopback pair without sccache, closed cleanly, costs 123 to 126 s; the same pair closed with a reset leaves no TIME_WAIT and costs 1 s.

    • The TIME_WAIT is the client’s. sccache 0.18.0’s client always closes first (commands.rs drops the connection after reading the response; the server reads to EOF), so every compile and every --show-stats leaves one on its ephemeral port. Killing the server therefore cannot help: compile + stats + Stop-Process -Force still cost 128 s.

    • Nothing in the container removes it. SetTcpEntry(DELETE_TCB) refuses a TIME_WAIT row (317). The registry value inside the container changes nothing (the stack is the host’s). SCCACHE_SERVER_UDS is ignored on Windows: 0.18.0 still listens on 127.0.0.1:4226.

    • The graceful stop protects no cache write in this chain. With SCCACHE_MULTILEVEL_CHAIN unset (webdav only), each compile awaits its PUT before the client gets CompileFinished. Under disk,webdav the webdav writes are detached tasks that the stop does not wait for either (multilevel.rs, write_remaining_levels_async). Two fixes, both owner decisions; the second is in place:

    • A patched sccache. The client sets SO_LINGER {on, 0} after connecting, and the server takes the resulting reset as a disconnect rather than logging Failed to bind socket. The patch is 56 lines on v0.18.0 (net.rs, server.rs, a windows-sys feature). The hub would build sccache from source again, as before 2026-09-18 (#114). The end step then kills the server instead of --stop-server, which also resets its pooled webdav connections. That removes the whole tail. The probe that builds and times it is written but has not run.

    • The host’s TcpTimedWaitDelay at 30 s, adopted 2026-10-14. Install-NewHost.ps1 sets it and Test-HostSetup.ps1 checks it (windows-host-setup.md § C6); it takes effect after a reboot. In the from-base chain bk-20261014-cpu the sccache RUNs (ONNX Runtime, FFmpeg) reached DONE 29.9 s and 30.2 s after their last line, and the RUNs without a server 0.9 to 3.3 s. That is about 90 s less per sccache RUN, 12 to 21 min per chain at the counts above. The servers that the launchers alone started (hailo, merge 2/9) need the same end step. In the same function, webdav NotFound (permanent) 404s are cache misses rather than failures, and the -Last 60 dump contradicts its “Failures in full” comment. Before upstream/hcsshim-teardown-timeout/ISSUE.md is filed, re-measure its “legitimate 117 s teardown”: that OpenCV specimen ran sccache as well. Rides the next full chain, since WindowsSourceBuild.Common is mounted into every media layer.

  • #182 — GStreamer’s meson build bypasses sccache on both arches, and its setup fetches ~50 wraps live (opened 2026-10-12, #153). The bypass is FIXED IN SOURCE the same day: meson’s compilers/detect.py passes an sccache prefix to the MSVC compiler class but drops it for clang-cl. Build-GstreamerFromSource.ps1 now builds a clang-cl.exe shim (an x64 host exe, so the arm64 cross file can use it) that runs "<sccache>" "<clang-cl>" <args>. It hands the shim to meson as CC/CXX, and the stage fails when sccache counted fewer than 100 compile requests. A buildctl probe on bk-windows-toolchain showed meson 1.12.1 detecting it as clang-cl 23.1.3 with lld-link, the linker it had before, and every ninja compile reaching sccache. The compile it saves is 663 s, 515 s and 612 s per merge rebuild in the swept chains. The next full chain proves it on GStreamer. Still open: meson setup takes 422 to 557 s and downloads 48 to 51 wrap sources live (the prefetch covers about 9), ninja then regenerates the build files once more (79 s, 161 s), and gst-plugins-rs (135 to 151 s) runs without RUSTC_WRAPPER. Put meson’s packagecache on a cache mount or prefetch every wrap, find the regenerate trigger with ninja -d explain, and set RUSTC_WRAPPER.

  • #183 — vendored code compiles under clang-cl’s /Wall: 1.36 M warning lines in one arm64 merge vertex (opened 2026-10-12, #153; BACKLOG.md CON89’s VVL item). VVL and x265 are FIXED IN SOURCE the same day: both builds run with CL=/w, which clang-cl reads before its own flags, and /w still beats a later -Weverything (probed in :winamd64: 3 warnings without it, 0 with it). The first fix used _CL_, which clang-cl appends after them; CMake calls clang-cl as ... -- <source>, so /w landed after the -- and read as a file name, failing x265’s compiler check in bk-20261012-190400. sccache hashes neither variable, and /w changes no code. Abseil in ORT and OpenCV’s matx.hpp stay open; they warn from inside our own builds, where a blanket /w would hide our warnings too. VVL’s GCC-style -Wall reaches clang-cl as /Wall, which is -Weverything. In bk-20261010-121545 that vertex is 6,228,195 lines and 752 MB of the 768 MB merge log (97.9 %); the “about 110k” CON89 quoted is one translation unit, vk_validation_error_messages.cpp. The same class, smaller: x265’s 27,423 -Wunsafe-buffer-usage lines in ffmpeg, abseil’s 19,001 to 29,957 -Wdeprecated-attributes in onnx, 7,200 from OpenCV’s matx.hpp. Fix: -w for vendored code (VVL’s $common, x265 in Build-FfmpegCodecs.ps1, ORT’s abseil flags), retry VVL with the sccache launchers back (cleared after the server hung twice on that unit), build it at -j19 rather than -j9, and $null = the two Invoke-GitClone calls that print a stray True. Rides the next full chain with CON89.

  • #185 — chain RUNs fetch live on every rebuild, and three fetches are unpinned (opened 2026-10-12, #153). Against the everything-in-the-image rule, a rebuild fetches the PyTorch clone and submodules (262 s in bk-20261011-233025), the LiteRT sources (88 s, 244 s), the IREE clone (~218 s), the aarch64 rust-std, the crates.io index for gst-plugins-rs, CPython’s PCbuild externals (the toolchain layer deletes them, and arm64 builds the target CPython twice per chain) and 32 wheels the uv cache mount still downloaded for torch. The three unpinned fetches are pinned in source (2026-10-12): Build-LitertLmBazel.ps1 took bazelisk from releases/latest, fell back to Adoptium’s latest/21/ga JDK and installed git-lfs through scoop, none SHA-checked. Get-PinnedTool now fetches bazelisk v1.29.0, Temurin 21.0.5+11 and git-lfs v3.8.0 by their versions.env pins (BAZELISK_WINDOWS_*, TEMURIN_JDK_WINDOWS_*, GIT_LFS_WINDOWS_*) into the tool cache, refuses any other bytes and re-fetches a corrupt cache entry. bazelisk itself gave way to the pinned bazel binary (LITERT_LM_BAZEL_WINDOWS_*, the clone’s .bazelversion) on 2026-10-13 (CON92). Still open: serve pinned source archives from a cache mount keyed by commit, bake the aarch64 rust-std into the toolchain image, and build the target CPython once and keep its externals. Rides the next full chain.

  • #178 — fold Expand-PinnedArchive into Expand-SourceTarball at the next media rebuild (opened 2026-09-25). Expand-PinnedArchive (WindowsMigraphx.Common) is Expand-SourceTarball’s two 7-Zip passes, except that it skips the links 7-Zip 25+ refuses as dangerous instead of failing (flatbuffers 25.12.19 carries nine in-tree ones). It lives apart only because WindowsSourceBuild.Common is mounted into every media layer, so an edit there re-keys the whole media lane. At the next deliberate media rebuild, give Expand-SourceTarball that tolerance, point Save-PinnedSource at it, delete Expand-PinnedArchive, and delete its row in docs/scripts/code-dupes.allow (the WindowsMigraphx.Common ↔ WindowsSourceBuild.Common pair). Rebuild-coupled like CON34 in BACKLOG.md.

  • #177 — the image history still names the build host’s endpoint (opened 2026-09-23). Since 2026-09-23 SCCACHE_WEBDAV_ENDPOINT reaches the compiling RUNs as an ARG, so no published ENV carries it and the publish gate refuses one that does. But BuildKit writes a RUN’s build args into that layer’s history created_by (|3 SCCACHE_WEBDAV_ENDPOINT=http://… RUN …), and the final image inherits every one. Nothing reads history at run time, so consumers are unaffected; it still publishes an RFC1918 address. The fix is to pass the endpoint as a secret (RUN --mount=type=secret,id=sccache_endpoint,env=SCCACHE_WEBDAV_ENDPOINT, with buildctl --secret), which records nothing. Verified on WCOW 2026-10-12 (buildkitd v0.33.0, a buildctl probe over bk-windows-toolchain): the env= form sets the variable in its RUN, the next RUN no longer sees it, and no file is left in C:\, C:\run, %TEMP%, C:\Windows\Temp or C:\ProgramData. Its value also stays out of the cache key, which an ARG’s does not. Never the file form on WCOW: a target=C:\… secret mount stays in the snapshot, so the next RUN still reads the file and the layer carries it (an upstream defect to report, owner’s call). Left: change all ten compiling stages at one planned re-key. Context: windows-build-resources.md § What the published image carries.

  • #155 — LiteRT QNN: the real flags, and why wiring them now would be a fourth silent no-op. RESEARCHED 2026-08-31. DO NOT integrate before reading this. The genuine switches, all verified defined at v2.2.0: LITERT_ENABLE_QUALCOMM (litert/vendors/CMakeLists.txt:330, default OFF) and QAIRT_HEADERS_DIR (:20), where setting the headers dir AUTO-FORCES the former ON (:331-334). QAIRT is consumed as headers only, at configure time — no import libs, no link-time dependency; backends are LoadLibrary’d by name at runtime. NOTE LITERT_ENABLE_NPU (litert/CMakeLists.txt:72) is NOT the QNN switch: its only effect is a #cmakedefine01 in build_config.h, and it does not gate add_subdirectory(vendors). Why it is not wired yet — five upstream defects, all Windows-specific:

    1. The dispatch DLL exports nothing on Windows: the SHARED target sets no export macro, no WINDOWS_EXPORT_ALL_SYMBOLS, no .def. It would build green and be unloadable — exactly #154’s failure mode, one level deeper.

    2. -Wl,--whole-archive / --no-undefined are passed on every non-Apple platform, Windows included; only if(APPLE) is carved out. GNU-driver syntax under lld-link.

    3. Both dynamic_loading.cc and dynamic_loading_windows.cc compile on Windows.

    4. The Windows branch of the vendor link lines is empty.

    5. litert/vendors/CMakeLists.txt does three unconditional configure-time downloads — NeuroPilot from AWS S3, and QAIRT 2.47.0.260601 from softwarecenter.qualcomm.com with no EXPECTED_HASH and no error check — whenever the corresponding *_HEADERS_DIR is empty. Two are escapable by pointing them at a dummy dir; the QAIRT one is escaped by setting QAIRT_HEADERS_DIR to our staged 2.44 tree, which also stops the version skew. The litert/ tree is also a SEPARATE top-level project (project(LiteRT VERSION 1.4.0)), not a flag on the tflite/ tree we configure — a second configure, and it pulls tflite in EXCLUDE_FROM_ALL, so tensorflowlite_c must be named explicitly. Upstream has never configured this tree for Windows: the only CMake CI is linux-x86_64/Android. And the arch gate cannot prove any of it — QNN backends are LoadLibrary’d by name, so a dispatch DLL adds zero static import edges. A green gate would mean nothing here.

  • #163 — -ConcurrentAux has never been used: ~24 min idle-capacity per full amd64 chain. 43 manifests, zero concurrent runs; litert 3535 s + tvm 1425 s always sequential; the 19 GB half-budget path works and no longer re-keys. The three ConcurrentAux fixes it was waiting on landed on 2026-09-17 (archive 2026-09-17), so it is unblocked: default it for full amd64 chains after one owed measurement — litert at 19 GB must not exceed the hidden 1425 s (its bazel half grew 18→59 min since July).

  • #176 — CUDA for Windows ARM64. PHASE 1 LANDED 2026-09-20 (built, smoke adapted; running it still needs a device). Opened 2026-09-19 on new upstream facts that supersede #122; the x64 host can cross-compile CUDA to ARM64 (NVIDIA’s Windows-on-Arm Porting Guide; probe-proven in out/probe-cuda-cross2/ the same day), so the device is needed to run the result, not to build it. Phase 1, all in tree and built:

    • Install-Cuda.ps1 -TargetArch arm64 stages the arm64 redist payload (cuda_cudart, libcublas, libcufft, libcurand, libnvjitlink; SHA pins in versions.env from redistrib_13.4.2.json) into the x64 toolkit’s root as lib\arm64 / bin\arm64, plus cudnn-windows-arm64-<pin>_cuda13.4-archive.zip (NVIDIA names the arm64 archive _cuda13.4, the x64 one _cuda13).

    • Get-NvccCudaCmakeArgs drives Hostx64\arm64\cl.exe + nvcc --use-local-env on the cross lane; Get-CudnnLibraryDir picks lib\arm64; Test-CudaWindowsArm64Payload is the cross POSITIVE signal — no host GPU probe decides a target flag.

    • Build-OnnxFromSource.ps1 builds the ORT CUDA EP for arm64: the 2026-09-20 run bk-20260920-031232 produced onnxruntime_providers_cuda.dll 0xAA64 in the win_arm64 wheel (2065-object ORT build), merge arch gate 1011/0, smoke 119/1/15 → the one red was the cuDNN run-probe, now a link + PE-machine assert (Assert-NativeLinkRun -CrossLinkOnly, same 1:1 substitution as §14).

    • -Gpu -TargetArch arm64 is a supported driver combination; EXPECT_GPU rides the cross smoke so a lost CUDA env reds instead of skipping. PHASE 2 LANDED 2026-09-20 (bk-20260920-203631, 2:45:53): OpenCV, GenAI and TVM build their CUDA paths for arm64 too.

    • Payload grew libnpp/libcusolver/libcusparse (OpenCV links CUDA::nppial/…; the first phase-2 configure died on the missing NPP targets).

    • OpenCV: arch-aware host compiler + --use-local-env + explicit arm64 CUDNN_LIBRARY; its one x86 intrinsic in the CUDA tree (_mm_popcnt_u64, cudafilters wavelet_matrix_2d.cuh) is patched to a software popcount (patches/opencv_contrib/002-..., probe-proven; nvcc’s device pass rejects __builtin_popcountll, so a guard tweak alone was not enough).

    • GenAI: onnxruntime-genai-cuda.dll 0xAA64 in the win_arm64 wheel.

    • TVM: USE_CUDA/CUBLAS/CUDNN=ON with the arm64 libs named explicitly (its legacy FindCUDA hardcodes lib\x64), CUDA_HOST_COMPILER=Hostx64\arm64\cl, CUDA_NVCC_FLAGS=--use-local-env.

    • Smoke 120/0/15; arch gate 1047/0; opencv_cuda*500.dll installed under lib\opencv5\arm64\vc18\bin. STILL OPEN: classic TensorRT (x64-only; TensorRT-RTX unwired) and the first RUN on an RTX Spark/N1x-class device. Do not re-derive #122’s August reasoning — the archive entry records what it rested on.

CLOSED (pointers — full narratives in the dated archives)#

  • #181 — the LiteRT layer no longer exports bazel’s trees. DONE 2026-10-12. Build-LitertLmBazel.ps1 stops the bazel server and removes C:\bzl, C:\llm, C:\bzl-tools and the bazel log after the install’s contract check; the merge copies only lib\litert-lm out of the layer. Proven on a local media-litert stage (bk-20261012-152007): 8.7 GB removed (6.66 + 1.72 + 0.33 GB), and the layer’s export took 52.9 s, against 1,126 s in bk-20261011-184301. Its unpack took 1,047 s, against 396 s there and 1,337 s in bk-20261011-233025, so unpack time follows the host, not this layer, and the stage came to 2,415 s against 2,728 s. bazel’s user root (1.21 GB) and bazelisk’s cache (0.11 GB) stayed; since CON92 the user root is C:\bzl-user and leaves with the rest, and bazelisk is gone. Bazel’s progress now reaches the stage log once per 30 s. Not done: a remote action cache for the 3,603 actions every rebuild compiles afresh.

  • #180 — the patched clang-cl relinks reproducibly. DONE in source 2026-10-12. CMake’s default compiler for the LLVM build was MSVC cl, and LLVM appends /Brepro only for a Clang compiler, so each relink stamped clang-cl.exe with the link time (0x6ACBBE2C in bk-20261011-184301, 0x6AC6CAC6 in the published :winamd64, same 92,396,032 bytes) and sccache, which keys on the compiler’s bytes, missed nearly every media compile after it. Build-LlvmFromSource.ps1 now builds with the base image’s pinned clang-cl (LLVM_USER_BIN) and fails the stage when the installed clang-cl.exe still carries a link-time stamp (Assert-ReproducibleLink). Proven on a local toolchain stage (bk-20261012-144845, 18:23): LLVM compiled cold with clang-cl in about 12 min, and clang-cl.exe carries the content hash 0x35341D1A. The next full chain rebuilds the media lane once against it; the relink after that is the one that keeps its hits.

  • #184 — a stopped BuildKit run keeps its record. DONE 2026-10-12. Build-Buildkit.ps1 writes bk-<run>-manifest.txt with status=ok or, from its finally, status=failed:<stage> and every finished stage, in the invariant culture (a German host had written patched-llvm=778,9 beside total_s=13058). bk-<run>-driver.log records the run, each stage’s start, end and failure, every retry verdict with the line that made it transient, and every disk-gate verdict (Write-DriverRecord). The resource CSV gains diskFreeGB for the build drive and -Summarize a MinDiskFreeGB column. buildctl’s output is decoded as UTF-8. Left out: a --progress rawjson copy, since buildctl takes one progress mode; the driver log’s stage stamps place the time before a stage’s first RUN.

  • #153 — the step-time / silent-retry sweep. DONE 2026-10-12. The 2026-09-01/02 logs it named were pruned (only manifests survive), so it ran over the three complete chains left, 51 logs: bk-20261011-184301 (amd64, 13,058 s, smoke 268/0/1), bk-20261011-233025 with -20261012-051453 (amd64 rocm, 19,592 s to the disk gate, then final and smoke) and bk-20261010-121545 (arm64, 7,211 s, smoke 147/0/17). The 441.3 s lifecycle floor is gone: a trivial RUN cmd /c exit 0 takes 4.9 to 9.0 s. One silent retry in three chains: rocm-llama’s first RUN over the fresh migraphx image hit ActivateLayer 0x20, and attempt 2/3 went green for about 70 s; the merge fan-in self-heal did not recur (every merge log is attempt 1/5, with no ActivateLayer). No clipped output in the 51 logs, the 6.2 M-line VVL vertex included. The rule for reading step times now: subtract ~120 s from each RUN that started an sccache server, and nothing from the others (#179). What the sweep found beyond its question is #179 to #185. Before 2026-08-30 the corpus was pruned too, so those forensic conclusions stay unverifiable.

  • **#158 / #159 / #160 / #164 / #167 / #168-#174 / #175 — the 2026-09-17 batch: the thirteen audited defects, the eight settled-probe deletions, the compiler-rt verify+tar port, LLVM through sccache, smoke section 23, the comment-discipline wave and the three neighbourhood checks. Landed in 2bb0410f (2026-09-18), unbuilt when archived. Archive: windows-backlog-archive-2026-09-17.md.

  • The 2026-09-03 doc-drift findings and the PascalCase anchors — all six doc defects fixed, deps.json’s sccache attribution corrected, and verify_doc_links.py green over the renamed script headings. Archive: same page.

  • #152 — the 2026-08-31 wave: PROVEN BY BUILD 2026-09-01/02. The dual-lane rebuild ran it for real, and the gate did its job: the wave’s compiler-rt ride-along had set COMPILER_RT_BUILD_SANITIZERS=OFF, the amd64 smoke gate failed on exactly the ASAN probe, the fix (=ON) re-ran and the lane closed at the best recorded state 222/0/0 ([PASS] AddressSanitizer …, clang_rt.asan_dynamic-x86_64.dll installs verified in the toolchain log); arm64 closed green at its baseline 97/0/15. The predicted re-pay was real (full LLVM + media on both lanes). Narrative: changelog-archive-2026-09-07.md § 2026-09-01/02.

  • #149 — the c9586c1^ warm/materialize rollback recipe: DEAD, not stale. FOUR independent breakages (every script path, the missing TargetArch + Tvm modules, the swapped media-core order, the QAIRT pin). The restore recipe is now a derivation rule in windows-build-lanes.md § Restoring the warm/materialize rollback — derive each RUN from the stage that runs that script TODAY. Archive: windows-backlog-archive-2026-08-31.md § Resolved 2026-08-31 (second pass).

  • #154 — QNN is integrated in ONE framework, not five: TVM, IREE and LiteRT were passing CMake flags upstream never defined, so CMake dropped them silently while all three logged success. Flags and banners removed on both lanes; the class is gated by Assert-CmakeArgsConsumed. The surviving lead is #155. Archive: same page.

  • #156 — LiteRT-LM QNN: DECLINED with reasons (also a standing directive below). Archive: same page.

  • #147 / #148 / #150 / #151 — the 2026-08-31 wave: classic driver deleted, the patched-llvm whole-dir module mount narrowed (and gated), TVM + GStreamer helpers moved to their leaf modules, Set-StrictMode plus the four latent bugs it exposed, the deliberate declines, the dead build.ps1 permission entries, and the CHANGELOG archive split. Seven commits, 7cc2e95f^..a5eac0dd. Full narratives: windows-backlog-archive-2026-08-31.md § Backlog wave #147-#151. Its declines are also a standing directive below.

  • #121 — QNN EP: BUILD-TIME PATH PROVEN 2026-08-31 on the full :winarm64 cross run — SDK qairt-2.44.0.260225 (SHA-pinned), onnxruntime_USE_QNN=ON with the aarch64-windows-msvc backend set, QNN provider built, QNN runtime copied beside all five installs — but only ORT consumes it (#154); arch gate 1168/0, smoke 97/0/15. Execution verification still needs a Snapdragon host. Archive: this entry.

  • -ResumeStage BK equivalent — CLOSED 2026-08-29 (no BK equivalent needed). The classic lane’s -ResumeStage preserved a stopped container (hours of compile state), injected a fix via docker cp, committed to a partial image, and re-ran from a specific stage. BuildKit has no intermediate containers to preserve — but it caches completed RUN vertices, so re-running buildctl automatically resumes from the last cached step. Script fixes land via bind mounts (re-read at solve time), so the “inject a fix and re-run” case is covered natively. No BK equivalent is needed.

  • #135 — LLVM 23.1.0 AArch64 codegen: DONE 2026-08-29. BUILD_PATCHED_LLVM=1 is now the DEFAULT (Dockerfile ARG + driver default), workarounds removed from Build-OpencvFromSource.ps1. -StockLlvm is the opt-out. Items 1+3 closed; item 2 (NINJA_KEEP_GOING census) is now unnecessary (root cause fixed); item 4 filed as llvm#219200. Archive: windows-backlog-archive-2026-08-26.md § #135.

  • #135 follow-up — patched toolchain lacks the aarch64 compiler-rt — DONE 2026-08-31. The source-built C:\llvm-patched ships clang_rt.builtins-x86_64.lib only, so the arm64 GStreamer link died on __udivti3 (2026-08-30 cross run, merge stage). The merge stage now self-heals: Build-GstreamerFromSource.ps1 § 5d mines clang_rt.builtins-aarch64.lib from the LLVM release archive on the cross lane (same recipe as Install-ScoopTools.ps1). Chosen over adding the lib to the toolchain layer because the media branches derive FROM bk-windows-toolchain, so that would re-pay ~2 h of media compiles for one lib. Toolchain-level fix LANDED 2026-08-31 (bd150ae1, Install-TargetCompilerRt in Build-LlvmFromSource.ps1) and shipped with the 2026-09-01/02 toolchain rebuilds — the merge-stage self-heal is now the never-firing fallback. Nothing left here. Regression: SourceBuild.GstreamerCompilerRt.Tests.ps1. Docs: docs/windows-cross-builds.md § aarch64 compiler-rt. Second unmasked failure fixed in the same window: the speculative cross-lane opus intrinsics enablement (2026-08-30) is REVERTED to the proven disabled state — under clang-cl aarch64 the RTCD path applies -mfpu=neon (ARM32-only flag) and its CPU probe uses MSVC’s __emit (absent from clang-cl). Working enablement recipe for a future TESTED window: -Dopus:intrinsics=enabled -Dopus:rtcd=disabled (presumes NEON+dotprod; needs a real-device smoke). Final gate fix in the same window: the arch-gate import walk gained the Qualcomm FastRPC pair libcdsprpc.dll/libadsprpc.dll as device-OS allowances (imported by the QAIRT HTP stub DLLs staged by Copy-QnnRuntime; they ship in every Windows-on-Snapdragon OS image, never in the SDK zip).

  • #133(d) — LiteRT-LM CMake port: CLOSED 2026-08-29 (owner decision — staying on Bazel).

  • #134 — post-#133 cleanup wave: DONE 2026-08-29. amd64 acceptance PASSED (smoke 192/0/1, arch gate 1134/0). Narrative: changelog-archive-2026-09-07.md § 2026-08-29 — amd64 acceptance build GREEN, and changelog-archive-2026-08-28.md § #134 acceptance run. NOT in windows-backlog-archive-2026-08-26.md — that archive (line 490) names #134 among the entries that deliberately stayed live.

  • Layer headroom dispute — SETTLED 2026-08-28. The final image sits at ~75 layers (counted from the inherited chain’s instructions: base 16 + nvidia 3 + toolchain 4 + media-merge 15 + torch 3 + final 2 = 43, + 20 ENV + ~12 servercore = ~75). The ~108 figure was the pre-ENV-consolidation count. Updated in docs/windows-build-invariants.md.

  • #122 — CUDA on arm64: CLOSED 2026-08-28 (owner decision — at the time NVIDIA shipped no Windows-on-ARM CUDA). REOPENED as #176 on 2026-09-19: CUDA 13.4.2 ships windows_arm64 installers, and x64→ARM64 cross-compilation is documented and probe-proven here — the device is needed to run the result, not to build it. Archive: windows-backlog-archive-2026-08-26.md § #122.

  • #136 — VS RUN caching: SOLVED + DEPLOYED 2026-08-26. Archive: windows-backlog-archive-2026-08-26.md § #136.

  • #137 — sccache: DONE 2026-08-28, LANDED, SUPERSEDED 2026-09-18. The SCCACHE_GIT_REV=8ab39266 source build landed and the full arm64 chain rebuilt green through it on 2026-08-31; 0.18.0 (2026-09-14) ships everything that pin carried, so the source build retired and the base now installs the released zip (Install-RustToolchain.ps1). The next base build re-pays the rust layer once.

  • #134 free follow-ups — ALL DONE 2026-08-28: smoke floor recalibrated (85→66), §19 PROVISIONAL marker removed, pin parity updated for TVM_COMMIT, TVM fixtures fixed (single-quoted `n → real newlines), three merge-stage test suites added (39 tests), resource sampler wired into Build-Buildkit.ps1.

  • #133 (a)+(b)+(c) — DONE 2026-08-26. Archive: windows-backlog-archive-2026-08-26.md.

  • #128, #129 — DONE 2026-08-25/26. GStreamer webrtc/nice, OpenCV NEON dispatch.

  • #124, #125, #126, #127 — DONE 2026-08-25. vcruntime140, sitecustomize, staged deps, import walk.

  • #31 [owner decision] registry push — push the verified images to a registry instead of local-only tags. Parked until the owner wants it (#59 branch protection was DECLINED, #31 was not).

STANDING DIRECTIVES (survive their archived entries — do not re-litigate)#

  • #157 and #162 are DECLINED by the owner (2026-09-18). The QNN payload keeps its five copies (Copy-QnnRuntime unchanged), and versions.env stays the whole-file COPY into base — one file, no generated Windows subset and no bake relocation. The ~4 h re-key a Linux-only pin edit costs is accepted. Do not re-propose either without new facts (a measured cost that changes the trade-off, or an upstream that ships WebNN where it matters).

  • NEVER trim CUDA_ARCHITECTURES as a speed lever (the current set is 86;87;89;120, owner decision 2026-09-23 — 80/A100 retired, 120/Blackwell added; in ALL builds, incl. dev iterations; pinned by Pins.CanonicalValues).

  • CUDA compiles go THROUGH sccache — SCCACHE_CUDA_LAUNCHER is DEFAULT ON since 2026-08-18 (Dockerfile.media-builder’s ARG SCCACHE_CUDA_LAUNCHER="1"; the OpenCV stage via its own ARG OPENCV_CUDA_LAUNCHER="1"). Do not flip it off silently — the onnx stage then re-pays ~25 min of CUDA compiles per rebuild. Opt out per run with -BuildArg SCCACHE_CUDA_LAUNCHER=. CORRECTED 2026-08-31: this directive read “CUDA compiles stay BARE nvcc” and had been wrong since 2026-08-18. The 2026-08-10 miscompile it rested on was root-caused to sccache’s dryrun quote-collapse and fixed upstream (mozilla/sccache#2811, merged 2026-08-19); patch 006 is retired (the “Patch 006” comment in Build-OnnxFromSource.ps1). A directive is the worst place for a stale rule — acting on this one would have cost ~25 min per rebuild against an explicit Dockerfile warning.

  • Do NOT collapse the media-core checkpoints (#72: export is ~1.2% of the chain; resume granularity is worth more than ~60 s).

  • Do not re-propose branch protection (#59) or a scheduled nightly/weekly chain run (#111) — DECLINED by owner 2026-08-17; manual launches are the verification cadence.

  • Do not re-propose #147’s declines (#148): deleting Get-LlvmMasmCmakeArg or narrowing the WindowsSourceBuild.Common re-exports (external-consumer API), Set-StrictMode on the two consumer modules or the two dot-sourced scripts, or moving the handoff helpers off the facade.

  • No logging-idiom sweep (#110): chain scripts use Write-Host, gstreamer keeps its structured log, Write-BuildLog stays host-driver territory; enforcement is review, not a cache-busting mass edit.

  • Do not re-propose QNN in LiteRT-LM (#156, 2026-08-31). It is declined, not deferred. QNN only pays on arm64, where LiteRT-LM does not build (no windows_arm64 Bazel config, no prebuilts, build:windows --copt=/arch:AVX2 on every Windows compile, cpuinfo’s [restrict static 1] under clang-cl). On x64 it is worse than pointless: QAIRT ships no QnnHtpV*Stub.dll for x86_64-windows-msvc, so there is no path to a Hexagon DSP at all. litert_lm_main also parses --litert_dispatch_lib_dir and never reads it, and the NPU models are EAP-gated. Re-open only if upstream ships a windows-arm64 target.

  • Do not re-propose #147’s declines (#148, 2026-08-31). Each was considered and refused with a reason: deleting Get-LlvmMasmCmakeArg or narrowing the WindowsSourceBuild.Common re-exports (the modules are external-consumer API — a zero-references audit is not evidence of zero consumers); Set-StrictMode on WindowsFlutter.Common / WindowsContainerLog.Common (same rule, and a module does not inherit its caller’s strict mode), on Initialize-CiEnvironment.ps1 or Export-LitertLmBridge.ps1 (dot-sourced — it leaks, or is a no-op); moving Export-/Import-BuildHandoff off the hot facade (they are the #149 rollback payload). Full reasoning in the 2026-08-31 archive § Backlog wave.

  • GES _commit retry is DORMANT INSURANCE (#77): re-open only if its retry marker reappears in a gstreamer build.

  • Restore disk,webdav only after WCOW cache mounts are PROVEN (#99 re-verification recipe in the archive; also listed under Pending).

Pending host/upstream actions (not refactors — do not let these evaporate)#

The elevated between-runs window (buildkitd step-log env restore, GC-budget deploy = #34, poisoned probe-chain prune, diagnostic tag cleanup) and the dufs SYSTEM-service migration were APPLIED by the owner 2026-08-13 — see the archive addendum. Sanity-check the GC deploy with buildctl debug workers -v — the output says Reserved space: 161.0612736GB for a healthy 150 GiB pin, NOT “150GB”: the toml takes GiB and buildctl prints GB, and the labels differ too.

  • UPSTREAM, consolidated 2026-08-17 (was scattered across #99’s body and two Open-items entries):

    1. moby/buildkit — WCOW cache mounts lose writes into an inherited directory. Cause, A/B measurements and the 2-minute repro are in the archive (#99). Strengthen before filing: reproduce with PLAIN file writes (no sccache). Goes to moby/buildkit, NOT mozilla/sccache.

    2. mozilla/sccache#2808 addendum — the issue’s “WebDAV cache was largely empty” reasoning is now explained by the BuildKit mount defect (writes never reached the remote because L0 failed first under the default write-error-policy=l0). Core findings (nvcc deadlock + miscompile) stand; a two-sentence correction protects the report’s credibility. Also note CUDA was launcher-off by default only between 2026-08-10 and 2026-08-18 and has been launcher-ON since (post-#2811); the repro is explicit either way — Invoke-SccacheCudaLlmDeadlock.ps1:108 passes SCCACHE_CUDA_LAUNCHER=1 itself.

    3. Restore disk,webdav once WCOW cache mounts stop losing inherited writes — owner intent; two-step re-verification recipe in the archive (#99): probe twice (ON-mount row must be clean on the SECOND, inheriting run), then one media build with the chain re-enabled and genai at 0 write errors.

  • hcsshim follow-ups still unfiled (package README status header, re-checked 2026-08-21): the ISSUE.md issue and the Windows-Containers#547 comment for microsoft/hcsshim#2855 (the draft PR itself IS filed; the package’s submission recipe is now marked HISTORICAL so nobody files a duplicate).

  • Post the upstream issues — POSTED 2026-08-13: mozilla/sccache → https://github.com/mozilla/sccache/issues/2808 (nvcc deadlock + miscompile), google-ai-edge/LiteRT-LM → https://github.com/google-ai-edge/LiteRT-LM/issues/3245 (CMake-lane staleness, four findings). POSTED 2026-08-24: opencv/opencv#29788 (dnn/ORT char* vs wchar_t, from out/upstream-issue-opencv-ort-wchar.md) — now closed: upstream fixed it on 5.x (opencv/opencv#29309) after the 5.0.0 tag we pin, so our 004-dnn-ort-profiling-wchar.patch stays until OPENCV_VERSION moves past it (upstream-windows-patches.md § Already filed, checked 2026-09-18). NEW DRAFTS (2026-08-24 evening, not posted — owner’s call), both found by #116’s first cross runs: out/upstream-issue-iree-host-bin-dir-exe.md — IREE_HOST_BIN_DIR composes host tool paths without .exe on a Windows host; out/upstream-issue-iree-elf-arch-arm64-msvc.md — MSVC_C_ARCHITECTURE_ID MATCHES 64 matches ARM64, archiving the x64 MASM object into an ARM64 library. That one is packaged as a ready-to-send PR since 2026-09-02, still not posted: windows/upstream/iree-elf-arch-x64-match/.

  • REGISTRY of unfiled drafts under out/ (added 2026-08-31 — the section title says “do not let these evaporate”, and four of these were named nowhere): upstream-buildkit-wcow-cache-mount-draft.md (item 1 above; its own header records the one strengthening step still open), upstream-sccache-2808-addendum.md (item 2), upstream-issue-iree-host-bin-dir-exe.md, upstream-issue-iree-elf-arch-arm64-msvc.md, upstream-issue-meson-summary-build-subproject.md (DRAFT, not filed; verified against meson 1.12.0), upstream-issue-opencv-softfloat-neon.md, upstream-comment-for-win-14977.md (a comment for docker/for-win#14977), and upstream-llvm-aarch64-seh-instsize.md — that last one is not llvm#219200: two commits are written, built and green on branch aarch64-instsize-verify, UNPUSHED. Already posted and needing no entry: upstream-issue-sccache-nvcc.md (#2808), upstream-issue-litert-lm-cmake.md (#3245), upstream-issue-opencv-ort-wchar.md (#29788).

  • Post-run diagnostics queue — DEMOTED 2026-08-14: the sccache half is CLOSED. The forensics had escalated this to “sccache has never worked, on any run” (0 hits / 189,861 failed writes across 94 stat blocks). That was stale evidence, not a live defect — the newest sccache stats in the whole corpus are from 2026-08-13 19:43, the dufs SYSTEM-service migration landed the same day, and every run since had media-core CACHED, so nothing could have shown the improvement. A direct probe (real cache mount, real WebDAV endpoint, one TU compiled twice) returned miss → store → HIT, 0 write errors, 0 read errors, and confirmed HEAD/GET/PUT succeed from inside a container — the direction Assert-SccacheEndpoint never tests. SCCACHE_ERROR_LOG is no longer “the missing artifact”: the cache mount was found EMPTY, so there was never a log to recover. What remains here — RE-AUDITED 2026-08-31, and only item (2) is still open: (1) CLOSED 2026-08-18/19. The exact-TU replay exists and ran (Test-OnnxTuReplay.ps1, bias_softmax_impl.cu): it root-caused the miscompile to sccache’s dryrun quote-collapse (verified bare 3189 == wrapped 3189 symbols; mozilla/sccache#2811 merged 2026-08-19), and the nvcc/CUDA crash proved to be #99 collateral, gone under a healthy backend. #75’s -j ladder is no longer silent either — it warns before and after (archive 2026-08-21, item 75). (2) RE-SCOPED 2026-09-18, run measured: one Test-BuildCopy.ps1 -Heavy smoke after the poisoned-chain prune. With the RDNA4 dGPU ENABLED (RX 9070 XT, Status OK) it is red on both lanes with the known signature — ActivateLayer 0x20 on the COPY commit, failed to kill process on cancel on the heavy RUN — which is the documented RDNA4 host behaviour, not evidence about the prune. Run it inside the build window after Set-Rdna4Gpu.ps1 -Disable (the chain’s Assert-NoActiveRdna4Gpu preflight requires that anyway); a green heavy probe with the dGPU enabled is the only thing that would justify -SkipRdna4Gate. (3) CLOSED. The at-scale hit rate was measured twice on real media builds: the 2026-08-18 base ride (100.00 % CUDA/PTX/CUBIN, 207/816 hits) and the 2026-08-19 opencv run (99.97 % overall; opencv stage ~13 → ~4.3 min). VERIFIED 2026-08-14: the buildkitd service env now really does carry BUILDKIT_STEP_LOG_MAX_SIZE=-1 + ..._MAX_SPEED=-1 (checked at the service registry key, and today’s base build no longer emits the clip warning) — so the next full chain would be the FIRST fully-captured one. That chain has since run (the 2026-08-18 full-chain green), so a re-run of the forensics against a captured corpus is now possible and has not been done. The 49-run corpus analysed above predates this: 28 of those logs contain real clip events, the green reference run is 49 % blind in its merge step, and historical ONNX steps are 83 % blind. Re-run the forensics against a full captured chain.