Windows build invariants (do not regress)

Contents

Windows build invariants (do not regress)#

Load-bearing fixes for the Windows lane. Preserve them or builds slow down, ship broken, or start lying about their own results. Every entry here was found live — the date in a heading’s body is when it cost a build.

These are agent-behavioral rules: what you must not do when editing the Windows chain. They are deliberately separate from the reference data (component matrix, script tables, build sequences), which lives in windows-builds.md, and from the symptom lookup, which lives in failure-modes.md.

Editing one of these? The rule and its evidence travel together. If you change a mechanism, update the entry in the same commit — a rule whose evidence no longer matches the code is worse than no rule.

Contents#

Shell and PowerShell discipline

Gates that must stay armed

Diagnosing: probes and evidence

Logs

Layers, scratch and image size

Lanes, isolation and CPU

Container networking (CNI)

Build inputs and toolchain

Shell and PowerShell discipline#

pwsh 7 everywhere — no powershell.exe, no cmd SHELL#

pwsh (PowerShell 7) everywhere — owner policy 2026-08-04. Every SHELL directive, every in-container exec, every script runs pwsh 7. The ONLY sanctioned Windows PowerShell 5.1 context is the single bootstrap RUN in Dockerfile.base that installs pwsh itself (nothing else may run in that window). Never add powershell/powershell.exe invocations or cmd SHELL directives; cmd.exe /c may appear only inside Invoke-ShieldedNative and the documented bespoke sites.

Never cross a pwsh process boundary with an array parameter#

Never cross a pwsh process boundary with an array parameter. & pwsh -File script.ps1 -Param 'a','b' delivers ONE literal string INCLUDING the quote characters, not a two-element array. Cost (2026-08-18): the deadlock repro’s -BuildArg pair reached buildctl as one mangled undeclared ARG name, buildctl silently discarded it, and an 88-minute repro measured nothing while reporting success. Invoke repo scripts DIRECTLY (& 'windows\Build-Buildkit.ps1' …) when already in pwsh 7; Build-Buildkit.ps1 now throws on non-identifier -BuildArg keys so the flattened form fails loudly instead of vanishing downstream.

Never put double quotes inside shell-form RUN lines#

Never put DOUBLE QUOTES inside shell-form RUN lines in the Windows Dockerfiles. The dockerfile frontend strips embedded " from the command string before it reaches the pwsh SHELL (three incidents on 2026-08-04: "$env:TEMP\*" became bare $env:TEMP\* → ParserError; the pwsh-written Directory.Build.props lost its XML attribute quotes → MSB4024). Use single quotes / ''-doubling / string concatenation instead; XML attributes may legally use single quotes.

Import-Module -Force only at entry-script top level#

Import-Module X -Force is safe ONLY at ENTRY-script top level — never in a module, and never in a script that gets &-invoked FROM module scope. A forced re-import from module context unloads the caller’s top-level copy and rebinds it into the module’s private session state (probed 2026-08-05: Import-Versions.ps1’s Import-Module Shared -Force, run via Import-CanonicalVersions, made Resolve-DirectoryPath CommandNotFound at gstreamer top level and killed the merge-warm solve). Nested imports use the guarded form if (-not (Get-Module -Name 'X')) { Import-Module $path }. Regression pin: import Shared→Installer→SourceBuild.Common, run Import-CanonicalVersions, then Get-Command Resolve-DirectoryPath must still resolve. The “REPO-COMPLETE since 2026-08-05” claim that stood here was WRONG, and it cost a 53-minute compile on 2026-08-21. Every leaf builder (Build-OnnxFromSource.ps1, Build-OpencvFromSource.ps1, and the FFmpeg, GStreamer, TVM, LiteRT and IREE builders, …) still opened with Import-Module $modulePath -Force — and those are precisely “scripts &-invoked from module scope”: the chain runs them in-process via & (Join-Path $ScriptDir $stage.Script). ONNX built green for 53 min; the chain tail then died on The term 'Stop-LingeringBuildProcess' is not recognized, because -Force had removed the module instance Invoke-SourceBuildChain was still running inside. Read the asymmetry in the log — it is the fingerprint of this bug: the EXPORTED call one line earlier (Write-SccacheStats) succeeded, because exported names resolve through the global command table, while the UNEXPORTED helper existed only in the destroyed scope. A prose rule with no gate is a suggestion: windows/scripts/tests/Modules.ForceImportScope.Tests.ps1 now discovers the leaves from the $stages tables and fails on any -Force among them (with a rot guard, so it cannot pass vacuously).

Never splat a string array of -Param-shaped tokens#

Never splat a string ARRAY containing -Param-shaped tokens onto a PowerShell script/function — array splatting binds strictly BY POSITION. & $script @('-ResumeFrom','OpenCV') delivers -ResumeFrom as the VALUE of parameter 1 (silently wrong without CmdletBinding, “positional parameter cannot be found” with it) — this killed the opencv warm solve on 2026-08-04 and reproduces identically on host pwsh 7.6. Route such argv through a child process instead (& pwsh -NoProfile -File $script @argv — native argv is re-parsed into named parameters; Invoke-BkWarm.ps1 is the reference), or splat a HASHTABLE. Splatting arrays onto native executables stays fine.

Four more pwsh traps: bareword comma-attributes, switch-name collisions, glued parameter tokens, null env restores#

Four more pwsh traps — (a) and (b) found live 2026-08-10 in Test-BuildCopy.ps1, (c) and (d) on 2026-08-25 in the arm64 target-cpython and LiteRT stages (regression pins: windows/scripts/tests/Native.ArgQuoting.Tests.ps1 for (a)–(c), which are AST traps in Invoke-Lint.ps1, and SourceBuild.HostArchLibEnv.Tests.ps1 for (d)): (c) A parameter token glued to its argument PARSES: Get-PeFileMachine -Path$staged.FullName is a parameter literally NAMED Path$staged (the .FullName becomes a positional argument) and dies at runtime with “A parameter cannot be found that matches parameter name ‘Path$staged’”; -Path(Join-Path ...) binds the paren expression positionally, right only by accident. Four such calls survived a refactor, the parse gate and 662 tests and surfaced 90 s into the arm64 regression. Native tools are exempt — for clang-cl/nasm the glued token IS the syntax (-Fotiny.o, -I<dir>) — the detector inspects Verb-Noun commands with an approved verb plus Assert-*. (d) Restoring a captured $null with [Environment]::SetEnvironmentVariable does not unset the variable. PowerShell binds $null to the string parameter as '', and pwsh 7.6 then leaves NAME= DEFINED-EMPTY in the process block (Test-Path env:NAME → True, and every native child inherits it as set). Measured 2026-08-25, arm64 runs 16/17: the LiteRT stage never enters VsDevCmd, so LIB is unset there and lld-link auto-detects the MSVC/SDK directories — until the host-flatc pass “restored” LIB= and the very next target try-compile died with could not open 'kernel32.lib' (IREE/TVM restore a real value and never noticed). Save-and-restore code must branch: if ($null -eq $saved) { Remove-Item Env:NAME } else { SetEnvironmentVariable } ([NullString]::Value also works); pin: windows/scripts/tests/SourceBuild.HostArchLibEnv.Tests.ps1. (a) a BAREWORD comma-attribute native argument (buildctl --output type=local,dest=$outDir) parses as an ArrayLiteral and the exe receives the VERBATIM SOURCE TEXT — no variable expansion, no comma split (buildctl exported into a directory literally named $outDir). Double-quote the whole attribute string (--output "type=image,name=$tag"). (b) Variable names are case-insensitive, so a local $docker = "...\docker.exe" in a script declaring [switch]$Docker assigns to the switch-typed PARAMETER variable and throws Cannot convert ... String to ... SwitchParameter at runtime — the probe’s docker lane had never executed. Never reuse a switch parameter’s name for a local variable.

Windows stage scripts must end with an explicit exit 0#

Windows stage scripts must end with an explicit exit 0. pwsh -File propagates the LAST native command’s exit code: a fully green LiteRT-LM build was declared failed because the final cleanup rmdir exited 145 (ERROR_DIR_NOT_EMPTY). Real failures throw (EAP=Stop + gates); reaching the end IS success — say so explicitly.

Production scripts run under Set-StrictMode -Version Latest#

Every entry script gets Set-StrictMode -Version Latest, and adding it is a BUG HUNT, not a formality (2026-08-31). Adding it to seven scripts surfaced four latent defects — three of them on NORMAL SUCCESS PATHS. Two traps account for all four, and both are about .Count/.Sum on a pipeline result: (a) @()-wrap a pipeline result before touching .Count — on pwsh 7.6.5 a one-element result is the scalar (.Count throws) and an empty result is AutomationNull (.Count throws too), so the working and the empty case both fail while only the two-or-more case passes. Set-TensorrtTree.ps1’s $dllDirs hit exactly this on its success path (TensorRT 10+/11 keep the DLLs in bin only, so precisely one dir survives the filter), and Copy-CudaRuntime.ps1’s $roots would have re-broken the arm64/CPU merge lane that the 2026-08-23 degrade-cleanly fix unblocked. (b) Measure-Object -Property emits NOTHING for empty input, so an inline (… | Measure-Object Length -Sum).Sum throws on an empty directory — bind the result first (Clear-SccacheMount.ps1, empty cache dir). StrictMode is INHERITED across &-invocation, so a script can already be under it without saying so. Debug-LitertlmLink.ps1’s (Get-Command 'llvm-nm.exe' -EA SilentlyContinue).Source was a LIVE bug for that reason: its only caller, Build-LitertLmFromSource.ps1, sets StrictMode, and a missing llvm-nm — the case the code was written to handle — threw instead. Deliberately EXCLUDED: modules and dot-sourced scripts (WindowsFlutter.Common, WindowsContainerLog.Common, Initialize-CiEnvironment.ps1, Export-LitertLmBridge.ps1). A module does not inherit its caller’s strict mode, and a dot-sourced script leaks it into every caller — in-repo and external — so those are behaviour changes, not hygiene.

Gates that must stay armed#

Every BK chain ends with a mandatory smoke gate#

Every BK chain ends with a MANDATORY smoke gate — do not route around it. Build-Buildkit.ps1 solves windows/Dockerfile.smoke-gate against the finished image after final, and a failure fails the chain (backlog #44). It is the only smoke gate since the classic driver’s deletion on 2026-08-31 (mechanism and gating history: docs/windows-builds.md § Smoke Testing). Three rules when touching it: it must run through entrypoint.cmd (a bare RUN bypasses ENTRYPOINT and loses VsDevCmd + the ASAN runtime dir); it bind-mounts the current script rather than the image’s baked copy, so a smoke-test fix is re-verifiable without an image rebuild; and it enforces coverage floors (-SmokeMinPassed/-SmokeMaxSkipped, exit 3), because a run that asserted nothing used to print “All smoke tests passed!” and exit 0. -SkipSmokeGate is for chain iteration only. On -TargetArch arm64 the gate is arch-SPLIT (2026-08-24). Test-Container.ps1 mostly verifies by EXECUTING the staged binaries, and Windows x64 has no ARM64 emulation — so the payload sections keep floor 0 on arm64 and the driver reports them NOT APPLICABLE, never “passed”, while the host-toolchain and static sections (1-6, 14-16, 19 arch-filtered: TORCH_APP_DIR dropped, 23 without the torch-baked script, and 25, the ORT census) RUN with their OWN floor column, and the healthcheck likewise runs its host-tool checks, skipping only payload execution. Sections 14/15 compile FOR the target and assert the produced PE machine instead of running (ASAN skipped there — LLVM’s win-x64 package ships no aarch64-windows ASAN runtime). The arm64 floor values and the measured green runs live in docs/windows-builds.md § Smoke Testing. Never lower the amd64 floors toward the arm64 column: a reduced -SmokeMinPassed would leave a number a later amd64 change could quietly be measured against, which is exactly how this gate became decorative once before. The cross lane’s execution-side verification is Test-TargetArch.ps1 (PE machine type over C:\runtime AND the fanned-in site-packages, .lib archives included, inside the merge stage, with its own floor, ARCH_GATE_MIN_INSPECTED = 580 on arm64) plus the fact that every artifact linked at all — neither proves the code RUNS, and nothing available on this host can.

Never rewrite upstream sources with a bare string(REPLACE)#

Never rewrite upstream sources with a bare string(REPLACE). Use patch_replace_required / patch_regex_replace_required from patches/litert-lm/patch-assert.cmake, which FATAL_ERRORs when the pattern matched nothing (backlog #56). The old pattern printed “Patched …” unconditionally, so an upstream reformat silently restored the defect the patch existed to fix — for sentencepiece’s duplicate ABSL_FLAG(minloglevel) that means a link-clean litert_lm_main.exe that aborts on every run. Patches.CmakeNoOpGuards.Tests.ps1 fails any unguarded replace; a genuine non-source replace opts out with a patch-assert-exempt marker AND a reason.

AVX-512/AMX flags never go in global CXX flags#

AVX-512/AMX flags NEVER go in global CXX flags (final polarity, settled 2026-08-03). Globally, clang may emit AVX-512 anywhere — the in-tree protoc AND onnxruntime.dll’s static initializers both crashed with STATUS_ILLEGAL_INSTRUCTION at run/load time on the AVX2-only build host (the import assert catches this). But entirely without the flags, MLAS’s arch TUs fail to COMPILE (clang-cl gates intrinsics behind target features; MSVC doesn’t). The settled design: Build-OnnxFromSource.ps1 appends Get-WindowsTargetKernelSimdFlags -Arch per-TU (the name this line carried until 2026-08-24, Get-WindowsX86Avx512Flags, was a zero-caller compat shim until its deletion on 2026-08-26) to exactly the MLAS arch FLAGS = lines in build.ninja post-configure (runtime-dispatched kernels — the only code allowed to assume the features) and asserts the tagged count against Get-MlasKernelTuMinimum — a THROW, not a log. Two field lessons shape that floor: on aarch64 the x86 pattern matches nothing and a no-match patch succeeds (why the pattern is arch-parameterized), and on 2026-08-24 the amd64 lane broke with the floor PRESENT but too low — ORT v1.29.0 added six AVX-512 TUs outside intrinsics/, the stale pattern still matched 5 ≥ floor 4, and five kernels failed to compile. Floor rule: it must be high enough that the previous broken state trips it (now 8, against the 11 that v1.29.0 matched; the stale pattern’s 5 fails loudly). When bumping ONNXRUNTIME_VERSION, re-measure BOTH arches’ patterns against the new MLAS tree — the 1.29 bump re-measured only aarch64 and amd64 paid for it. Don’t “simplify” in either direction.

The mandatory GStreamer plugin set is a contract, never auto#

The mandatory GStreamer plugin set is a CONTRACT, never auto (2026-08-07). Get-RequiredGstPlugin (WindowsGstPlugins.Common.psm1) is the SINGLE definition of which integrations must exist — libav, opencv, onnx, webrtc, nice, tflite since 2026-08-25 (#128), plus WebRTC’s run-time set dtls, srtp, sctp, rtpmanager, openh264 and the gst-plugins-rs rswebrtc/rsrtp since 2026-10-05 (CON28), the same thirteen on BOTH lanes — enforced at four points that must never disagree: the pkg-config pre-flight in Build-GstreamerFromSource.ps1, meson features set to enabled (meson’s auto means skip silently — never use it; tflite’s flag is presence-driven -Dgst-plugins-bad:tflite=enabled), a post-install gate that throws (by exact DLL name, gst<name>.dll), and smoke-test assertions that fail, ending in a WebRTC loopback. Any arch filtering lives in the CONTRACT, never in a caller — teaching only one enforcement point about an arch would recreate the 2026-07-11 regression. tensorfilter is an NNStreamer element, not a GStreamer plugin — never add it to the set. Deliberate exception: -SkipPluginGate, which marks the image unshippable. The root causes, per-plugin mechanisms and the hardened export-marker gate: docs/windows-builds.md § Mandatory GStreamer plugins (the contract); the cross-lane arch conditionals: docs/windows-cross-builds.md § The merge stage on arm64.

A missing stage artifact is a THROW, not a warning#

A missing stage artifact is a THROW, not a warning. media-litert once “completed” without litert_lm_main.exe (configure had failed; the script only warned) and the degraded image would have shipped through merge/final. Every stage’s terminal artifact check must throw (debug escape hatches env-gated, e.g. LITERTLM_KEEP_BUILD_TREE).

The host gets artifacts, never CMake state#

Sync-FastLocalArtifactsToHost must keep CMakeCache.txt in /XF and CMakeFiles in /XD (2026-09-06). The Flutter lane builds in a container-local fast root and robocopies the result back onto the bind-mounted host build tree (Sync-FastLocalArtifactsToHost in windows/scripts/modules/WindowsFlutter.Common.psm1 — consumer API, no in-repo caller). A CMakeCache.txt is bound to the directory and the generator that wrote it, so copying one back poisons the NEXT run: flutter build windows --config-only reconfigures the host tree with the Visual Studio generator, finds the scratch cache and aborts — “The current CMakeCache.txt directory C:/ws-mnt/build/windows/x64/CMakeCache.txt is different than the directory c:/kataglyphis_fast_build/build/windows/x64 where CMakeCache.txt was created”, followed by “Does not match the generator used previously: Ninja”. CMakeFiles is excluded for the same reason. It hid for weeks because the consumer swallowed the exit code and still reported every step green, so the step failed on every run without anyone seeing it. Widen the copy with artifacts whenever a consumer needs one; never let build STATE cross back.

ONNX Runtime has exactly one source: the chain (owner rule 2026-09-23)#

Every ONNX Runtime consumer uses the ORT this chain builds: Build-OnnxFromSource.ps1 → ONNX_ROOT, plus its wheel in C:\runtime\wheels. Never a NuGet package, a PyPI wheel, a release zip, a plugin-EP wheel, a pyke build or Windows ML’s System32 copy — no exceptions. Two consumers broke this unseen until 2026-09-23. OpenCV’s dnn fetched onnxruntime-win-x64-1.25.1.zip at configure time, and GenAI’s ortlib.cmake restored Microsoft.ML.OnnxRuntime.DirectML 1.24.4 from a feed; neither was hash-pinned. The image looked clean both times, because neither copy’s DLLs shipped. That is why a gate on the shipped bytes alone is not enough.

Do not regress any of these:

  • OpenCV keeps -DHAVE_ONNXRUNTIME=ON, the nested-header shim and its CMake hook directory. GenAI keeps ORT_HOME and both FETCHCONTENT_SOURCE_DIR_* pointed at the empty dir, and never sets USE_WINML (it replaces ORT_HOME with a Windows ML NuGet ORT).

  • Every consumer build ends in Assert-ChainOrtOnly (G2), with at least one log, before its tree is removed: OpenCV, GenAI, FFmpeg, GStreamer and the AMD GPU EP. WindowsOrtProvenance.Build.psm1 reaches those five RUNs as a per-file mount at C:\bkmnt\ortmods\ only. Never put it in buildmods, migraphxmods, WindowsSourceBuild.Common or Build-MediaCoreAll.ps1, and never merge it into the census module: one edit would re-key five compile RUNs.

  • Smoke section 25 (the ORT census) and its STAMP assertion stay armed on every lane, the arm64 bundle included. An exemption goes into $ortCensusExemption with a reason, and an in-box Windows ML ORT is never exempted (INBOX refuses it). Never delete or patch files under C:\Windows in a layer (fix11 refuses a line doing so to an ORT or Windows ML file); a base that ships one keeps the previous WINDOWS_BASE_DIGEST: § The in-box ONNX Runtime. Its $script:OrtPathMarker matches a whole NUL-terminated source path, never the bare directory, which oxidant.dll carries as a string: § What “the chain ORT” is. A PE that exports OrtGetApiBase (not as a forwarder) is an ORT instance under any name: § An ORT under another name.

  • Section 19’s crate-env row and section 21’s DirectML and chain-wheel checks run on every amd64 lane, with no lane condition above them.

  • WindowsOnnx.Common refuses NuGet ORT, and the WebGPU EP is built in-tree, never installed as a plugin wheel.

  • Renaming a gate function means updating F11_GATE_CALLS in linux/scripts/verify-critical-fixes.sh in the same change. fix11 also greps this heading verbatim, so do not reword it.

The six guards, every consumer, the stamp contract and what each guard does NOT cover: onnxruntime-single-source.md. The Linux side of the rule is in AGENTS.md § Linux Build Rules.

A build-host setting is an ARG, never ENV — the publish gate refuses one#

Anything that describes the BUILD HOST reaches a RUN as an ARG and never lands in an ENV (2026-09-23). :winamd64 shipped SCCACHE_WEBDAV_ENDPOINT=http://192.168.188.116:5000 from four ENV blocks, and on every consumer runner sccache 0.18 died on the unreachable endpoint, so CMake reported clang-cl broken. Do not regress any of these:

  • The three build-host sccache names (SCCACHE_WEBDAV_ENDPOINT, SCCACHE_MULTILEVEL_CHAIN, SCCACHE_FORCE_LOCAL) are ARGs with no default, declared in EVERY compiling stage above its RUN. An ARG does not cross a FROM, and the media-core chain crosses solves (FROM ${MEDIA_CORE_*_IMAGE}). Never “restore” one as an empty ENV: that is the shape that leaked.

  • A new compiling stage (a RUN mounting both C:\sccache and C:\sccache-logs) declares ARG SCCACHE_WEBDAV_ENDPOINT in its own stage, or the static lint fails it — without the ARG it compiles uncached and says nothing.

  • The publish gate stays armed at all three points: on every BASE_IMAGE a stage inherits that the run did not build (in Invoke-BkStage, before the solve), on the fresh toolchain, and on the final image after the smoke gate and before any export or push. -SkipSmokeGate skips none of them. An ENV crosses FROM, so a parent built before a fix carries the leak into every stage after it; restart a chain across such a fix, never resume one. Never give the gate an ARG after FROM or route it through the entrypoint; either changes the environment it grades.

  • The same holds for any future build-host knob — a LAN mirror, a proxy, a cache server: ARG, never ENV. The gates refuse a LAN address in any ENV value.

The table of which sccache variables ship and why, the three gates and what they cannot see (hostnames, image history — backlog #177): windows-build-resources.md § What the published image carries.

Diagnosing: probes and evidence#

When a probe says the product is broken, suspect the probe first#

When a probe says the product is broken, suspect the probe first — and always run a known-good control. Three probes lied on 2026-08-14 before one told the truth, each looking exactly like a product defect: (1) a hand-rolled [DllImport("kernel32")] LoadLibraryW(string) marshals string as ANSI by default, so a UTF-16 API answered “module not found” (126) for all 14 TensorRT DLLs — the repo’s Assert-DllLoads (WindowsSmokeTest.Common.psm1) has always declared CharSet=CharSet.Unicode; use the existing helper, don’t re-declare P/Invoke. (2) A hand-written probe Dockerfile without # escape=\`` let the default ` escape eat the s in target=C:\sccache, so a cache mount silently did not exist. (3) /Fo:C:\x.obj (colon) makes sccache build C:\:C:\x.obj and fail with a misleading “failed to zip up compiler outputs”; MSVC syntax is /Fo<path>. In each case the control is what exposed it — a VCRUNTIME140.dll that obviously loads in an MSVC-built image, a mount that should exist. A probe with no control cannot distinguish “broken product” from “broken probe”.

Reproducing the ENVIRONMENT is not reproducing the FAILURE#

A probe that reproduces the ENVIRONMENT does not necessarily reproduce the FAILURE — and until it does, it can clear nothing. Test-SccacheWrite.ps1 ran the real image, the real cache mount ids and the real ENV, and for two days every configuration it pronounced clean then failed in the next 90-minute build: a repaired cache tree, a fresh SCCACHE_DIR, the multilevel chain, 16-way concurrency, 239-character paths. It was not lying — its writes really did succeed. It was simply too SMALL: the defect needs ~250 objects written into a directory a PREVIOUS container populated, and every section until then wrote a single object. Two rules from that: (a) treat a green probe as a hypothesis to test in a real build, never as clearance — only per-stage --show-stats numbers from a build settled anything here; (b) when a probe and the product disagree, the next move is to make the probe BIGGER along the dimension you have not varied, not to trust it. See backlog #99 for the full list of dead hypotheses.

A probe can destroy its own experiment#

A probe can destroy its own experiment — read the setup output, not just the verdict. The same script sweeps anything in the cache root that is not a hex bucket off the mount as debris; that quietly deleted the inheritance fixture a later section depended on, and the run reported “0 files inherited” — which reads as “the cache mount lost 250 objects”, i.e. exactly backwards. The directory listing printed at the top of the log is what exposed it. Keep fixtures on an explicit allow-list ($probeOwned), and when a probe’s result is surprising, check what the probe DID to the system before believing what it says about the system.

Aggregate evidence has a shelf life#

Aggregate evidence has a SHELF LIFE — check the newest sample’s timestamp against the last fix. The log forensics concluded “sccache has never worked” from 0 hits / 189,861 failed writes across 94 stat blocks. Every one of those samples predated the dufs SYSTEM-service migration on the same day, and no run since had exercised sccache — a direct probe showed miss → store → HIT with 0 write errors. Confident conclusions about a state that no longer exists are the failure mode of corpus-wide aggregation; date the newest sample before trusting the aggregate.

Logs#

A daemon’s log is only as durable as its last flush#

A daemon’s log is only as durable as its last flush — stop the server before the RUN ends. SCCACHE_ERROR_LOG is written by the sccache SERVER, and SCCACHE_IDLE_TIMEOUT=0 means it never exits on its own, so BuildKit tears the RUN’s process tree down with the log still buffered and nothing reaches the mount. Complete-SourceBuildChain now calls sccache --stop-server as the chain epilogue (after every Write-SccacheStatsToStderr, which needs a live server), which also flushes the async webdav write-through tail. Invoke-SourceBuildChain’s prologue is the other half: it stops the server, truncates the error log, and starts the server explicitly from C:\. The truncation is not tidiness — the log lives on a SHARED mount and only appends, so the epilogue’s dump was replaying a PREVIOUS run’s failures verbatim (50,928 lines / 12,413 error lines) and cost a full false alarm before anyone compared its timestamps to the run’s start time. Dump -Last N, never -First N, on any log that accumulates. Diagnostic value of this one: the path, the level and the mount were all correct for days while three separate hypotheses were chased — LRU pruning, wrong location, unset SCCACHE_LOG — because a hand probe that waits a few seconds with the server alive ALWAYS saw content, and a real build never did. When a log is empty only after real runs, suspect lifetime before correctness.

Never put a log inside a directory another tool prunes#

Never put a log inside a directory some OTHER tool owns and prunes. SCCACHE_ERROR_LOG was set to C:\sccache\logs\sccache-error.log — inside SCCACHE_DIR, the directory sccache itself manages by LRU. The dir-creation code runs, the cache mount persists (236 MB of content survived), and the logs\ directory is gone anyway: sccache pruned it. That is why sccache’s own error log was unobtainable through an entire multi-day investigation — it was deleted by design, not lost by accident, and its absence is what left the genai write failures undiagnosable (backlog #90). It now lives on its own cache mount — C:\sccache-logs (id=sccache-logs-winamd64), mounted next to the sccache mount on every compiling RUN — and is listed in windows/buildkitd.toml’s tier-0 inventory, which must stay in step with that budget. Sibling rule to the one below: a log must live where nothing else has a delete policy over it.

A build log written inside the build dir dies with the solve#

A build log written inside the build dir DIES WITH THE SOLVE — always use Get-PersistentBuildLogPath. When a vertex fails, BuildKit discards the container filesystem, so a log at $buildDir\x-build.log is gone exactly when it is needed and the only diagnosis left is the 50-line tail Invoke-NinjaBuildWithRetry prints. The helper (WindowsSourceBuild.Common.psm1) therefore puts logs on the persistent C:\sccache-logs mount (derived from SCCACHE_ERROR_LOG’s parent, so the two cannot drift apart) with one .prev generation, falling back to the build dir only when no mount exists. Never $SCCACHE_DIR\logs — it wrote build logs straight into sccache’s own LRU-managed cache ROOT, which is the same mistake #90 had already fixed for the error log; corrected 2026-08-16. Pass the result as -LogFile; never hand-roll the path, and never omit -LogFile (the ONNX GenAI builder did, and produced no ninja log at all). This lived as ONE inline block in Build-OnnxFromSource.ps1 for months while opencv/iree/tvm/litert silently lost their logs — hence a shared helper (backlog #43). Corollary of the owner’s standing “never swallow logs” rule.

Never swallow logs — display may truncate, persistence must not#

NEVER swallow logs — display may truncate, persistence must not (owner directive 2026-08-10). Every tool that shows -Last N lines Tee’s the FULL stream to out\build-logs\ first and prints the path; sccache’s server error log persists on its OWN cache mount (SCCACHE_ERROR_LOG=C:\sccache-logs\sccache-error.log in Dockerfile.media-builder — NOT inside SCCACHE_DIR, see #90 — the 2026-08-10 nvcc-decomposition postmortem had only client-side 10054s because the server died with its logs); build stats go to STDERR (survives the step-log clip); and BUILDKIT_STEP_LOG_MAX_SIZE=-1/MAX_SPEED=-1 on the buildkitd service is a REQUIRED host setting — a Stevedore reinstall/repair wipes the service env silently (found empty 2026-08-10; that clip hid guard verdicts and stats for three runs). Build-Buildkit.ps1 refuses to launch without MAX_SIZE=-1 (Assert-BuildkitdStepLogEnv; -SkipStepLogGate skips it for one launch). Verify with Install-NewHost.ps1 -ReportOnly; re-apply + Restart-Service buildkitd only between builds.

Layers, scratch and image size#

Scratch must be scrubbed INSIDE the layer that created it#

Scratch must be scrubbed INSIDE the layer that created it — image layers are additive. Deleting package-manager scratch (NuGet restore, pip cache, %TEMP%, INetCache) from a LATER layer only writes a whiteout; the bytes still ship. Until 2026-08-07 only the last media-core partition scrubbed, so the onnx/opencv/ffmpeg/litert/tvm/gstreamer layers each carried their own forever. Every chain wrapper now takes -ScrubAfter and every heavy RUN in the BK lane passes it; the shared epilogue is Complete-SourceBuildChain. Safe because Clear-BuildScratch targets $env:TEMP (the container profile temp) — Install-Vs.ps1 repoints $env:TEMP to C:\temp but only inside its own process in the base layer, so C:\temp\cpython and the mounted script/patch trees are never touched. Check that before widening it further: a scrub of C:\temp would delete the CPython tree the merge stage fans in.

A committed layer can never be shrunk later#

A committed layer can never be shrunk later, so scrub INSIDE the container. The BK lane — the only lane since 2026-08-31 — passes -ScrubAfter to the media branch and merge/GStreamer runs (Clear-BuildScratch: pip cache, ~\.nuget, %TEMP%, INetCache). It was added there first; the classic lane went without it until 2026-08-07 and shipped debris the BK lane’s images did not. Source trees are a separate mechanism — each leaf script calls Remove-SourceBuildTree itself. The toolchain stage is excluded on purpose: its CPython tree at C:\temp\cpython IS the deliverable.

Preserve committed line endings when editing a COPY’d .psm1/.ps1#

Preserve committed line endings when editing a COPY’d .psm1/.ps1. Media modules are LF, some build scripts CRLF; core.autocrlf=true plus some editors can flip a whole file, busting the media layer cache. .gitattributes pins these -text; after editing, confirm git diff shows only your change, not a whole-file EOL flip.

Windows images have a hard 125-layer cap#

Windows images have a HARD 125-layer cap — it binds any image LOADED INTO DOCKER, whichever builder produced it. The final stage died with max depth exceeded on 2026-08-03 because the merge Dockerfile carried ~28 separate ENV lines, one layer apiece, under the since-deleted classic builder. BuildKit keeps metadata in the image config, so a BK solve spends layers only on RUN/COPY/ADD — but -FinalTar hands the result to docker load, which enforces the ceiling again. Rule: in every windows Dockerfile, consolidate ENV/metadata into single instructions (see Dockerfile.media-merge-builder’s one big ENV, layers 114→86). When adding stages/instructions, check headroom: docker inspect <tag> --format '{{len .RootFS.Layers}}' chain-wide; the final image currently sits at ~75 layers (settled 2026-08-28 by counting the inherited chain’s RUN+COPY+ADD+ENV instructions: base 16 + nvidia 3 + toolchain 4 + media-merge 15 + torch 3 + final 2 = 43, plus 20 ENV layers and ~12 from the servercore base = ~75). The earlier “~108/125” figure was the pre-ENV-consolidation count — the merge-builder’s 28 ENV lines → 5 blocks alone removed ~23 layers.

A layer that replaces a lower layer’s file stores its name lowercased#

A Windows container layer that deletes or moves away a file or directory a lower layer holds, then creates the same name again, stores the new entry lowercased. Measured 2026-10-07 in buildctl solves on servercore ltsc2025 (BuildKit v0.33.0, containerd v2.4): Diff.py came back as diff.py, and a moved-away DirUp as dirup. An in-place overwrite keeps the name. So do a rename, a delete and recreate of a file the same layer created, and a delete and a create in two different layers. BuildKit’s COPY replaces every file it overwrites, so a COPY lowercases each file that already exists at its destination. Directories it merges into keep their names.

NTFS opens either spelling, so nothing notices, until CPython’s import, which compares the directory listing exactly. The fan-in’s two site-packages COPYs shipped Cython\shadow.py, and import Cython failed in the published image (failure mode). Rules:

  • Never fan two trees that share files into one path with COPY. Merge them in one RUN from bind mounts with an in-place copy, as the media fan-in does with Merge-SitePackageTree. Bind a stage with one RUN layer over the branch image, not the image itself (failure mode).

  • Install a Python distribution once, in the layer that needs it. A pip install --force-reinstall, or an upgrade over a lower layer’s install, moves the old package away and creates it again, so its top-level names come back lowercased.

  • The RECORD check is the gate. Find-RecordCaseMismatch runs at the end of the merge and in the smoke gate’s section 2 over the base interpreter.

docker commit inherits the container’s Cmd#

HISTORICAL since 2026-08-31 — no run+commit site remains in the repo (build.ps1 and Invoke-RunCommitStage are deleted). Kept because nothing gates this: a re-introduced docker commit would repeat it silently.

docker commit inherits the container’s Cmd — always commit with --change 'CMD ["pwsh"]' (classic lane, fixed 2026-08-07). The run+commit stages launched the container with a build-script argv, and commit captured it as the image’s CMD: local/kataglyphis:windows-media and windows-torch shipped a CMD that RE-RUNS the GStreamer build, so docker run -it local/kataglyphis:windows-media starts recompiling over C:\runtime instead of opening a shell. The FINAL image was never affected (a Dockerfile ENTRYPOINT resets an inherited CMD), which is why it hid for months. Any new run+commit site must carry the same --change.

Lanes, isolation and CPU#

media-core builds via run+commit — never re-add --isolation process#

HISTORICAL since 2026-08-31 — Invoke-RunCommitStage was deleted with build.ps1; media-core is a plain BK solve at full CPU count. The docker build finding below is still true of this host, and the re-test pointer is still a valid hand run.

media-core built via run+commit for CPU parallelism — never re-add --isolation process. docker build is 2-CPU-capped here and process isolation cannot commit layers (hcsshim::ActivateLayer 0x20, reproduced even for a 100 MB dummy). media-core therefore ran as docker run --isolation hyperv --cpu-count $MediaCoreCpus + docker commit (Invoke-RunCommitStage) — the only way to get >2 CPUs and a committable image. Regression symptom of the era: -j2 in out\windows-build-logs\media-core.log, or ActivateLayer on any commit. Full rationale: docs/windows-build-lanes.md § Build isolation and CPU parallelism. Before assuming this is still needed after a Docker/Windows/base-image upgrade, re-check with windows/scripts/diagnostics/Test-ProcessIsolationCommit.ps1 — if it reports BUG GONE, process isolation for docker build is usable again and the workaround can be retired (see windows-build-lanes.md § Re-testing process isolation on new versions).

Parallelism is memory-bounded, not CPU-bounded#

Parallelism is memory-bounded, not CPU-bounded — and the defaults ARE the max. Get-BuildJobCount = min(ProcessorCount, MEMORY_LIMIT_GB / MemGBPerJob); every BK RUN is process-isolated and sees all host logical processors (32 here). ONNX is tuned to ~4 GB/job (its CUDA/AVX-512 TUs are the RAM-heaviest; the -j2 incremental retry absorbs the occasional OOM) → ~-j10 at the auto-detected -MediaMemoryGb 39 (61 GB usable − 22 GB host reserve). Do not “optimize” by raising the memory cap or cutting -HostReserveGb: the verified maximum envelope for this host (32 CPUs / 39 GB; media-core bottomed the host at 0.2 GB free and survived; 53 GB deadlocked it) is documented in docs/windows-build-resources.md § Maximum resource envelope — average CPU of ~35–45 % during compiles is the expected memory-bound signature, not a tuning failure. You cannot reach -j32 on ONNX: 32 heavy TUs need ~128+ GB — RAM per job, not core count, is the ceiling; the real speed levers are more physical RAM and the sccache WebDAV remote. The local L0 disk tier is DISABLED BY DEFAULT since 2026-08-16 — and the fault is BuildKit, not sccache. A BuildKit cache mount on Windows loses writes once its directory holds objects an EARLIER RUN wrote: 158 of 250 failed on the mount vs 0 of 250 into a plain container directory, same program, same moment; a FRESH directory on the same mount takes all 250. That is why opencv/genai (which inherit onnx’s cache dir) failed ~100 % of writes while onnx, filling its own, failed 1.9 %. SCCACHE_MULTILEVEL_CHAIN is an ARG with no default in every compiling stage (unset means WebDAV only); restore disk,webdav with -BuildArg SCCACHE_MULTILEVEL_CHAIN=disk,webdav only after re-verifying against a newer buildkit (recipe: Test-SccacheWrite.ps1 in docs/windows-builds.md; full measurement: docs/windows-backlog-archive-2026-08-17.md § P0d, #99). Do not “fix” this in sccache.

buildctl builds (non-admin), nerdctl runs and inspects (admin)#

BOTH lanes are supported: buildctl (NON-ADMIN) builds the chain, nerdctl (ADMIN, always) runs/inspects/administers — and can build too. Verified 2026-08-07: nerdctl build with BUILDKIT_HOST=npipe:////./pipe/buildkitd produced and stored an image; nerdctl run --network nat gets a routable IP. The admin requirement is upstream, not a misconfiguration — nerdctl opens \\.\pipe\containerd-containerd for EVERY subcommand (even build --output type=tar), and containerd has no --group equivalent to buildkitd’s --group docker-users; checked against its full flag set and default config, --address only moves the pipe. Do NOT attempt pipe-ACL hacks (recreated on every restart; containerd access is machine-admin) and do NOT re-litigate this — the legitimate route is an upstream containerd feature request. The chain keeps using buildctl deliberately: nerdctl build is a wrapper around the same buildkitd, does not expose the load-bearing --opt image-resolve-mode=local, and would force every unattended build to run elevated. Always pass --namespace buildkit (the bk-* images live there). Recipes + traps: docs/windows-build-lanes.md § nerdctl lane.

Never pass a command to nerdctl run of an image with an ENTRYPOINT#

Never pass a command to a nerdctl run of an image that has an ENTRYPOINT — it is appended as entrypoint ARGUMENTS, not substituted. On bk-winamd64 that exits 255 instantly and leaves a container whose rm -f then BLOCKS for up to 45 minutes, because the patched shim waits for a teardown instead of force-terminating (right for builds, painful interactively). Use no command (the final image’s entrypoint.cmd starts pwsh itself) or --entrypoint. Zombie recovery, safe only when the container did no real filesystem work: Get-Process containerd-shim-runhcs-v1,CExecSvc | Stop-Process -Force then rm -f again. Exit 3221225786 = 0xC000013A = the container was Ctrl+C’d.

Invoke-BkStage -MaxAttempts — the media merge stage passes 5#

Invoke-BkStage -MaxAttempts defaults to 3; the media MERGE stage passes 5. It fans in three branch images, so it does far more mount work than any other stage and flakes proportionally — 2026-08-06 it burned its whole 3-attempt budget (two failed to mount {windows-layer} failures, green only on the last try). Retries are cheap: completed RUN vertices stay cached, only the failed finalize/export re-runs. Raise the per-stage budget rather than the global default.

The classic lane’s fallback needs a RUNNING dockerd#

HISTORICAL since 2026-08-26 — this no longer guards a usable lane; build.ps1 was retired then and deleted 2026-08-31 (rationale: windows-build-lanes.md). Kept because the host observation below is still true and still bites docker.exe publish/inspect work: a Stevedore host can have docker on PATH with the daemon Stopped. Heading unchanged on purpose — it is a live anchor target.

The classic docker lane’s “always-working fallback” needs a RUNNING daemon, and on a Stevedore host that daemon is the stevedore SERVICE. stevedore IS dockerd (...\Stevedore\dockerd.exe --run-service --service-name stevedore --host npipe:...dockerDesktopWindowsEngine --containerd=npipe:...). Found Stopped on 2026-08-07 while the BuildKit lane ran happily — i.e. the documented fallback was unavailable and nothing said so. docker.exe sitting on PATH proves nothing — and since 2026-08-31 nothing fail-fasts on it: Assert-DockerDaemon was deleted with build.ps1. Test-HostSetup.ps1 reports the service state (WARN: “docker.exe publish/inspect unavailable”), which is the only check left. Starting it is NOT a safe reflex: a dockerd start recreates the nat HNS network and can move the subnet out from under 0-containerd-nat.conf, leaving BuildKit containers with unroutable IPs — so start it deliberately, then RE-CHECK the CNI subnet (Build-Buildkit.ps1 fail-fasts on that drift with the exact fix).

Two BK-driver guards to know before debugging around them#

Two guards added 2026-08-07 that change how the BK driver behaves — know they exist before debugging around them. (1) Invoke-TransientCooldown now takes -PreviousTail and refuses to retry a byte-identical failure: a flake changes between attempts, a poisoned snapshot does not, and the old behaviour burned the whole retry budget on ImportLayer 0xb7. The comparison strips buildkit’s per-line elapsed-time prefixes, which differ every attempt. One class is retried even when identical: snapshot-mount contention (failed to mount {windows-layer, failed to calculate checksum of ref; -RetryDespiteIdenticalPattern), measured to go green on a later attempt. (2) Invoke-BkStage gates disk per stage with a stage-aware floor (Get-StageDiskFloorGb: 60 GB for the CUDA sdk, 55 for the ONNX solve, 40–45 for the rest; recalibrated since the first table’s 80 GB media floor refused a good rebuild — windows-build-lanes.md § Driver preflight gates and isolation policy) because the start-of-run check passed at 164 GB while the chain still walked to 23 GB inside one heavy stage. Both honour the existing override switches; both were BK-only when added, and since 2026-08-31 there is no other driver to differ from.

Container networking (CNI)#

The CNI nat config must exist as BOTH .conf AND .conflist#

The CNI nat config must exist as BOTH 0-containerd-nat.conf AND 0-containerd-nat.conflist — the two clients disagree and each breaks silently without its own file. NEVER “convert” one into the other.

  • buildkitd needs the .conf. Without it, RUN steps get no network adapter at all — not a DNS problem: measured 2026-08-07 with a probe container, ipconfig was EMPTY and a raw TCP connect to a literal GitHub IP returned “unreachable network”; the containerd debug log showed the HcsCreateComputeSystem spec for buildkitsandbox with Storage, MappedDirectories and MappedPipes but no networking block. Surfaces as Could not resolve host: github.com at the first downloading RUN.

  • nerdctl needs the .conflist. It indexes plugins[0] with no length check and dies index out of range [0] with length 0, in network create (netutil_windows.go:40) AND in run (container_network_manager.go:857). Upstream nerdctl bug (it should return an error), worth reporting.

This entry previously said the opposite (“containerd and BuildKit read either”) and that error cost a launched chain on 2026-08-07: the .conf was converted away to fix nerdctl, which silently killed buildkitd’s networking, and nothing caught it because no chain build ran in between. Restoring the .conf beside the .conflist + Restart-Service buildkitd -Force fixed it on the spot (IPv4 172.31.44.107, GW 172.31.32.1, DNS 192.168.188.1, github.com resolved). Build-Buildkit.ps1 now fail-fasts via Get-CniConfFormIssue. When you edit one file, edit both — nothing detects the two drifting apart in CONTENT, only absence.

Two guards, two different failures, do not conflate them: Get-CniNatSubnetDrift judges only subnet-vs-adapter drift and reads .conflist then .conf (its “file absent = nothing to judge” contract means a conf under any other name silently disables it — and it passed green the entire time the lane had no network). Get-CniConfFormIssue judges presence of the form each client needs. ALWAYS re-run a network canary after touching either file — it is load-bearing for every media compile via sccache.

The CNI .conf is DERIVED from the .conflist, not hand-edited#

The CNI .conf is DERIVED from the .conflist, not hand-edited (2026-08-07). Set-ContainerdConfig.ps1 rewrites it via ConvertFrom-CniConfList whenever the two differ, so the two forms the clients each require cannot drift in content. Edit the .conflist and re-run that script; a multi-plugin conflist is refused rather than truncated to plugins[0].

Build inputs and toolchain#

TVM builds its OWN minimal LLVM — do not “simplify” it away#

TVM builds its OWN minimal LLVM (#47, 2026-08-17) — do not “simplify” it away. Scoop’s LLVM (official Windows installer) ships NO llvm-config.exe and no dev libs anywhere (probed: 0 hits), so every earlier Windows TVM was silently USE_LLVM=OFF — no CPU codegen, tvm.build for llvm targets dies at runtime. The official clang+llvm-*-windows-msvc dev tarball is NOT the fix: its static libs are /MT and want xml2s.lib, fatally mismatched against this /MD chain (verified by one link attempt). The heal in Build-TvmFromSource.ps1 builds a SHA-pinned llvm-project from source (targets from Get-TvmLlvmTargetList: X86;AArch64;NVPTX, plus AMDGPU on the ROCm spike; no xml2/zlib/tests, LLVM_ENABLE_DIA_SDK=OFF — no ATL in these Build Tools, RTTI ON, full-:FILEPATH archiver) and passes USE_LLVM=<path>/llvm-config.exe. Since the patched toolchain LLVM (C:\llvm-patched, AArch64;X86) sits first on PATH on amd64, TVM links that one, and the heal runs only under -StockLlvm and on the rocm lane’s TVM_ROCM=1 spike, where Get-TvmLlvmChoice treats a PATH llvm-config without AMDGPU as absent. Keep the heal: it is the spike’s only AMDGPU-capable LLVM. The ~6 min sccache-warm figure is for the X86;AArch64;NVPTX build; with AMDGPU it is unmeasured.

freedesktop/videolan GitLab downloads must go through Invoke-WrapDownload#

freedesktop/videolan GitLab downloads MUST go through Invoke-WrapDownload (curl-native UA + gzip/bzip2 magic-byte check, in WindowsMeson.Common.psm1 since #134 — a merge-lane leaf module, NOT in the shared buildmods closure) — the Anubis anti-scraper in front of those hosts answers browser UAs without JS with an HTTP-200 HTML challenge page, which is exactly what the shared Invoke-DownloadWithRetry sends. Also strip .git from GitLab /-/archive/ URLs (with it, GitLab serves HTML even to curl). Both burned a merge run each on 2026-08-17; do not “unify” wrap fetching back onto the shared helper.

graphene needs -Dgraphene:sse2=false plus the meson.build patch#

graphene builds only with -Dgraphene:sse2=false plus the post-setup meson.build patch dropping its -Werror=undef (it outranks our c_args — last flag wins; clang-cl defines no __GNUC__, and its MSVC SIMD path calls SSE4.1 intrinsics unguarded). graphene entered the build for the FIRST time when #88 made every wrap actually arrive — expect more first-ever subprojects to surface clang-cl corners after wrap fixes.

In RUN --mount=...,from=<stage> the SOURCE path is Unix-style#

In RUN --mount=...,from=<stage> the SOURCE path is Unix-style with NO drive letter, even on Windows containers. source=C:\bkmods is normalised to /C:/bkmods and fails the solve with failed to calculate checksum of ref ...: "/C:/bkmods": not found — it is a PARSE-time/cache-key failure, so it dies the moment the stage is reached, not inside the container. Write source=/bkmods; the COPY that populates the stage keeps the Windows form (C:\bkmods), and target= stays Windows-shaped too. Only the from-stage source is Unix. Cost a chain launch on 2026-08-07 (buildmods closure stage in Dockerfile.media-builder / .media-merge-builder). Verify a from-stage mount with a Test-Path assertion through it, NOT with Get-ChildItem — an empty mount lists cleanly and exits 0, so a bare listing proves nothing.

A whole-directory modules mount puts EVERY module in the cache key#

Never bind-mount windows/scripts/modules as a DIRECTORY — mount the per-file closure the script imports (2026-08-31). Every file under a directory mount is part of that RUN’s cache key, so one host-only .psm1 edit re-keys the whole vertex. Dockerfile.toolchain-builder’s patched-llvm RUN did exactly this, and it is the DEFAULT toolchain target (-StockLlvm is the opt-out, #135): editing any module — WindowsBuildDriver.Common.psm1 included, which no container ever loads — re-paid a full LLVM 23.1.0 compile plus every media lane derived from that image. That RUN now mounts the six modules Build-LlvmFromSource.ps1 actually imports, and BuildKit.ModuleClosure.Tests.ps1 fails any windows Dockerfile that re-introduces the directory form. Dockerfile.probe is exempt BY DESIGN (PROBE_NONCE busts its layer regardless, and it dispatches arbitrary diagnostic scripts — its own header says so). Corollary: “a module edit is cheap” is a claim about the MOUNTS, not about the module.

The “unreferenced” windows/scripts modules are external-consumer API#

The “unreferenced” windows/scripts modules are EXTERNAL-CONSUMER API — never delete (owner decision 2026-08-04). Flutter/CMake/CodeQL/MSIX/ Slang/Vulkan/PerfBaseline/WasmOpt/AppRunner/ContainerBuild.Reuse/Uv/ Build.Common/WebDav/Toolchain/Config/Formatting/ContainerLog (verified live 2026-08-21: OxidANT’s container scripts import it — Invoke-StevedoreBuild + rust-build/test-all — vendored into BeschleunigerBallett and OmniAccelerANT) plus scripts/rust/ and scripts/python/ are the shared build framework other Kataglyphis repos consume (this repo IS the upstream). One exception, by owner decision 2026-10-05: scripts/rust/Build-Windows.ps1 had no consumer and is gone. OxidANT owns its Windows Rust build, and the script’s one shared piece, Get-ANTfrastructurePin, lives in WindowsScripts.Shared.psm1. Repo-internal reference audits will flag them as dead — they are library surface. Keep them lint-clean; do not rename exported functions without checking external consumers. Their build-cache cost is zero only while every mount stays per-file (§ A whole-directory modules mount puts EVERY module in the cache key). Applied 2026-08-31, do not re-litigate: the driver-cleanup wave kept Get-LlvmMasmCmakeArg (zero in-repo callers) and four WindowsSourceBuild.Common re-exports on this rule alone — the audit that flags them next will be the same audit, not new evidence.

WindowsOnnx.Common and WindowsMediaRuntime.Common stay exported but are chain-only (2026-09-23). AccelerANTgine imports both by name and calls Copy-MediaRuntimeBundle, so neither is deleted; neither can reach a NuGet ONNX Runtime any more:

  • Install-OptionalNuGetPackage refuses any id matching onnxruntime or Microsoft.(Windows.)AI.MachineLearning before nuget runs. After any other install it fails when a package directory in the output holds ORT: as a dependency, bundled under another id, or left by an earlier run.

  • Get-OnnxPackageLayout keeps its parameters and throws, pointing at Get-OnnxChainLayout. That function describes the chain install: ONNX_ROOT\bin\onnxruntime.dll for the target machine, and ONNX_GENAI_ROOT with lib\ (or bin\) onnxruntime-genai.dll and no ORT file of its own. A NuGet tree or a release zip throws on both halves.

  • Copy-MediaRuntimeBundle stages the chain directories last, so the chain’s onnxruntime.dll and DirectML.dll win every name collision, never walks runtimes\<rid>\native, throws on an ORT DLL in any other directory, and then requires every staged ORT or GenAI DLL to carry the chain’s bytes.

Whether the bytes under ONNX_ROOT really are the chain build is the ORT census’s question, not these modules’ (onnxruntime-single-source.md).

Rust: rustup WITH a default toolchain is the sole provider#

Rust: rustup WITH a default toolchain is the sole provider — never a toolchain-less rustup, never a second provider (no scoop rust). Polarity INVERTED by the Flutter-Cargokit fix: Cargokit (flutter_rust_bridge-style plugins) hard-requires rustup and aborts with “rustup not found in PATH.” otherwise, so Install-RustToolchain.ps1 runs rustup-init -y --default-toolchain <RUST_VERSION> --profile minimal (pinned since 2026-10-05) and Install-ScoopTools.ps1 installs NO rust. CARGO_BIN (= ...\.cargo\bin, the rustup proxy dir) sits ahead of scoop’s shims on PATH by design. The failure the old “never rustup” rule guarded against was narrower than the rule: a toolchain-less rustup (--default-toolchain none) drops proxy shims that resolve no toolchain (“no default toolchain configured”); installed WITH a default they resolve correctly. Do not re-add scoop install main/rust alongside — one provider only. Details: docs/windows-builds.md § Rust toolchain.

versions.env is the single source of truth#

versions.env is the single source of truth. Build-Buildkit.ps1 forwards the pins a stage declares as --build-arg (the media branches’ through Get-MediaBranchVersionArg), and anything else a script reads falls back to the versions.env the base baked or a RUN bind-mounts — the driver’s filtered windows/generated/versions.env, so a Linux-only pin bump leaves the chain cached (CON87, windows-builds.md § Import-Versions.ps1); the smoke test and scripts derive expected values from it (e.g. CMake URL from CMAKE_VERSION; LLVM_RELEASE pins the LINUX clang, LLVM_WINDOWS_VERSION the Windows one — separate on purpose). Don’t hardcode versions in scripts or Dockerfiles. Anything that produces or shapes compiled output belongs here; tools the build merely invokes may float, and Install-ScoopTools.ps1 splits its installs into exactly those two blocks.

CMake cross configures must carry the ASM language target too#

CMake cross configures must carry the ASM language target too (2026-08-24). Get-CMakeCrossArgs (WindowsTargetArch.Common.psm1) sets CMAKE_ASM_COMPILER_TARGET/CMAKE_ASM_FLAGS_INIT alongside the C/CXX pair, and it OWNS them — never hand a project ad-hoc ASM cross flags. Before it did, any project enabling the ASM language assembled with clang’s X64 default target — signature: brackets expression not supported on this target on aarch64 .S sources — and an aarch64 -march handed to that x86-targeted driver was misread as a CPU name (unknown target CPU 'armv8.2-a+fp16'), which sent the first diagnosis chasing a nonexistent driver gap. amd64 is untouched: the cross args stay empty there.

vcpkg ships zlib ONLY#

vcpkg ships zlib ONLY (protobuf removed 2026-08-03). Nothing consumed vcpkg protobuf — every source build brings its own (ONNX _deps, LiteRT-LM’s protobuf_external + downloaded version-matched protoc), and LiteRT-LM even had to hide vcpkg’s protobuf headers to avoid version skew. Don’t re-add it “for convenience”; it costs ~15 min of base build and creates header-leak hazards.

Python bindings plumbing is load-bearing#

Python bindings plumbing is load-bearing (added 2026-07-13; full detail in docs/windows-builds.md § What is verified: native vs. Python). (1) The sitecustomize.py shim written by Initialize-PythonPlatformTag fixes clang-built CPython’s win32 platform misreport (pip pulls 32-bit wheels without it) AND registers native DLL dirs (os.add_dll_directory; python ignores PATH for pyd deps) — never remove it. (2) OpenCV must keep WITH_MSMF=OFF and WITH_OBSENSOR=OFF: both hard-import Media Foundation, absent on Server Core — either ON makes videoio and the cv2 pyd unloadable. (3) Always @()-wrap Save-PythonWheel results: PS unwraps a 1-element array so [0] becomes the first character and pip once installed the PyPI package literally named c. (4) Binding asserts go through Test-PythonImport (cmd.exe-shielded): tvm writes warnings to stderr on successful imports, which raw & under EAP=Stop turns into false failures. (5) Wheels live at C:\runtime\wheels (PYTHON_WHEELS); the OrchestrANT torch step resolves the app’s LATEST tag per build and its wheel-smoke suite gates the final docker build — both amd64-lane facts. On arm64 (#120, 2026-08-24) the target CPython ships at C:\runtime\python (source-built, PCbuild\build.bat -e -p ARM64, ClangCL props, PE 0xAA64 verified in-stage) and the wheel store holds the onnxruntime/onnxruntime_genai_directml/av wheels tagged cp314-win_arm64 plus cv2.cp314-win_arm64.pyd in the target site-packages — staged, never installed or imported (nothing here can run aarch64). Load-bearing rules on that lane: the HOST interpreter runs every build and the TARGET import lib is linked (Get-TargetBuildPython — .Exe host, .Lib target; never conflate them); wheels go through Invoke-PythonWheelBuild -CrossStage (one call for both lanes: on cross it appends the target --plat-name, stages, and runs Assert-WheelTargetArch — PE machine + EXT_SUFFIX name tag of every member; natively it installs + import-asserts); the shim pins EXT_SUFFIX to the target while get_platform() stays host — two different facts (what this interpreter builds vs what pip may download); ORT/GenAI take Python_* hints (GenAI additionally the legacy PYTHON_* trio for vendored pybind11) — Python3_* is silently ignored, see SourceBuild.FindPythonPrefix.Tests.ps1. The GenAI wheel’s ORT requirement is rewritten to plain onnxruntime before packing (both lanes, #126): upstream’s setup.py.in derives onnxruntime-directml/-gpu from the package name, but our combined ORT wheel is named onnxruntime — unpatched, the edge is unsatisfiable on arm64 (no win_arm64 onnxruntime-directml exists) and pulls a second ORT over ours on amd64. The cross merge then stages the target’s runtime deps (Copy-TargetPythonDeps.ps1: host pip download --platform win_arm64 --python-version 3.14 for what the bundle does not provide; gate: every wheel pure or win_arm64, every Requires-Dist resolves inside C:\runtime\wheels) and the arch gate walks every PE’s import table (Test-TargetArch.ps1 -ImportWalk, #127: each import must resolve to a bundle DLL, an api-ms-/System32 name — CRT DLLs excluded from System32 on cross — the driver allowlist, or the client-OS list -ClientOsPattern for DLLs a Windows client SKU ships but Server Core does not: dsound, mf*, winspool.drv). Its first run (arm64 run 13) found 13 real unresolved imports — the OpenSSL runtime DLLs nobody installed, the client-OS set, and vcruntime140_threads.dll — so treat a green walk as the gate it is, not decoration. It is a hard gate on cross lanes only: on the native lane the deliverable is the image, whose PATH carries the host CPython, scoop’s OpenSSL and the toolkits, so the same walk reports (amd64: 186× python314.dll + 8× python3.dll from the host interpreter outside the roots, 6× libcrypto/libssl-4-x64) and never throws. (6) PyAV is built from sdist against OUR FFmpeg (setup.py --ffmpeg-dir) — PyPI’s av wheel is unloadable on Server Core (bundled avdevice imports desktop-only AVICAP32.dll); in headless code request software encoders by name (mpeg4) — the generic h264 alias resolves to the hardware h264_d3d12va. (7) IREE (media-tvm branch, TVM→IREE chain) is a shallow-submodule git clone (release tarballs lack LLVM); its wheels come from the ninja build tree’s synthesized compiler/+runtime pip dirs with --no-build-isolation — plain pip wheel of the repo would rebuild all of LLVM in an isolated tree. Native tools at IREE_ROOT (C:\runtime\iree); CUDA HAL/target need no nvcc (PTX via NVPTX, driver dlopens nvcuda.dll).

Contents