Common failure modes — symptom lookup

Contents

Common failure modes — symptom lookup#

Start from the error message. Every entry below is keyed by what you actually see, then names the cause and the fix. All of them were hit live in this repo; the dates are when.

Two neighbours, so you land on the right page:

Several rows here point at deeper write-ups in windows-builds.md and its sibling pages. This page is the triage index: enough to recognise the failure and act, with a link when the full history matters.

Contents#

Linux and cross-lane

Windows: the layer store (hcsshim)

Windows: container networking

Windows: buildkitd and the store

Windows: Stevedore and the docker service

Windows: build content and toolchain

The shortlist AGENTS.md carried#

Moved out of AGENTS.md on 2026-09-15 (owner decision D10), unedited except for this heading, the relative links and a failure count that had gone stale. The RULES stayed there; this is the reference behind them.

Symptom → cause → fix for every failure seen live on both lanes, keyed by the error message you actually get: docs/failure-modes.md. Grouped as Linux/cross-lane · the Windows layer store (hcsshim) · container networking (CNI) · buildkitd and the store · Stevedore and the docker service · build content and toolchain.

Four reflexes that page encodes — worth holding before you need them:

  1. Check free disk FIRST on any weird hcsshim failure. Disk exhaustion wears three different costumes and only one of them names the disease.

  2. Prefer letting a doomed solve fail cleanly over killing it. A clean finalize failure leaves no debris; a buildctl kill mid-finalize manufactures the deterministic 0xb7 that then costs a -NoCache re-run.

  3. After ANY red finalize, REBOOT before further A/B tests. A wedged hcs state falsifies every experiment run after it.

  4. Identical step timings mean a TIMEOUT, not slow work. When every RUN lands on the same number, decode that number against the shim teardown knob before debugging the workload (the 2026-08-31/09-01 incident: 2841.2 s byte-identical = lost exit notification × 45 min constant; a 240 s probe kill then misread it as a hard wedge for a day). Timing table + re-mitigate command: docs/failure-modes.md § “Every RUN step reports DONE 2841.2s”.

Linux and cross-lane#

exec format error on a foreign-arch build#

Symptom. exec format error

Cause. QEMU/binfmt not registered after host reboot

Fix. linux/scripts/setup-rootless-binfmt.sh — NOT the tonistiigi container: that installs into the wrong namespace under rootless nerdctl (see Host prerequisite: QEMU/binfmt in docs/linux-cross-builds.md; this row prescribed exactly that dead-end until 2026-08-24)

no space left on device#

Symptom. no space left on device

Cause. Disk full from cached images/artifacts

Fix. In order: (1) linux/host-config/prune-safe.sh (spares compile caches), (2) nerdctl rmi of specific already-pushed tags, (3) nerdctl system prune -a -f ONLY with no chain running — it deletes ALL non-container-referenced images INCLUDING tagged cross-stage locals (bit us mid-run 2026-08-18; registry-pinned handoffs survived via re-pull)

Stale downstream images#

Symptom. Stale downstream images

Cause. Base image rebuilt but downstream not refreshed

Fix. Use --verify-chain or rebuild from replaced stage

no active session / grpc: the client connection is closing mid-chain#

Symptom. no active session / grpc: the client connection is closing / DeadlineExceeded mid-chain (cache reads, layer export, pushes)

Cause. buildkitd session rot after ~1-2 h of parallel load (BKD1, bit 6× on 2026-08-19/20)

Fix. Let the worker retries absorb one-offs; on a repeat: stop chain → systemctl --user restart buildkit.service → relaunch (cache mounts provably survive; builds fast-forward)

registry_pin_ref fails on a fresh push#

Symptom. registry_pin_ref fails on fresh push

Cause. Registry hasn’t propagated the new manifest

Fix. Now uses retry() with 5 attempts; wait a few seconds and retry

Terminal freeze during a long build#

Symptom. Terminal freeze during long build

Cause. Build output overwhelms terminal

Fix. Use setsid / disown for very long builds

LiteRT configure: fatal: expected flush after ref listing#

Symptom. Every LiteRT/TFLite cmake configure fails cloning eigen; all three android lanes down inside one window (2026-08-21)

Cause. tools/cmake/modules/eigen.cmake declares a single GIT_REPOSITORY (gitlab.com/libeigen/eigen), so a momentary gitlab outage is a total outage

Fix. Already handled by linux/scripts/03-media/build/litert/android/litert-eigen-fetch.sh, sourced by both LiteRT build scripts. It sets two knobs of upstream’s OverridableFetchContent and pins no commit of its own (the tag stays upstream’s):

  • GIT_REPOSITORY_AND_TAG_TO_URL_eigen=ON turns the clone into an archive download of the same pinned commit — how the C-API/wheel configure paths already fetch it;

  • <content>_MATCH/_REPLACE rewrite that archive URL. A ; in the replacement makes it a cmake list, which ExternalProject downloads “in turn until one succeeds”.

The second URL is TensorFlow’s own mirror of the same gitlab path (the tf_mirror_urls() rule in third_party/eigen3/workspace.bzl), verified byte-identical on 2026-08-23: both URLs for commit ea13a98d… returned sha256 35c6126e…, 2870994 bytes. It degrades safely — if the regex ever stops matching, the URL is left untouched and the fetch behaves as it does today.

logging.sh: line NNN: action: unbound variable instead of the real error#

Symptom. A build stage dies and the only diagnostic in the log is .../01-core/logging.sh: line 119: action: unbound variable. The command that actually failed is nowhere. Hit on 2026-08-30, where it stood in for a parallel-GCC apt-lock failure.

Cause. _install_trap defined on_err as a NESTED function that read its reporting action (err or warn) out of the installer’s own local action — i.e. through dynamic scope, which only resolves while _install_trap is still on the call stack. An ERR trap fires long after the installer has returned, so under set -u the handler aborted on its first line. Two things were lost, not one: that abort replaced the real error text, and the action it existed to run never ran — install_err_trap never exited 1 and install_warn_trap never printed. Every script that sources logging.sh inherited it.

Fix. Fixed 2026-08-31. on_err is a top-level function, and _install_trap bakes the resolved action into the trap string with printf -v '%q' so it is expanded AT INSTALL TIME, while LINENO and BASH_COMMAND stay escaped and expand AT FIRE TIME (reverse that escaping and every failure is reported at the install site’s line number, forever). _LOG_TRAP_ACTION keeps the two-argument call working for build-gcc.sh, which tears the trap down and re-arms the bare string by hand around its configure step. Regression cover: linux/scripts/tests/test-logging-err-trap.sh. Behaviour note: inside a set +e window install_err_trap now exits 1 where it used to print noise and carry on. No caller has such a window today; a new one would feel it.

A fault-injection test passes and proves nothing (grep is ugrep)#

Symptom. A regression suite is green — and stays green when you break the very thing it guards. Its fault-injection arm (a stub that fails when its argv matches a pattern) never fires, and nothing reports that: a pattern matching nothing looks exactly like a run with no fault injected.

Cause. The pattern began with -- and was handed to grep positionally. grep on this host is ugrep, which parses a leading--- pattern as a command-line option instead of as the pattern. Found 2026-08-31 in linux/scripts/tests/test-iree-wheelhouse-stages.sh, whose header claimed five || return 1 call sites were covered when two were.

Fix. Pass the pattern behind -e: grep -qE -e "$pat". The general rule — a regression test has to be SHOWN to go red before it counts, and its header must say what it does not cover — is in cross-build-verification.md, along with the second half of this same incident: with injection finally working, the obvious rc == 1 assertion still passed on every mutation.

GENAI-BIND SKIP reported green while the native binding is broken#

Symptom. The runtime-image smoke prints GENAI-BIND SKIP: onnxruntime_genai is not installed and the run stays green — on an image where that wheel IS installed and only its native library refuses to load.

Cause. The smoke collapsed two unrelated situations into one exit code (3 = skip): “this arch ships no genai wheel”, which is benign and is the ARCH-PARITY table’s business, and “the distribution is installed but importing it raises”, which is a defect. Nothing else in the chain can see the second one: smoke-torch-venv.sh’s installed_version() falls back to importlib.metadata when the import raises, so it cheerfully reports the pinned version of a package that cannot be imported, and the ARCH-PARITY table reads only dist-info directory names. A riscv64 binding with an unresolved __atomic_* or a missing NEEDED would have shipped behind three green gates. The lane this was found on is written up in gen1-riscv64-genai.md.

Fix. Fixed 2026-08-31 in smoke_genai_py (06-packaging/smoke-common.sh): on an import failure the program asks importlib.metadata whether the DISTRIBUTION is present. Present → exit 1 with GENAI-BIND FAIL: … is INSTALLED but not importable, pointing at the .so’s NEEDED/undefined symbols; absent → the SKIP. Transferable form: an exit code meaning “there was nothing to test” must not be reachable from a state meaning “the thing under test is broken”.

The documented GENAI_ALLOW_RISCV64 back-out does not reach the smoke#

Symptom. A lane toggle is set in versions.env, the build honours it, and the runtime-image smoke still reddens by asserting the component the toggle switched OFF. Exporting the variable in the host shell before running the smoke changes nothing.

Cause. Two boundaries, and the value crossed neither of them. As far as the host shell is concerned a nerdctl run begins with an empty environment, so exporting the toggle before the smoke is invisible inside the container. And the ARG/ENV that carries it is declared on a BUILDER stage, while the final stage of Dockerfile.media descends from a different parent — so the shipped image holds no copy of it either. The baked smoke-torch-venv.sh therefore read the toggle as unset every time and took its default arm: the escape hatch the docs promised had never once worked. Depth on this particular toggle: gen1-riscv64-genai.md.

Fix. Fixed 2026-08-31: smoke-runtime-image.sh reads the value host-side out of versions.env (_rt_versions_env_pin, empty on a miss = “not asserted”) and forwards it with an explicit -e on the nerdctl run, so the smoke asserts the policy the build was configured with. A new toggle has to be checked on both hops — build ARG into the SHIPPED stage, host env into the container — and a verifier should read a marker the producer actually wrote rather than re-derive the decision: a producer defaulting :-false against a verifier defaulting :-true disagrees in the failing direction.

An unresolved NEEDED in a library that nothing scans#

Symptom. Nothing at all, which is the point. A shipped .so carries a dependency that cannot be resolved, every gate is green, and the failure surfaces as an import or dlopen error on the target.

Cause. 03-media/runtime/validate-media-runtime.sh hunts unresolved NEEDED entries over ARTIFACTS (the gst / libcamera / ffmpeg binaries) plus the GStreamer plugin directory. Its LIB_DIRS sweep is a DIFFERENT check — ELF machine only, and advisory. An install prefix in neither list is scanned by nobody; /usr/local/lib/onnxruntime-genai/lib was in neither until 2026-08-31. The concrete thing that hid behind it: GenAI’s CMake, unlike upstream onnxruntime’s, carries no libatomic probe, so a riscv64 build can need -latomic and only say so at load time. That lane: gen1-riscv64-genai.md.

Fix. Fixed 2026-08-31 — the prefix joined LIB_DIRS (resolution root plus the advisory arch sweep) AND its lib/ is now walked by scan_plugin_directory (misnamed but generic, so it was reused rather than copied), feeding the existing missing-soname machinery. A [ -d ] guard makes it a no-op when the lane is off. The check to carry away: when an image gains an install prefix, ask which scanner covers it — and remember that an ELF-machine sweep is not a dependency scan.

A prune step deletes the wheel a later step requires#

Symptom. Again nothing yet — the step that would have broken was disarmed by an unrelated accident. install_project_environment prunes conflicting wheels before build_uv_sync_args goes looking for the local ones, and one prune glob matched a wheel the later step requires.

Cause. 03-media/runtime/assemble-torch-app.sh’s prune_conflicting_onnx_wheels carried a *genai*.whl glob on its DEFAULT ONNX_PACKAGE=onnxruntime arm. Far too broad: it also names the plain CPU genai wheel this media lane now produces for every architecture, which the uv-sync step and the ARCH-PARITY table both rely on being present. Had the deletion succeeded, the earlier GENAI-DRIFT bug would have returned — absent a local wheel, the resolver quietly prefers the app lock’s PyPI build. It survived only because /opt/wheels happens to be mounted read-only, so the removal errored and || true discarded that error; flipping the mount to writable would have broken all three architectures at once. Full narrative: gen1-riscv64-genai.md.

Fix. Fixed 2026-08-31 by narrowing the glob to the GPU-flavoured variants the arm actually means (*genai_cuda-*, *genai_rocm-*, *genai_directml-*), beside its *_gpu-* / *_migraphx-* neighbours. The lesson is about || true on a destructive step: it hides the failure AND the fact that the step was wrong, so the bug sits latent until something unrelated arms it — here, a read-only mount becoming writable.

The disk guard aims at the wrong number#

Symptom. The chain prunes at 40G free between stages, reports success, and the NEXT stage refuses anyway: [ERROR] runtime lane refused: 56G free, ~120G needed (2026-09-02: six manual prunes; 2026-09-03: the Flutter ship build, attempt 1).

Cause. Two different numbers. CROSS_DISK_GUARD_GB (40) is a floor for this stage; the runtime lane needs CROSS_RUNTIME_LANE_GB (120) per wrapper build, and --only runtime additionally pulls the three cross-android-<arch> images (~40G uncompressed each) as its artifact source. A guard that reclaims to a fixed floor is therefore satisfied exactly when the lane is not.

Fix. _chain_stage_disk_guard in build-cross-chain.sh reclaims to what comes next (_chain_runtime_lane_need_gb), and the launch-time preflight warns when the run will enter the lane with less than stage cost + lane need.

When the guard reclaims nothing. buildctl du showing every regular record Reclaimable: false is not a full cache — it is killed chains’ leaked leases (2026-09-03: 351 records / 251 GB pinned, prune-safe.sh freed 0). Leases die with the daemon: systemctl --user restart buildkit.service (no build running), then PRUNE_KEEP_GB=<N> linux/host-config/prune-safe.sh. --keep-storage bounds the WHOLE store, and the non-candidates (cache mounts ~166G + source.local ~10G) count toward it; since CON53 (2026-10-01) the script and the chain’s fallback add them, so <N> is GB of layer cache, not of the store. Local :latest-<arch> images are re-pullable and not build inputs; nerdctl rmi them last. Restarting the daemon is the BKD1 remedy above wearing a disk costume.

A renamed or dropped distro package kills a stage hours in#

Symptom. A media or toolchain stage dies four hours into a rebuild with E: Unable to locate package <name> — or, worse, does not die: an install_target_packages ... || true swallows it and the feature silently vanishes from the shipped image.

Cause. Ubuntu renames and drops binary packages between releases, and this tree pins a rolling one (UBUNTU_CODENAME in versions.env). 26.04 renamed libfreetype6-dev to libfreetype-dev, replaced libopenexr-3-dev with libopenexr-dev, and libvvdec-dev never existed on ports at all. Nothing noticed for months because warm apt caches still answered for the old names — the tree was un-buildable from scratch and no one knew, because nothing ever built from scratch.

Fix. linux/scripts/verify_package_names.py, wired into preflight.sh as the pkg-names check. It extracts every distro package name linux/scripts/** asks for and resolves each against the live Ubuntu indices for the pinned codename — archive.ubuntu.com for amd64, ports.ubuntu.com for arm64/riscv64 — before a build starts rather than four hours in.

What it reads, and how a verdict is reached:

  • Sources: install_target_packages / install_optional_target_packages / install_host_packages / install_deps_preamble / apt_install call sites, bare apt-get install lines, *_packages and *_pkgs array literals, append_unique_packages / append_available_packages in 01-core/package-lists.sh, the _CPYTHON_EXT_DEV_PKG_TABLE rows in 01-core/cpython-dev-packages.sh, and the soname map at 03-media/runtime/so-package-map.txt. Names occur one-per-line AND several-per-line; a per-line assumption is how an earlier audit reported the wrong count, so --list prints exactly what was extracted.

  • UNGUARDED requests FAIL the gate: a missing name aborts the whole apt transaction and kills the stage. GUARDED ones (|| true, a || fallback chain, a self-filtering helper, an enclosing apt probe) only WARN — the cost there is one wasted apt round-trip per run, not a dead build.

  • A name apt can still install through Provides: counts as present but is reported: a virtual name disappears the next time the provider is renamed.

  • Names from a non-Ubuntu repo (NVIDIA/ROCm/TensorRT) are UNVERIFIABLE, never dead; that file list lives in the script.

  • An array no installer ever consumes is reported as unchecked, not silently dropped — that is what keeps sdkmanager component lists out of the verdict.

  • Not covered: package names written directly in a Dockerfile RUN (Dockerfile.toolchain’s binutils-dev is the only one today), and anything a script assembles at runtime rather than spelling out.

Offline is a SKIP, never a pass. Indices are cached per codename with a 6h TTL (PKG_NAMES_TTL, PKG_NAMES_CACHE_DIR): ~13s cold, ~1.5s warm. With no network and no cache the check says so loudly and exits 0; with a stale cache it verifies anyway and says the cache was stale.

The transferable lesson. The gate carries its own extraction self-check — known package names that MUST be found (a several-per-line array row, a backslash-continued call, a package-lists.sh helper) and decoys that must NOT be (an sdkmanager component, a table column, a word from an echo string). Break the extractor and the check exits 2 before it ever reports green. A gate whose input extraction can silently degrade to nothing is the failure this repo keeps re-learning; a scanner has to prove it is still scanning.

apt: libc6:i386 install is unsatisfiable after an archive/ports drift#

Symptom. The android stage dies a few seconds into the SDK install, on every arch, with:

E: Unable to satisfy dependencies. Reached two conflicting assignments:
   1. libgcc-s1:arm64 is selected for install
   2. libgcc-s1:arm64 Depends libc6:arm64 (>= 2.35)
      but none of the choices are installable:
      - libc6:arm64 is selected for removal because:
        1. libc6:i386=2.43-2ubuntu2.4 is selected for install
        2. libc6:arm64 Breaks libc6:i386 (!= 2.43-2ubuntu2.3)

The arch named varies by lane: arm64 on the riscv64 image, riscv64 on the arm64 one, both on amd64.

Cause. Two facts combine. The compiler base installs libc6 for BOTH foreign arches, but Dockerfile.media’s apt reset wipes all sources and writes only the build host’s and the current target’s. When archive.ubuntu.com publishes a libc6 update before ports.ubuntu.com (measured 2026-09-11: i386 2.4 on archive, arm64/riscv64 still 2.3 on ports), the freshly installed libc6:i386 Breaks the frozen foreign copies — and apt cannot upgrade an arch it has no source for. The mirrors were back in sync minutes later; the failure mode is the drift, not the package.

Fix. cross_ensure_installed_foreign_arch_sources (01-core/cross-apt.sh), called from android-sdk.sh before apt-get update: it writes a per-arch ports source for every installed foreign arch that has none, so apt moves all libc6 arches to the new version together. Guarded by test-cross-apt.sh, including a whole-line wiring check — the first mutation run showed a substring check over the file still passes with the call deleted, because the helper’s name appears in a comment.

Retry after a drift. Nothing in the image needs rebuilding. Wait for the mirror to catch up and re-run the stage (--from-stage android, ancestry checked) — the failed run left no partial tags (0/7 cache stages exported).

A from-source CPython silently drops an extension module#

Symptom. import ssl / import sqlite3 / import lzma raises ModuleNotFoundError inside a shipped image, or an interactive python3 has no line editing. The toolchain stage was green: make -k || true skips an extension whose dev header was missing at configure time and says so only in passing.

The 2026-08-09 incident was libsqlite3-dev: the host closure got sqlite transitively through GUI dev packages, the cross-target install list forgot it, and half the Python ecosystem imports sqlite3 on the way to something else.

One table, two consumers. 01-core/cpython-dev-packages.sh holds every row as <dev-package> <required|optional> <ext-module>..., and nothing else may keep a second list:

consumer

reads

verdict

package-lists.sh base_image_os_packages

cpython_ext_dev_packages

the HOST closure installs the same set

build_python.sh _python_cross_stage_target_dev_pkgs

cpython_ext_dev_packages + cpython_ext_dev_packages_required

one atomic apt install, then a per-package dpkg-query; a missing REQUIRED package is fatal

build_python.sh _python_dynload_audit

cpython_ext_modules

audits each staged lib-dynload; a missing .so warns, on every row

The class column therefore governs the package, not the .so. The audit is warn-only on purpose: promoting the required rows to fatal there flips all three arches at once and only a cross rebuild can price that, so the decision stays open rather than being smuggled in with a refactor. The free-threaded trees are the exception: they ship as the image’s runtime interpreter, so a missing _ssl, _hashlib, _sqlite3, zlib, _bz2, _lzma or _ctypes is fatal there (consumer-image-contract.md § The free-threaded Python). Until 2026-09-05 the audit carried its own hand-written array instead — seven modules that had never gained readline after LOG23 added it to the table, which is the same desync in miniature.

The parsing trap. build_python.sh runs under IFS=$'\n\t', where an unpinned read does not split on spaces: a row naming two modules (libssl-dev required _ssl _hashlib) would arrive as one word and the audit would look for an extension called _ssl _hashlib. Every accessor in the table file pins IFS=' ' on its read for exactly that reason, and tests/test-cpython-ext-table.sh runs cpython_ext_modules and cpython_ext_dev_packages under both values of IFS.

What only a real build shows. Whether the five modules the audit gained on 2026-09-05 (_zstd, readline, _curses, _uuid, _decimal) actually land in lib-dynload on arm64 and riscv64. Four of them are optional rows and readline is required, which makes its package fatal at the install step, not its .so. The audit warns on every row, so a warning there is information, not a failure, and the toolchain smoke’s own stdlib battery is what turns a genuinely broken interpreter red.

The delete guard denies its own legitimate work#

.claude/hooks/guard-destructive-deletes.py matched a delete VERB and a PROTECTED PATH independently, anywhere in the command. So any command that both deleted something harmless and merely mentioned a system path was denied:

rm -rf scratch && cc -o x /opt/gcc-16.2.0/bin/gcc     # denied: "a system directory"

The /opt here is a compiler, not a delete target. This fired five times on 2026-09-02 against a scratch directory under the job tmpdir, each time costing a tool call and a rewrite. Two earlier variants of the same shape are recorded in the file’s own header: --rm matching \brm\b, and sed 's/^/  /' reading as the filesystem root.

The fix. Split the command on ;, &&, ||, | and newlines, never inside a quoted span (_split_outside_quotes), and run the protected-path patterns only on the segments that actually carry a delete verb. A preceding cd target is carried into later segments, so cd /usr && rm -rf * still denies — the relative delete cannot escape the directory it was given.

The transferable lesson. A guard whose two halves are matched independently across a whole command is not checking a relationship, it is checking co-occurrence. Co-occurrence guards look strict and behave randomly: they deny safe work and, worse, teach the operator to phrase commands to avoid the guard rather than to be safe. Bind the dangerous verb to its own argument. Both directions are pinned in test-delete-guard.sh — widening the scope back to the whole command turns the allow-cases red, and dropping the cd tracking turns the deny-cases red; the recorded mutation, delete-guard.quote-aware-split, pins the quote-aware split.

OpenCV: std::complex breaks on a shadowed complex.h#

The riscv64 opencv-gst stage dies compiling modules/core/src/hal_internal.cpp:

error: expected unqualified-id before '_Complex' [-Wtemplate-body]
  540 |     int ldsrc1 = (int)(src1_step / sizeof(std::complex<fptype>));

The caret sits on complex inside std::complex, because complex is a macro by then. hal_internal.cpp includes the C header at line 50, under HAVE_LAPACK, and precomp.hpp has already pulled in <complex> — so the class is declared correctly and only later uses of the token expand.

In C++, <complex.h> is supposed to reach libstdc++’s wrapper, which includes glibc’s header and then removes the damage:

# include_next <complex.h>
# ifdef _GLIBCXX_COMPLEX
#  undef complex          // <- the guarantee
# endif

glibc’s own /usr/include/complex.h has #define complex _Complex with no __cplusplus guard, so whoever wins the lookup decides whether C++ still works. CMake passes OpenCV -isystem /usr/include, and -isystem is searched before the compiler’s C++ directories — so glibc’s header wins and the #undef never happens. Measured with the riscv64 cross g++:

flags

<complex.h> resolves to

complex macro

none

…/riscv64-linux-gnu/include/c++/16.2.0/complex.h

not defined

--sysroot=/ -isystem /usr/include

/usr/include/complex.h

#define complex _Complex

Reproduced in three lines — #include <complex>, #include <complex.h>, then one std::complex<float> — and it fails on every target compiler once the flag is present, so this is not architecture-specific in nature.

The fix. build-opencv.sh writes a two-branch complex.h shim next to the source tree and prepends it with -I. GCC searches every -I before every -isystem, so the shim wins regardless of what CMake appends — the same precedence trick the file already uses to pin our FFmpeg prefix. The shim restores libstdc++’s behaviour in C++ and is a transparent pass-through in C, which matters because the same flag string is handed to both CMAKE_C_FLAGS and CMAKE_CXX_FLAGS (OpenCV compiles C third-party code such as openjp2 that legitimately wants the complex macro).

Why it surfaced only now. It is not a regression. The previous run never compiled this file: libcamera failed earlier on an unrelated link error, and BuildKit builds those stages in parallel, so the whole lane died first. Fixing libcamera let the build get far enough to reach a defect that was always there.

Still open: which CMake package contributes -isystem /usr/include on riscv64 and not on the other lanes. ZLIB_INCLUDE_DIR=/usr/include is passed on every cross arch and all three resolve zlib identically, so that alone does not explain it. The shim makes the build immune either way; the injector is worth finding so the flag can be removed at the source.

A smoke that never passed and always excused itself#

validate-media-runtime.sh carried a “QEMU Cross-Arch Binary Smoke” that ran each built binary under qemu-user and, on failure, printed:

WARN: /usr/bin/qemu-riscv64 gst-launch-1.0 --help failed
      (may be expected for cross builds without full sysroot)

Three things were wrong with it at once, and each alone would have been enough:

  • Both branches were echos. No counter, no variable, no exit code — there was no path from that loop to a failure. The ELF block directly above it does it properly: it counts core_mismatches and exit 1s.

  • The excuse covered every case it could ever see. The block only ran under cross_build_is_active, so “may be expected for cross builds” applied to 100% of its invocations. That is not a diagnosis, it is a blanket.

  • The failure was mechanically guaranteed. qemu-user needs the target’s ld.so and libraries via -L or QEMU_LD_PREFIX; neither was passed, and QEMU_LD_PREFIX appears nowhere in the repo. Every binary died in the dynamic loader before main — exactly the “full sysroot” the message names, which makes it a fixable bug, not an expected condition.

Final score before removal: 7 failures, 0 passes, across arm64 and riscv64 in one run.

Removed rather than repaired, 2026-09-02. Repairing it needs a multi-hour media build to prove the sysroot prefix actually works, and shipping an unproven-but-now-fatal gate is how you lose the next run. Functional coverage already exists where it belongs: the runtime-image smoke boots the real per-arch image and gates the manifest on it — that is what caught the riscv64 venv gap the same day.

The transferable lesson. A check whose failure branch is always excused teaches its readers that WARN lines are noise, which is worse than having no check: it costs attention on every run and buys nothing. When you find one, the question is not “how do I make this pass” but “what would it take to make this able to fail” — and if you cannot answer that today, delete it and say where the real coverage lives.

A trailing conditional fails the whole script#

The 2026-09-03 runtime build died on amd64 in setup-torch-venv.sh with exit code: 1 and no error text — the last lines were a healthy uv pip install --no-deps ml_dtypes / Checked 1 package in 2ms. The cause was one line at the END of reconcile_local_wheels in assemble-torch-app.sh:

  [ "${have_torch_family}" = "true" ] && _backfill_torch_runtime_deps
}

A function returns the status of its last statement. With no local torch wheel (every amd64/arm64 build — torch comes from uv sync there) the test is false, the && list is 1, so the function returns 1; the caller runs under set -euo pipefail and exits. Nothing printed anything, because nothing had failed in the ordinary sense. riscv64, the only arch WITH a local torch wheel, was the only arch that would have passed.

Two things let it through:

  • set -e is not tripped by the list itself. a && b is exempt from -e, which is why the idiom feels safe; the trap is only that its status becomes the function’s status, and the bare call reconcile_local_wheels in the caller is NOT exempt.

  • The characterisation test ran without set -e and never looked at $?. It pinned the sequence of uv calls the function makes (its stated purpose) and so proved the refactor behaviour-preserving for every path — including the path that now returned 1, because a return status is not a uv call.

Fix: an if ... fi (or || true when the right-hand side is genuinely optional). The test harness now runs the function under set -eu and asserts 0 for a non-torch wheel set; against the broken revision that case fails (expected '0', got '1').

Where else this hides. A census of cond && cmd as the last statement before a closing brace finds 17 sites in the tree; almost all are deliberate predicates ([ -n "${_t}" ] && [ "${_t}" != "${_b}" ]) whose callers use them in if/||. The bug shape is narrower: an action on the right-hand side and a bare call site under set -e. A regex on the function alone cannot tell the two apart, so the gate that landed for it, preflight slug trailing-conditional (linux/scripts/verify_trailing_conditional.py), flags every function that ends on a conditional and freezes the reviewed predicates in linux/scripts/trailing-conditional.allow: code-quality-tooling.md § Trailing-conditional returns. It cannot see call sites, so a function-level test must still assert the exit status on the “nothing to do” path, not only the calls it makes.

The copied Rust toolchain is the builder’s arch#

Symptom. The arm64 package stage dies with exit code: 127 and no command not found line — the last output is Keeping existing /usr/local/cargo/bin/rustup (rustup toolchain wins over PATH lookup) (2026-09-03). In the shipped :latest-cross-arm64 of 2026-09-01:

$ rustc --version
bash: /usr/local/cargo/bin/rustc: cannot execute: required file not found
$ ls /usr/local/rustup/toolchains
1.98.0-x86_64-unknown-linux-gnu  nightly-2026-06-28-x86_64-unknown-linux-gnu
$ du -sh /usr/local/rustup      # 2.0G

Cause. Dockerfile.package COPYs /usr/local/{rustup,cargo} from artifact-source, and artifact-source is the amd64 host image of the cross chain (--platform=${ARTIFACT_PLATFORM}). Everything else it copies is a target tree (a cross-built /opt/gcc-<v>, /opt/llvm-target, /opt/opencv5), but the Rust toolchain there is the builder’s own x86_64 install with aarch64/riscv64gc targets added — that is what a cross-compiler needs and exactly what a native arm64 image cannot run. Every arm64/riscv64 runtime image shipped 2 GB of dead x86_64 binaries at the front of PATH (/usr/local/cargo/bin precedes /usr/bin), so cargo was broken rather than falling back to Ubuntu’s.

Why nothing caught it. Three layers, each blind in its own way:

  • wire_cargo_symlinks checked -x (true for a foreign ELF) and kept the shims.

  • report_rust_provenance’s hard gate reads RUST_VERSION, which did not reach the script until 7eca29c6 (2026-09-03) — the day it started running was the day it failed. Before that: NOTE: RUST_VERSION unset; cannot verify.

  • The runtime smoke’s ADV/HAVE table (_advert_verdicts) turned an unreadable actual value into SKIP, so rustc --version failing under 2>/dev/null was reported as “could not read”, not as a mismatch. That arm is fatal since 2026-09-04 (UNREAD); see docs/cross-build-verification.md, gate A.

Why 127 with no message. _got="$("${CARGO_HOME}/bin/rustc" --version 2>/dev/null | awk …)": the loader’s cannot execute goes to /dev/null, pipefail propagates rustc’s 127 through awk, and set -e exits on the assignment. The pattern is fine for a probe whose result is tested; it is not fine when the probe’s failure is itself the fault.

Fix. setup-package-image.sh ensure_native_rust_toolchain (before wire_cargo_symlinks): if ${RUSTUP_HOME}/toolchains/ holds no *-<own triple> dir, wipe ${RUSTUP_HOME} and everything in ${CARGO_HOME} but its registry cache mount, and run the same install-rust.sh the toolchain stage uses, with BUILD_MODE=native and RUST_INSTALL_CARGO_C=0 (compiling cargo-c under QEMU would cost an hour; the cargo-c apt package is what wire_cargo_symlinks links in that case — one minor version behind the pin, an accepted skew). amd64 is a no-op: the copied toolchain is native there. The native install downloads the pinned stable + clippy + rustfmt + wasm32-unknown-unknown and the pinned nightly, same set as amd64, so the three arches ship the same toolchain surface. Since 2026-10-13 prune-host-payload.sh empties the foreign toolchain in artifact-source, before the COPY, so the 745 MB x86_64 layer no longer ships only to be deleted; this function then installs over an empty tree (why).

Guards. check_rust_toolchain in smoke-runtime-image.sh executes rustc --version in the shipped image, requires the RUST_VERSION pin, reads the active toolchain’s host triple (rustup show active-toolchain) and requires cargo-cbuild — it fails on the 2026-09-01 image (tests/test-runtime-image-gates.sh). The same class applies to any host tree copied from artifact-source: /opt/flutter ships bare from the amd64 sdk stage, and bootstrap_flutter_sdk (setup-package-image.sh) caches the target-arch Dart SDK, which check_flutter verifies the same way (Bootstrapping Flutter in the package stage). The general audit for that class is check_manifest_tree_arch: it reads the ELF machine of everything under every tree in runtime-artifacts.manifest and fails when a foreign image carries builder-arch objects — docs/artifact-copy-completeness.md#the-shipped-trees-must-carry-the-images-own-arch

A packaging script dies with no message#

Symptom. A Dockerfile.package / Dockerfile.torch RUN ends with did not complete successfully: exit code: N and the last line of output is an ordinary progress line. Nothing names a command, a line, or a reason. Seen twice in the 2026-09-03 18:13 run: the arm64 package stage (exit code: 127, last line Keeping existing /usr/local/cargo/bin/rustup) and the amd64 torch stage (exit code: 1, last line Checked 1 package in 2ms).

Cause. set -e exits silently by design. The three image-side scripts — setup-package-image.sh, setup-torch-venv.sh, assemble-torch-app.sh — are several hundred lines of small functions, so the failing command can be any of them, and the usual suspects say nothing on the way out:

  • a command substitution under pipefail whose stderr is discarded (_got="$(rustc --version 2>/dev/null | awk …)" — an unrunnable foreign-arch rustc produces exit 127 and not one byte of output);

  • a helper whose last command is a failed test, which becomes the helper’s own return value (see A trailing conditional fails the whole script);

  • any uv/git call that fails with its diagnostics already consumed.

Fix. The three scripts run set -Eeuo pipefail and install_err_trap (01-core/logging.sh), so a set -e death now prints [ERROR] Command failed (line N): <the command> before it exits. -E is the load-bearing flag: without errtrace the trap is not inherited by shell functions, which is where all the work happens. Dockerfile.torch COPYs logging.sh into /opt/scripts/core/ for this.

The probe that produced the arm64 127 also names its own failure now: an unrunnable ${CARGO_HOME}/bin/rustc reports does not execute and points at ensure_native_rust_toolchain.

Reading it. The trap reports the line of the failing command, not of the caller. Match it against the script in the image (nerdctl run --rm <image> sed -n '<N>p' /opt/scripts/packaging/<script>; assemble-torch-app.sh lives in /opt/scripts/03-media/final/), because the line numbers move with every edit to the file.

A callee invoked in an if ! condition runs with errexit off#

Symptom. A multi-hour builder fails, the chain carries on, and the stage reports success. Nothing in the log says the exit code was thrown away.

Cause. Bash suspends errexit for the entire dynamic extent of a command used as a condition — the callee’s body, and everything it calls. So a function written to rely on set -e for its aborts raises nothing when it is reached through if ! f …, f && …, f || … or a condition-context substitution. gcc.sh:384 reaches build_canadian_native_gcc_for exactly that way, and inside it the invocation of build-gcc.sh was a plain command: its non-zero exit was discarded, and the missing native GCC only surfaced arches later.

Fix. Inside such a function every failure has to be raised explicitly. The builder invocation now ends || die "Canadian native GCC build FAILED for …", and the two [ -x … ] checks on the produced gcc/g++ die on their own rather than through errexit. tests/test-gcc-errexit-contract.sh holds both halves: one characterisation case that runs a real bash -c and records that if ! f really does suppress errexit inside f — the mechanism, not our code — and contract cases over gcc.sh itself asserting the explicit raises are still there.

The check to carry away. Before trusting set -e inside a helper, grep for how the helper is called. A guard that only works in the default context is not a guard; it is a coincidence.

A checksum probe that cannot reach the server reads as “nothing to verify”#

Symptom. The build prints No sha512.sum found on server; continuing. and goes green. The tarball it just unpacked was verified by nothing.

Cause. The GCC tarball is fetched mirror-first (ftpmirror) for speed, while both proofs — sha512.sum and the detached .sig — exist only on gcc.gnu.org. Verification was gated on a wget --spider probe of that host, so any non-200 answer — outage, 5xx, DNS — took the “the server has no checksum” branch and returned 0. The one event the mirror exists to survive was also the event that silently disarmed the integrity check on the mirror’s bytes.

Fix. In build-gcc.sh, an unreachable or absent sha512.sum now goes through _gcc_sha_unverified_or_die, as does a sha512.sum that downloads but carries no entry for this tarball; a sha512.sum that probes but fails to download was already fatal. The .sig path routes through _gcc_gpg_require_or_warn, so GCC_REQUIRE_GPG=1 makes a missing signature fatal too. tests/test-gcc-tarball-verification.sh extracts the helpers — build-gcc.sh is a top-level script, so they cannot be sourced — and drives each branch with the probe stubbed.

The check to carry away. “The proof was not reachable” and “there is no proof” are different answers to different questions. Any verifier that maps the first onto the second has stopped being a verifier at the moment it matters most.

A soname with no map entry is resolved by an apt-cache prefix guess#

Symptom. Nothing visible. validate-media-runtime.sh reports a missing soname as resolved, the media runtime installs a package, and the wrong (or wrongly versioned) library ships.

Cause. Three hand-maintained truths claim to say which apt package provides a media runtime library, and only one of them is reachable at build time:

  1. the codec-lib baseline in 06-packaging/setup-torch-venv.sh :: _install_cv2_runtime_apt — the torch stage’s fallback install list when the ffmpeg manifest is absent;

  2. the SONAME→package map 03-media/runtime/so-package-map.txt, which validate-media-runtime.sh uses to resolve a missing soname deterministically;

  3. /opt/ffmpeg/runtime-apt-packages.txt (emit_runtime_apt_manifest in build-ffmpeg.sh) — the ground truth, which exists only inside a built image, so no static check can reach it.

Where (2) has no entry, resolution degrades to dpkg-query and then to apt-cache search "^${base}[0-9]" | head -1 — an arbitrary first hit. And known_so_packages_load is last-wins, so a duplicate soname naming a different package is a silent override rather than an error. A baseline bump (libx265-215 → libx265-2xx) that forgets the map degrades exactly that one library to guessing, and says nothing.

Fix. tests/test-codec-so-map-convergence.sh freezes the statically checkable convergence of (1) and (2): the map is well-formed (SONAME<TAB>package, * family keys allowed), no soname maps to two different packages, and every versioned runtime lib package hardcoded in the baseline appears as a mapping target. KNOWN_UNMAPPED is a ratchet, not an excuse list — it records the divergence that already existed on 2026-08-24, because inventing map entries would mean inventing sonames, and it fails in both directions so the list can only shrink honestly. Truth (3) stays out of a static test’s reach; that is stated rather than papered over. ONNX Runtime is denied in code (ort_is_denied_soname, ort-runtime-gate.sh), so a missing or edited map cannot reopen the 2026-08-27 path, where the guess pulled a distro libonnxruntime1.x.

The NVIDIA apt keyring 404s on ubunturesolute#

Symptom. Dockerfile.nvidia dies in its first RUN fetching .../compute/cuda/repos/ubunturesolute/<arch>/cuda-keyring_1.1-1_all.deb (404).

Cause. NVIDIA’s repository path is the Ubuntu version digits (ubuntu2604), not the codename.

Fix. setup-cuda-repo.sh builds it from UBUNTU_VERSION. A keyring SHA in versions.env names one repository’s file; move both pins together when the Ubuntu base moves.

nvcc rejects the image’s GCC 16#

Symptom. #error -- unsupported GNU version! from nvcc’s host_config.h; or, after working around it with CUDAHOSTCXX=g++-15, GLIBCXX_3.4.xx not found at link time.

Cause. nvcc checks the host compiler’s version. Pairing CUDA objects built by GCC 15 with a GCC 16 libstdc++ is what produced the GLIBCXX errors.

Fix. Keep GCC 16 and set NVCC_PREPEND_FLAGS=-allow-unsupported-compiler (Dockerfile.media, Dockerfile.package). Never downgrade the host compiler.

A CUDA compile is Killed though average memory looked fine#

Symptom. A bare Killed in the ONNX Runtime GPU step; the kernel log shows cicc at 3-6 GB.

Cause. Heavy CUDA translation units are staggered, so the average RSS hides the peak.

Fix. CUDA_MB_PER_CICC (default 3500) sizes the GPU job count against the peak. Raise it before adding swap.

An exception through std::call_once segfaults in libunwind#

Symptom. A process dies with SIGSEGV where a library should have reported an error. The backtrace runs pthread_once → _Unwind_Resume (GCC’s libgcc_s) → __gcc_personality_v0 → __libunwind_Unwind_GetLanguageSpecificData (libunwind.so.8) → address 0. Seen 2026-10-01 in nvinfer, whose TensorRT engine build threw inside std::call_once.

Cause. Two unwinders in one process. libunwind.so.8 exports the _Unwind_* symbols too, and it was ahead of libgcc_s.so.1 in the lookup scope: libgstreamer-1.0 and libcamera-base both link it for their backtrace support. glibc’s pthread_once cleanup resumes the exception through libgcc_s directly, whose personality routine then calls libunwind’s getter with a libgcc_s context. Measured with a ten-line std::call_once that throws and catches: it crashes when libunwind.so.8 is loaded first (also when libcamera-base is dlopen()ed RTLD_LOCAL first), and passes otherwise. A C host such as gst-launch-1.0 or Python always loads libgstreamer before any C++ library.

Fix. GStreamer core and libcamera build without libunwind (-Dgstreamer:libunwind=disabled, libcamera’s -Dlibunwind=disabled), in every variant (owner decision 2026-10-01: everywhere, not only where DeepStream runs). Their backtraces fall back to glibc’s backtrace(). Both halves are measured:

  • GStreamer. GStreamer 1.29.2 core rebuilt that way: the same nvinfer failure reports TensorRT’s error and an element error instead of a segfault.

  • libcamera (2026-10-01, build-libcamera.sh run twice in a container FROM the published :latest amd64, v0.7.2, GCC 16.2). Without the flag meson finds libunwind 1.8.3 and libcamera-base.so.0.7.2 NEEDs libunwind.so.8; a C host that dlopen()s it first (either RTLD_GLOBAL or RTLD_LOCAL) and then a C++ library whose std::call_once throws dies with SIGSEGV (rc 139). With the flag the NEEDED list has libgcc_s.so.1 instead and the exception is caught. A C++ executable that loads it later passes either way: its own libgcc_s is already first.

What else in the image needs libunwind.so.8 (readelf over every ELF, all three arches, 2026-10-01): under /opt and /usr/local only those two libraries, directly or through them. Outside it, distro packages only: Xvfb (amd64, arm64) and gperftools’ libprofiler and libtcmalloc* (amd64). Those are Ubuntu’s builds; a C host that preloads libprofiler brings the hazard back. The runtime smoke’s check_no_libunwind_closure fails any shipped file under /opt or /usr/local that needs it, and tests/test-no-libunwind.sh pins both flags and that verdict. libunwind-dev stays installed (several install-deps.sh and the runtime image ship it), so the flags are the guard: do not set them back to auto to “get better backtraces”.

lavapipe segfaults building an acceleration structure on arm64#

Symptom. A Vulkan test on lavapipe (Mesa’s CPU device) dies with SIGSEGV as soon as a draw builds an acceleration structure; the core shows a 4-lane st1 in rs_scatter_smem. Seen on arm64 in BeschleunigerBallett run 36746313937; amd64 passes.

Cause. Mesa 26.0.8’s lavapipe compiles its BVH radix sort (lvp_acceleration_structure.c, subgroup_size_log2 = 3) for 8-lane subgroups. llvmpipe’s subgroup is its vector width / 32: 256 bits (8 lanes) on AVX2, but 128 (4 lanes) on arm64 NEON and on riscv64. The scatter then writes through garbage addresses. Measured 2026-10-01 with vulkaninfo in the published :latest: subgroupSize 8 on amd64, 4 on arm64 and riscv64, and 8 on all three with LP_NATIVE_VECTOR_WIDTH=256.

Fix. The package image sets ENV LP_NATIVE_VECTOR_WIDTH=256 on every arch (CON44), and the runtime smoke’s check_lavapipe_subgroup fails an image whose variable or lavapipe subgroup is anything else. Retire both once the image’s Mesa carries upstream ebcfbe60 (2026-08-22), which deletes that sort.

Headless Chrome prints nothing under qemu-user#

Symptom. In an emulated arm64 container, chrome --headless --dump-dom <url> prints nothing and exits 134; stderr shows `qemu: uncaught target signal 5 (Trace/breakpoint trap)

  • core dumped`. The package stage’s Chrome check fails this way when it builds arm64 under QEMU.

Cause. Chrome forks its renderers from a zygote, and the forked children die under qemu-user. Measured 2026-10-01 with Chrome for Testing 154: --no-zygote renders the page in 21 s.

Fix. Start Chrome through /usr/local/bin/chrome ($CHROME_EXECUTABLE), which passes --no-zygote: consumer-image-contract.md.

Chrome stops with No usable sandbox!#

Symptom. FATAL:content/browser/zygote_host/zygote_host_impl_linux.cc:129] No usable sandbox! If you are running on Ubuntu 23.10+ …, exit 134, as uid 1001 in a container.

Cause. The renderer sandbox needs user namespaces or a setuid-root helper. The image ships no setuid helper, and the host decides about user namespaces. Measured 2026-10-01: the bare binary started on amd64 under Rancher Desktop and died this way on arm64 under qemu-user.

Fix. The same wrapper passes --no-sandbox: consumer-image-contract.md.

An arm64 app dies in ndk_translation on an API 30 emulator#

Symptom. An arm64-v8a APK installs on an x86_64 Android 11 (API 30) emulator, the Flutter engine starts, then the process dies with SIGILL. logcat shows ndk_translation: Undefined instruction 0x7ee1b800, and the tombstone ends in libndk_translation.so DecodeSimdScalarTwoRegMisc.

Cause. API 30’s ARM translator (ndk_translation 0.2.2) lacks some ARMv8.0 instructions; 0x7ee1b800 is scalar FCVTZU. Seen 2026-10-01 with OmniAccelerANT’s release APK.

Fix. An API 35 system image, whose translator runs the same APK. The image pins it: consumer-image-contract.md.

The Vulkan variables name an arch-specific SDK dir#

Symptom. After the entrypoint, env | grep /opt/vulkan/ names /opt/vulkan/<version>/<arch> rather than /opt/vulkan/active. Natively that is the right SDK; a riscv64 test under QEMU in the amd64 image then loses Vulkan, both in CMake and in the loader (the two messages).

Cause. The entrypoint, not the login shell, sources LunarG’s setup-env.sh. That script sets VULKAN_SDK to $(readlink -f <its dir>)/$(uname -m), so in the amd64 image every variable derived from it names /opt/vulkan/<version>/x86_64: VULKAN_SDK, VK_ADD_LAYER_PATH, PATH, LD_LIBRARY_PATH, PKG_CONFIG_PATH, CMAKE_PREFIX_PATH. Measured 2026-10-01 in the published :latest amd64 child; with --entrypoint bash the image ENV’s /opt/vulkan/active survives. Under QEMU_LD_PREFIX a riscv64 process re-roots an absolute path into the riscv64 sysroot, which has no x86_64 dir, so it falls through to the amd64 files. /opt/vulkan/active exists in both and names each one’s own SDK.

Fix. vulkan_env_prefer_active_link (01-core/vulkan-env.sh) rewrites every component under the link’s target back to <root>/active, without duplicates; vulkan_env_source and the entrypoint call it (CON48). Native runs are unchanged, since the link resolves to the same dir. The runtime smoke’s boot probe fails an image that still prints a /opt/vulkan/<version>/ component (vkarch=), and requires VK_ADD_LAYER_PATH (vkadd=yes) as its proof that the entrypoint sourced setup-env.sh. The :latest of 2026-10-03 carries the fix: after its entrypoint every Vulkan variable names /opt/vulkan/active (checked 2026-10-06), so riscv64_cross_env no longer sets VULKAN_SDK and VK_ADD_LAYER_PATH itself.

The core registry blacklists libgstvalidatessim.so#

Symptom. gst-inspect-1.0 -b lists libgstvalidatessim.so, and gst-inspect-1.0 <path>/validate/libgstvalidatessim.so says it “appears to be a GStreamer plugin, but it failed to initialize”. No Failed to load plugin line, no missing symbol (ldd -r is clean). amd64 only: cross builds disable gst-devtools.

Cause. gst-devtools installs its validate plugins under <libdir>/gstreamer-1.0/validate/, and the core registry scans plugin directories recursively, so it loads them too. The SSIM plugin’s plugin_init returns FALSE when gst-validate is not initialized, and the registry blacklists a plugin whose init fails. Under gst-validate-1.0, which keeps its own registry, it loads and works.

Fix. patches/gstreamer/007-validate-ssim-register-outside-validate.patch returns TRUE there instead, registering nothing, like the gapplication plugin already does (upstreamable-patches.md § 22). The runtime smoke’s plugin-health check now fails on any blacklisted plugin that _PARITY_GST_KNOWN_BROKEN does not document, and check_gst_validate_ssim runs the override under gst-validate on amd64.

DeepStream: nvstreammux stops the pipeline with reason error (-5) and no message#

Symptom. Any pipeline through nvstreammux fails at preroll; the only error is h264parse0: Internal data stream error … streaming stopped, reason error (-5).

Cause. The legacy mux, NVIDIA’s prebuilt default, reads GstMapInfo.data after gst_buffer_unmap(), and GStreamer ≥ 1.28 clears it on unmap.

Fix. USE_NEW_NVSTREAMMUX=yes, which the image sets with DeepStream: linux-accelerator-images.md § nvstreammux: the new one only.

DeepStream: Unable to load library: libnvinfer_builder_resource_sm75.so.10.16.1#

Symptom. nvinfer cannot build an engine: IBuilder::buildSerializedNetwork: Error Code 6: API Usage Error (Unable to load library: libnvinfer_builder_resource_sm75.so.10.16.1 …).

Cause. TensorRT 10 dlopen()s its builder resources by file name, and ld.so.cache holds them only under their do_not_link_against_* SONAME.

Fix. The links deepstream.sh puts into /usr/lib/<triplet>/: linux-accelerator-images.md § TensorRT builder resources, by file name.

DeepStream: nvv4l2decoder: /dev/nvidia0 “isn’t a v4l2 driver”#

Symptom. Error getting capabilities for device '/dev/nvidia0': It isn't a v4l2 driver, then Failed to open decoder.

Cause. The distro libv4l2 did not load NVIDIA’s NVDEC plugin, which it looks for only in its own plugin directory.

Fix. The plugin link deepstream.sh stage-runtime makes: linux-accelerator-images.md § NVDEC through the distro libv4l2.

A no-push wrapper build cannot find its own android image#

Symptom. In Dockerfile.torch’s wheels-source stage: cross-android-...: not found; with an OCI context added, unable to get info about digest ... content sha256:...: not found; extracting with nerdctl cp, cannot use cp with stopped containers in rootless mode.

Cause. The android tag exists only in containerd’s store, which BuildKit’s OCI worker cannot see. nerdctl maps every oci-layout:// context onto one store id (parent-image-key), so a second one beside runtime_package makes one of them unresolvable. Rootless nerdctl refuses cp from a stopped container.

Fix. runtime_wheels_context_dir extracts /opt/wheels with nerdctl export | tar into a directory context. Never pass two oci-layout:// contexts to one nerdctl build. Since 2026-09-24 RUNTIME_WHEELS_SOURCE=export is the other way: its own export build reads the android OCI layout (the only oci-layout:// context of that build), after proving it is the same image (both deliveries).

The torch RUN idles for minutes before uv venv#

Symptom. The wrapper’s torch RUN (the one that mounts wheels-source) writes nothing for minutes. The first /opt/venv file is dated 161 / 565 / 661 s (amd64 / arm64 / riscv64) after the COPY setup-torch-venv.sh layer (2026-09-22 lane), where the day before it was 96 / 1 / 2 s with the same scripts.

Cause. Likely, not proven: to bind-mount /opt/wheels (about 0.1 GB), BuildKit must hold and checksum the whole android snapshot (about 12.7 GB in 96 layers). When a prune or GC evicted that snapshot after the package build, the wrapper pulls and extracts it again first. A lock on the sharing=locked apt cache mounts would look the same from inside the RUN.

Fix. RUNTIME_WHEELS_SOURCE=export stages the wheelhouse before the package build, so the wrapper no longer depends on the snapshot. To tell the causes apart, compare the wheels-source vertex’s extracting lines with the RUN vertex’s start: the recipe.

RUNTIME_WHEELS_SOURCE=export stops a runtime lane#

Symptom. One of:

  • RUNTIME_WHEELS_SOURCE=<x>: expected auto, image or export (exit 2, at chain or lane start);

  • the android layout <dir> holds <digest>, but <ref> is <digest> in containerd (or is not present in containerd);

  • needs ARTIFACT_CONTEXT_MODE=oci to prove the layout is <ref>;

  • no wheel in <dir>/opt/wheels, exported from <ref>;

  • the wheelhouse staged for <arch> at <dir> is missing or changed since its export;

  • from BuildKit, target stage "wheels-export" could not be found.

Cause. Export mode refuses anything that would change where the wheels come from, or mount bytes it did not seal. In order: a typo; a --no-push artifact layout that is not the image image mode would mount, which is every --no-push chain whose android tag is published (every default or variant one on the amd64 cross host): such a chain threads no pin, so the lane names the registry’s digest (<ref> is <repo>@sha256:...) while the layout holds this run’s android; a directory artifact with no digest to compare; an android image without a wheelhouse; a staged directory changed or removed between the export and the wrapper; a --wrapper-dockerfile without the export stage.

Fix. RUNTIME_WHEELS_SOURCE=image is the pre-2026-09-24 delivery and needs none of these. Nothing falls back on its own. For a digest mismatch, use it too: rebuilding android in the same run is exactly the published-tag case above. Never re-export the layout from the ref the error names, because the package build copies from that layout and would switch to the published android too (why).

A GPU venv ships two onnxruntime distributions#

Symptom. ARCH-PARITY: MORE THAN ONE onnxruntime distribution (onnxruntime_gpu and onnxruntime_webgpu); the build log has rm: cannot remove '/opt/wheels/...': Read-only file system.

Cause. /opt/wheels is a bind mount, read-only by default, and the variant prune’s rm -f ... || true swallowed the error.

Fix. Dockerfile.torch mounts it rw (BuildKit discards the writes after the RUN) and the prune has no || true. CPU images never showed this because they carry no _gpu wheel. Since 2026-09-23 the ORT census at the end of install_project_environment fails the torch stage on two owners, before any smoke runs.

The torch stage fails with ORT-CENSUS FAIL#

Symptom. The torch stage stops with lines like these:

  • ORT-CENSUS FAIL onnxruntime-ep-webgpu 0.4.0 at C:\opt\OrchestrANT\.venv\Lib\site-packages is not a chain wheel: the store C:\runtime\wheels has no wheel of that name and version

  • ... 52 file(s) differ from the chain wheel's bytes, e.g. ...

  • the onnxruntime import package has 2 owners (...), expected exactly one

  • on Linux, before uv sync: ERROR: no chain ONNX Runtime wheel in /opt/wheels for ONNX_PACKAGE=...

  • on Windows, before uv sync: uv.lock pins onnxruntime-genai, ... and C:\runtime\wheels has no chain wheel of that family

It comes from Linux assemble-torch-app.sh, from Windows Build-TorchApp.ps1 (install or verify, including the rocm-1 stage), or from smoke section 21’s app-verify assertion.

Cause. Every ONNX Runtime in the app venv must be a chain wheel (onnxruntime-single-source.md). linux/scripts/03-media/runtime/ort-venv-census.py runs in the venv’s own interpreter (python -I); Build-TorchApp.ps1 embeds the same program. It treats as ORT every distribution named onnxruntime or onnxruntime-* and every one that ships files into onnxruntime, onnxruntime_genai or onnxruntime_extensions, so ort-nightly counts. Each must be byte-identical to the store wheel of its name and version: every payload file with the same SHA256, and nothing installed that the wheel lacks. Name and version are not enough, because PyPI publishes onnxruntime 1.31.0 for cp314 too. It also requires one owner of onnxruntime, the import resolving to that owner, no unowned file in those package directories, and a ORT-CENSUS PASS line: exit 0 without one is a failure.

Fix. Put the flavour into the chain store; never install it from PyPI and never exempt it.

  • The reconcile already uninstalls every candidate --purge-list names before it force-installs the store, so a foreign one was installed after the reconcile. The rocm lane’s PyPI onnxruntime-ep-webgpu was exactly that, until the WebGPU EP moved into the chain ORT.

  • A missing chain wheel is a media-stage problem. On Linux the cross ORT wheel is built only when cross_target_python_dev_ready.

  • On nvidia the chain GenAI is onnxruntime-genai-cuda or -trt-rtx, not onnxruntime-genai. ARCH-PARITY, VENV-SET and the GEN1 binding check count any onnxruntime-genai(-<flavour>) as the GenAI (_pkg_count, genai_dist_version); do not add an onnxruntime-genai arm to _venv_pkg_exempt, which would read STALE on every image that ships the plain name.

Not covered here: whether the store wheel itself was compiled by the chain (the image census), files installed outside site-packages, and ORT vendored under another distribution and package name.

The torch stage fails with CHAIN-WHEEL FAIL#

Symptom. The Linux torch stage stops after the torch pins with lines like these:

  • CHAIN-WHEEL FAIL onnxruntime-genai: the venv has 0.15.2 from an index; the chain staged onnxruntime_genai-0.15.2-cp314-cp314-linux_x86_64.whl (0.15.2)

  • CHAIN-WHEEL FAIL ai-edge-litert: not in the venv; the chain staged ai_edge_litert-2.2.0-...whl (2.2.0)

  • then CHAIN-WHEEL FAIL: N staged wheel(s) are not what the venv imports

Cause. assert_chain_wheels_installed in assemble-torch-app.sh requires every wheel in /opt/wheels to be the installed distribution of its name, read from the direct_url.json uv writes (file:///opt/wheels/<file>). Same name and version are not enough: OrchestrANT’s lock names PyPI builds of ai-edge-litert, onnxruntime-genai and IREE, and a PyPI build of the same version is not the chain’s. Exempt: TVM (optional, its install may fail), OpenCV when the source-built bindings are staged (removed on purpose), and IREE on riscv64 (non-fatal there). The lock cannot name /opt/wheels itself: off the image uv lock stops with Failed to read --find-links directory: /opt/wheels, and no PEP 508 marker tells the image from a dev host (linux-cross-builds.md § The chain wheels and the app’s lock).

Fix. Find who installed the PyPI build after reconcile_local_wheels, the one place that installs the store. The sync must name the family in --no-install-package (collect_locked_local_skip_packages, and the GenAI arm of build_uv_sync_args); a family missing there lets the lock install it. enforce_torch_version_pins reinstalling torch over a staged torch wheel reads the same way. Never exempt the wheel.

A build or smoke stops on an ONNX Runtime that is not the chain’s#

Symptom. One of:

  • Windows, at the end of a consumer build: ORT gate (<consumer>): N finding(s), the build reached an ONNX Runtime other than the chain's, after FAIL: lines naming a file, a record path or a log line.

  • Linux, the same place: ORT-GATE FAIL (<consumer>): ... lines, then ORT-GATE FAILED.

  • no build record names the chain ONNX Runtime ...: the consumer built without ORT at all. On Windows FFmpeg that is a missing chain header.

  • The smoke: ORT census: STAMP ...\ort-provenance\<consumer>.json -- missing, or not naming this image's chain core lib.

  • Linux OpenCV after cmake: configure log: ... DNN: Downloading ONNX Runtime, ONNXRT_ROOT_DIR=...3rdparty/onnxruntime..., or no 'ONNX Runtime: YES (ver X)'; before cmake, OpenCV must build against the chain ONNX Runtime at /usr/local/lib/onnxruntime-cpu.

  • Linux apt or loader: FAIL: this apt install would pull a distro ONNX Runtime, FAIL: a distro ONNX Runtime is installed beside the chain build, FAIL: a loader path reaches an ONNX Runtime that is not the chain build.

Cause. A consumer compiled, linked, fetched or loaded an ORT that is not the chain build. An upstream bump that re-enabled a download is the usual source. The STAMP case means the consumer’s layer predates the build gate or ran against another ORT build. On Linux, Ubuntu’s gstreamer1.0-plugins-bad depends on libonnxruntime1.x, which is how apt pulled a distro ORT on 2026-08-27 (12 packages, 62.9 MB).

Fix. Remove the source, never loosen the gate: restore the pre-set, ORT_HOME or fetch block the consumer lost; drop the package that drags a distro ORT in; replace a stray copy with a link to the chain file. For STAMP, rebuild that consumer’s stage. Which gate sits where, and what each message means: onnxruntime-single-source.md § Failure messages.

The wrapper smoke fails clang --version after a partial rebuild#

Symptom. COMPILER FAIL [clang-version]: clang --version: 23.1.0 (expected 23.1.1) when only the later stages were rebuilt.

Cause. The lower stages were built before a pin wave; the runtime lane asserts today’s versions.env.

Fix. Rebuild from the stage the pin belongs to. To finish a chain on its own older pins, export the payload pins (LLVM_RELEASE, ONNXRUNTIME_VERSION, …) for the runtime run only; the runtime base installs its own and needs today’s.

fatal error: sanitizer/common_interface_defs.h: No such file or directory on arm64/riscv64#

Symptom. A -fsanitize=address build in the arm64 or riscv64 image stops on that header (abseil’s dynamic_annotations.h includes it). Past it, the link cannot find libasan/libubsan. The same build passes in the amd64 image. Since 2026-09-23 the chain stops earlier instead, with one of libsanitizer installed no headers/libasan for <triplet> (compiler stage), ERROR: /opt/gcc-16.2.0 lacks the sanitizer runtime: ... (android swap) or COMPILER FAIL [gcc-sanitizers] (wrapper smoke).

Cause. The arm64 and riscv64 cc is the Canadian-native GCC, and build-gcc.sh built only libgcc, libstdc++ and libatomic for any --target build.

Fix. Rebuild the chain from the compiler stage; build-gcc.sh now builds libsanitizer when host == target. A partial rebuild on older images fails on purpose: --from-stage sdk|media|android at the android swap, and a runtime-lane run on the published android images at the wrapper smoke. Mechanism, gates and cost: cross-build-verification.md#the-native-gcc-ships-libsanitizer.

A Jetson GPU container sees no GPU#

Symptom. unresolvable CDI devices nvidia.com/gpu=all; or NvRmMemInitNvmap failed: error Permission denied then No CUDA GPUs are available; or OCI runtime create failed: unknown version specified.

Cause. Rootless nerdctl on Tegra: the CDI spec is hidden by rootlesskit’s /run, the GPU nodes belong to the host video group a rootless process drops, and Ubuntu’s crun is too old for containerd 2.x.

Fix. The three flags and the crun install: linux-host-setup.md § B2b.

A USB camera delivers half its frame rate#

Symptom. A V4L2 capture sits at exactly 15 fps while the camera advertises 30.

Cause. CAP_PROP_BUFFERSIZE=1: with one V4L2 buffer the camera cannot capture while the last frame is dequeued.

Fix. Leave the buffer count alone and hand on only the newest frame from a capture thread, as linux/jetson-webcam/app.py does.

A push leaves the repo bare: core.bare and core.worktree do not make sense#

After a git push, every git command in the checkout prints warning: core.bare and core.worktree do not make sense, git status stops with fatal: unable to set up work tree using invalid config, and git check-attr finds nothing, so Dockerfile.EolAttributes.Tests.ps1 fails and lists ~64 files that .gitattributes plainly covers. The gitdir’s [core] section now ends in bare = true.

Cause. Git exports GIT_DIR (and GIT_WORK_TREE, GIT_INDEX_FILE, GIT_PREFIX) to hooks. The pre-push hook ran the mutation gate, whose fixtures call git -C <tmp> init with that variable still set, so each init re-initialised the REAL gitdir instead of the temp one. For a gitdir whose path does not end in /.git — every submodule’s .git/modules/<name> — git guesses “bare” and writes core.bare = true. The pre-commit hook already cleared these variables; pre-push did not. Seen 2026-09-22 on this repo checked out as a submodule: the config changed mid-push, while the branches and the index stayed untouched.

Fix. git config --file <gitdir>/config core.bare false, where <gitdir> is the path in the checkout’s .git file. The pre-push hook now clears the same four variables as pre-commit, mutation-tested as mutations.push-clears-git-env.

HailoRT’s configure is slow while the cache looks healthy#

Symptom. arm64’s Hailo RUN spends most of its time before CMake generates (384 s of 595 s in the 2026-09-22 lane), yet every compile the cache counts is a hit. The nested external/protobuf-build/CMakeCache.txt has no CMAKE_CXX_COMPILER_LAUNCHER. Since 2026-09-24 a build can stop instead, with [hailo] ERROR: the nested build compiled N objects and the cache saw R requests or ... no longer spawns its nested build through 'env -i HOME=$ENV{HOME} ...'.

Cause. HailoRT compiles a host protobuf inside its own configure, under env -i. The launcher environment setup_ccache exports never reaches that build, whatever the resolver decides. Two shapes seen while fixing it:

  • sccache --show-stats fails with Mismatch of client/server versions? (tag for enum is not valid). A clean PATH puts /bin first and finds the distro sccache 0.13; when no server answered it, it started a 0.13 server on the socket the pinned 0.17 client then talks to.

  • The probe sees 0 requests after another user’s run in the same container. SCCACHE_ERROR_LOG is /tmp/sccache.log for every user, and under fs.protected_regular=2 the second user, root included, cannot open the first user’s file, so its server does not start.

Fix. HAILO_NESTED_CACHE=carry, the default, carries the cache environment and the parent’s own sccache into that build, and its gate stops a carry build the cache did not reach. HAILO_NESTED_CACHE=off is the build as it was before 2026-09-24. Run the probe in a fresh container, or give each user its own SCCACHE_ERROR_LOG. The carrier and its gate.

import hailo_platform fails: _pyhailort is a 4 KB module#

Symptom. In an image built before the HAILO_PYHAILORT_IPO switch, or with HAILO_PYHAILORT_IPO=upstream: import hailo_platform fails, and _pyhailort.cpython-314-<arch>-linux-gnu.so is about 4 KB. readelf --dyn-syms lists no PyInit__pyhailort, only a __gnu_lto_slim object. The build date does not say which build an image had; the module does. With the switch at its default, off, the build stops instead: [hailo] ERROR: ... does not export PyInit__pyhailort.

Cause. Upstream forces IPO in pyhailort’s CMake. GCC writes slim LTO objects, and lld, the linker setup_lld_linker selects, cannot read them, so the link yields a module with no code.

Fix. HAILO_PYHAILORT_IPO=off, the default, patches the forced IPO out of the cached source. pyhailort.

Windows: the layer store (hcsshim)#

hcsshim::ActivateLayer 0x20 on an AMD Radeon host#

Symptom. Windows-container layer commit/finalize dies hcsshim::ActivateLayer 0x20 on an AMD Radeon host (buildkit failed to import/reimport snapshot, on ANY layer writing into an existing parent dir; docker-classic legacy dies mkdir \\?\Volume{<GUID>}\C:. invalid dir name) — BOTH lanes, process AND Hyper-V isolation

Cause. RESOLVED 2026-08-10: an ENABLED AMD RDNA4 dGPU (RX 9xxx + Adrenalin) locks freshly-written container layers (upstream docker/for-win#14977, open; same-boot A/B-proven). Failed finalizes additionally WEDGE hcs state until a REBOOT (survives service bounces + vmcompute restart).

Fix. Probe with Test-BuildCopy.ps1 -Heavy (only a -Heavy-green verdict counts), then build inside the Set-Rdna4Gpu.ps1 -Disable window — Build-Buildkit.ps1 enforces that via Assert-NoActiveRdna4Gpu. After ANY red finalize, reboot before testing anything else: a wedged host falsifies every later experiment. Full A/B history and the superseded 2026-08-09 verdict: windows-build-lanes.md.

ExportLayer 0x3, spawn flakes, ExportLayer 0x70 — disk exhaustion in costume#

Symptom. ANY weird hcsshim failure on the BK lane: ExportLayer 0x3 (path not found) at finalize, spawn flakes ('cmd.exe' is not recognized), ExportLayer 0x70 (disk full)

Cause. Disk exhaustion in costume — the bk image generations stack 30–40 GB per rebuild cycle; only 0x70 names the disease (all three hit on 2026-08-03)

Fix. Check free disk FIRST. buildctl prune --all + docker image prune -f (non-admin); bk-* image generations need admin. Full playbook: docs/windows-build-lanes.md § Store GC.

ExportLayer 0x3 at finalize of a heavy media layer, disk is fine#

Symptom. BK lane, disk is FINE: failed to reimport snapshot: hcsshim::ExportLayer ... (0x3) at finalize of a heavy media layer (GenAI/OpenCV class) — deterministic, every fresh snapshot, both finalize paths, survives host reboot

Cause. ROOT CAUSE FOUND & FIXED 2026-08-06: the runhcs shim’s hardcoded tearDownTimeout = 30s (cmd/containerd-shim-runhcs-v1/task_hcs.go) terminates the heavy-churn silo teardown mid-hive-flush (real duration: 117 s measured for OpenCV), permanently poisoning the scratch vhdx. Calm exits (ONNX ninja, cpython, LiteRT, torch) finish teardown under 30 s and never tripped it.

Fix. Solved at the root by a patched runhcs shim, already deployed. The part to remember is the maintenance: every Stevedore/containerd update overwrites it, so Publish-ShimPatch.ps1 -ReportOnly belongs in your post-update routine — Assert-ShimPatch gates the BK lane on it by SHA256 — and one OpenCV canary should follow any update. 0x3 is deliberately excluded from the driver’s transient-retry pattern: it must fail loudly. Root cause, constants and the rollback path: windows-build-lanes.md.

Every RUN step reports DONE 2841.2s — the same number, whatever it runs#

Status 2026-10-12: not seen since. The three complete chains of 2026-10-10/11 (bk-20261010-121545, bk-20261011-184301, bk-20261011-233025) run a trivial RUN in 4.9 to 9.0 s, the healthy band of the playbook below, with the 5 min knob still deployed. Their one fixed cost is a different shape: ~120 s between a RUN’s last line and its DONE, only in RUNs that started an sccache server (windows-refactor-backlog.md #179).

Symptom. BK lane, since the 2026-08-31 15:12:59 boot: every RUN step on a servercore base finishes GREEN but takes 2841.2 s, byte-identical across independent solves and independent of what the step does — RUN echo probe > C:probe.txt costs exactly as much as a real build step. Everything else in the solve is normal (COPY 5.1 s, export 1.7 s, unpack 5.2 s). Host is idle throughout: 4 % CPU, disk queue 0, no I/O.

Cause. The container’s shutdown/exit notification is lost, so the shim waits out its whole teardown timeout. 2841.2 = ~141 s container boot + 2700 s = the 45 min tearDownTimeout this repo patches in (local-45min-deployed.patch). The RUN itself succeeds: on a stuck container docker logs prints the command’s own output and docker top shows no cmd.exe — the process ran and exited. Reproduced straight through dockerd (no shim, no buildkit), so it sits below both lanes in hcs/vmcompute — the Windows-Containers#547 family the entry above is about, but escalated from “filesystem-heavy containers only” to every container. nanoserver:ltsc2025 starts and exits in ~2 s; only the servercore service stack is affected. Same shape of step took 7.7 s / 9.2 s on 2026-08-31 03:53, so it is a ~350× regression, and a reboot does not clear it. It needs ltsc2025 AND process isolation together, measured on the same host and day with the same Dockerfile: servercore:ltsc2025 441.3 s · servercore:ltsc2022 1.8 s · ltsc2025 under --isolation=hyperv 3.5 s · nanoserver:ltsc2025 ~2 s. An older user-mode image on the identical 26200 kernel is fine, so “process isolation is broken here” is not the shape of it. That fits the silo-only artefacts — the WcSandboxState\Hives\*_Delta flush failures logged as Kernel-General 6, and the orphaned bindflt instances ISSUE.md already counted — and it is diagnosis only, not an escape: buildkitd v0.32 has no Hyper-V isolation option, and Hyper-V isolation is capped at 2 CPUs on this host.

The 141 s half is named. Inside the container, SCM logs 7022 The LSM service hung on starting exactly 140.07 s after RpcSs comes up, and every other service starts within the same second afterwards. LSM (Local Session Manager) then sits in START_PENDING forever — a scan of all 121 services in a live container finds it as the only one not in RUNNING or STOPPED — and it reports NOT_STOPPABLE, IGNORES_SHUTDOWN. Under --isolation=hyperv the same image has LSM RUNNING. But disabling LSM does not help: an image committed with LSM Start=4 (verify with sc qc LSM, not just reg query — a quoted key path passed through PowerShell to reg.exe silently creates a key named LSM") published its output and then still sat in teardown for >13 min before that run was cut short by an unrelated force-remove — so the bound is “>13 min”, not a completed 2841 s, but it is already 100× the 7.7 s baseline. LSM is a co-symptom, and it is not worth chasing economically: 2700 of the 2841 s are the timeout, so the teardown is the only lever that matters — removing the whole 141 s boot stall would save 5 %.

Fix. UNRESOLVED as of 2026-09-01 — but do not spend the day re-falsifying what is already falsified: Defender RTP (reproduces with RTP off), the Defender platform (unchanged on disk since 2026-08-05), CNI/HNS (ADD succeeds, IP+gateway match the host nat adapter; --network none stalls identically), disk space (515 GB free), disk health (disk event 51 fires only on successful teardowns), the RDNA4 dGPU (cleanly disabled, Code 22), the shim patch (SHA256 matches), the buildkitd GC policy (active — buildkitd reads C:\ProgramData\buildkitd\buildkitd.toml by default, so a missing --config in the service ImagePath is a non-issue), and any new KB or driver (nothing was installed that day). Post-mitigation falsifications (2026-09-01): a Defender engine/SIU change (engine constant 1.1.26070.7 across good and bad periods; SIU updates landed just as frequently during the good window, the fast 03:53 build ran right after one), and in-container Windows Update (a layer with wuauserv+UsoSvc Start=4 — reg output confirmed applied — still burns the full teardown cap on the next RUN); disabling LSM was falsified above. The 141 s boot hang is NAMED (2026-09-02, wait-stack captured). LSM’s service-start thread in the silo’s svchost -k DcomLaunch -p sits in:

ntdll!NtWaitForSingleObject                                  <- never signalled
KERNELBASE!WaitForSingleObjectEx
lsm!CEventDispatcher::CEventListEntry::WaitForSessionEvent
lsm!CEventDispatcher::WaitForSessionState
lsm!CService::Start
lsm!ServiceMain

Two dumps 30 s apart carry the byte-identical stack and the thread reports 0.000 s user time — a fully static wait, not slow progress. So LSM never completes CService::Start because a session-state event inside the silo is never signalled; SCM then times it out (event 7022) and the rest of the boot proceeds. Reproduce with windows/scripts/diagnostics/Get-LsmWaitstack.ps1 (elevated; it grabs the next container start and dumps twice). Two notes from doing it: the script must find the silo through the process TREE (a new wininit.exe → its services.exe → their svchosts) because Win32_Process.ExecutablePath/CommandLine are empty for silo processes even elevated; and the dumps land SYSTEM-owned, so icacls <dir>\*.dmp /grant <user>:R before analysing them unelevated. cdb.exe ships inside the WinDbg store package (…\Microsoft.WinDbg_*\amd64\cdb.exe). The waited object is identified, its signaller is not: Find-LsmEventHolder.ps1 reads R10 as described below and finds an auto-reset Event, HandleCount 2, whose only handle any process holds is LSM’s own (draft § The waited object) — and one wrong way to try is recorded here because it looked convincing: reading the FIRST KERNELBASE!WaitForSingleObjectEx in the process names the SCM dispatcher’s own idle wait (sechost!ScDispatcherLoop), which every service process has, not LSM’s. Everything derived from that handle (HandleCount 2, “exactly one holder”) described the wrong object and was retracted 2026-09-02; the R10 re-read of the right object later measured the same two numbers. kb’s argument columns are home-space reconstructions and untrustworthy for a blocked wait; read R10 of the thread whose stack actually shows lsm!CService::Start (the x64 syscall stub does mov r10,rcx, so R10 still holds argument 1). Silo handles also resolve as Name <none> from outside, so the lsm!CEventDispatcher frames stay the reliable pointer. This is the concrete, reportable core for microsoft/Windows-Containers#547, which so far only describes the symptom — ready-to-file draft: upstream/windows-containers-lsm-session-event-hang.md.

Recurrence playbook — decode the timing before debugging anything. The signature is identical RUN durations regardless of what the step does. A RUN that started an sccache server adds the host’s TIME_WAIT on top of these: ~30 s with TcpTimedWaitDelay 30 s, ~120 s at the Windows default. That tail is #179 (windows-host-setup.md § C6), not the shim. On the current 5 min knob: ~10 s healthy (0.9 to 3.3 s measured on the fork build 04a9bdaa, 2026-10-14) · ~180 s the shim is env-configurable but the containerd Environment value is gone (a Stevedore update wiped it — silent stock 30 s; Set-ContainerdConfig.ps1 restores it) · ~450 s knob active, defect present · ~2841 s the 45 min constant build is back. Re-mitigate with Publish-ShimPatch.ps1 -ShimPath out\shim-builds\containerd-shim-runhcs-v1-fork-04a9bdaa.exe -ServiceEnvironment CONTAINERD_SHIM_RUNHCS_V1_TEARDOWN_TIMEOUT=5m (if the binary is lost: Install-NewHost.ps1 -SkipCni -SkipGcPolicy -SkipDufs rebuilds it from the pinned fork commit 04a9bdaa and deploys it with the 5m knob; by hand: fork branch feature/configurable-teardown-timeout, go build .\cmd\containerd-shim-runhcs-v1). Confirm a suspected hang cheaply with docker logs/docker top on the stuck container — output present + no process = lost notification, not a hung command. And never conclude “wedge” from a killed probe: give any timing probe ≥15 min. Diagnose the rest with admin: Get-ComputeProcess, fltmc instances -f bindflt (the orphaned-instance signature from hcsshim-teardown-timeout/ISSUE.md), ctr -n buildkit tasks ls, the Hyper-V-Compute logs. Do not judge a step by its console silence and kill buildctl — that manufactures the 0xb7 debris two rows up; a 240 s timeout is what made this look like a hard wedge for a day. The practical lever is the timeout, and 45 min was never the requirement: the measured legitimate worst-case teardown on this host is 117 s (hcsshim-teardown-timeout/ISSUE.md, OpenCV), so 2700 s is 23× the number it was raised to cover. Deploying the upstream-env shim variant — Publish-ShimPatch.ps1 -ShimPath <fork build> -ServiceEnvironment CONTAINERD_SHIM_RUNHCS_V1_TEARDOWN_TIMEOUT=5m (a later retune is Set-ContainerdConfig.ps1 -TeardownTimeout <duration> — no rebuild, no redeploy) — keeps ~2.5× headroom over the real worst case while capping the pathological case at 5 min, taking a RUN step from ~47 min to ~7 min without rebuilding the shim for every retune. The value is a Go duration string and a bare number is not one: =300 fails time.ParseDuration, and the patch treats an unparseable value as unset, so it would silently give you the stock 30 s. The task-close timeout is then derived automatically to 2*5m+30s. Force-terminating at the cap is known to be safe here: docker stop on a hung container returns in 91 s and the layer still exports (exporting layers 1.7s, unpacking 5.2s). Still an owner decision — it trades against the ExportLayer 0x3 corruption the 45 min was raised to prevent. Do not conclude the patch itself is obsolete: checked 2026-09-01, 2026-09-22 and 2026-10-14, upstream main (56195bbd, 0e1f18b7, bd97a054) still hardcodes every 30 s constant, hcsshim#2855 sits unanswered, and Windows-Containers#547 is closed without a fix — dropping to stock re-opens the 0x3 A/B of 2026-08-06 the moment the host is healthy again. Build the shim from the fork branch (Kataglyphis/hcsshim@feature/configurable-teardown-timeout — owner directive; code-identical to the in-tree patch, current-main base incl. the e6580439 leaked-layer-reader deadlock fix), not from the in-tree patch file.

hcsshim::ActivateLayer failed (0x20) during build#

Symptom. hcsshim::ActivateLayer failed (0x20) during build

Cause. Windows Defender scanning new layer files + containerd snapshot contention

Fix. Exclude the container stores from Windows Defender: windows\scripts\host\Sync-DefenderExclusions.ps1 (elevated) applies the full set, C:\ProgramData\containerd and C:\ProgramData\nerdctl among them.

ActivateLayer 0x20 "file used by another process" on commit#

Symptom. hcsshim::ActivateLayer failed ... 0x20 "file used by another process" on commit

Cause. --isolation process was used for a docker build — it cannot commit layers on this host

Fix. Never pass --isolation process to a docker build. Build through windows/Build-Buildkit.ps1: buildctl commits process-isolated layers, so every RUN gets all host CPUs (the docker run --cpu-count N + docker commit path went with build.ps1 on 2026-08-31). Not Defender/Search/SysMain (all ruled out).

ImportLayer ... (0xb7) "already exists" — deterministic, burns the retry budget#

Symptom. BK lane: failed to commit … during finalize: failed to reimport snapshot: hcsshim::ImportLayer failed in Win32: cannot create a file when that file already exists (0xb7), DETERMINISTIC — identical snapshot IDs on every attempt, burns the driver’s whole retry budget

Cause. A half-committed snapshot is in the way. Prime cause, measured 2026-08-07: killing buildctl mid-finalize leaves exactly this debris (a chain was aborted deliberately at 23 GB free to escape the disk danger band, and the next run died three times on the same IDs). prune does NOT clear it — it is not a reclaimable BK cache record (495 MB returned, nothing relevant); the transient-retry engine cannot help either, because the failure is deterministic, not a flake.

Fix. -NoCache on the affected stage only — e.g. .\windows\Build-Buildkit.ps1 -Gpu -Stages sdk -NoCache. Re-running the RUN yields a NEW layer digest (its output is not bit-identical), hence fresh chain IDs downstream, and the poisoned snapshot is simply no longer in the path. Verified 2026-08-07: the stage that had failed 3× exported cleanly, Done in 00:17:10. Prefer this over a CACHE-BUST comment edited into the RUN’s script (the old technique): same effect, costs one stage re-run, leaves NO trace in the source. Only reach for a source-level cache-bust when the debris sits in a layer you cannot isolate with -Stages. Corollary: prefer letting a doomed solve fail cleanly over killing it — a clean finalize failure leaves no debris, a kill does.

ImportLayer ... (0xb7) on the SAME chain-IDs across retries#

Symptom. BK lane: hcsshim::ImportLayer failed ... (0xb7) "already exists" on the SAME chain-IDs across retries

Cause. Persistent snapshotter debris from an earlier low-disk finalize failure — NOT transient, buildctl prune cannot reach it (0B reclaimable)

Fix. Non-admin sidestep: cache-bust the layer above (any content change to the COPY’d/mounted file → new chain-IDs; live examples in the 2026-08-06 headers of Install-ScoopTools.ps1 and Install-Vcpkg.ps1, the latter this exact 0xb7 case). Admin fix: prune/GC under the active gcpolicy.

failed to reimport snapshot / failed to write compressed diff — the hcs-temp flake family#

Symptom. BK lane: failed to reimport snapshot / failed to write compressed diff at finalize/export

Cause. hcs-temp flake family (2026-08-05): realtime scanner racing C:\WINDOWS\SystemTemp\hcs* scratch, and/or low disk (<~25 GB free makes hcsshim “weird” before disk-full)

Fix. Auto-retried by the BK driver’s transient pattern. Root remedies (applied 2026-08-05): Defender exclusions for buildkitd/containerd + their ProgramData dirs; keep ≥40 GB free; gcpolicy active. ALWAYS check free disk first — disk-full mimics the same message.

Finalize dies unknown stream ID 9 on a Windows Update .msu#

Symptom. BK lane: a RUN finishes, then finalize dies failed to reimport snapshot: Files/Windows/SoftwareDistribution/Download/<id>/Windows11.0-KB…-x64.msu: unknown stream ID 9, byte-identical on every retry (same snapshot IDs — the RUN result is cached, only the finalize re-runs)

Cause. Windows Update ran INSIDE the build container (servercore ships wuauserv + UsoSvc, trigger-started, and the container has network) and dropped an .msu into the update spool during the RUN; BuildKit’s Windows layer writer cannot carry that file’s alternate stream. Measured 2026-08-25 on the amd64 media-core-onnx stage (150 s RUN, KB5120233)

Fix. Prevention, not cleanup: Disable-ContainerWindowsUpdate (WindowsSourceBuild.Common.psm1) stops + disables both services and sets NoAutoUpdate=1 as the first step of every build script (Initialize-SourceBuildEnvironment) and reports the spool count — an entry inherited from the parent image (1 item at RUN start on every media stage) is harmless, only a file written during the RUN lands in the diff. Nothing under C:\Windows is deleted by any script (protected-root rule). A layer that already carries a download is fixed by re-running its RUN with the guard in place (any module edit re-keys it); retrying the same cached RUN cannot help.

exporting layers prints nothing for 20+ minutes#

Symptom. BK lane: exporting layers (or the following unpacking to …) prints NOTHING for 20+ minutes after a heavy amd64 layer (the LiteRT-LM Bazel output is the measured case) — or the merge fan-in COPY --from=media-core C:\runtime C:\runtime sits silent for 10–15 minutes on a step that took 5 s the run before — and every daemon — buildkitd, containerd, vmcompute, wcifs — sits at 0.00 s CPU, no shim/container process is left, and buildctl debug workers still answers in <1 s

Cause. Not a hang — a slow, mostly kernel-side layer export/unpack. Measured 2026-08-24/25: exporting layers 1216.3s done after 20 idle-looking minutes, then unpacking … 1277.0s done, and the stage went OK at 58:08 (17 min of actual build); arm64 run 12 (2026-08-25): the fan-in COPY finished DONE 847.4s after 14 idle-looking minutes (run 11: 5.1 s), no eventlog entry, nothing written under the buildkitd root

Fix. Wait. Killing buildctl here is exactly the “mid-finalize” kill that manufactures the 0xb7 debris two rows up and costs a -NoCache re-run. Bound the wait (a watcher on the log’s mtime) instead of judging by CPU: the export shows no user-mode activity by design.

Every new RUN sits silent while C: gains hundreds of GB#

Symptom. BK lane: a RUN header prints and nothing follows, for every step and every solve, even a trivial probe. No containerd-shim-runhcs-v1 process exists, and ctr -n buildkit containers ls lists the waiting containers while tasks ls is empty. containerd does thousands of small file operations per second at little CPU. Free space on C: climbs by GB per minute, and one rm-<n> directory under C:\ProgramData\containerd\root\io.containerd.snapshotter.v1.windows\snapshots has a current LastWriteTime.

Cause. containerd is deleting snapshots, and a new container waits for it. A new container needs a fresh snapshot, and while the sweep ran none was created (the rm-<n> directory is the one in flight). Measured 2026-09-29: containerd deleted about 440 GB, most likely the 27-28 Sep rocm runs’ cache after buildkit’s [history] retention (48 h, 10 entries) let go of it. New RUNs stalled from about 20:37 to 21:31, and buildctl du ended at 24 GB. Nothing failed: the waiting RUNs started on their own the second the sweep ended.

Fix. Wait, and do not kill buildctl (the 0xb7 row above). Watch C:’s free space: while it grows, the sweep is running, and the RUN starts when it stops. Afterwards the next chain solve is colder, because the sweep took the unpinned cache of earlier runs with it (tagged bk-* images are kept).

A stage fails instantly with exit code: 1 and zero container output#

Symptom. BK lane: a stage fails instantly with exit code: 1 and ZERO container output — no script banner, no stderr, deterministic across retries

Cause. Two solves racing on the same freshly-invalidated ancestor stage. Measured 2026-08-07: a second Build-Buildkit.ps1 was started while the main chain ran, right after a change to the common stage invalidated it for BOTH. Each solve tried to build the same new snapshot chain; one died before its process ever started, hence no output. NOT a script bug — a probe running the identical mounts, module import and Initialize-SourceBuildScript against the same base passed cleanly.

Fix. Do not run a second solve that shares an ancestor stage you just invalidated. -ConcurrentAux is safe because its two branches sit on an ALREADY-BUILT common ancestor. Wait for the running chain, then start the second build. If you must parallelise, first build the shared ancestor once on its own.

A Python module imports nowhere though its distribution is installed#

Symptom. In the :winamd64 and :winarm64 images of 2026-10-04, C:\temp\cpython\PCbuild\amd64\python.exe -c "import Cython" fails with ModuleNotFoundError: No module named 'Cython.Shadow', while pip list shows Cython 3.3.0. Cython\ holds shadow.py, utils.py and stringiotree.py where cython-3.3.0.dist-info\record names Shadow.py, Utils.py and StringIOTree.py. A pip install --force-reinstall cython in a container then leaves a lowercase cython\ directory, and the import still fails.

Cause. A Windows container layer that replaces a file a lower layer holds stores the name lowercased (invariant), and the media fan-in replaced thousands. It COPYed media-core’s whole site-packages and then media-tvm’s. Both branches install the same Cython, numpy, pip, setuptools, wheel and packaging into the toolchain’s CPython, so the second COPY lowercased every file of those six distributions, 451 RECORD entries, and the first COPY the toolchain’s README.txt. Distributions only one branch carries (av, onnxruntime, tvm, iree) kept their names, and so did meson, which built installs later. Measured 2026-10-07 on the local stage images: bk-windows-media-core and bk-windows-media-tvm each import Cython.Shadow, and bk-windows-media is the first image that cannot. NTFS opens either spelling, so pip, importlib.metadata and every DLL load still work. CPython’s import compares the directory listing exactly, and Cython is the one distribution of the six with CamelCase module files. The force-reinstall hits the same trap: pip moves the old Cython\ away and creates it again in the same layer.

Fix. Since 2026-10-07 the fan-in merges both trees in one RUN from bind mounts, never by COPY (Merge-SitePackageTree in WindowsSitePackages.Common.psm1). robocopy overwrites in place, so names survive. The merge stops when the branches carry one distribution at two versions, or when any RECORD entry is spelled otherwise on disk afterwards. Both branches install cython==PY_CYTHON_VERSION. The smoke gate’s section 2 imports Cython.Shadow and runs the same RECORD check over the base interpreter. In an image built before the fix, pip uninstall -y cython in one RUN and pip install cython==3.3.0 in the next restores it: a delete and a create in different layers keep the name. A venv never had the fault, since its files are new.

A bind mount of a branch image lists its files but opens none#

Symptom. The media fan-in’s site-packages merge fails with robocopy ERROR 3 (0x00000003) … The system cannot find the path specified for every entry of C:\bkmnt\site-packages\media-tvm, files and directories alike, while the media-core tree beside it merges. The first attempt after the branch image was unpacked may fail earlier, with failed to activate layer …: hcsshim::ActivateLayer failed in Win32: The process cannot access the file because it is being used by another process. (0x20), and the driver’s retry then reaches the robocopy failure.

Cause. Not established. A RUN that bind-mounts the media-tvm image sees its listing (49 entries) but cannot open cython.py. Mounted alone or beside media-core, the result is the same. The same files open in a RUN on that image, and through a bind of a stage that adds one empty RUN layer on top. Measured 2026-10-09 with buildctl probes, after a drastic buildctl prune and a full chain rebuild. It held across a -NoCacheStage media-tvm rebuild (a new top layer) and a restart of containerd and buildkitd. The same merge had passed in three solves on 2026-10-07/08.

Fix. The merge binds media-core-view and media-tvm-view, two stages that add RUN cmd /c exit 0 on top of each branch image (Dockerfile.media-merge-builder). The COPY --from= steps keep reading the images themselves, which works. The view layer never reaches the image, since only the merged files are kept.

Windows: container networking#

Could not resolve host: github.com seconds into the first downloading RUN#

Symptom. BK lane, seconds into the first downloading RUN: Could not resolve host: github.com / git clone failed (exit 128)

Cause. The CNI .conf is missing — buildkitd then gives the container NO NETWORK ADAPTER AT ALL (not a DNS fault). Confirm in 30 s with a probe RUN: ipconfig prints nothing and a raw TCP connect to a literal IP fails “unreachable network”; the containerd debug log shows the HcsCreateComputeSystem spec with no networking block. Usual cause: someone “converted” 0-containerd-nat.conf → .conflist to fix nerdctl (2026-08-07, cost a launched chain). The subnet-drift guard does NOT catch this and stays green.

Fix. Restore it (admin): pwsh -File windows\scripts\host\Set-ContainerdConfig.ps1 derives the .conf from the .conflist (never hand-edit a copy — invariant), then Restart-Service buildkitd -Force. Keep BOTH files — buildkitd needs .conf, nerdctl needs .conflist. Build-Buildkit.ps1 now fail-fasts (Get-CniConfFormIssue) and Test-HostSetup.ps1 FAILs on a missing .conf.

The remote name could not be resolved — CNI nat subnet drift#

Symptom. BK RUN steps: The remote name could not be resolved (and even direct-IP queries time out)

Cause. CNI nat subnet drift: dockerd restarts recreate the nat HNS network on a new subnet; the static CNI conf then hands out IPs whose gateway doesn’t exist

Fix. Update ipam.subnet/GW in the authored C:\Program Files\containerd\cni\conf\0-containerd-nat.conflist to match the live vEthernet (nat) adapter (ipconfig), re-derive the .conf with windows\scripts\host\Set-ContainerdConfig.ps1, then Restart-Service buildkitd -Force (admin). Build-Buildkit.ps1’s preflight guard (Get-CniNatSubnetDrift) detects this and prints the fix.

nerdctl DNS failure in build#

Symptom. nerdctl DNS failure in build

Cause. BuildKit container can’t resolve hostnames on Windows without a CNI nat CONFIG (--dns and --network host unsupported)

Fix. Fixed 2026-08-03: install 0-containerd-nat.conf into C:\Program Files\containerd\cni\conf\ (nat.exe was already in …\cni\bin) — buildkitd RUN steps then have full NAT+DNS. Today both forms are installed (the .conflist authored, the .conf derived by Set-ContainerdConfig.ps1). There is no docker.exe fallback: the classic driver went on 2026-08-31 and the Dockerfiles need BuildKit.

Windows: buildkitd and the store#

A BK compile step freezes silently after [output clipped, log limit 2MiB reached]#

Symptom. BK compile step freezes silently minutes in (0 % CPU, zombie ninja, no log growth) right after [output clipped, log limit 2MiB reached]

Cause. buildkitd step-log clip deadlock (Windows buildkitd v0.32): after the 2 MiB clip the stdio pipe stops being drained; every process blocks on its next write. ONNX’s warning flood hits the clip in ~3 min

Fix. Set service env BUILDKIT_STEP_LOG_MAX_SIZE=-1 + BUILDKIT_STEP_LOG_MAX_SPEED=-1 on buildkitd (registry MultiString Environment), Restart-Service buildkitd -Force. See docs/windows-build-lanes.md § Getting it going, step 4. Build-Buildkit.ps1 refuses to start without it (Assert-BuildkitdStepLogEnv; -SkipStepLogGate for one launch), and Install-NewHost.ps1 sets it.

buildctl prune returns Total: 0B no matter what you pass#

Symptom. buildctl prune returns Total: 0B no matter what you pass

Cause. CHECK du -v FOR Shared: true FIRST — that is almost always the answer. Shared records are pinned by containerd IMAGE TAGS and NO prune flag can take them; prune only ever gets the Private slice, and Reclaimable describes the LEASE state, not what prune will hand back. reservedSpace is a red herring for this symptom. Otherwise: refs pinned by BUILD HISTORY (every record incl. failed attempts pins its refs indefinitely) and/or by named bk-* image generations

Fix. In order: (1) buildctl du -v and look for Shared: true — those records are pinned by containerd image tags and no prune flag can take them, so the lever is an admin nerdctl --namespace buildkit rmi of dead stage tags. (2) Non-admin and safe mid-build: buildctl prune --free-storage <MB> — a minimum-free target, not an amount to delete. (3) prune-histories second, never first. (4) Look for a superseded lineage, the big hidden reclaim. The full decision procedure with every measurement: windows-build-lanes.md.

buildctl local export dies file already closed on a Windows rootfs#

Symptom. buildctl local export of a Windows image dies error from receiver: write ...\Boot\Fonts\<font>.ttf: file already closed (nondeterministic file; every layer had already committed)

Cause. The type=local exporter cannot receive a full Windows rootfs client-side — NOT a host/commit defect (measured 2026-08-10 on a host whose type=image,...,unpack=true export of the same solve was green)

Fix. Export type=image (what Build-Buildkit.ps1 and, since 2026-08-10, Test-BuildCopy.ps1 do) or the tar-stream type=docker,dest=<file> (-FinalTar); never judge host health from a type=local export of a Windows image.

Windows: Stevedore and the docker service#

cannot access containerd socket ... Zugriff verweigert (non-admin)#

Symptom. nerdctl ...: cannot access containerd socket ... Zugriff verweigert (non-admin)

Cause. containerd’s pipe is admin-only (its service lacks --group docker-users, unlike dockerd/buildkitd)

Fix. Use docker.exe (dockerd pipe) or buildctl (buildkitd pipe) — both are docker-users-accessible. nerdctl needs an elevated shell.

runtime "com.docker.hcsshim.v1" binary not installed#

Symptom. Stevedore docker build: runtime "com.docker.hcsshim.v1" binary not installed

Cause. Service default runtime uses hcsshim-v1 shim which isn’t shipped

Fix. Change to runhcs-v1: sc config stevedore binPath="..." --default-runtime=io.containerd.runhcs.v1" (see docs/windows-stevedore-and-docker.md § Fix 2)

failed to create TTRPC connection#

Symptom. Stevedore docker build: failed to create TTRPC connection

Cause. Shim binary mismatch (runhcs copied as hcsshim)

Fix. Remove the bad shim copy: del "C:\Program Files\Stevedore\bin\containerd-shim-hcsshim-v1.exe". Apply Fix 2 instead.

Stevedore service won’t start (1053 timeout)#

Symptom. Stevedore service won’t start (1053 timeout)

Cause. Windows Defender blocking dockerd.exe OR stale daemon.json from Docker Desktop

Fix. Add-MpPreference -ExclusionProcess "dockerd.exe" AND delete C:\ProgramData\docker\config\daemon.json

error getting credentials - err: exit status 1#

Symptom. error getting credentials - err: exit status 1

Cause. wincred credential helper fails because dockerd runs as SYSTEM without interactive session

Fix. OK to ignore for public images (MCR, GitHub). Use nerdctl pull instead for images that need auth, or set "credsStore":"" in docker config.

Windows: build content and toolchain#

Windows media build crawls; Building with ninja -j2#

Symptom. Windows media build crawls; Building with ninja -j2 in the stage log (out\windows-build-logs\bk-<runid>-Dockerfile.media-builder-media-core-*.log)

Cause. -j2 is the FLOOR of Get-BuildJobCount (max(2, min(cores, MEMORY_LIMIT_GB/4))), so the container resolved a near-zero memory budget — BK RUN steps are process-isolated with all host CPUs, so this is memory-bound, not a CPU cap. (Historical: on the classic lane the same line meant media-core had fallen back to a 2-CPU docker build instead of Invoke-RunCommitStage; that driver went on 2026-08-31.)

Fix. Check the driver’s preseed: memory-limit-gb=<N> published line and that the stage carries SCCACHE_WEBDAV_ENDPOINT — #51 publishes the budget to the WebDAV rather than as an ARG/ENV (both are cache keys), and a container that cannot read it falls back to CIM host RAM. Worked numbers: windows-build-resources.md § Media fan-out and memory budgeting.

Rust smoke test: “rustup could not choose a version of cargo/rustc”#

Symptom. Rust smoke test fails: “rustup could not choose a version of cargo/rustc”

Cause. A toolchain-less rustup (proxy shims in CARGO_BIN that resolve no toolchain) — e.g. rustup-init --default-toolchain none, or an image from before the Cargokit fix

Fix. rustup WITH a default toolchain (RUST_VERSION, pinned since 2026-10-05) IS the sole provider (Install-RustToolchain.ps1); CARGO_BIN on the rustup path is by design. Fix with rustup default $env:RUST_VERSION; never add a second provider (no scoop rust) (windows-build-invariants.md).

A GitLab download “succeeds” with HTTP 200 but is a few KB#

Symptom. A download from gitlab.freedesktop.org / code.videolan.org “succeeds” (HTTP 200) but is a few KB and extraction fails — e.g. GStreamer wraps “downloaded but extraction into X failed”

Cause. Anubis anti-scraper: browser User-Agents without JS get an HTML challenge page as a 200 (the shared Invoke-DownloadWithRetry sends a browser UA). Second variant: .git left in a GitLab /-/archive/ URL serves HTML even to curl. Both burned a merge run on 2026-08-17.

Fix. Both halves live in Invoke-GstWrapProvisioning (in WindowsMeson.Common.psm1 since 2026-08-31 — the merge-lane leaf mounted by Dockerfile.media-merge-builder only, so editing it costs the GStreamer layer and not the whole media chain): it fetches via Invoke-WrapDownload (curl-native UA + gzip/bzip2 magic-byte check) and strips .git from GitLab archive URLs in both branches. Diagnosis in 10 s: read the first bytes — <!doctype html> = challenge page, not a corrupt archive.

TVM: llvm-config.exe not found on PATH#

Symptom. tvm stage: TVM: llvm-config.exe not found on PATH (#47 gate)

Cause. Scoop LLVM never ships llvm-config or dev libs — TVM was silently USE_LLVM=OFF (no CPU codegen) until 2026-08-17. NOT a broken PATH.

Fix. The self-heal in Build-TvmFromSource.ps1 builds a pinned minimal LLVM from source (windows-build-invariants.md). On amd64 TVM links the patched toolchain’s llvm-config, so the heal runs only under -StockLlvm and on the rocm TVM_ROCM=1 spike. If the gate throws, check the heal’s download/SHA pin for the current LLVM_WINDOWS_VERSION — do NOT fall back to the official /MT dev tarball or USE_LLVM=OFF.

TVM ROCm spike: C:\llvm-patched\bin\llvm-config.exe has no AMDGPU target#

Symptom. A -Variant rocm run, media-tvm RUN: TVM ROCm spike: C:\llvm-patched\bin\llvm-config.exe has no AMDGPU target (targets-built: 'AArch64 X86') -- the rocm codegen cannot emit hsaco, thrown before TVM configures. IREE, in the same RUN, never runs.

Cause. Until 2026-09-23 TVM built its own AMDGPU-capable LLVM only when PATH had no llvm-config, and on amd64 the patched toolchain LLVM (AArch64;X86) is always on PATH.

Fix. Fixed in Build-TvmFromSource.ps1 (Get-TvmLlvmChoice): on the spike a PATH llvm-config without AMDGPU counts as absent, and TVM builds a minimal LLVM with AMDGPU (windows-rocm.md § IREE and TVM). If the message names C:\llvm-patched, the choice was bypassed; if it names C:\temp\llvm-dev\install\bin\llvm-config.exe, the minimal build ignored its target list. -NoRocmSpikes is the workaround.

TVM: no member named 'matchIntrinsicSignature' in namespace 'llvm::Intrinsic'#

Symptom. amd64 TVM compiler build: codegen_llvm.cc(1119): error: no member named 'matchIntrinsicSignature' in namespace 'llvm::Intrinsic' plus 7 more errors on MatchIntrinsicTypes_*, then ninja: build stopped. The arm64 lane is unaffected (runtime-only, no compiler).

Cause. TVM 0.26’s codegen_llvm.cc uses llvm::Intrinsic::matchIntrinsicSignature / MatchIntrinsicTypes_* which were removed or renamed in LLVM 23.1.0. The forced LLVM_WINDOWS_VERSION bump from 22.1.8 to 23.1.0 (scoop reshaped the artifact, #135) broke the TVM compiler’s LLVM API surface. The llvm_module.cc LLJITBuilderState::ObjectLinkingLayerCreator conversion also fails (API change).

Fix. Fixed (#134; rebuilt green 2026-09-02): TVM_COMMIT in versions.env pins TVM to upstream main (994e0216), which carries the TVM_LLVM_VERSION >= 230 guards, and Build-TvmFromSource.ps1 prefers it over TVM_REF. TVM v0.27.0 ships those guards (#20189), and TVM_COMMIT now pins that tag’s commit; it stays set, because a branch named v0.27.0 shadows the tag (linux-cross-builds.md). The arm64 lane needs no fix — it builds runtime-only and never compiles codegen_llvm.cc.

meson cross: Summary section 'Build environment' already have key 'host cpu', then Subproject "subprojects/glib" required but not found#

Symptom. arm64 GStreamer merge (cross file + native file): glib-2.86.3/meson.build:2777:0: Exception: Summary section 'Build environment' already have key 'host cpu' right after glib’s build-machine configure reached its summary, then gstreamer| Subproject subprojects\glib-2.86.3 is buildable: NO (disabling), dozens of Dependency 'libpcre2-8' is required but not found re-tries, and finally libnice-0.1.23/meson.build:214:4: Exception: Subproject "subprojects/glib" required but not found → gst-libs/gst/webrtc/nice/meson.build:16:14: ERROR: Subproject "subprojects/libnice" required but not found (arm64 run 25, 2026-08-26). The host glib configured fine minutes earlier.

Cause. Three meson bugs around “build-only” subprojects (1.12.0 and master, checked 2026-08-26). Nothing in the monorepo asks for a build-machine glib; meson’s gnome module does (mkenums_simple → find_tool → dependency('glib-2.0', native: true, required: false)), so under forcefallback glib’s meson.build runs a second time for the build machine. (1) Interpreter.summary is keyed by NAME and shared across interpreters → the second run throws at its summary({'host cpu': …}). (2) do_subproject’s failure paths call disabled_subproject(subp_name, exception=e) without for_machine (default HOST) → the failed build-machine holder overwrites the healthy host glib holder — that is the poison: libnice’s anonymous dependency('', fallback: ['glib', 'libglib_dep']) reaches glib by NAME and inherits the failure (the gio-2.0 lookup survives through the override table), and every later native: true request re-runs the whole failing configure because the BUILD key never got the disabled entry. (3) configure_file writes to the unprefixed self.subdir while targets/include dirs of a build-only subproject live under build.<subdir> → the build machine’s glibconfig.h/config.h/fficonfig.h overwrite the host’s, and a build-machine compile cannot find them (run 27). Not a toolchain problem — the build-machine sanity checks link x64 via /vctoolsdir+/winsdkdir and its libffi assembles with the x64 cl/ml64 named in the native file (backlog #128 runs 23–27 for the earlier costumes of this chain).

Fix. Invoke-MesonBuildSubprojectPatch (in WindowsMeson.Common.psm1 since #134, 2026-08-26 — a merge-lane leaf module mounted by Dockerfile.media-merge-builder only, so editing it costs the GStreamer layer and not the whole media chain; it lived inside Build-GstreamerFromSource.ps1 until then because the only module home was the shared buildmods six) patches the pip-installed mesonbuild/interpreter/interpreter.py before meson setup: for_machine=for_machine at both disabled_subproject failure sites, and the project prefix on configure_file’s output path and returned File. Bug (1) is deliberately left alone — a build-machine glib that configures is one that compiles (5772 targets) into more unprefixed paths, and nothing consumes it; with (2) fixed its failure is recorded under BUILD only, with (3) fixed it clobbers nothing, gnome’s find_tool falls through to the host override as designed, and libnice/webrtc configure against the host glib. Idempotent by marker; THROWS when a site count is off so a newer meson cannot silently skip it and resurface as the libnice error two hours later. Fixture test SourceBuild.MesonBuildSubprojectPatch.Tests.ps1; upstream draft out/upstream-issue-meson-summary-build-subproject.md (all three). If you see this on a meson newer than 1.12.0, check whether upstream fixed it before re-anchoring the regexes.

Windows base: scoop cannot install a pinned tool, 404 on the installer#

Symptom. Dockerfile.base dies in the scoop layer, ~14 minutes in (i.e. after the Visual Studio Build Tools layer), with The remote server returned an error: (404) Not Found. naming an installer URL, then Where-Object: Cannot bind argument to parameter 'AppName' because it is an empty string — scoop’s own follow-on error, not the cause. Both halves of this hit on 2026-08-26, twenty minutes apart.

Cause. The pin names a version whose WINDOWS artifact does not exist. Two distinct ways in:

  • The upstream reshaped the artifact. LLVM changed its Windows packaging at 23.1.0 from .exe to .msi, and the scoop main/llvm manifest followed within the hour. scoop install app@<ver> synthesises its URL from the CURRENT manifest’s autoupdate template, so the still-valid 22.1.8 pin began asking for LLVM-22.1.8-win64.msi, which was never published. Measured: LLVM-22.1.8-win64.exe 206 / .msi 404; LLVM-23.1.0-win64.exe 404 / .msi 206. A pin can go uninstallable without the pinned version changing.

  • The upstream versions per platform and the key is shared. LunarG publishes the Vulkan SDK per platform and Windows lags: https://vulkan.lunarg.com/sdk/latest.json read {"linux":"1.4.357.1","mac":"1.4.357.1","windows":"1.4.357.0"}. VULKAN_VERSION feeds the linux base/sdk AND the windows scoop layer, and the bump tool resolved it from latest/linux.txt.

Fix. Check the artifact, not the version number:

curl -sS -o NUL -w '%{http_code}' -L --max-time 30 -r 0-0 <installer-url>   # 206 = there, 404 = not
curl -s https://raw.githubusercontent.com/ScoopInstaller/Main/master/bucket/<app>.json | grep -A2 autoupdate

spec_vulkan in bump_versions.py now reads latest.json and pins the OLDEST platform this repo installs — a shared key can only carry a version that exists on every lane consuming it — and prints the disagreement. Build-Buildkit.ps1’s Vulkan preseed probes the installer URL before any solve and throws on 404 ONLY (403/5xx/timeouts stay fail-open, so a LunarG outage cannot block a build the container could still complete). Note the preseed had ALREADY warned “host download failed (exit 22)” and fallen through by design; a 404 is a wrong pin, not an outage.

AArch64 cross compile aborts with error: fixup value out of range#

Symptom. A cross TU aborts with error: fixup value out of range, or with error: value evaluated as <N> is out of range. — no source location, no fixup kind, no instruction. Appeared with clang-cl 23.1.0; 22.1.8 compiled the same tree. Observed in OpenCV’s CPU-dispatch TUs and in the bundled protobuf.

Cause — ONE defect, reaching two passes that both pick an encoding from an ESTIMATE of block offsets and are then contradicted by the assembler. Under async EH (/EHa, which OpenCV passes) AsmPrinter emits a NOP after an EH_LABEL whose next instruction can fault, but getInstSizeInBytes reports EH_LABEL as a zero-size meta-instruction — so every MIR-level block-size estimate is 4 bytes short per label. AArch64CompressJumpTables then picks a 1-byte jump-table entry for a span that does not fit (1), and BranchRelaxation leaves a branch that cannot reach its target (2).

This entry has now been wrong twice, in opposite directions; both corrections are kept because each wrong version was acted on. 2026-08-27 it said “one signature at two sites” with a shared cause — refuted, and the investigation split. Later the same day it said the two were unrelated, that (2) was llvm#202716 and that only a toolchain move to LLVM main would retire /Ob1 — also wrong. A census is recorded as settling it — 1,869 objects green with BOTH workarounds off, on pinned 23.1.0 plus only the two getInstSizeInBytes patches, a compiler containing no llvm#202716 — but it was run by hand and left no log, so it is a claim, not evidence. It was acted on anyway: #135 closed 2026-08-29 with the patched toolchain as the default and both workarounds removed (below). Full evidence in windows-backlog-archive-2026-08-26.md § #135.

  1. Jump-table entry width → value evaluated as <N> is out of range. AArch64CompressJumpTables selects 1-byte entries whenever span>>2 fits in 8 bits — a ceiling of 255×4 = 1020 bytes. Every N measured here sits JUST past it: 256 (= 1024 B, four bytes over), then 258, 259, 260, 262, 272, 281, 284. Offender class: the bundled libprotobuf (descriptor.cc, generated_message_reflection.cc, wire_format.cc), which is known-fragile on windows-arm64 — but read that citation carefully (protobuf#24758 is a LOOSER match than it looks: Ruby/upb, not descriptor.cc, and its symptom is Failed to evaluate function length in SEH unwind info, i.e. llvm#47432 — the same bug this page already warns about under -align-all-*, and NOT the jump-table overflow). It was closed as an LLVM bug with nothing to fix in protobuf, which is the useful precedent: the offender library is the trigger, not the defect. Root cause, FOUND 2026-08-28 — FIXED by the patched toolchain (llvm#219275 + #219276, BUILD_PATCHED_LLVM=1, now the default). EH_LABEL under /EHa emits a 4-byte nop counted as zero by getInstSizeInBytes; the pass is sound given correct instruction sizes. The workarounds (+force-32bit-jump-tables and per-TU /Ob1) have been REMOVED from Build-OpencvFromSource.ps1. The stock scoop clang-cl still has the bug — use -StockLlvm only for patch debugging.

  2. Branch relaxation → fixup value out of range. In median_blur.dispatch.cpp, /O2 collapses the whole baseline median filter into ONE function — cv::cpu_baseline::medianBlur, 8,465 instructions ≈ 33,860 bytes — and inside it

    tbnz  w9, #31, .LBB546_847

    has to reach a block ~32,916 bytes away (counted from the emitted listing, instructions × 4). tbz/tbnz carry a 14-bit displacement: ±32,768 bytes. It misses by roughly 150 bytes — a hair, not an order of magnitude. LLVM’s BranchRelaxation pass exists to catch exactly this and rewrite the branch; it did not, because its layout estimate came out short. Root cause: the same EH_LABEL undercount as (1) — ~150 bytes is 37 labels at 4 bytes each. Retired by the same two patches on pinned 23.1.0; measured 2026-08-28, /Ob1 off, this TU compiles clean. Superseded claim, kept because it was acted on: this once read “root cause FOUND, fixed upstream by c6e184686cd7, only a toolchain move to main retires /Ob1”. The census compiler contains no such commit, so #202716 is not the cause here. What the earlier branch-relax-tbz.mir revert actually showed was that the commit is load-bearing for upstream’s own test — never evidence about median_blur.dispatch.cpp.

Fix — HISTORICAL (both removed 2026-08-29; the patched toolchain above is the fix): two settings, one per site, both cross-lane only.

  • (1) -Xclang -target-feature -Xclang +force-32bit-jump-tables, whole build. This disables the compression pass — if (ST.force32BitJumpTables() && !MF->getFunction().hasMinSize()) return false; (AArch64CompressJumpTables.cpp, 23.1.0) — so every table keeps 4-byte entries: .word .LBB0_2-.Ltmp0 / ldrsw, ±2 GB, instead of .hword (.LBB0_2-.LBB0_2)>>2 / ldrh. Cost on a reproducer: 4522 → 4650 bytes of object, ~2.8 %, all of it jump-table DATA; full /O2 retained.

  • (2) /Ob1 on the two offending TUs — median_blur.dispatch.cpp and multiview_calibration.cpp — were appended to their build.ninja FLAGS lines by Build-OpencvFromSource.ps1. It does not lower the optimisation level: every kernel keeps /O2, vectorisation and unrolling. It stops the inliner from gluing file-static helpers into one oversized function; on median_blur the largest function drops 33,860 → 10,620 bytes, i.e. 3.1× headroom under the ceiling instead of a 148-byte miss. -mllvm -inline-threshold=100 and =25 do NOT achieve this (measured, both still fail): those helpers have a single call site and are inlined regardless of threshold.

    Get the list by census, not one rebuild at a time. NINJA_KEEP_GOING=1 (honoured by Invoke-NinjaBuildWithRetry) turns the stage into ninja -k 0, so one run compiles all 1,870 objects and reports EVERY offender instead of stopping at the first. That is how the list above was closed at two. It matters again only if the patched toolchain is dropped: the ceiling is a property of what the inliner produces, so a new offender is one source change away. (No Dockerfile declares NINJA_KEEP_GOING as an ARG, so -BuildArg NINJA_KEEP_GOING=1 is silently dropped.)

+force-32bit-jump-tables and -mllvm -aarch64-enable-compress-jump-tables=false are the same thing — byte-identical .asm from clang-cl 23.1.0, verified locally on 2026-08-27. An earlier version of this page claimed the feature “keeps the pass enabled with its adr check intact” while the -mllvm flag removes it, and used that difference to explain (2). Both halves were wrong. With the pass off, the adr that materialises a table base is self-relative (.Ltmp0: sits on the adr itself, displacement 0) and cannot go out of range. The target feature is preferred only because it is a supported spelling where -mllvm is a debug knob.

Do NOT reach for these — each one cost a run:

  • -fno-jump-tables for (2): no jump table is involved; it fails identically (measured).

  • -align-all-* to nudge an estimate: the padding makes the function length unevaluable for the Windows SEH unwind writer and clang-cl dies with Failed to evaluate function length in SEH unwind info (llvm#122707, a duplicate of llvm#47432).

  • /Od or /O1 to dodge a pass: they move the failure to the next TU (measured three times) and cost code quality on a lane whose point is a real build. A 148-byte miss flips on ANY perturbation, which is exactly why blunt flags keep appearing to “work”.

  • -max-jump-table-size: caps the entry COUNT while the ceiling is a BYTE SPAN. Accepted by the driver, no effect.

Diagnose it in minutes, not in build-hours. The message carries no location because it comes from the MC layer, after codegen. /FA makes clang-cl assemble its own listing, which puts a file:line and the offending instruction on the error:

clang-cl.exe --target=aarch64-pc-windows-msvc /O2 /c /FA /Fat.asm /Fot.obj t.cpp

One trap on 23.1.0: that listing does not round-trip. A catch funclet’s block address prints as add x0, x0, .LBB0_903 with the :lo12: specifier MISSING, so LLVM’s own assembler stops there with expected compatible register, symbol or integer in range [0, 4095] — before reaching the fixup you are chasing. Direct object emission is unaffected (it gets the specifier right), so repair the listing and assemble that instead:

(Get-Content t.asm) -replace '^\s*add\s+(x\d+),\s*(x\d+),\s*(\.L\S+)\s*$', "`tadd`t`$1, `$2, :lo12:`$3" | Set-Content t2.s
clang.exe --target=aarch64-pc-windows-msvc -c t2.s -o t2.obj    # the error now has a line number

That printing bug reproduces in 15 lines — one try/catch with a body big enough to push the continuation block past 4 KB — and deserves its own upstream report.

Work against a LOCAL toolchain, not the container: it turns 4-minute lane runs into 2-second experiments.

curl -sSfL -o llvm.tar.xz https://github.com/llvm/llvm-project/releases/download/llvmorg-<ver>/clang+llvm-<ver>-x86_64-pc-windows-msvc.tar.xz
tar -xf llvm.tar.xz

Then read the listing: .byte/.hword/.word says which jump-table width was chosen, adr vs adrp says how a base is reached, and the instruction count between a branch and its target label — ×4 for bytes — says whether relaxation failed and by how much.

Deciding whether a new stock compiler can replace the patched toolchain. Do not answer that from a lane run, and do not answer it from a synthetic reproducer — a 148-byte miss flips on any perturbation, so a hand-written test case proves nothing about these TUs. Invoke-LlvmAarch64Layout.ps1 is the A/B: it freezes the five real offenders as preprocessed .i files (-Capture, once per OpenCV bump) and then compiles them with the workaround OFF against any candidate clang-cl. Because a frozen .i has no #includes left, the run phase needs no OpenCV tree, no VsDevCmd and no container — only a compiler that can target aarch64-pc-windows-msvc.

It gates every verdict on its own control arms: the stock compiler must still reproduce the abort with the workaround off, and must still compile clean with it on. If either fails the run reports INVALID and exits 2, because a green candidate under a broken control is not evidence — it is a stale corpus. A FIXED verdict is necessary and not sufficient: the frozen set is a census taken at one commit, and the ceiling is a property of what the inliner produces, so dropping the patched toolchain (BUILD_PATCHED_LLVM=1) still requires a full arm64 OpenCV build over all ~1,870 objects. Its command-line surgery — in particular that both spellings of the jump-table workaround are stripped, so the “off” arm is genuinely off — is pinned by windows/scripts/tests/Diagnostics.Llvm135Repro.Tests.ps1 against the real ninja command line.

A source build produces UNPATCHED sources and says SKIP: ... (already applied)#

Symptom. Invoke-SourcePatch reports SKIP: <patch> (already applied) for every patch on a fresh source tree, the build succeeds, and the artefact behaves as though no patch was ever applied. No error anywhere.

Cause. Invoke-SourcePatch falls back to patch.exe when the source tree is not a git repo, and until 2026-08-27 it passed the patch file as a POSITIONAL argument. To GNU patch a bare path is the file to be patched, not the patch — so it read an empty patch from stdin, changed nothing and exited 0. The helper’s reverse-check reads exit 0 as “already applied” and skips. Measured against a file containing only the word placeholder: both the forward and the reverse dry-run returned 0 and printed nothing. With -i the same call returns 1 and prints Hunk #1 FAILED.

Only non-git trees were affected. OpenCV and opencv_contrib are git clones and take the git apply path, where the positional form is correct — which is why this sat undiscovered until the LLVM release tarball became the first non-git consumer.

Fix. Already applied: the three patch.exe scriptblocks in WindowsSourceBuild.Patches.psm1 now pass -i $PatchFile.

The transferable lesson. This was caught only because Build-LlvmFromSource.ps1 asserts on the PATCHED CONTENT afterwards (if ($text -notmatch 'eh-asynch') { throw ... }), not on the patch step’s exit code. A source build whose patch silently vanishing would matter should check the result, not the return value — otherwise you ship a compiler that reports the right version, passes the provenance gate, and still has the bug you patched out.

atlbase.h not found when building LLVM in the container#

Symptom. DIASupport.h(25): fatal error C1083: ... "atlbase.h" partway through an LLVM source build.

Cause. LLVM’s PDB/DIA support needs ATL, which the container’s VS Build Tools installation does not include.

Fix. -DLLVM_ENABLE_DIA_SDK=OFF (already in Build-LlvmFromSource.ps1). DIA only powers PDB symbolisation in the LLVM tools; the compiler itself needs none of it.

A build script dies with The term ... is not recognized, in the container only#

Symptom. A build script dies with FATAL ERROR: The term 'Resolve-BuildMachineMsvcTool' is not recognized as a name of a cmdlet, function, script file, or executable program. — well into a compile stage, on the build host only. The same script runs fine on a dev box.

Cause. The function exists and is exported by its owning module, but the script reaches that module INDIRECTLY through WindowsSourceBuild.Common’s re-export list, and the name was never added there. Module-internal use never needs an export entry; a direct script call does. On a dev box the whole modules directory is on disk and earlier imports pollute the session, so nothing fails. Export-ModuleMember also silently ignores names with no matching function, so the reverse mistake is equally quiet. This class has cost two incidents (#113/verify12, and #134 two hours into arm64 run 37).

Fix. Exporting is TWO edits: Export-ModuleMember in the owning module AND the re-export list in WindowsSourceBuild.Common.psm1. Modules.ScriptCallClosure.Tests.ps1 proves in a fresh pwsh — importing only what the script itself imports — that every module function a build script CALLS resolves; Modules.ReExport.Tests.ps1 checks the other direction. Neither replaces the other, and the check takes seconds where the build takes hours.

A consumer’s CMake says clang-cl “is not able to compile a simple test program”#

Symptom. A consumer’s Windows CI lane, building inside :winamd64, stops at CMake’s compiler check: The C compiler "clang-cl.exe" is not able to compile a simple test program. Each sccache client in the check’s log waits 10-12 s and exits 2; the sccache server log (if you can find it) says tcp connect error. The lane’s own output may show none of this: consumer wrappers ran $null = Invoke-ContainerBuild, which discarded the build’s stdout until 2026-09-23. Seen on AccelerANTgine and OmniAccelerANT (runs 35921977157, 35912798986).

Cause. The published image carried the build host’s SCCACHE_WEBDAV_ENDPOINT (a LAN address) in its ENV. The consumer’s wiring turns the sccache launchers on whenever sccache is on PATH; sccache 0.18 checks its storage when the server starts, cannot reach the endpoint from a runner, and exits, so every compile behind the launcher fails.

Fix. A hub at or after 2026-09-23 removes an unreachable endpoint before it wires the launchers (Clear-UnreachableSccacheEndpoint, one WARN), so bumping the hub pin is enough on an old image. Images built from that hub no longer carry the variable, and three gates refuse one that does: windows-build-resources.md § What the published image carries. By hand on an old hub: clear SCCACHE_WEBDAV_ENDPOINT in the container before the build.

CodeQL database create hangs for hours at near-zero CPU#

Symptom. A Windows -CodeQL run (Invoke-BuildCodeQL) sits in database create for hours with seconds of CPU. Since 2026-10-05 the phase watchdog kills it and names the phase (CodeQL database create did not finish within … min). Inside the container, a cargo metadata started by CodeQL’s Rust extractor, or a traced build step, waits beside an sccache.exe that has no arguments.

Cause. The image sets SCCACHE_IDLE_TIMEOUT=0, so sccache’s server never exits. A client started under the CodeQL tracer spawns the server, the server keeps the client’s output pipe open, and the caller waits for an end-of-file that never comes. Killing the server releases one call, and the next call starts a new server and blocks again (OmniAccelerANT, 2026-10-05). A cache hit would also hide the compile from the extractor.

Fix. Since 2026-10-05 Invoke-BuildCodeQL turns sccache off for the traced build: Disable-SccacheForTrace clears RUSTC_WRAPPER, CC_WRAPPER, CXX_WRAPPER and both CMake launcher variables and sets KATAGLYPHIS_NO_SCCACHE=1, which Initialize-BuildCacheEnvironment and Get-SccacheStatsText honour. A consumer that wires sccache itself must honour the marker too, including a preset’s COMPILER_CACHE=sccache (pass -DCOMPILER_CACHE=).

A Windows chain stops in seconds at [bk:publish-gate:bk-windows-…]#

Symptom. Right after its preflight, before the stage you expected, Build-Buildkit.ps1 fails in a stage labelled publish-gate:<tag> with LEAK [Process] SCCACHE_WEBDAV_ENDPOINT=… in the log and <tag> was not built by this run and failed the publish gate … Rebuild it.

Cause. The run inherits a parent image it did not build (its -Stages leaves out the stage that produces it), and that image predates the 2026-09-23 fix, so its config ENV still carries the endpoint. Every stage built on it would inherit it, up to the published image.

Fix. Put the producing stage in -Stages (after that fix: toolchain and everything after it) and start a fresh driver process; never resume a run that was started before the fix: windows-build-resources.md § An image this run did not build.

A declaration that masks its command’s exit status#

local x="$(cmd)" returns local’s status, not cmd’s. Under set -e the failure is invisible and x silently holds "".

Live example, fixed 2026-09-01 — _gst_monorepo_install (03-media/build/gstreamer/common/build-gstreamer-monorepo.sh):

local gst_stage="$(mktemp -d "/tmp/gst-stage.XXXXXX")"

/tmp is a tmpfs on every Dockerfile.media RUN, so a full tmpfs makes mktemp fail. With gst_stage empty the whole staging design inverts:

line

intended

with an empty value

meson install --destdir "${gst_stage}"

stage into a temp tree

--destdir '' → installs into the live root, running target post-install scripts there

[ -d "${gst_stage}${GSTREAMER_PREFIX}" ]

is anything staged?

[ -d /opt/gstreamer ] → always true

[ -d "${gst_stage}/usr/local" ]

as above

[ -d /usr/local ] → always true

rm -rf "${gst_stage}"

drop the staging tree

rm -rf '' → no-op

The sting: the only line reaching the build log is the same WARNING: GStreamer cross-install had errors a healthy cross run prints, so the broken run is indistinguishable in the log.

The gate. verify_masked_assignments.py (preflight slug masked-decls) fails on any NEW local/export/declare/readonly declaration containing a command substitution; the pre-existing sites (54 when frozen on 2026-09-01) are listed in masked-assignments.allow, keyed by file+variable so a site does not re-flag when something above it moves. Fixing one means deleting its line.

Two reasons shellcheck alone was not enough: SC2155 does not fire on local x="${y:-$(cmd)}" (verified), and lint-shell.sh gates at -S error, where a warning can never fail the build.

The fix shape:

local x
x="$(cmd)" || return 1

RV1-FREETYPE: riscv64 OpenCV freetype/harfbuzz#

RV1-FREETYPE — FIXED (2026-08-24); the coming rebuild is the final validator. History (2026-08-23 investigation, still true): riscv64 had no TARGET harfbuzz dev surface at configure time — pass-2 (FROM gstreamer, libfreetype-dev pre-satisfied) got only the RUNTIME libharfbuzz0b — so ocv_check_modules(HARFBUZZ harfbuzz) resolved a HOST harfbuzz (find_library fall-through under CMAKE_FIND_ROOT_PATH_MODE_LIBRARY=BOTH) → “file in wrong format” at link. The ports dev package stays banned (libharfbuzz-dev:riscv64 → Depends: libglib2.0-dev = RV1-GST-PC poison), so install-deps.sh now stages a PIC STATIC target harfbuzz (HB_HAVE_FREETYPE=ON, hb-ft glue included) at /usr/ with a cmake-generated absolute-prefix harfbuzz.pc whose freetype dep is promoted to Requires: pkg-config then emits -lharfbuzz -lfreetype, so HARFBUZZ_LIBRARIES becomes [libharfbuzz.a, target libfreetype.so] and the module link ( FREETYPE_LIBRARIES HARFBUZZ_LIBRARIES) still resolves the archive’s FT_* refs under -Wl,–no-undefined — a DSO BEFORE the archive alone does NOT (proven by a local link experiment 2026-08-24: objects+ft.so+hb.a fails; +ft.so after the archive links clean). Determinism, both passes:

  • our pkgconfig dir is prepended to PKG_CONFIG_PATH, which outranks PKG_CONFIG_LIBDIR — pass-2 puts /opt/gstreamer first in PKG_CONFIG_PATH and its meson-subproject harfbuzz may export a competing harfbuzz.pc; ours must win;

  • the pkgcfg_lib_* cache vars FindPkgConfig/ocv_check_modules resolve libraries through are pre-seeded with absolute TARGET paths, so no find_library ever runs, let alone falls through to a host lib (OCV-FF1’s determinism discipline, pinned by file path instead of -L ordering). Static harfbuzz keeps the runtime surface: the only new NEEDED is libfreetype.so.6, which validate-media-runtime’s so-package-map already resolves to libfreetype6 (and the gstreamer stack pulls it in on riscv64 today anyway). If install-deps could NOT stage the static harfbuzz, keep the module hard-OFF rather than let detection wander back to host libs — absence then still surfaces as the wheel smoke’s riscv64-only opencv-freetype warning, and after a green rebuild that warning’s “expected on riscv64” status is STALE (parity follow-up for the orchestrator).

A half-fetched dependency that every retry inherits#

ONNX Runtime CPU build failed after 3 attempts on media-riscv64, 2026-09-02, with the same CMake error each time:

add_subdirectory: The source directory
  .../_deps/dawn-src/third_party/spirv-headers/src
does not contain a CMakeLists.txt file.

Dawn’s dependency fetch was interrupted, leaving the directory present but EMPTY. _deps is in the image layer, not a cache mount, so a fresh run recovers — but the three retries inside one RUN all inherited the ruin, so the retry bought nothing. amd64 and arm64 passed the identical build; only the riscv64 run hit the transient failure.

30-build-native.sh now drops any *-src/third_party/*/src that carries no CMakeLists.txt before each attempt. A retry that inherits the state that broke the previous attempt is not a retry.

RVV changed what the optimizer can prove#

media-riscv64 failed on 2026-09-02, after the RVA23 switch, in vvdec:

thirdparty/simde/x86/ssse3.h:370: error: 'r_.simde__m128i_private::i8'
may be used uninitialized [-Werror=maybe-uninitialized]   → vvdec_x86_simd

SIMDe emulates x86 intrinsics on non-x86, so a riscv64 cross build really does compile that header. -Wmaybe-uninitialized is optimization-dependent, and enabling vector plus Zba/Zbb changes what the optimizer can prove — a diagnostic that did not fire before the ISA change now does, in upstream third-party code.

install-vvdec.sh waives it exactly where the same file already waives -Wno-error=unused-but-set-variable. The waiver is scoped to vvdec, not global: this is a false-positive class in one dependency, not a reason to stop treating warnings as errors elsewhere.

Expect more of this shape as the vector baseline lands. Waive per component, never tree-wide, and record why here.

webrtcbin’s action signals never reply; ffi_prep_cif returns FFI_BAD_TYPEDEF#

create-offer, get-stats and create-data-channel on webrtcbin do nothing, not even the closed error a webrtcbin in NULL state must answer with. Signals without arguments still work. GLib’s generic marshaller runs every signal with arguments through libffi, and ffi-7.dll exported its ffi_type_* descriptors as zeros: ffi_type_sint32 size=0 align=0 type=0, so ffi_prep_cif failed and g_cclosure_marshal_generic returned without calling the handler. The build log shows the cause as 60 lld-link: warning: duplicate symbol: ffi_type_* lines at Linking target subprojects/libffi/src/ffi-7.dll, a warning only because of /FORCE:MULTIPLE. Mechanism and fix: windows-builds.md § libffi’s type exports. Assert-LibffiTypeExport now fails the build on it.

dtlsconnection ... unexpected eof while reading at the first DTLS step#

Signalling and ICE complete, then both peers fail with ssl error: ...:error:0A000126:SSL routines::unexpected eof while reading:ssl\record\rec_layer_d1.c:254 from gst_dtls_enc_change_state, and the consumer reports Connection state failed. OpenSSL 4.0 takes the empty read GStreamer 1.29.2’s DTLS BIO returns as end of file: windows-builds.md § DTLS with OpenSSL 4. If it comes back after a GSTREAMER_VERSION bump, check that the build log still says the patch applied or that upstream carries it.

Contents