# Cross-build verification & failure-class catalog This document catalogs the classes of failure hit during the base→`:latest-cross` (now `:latest`) rebuild campaigns and maps each to the fast check that catches it *before* a multi-hour QEMU build. It is the reference for the pre-flight verification workflow (see "Pre-flight" below). The guiding principle: **every error we debugged interactively should become a check that fails in seconds, not after a 30–60 min emulated build.** > **Scope: this page is the LINUX cross lane.** Its failure classes come from the > `base→:latest-cross` campaigns and its checks assume QEMU, ELF and a runnable > target. The **Windows** arm64 cross lane is documented separately in > [`windows-cross-builds.md`](windows-cross-builds.md), and its verification story is > deliberately different in one decisive way: **nothing it produces can be executed at > all.** Windows x64 has no ARM64 emulation, so there is no Windows counterpart to the > "compile+link+RUN under qemu" half of class 4 below — only the static half. That lane's > equivalent of the per-arch ELF/machine check is `windows/scripts/build/Test-TargetArch.ps1` > (PE `Machine` field over the whole install prefix, with a minimum-inspected floor so a > lane that staged nothing cannot pass green), and its smoke gate runs the host-toolchain > sections against an arm64-specific floor column (76/20; the green 97/0/15 was measured > under 69/20, before sections 19 and 25 grew), reporting > only the payload-execution sections NOT APPLICABLE (since 2026-08-24 — before that the whole > gate was reported NOT APPLICABLE). ## Failure classes (from build history) | # | Class | Representative bug(s) | Fix commit(s) | Caught early by | |---|-------|----------------------|---------------|-----------------| | 1 | Script not COPY'd into a stage → sourced fn missing at runtime (`command not found`, exit 127) | `media_load_arch_flags` not found (03-media/core never COPY'd into Dockerfile.package) | `da41e19` | **sourced-scripts-present** static check (`verify_script_copy_coverage.py`) | | 2 | Relocated native GCC/G++ can't find `/usr/include` for source builds under QEMU — C *and* C++ (`#include_next`) | `string.h: No such file` (Pillow); ``→`stdlib.h: No such` (numpy) | `3c623fa`, `349e32b`, `dc93d11` | **compile smoke test** (C + C++ `#include_next` + `jpeglib.h`) in `validate-compilers.sh` | | 3 | Missing dev headers for QEMU source builds | `jpeglib.h` missing for Pillow (`libjpeg-dev`) | `3c623fa` | same compile smoke test (header presence probe) | | 4 | Cross toolchain artifact wrong-arch / not runnable on host | `/opt/llvm-target` clobbered by shared compiler; non-runnable `llvm-config`; missing target linker | `8e66c5f`, `fb634a3`, `b1dd72e`, `312a4d8` | **`validate-compilers.sh`** per-arch ELF/machine check (build-time) **+ compile+link+RUN under qemu** in `smoke-runtime-image.sh` (`bcbd19d`) | | 5 | venv/wheel install collision & bad seeding | apt numpy seeded into venv without dist-info → uv install `File exists` | `9f07334` | **torch-venv integrity** smoke (`smoke-torch-venv.sh`) | | 6 | Undefined/typo'd bash function or quoting bug | `tvm-detect` undefined; verify-parity venv quoting | (dedup passes) | **shellcheck gate** (`-S error`) in pre-commit | | 7 | Include-flag construction bugs | bare `-I -I -I` broke Abseil C++17 probe; missing pybind11/numpy include dirs | `ab3776b`, `5412ec4` | shellcheck + compile smoke test | | 8 | Dockerfile ARG / mirror / cache drift | apt not `-y`; cache-to 400; ARG≠versions.env | `bf49676`, `4f27634` | `verify-arg-consistency.sh`, `verify-ubuntu-mirror-consistency.sh` (exist) | ## Existing infrastructure (reuse, don't duplicate) - **Shared helper for class 2/3:** `01-core/common.sh` → `append_cross_idirafter ` already appends `-idirafter /usr/include{,/}` to `CPPFLAGS/CFLAGS/CXXFLAGS`. Used by `build-libcamera.sh`, `build-gstreamer-monorepo.sh`. `setup-torch-venv.sh` and `swap-native-gcc.sh` carry deliberate inline copies, because they run without `01-core/common.sh`; `verify-critical-fixes.sh` fix6 keeps them in sync. - **Compiler validation (class 4):** `06-packaging/validate-compilers.sh` emits `ARTIFACT COMPILER VERIFICATION PASSED for `; validates the `versions.env`-pinned GCC/Clang chain and per-arch ELF machine type. The versions are *not* baked into the script — it reads `GCC_VERSION` and `LLVM_RELEASE` from the environment, which the wrapper-smoke stage passes in as build ARGs (`Dockerfile.package`, the `wrapper-smoke` stage). The clang checks refuse an unset `LLVM_RELEASE` (`${LLVM_RELEASE:?…}`); the `${GCC_VERSION:-16.2.0}` fallback literals in `validate-compilers.sh` are dead in the build path. Extend here for the compile smoke test. - **Smoke framework:** `06-packaging/smoke-common.sh` (`pass`/`fail`/`FAILURES`); smoke tests are `06-packaging/smoke-.sh` and `source smoke-common.sh`. - **Static host verifiers wired into `linux/host-config/git-hooks/pre-commit`:** the hook runs the cheap subset of `preflight.sh` slugs listed in its `_FAST_SLUGS` — `arg-consistency` (`01-core/verify-arg-consistency.sh`) and `copy-coverage` (the sourced-scripts-present check) among them — plus `shellcheck` on the staged shell files. `critical-fixes` (`verify-critical-fixes.sh`) and `version-snapshot` (`sync_versions.py --check`) are not in that subset; `make preflight` runs them. ## Pre-flight Run **`linux/scripts/preflight.sh`** before `build-cross-chain.sh`. It runs every fast (no-build) check in seconds/minutes so whole failure classes are caught before a multi-hour QEMU rebuild. All checks run even if one fails; the script exits non-zero if any did. Since 2026-08-08 preflight also validates the stage graph itself (slug `stage-graph` — parent refs, dockerfile existence, tag resolution, cycles); previously that ran only at build kickoff. The script-tests slug now prints an assertion aggregate (` suites, assertions`, `run-tests.sh:31`) — a sudden drop in those numbers is the alarm it looks like: the harness fails suites that run zero assertions, and the aggregate makes shrinking coverage visible. The current figures are deliberately not restated here — they grow with every suite that lands, so a stale baseline in this doc would make a real collapse read as growth (`run-tests.sh:30` picks a deliberately absurd "24 suites, 3 assertions" as its own example for that reason). Read them with `bash linux/scripts/tests/run-tests.sh`. ### The `linux/scripts/tests` suites Conventions for the bash unit suites behind the `script-tests` slug. They exist so a decomposition or a flag rewrite is provably behaviour-preserving in seconds, on a lane where the alternative evidence is a multi-hour QEMU build. **Discovery is a glob, not a registry.** `run-tests.sh` runs every `linux/scripts/tests/test-*.sh` in its own `bash` process (nothing leaks between suites) and skips `test-harness.sh` by name. Adding a file adds a suite; there is nothing to register — and deleting one removes coverage quietly, which is exactly what the assertion aggregate above is for. **The harness is the whole API.** `source test-harness.sh`, then `t_case ` to label the assertions that follow, `t_assert_eq [msg]`, `t_assert_contains [msg]`, `t_assert_ok `, `t_assert_fails `, and `t_summary` as the last line. `t_summary` exits non-zero when an assertion failed **and when the suite ran none at all** — a gutted suite must not read as a pass. **A Git Bash host may skip what it cannot host; Linux may not** (CON61, 2026-10-05). Some suites need what Git Bash lacks: real symlinks, mode bits, a POSIX `python3`, `jq`. Such a suite says so with `t_skip_unless ` after sourcing the harness. On Git Bash that prints `SKIP [suite] this host lacks ` and exits 77, and `run-tests.sh` lists the suite as skipped. On Linux the same line fails the suite, so CI never runs less than it claims. When only one case needs it, `t_needs ` right after its `t_case` waives that case's failures, and `t_summary` prints how many it waived. The probes are `t_posix_symlinks`, `t_posix_modes`, `t_posix_python` and `t_is_elf`. `test-host-skips.sh` holds both verdicts to that split. A Windows-only failure in the code under test is a bug, not a skip. Two gates printed `linux\scripts\…` there, and they print POSIX paths now. **Scripts that end in `main "$@"` cannot be sourced.** Several subjects are executables, not libraries. The idiom is to cut the region under test out with `awk` and `eval` it: `test-vulkan-target-decomposition.sh` extracts each `_vulkan_target_*` helper by name, `test-iree-wheelhouse-stages.sh` takes the block from `_iree_check_prereqs` through `build_iree_wheels`. Always assert that the extraction caught what it claims (both suites do) so a rename degrades into a failing assertion instead of an empty one. **A golden trace that normalises paths must derive its temp root from `TMPDIR`.** `test-vulkan-target-decomposition.sh` folds the `mktemp -d` build directory out of the cmake argv before comparing; that `sed` matched a literal `/tmp` until 2026-09-04, while both the fixture and `vulkan.sh` use `mktemp -d`, which honours `TMPDIR`. Under any override the suite failed 5 of 35 assertions on un-normalised paths — green everywhere it was ever run, red on the first machine that sets `TMPDIR`. The pattern is now built from `${TMPDIR:-/tmp}` with every non-word byte escaped for the ERE, and a final case re-runs the whole suite under a different `TMPDIR` so the hardcoding cannot come back. `test-base-image-parse-options.sh` shows the two-mode variant: real `bash base-image.sh …` processes for the error paths — every parse error dies before `main` dispatches, so nothing touches apt, the network or the host — plus the extracted parser with a stub `die()` for the paths whose only observable effect is a variable assignment. Where the subject *is* a library (`llvm-cross.sh`, `tvm-config.sh`) source it directly and stub its collaborators. **Stub EXECUTABLES on PATH vs shell-function stubs.** A collaborator invoked as a plain word can be a shell function. One invoked through `env` — IREE's host stage runs `env -u … cmake …` — or through any other exec cannot see a shell function at all, so it needs a real file, `chmod +x`, on a PATH the suite prepends. Get this wrong and the stub is silently bypassed while the host's real tool runs. **A failing `set -e` is only observable across a PROCESS boundary.** `set -e` is disabled for the whole dynamic extent of a command whose status is tested, and `t_assert_ok` / `t_assert_fails` test status with `if` — so a subject that fails *only* through errexit cannot fail inside a subshell the harness calls. Drive it in a fresh `bash` process instead. `test-base-image-parse-options.sh` needs this for one arm: a trailing flag with no value at all reaches a `shift 2` that runs out of arguments, which is a failure through errexit and nothing else. **Mutation coverage is the rule.** A new regression test must be *demonstrated* to go RED when the thing it guards is broken, and its header must state honestly what it does and does not cover. The IREE suite is the worked example, and it failed this twice on 2026-08-31 before it passed: 1. Its fault-injection stub matched nothing, so every "failure" case was really a success case. The pattern started with `--` and was passed positionally to `grep`, which here is **ugrep** and read it as an option. `grep -qE -e "$pat"` is the portable spelling; the symptom entry is in [`failure-modes.md`](failure-modes.md#a-fault-injection-test-passes-and-proves-nothing-grep-is-ugrep). 2. With injection working, `rc == 1` STILL did not discriminate. Deleting `|| return 1` from a build-stage call site let execution fall through into `_iree_package_wheels`, which bailed at its own missing-project-directory guard and returned 1 anyway — **the right answer for the wrong reason**, and with a misleading "produced no runtime/ wheel project" line standing in for the real configure failure and its log. The cases therefore also assert the packaging diagnostic is ABSENT, which is what actually pins the early abort; all five call sites are now verified to fail at least one assertion when their `|| return 1` is removed. A suite nobody has watched fail is documentation with a green tick on it. Per suite, what the assertions actually pin (so each file needs only a one-line header): | Suite | Subject | Pinned | |-------|---------|--------| | `test-logging-err-trap.sh` | `01-core/logging.sh` ERR trap | Every case runs a real `bash -c` under `set -Eeuo pipefail`, the only place the dynamic-scope bug reproduces. warn reports and lets the script finish; err reports and exits 1; neither dies on an unbound variable; the reported line is the FAILING one, not the install site; the command survives spaces, quotes and `$`; a trap installed inside a function still fires after that function returned and from inside another; the last install wins; and `build-gcc.sh`'s hand-re-armed two-argument `on_err` keeps err semantics. | | `test-tvm-llvm-resolve.sh` | `05-frameworks/tvm.sh` `resolve_tvm_llvm` and `tvm-detect.sh` `detect_native_llvm_cmake_dir` | Which LLVM TVM links (BACKLOG CON35): a native build takes the CMake package the image ships as `/usr/local/llvm-target` (found under `lib` or `lib64`), never llvm-config's apt bootstrap; a caller's `TVM_LLVM_DIR` turns `USE_LLVM` on; a cross build keeps its target's package; without one, llvm-config as before. | | `test-tvm-cmake-args.sh` | `05-frameworks/tvm-config.sh` `append_tvm_cmake_args` (15 positionals → named options) | Golden argv captured from the pre-refactor implementation byte for byte, including `-D` ORDER (a later flag overrides an earlier one) and the exact `${CMAKE_*:+ …}` suffix handling; the Vulkan / cross-linker / `LLVM_DIR` / `CMAKE_IGNORE_PATH` / CUDA-OpenCL normalisation arms; `resolve_qnn_sdk` exporting `TVM_QNN_HOME` non-locally, which `tvm.sh`'s post-install staging depends on; and that an unknown, missing or value-less option is a HARD error rather than a silently wrong feature set. | | `test-vulkan-target-decomposition.sh` | `02-toolchain/vulkan.sh` `_build_vulkan_targets` + `_vulkan_target_*` | A golden trace of the cmake argv per component, the headers→loader→SPIRV-Tools→glslang order, the attempted/ok counter wording, the env-shaped all-failed verdict and its `VULKAN_CROSS_STRICT=1` promotion to fatal, the `source/glslang-main` checkout fallback, the `/usr/local/bin` alias plumbing (recorded through a stub `SUDO`, never run) — and that `_vulkan_target_link_glslang_aliases` keeps its explicit `return 0`, since the helpers are called as plain statements and its final false `[ -e ]` test would otherwise trip errexit and kill the SDK stage. Also `_vulkan_prune_sdk_sources`: the 3.9 GB `./vulkansdk` build tree goes in the SAME RUN that made it, AFTER the cross targets consumed it, and a tree without one is a no-op rather than an errexit abort. | | `test-vulkan-host-sdk-prune.sh` | `06-packaging/prune-vulkan-host-sdk.sh` + its wiring in `Dockerfile.package` | The builder-arch prefix goes only where the image runs its own: amd64 keeps `x86_64` because that IS its prefix; a foreign arch drops it only behind a REAL (non-symlink) `/lib/libvulkan.so.1`, so the `setup-vulkan-symlinks.sh` fallback link is not mistaken for a cross build and a lane whose loader failed keeps the tree and says so; an arch `platform.sh` cannot map is a hard error, not a guess; and the prune runs in `artifact-source` AHEAD of the `/opt/vulkan` COPY, where it is still worth bytes. | | `test-host-payload.sh` | `06-packaging/prune-host-payload.sh`, `runtime_host_payload` and their wiring | `HOST_PAYLOAD=on` touches nothing; `off` empties the Android SDK, `/opt/android` and a Rust toolchain that is not the target's, keeps every directory, the cargo registry and an existing payload-off marker, and writes the marker otherwise. The prune runs in `artifact-source` ahead of all four COPYs, and the package stage's eight Android variables follow it. The resolver puts amd64 alone on by default, honours `HOST_PAYLOAD_ARCHES`, turns off every artifact no amd64 host built, and refuses a bad list in all three orchestrators. | | `test-llvm-cross-stanza.sh` | `02-toolchain/llvm-cross.sh` helpers split out of `_llvm_cross_setup_and_build` | One `-Wl,-rpath-link` per EXISTING directory in order, an out-array reset rather than appended to, a failing resolver tolerated; both compiler launchers or neither; the superset shape (projects/runtimes, `LLVM_USE_HOST_TOOLS`, `CLANG_TABLEGEN`) not regressing to the core-only one that leaves `libLLVMSupportLSP.a` unbuilt; the nested `CROSS_TOOLCHAIN_FLAGS_NATIVE` string with and without a launcher; the three injected arg groups landing at their original insertion points in the configure argv; and exactly four build/install calls — including the explicit `--target llvm-config` (TVM reads it out of `/opt/llvm-cross`) and `--strip` on both installs. | | `test-iree-wheelhouse-stages.sh` | `05-frameworks/torch/build-app-wheelhouse.sh` `_iree_*` stages | The dynamic-scope couplings a refactor can silently sever — `ccache_cmake_args` / `_iree_launcher` reaching BOTH build steps, the `CCACHE_*`/`SCCACHE_*` exports surviving the helper boundary, `wheel_platform` reaching the retag, `iree_wheel_projects` reaching packaging so the cross lane ships the runtime wheel only, the target-python sysconfig export surviving into wheel-packing — plus the native and cross flag sets, the QNN arm, the five mutation-covered `|| return 1` call sites described above, and the CON81 settle: each target tree configures twice before its build (a failed second configure stops it before packaging), the host tree once. | | `test-base-image-parse-options.sh` | `01-core/base-image.sh` `parse_options` (116-line nest → data table) | The asymmetries a "harmonising" rewrite would eat: `--archive-url` rejects an empty value while `--ports-url` accepts one verbatim; both also flip `USE_FAST_UBUNTU_MIRROR=true` while `--rewrite-security` does not and validates through `parse_bool_flag`; `install-vulkan-runtime-files` is the ONLY command taking positionals (stopping at `--` or the first non-flag, remainder into `REMAINING_ARGS`, and it alone sets that variable); a repeated flag keeps the LAST value; and the per-command, no-arg and catch-all arms each die with their own message verbatim. | ### In-image verification gates & their escape hatches (audit round 2) **Retiring a transitional branch: probe, don't TODO.** A compatibility arm that exists only for artifacts built before some fix should decide by *probing the artifact* for a marker of the post-fix version (e.g. `grep -q STV_REQUIRE_GENAI` in the image's baked smoke), never by a "delete me later" comment. Only a POSITIVE probe may flip the verdict, so a probe that cannot run stays tolerant rather than failing an image it simply could not inspect. Lesson recorded when the riscv64 GenAI transitional arm was removed, 2026-08-31. The 2026-08-08 audit closed a set of gates that previously could not fail. Each hard gate has ONE explicit, documented escape hatch — set it only for a deliberately reduced image, never to "get the build green": | Gate | Where it runs | Escape hatch / opt-in | |------|---------------|----------------------| | riscv64 app-wheelhouse must contain real `*.whl` (a `.placeholder`-only dir fails) | `verify-media-artifacts.sh app-wheels` (Dockerfile.media) | `ALLOW_EMPTY_APP_WHEELS=1` | | `/opt/venv` must exist in the package wrapper image (even torch-less images ship a venv with a `.torch-missing` sentinel) | `smoke-torch-venv.sh` via wrapper-smoke | unset `STV_REQUIRE_VENV` (only stages that legitimately ship no venv) | | CUDA/cuDNN/TensorRT/NCCL completeness | `verify-cuda-stack.sh` (Dockerfile.nvidia) | default is warn-only; `CUDA_STACK_STRICT=1` is the OPT-IN hard gate for images that claim a complete stack | | One ROCm tree: every installed `amdrocm*` package whose name carries a release carries `ROCM_VERSION` | `setup-rocm-repo.sh` (Dockerfile.amd), after the last install | none. A second release means a versionless name came back ([`linux-accelerator-images.md`](linux-accelerator-images.md#the-rocm-release-is-in-every-package-name)) | | MIGraphX is installed and loadable: `migraphx.hpp` exists somewhere under `/opt/rocm`, and `ldconfig -p` lists `libmigraphx_c` | `setup-rocm-repo.sh` (Dockerfile.amd), after it writes `rocm.conf` | none. Since 10.1 MIGraphX sits in `extras-/`, off `/opt/rocm/lib` ([`linux-accelerator-images.md`](linux-accelerator-images.md#where-rocm-101-puts-migraphx)) | | TVM presence/version per arch | `smoke-torch-venv.sh` — a HARD assert on every image since `EXP_TVM` is set from `versions.env` `TVM_REF` in `smoke-torch-venv.sh`, not opted into | none. A media build that ships without TVM fails the per-arch wrapper smoke and **blocks the manifest**, at the end of a multi-hour chain. Dropping TVM from a lane means removing the pin, not expecting a warning | | ELF architecture of shipped binaries | `validate-media-runtime.sh` — runs on EVERY scan since 2026-08-08 (a clean dependency scan used to `exit 0` before it) | `MEDIA_ELF_MISMATCH_FATAL=0` downgrades to warning | | litert / genai / opencv-core produce real artifacts | `verify-media-artifacts.sh` | none — these verify stage-specific files now; genai mirrors its producer's legitimate cross-build skip, which since GEN1 (2026-08-31) covers only NON-arm64/riscv64 cross targets and a riscv64 lane switched off with `GENAI_ALLOW_RISCV64` | | `onnxruntime_genai`'s native binding really works — version == the versions.env pin, the loaded pybind `.so` is TARGET-arch ELF (read from its own `e_machine`), and native code RUNS (`og.Tensor` numpy round-trip, the capability predicates, `og.Config` rejecting a non-model path from C++) | `smoke-runtime-image.sh` `check_genai_binding` (payload: `smoke-common.sh` `smoke_genai_py`) | none — but an absent wheel is a SKIP, not a failure (presence is the ARCH-PARITY table's assertion). Set `GENAI_MODEL_DIR` to a real model directory to arm the fourth tier, which calls `generate()` and asserts on TOKEN CONTENT; no model ships in these images, so that tier reports UNPROVEN by default | | Every artifact tree in `runtime-artifacts.manifest` carries THIS image's ELF machine — `artifact-source` is the builder's image, so a tree installed on the host (rustup, Flutter) ships x86-64 into a foreign one | `smoke-runtime-image.sh` `check_manifest_tree_arch` | none. Exemptions name the TREE (`/opt/android`, `/opt/android-sdk`), never an arch, so a new host-installed tree fails by default — docs/artifact-copy-completeness.md#the-shipped-trees-must-carry-the-images-own-arch | | Vulkan cross-components (loader / SPIRV-Tools / glslang) — all three failing at once is an env-shaped toolchain cause | `vulkan.sh` | default is advisory (WARN); `VULKAN_CROSS_STRICT=1` is the OPT-IN promotion to fatal | | Vendored-wheel SOABI vs target triple — a native `.cpython-*.so` carrying a SOABI for a different arch than the target triple is a host-SOABI leak that only fails at `import` | `verify-wheels.sh` (triple derived from `TARGET_ARCH`, **not** the running interpreter) | default is advisory (WARN); `WHEEL_SOABI_STRICT=1` is the OPT-IN promotion to fatal | | OpenCV configured and installed against the chain ORT: no dnn download, the summary at the chain version, the library in the chain, no ORT file of its own under `/opt/opencv5` | `build-opencv.sh` via `opencv-ort.sh` (both passes) | none | | FFmpeg links the chain ORT: its `-L` first, every ORT reference in `ffbuild/config.mak` resolving into the chain; a missing chain or a probe that cannot link it is fatal | `ffmpeg-dnn-backends.sh` `ffmpeg_ort_link_findings`, called by `configure_ffmpeg` | none | | `libgstonnx.so` links and includes the chain ORT only (read from `build.ninja`) | `build-gstreamer-monorepo.sh` `_gst_monorepo_onnx_ort_gate` | none | | No apt ORT package, and every loader-path `libonnxruntime*` resolves into the chain | `validate-media-runtime.sh` (apt plan + dpkg), `configure-runtime.sh` `ort_runtime_gate` | none | | Every ORT consumer build ends in the ORT build gate, which writes the stamp the census requires | `ort_assert_chain_only` in `build-opencv.sh`, `build-ffmpeg.sh`, `build-gstreamer-monorepo.sh`, `verify-genai-ort.sh` | none | | Clean stop of a running chain — reaps the orphaned nerdctl/buildctl child subtree; **never `pkill` the orchestrator, that orphans them** | `bash linux/scripts/stop-cross-chain.sh` (finds the run via its pidfile, falling back to a bracket-trick pgrep) | n/a — operational tool, not a gate | #### Verify the shipped BYTES, never the push Backlog RTCACHE3; 2026-08-15 → 2026-08-16. The 2026-08-15 S2 saga shipped `:latest-cross` STALE five times with every static gate and all smokes GREEN — the manifest, smokes, and push were all byte-identical to a prior run. The real cause was the `--output type=image` tagging bug (see [`linux-cross-builds.md` § versions.env feature toggles](linux-cross-builds.md#versionsenv-feature-toggles-linux-lane), "Why the explicit `RUNTIME_NO_CACHE`"), NOT a cache; media+android were always fresh. This is now GATED automatically: `verify-shipped-wrapper.sh` runs in `build-runtime-manifest.sh` in its OWN pass over every arch, before the boot smokes and before the manifest is assembled — it lists each wrapper's rootfs (`nerdctl export | tar -t`, arch-agnostic, no emulation) and asserts the `/opt/ffmpeg` lib set matches the versions.env toggles (`FFMPEG_ENABLE_TF` → `libtensorflow` present/absent, ffmpeg intact). A mismatch aborts before `:latest` goes live; `WRAPPER_CONTENT_GATE=0` makes it advisory. To spot-check by hand: pull the wrapper and grep for the expected lib set. The two-pass order is load-bearing and was learned the hard way. Both checks used to share ONE loop with the boot smoke first, so a smoke failure on arch N skipped the content gate for arch N *and every arch after it*. On 2026-08-27 riscv64's smoke failed on an ARCH-PARITY arm, and the riscv64 wrapper that then shipped in the index had never been content-checked at all. Content gate first is also simply cheaper: `verify-shipped-wrapper.sh` is a tar listing — no boot, no emulation — so it runs anywhere, while the smoke needs QEMU for foreign arches. That gate covers each wrapper's CONTENT. The INDEX went ungated until 2026-08-27: the only manifest check in the chain was `nerdctl manifest inspect >/dev/null` (`build-runtime-manifest.sh`), which proves existence, not freshness — so an index can be created, pushed and reported green while one child still points at a previous run. `linux/scripts/verify-manifest-freshness.sh` closes that, registry-only (no pull, no emulation): it asserts each index child equals the digest its per-arch tag resolves to, and that all children share one `org.kataglyphis.run-id`. Run it with `EXPECT_RUN_ID` — measured on a live stale index, neither assertion suffices alone, because a wholesale-stale ship is perfectly self-consistent: child and tag agreed (both old), and all three run-ids matched (all from the previous run). Only pinning to the run that just built distinguishes the two. `build-runtime-manifest.sh` now runs it right after it writes the index, with `EXPECT_RUN_ID` set to the chain's run id. It is advisory there, because the index is already pushed: `MANIFEST_FRESHNESS_STRICT=1` makes it fatal and `MANIFEST_FRESHNESS_GATE=0` skips it. preflight does not run it. **`:latest-cross` was re-shipped 2026-08-16** (fresh amd64 `509027696e16` / arm64 `bdb46c953954` / riscv64 `28e3ded96f72`) carrying the Batch-2 fixes; that full-media rebuild flushed out two bugs the runtime-lane validations miss because they skip smoke-media — (1) smoke-media's native cv2 import must be gated on numpy being importable (numpy is a `/opt/venv` packaging dep, absent in the media BUILD sandbox → defer to the runtime smoke, like onnxruntime), and (2) `configure-runtime.sh` runs a SECOND time in the package stage, so its gstreamer-multiarch resolver must drop/skip the pre-existing `lib/multiarch` symlink before resolving or it re-points it at itself — and its dev-surface check must be a WARN, not a fail-loud exit (a script that runs twice must not carry a build-breaking assert; the pkg-config `verify_consumer_dev_surface` gate is the authority). `preflight.sh` keeps its check list in one place — the `KNOWN_SLUGS` array (`preflight.sh:36-52`, 37 slugs), which is also the vocabulary `PREFLIGHT_ONLY=` and `PREFLIGHT_SKIP=` accept. **That array is the authority for both membership and run order** — the table below groups them by kind and will drift if a slug is added without touching it. | Slug | Script | Catches | |------|--------|---------| | `crlf-guard` | inline (`git ls-files --eol` over `lint-shell.sh --list-files`) | a tracked shell script — `*.sh` or an extension-less file on a shell shebang — whose working tree carries CR bytes (`w/crlf`, `w/mixed` or `w/-text`) | | `shellcheck` | `lint-shell.sh` | classes 6, 7 — `shellcheck -S error` over every file `lint-shell.sh --list-files` names (427 on 2026-09-25); `linux/host-config`'s operator tools joined the sweep on 2026-08-27, before that seven scripts sat outside it | | `copy-coverage` | `verify_script_copy_coverage.py` | class 1 — a referenced `/opt/scripts` path never COPY'd/mounted into its image | | `context-paths` | `verify_dockerfile_context_paths.py` | class 1's other half — a COPY/`--mount=type=bind` source that no longer exists in that Dockerfile's build context, which BuildKit fails at context checksum before instruction one | | `critical-fixes` | `verify-critical-fixes.sh` | classes 2, 3 — the host half (fix5-fix11; fix11 is the ONNX Runtime single-source denylist, which reads `windows/` too); the /opt-probing half is [`smoke-critical-fixes.sh`](#the-in-image-half-of-critical-fixes), which no build stage runs | | `patch-integrity` | `verify-patch-integrity.sh` | a malformed unified diff, or an orphaned patch nothing references | | `artifact-parity` | `verify-artifact-copy-parity.sh` | `Dockerfile.package`'s artifact-COPY lane — missing artifact-source stage, undocumented src/dst relocation | | `arg-consistency` | `01-core/verify-arg-consistency.sh` | class 8 — ARG names/values vs `versions.env`, plus their forwarding | | `version-snapshot` | `docs/scripts/sync_versions.py --check` | class 8 — version snapshots / inline markers / deps table out of sync | | `pkg-names` | `verify_package_names.py` | a distro package name the tree asks apt for that no longer resolves against the live Ubuntu indices, at an UNGUARDED call site; SKIPs with no network and no cache | | `advert-keys` | `verify_advertised_keys.py` | a version-shaped `ENV`/`ARG` the runtime image advertises that the smoke neither checks nor excuses with a written reason | | `doc-links` | `docs/scripts/verify_doc_links.py` | broken relative links, dead anchors, `file.md § Heading` refs, missing `INDEX.md`/toctree coverage | | `doc-dupes` | `docs/scripts/verify_doc_dupes.py` | a passage copied into a second page; deliberate overlap is budgeted in `docs/scripts/doc-dupes.allow`, which itself fails when an entry goes stale | | `sbom` | `docs/scripts/generate_sbom.py --check` | the committed curated SBOM drifting from `deps.json` + `versions.env` | | `env-knobs` | `lint-env-knobs.sh` | a consumed `${VAR:-}` knob with no owner; advisory unless `KNOB_GATE=1` — stale allow rows always fail | | `mirror-consistency` | `01-core/verify-ubuntu-mirror-consistency.sh` | class 8 — a Dockerfile missing the canonical Ubuntu mirror ARGs | | `runtime-paths` | `04-runtime/verify-runtime-paths.sh` | class 8 — `PATH`/`LD_LIBRARY_PATH`/`PKG_CONFIG_PATH` drifting from `runtime-paths.env` | | `dockerfile-lint` | `lint-dockerfiles.sh` | hadolint findings (policy in `.hadolint.yaml`; bootstraps a pinned, SHA-verified binary when none is on PATH) | | `workflow-lint` | `lint-workflows.sh` | actionlint findings in workflows and composite actions | | `python-lint` | `lint-python.sh` | ruff — hard-fails on the real-error classes (syntax, undefined names); the full ruleset is advisory | | `secret-scan` | `lint-secrets.sh` | gitleaks over the working tree (`detect --no-git`), ENFORCING — triaged false positives are value-pinned allowlist entries in `.gitleaks.toml`, each carrying a written justification (`.gitleaksignore` is a superseded, deliberately empty stub: its `file:rule:line` fingerprints stopped matching on the first unrelated edit) | | `android-parity` | `01-core/verify-android-stage-parity.sh` | the five parallel Android library stages diverging beyond `ANDROID_LIB` | | `script-tests` | `linux/scripts/tests/run-tests.sh` | unit-test regressions in the tag/build-arg/disk-guard logic; prints the assertion aggregate above | | `stage-graph` | inline `cross_stage_validate_graph` | bad parent refs, missing dockerfiles, unresolvable tags, cycles | | `stdout-returns` | `verify_stdout_returns.py` | a function whose stdout is captured by `$(...)` logging to stdout, poisoning its return value | | `code-dupes` | `docs/scripts/verify_code_dupes.py` | token-normalised duplication across shell, Dockerfiles and non-`docs/` Markdown — it sees *renamed* clones | | `masked-decls` | inline | `local x=$(...)` / `export x=$(...)`, where the declaration masks the command's exit status | | `trailing-conditional` | `verify_trailing_conditional.py` | a function whose LAST statement is a bare test or an `&&` list, so it returns that test's status and a caller under `set -e` dies on the nothing-to-do path; deliberate predicates are frozen in `trailing-conditional.allow` | | `comment-size` | inline | comment blocks over 10 lines, against a frozen baseline — prose belongs in `docs/` | | `code-size` | `verify_code_size.py` | shell/Python functions over 80 lines and shell/Python/Dockerfile files over 800, against `function-size.allow` / `file-size.allow` | | `code-complexity` | `verify_code_complexity.py` | cyclomatic complexity over 15 or nesting over 5 in the `code-size` scan set, against `code-complexity.allow` — the shell counter is heredoc-, comment- and quote-aware | | `dead-functions` | `verify_dead_functions.py` | a NEW shell function defined under `linux/scripts` or `linux/host-config` and named nowhere else; dispatch the scanner cannot see is frozen in `dead-functions.allow` | | `shellcheck-warnings` | `verify_shellcheck_warnings.py` | a new or grown `shellcheck -S warning` finding per (file, code), over exactly `lint-shell.sh --list-files`, against `shellcheck-warnings.allow` | | `mutations` | `docs/scripts/verify_mutations.py` | a test that CANNOT fail: each recorded mutant neuters one guarantee and the named test must go red | | `shared-config` | inline (`bash shared/config/sync-shared-config.sh --repo-root . --check`, the bash twin, because no hub Linux image ships pwsh) | ANTfrastructure's own root `.cmake-format.yaml` — a consumer copy of the canonical file beside the sync script — drifting from it or deleted; the root `.antfrastructure-shared.manifest` declares it as the one taken file, and the other four canonical names have no root copy here, so they are simply not declared | | `cmake-format` | inline (`lib/code-quality.sh` + `cmake-format --check`) | a repo-owned CMake file (`cmake/`, any `CMakeLists.txt`) deviating from the root `.cmake-format.yaml` — including CRLF endings, which `line_ending: unix` treats as drift; the `windows/scripts/patches/` shims are excluded because their bytes are Windows layer-cache keys | | `gate-registry` | `verify_gate_registry.py` | the meta-gate — a slug with no proof (no suite naming its script, no mutation) and `docs/code-quality-gates.md` drifting from what it derives; unproven slugs are frozen in `gate-proofs.allow` and may only leave it | Every check with a script is runnable standalone (same command); `crlf-guard` and `stage-graph` are inline in `preflight.sh` and have no separate entry point. The pre-commit hook (`linux/host-config/git-hooks/pre-commit`) runs the whole-tree gates that are cheap — `PREFLIGHT_ONLY=` the 18 fast slugs in `_FAST_SLUGS` (`:88-91`), 6.4 s combined — and then three blocks scoped to the STAGED content, so nothing slow runs over the whole tree: `shellcheck -S error` plus the warning ratchet on staged `.sh` files (`:102-119`, the binary resolved through `lint-shell.sh --print-bin`, its one owner), the doc-duplication gate when `docs/*.md` moved, and `verify_mutations.py` on a SAMPLE of the entries whose target is staged — at most `PRECOMMIT_MUTATION_CAP` (default 16), newest first, and it prints `SAMPLED n of m` whenever it cut, because a green hook must not imply coverage it did not pay for. Measured end to end 2026-09-04: **8.0 s** for a one-file commit (no sampling — all 5 matched entries ran) and **27.2 s** for the 43-file commit `9a5bf8dd` (sampling 6 of 132), against **5m26s** for that same commit before the cap; those 132 entries uncapped are **322 s** on their own. `make preflight` and CI still run every entry. Why it samples rather than running `--changed` whole: [`code-quality-tooling.md`](code-quality-tooling.md#the-pre-commit-hooks-cost-budget). `shellcheck-warnings` is deliberately NOT a fast slug: whole-tree it was 22 s against 0.2 s per staged file (measured 2026-09-04). ### In-image smoke tests (need a built image, not part of preflight) These validate a built/pulled image and also run during the build to fail fast: > **Two-environment semantics of `smoke-media.sh` (since 2026-08-10):** the > suite runs TWICE — once inside the media build sandbox (Dockerfile.media, > loader NOT yet wired: `/opt/ffmpeg` libs and `/opt/venv` are unreachable > there) and once at the packaging stage (Dockerfile.package, loader fully > configured). In the sandbox run, three gates deliberately DEFER instead of > failing: the `onnxruntime_genai` Python import (its wheel installs into > `/opt/venv` only at packaging; `smoke-torch-venv.sh` is the functional > gate), the gst `libav` plugin load (links the source-built ffmpeg libav* > — gated on ffmpeg-executability; libtensorflow is bundled only when > `FFMPEG_ENABLE_TF=1`, which defaults OFF since S2, 2026-08-14), and > ffmpeg's own execution. Auditing coverage by the media-stage log alone > therefore UNDER-counts what is enforced — the packaging-stage run is the > strict one. - **Native source-build header preflight** — inside `setup-torch-venv.sh` (`verify_native_source_headers`): compiles tiny C / C++ / jpeglib probes with the same compiler+flags the pip build uses, so a header/sysroot regression (classes 2/3) aborts in <1s instead of after a ~9-min numpy/pillow compile. - **Torch venv integrity** — `06-packaging/smoke-torch-venv.sh`: imports numpy/torch/torchvision/PIL/cv2/contourpy (+ torch↔numpy ABI bridge) from `/opt/venv` (class 5). Wired into the `wrapper-smoke` target's smoke set (`validate-compilers.sh`, `smoke-media.sh`, `smoke-torch-venv.sh`, `smoke-cross-all-arches.sh`); skips cleanly if no venv. Run standalone: `VENV=/opt/venv smoke-torch-venv.sh`. - **Runtime-image boot + functional smoke** — `06-packaging/smoke-runtime-image.sh `, run per-arch by `build-runtime-manifest.sh` against the freshly built wrapper. Boots the actual published image and, under **binfmt/qemu for the cross arches**, runs real workloads *on-target*: - ML imports (`onnxruntime`, `numpy`, `torch`) + `ffmpeg -version` (pipefail-guarded so a missing `.so` can't pass silently); torch-less sentinel flagged. - **ML version-pin assertion** (fail) — not just *importable* but the *correct versions*. Delegates to `smoke-torch-venv.sh` (assert-only). Two authorities, but since GENAI-DRIFT (2026-08-23) they are **no longer unioned** — whichever one OWNS the package decides (`smoke-torch-venv.sh` `assert_pinned_versions`, implemented as `allowed = pin_set(build_pin) if build_pin else set(from_lock)`). A versions.env **build pin** wins outright for everything we build or force-reinstall from a **local wheel** (riscv64 torch/vision, the source-built onnxruntime, ai-edge-litert, onnxruntime-genai) on every arch that builds it; the lock's opinion is printed (`[uv.lock says …; build pin wins]`) but not accepted. uv.lock governs only what uv resolves and we do not build (numpy/pillow/contourpy + the amd64/arm64 torch/vision/onnx wheels when unpinned). The comparison uses each module's `__version__`, which for the source-built onnxruntime intentionally differs from its pip dist metadata — the build pin (`ONNXRUNTIME_VERSION` in `versions.env`) is what governs. The old union is exactly how arm64 shipped onnxruntime-genai 0.14.0 (from the lock) against a `v0.15.2` build pin and still printed OK. One carve-out mechanism survives: `KNOWN_DRIFT` in `smoke-torch-venv.sh` holds dated, exact `(dist, arch, installed, expected)` quadruples (with a reason), each printed as a loud `!!` and counted in the summary; anything that is not such a quadruple still FAILS, and a new version on either side re-arms the assert by itself. The list is empty on 2026-09-25. Also checks the `+cpu`/`+cu130` build variant vs `PYTORCH_EXTRA` and the OpenCV major. Catches a wrong version silently slipping in (lock drift, a stale local wheel, a floated index) — the class a presence/import check can't see. - **Native `/opt` `.so`-closure gate** — `ldd` over the ffmpeg/opencv5/libcamera/ vulkan payload; any unresolved soname fails the gate. Generalises the ffmpeg check to the whole native stack (the class that shipped libopencore-amrwb.so.0- broken ffmpeg). Venv Python extensions are excluded (import-time lib paths defeat bare `ldd`; the import checks are their gate). - **ARCH-PARITY table** (fail) — `smoke-runtime-image.sh` sees ONE image at a time (`build-runtime-manifest.sh` invokes it per arch), so this is **table conformance, not a cross-arch diff**. Every `/opt` prefix and component wheel NAMED in `_PARITY_PREFIXES`/`_PARITY_WHEELS` must be present on this arch unless `_parity_exempt` documents its absence; exactly one `onnxruntime` distribution, the flavour `_parity_ort_flavor` names, may be installed. The **blind spot** is a one-sided EXTRA: a component present on one arch and absent from the table is invisible to the loop, so the untracked `/opt` prefixes are printed (INFO) every run and a human diff of the three per-arch logs is what catches it — putting a component under the gate means adding it to the table. A documented exemption whose component turns out to be PRESENT **fails**, and so does a `_PARITY_GST_KNOWN_BROKEN` entry whose plugin starts loading again (two independent signals must agree: absent from the scanner's failure list AND `gst-inspect-1.0` loads the file directly). Warning there instead would let the table rot underneath a green run; failing makes it self-correcting, since the fix is the one-line deletion the message names. - **CONSUMER CONTRACT** (fail) — the properties a consuming CI lane depends on and cannot repair from inside a read-only overlay layer, one row each in `_CONSUMER_CONTRACT_ROWS`: compiler caches outside `/workspace` and writable, `$RUSTUP_HOME/tmp` and `$CARGO_HOME` writable, `ANDROID_HOME` set with `platform-tools` present *and* on `PATH`, a JDK, a readable `appimagetool`, every path under `/opt/flutter` owned by the runtime uid, and, added since, the staged flatpak and AppImage runtimes, the web-lane tools and the ORT crate environment. One probe, run as the image's own `Config.User` — as root every directory answers writable, so a probe reporting any other identity fails the gate outright. The contract itself, its per-arch exemption table and the 2026-09-04 defects that motivated it: [`consumer-image-contract.md`](consumer-image-contract.md#the-contract). - **GStreamer plugin health** (warn) — lists plugins whose runtime `.so` is absent (they degrade gracefully); surfaces app-critical regressions like `webrtcbin2`→`librice-proto.so.0`. **GStreamer core pipeline** (fail) — `videotestsrc ! videoconvert ! fakesink`. - **onnxruntime inference** (fail) — runs a tiny embedded Add model and asserts the output (proves the CPU EP executes, not just imports). **cv2 encode/decode** roundtrip. **Application import** — the shipped venv must `import orchestrant`. - **Native compiler battery compile+link+RUN** — a 9-case battery with the image's `gcc`/`g++`: C hello (stdout), pthreads, libm, libatomic; C++ hello (libstdc++), **exceptions+STL** (throw/catch + `std::sort`), std::thread, `-flto`, and `-fsanitize=address,undefined`. The first eight run their binary on-target. The sanitizer case runs its binary only where the build host's arch is the image's ([§ The native GCC ships libsanitizer](#the-native-gcc-ships-libsanitizer)). This is what upgrades class 4 from a static ELF/machine check to genuine **execution** proof: a cross arch's binary can't run on the x86_64 build host, so the shipped native GCC (esp. the riscv64 `--with-isa-spec` toolchain) was previously never actually executed — under qemu here it is. The **exceptions+STL** case is the regression guard for the `swap-native-gcc.sh` **wrapper** fix (`c46da5f`): the compiler reaches the runtime image's system headers via command-line `-idirafter` wrappers, *not* an installed `specs` file — a specs file silently drops `-lgcc_s`/`--eh-frame-hdr` and makes every throwing C++ program terminate at runtime. Gate `RUNTIME_COMPILER_SMOKE=0` to skip just this; `RUNTIME_FUNCTIONAL_SMOKE=0` skips all functional checks; `RUNTIME_IMAGE_SMOKE=0` (in `build-runtime-manifest.sh`) skips the whole runtime-image smoke (e.g. a host without a qemu handler for a foreign arch). #### Shipped-truth gates (2026-09-01) — measure the artefact, not the log Two classes were invisible to every gate above because every gate above reads what the *build* claims. These read the *shipped bytes*. Both live in `06-packaging/smoke-runtime-image.sh` and share ONE in-image probe (`run_shipped_truth_probe`, ~13 s per image (measured on arm64; the three gates together ~15 s), no network, no host state). **Why one probe that only prints facts.** The probe emits `ADV`/`HAVE`/`PKG`/ `REQ`/`DANG` lines and an `RTPROBE_DONE` marker; every verdict is reached afterwards on the host by `_advert_verdicts` and `_venv_set_verdicts`, which are pure `text in → verdict lines out`. That split is deliberate: the judging code can be driven from a *recorded* probe capture, so its failure paths are provable without a doctored 27 GB image — and without a hand-copied replica of the logic, which is exactly how a test in this repo once stayed green while the code it guarded was gutted. **A. Advertised env versions == actual** (`check_advertised_versions`, fail). Every key in `_ADVERTISED_VERSION_KEYS` (`PYTHON_MAJOR_MINOR`, `GCC_VERSION`, `LLVM_RELEASE`, `GSTREAMER_VERSION`, `VULKAN_VERSION`, …) is compared against what the image *has* — the venv interpreter, `gcc -dumpfullversion`, `clang --version`, `gst-inspect-1.0 --version`, the resolved `/opt/vulkan/active` path. This exists because all three wrappers shipped Ubuntu's Python **3.14.4** while advertising `PYTHON_VERSION=3.14.7`; an env label that contradicts the interpreter is worse than no label, since everything downstream reasons from it. There is **no exemption arm** — a version label that disagrees with the artefact is never a documented state. There is no `SKIP` arm either. Both "the image did not tell us" answers are **fatal**: a key the image does not set at all (`UNSET`) and a key whose actual value the probe could not read (`UNREAD`). The second is the exact shape the builder's `rustc` shipped in — it exited 127 on every foreign image for months while this gate printed a green `SKIP` — and the first is how a row that *can only ever* `SKIP` gets in, the same hole `verify_advertised_keys.py` emptied `FROZEN_UNPROBED` to abolish. A verdict verb no arm handles is fatal too: a silently dropped row is that failure wearing a new name. A key the image deliberately does not advertise belongs in `verify_advertised_keys.py`'s `EXCUSED` table rather than in a row that cannot fail — `PYTHON_VERSION` is `ARG`-only by design (the python-ci drivers read it as their own knob, so an image ENV would turn every default into the full X.Y.Z; the package stage checks the staged interpreter reports it exactly), and `PYTHON_MAJOR_MINOR` is the ENV that *is* checked. **B. Venv package set vs the app's own declared graph** (`check_venv_package_set`, fail). Measured live, the shipped venvs held amd64 = 153, arm64 = 165, **riscv64 = 66** distributions; 99 packages were simply gone on riscv64 and nothing noticed, because ARCH-PARITY only conforms a fixed *name table* of `/opt` prefixes and component wheels and never compares the venv set at all. The gate does **not** carry a checked-in list of 165 names and does **not** carry a count anybody can lower. The expected set is derived from the image's own metadata: the app distribution's `Provides-Extra` requirement edges, with environment markers evaluated *inside the image*. Upstream already encodes the legitimate riscv64 divergences as markers (`mlflow; platform_machine != "riscv64"`), so those are honoured for free and never need a local exception. Which extras count is a two-line **contract**, not a threshold: `_VENV_CONTRACT_EXTRAS="ml-ai docs"` (what `assemble-torch-app.sh` always requests) plus whichever `pytorch-*` extra the image itself advertises in `PYTORCH_EXTRA` — reading the image's own advertisement is what keeps a `pytorch-cpu` wrapper from being asked for the ROCm extra's wheels. A second arm needs no configuration at all: any **dangling edge** — an unconditional requirement of an installed distribution that is not installed — fails, which is what covers the transitive tail the extras graph cannot see. Documented divergences go in `_venv_pkg_exempt`, keyed `::` (`DEP` for a dangling edge) and following `_parity_exempt`'s contract exactly: listed = reviewed, and an arm whose package turns out to be **present** fails and names the line to delete, so the table cannot rot in place. Today it holds three arms — `cv2` is the source-built `/opt/opencv5` binding rather than the PyPI `opencv-python` wheel; `onnxruntime` ships under its flavour name (`onnxruntime_dnnl` / `_webgpu`), which `_parity_ort_flavor` asserts; and riscv64 ships no `scipy`, `scikit-learn` or `pandas` (no riscv64 wheels on PyPI, and a source build means BLAS/LAPACK and Fortran — an owner decision measured 2026-09-02). An extra that yields **no** requirement edges fails (renamed upstream, or truncated metadata) and so does a run that asserted nothing: an empty set is not a pass. **What these two do NOT cover.** They still see one image at a time, so a component present on *every* arch but wrong on all of them is invisible; a package installed at the right name but the wrong *version* remains the ML version-pin assertion's job; anything the app does not declare as a requirement (a transitively-pulled tool nobody depends on) is outside the graph; and the advertised-version table covers only the keys listed in `_ADVERTISED_VERSION_KEYS` — a new version-carrying `ENV` in `Dockerfile.package` is unguarded until it is added there, which is what the `advert-keys` gate enforces. **Mutation record (both gates proven red, then green).** Against the real locally-built `latest-cross-arm64`/`-riscv64` wrappers: | mutation | result | | --- | --- | | none (as shipped) | A **fails** on the live `PYTHON_VERSION` 3.14.7-vs-3.14.4 defect on both arches; B green on arm64 (18 edges), **10 fails** on riscv64 (`optuna`, `pandas`, `scikit-learn`, `scipy`, the four `docs` packages, `flatbuffers`, `protobuf`) | | advertise `PYTHON_VERSION=3.14.4` (the corrected label) | A **green**, 6/6 | | advertise `GSTREAMER_VERSION=9.9.9` | A fails on that key too — a second, independent row can red | | `pip uninstall captum scipy`, image re-committed | B goes **red**: 2 extra-requirement fails + 3 dangling-edge fails; unmutated image green | | recorded probe, `RTPROBE_DONE` stripped | both gates **fail** ("a gate that cannot run is not a pass") | | recorded probe, every `ADV` value blanked | A **fails** — one fatal `UNSET` per row. Before 2026-09-04 those rows were loud `SKIP`s and only the vacuous-pass guard caught the blanking | | recorded probe, `PKG opencv-python` added | B **fails**: the documented exemption no longer applies, message names the arm to delete | | recorded probe, `REQ docs …` removed | B **fails**: refuses to assert an empty extra | | recorded probe, `VENV ABSENT` injected | B prints a loud `SKIP` (never a pass) | ##### E. ORT single source **What it checks** (`check_ort_census`, fail). This is the ORT census, G1 of the ONNX Runtime single-source rule ([`onnxruntime-single-source.md`](onnxruntime-single-source.md)). Every ORT binary in the image must be the chain build, byte for byte, and every importer must resolve to it through the image's own loader order. **How.** `06-packaging/ort_census_probe.py` runs INSIDE the image as one process, so the loader facts are the image's own and nothing needs `readelf` under QEMU. It finds ORT by name (`libonnxruntime*.so*`, `onnxruntime_pybind11_state*`) and by content: the whole `onnxruntime/(core|contrib_ops)/….cc` source-file paths, ending in NUL, that every ORT build embeds through `__FILE__` (a directory string alone is a consumer, not ORT: [`onnxruntime-single-source.md`](onnxruntime-single-source.md#what-the-chain-ort-is)), plus the ABI markers `OrtGetApiBase`, `CreateEpFactories` and `RegisterCustomOps`. An ELF whose dynamic symbols DEFINE `OrtGetApiBase` is an ORT instance under any name, with or without fingerprints: the probe looks the symbol up through `DT_GNU_HASH` or `DT_HASH`, as `ld.so` does, and prints such a file's BIN line with `def` ([`onnxruntime-single-source.md`](onnxruntime-single-source.md#an-ort-under-another-name)). The probe prints a relative build root as `.`; `-` means no fingerprint at all. It reads every ELF or wasm file of at least 1 KiB and every archive member, and emulates `ld.so` per importer: RPATH, `LD_LIBRARY_PATH`, RUNPATH with `$ORIGIN`, `ld.so.conf`, then the default dirs. `ort_census_verdicts` in `06-packaging/check-ort-provenance.sh` is pure: probe text in, verdict lines out. **The reference set** is read from the image, never hard-coded: the prefixes `/usr/local/lib/onnxruntime-{cpu,gpu,web}` and `/opt/android/onnxruntime`, plus `/ort-provenance.sha256`, the chain wheel's native members, which `03-media/runtime/collect-artifacts.sh` writes because the runtime image never holds the wheel. On cross media `03-media/runtime/repair-wheels.sh` strips and retags those wheels, so it re-points each row at the wheel it ships (`ort_manifest_rows check` before, `follow` after); without that the census called our own stripped ORT STALE. That writer lists only wheels of the build's `ONNXRUNTIME_VERSION` and fails when it cannot resolve one. The chain build roots are `/opt/onnxruntime` and `/opt/onnxruntime-android`. A manifest row carries no build roots, so the census grades the installed bytes that match it: a manifest sha whose bytes name a non-chain root is FOREIGN. | Verdict | Meaning (every one fatal except EXEMPT) | | --- | --- | | `FOREIGN` | another build root (PyPI `N:\_work\1\s`, Windows ML `C:\__w\1\s`, pyke, …), or relative paths only (mixed with a chain root the verdict is `STALE`, with a foreign root `FOREIGN`); a reference whose roots are all relative is FOREIGN too | | `STALE` | the chain's root, but not this chain's bytes | | `UNPROVEN` | an ORT name or an `OrtGetApiBase` definition with no fingerprint and foreign bytes, or an unreadable ORT file or archive | | `ELSEWHERE` | a chain-identical copy outside the chain prefixes and `*/site-packages/onnxruntime/capi` | | `UNRESOLVED` | an importer's `ld.so` order lands on nothing, or on a non-reference file | | `UNREGISTERED` | an ORT-ABI user outside `ort_census_contract` | | `STAMP` | a present consumer without a G2 stamp naming this image's chain core library | | `DIST` | two distributions own `onnxruntime/` | | `NONE` | no probe marker, an empty reference set, or no ORT found | | `EXEMPT-STALE` | an `_ORT_CENSUS_IMAGE_EXEMPT` entry (`::`) that no longer matches | STAMP is armed while `linux/scripts/03-media/ort-provenance.sh` defines `ort_assert_chain_only`. The four Linux consumers write `/ort-provenance/.json` at build time, and the prefixes travel whole into the runtime image, so an image built before G2 fails STAMP until its consumer stages are rebuilt. **What it does NOT cover.** Header-only provenance: a consumer compiled against foreign headers or import libraries ships no foreign bytes, which is G2's job. Files the image user cannot read (`/root` is not scanned). Which copy a `dlopen` by absolute path picks at run time. An ORT that is renamed, stripped of its fingerprints AND built so it no longer exports `OrtGetApiBase`; an archive member is found by name and fingerprint only. The call sits inside the `RUNTIME_FUNCTIONAL_SMOKE=1` branch of `main()`. **Consumer use (G6).** Inside the image the app was built in, run `bash third_party/ANTfrastructure/linux/scripts/06-packaging/check-ort-provenance.sh ` (`--reference DIR`, `--manifest FILE`, `--exempt ARCH:PATH:WHY`). Exit 0 is clean, 1 is findings, 2 is usage; `--image` grades the system it runs on. ### The in-image half of critical-fixes The `critical-fixes` battery was two scripts wearing one name, and the split is what let its preflight slug leave `gate-proofs.allow`. `verify-critical-fixes.sh` now holds fix5–fix11, every one of them a grep over the REPO TREE, so preflight can run it off-target and `test-critical-fixes.sh` can knock out one guarded line at a time in a fixture tree. Each row runs only the fix it knocks out, extracted from the gate; one case runs the real gate, which proves a FAIL reaches the exit code. `06-packaging/smoke-critical-fixes.sh` holds fix1–fix4, the probes that only mean anything inside a built image. It is **not wired into any build stage**: run it against a shipped image with `nerdctl run --rm --platform linux/ -v :/repo:ro --entrypoint bash /repo/linux/scripts/06-packaging/smoke-critical-fixes.sh`. `CF_SMOKE_ROOT` prefixes every absolute path it reads, which is what makes the suite able to prove it on a host. **Nothing had ever run those four probes anywhere.** On the host they all skip — except fix4, which on an amd64 host asserts that an amd64 `cc` reports amd64. Run for the first time on 2026-09-05, against the images this repo had actually shipped, they reported **four failures, and all four were the probe being wrong**: | probe | what it asserted | what the image had | |---|---|---| | fix1 | a literal `prefix=/opt/python-cross` line in `python-3.14.pc` | `prefix=${pcfiledir}/../..` on all three arches — the RELOCATABLE form, which resolves to exactly the right directory. Now the probe resolves `${pcfiledir}` the way pkg-config does and judges where the prefix LANDS | | fix2 | `absl/types/span.h` under `/opt/litert/include`, `/usr/local/include/litert` or `/usr/local/include/tensorflow/lite` | `install_abseil_headers` installs into `/usr/local/include`, which the probe never looked at. It also treated the existence of `/usr/local/include/tensorflow` as proof LiteRT was present — and that path ships as an EMPTY two-directory stub | fix3 and fix4 pass, and that is the first evidence either has ever produced: `cross-android-amd64` has no dangling `lib-dynload` symlink on any of the three staged arches, and `cc -dumpmachine` reports `x86_64` / `aarch64` / `riscv64` in the matching `latest-cross-*` image. (`readelf` is absent from the foreign runtime images, so the ELF-machine arm of fix4 returns early there.) **With fix2 corrected it finds a real defect, on all three shipped arches.** `latest-cross-{amd64,arm64,riscv64}` each ship 1322 LiteRT/TFLite headers, **707 of which `#include "absl/…"`**, plus `litert.pc`, `tensorflow-lite.pc`, `tensorflowlite_c.pc` and the libraries — and no `absl` directory at all. A consumer that compiles against the prefix those `.pc` files advertise fails on the first include. The cause is `06-packaging/copy-media-payloads.sh`, which copied `/usr/local/include/{tflite,tensorflow,flatbuffers,c}` out of the artifact image and left `/usr/local/include/absl` — 384 headers, 4.6 MB, present in all three `cross-android-*` images — behind. It is in the copy list now; only a rebuild proves the bytes. ### Foreign-arch execution needs QEMU binfmt — registered **without sudo** The foreign-arch smokes above only mean something if the image's binaries can actually *execute* on the build host. That requires a `binfmt_misc` QEMU handler. **`build-runtime-manifest.sh` registers it automatically, with no sudo**, before the smoke loop (`ensure_foreign_binfmt` → `linux/scripts/setup-rootless-binfmt.sh`). Opt out with `RUNTIME_REGISTER_BINFMT=0` (e.g. a rootful/CI host where qemu is already registered via `docker run --privileged tonistiigi/binfmt` or `update-binfmts`). Two dead-ends to know about, because both *look* like they work and don't: - **`nerdctl run --privileged tonistiigi/binfmt --install …` (rootless)** registers binfmt inside the throwaway container's own user namespace, which `--rm` destroys. It prints "arch OK" but never reaches the namespace where builds/runs happen. - **BuildKit's embedded `/dev/.buildkit_qemu_emulator`** only wraps the *top-level* process of a `RUN`. The shell starts, but its first *child* exec (`uname`, `mktemp`, `gcc`, `python`, `ffmpeg`) dies with **`Exec format error`** — so it cannot run any real multi-process smoke. `setup-rootless-binfmt.sh` is the working no-sudo path on a rootless containerd/BuildKit host: buildkitd runs `nsenter`'d into containerd's rootlesskit namespace, so `nerdctl run` and `nerdctl build` **share one persistent namespace**. The script extracts the static `qemu-` emulators from `tonistiigi/binfmt`, enters that shared namespace via `containerd-rootless-setuptool.sh nsenter` (where the mapped uid-0 *does* hold `CAP_SYS_ADMIN` over its own mounts — no host sudo), and registers each with flags **`POCF`**. The **`F` (fix-binary)** flag is the crux: the kernel opens the interpreter fd at registration time, so emulation is inherited into the nested build/run namespaces where the qemu path isn't even mounted — which is exactly what makes *child* execs work. Register once per boot (or install the `systemd --user` unit with `--install-service`); verify with `--verify`. ## Dedup & factoring notes (2026-07) The tree has been through several dedup passes already; remaining duplication is largely **deliberate** and should not be "fixed": - `cross_build_is_active` / `install_host_packages` etc. are re-defined as **fallbacks** in several modules so each can be sourced standalone. Removing them breaks isolated use. Three owners are live and none is redundant: `01-core/common.sh` holds the documented approximation, `_gst_monorepo_env_setup` a strict superset for stage contexts where 01-core is never sourced, and `01-core/install-deps-preamble.sh` the trivial native `false` stubs. A fourth, in `media_common_init`, was deleted on 2026-09-05: the same function asserts with `declare -F` that `cross_build_is_active` exists and returns 1 when it does not, sixteen lines above the guard, so that copy could never execute. "Sourced standalone" is the test — a fallback that an assertion above it has already refused is dead, not defensive, and `test-cross-fallback-parity.sh` pins the assertion so the deletion stays justified. - `build-libcamera.sh`'s inline `-idirafter /usr/include` (generic) plus its `append_cross_idirafter` call is a **native/cross fallback pair**, not a copy. - The `-idirafter` logic in `setup-torch-venv.sh` + `swap-native-gcc.sh` duplicates the canonical helper `append_cross_idirafter` (`01-core/common.sh`). It is **inlined on purpose**: the torch/android stages do not COPY `common.sh` (nor its `load-versions-env.sh` chain), and pulling that in for six flag lines is not worth the surface on a verified critical path. The copies are cross-referenced to the helper and kept in sync by `verify-critical-fixes.sh` **fix6** — the pattern for necessary duplication: guard it, don't hide it. ## Hardening pass (2026-07) — landed + residuals **Landed** (see `verify-critical-fixes.sh` **fix7** for the regression guards): - **Build cache** — replaced the self-defeating registry `-buildcache` (whose `--cache-to` was gated out by `NO_CACHE_EXPORT`, so nothing ever cache-hit) with a **local** buildkit cache + inline cache on push. This is why full base rebuilds no longer recur on the same host. - **Base cache scope** — base RUNs bind-mount only `01-core` (+ `02-toolchain` for shared tooling), so editing a media/android script no longer busts the base image. - **Supply-chain** — the sole floating external base (`ubuntu:26.04`) is now digest-pinned by its multi-arch manifest-**list** digest (`UBUNTU_DIGEST`). - **Non-root runtime** — `/workspace` is `chown`ed to `kataglyphis`; `PYTHONDONTWRITEBYTECODE=1`; the build-only fake `sudo` shim is stripped from the shipped image. - **Robustness** — image-wide `apt` retries (`80-retries`); real-pipe + `PIPESTATUS` so a failing build's log tail is flushed and its true exit code returned; `parallel-loop.sh` names the failed arch. - **Smokes** — every previously-orphaned smoke now runs: `smoke-toolchain` (toolchain, `Dockerfile.toolchain`), `smoke-android` (android, `Dockerfile.android`), the wrapper-smoke set `validate-compilers` + `smoke-media` + `smoke-torch-venv` + `smoke-cross-all-arches` (`Dockerfile.package`, `FROM package AS wrapper-smoke`). The wrapper-smoke stage is a separate `--target wrapper-smoke` build run by `_runtime_run_package_smoke` in `runtime_build_chain`, between the package and wrapper builds (`WRAPPER_SMOKE_GATE=0` to skip) — it is NOT reached by the package build's own `--target package`, which prunes everything after `package`. Also host-side `smoke-runtime-image` (in `build-runtime-manifest.sh`, `RUNTIME_IMAGE_SMOKE=0` to skip). The one deliberate exception is `smoke-vulkan`, which is wired into **no** stage: it probes the full Vulkan *SDK* (`/opt/vulkan/active`, `vulkan/vulkan.h`, `libvulkan.so`) while this cross build installs only the Vulkan *runtime* (`libvulkan.so.1` + ICD/layer JSONs), so it can never pass here. `verify-critical-fixes.sh` hard-fails preflight if any Dockerfile RUNs it; it is retained as a standalone host tool for image variants that DO ship the SDK (see the note after the `wrapper-smoke` stage in `Dockerfile.package`). - **Reproducibility (opt-in)** — `clone_or_update_repo` and `build-ffmpeg.sh` accept a 40-hex commit SHA; `OPENCV_COMMIT`/`OPENCV_CONTRIB_COMMIT`/ `FFMPEG_COMMIT` (empty by default = keep the `OPENCV_VERSION`/`FFMPEG_VERSION` ref, which has been a release tag rather than a branch since 2026-08) freeze those sources to an immutable commit for a release build. - **Attestations (opt-in)** — `BUILD_ATTEST=1` attaches SLSA provenance + SBOM to pushed images. **Residual supply-chain gaps, as listed in 2026-07 — four of five are closed now.** Each was a `curl`/`wget` without a checksum. Four now go through `download_verified_file` with a `*_SHA256` pin in `versions.env`, or refuse the unverified path: | Source | Site | Now | |--------|------|------| | Flutter SDK tarball | `flutter/setup-flutter.sh` | `FLUTTER_SDK_SHA256` | | Android cmdline-tools zip | `android-sdk.sh` | `ANDROID_CMDLINE_TOOLS_SHA256`; NDK/build-tools are sdkmanager-verified | | freetype source | `opencv/install-deps.sh` | **still open.** apt's `libfreetype-dev` comes first (`\|\| true`); the source fallback, `cross_compile_cmake_lib_from_source` (`01-core/cross-env.sh`, also used for harfbuzz and libpng), fetches a tag tarball or a shallow `git+` clone with no checksum, and never hard-fails | | GStreamer-Android universal | `android/build-gstreamer.sh` | `GSTREAMER_ANDROID_UNIVERSAL_SHA256` (the prebuilt fallback path) | | `rustup-init` / NodeSource | `install-rust.sh`, `onnxruntime/build/10-deps.sh` | `rustup-init` is downloaded to a file and checked against `RUSTUP_INIT_SHA256`, never piped into `sh`; the NodeSource fallback is refused in favour of the base image's pinned Node | **Deliberate keep:** the `-dev` header packages in `setup-torch-venv.sh` land in the final image. This is a cross-**dev** container that compiles Python wheels from source under QEMU at build time, so the headers are load-bearing; splitting build-deps from runtime-deps would risk the source-build path for marginal size savings. Left as-is by design. ### The LLVM triple and the compiler-rt directory LLVM creates `lib/clang//lib//` from the triple it was **configured** with (`LLVM_DEFAULT_TARGET_TRIPLE`), while the clang **driver** searches that path under the triple it normalizes the target to. The two spellings must be the same string or the runtimes are installed where nothing looks for them. The symptom is specific and easy to misread. A plain compile and link succeed, because they need no compiler-rt; anything that does — `-fprofile-instr-generate -fcoverage-mapping`, `-fsanitize=address` — fails at link with ``` ld.bfd: cannot find /usr/local/llvm-target/lib/clang/23/lib/x86_64-unknown-linux-gnu/libclang_rt.profile.a ``` while the file is really at `.../lib/x86_64-linux-gnu/`, and `clang -print-runtime-dir` reports `(runtime dir is not present)`. gcc jobs in the same lane pass, which is the corroboration: only clang needs compiler-rt. `llvm_cross_clang_triple` in `02-toolchain/llvm-cross.sh` derives the LLVM spelling from the Debian multiarch one by inserting the vendor field (`x86_64-linux-gnu` → `x86_64-unknown-linux-gnu`). It is derived rather than listed because it *was* a per-arch `case` list, and that list normalized arm64 and riscv64 and never mentioned amd64 — so x86_64 alone was configured with the Debian spelling and shipped a clang whose runtime directory the driver could not find. A list can miss an arch; a derivation cannot. A string that is not a Debian multiarch triplet is refused, because falling through to the input is exactly what produced the wrong value. `test-llvm-cross-stanza.sh` pins all three arches by name. ### Advertised version keys (`advert-keys`) `linux/scripts/verify_advertised_keys.py` globs `linux/Dockerfile.*` for every version-shaped `ENV`/`ARG` and fails when one is neither checked by the smoke's advertised-vs-actual gate (`_ADVERTISED_VERSION_KEYS`) nor listed in `EXCUSED` with a reason. A stale excuse fails too, so the table cannot rot. The smoke's HAVE side must be an *independent* measurement, never a re-read of the value the image advertises. `VULKAN_VERSION` used to fail this: it parsed the version out of `/opt/vulkan/active`, a directory named by the very ARG being checked, so the row could never disagree. It now reads the loader (`vulkaninfo`), falling back to `VK_HEADER_VERSION` in the SDK header. Both sides are normalised before comparison: a leading `v` is stripped from the advertised git tag, and the measured value is cut to its numeric prefix so a `.dev0+` trailer (as `iree-base-runtime` carries) still compares equal. ### Distro package names (`pkg-names`) `linux/scripts/verify_package_names.py` resolves every package name the tree asks apt for against the live Ubuntu indices. Two properties matter: - A **partial** index fetch is not a pass. If any component (`main`, `restricted`, `universe`, `multiverse`) fails to download, the truncated name set is discarded and not cached — otherwise a mirror hiccup would report live packages as dead. - The vendor-repo exemption applies to vendor-shaped **names**, not whole files. `setup-rocm-repo.sh` installs `curl`, `gpg` and `ca-certificates` from plain Ubuntu before it adds the vendor repo; those stay failable. Offline is a `SKIP`, never a pass. ### Runtime image push retries `01-core/runtime-build-fns.sh` retries a runtime image push, because a ~8 GiB wrapper upload can reset mid-transfer. Only the upload repeats: `PUSH_MAX_ATTEMPTS` (default 4) and `PUSH_RETRY_BASE_SECS` (default 15 s). A **permanent** failure is not retried. An unpushable `not found` — the build never created the tag — once burned all four attempts and buried the real cause in retry noise. The classification comes from the cross lane's `_cross_stage_push_error_is_transient` (`01-core/cross-stage-build.sh`); when that helper or the temp log is unavailable, every failure is retried as before. ### cross-apt: pkg-config libdir resolution DUP1: was a hand-rolled uname->triplet case here. platform.sh's arch_deb_multiarch_triplet_for is the SSOT. Mount/bake map independently RE-AUDITED 2026-08-24 (grep of every RUN/COPY block referencing this file — do not re-litigate without re-running that grep): platform.sh is co-mounted in ALL 5 Dockerfile.toolchain RUNs that mount this file (blocks at 70/124/187/245/266) and in both Dockerfile.media litert RUNs (per-file mounts @414, whole-01-core mount @432). Dockerfile.sdk is the ONLY image that bakes this file (COPY block @91) and bakes platform.sh beside it (:60). Dockerfile.android and Dockerfile.torch do NOT ship cross-apt.sh at all (an earlier note listed them; that was vacuous — torch bakes cross-env.sh WITHOUT this file, so sourcing cross-env.sh there dies loudly at its `source cross-apt.sh`, never reaching this fallback silently). Plus cross-env.sh:10 and 01-core/common.sh:22 both source platform.sh before this file, and the missing-helper branch below still warns loudly if a future RUN forgets the co-mount. (Only apt_sources_set_architectures is contracted to work when cross-apt.sh is sourced STANDALONE — see 02-toolchain/android-sdk.sh:8 — and it stays dependency-free.) Update, 2026-09-25: `Dockerfile.android` now bakes `cross-apt.sh` too, for `cross_ensure_installed_foreign_arch_sources`, with `platform.sh` COPY'd beside it. So "Dockerfile.sdk is the ONLY image that bakes this file" and "Dockerfile.android ... do NOT ship cross-apt.sh" above no longer hold; the co-mount rule they illustrate still does. The two ways this lookup can come back empty are NOT the same failure and must not degrade the same way. A single `... 2>/dev/null || true` collapsed both into silence: * helper MISSING (rc 127, command not found) = a WIRING bug — platform.sh was not co-mounted/sourced. Every host-arch pkgconfig dir silently vanishes from PKG_CONFIG_LIBDIR and host tools (xcb & friends) start failing to configure with no hint as to why. Say so, loudly. * helper present, arch UNRECOGNISED (rc 1) = expected degradation. The candidate is skipped and we stay quiet, exactly as before. ### Rewriting a deb822 sources file in place `_apt_sources_rewrite ` (cross-apt.sh) is the one owner of that operation; `apt_sources_set_architectures` and `cross_align_host_apt_pockets` both go through it, and it was extracted the moment the second one existed — the two had drifted into an eight-line identical run before the dupes gate said so. Three properties are load-bearing, each paid for by a real failure: * **The temp file lives NEXT TO the target, not in `$TMPDIR`.** Several RUNs mount `/tmp` as a tmpfs (Dockerfile.sdk, Dockerfile.toolchain), which turns the final `mv` from an atomic rename into a cross-device copy. The random suffix keeps it out of apt's own `*.sources`/`*.list` globs, and out of `cross_prune_foreign_arch_apt_sources`' `ubuntu-ports*.sources` glob, for the moments it exists. * **Cleanup is EXPLICIT at every exit, never a trap.** This file is SOURCED, and a trap armed inside a sourced function stays armed and re-fires on the CALLER's return — the parallel-loop.sh RETURN-trap incident, which turned a fully green chain into exit 1. * **A failing awk must not reach the `mv`.** The original ran `mv` unconditionally, so a failing awk (ENOSPC on the temp, a shadowed or broken awk) REPLACED the host sources file with awk's truncated output and still returned 0. Every later `apt-get` in that RUN then died with "Unable to locate package" and no trace of the real cause. `mktemp` creates 0600 and `mv` carries that mode onto the target, so the helper chmods 0644 first — apt sources are 0644 root:root everywhere else in `/etc/apt/sources.list.d`. The `` argument only names the function in the two error messages; it keeps a shared helper from reporting failures under its own name. ### Host and target apt sources must expose the same pockets VK2/riscv64, 2026-09-08. `qt6-base-dev:riscv64` refused to install with ``` libappstream5:riscv64 : Depends: libcurl3t64-gnutls:riscv64 (>= 7.63.0) but it is not going to be installed libproxy1v5:riscv64 : Depends: libcurl3t64-gnutls:riscv64 (>= 7.16.2) but it is not going to be installed E: Broken packages ``` and every dependency in that list existed in ports at a satisfying version, so the message named the symptom and not the cause. The cause was a **pocket asymmetry** between the host and target sources, both written by `ubuntu_write_deb822_source`, whose 5th argument decides whether `-security` is appended: * `02-toolchain/python/build_python.sh` (`_python_cross_enable_multiarch_apt`, compiler stage) wrote `ubuntu.sources` for amd64 with `0` and `ubuntu-ports.sources` for arm64/riscv64 with `1`. * `Dockerfile.media` did the same for its own reset. `libcurl3t64-gnutls` is `Multi-Arch: same`, so the amd64 and riscv64 copies must be the **identical version**. Ports could see `8.18.0-1ubuntu2.5` (security); amd64 topped out at `8.18.0-1ubuntu2.4` (updates). No common version existed in apt's view, so the foreign-arch library became uninstallable — and with it every `Multi-Arch: same` library that has ever had a security-only upload. It sank the caps viewer because `qt6-base-dev:riscv64` pulls `libappstream5`/`libproxy1v5`, but nothing about it is Qt- or riscv64-specific. Proven by A/B on `:base`, same image, one line different: | amd64 stanza | amd64 candidate | riscv64 candidate | `qt6-base-dev:riscv64` | | --- | --- | --- | --- | | without `-security` | 8.18.0-1ubuntu2.4 | 8.18.0-1ubuntu2.5 | `E: Broken packages` | | with `-security` | 8.18.0-1ubuntu2.5 | 8.18.0-1ubuntu2.5 | installs | `archive.ubuntu.com` carries `-security` for amd64 (HTTP 200, `Architectures: amd64 amd64v3 arm64 armhf i386 ppc64el riscv64 s390x`), so the `0` bought nothing — it was not a mirror-coverage workaround. Both call sites now pass `1`, and the `mirror-consistency` gate asserts the invariant directly rather than the two literals: it runs the real `ubuntu_write_deb822_source` for a host arch and a ports arch and requires the two suite sets to match, then greps the call sites so a future writer cannot reintroduce the skew. **The rule is symmetry, not `-security` as such** — a host source that gains a pocket the ports source lacks breaks co-installation in exactly the same way, just in the other direction. Fixing the writers only helps a stage built after them, so the invariant is also enforced at the point of use: `cross_align_host_apt_pockets` (cross-apt.sh) runs inside `cross_configure_foreign_arch_apt_sources`, right after the `Architectures:` rewrite, and appends `-security` to the host sources when no stanza carries it. It is a no-op on the stock Ubuntu layout, where the pocket lives in its own `security.ubuntu.com` stanza — appending a second copy there would double-define the target and spam every apt call with "configured multiple times". When it does change the file it clears `_CROSS_ENV_APT_UPDATED`, because `install_target_packages` skips `apt-get update` when the flag is already set and the new pocket would otherwise never be fetched. One consequence worth stating: a custom `FAST_UBUNTU_MIRROR_URL` must now carry the security pocket for the host arch. Ports mirrors already had to (the ports source has always been written with it), so this is the same requirement applied to both halves rather than a new one. Three AS1 neighbours closed on 2026-09-17, all latent while every cross stage builds on `linux/amd64`: * **The host stanza follows the BUILD arch.** `Dockerfile.media` and `build_python.sh` pinned it to `amd64` regardless of the machine; a non-amd64 host therefore had no source for its own packages, and deb822 `Architectures:` replaces rather than intersects. They now use `ubuntu_arch_uses_ports` like `cross-apt.sh`, so the two halves cannot answer differently. * **An amd64/i386 cross TARGET gets an archive, not just an architecture.** `cross_configure_foreign_arch_apt_sources` returned early whenever the target was not a ports arch, so `dpkg --add-architecture` ran and nothing wrote a source. It now writes `ubuntu-archive-.sources` for amd64/i386 and `ubuntu-ports-.sources` for everything else, from the same table; the prune glob covers both prefixes, and `386` is in the archive arm because that is `arch_normalize`'s canonical spelling for i386. * **`use-fast-ubuntu-mirror.sh` rewrites `-security` by default.** With the fast mirror on and the knob at `false`, the host `-security` stayed on security.ubuntu.com while the target pocket came from the fast ports mirror, and `cross_align_host_apt_pockets` cannot see it because it compares suite names, never URIs. The default now matches `bootstrap_ca`'s (true, decided for the same reason); `FAST_UBUNTU_REWRITE_SECURITY=false` stays the explicit opt-out and the only way to keep the stock security host. A mirror without the security pocket for the host arch fails the media stage either way. Note what did NOT catch it: the failure is invisible to every static gate (the sources are correct in isolation), invisible to a same-arch build, and invisible until a `Multi-Arch: same` package actually receives a security-only upload. Before 2026-09-08 the two pockets happened to agree. ### `resolve_ci_version`: the version a CI run stamps `02-toolchain/rust/version_util.sh`. Resolve the version a CI run should stamp, from the sources every consumer has: a VERSION.txt at the repo root, else the ref name, else the run number. **A `Cargo.toml` is a version file too (2026-10-15).** Handed a file named `Cargo.toml`, it reads `[workspace.package]`'s `version`, else `[package]`'s, and never a dependency's or a member's `version.workspace = true`; with neither, the ref name and run number follow as below. A Rust repo then keeps its version in one place: OxidANT dropped its `VERSION.txt` for it, and passes `version-file: Cargo.toml`. `linux/scripts/tests/test-version-util.sh` holds both file kinds and the fallbacks. Lifted out of a consumer (OxidANT's scripts/compute_version.sh) because every repo with a CI lane needs exactly this and had to reimplement it around the primitives above. The guards are load-bearing, in this order: * only the FIRST NON-EMPTY line of VERSION.txt counts, CR-stripped and trimmed - a file written on Windows otherwise yields "2.3.4\r", which is not a valid version anywhere downstream; * a leading "v" is dropped, so a v-prefixed tag works as a ref source; * anything that still does not START WITH A DIGIT falls back to the run number, which keeps a branch name like "feature/x" from becoming a version. That guard is also why normalize_version's own "v" handling is unreachable from here. One deliberate behaviour change from the consumer version this replaces: `read` returns non-zero on a final line with no trailing newline, so a plain `while IFS= read -r line` DROPS it. A VERSION.txt written without a trailing newline was therefore ignored outright and the run number stamped instead - silently, since the fallback looks like a normal result. The `|| [[ -n "$line" ]]` guard below reads that last line. ### 03-media: data-driven per-arch skip flags (`media_load_arch_flags`) The repeated boolean per-arch skip decisions in the media install/build scripts (e.g. "skip target Csound packages on riscv64/arm64 cross") are consolidated into declarative flag files next to this one: arch-flags-{amd64,arm64,riscv64}.env — KEY=value lines only (MEDIA_SKIP_*; 1 = skip, 0/unset = do not skip), with the per-flag justifications kept as comments in those files. media_load_arch_flags resolves the effective target arch — cross_target_arch when a cross build is active, otherwise the native amd64 defaults — and sources the matching flag file if present. A missing file simply leaves every MEDIA_SKIP_* flag unset (= do not skip), matching the old is_cross_*-style helpers which only ever skipped on active cross builds. Consumers: - Scripts using media_common_init get the flags automatically (called at the end of media_common_init, after cross-env.sh is loaded). - install-deps-preamble based scripts source this file (container path /opt/scripts/03-media/core/common.sh first, repo layout second) and call media_load_arch_flags themselves; the preamble has already provided the cross_build_is_active / cross_target_arch helpers by then. ### smoke-torch-venv: what the venv assertions cover Assert installed ML-stack versions match their pins. Two authorities, but NOT unioned (GENAI-DRIFT, 2026-08-23): whichever one OWNS the package decides. - versions.env build-pin -> packages we BUILD or force-reinstall from a LOCAL wheel (riscv64 torch/vision, source-built onnxruntime, ai-edge-litert, onnxruntime-genai). The pin WINS outright, on every arch that builds it; the lock's opinion is printed but not accepted. - uv.lock -> everything uv resolves and we do not build (numpy/pillow/ contourpy, the amd64/arm64 torch/vision/onnx wheels when unpinned). The old union accepted either, which is exactly how arm64 shipped onnxruntime-genai 0.14.0 (from the lock) against a v0.15.2 build pin and still printed OK. That tightening exposed a drift whose PRODUCER-side fix was still open, and this assert is a hard release gate, so the known case was carried in KNOWN_DRIFT below: an exact (dist, arch, installed, expected) quadruple, dated, naming its backlog item, printed as a loud `!!` on every run and counted in the summary. Anything that is not that exact quadruple still FAILS. (KNOWN_DRIFT is empty on 2026-09-25: no case is tolerated.) It uses each module's __version__ (the actual runtime version) -- which for onnxruntime intentionally differs from its pip dist metadata (source-built lib vs locked wheel; the build pin governs). Also asserts the torch/vision build VARIANT (+cpu/+cu130/+rocm7.1) matches PYTORCH_EXTRA and that OpenCV's major matches OPENCV_VERSION, and runs the cv2 media backends for real (SMOKE-DEPTH a). ### smoke-media: the mandatory-plugin gate and its build-sandbox deferrals Mandatory-plugin gate (smoke-depth R1): meson `enabled` guards CONFIGURE, but a plugin that ships and then fails to dlopen was only a WARN-count. A present-but-unloadable plugin is exactly the observed class (webrtcbin2→librice-proto, gtk4→vkCreateWaylandSurfaceKHR). The `libav` plugin is special: this project's gst-libav links the source-built FFmpeg libav* (incl. libavfilter, which NEEDs the bundled libtensorflow.so.2). Those resolve only once configure-runtime.sh has wired the loader — the SAME reason the ffmpeg binary itself is deferred in the build sandbox below. So gate `libav` on ffmpeg executability HERE: if ffmpeg cannot run in this environment (sandbox), a libav load failure is that same deferral (INFO; re-tested by the packaging-stage smoke, Dockerfile.package, where the loader is wired); if ffmpeg DOES run here but libav still fails, that is a real defect. opencv/onnx have the SAME class of build-sandbox issue, not a link to ffmpeg: the opencv plugin links pass-2 OpenCV, which links the source-built GStreamer (a circular dep the build sandbox can't close); the onnx plugin links libonnxruntime.so which transitively needs libstdc++.so from the source-built GCC (a path the flat NEEDED scan in validate-media-runtime.sh doesn't catch but the dynamic linker hits at dlopen). Both pass validate-media-runtime's NEEDED scan and the runtime/packaging smoke (Dockerfile.package) where the loader is fully wired. Gate on the ffmpeg-executability proxy: if ffmpeg can't run here (build sandbox), defer opencv/onnx just like libav. ### slang-compile: shader compilation contract Caches the subdirectories under the source tree in SLANG_COMPILE_SUBDIRS, reused for every -I expansion. ORDER IS LOAD-BEARING and was previously whatever `find` happened to emit - i.e. filesystem order, which differs between a developer's ext4 checkout and the CI runner's overlayfs. `import ` resolves to the FIRST .slang on the -I list, so any two modules sharing a basename resolved differently on different machines. BeschleunigerBallett has exactly that: common/noise.slang (simplex noise + fbm) and compute/noise.slang (a noise-volume kernel). On CI the compute/ one won and tests/noise_test.slang, which wants the common/ one, failed to build every clang lane with error[E30015]: undefined identifier 'snoise' while the same tree built fine locally. Two rules, both deliberate: 1. sort, so the answer is reproducible anywhere; 2. hoist a top-level common/ ahead of the rest, because "shared module lives in common/" is already this driver's assumption (see the comment on the -I expansion below) and is the documented contract on the consumer side too - BeschleunigerBallett's buildIntegritySuite.cpp resolves imports with an explicit "then a common/ preference". Alphabetical order happens to give the same answer for common/ vs compute/, which is precisely why this must not be left to luck. The generated output tree is excluded: it holds no .slang sources, and feeding a build directory back in as an include path can only add ambiguity. ### `alt_install_and_set`: register an alternative and select it `01-core/common.sh`. Register an alternative and immediately select it. Runs privileged via run_priv (honors ${SUDO}). The --set is tolerant: a --set of the path just --installed effectively never fails, and a spurious failure must not abort a build running under `set -e`. Usage: alt_install_and_set [priority] \ [--candidate ]... # extra paths to search for the binary [--slave ]... # slave alternatives (multi-binary groups) Backward compatible with the historical ` [priority]` form. Extensions (so the 02-toolchain scripts can converge onto this helper): --candidate Add to the search list. The FIRST executable among {, candidates...} is the one registered. When one or more --candidate are given and NONE (including ) is executable, the function is a no-op (nothing to register). With no --candidate, is registered verbatim exactly as the original 3/4-arg form did (no executability check). --slave L N P Pass a `--slave L N P` group through to --install, so one call can register a multi-binary group (e.g. clang + its clang++/clang-format/… slaves). ### Swapping the native GCC in the shipped image --- Make the foreign-arch native GCC search runtime system headers --- The relocated Canadian-cross GCC keeps its compile-time --native-system- header-dir (/usr/${triplet}/include) baked in; that path is absent in the runtime image, so a bare `gcc hello.c` cannot find . The profile.d block above only helps make-style builds -- a bare `gcc`/`g++` reads neither CFLAGS nor CPPFLAGS, and CPATH cannot satisfy the C++ `#include_next`. We fix this with thin WRAPPER scripts that prepend the system include dirs on the *command line* (-idirafter, appended AFTER all built-in dirs so it never shadows libstdc++'s own headers and satisfies both C `<...>` and C++ `#include_next`). This is NOT done with an installed `specs` file: any installed specs file RESETS the driver's dynamically-computed link specs -- it silently drops `-lgcc_s` from `*libgcc` and `--eh-frame-hdr` from the EH link spec -- which makes every C++ program that throws terminate at runtime (rc=134, catch never fires) even though it links. Restoring those specs by hand is whack-a-mole and fragile under QEMU. Command-line -idirafter leaves the link specs untouched: validated on riscv64 that a bare `gcc hello.c`, simple C++, AND exception-throwing C++ (throw/catch, STL sort) all compile, link, and run correctly. amd64 never reaches this block (host GCC, no swap). ### The native GCC ships libsanitizer **Symptom.** In the arm64 or riscv64 image, a `-fsanitize=address` build stops with `fatal error: sanitizer/common_interface_defs.h: No such file or directory`. abseil's `dynamic_annotations.h` includes that header. With the header in place, the link still cannot find `libasan`/`libubsan`. The amd64 image is fine. Found on 2026-09-23 through AccelerANTgine's `linux-debug-GNU` preset, on the published `:latest` (index `ec4bb68b`, arm64 manifest `b867d353`, GCC layer `e95f585a`). **Cause.** Only amd64's `/opt/gcc-16.2.0` is a full `make` build. On arm64 and riscv64 it is the Canadian-native GCC (`gcc.sh` `build_canadian_native_gcc_for`: build is the build host, host and target are the image's arch), which `swap-native-gcc.sh` copies over the prefix in the android stage. `build-gcc.sh` built and installed only libgcc, libstdc++-v3 and libatomic whenever `--target` was set. So that GCC had no `include/sanitizer/` and no `libasan`, `libubsan`, `liblsan` or `libtsan`. **Fix.** When `--host` equals `--target`, `build-gcc.sh` adds `target-libsanitizer` to the `make` and `make install` targets (`_gcc_extra_target_libs`). No configure flag changes, and `gcc.sh` and the Dockerfiles are untouched. Which GCC takes which path: | GCC | `--target` / `--host` | libsanitizer | | --- | --- | --- | | Build-arch host GCC (`build_host_gcc`) | neither | already built by the full `make` | | Plain cross (`build_cross_gcc_for`), `-gcc` in the shared prefix | target only | not built, unchanged | | Canadian native (`build_canadian_native_gcc_for`), the arm64/riscv64 image's `cc` | both, equal | **built** | | A Canadian with host != target | both, different | not built; nothing in the tree produces one | In a Canadian cross the target libraries are compiled by the already-installed cross `-g++` (`CC_FOR_TARGET`/`CXX_FOR_TARGET`), not by the in-tree `xgcc`. libsanitizer is a raw-C++ target module like libstdc++-v3, which already builds this way here, and it uses the same `libc6-dev--cross` sysroot. Its runtime lands next to libstdc++: `lib64` on arm64 and amd64, `lib` on riscv64. Do not pass `--enable-libsanitizer`: it skips GCC's per-target support check. What GCC 16.2.0 builds per target (`libsanitizer/configure.tgt`): | Target | asan | ubsan | lsan | tsan | hwasan | | --- | --- | --- | --- | --- | --- | | x86_64 | yes | yes | yes | yes | yes | | aarch64 | yes | yes | yes | yes | yes | | riscv64 | yes | yes | yes | yes | no: GCC has no riscv memory tagging, so `-fsanitize=hwaddress` is rejected | **Three gates, and what each sees:** - `build-gcc.sh` fails the GCC build when the install left no `sanitizer/common_interface_defs.h` or no `libasan.so.*`. libsanitizer's own configure switches itself off without an error when a probe fails (`SANITIZER_SUPPORTED=no`), and then installs only `libsanitizer.spec`. - `swap-native-gcc.sh` checks the shipped `/opt/gcc-` on every build-host shape: the header, and `libasan`, `libubsan`, `liblsan` and `libtsan` (plus `libhwasan` on amd64 and arm64), each with the image's ELF machine. That includes the native Jetson and X100 lanes, whose own full-make GCC builds libsanitizer by default. It looks for the header under any triplet, because the full make's is config.guess-shaped (`x86_64-pc-linux-gnu`) and the Canadian one is not (`aarch64-linux-gnu`). It accepts the libraries in `lib64` or `lib`. - The wrapper smoke (`validate-compilers.sh`) compiles and links a TU that includes the header, with `-fsanitize=address,undefined`, and requires `libasan` and `libubsan` in `NEEDED`. It never runs the binary. `smoke-runtime-image.sh`'s compiler battery does, with `ASAN_OPTIONS=detect_leaks=0`, but only where the build host's arch is the image's. ASan and LSan are unreliable under qemu-user (LSan needs ptrace, and the shadow reservation can fail), so an emulated arch prints a skip line, never a pass. **Why the smoke TU shifts by a variable.** Both smokes return `(8 >> c) == 4`. The variable shift exponent is the one UBSan check that TU keeps. GCC 16 compiles C++ as C++20 by default (`c-opts.cc`), and C++20 defines a left shift of a signed value, so `c-ubsan.cc` emits no shift-base check; a constant exponent folds away as well. A TU with no UBSan call has `libubsan` in `NEEDED` only because the image's GCC links without `--as-needed`. Under `--as-needed` the linker drops it, and the smoke fails although the runtime is there. Measured on 2026-09-24 with the GCC 16.2.0 of a local amd64 `:latest-cross`: the first version's `(c << 3)` TU has no `__ubsan_` reference, and its smoke fails under `-Wl,--as-needed`; the `(8 >> c)` TU passes. `linux/scripts/tests/test-native-gcc-sanitizers.sh` covers all of them, and the `native-gcc-san.*` mutations prove each can fail. The two smokes run against a modelled GCC: its `g++` writes a script whose `NEEDED` lines the `readelf` stub prints, and knobs give the defect's two halves (no header, no `libasan`) and an `--as-needed` link. The battery's container body runs on that model, so its RUN gate and its report are tested by behaviour, not by text. Where the host's `g++` links `-fsanitize`, the wrapper smoke also runs against the real compiler. **Cost.** `build-gcc.sh` is in the compiler image's closure, so the change re-keys the whole Linux chain from the compiler stage down, on every arch. Base is untouched. The nvidia and rocm variants were never affected: each builds for its build host's own arch, whose GCC is the full make. Their chains re-key on their next run, because they build on the new shared sdk, with no change in their GCC. The GCC RUN gains two libsanitizer builds, about 190 source files each (171 on riscv64), each compiled twice by libtool. Its time is not measured. The shipped `/opt/gcc-16.2.0` grows by about 79 MB uncompressed on arm64, the size of amd64's set (79.3 MB in layer `236e40c6`, 62.7 MB of it the unstripped `lib*san.a` without hwasan). riscv64 has no hwasan, but its static archives are about 3x amd64's (`libstdc++.a` 171 MB against 55 MB, every archive in the prefix 269 MB against arm64's 90 MB), so expect roughly 190-200 MB there. The compiler, sdk, media and android images carry both Canadian prefixes, so they grow by roughly 270-280 MB, but they are not shipped. None of these figures is measured on a built image, and the compressed delta is not known. A partial rebuild on an image from before the change fails, by design, instead of shipping without the runtime. On arm64 and riscv64 that is two places: a `--from-stage sdk|media|android` run on the old compiler digest stops at the android swap, and a runtime-lane run on the published android images (`--from-stage runtime`, or `build-runtime-artifacts.sh` / `build-runtime-manifest.sh` directly) stops at the wrapper smoke with `COMPILER FAIL [gcc-sanitizers]`. So no arm64 or riscv64 wrapper can be rebuilt from a partial stage until a chain from the compiler stage down has run. `WRAPPER_SMOKE_GATE=0` gets past the second, and ships the defect. amd64 is not affected. **Not covered.** The amd64 image's plain cross compilers still have no target `libasan`, so `aarch64-linux-gnu-g++ -fsanitize=address` from amd64 still fails. The arm64 and riscv64 GCCs still lack libgomp (`omp.h`), libitm and gfortran. Both are recorded as decided in [`image-decisions.md`](image-decisions.md) § Deliberate — not gaps (CON22, 2026-09-26): no consumer uses them. TSan on riscv64 needs an sv39 or sv48 VMA, which is unverified on the X100. ### The native GCC has multiarch **Symptom.** In the arm64 image, CMake configured with GCC does not find libraries that sit in `/usr/lib/aarch64-linux-gnu`. BeschleunigerBallett's two GNU presets stop at `Could NOT find X11 (missing: X11_X11_LIB)`, while the same run's Clang preset finds X11 (run 36042437555). The amd64 image is fine. **Cause, measured 2026-09-25** on the published `:latest` (index `ec4bb68b`, the arm64 variant under qemu): | | arm64 `gcc` | arm64 `clang` | amd64 `gcc` | | --- | --- | --- | --- | | `gcc -print-multiarch` | empty | (no such option) | `x86_64-linux-gnu` | | CMake `CMAKE_LIBRARY_ARCHITECTURE` | empty | `aarch64-linux-gnu` | `x86_64-linux-gnu` | | `/usr/lib/` among the implicit link dirs | no | yes | yes | | `find_package(X11)` | `X11_X11_LIB-NOTFOUND` | found | found | CMake takes the architecture from an implicit link directory that ends in `/lib/`, and `find_library` searches `/lib/` only once it has one. The arm64 GCC is the Canadian native (the libsanitizer section above), and `build-gcc.sh` configures every `--target` build with `--with-native-system-header-dir`. GCC 16's `gcc/configure.ac` then turns `--enable-multiarch=auto`, its default, into `no` ("disabled auto check (configured with --native-system-header-dir)"). The amd64 GCC is a full `make` without that flag, so its auto-check finds multiarch. Linking still works on arm64 because `swap-native-gcc.sh` points `/opt/gcc-//lib` at `/usr/lib/`, but no path of that shape tells CMake the architecture. **Fix.** When `--host` equals `--target`, `build-gcc.sh` also passes `--enable-multiarch` (`_gcc_native_multiarch`, sharing `_gcc_is_canadian_native` with libsanitizer's helper). With `--with-sysroot=/` the driver then puts `/lib/` and `/usr/lib/` on the link line wherever they exist, as a distro GCC does. The flag changes only what the shipped driver does: a Canadian build compiles its target libraries with the installed cross `-g++`, never with the GCC being configured. Plain cross compilers keep their cross layout. **Gate.** `swap-native-gcc.sh` asks the relocated `/opt/gcc-/bin/gcc` for `-print-multiarch` and stops the build unless it prints the image's triplet. Where the build host cannot run the target binary it logs that and moves on, the tolerance the compile smoke next to it already has. `linux/scripts/tests/test-native-gcc-multiarch.sh` covers the helper, its wiring into the configure line, and the gate; the `native-gcc-ma.*` mutations prove each can fail. It cannot show what GCC's configure makes of the flag; only a chain run shows that. **Cost.** The same re-key as libsanitizer: the compiler stage down, on every arch. The two ship together in the next Linux `:latest` (`BACKLOG.md` CON11). ### Runtime stage context and ancestry annotations Append the image target for a runtime build to the nameref array. RTCACHE3 (root cause of the 2026-08-14 stale-ship saga): this used to emit the annotated `--output type=image,name=,annotation.*` exporter on the push path, on the assumption (see the now-corrected runtime_image_output_arg note) that it was "equivalent to -t ". It is NOT. Verified with a minimal busybox repro on this rootless nerdctl+containerd host: nerdctl build --output type=image,name=X → X is NOT in the local image store nerdctl build -t X → X IS in the local image store The exporter builds the image into buildkit's content store but never lands a local containerd tag. So the freshly built wrapper was invisible: the subsequent `nerdctl push ` (runtime_push_tag) and `nerdctl manifest create ` both resolved the STALE pre-existing local tag from an earlier run, and :latest-cross shipped byte-identical every time (amd64 stuck at 35c1f1df across five rebuilds). The annotations never reached the registry either — every run logged "wrapper tag(s) carry no run-id annotation … provenance unverifiable" — so nothing of value is lost by dropping the exporter. Use plain `-t` on BOTH paths: it reliably creates AND overwrites the local tag, which is what runtime_push_tag + the manifest step consume. Ancestry provenance came back on 2026-08-23 (XC3-INERT) as config LABELS, which ride the image config through `-t` and the push: `append_runtime_image_output` appends `ancestry.sh`'s `ancestry_label_args` (run id, parent digest, parent stage) on both call sites, package and wrapper. ### Measuring the torch RUN's wait before `uv venv` What `RUNTIME_WHEELS_SOURCE=export` is meant to remove, and how to tell whether it did. The two deliveries are in [`linux-cross-builds.md`](linux-cross-builds.md#the-wrappers-wheelhouse-two-deliveries). **Lines every runtime lane prints since 2026-09-24**, in both modes: | Line | Printed by | Says | |---|---|---| | `[runtime-timing] arch= step= secs= rc=` | `_runtime_timed`, host | wall time of `base`, `package`, `smoke`, `wrapper`, and `wheels-export` in export mode | | `[wheels] : files= bytes= sha256= (exported from )` | `lib-runtime-wheels.sh`, host, export mode | what was staged, and from which image | | `[torch-run] start epoch= arch=` | the torch RUN's first statement | when the RUN's process started | | `[torch-venv] wheelhouse files= bytes= sha256=` | `setup-torch-venv.sh`, before anything prunes | what the RUN mounted | | `[torch-venv:timing] epoch= +s` | `setup-torch-venv.sh`, after each step | the RUN's own timeline | `sha256` in both wheelhouse lines is the same function: sha256 over ` ./` lines for every file, sorted by path. So the host's `[wheels]` line and the RUN's `[torch-venv] wheelhouse` line must match, and image mode's RUN line must match export mode's. **Telling the causes apart.** The RUN's first-line elapsed time does NOT separate them. A cold snapshot and a lock wait both show a late first line, and part of a cold snapshot's cost lands inside the RUN vertex (8.2 s of a 27 s wait in the 2026-09-23 BuildKit probe on Rancher). Compare the `wheels-source` vertex with the RUN vertex's start instead: ```bash LOG=out/build-logs/wheels-ab-.log # the tee'd transcript: the runtime lane writes no stage log grep -nE '\[runtime-timing\]|\[wheels\] |\[torch-run\] start|\[torch-venv(:timing)?\] ' "$LOG" grep -nE '^#[0-9]+ \[wheels-source 1/1\] FROM' "$LOG" # its vertex id N, per build grep -nE '^#N (resolve|sha256:|extracting|DONE|CACHED)' "$LOG" grep -nE '\[disk-(watch|reclaim|buildkit)\]' "$LOG" # a prune between package and wrapper ``` - `extracting` lines under `wheels-source`, finishing just before the RUN vertex starts: the snapshot was cold. Export mode removes this from the wrapper build. - No extraction, and the RUN vertex started long before its first line: a lock. The next suspect is the `sharing=locked` apt cache mounts, not the wheelhouse. - In export mode the wrapper build shows no `wheels-source` pull or extraction at all. It transfers the small `runtime_wheels` context instead. **Before a lane, on the build host** (edits nothing, about a minute): this runs the same mount, cold and then warm, and checks ownership inside the RUN. ```bash mkdir -p /tmp/wheels-probe && cd /tmp/wheels-probe printf '%s\n' 'ARG WHEELS_IMAGE' 'FROM ${WHEELS_IMAGE} AS wheels-source' 'FROM busybox AS probe' \ 'RUN --mount=type=bind,from=wheels-source,source=/opt/wheels,target=/opt/wheels,rw echo "[probe] start $(date -u +%s)"; ls -ln /opt/wheels | head -3' > Dockerfile for i in cold warm; do nerdctl build --progress=plain --platform linux/arm64 --target probe \ --build-arg WHEELS_IMAGE=ghcr.io/kataglyphis/kataglyphis_beschleuniger:cross-android-arm64 . 2>&1 | tee "$i.log" done ``` The mount's `,rw` arrived on 2026-09-22 (fbc91813), between the fast and the slow samples. The Rancher probe found no cost for rw over ro on a warm source, but that was BuildKit 0.28.1 with a containerd worker, while the host runs a rootless OCI worker (0.31.2). On the host, drop `,rw` from the probe for one cold run to settle it. **The A/B.** Run it through the chain: the runtime lane's disk watch (`_chain_disk_watch_start runtime` in `build-cross-chain.sh`), whose mid-lane prune is the suspected eviction, runs only there. Both modes read the same published android, and nothing is published: ```bash for m in image export; do CROSS_LOCAL_CONTEXT_HANDOFF=0 RUNTIME_WHEELS_SOURCE=$m \ bash linux/scripts/build-cross-chain.sh --only runtime --no-push \ --target-arches amd64,arm64,riscv64 2>&1 | tee "out/build-logs/wheels-ab-$m.log" done ``` - `--no-push` hands the lane `--skip-manifest` and no `--push`, so no wrapper and no manifest leave the host. - `--only runtime` builds no android: the chain pulls the published `cross-android-`, and both modes resolve its registry digest. - `CROSS_LOCAL_CONTEXT_HANDOFF=0` is required. With the handoff on, the lane looks for an android layout this run never wrote. On the cross host `export` would also be refused there, as on every `--no-push` chain whose android tag is published ([why](linux-cross-builds.md#the-wrappers-wheelhouse-two-deliveries)). **Mechanism check only.** `build-runtime-manifest.sh` on its own runs the same two paths without the disk watch. Nothing then prunes between the package build and the wrapper, and image mode will usually show no wait at all. Use it to see that `export` stages, seals and mounts the wheelhouse and that both modes print one digest, never to judge the wait: ```bash for m in image export; do RUNTIME_WHEELS_SOURCE=$m bash linux/scripts/build-runtime-manifest.sh \ --image ghcr.io/kataglyphis/kataglyphis_beschleuniger:latest \ --target-arches amd64,arm64,riscv64 --skip-manifest 2>&1 | tee "out/build-logs/wheels-mech-$m.log" done ``` BuildKit keys a bind mount by its content, so the second mode may find the first one's torch RUN in the cache and skip the step you want to time. When the RUN shows `CACHED`, add `RUNTIME_NO_CACHE=1`, which re-runs the package build as well. The other way is to compare two real pushing chains, the first in `image` mode and the next in `export` mode; not two `--no-push` ones on the cross host, where `export` is refused. Accept `export` when, per arch: - the `image`-mode run reproduced the wait: `extracting` lines under its `wheels-source` vertex, finishing just before the RUN vertex starts, often after a `[disk-watch] G free < G DURING a stage` line (BuildKit's own GC evicts silently). Without that the arch is inconclusive, not a pass: a run that evicted nothing has no wait to remove; - the export-mode wrapper build has no `wheels-source` pull or `extracting` line; - `[torch-run] start` is within 10 s of the RUN vertex's start; - both modes print the same `[torch-venv] wheelhouse` digest, and that digest equals the host's `[wheels]` line; - `verify-shipped-wrapper.sh`, the ORT census and the boot smokes pass in both modes. Report the result per arch. amd64 waited 96 s even on the fast day, so its share may come from something else. One `export` run of a full `--no-push` chain on a native host (Jetson or X100) whose `-host` android tag was never pushed also exercises the layout digest check that only `ARTIFACT_CONTEXT_ROOT` reaches. On the cross host that check can only refuse. ### `cross_compile_cmake_lib_from_source`: a small library from source `01-core/cross-env.sh`. cross_compile_cmake_lib_from_source NAME URL[|MIRROR...] INSTALL_PREFIX SENTINEL [EXTRA_CMAKE_ARG...] URL may list '|'-separated fallback mirrors, tried in order until one lands (guards against a single-host outage silently disabling the lib). Each mirror is either a plain https tarball or a `git+#` spec (shallow git clone — reliable where the buildkit RUN can git-clone github.com but curl fails). Fetch a source tarball and cross-cmake build+install a small library into the target sysroot. For libraries whose Ubuntu Ports dev package is missing/broken, or whose OpenCV-vendored copy can't cross-build (freetype, libpng). No-op if SENTINEL already exists (e.g. an apt package already provided the lib). Behaviour-preserving extraction of the freetype/libpng blocks that were copy -pasted in 03-media/build/opencv/install-deps.sh: same cross toolchain (${triplet}-gcc/g++), same find-root scaffold. Per-library flags (FT_DISABLE_*, PNG_STATIC/PNG_HARDWARE_OPTIMIZATIONS, ZLIB_*, and any MODE_LIBRARY/INCLUDE override) are passed as trailing EXTRA_CMAKE_ARGs. Fetch goes through download_and_extract for curl --retry + temp-file hygiene (a hand-rolled `curl | tar` had neither). Never hard-fails: a download/build failure logs a WARN and returns 0 so the caller can degrade (e.g. OpenCV falls back to WITH_=OFF) instead of aborting the whole media stage. ### `python_uv.sh`: which interpreter `uv sync` is pinned to Pin the interpreter for the SAME reason uv_pip_install_requirements does, and it is just as load-bearing here: uv honours UV_PYTHON OVER the activated venv. The CI images export UV_PYTHON=/opt/venv/bin/python and run as the non-root user `kataglyphis`, so `--active` alone still resolves to that root-owned system venv and the sync dies with error: failed to remove file `/opt/venv/lib/python3.14/site-packages/...`: Permission denied (os error 13) Observed on both arches in OrchestrANT's lane on 2026-08-11, right after the extras fix let the resolve get this far. --python forces the writable local environment. Consult _CURRENT_VENV_PATH FIRST and VIRTUAL_ENV only as a fallback. The other order made the pin fire backwards: the images ALSO export VIRTUAL_ENV=/opt/venv, and `uv venv` only prints "Activate with: source .../activate" — it does not activate — so a caller that creates .venv_static_analysis never overwrites the inherited VIRTUAL_ENV. The pin then resolved to the very system venv it exists to avoid, logged "uv sync pinned to /opt/venv/bin/python", and died 2.5 minutes later with the exact permission error above (WebDavClient x64, 2026-08-12). _CURRENT_VENV_PATH is set by uv_venv_create/uv_venv_ensure/uv_venv_activate, so it names the venv THIS script owns — which is the one to sync into. ### `cmake-build.sh`: a writable `CARGO_HOME` The :latest-cross image ran as uid 1001 with CARGO_HOME=/usr/local/cargo owned by root, so the Corrosion/cargo half of the configure died with "failed to create directory /usr/local/cargo/registry" - which took the whole Linux lane down when combined with the tee exit-code masking in BeschleunigerBallett's `Linux.yml` (`reusable-linux.yml` since 2026-09-24; fixed there with shell: bash / pipefail). Redirect cargo to a writable home rather than requiring the image to hand us its own. Checked in order: 1. --cargo-cache-dir (CLI arg, survives container restarts when backed by a docker named volume or host bind mount) 2. $CARGO_CACHE_DIR (environment variable override) 3. $CARGO_HOME (image default; /usr/local/cargo, usually read-only) 4. $TMPDIR/cargo-home (fallback, lost when container exits) Probe registry/, not CARGO_HOME itself. /usr/local/cargo is writable in the cross image while /usr/local/cargo/registry underneath it is root-owned (populated by `cargo install cargo-c` during the image build), so the shallow `-w` test PASSED and the build then died anyway with error: failed to create directory `/usr/local/cargo/registry/cache/...` Caused by: Permission denied (os error 13) Same mkdir-then-test idiom the sccache/ccache loop below already uses. ### Per-arch version truth The runtime image ADVERTISES its component versions as env (`CMAKE_VERSION`, `NODE_VERSION`, …) and the shipped-truth probe in `smoke-runtime-image.sh` compares every one of them against the binary actually present. Two of them cannot be equal on every arch: * Kitware publishes no riscv64 CMake archive, so `install_cmake` takes the Ubuntu package there; * Node.js publishes no riscv64 tarball, so `install_nodejs` takes the Ubuntu package there. Both were true long before the probe could see it — the keys only became checkable when `d27cdee1` stopped 11 of 16 gate keys from SKIPping, and the first run that reached the gate failed with `ADVERTISES CMAKE_VERSION=4.4.3 but actually has 4.4.2` (and Node 26.8.1 vs 22.22.1). The label was wrong, not the image. `versions.env` therefore carries `_` overrides next to the base key, marked `# noforward` so they never become build args of their own: ``` CMAKE_VERSION=4.4.3 # noforward CMAKE_VERSION_RISCV64=4.4.2 ``` `append_version_build_args ` prefers the override for that arch, so `--build-arg CMAKE_VERSION=` carries 4.4.3 to amd64/arm64 and 4.4.2 to riscv64, and the image advertises what it has. A name that is *itself* a forwarded key (`GENAI_ALLOW_RISCV64`) is never read as another key's override. These are pins, not excuses: if the Ubuntu archive moves, ADV and HAVE diverge again and the build fails until the override is bumped. **A value whose characters the shell would interpret is quoted in the file and unquoted by the loader.** `CUDA_ARCHITECTURES=80;86;89;90` was written bare, so every plain `source versions.env` — `lint-python.sh` did exactly that to read `RUFF_VERSION` — ran `86`, `89` and `90` as commands and printed three `command not found` lines on every hook run. That reader now goes through `load_versions_env` like every other one, so the noise cannot come back the next time a value earns a metacharacter. The value is now `"80;86;89;90"`: `load_versions_env` strips one surrounding pair before exporting, so neither the exported variable nor the `--build-arg` it forwards gains a quote. That last half is the whole point — a quoted build arg reaches CMake as *data* and would defeat the `90` → `90a` transform, which is how the same bug landed once before. `test-version-forwarding.sh` asserts both halves through the real loader. ### Which shared library a consumer actually gets `/etc/ld.so.conf.d` is read in **sort order**, and several of our `/opt` trees have a distro package exporting the SAME soname. Measured in the shipped arm64 image on 2026-09-01: ``` libgstreamer-1.0.so.0 => /usr/lib/aarch64-linux-gnu/libgstreamer-1.0.so.0 (1.28.2, distro) libgstreamer-1.0.so.0 => /opt/gstreamer/lib/libgstreamer-1.0.so.0 (1.29.2, ours) ``` The distro copy won. It is not installed by us — `libgtk-4-1` pulls `libgstreamer1.0-0` back in *after* `03-media/runtime/install-deps.sh` purges it — so a consumer linking `-lgstreamer-1.0` got 1.28.2 while all 287 shipped plugins were 1.29.2. A core/plugin version split fails at runtime in confusing ways. Since 2026-09-26 the package stage drops that runtime again on amd64 (`setup-package-image.sh` `drop_redundant_distro_gtk4`, BACKLOG CON21): there `/opt/gstreamer` builds its own GTK 4, so nothing loaded Ubuntu's GTK 4 or its GStreamer, and the GStreamer registry, `gtk4paintablesink` and OpenCV's backend are unchanged without them. arm64 and riscv64 cross-build GStreamer without GTK, so their `libgstgtk4.so` needs Ubuntu's `libgtk-4-1` and the distro runtime stays; the `000-` confs below keep ours first there. `configure-runtime.sh` therefore writes `000-gstreamer.conf`, `000-ffmpeg.conf`, `000-opencv.conf`, `000-libcamera.conf`, `000-armnn.conf` and `000-onnxruntime.conf`, the same convention `000-llvm-target.conf` already used. ONNX Runtime needed one more step. OpenCV's dnn install rule (opencv 5.0.0 `modules/dnn/CMakeLists.txt:363-377`) copies the ORT it linked into `/opt/opencv5/lib`, which is first on `LD_LIBRARY_PATH` and dnn's RUNPATH, so that copy used to win every lookup. `opencv-ort.sh` now replaces each copy with an absolute link to the chain file of the same name and refuses bytes that differ from the chain's, so every lookup opens the one file under `/usr/local/lib/onnxruntime-cpu/lib`. `ort_runtime_gate`, at the end of `configure-runtime.sh`, fails the media and package builds when a `libonnxruntime.so*` or `libonnxruntime_providers_*.so*` in a loader directory resolves anywhere else, or when a distro ORT package is installed. It reads the build RUN's `LD_LIBRARY_PATH`, not the image's final ENV. Verify after any change with `ldconfig -p | grep -e ` **in the shipped image** — the build log cannot show this. ### The `01-core` source graph is layered The `01-core` modules form a small DAG that was audited clean once and that nothing enforced afterwards, so a convenience `source common.sh` added to a leaf would create a cycle or a load-order landmine that only detonates inside a Dockerfile stage — the worst place to find one. `tests/test-layer-order.sh` freezes the table: L0 true leaves that source no other `01-core` file (`logging.sh`, `load-versions-env.sh`, `path-helpers.sh`, `platform.sh`, `guard-helpers.sh`), L1 above them (`arch-mapping.sh`, `ubuntu-mirror.sh`, `downloads.sh`, `parallelism.sh`), L2 (`common.sh`), and L3 (`cross-env.sh` and its four `cross-*` siblings, with `cross-env.sh` additionally the aggregator). Each module may only reach DOWN. Two details are load-bearing. `platform.sh` is **L0, not L1** — it sources nothing, and both `arch-mapping.sh` and `ubuntu-mirror.sh` source it; the original sketch of this rule guessed otherwise. And no `01-core` file may source `artifact-common.sh` or `lib-orchestrator.sh`, which are top-level-only aggregators — the check strips comments before grepping, because `context-management.sh` and `version-forwarding.sh` *mention* `artifact-common.sh` in prose. Adjust the table only when the layering deliberately changes. ### Cross gobject-introspection: naming the wrappers before writing them Cross introspection under QEMU needs a set of wrapper executables, and the names have to agree across three phases that run minutes apart: the phase that writes them, Meson's own lookup, and the monorepo stage that execs them. `_gi_cross_detect_host_tools` (`03-media/build/gstreamer/common/pre-setup.sh`) is the single place those names are decided, and it communicates through file-scope variables by design. For `riscv64` it yields `/usr/local/bin/g-ir-scanner-riscv64-cross`, `/usr/bin/riscv64-linux-gnu-g-ir-scanner`, `/usr/local/bin/g-ir-scanner-ldd-riscv64-cross`, `/usr/local/bin/g-ir-scanner-riscv64-binary-wrapper` and `/usr/local/bin/meson-riscv64-exe-wrapper`, plus the two unsuffixed defaults `/usr/local/bin/g-ir-scanner` and `/usr/local/bin/ldd` that Meson finds on PATH. Two fallbacks in it are easy to lose in a rewrite and expensive to lose in a build. The host scanner is looked up with `PATH=/usr/bin:/bin` **first**, so a rerun does not recurse into the wrapper installed under `/usr/local/bin`. And an unqueryable `gobject-introspection` package falls back to `GOBJECT_INTROSPECTION_VERSION` rather than to an empty string — which is what a scratch cross stage actually sees. `tests/test-gi-cross-detect.sh` pins every name and both fallbacks by extracting the function (`pre-setup.sh` installs packages at top level, so it cannot be sourced). ### `gstreamer-env.sh`: the runtime env the entrypoint sources `04-runtime/gstreamer-env.sh` is what makes the `/opt/gstreamer` build reachable at runtime: it prepends the prefix to `PATH`, and prepends both the plain `lib/` and the multiarch `lib//` directory to `PKG_CONFIG_PATH`, `LD_LIBRARY_PATH`, `GST_PLUGIN_PATH` and `GI_TYPELIB_PATH`. The multiarch `gstreamer-1.0` plugin dir is precisely what `smoke-runtime-image.sh` looks for, so a regression here reads as a plugin-count failure three stages later. The triplet has three sources in a fixed order: `deb_multiarch_triplet` from `/opt/scripts/core/platform.sh` when that file is present, then `dpkg-architecture -q DEB_HOST_MULTIARCH`, then the literal `lib/multiarch` fallback. The script also carries an inline copy of `_path_contains` / `_path_prepend_unique` for the case where `/opt/scripts/core/path-helpers.sh` is absent — a copy that has to stay idempotent, since the entrypoint may source the script twice. `tests/test-gstreamer-env.sh` drives all of it on the host by copying the script and repointing its `/opt/scripts/core` lookups at a fixture, so the triplet branch, the fallback and the inline helpers each really run. ### TFLite for the GStreamer monorepo: three workarounds `_gst_monorepo_tflite_flags` in `03-media/build/gstreamer/common/build-gstreamer-monorepo.sh` was one 51-line function carrying three defects from three different years. It is now three named helpers called in order (backlog CL6), and the split is worth reading because the reason each exists is different: | helper | what it works around | | --- | --- | | `_gst_tflite_probe_flags` | cross builds only. `pkg-config` is asked for `tensorflowlite_c`, then `tensorflow-lite`, and the first that exists contributes `-idirafter ` to `CPPFLAGS`/`CFLAGS`/`CXXFLAGS` and `-L` plus `-Wl,-rpath-link,` to `LDFLAGS`. `-idirafter`, not `-I`: the target headers must lose to anything the toolchain already found | | `_gst_tflite_fix_pc` | an idempotent `sed` over `/usr/local/lib/pkgconfig/tensorflow-lite.pc`, deleting the stray `}` a since-fixed `generate_pkgconfig_file` left after `-ltensorflow-lite`. Older toolchain images still carry the bad `.pc`, so the repair has to happen at consume time, not at generate time. [`tests/test-pkgconfig-file.sh`](../linux/scripts/tests/test-pkgconfig-file.sh) owns both ends of that bug | | `_gst_tflite_symlink_for_meson` | Meson does not probe a library through `pkg-config`'s `-L`; it asks the cross compiler for `-print-file-name`, which searches only the toolchain's own directories. The fix is a symlink farm into every `/opt/gcc-*//lib*` and `/opt/gcc-*/lib/gcc//*/`, plus `LIBRARY_PATH` | Each helper is a no-op outside its own precondition — not cross, no `.pc`, no `libtensorflow-lite.so` — and each returns 0 on that path, which is what lets them be three bare calls under the file's `set -euo pipefail`. The suite proves exactly that: it extracts the three functions with `awk`, drives each one with a fixture `pkg-config`, `.pc` file and `gcc` layout, and asserts the skip paths leave the environment untouched and still exit 0. **What the suite cannot prove** is the thing the split is actually for: that an arm64 *and* a riscv64 monorepo stage still resolve TFLite. Meson's probe order, the real `.pc` files in the shipped toolchain image and the `-print-file-name` search dirs are all container facts. Until a chain runs both arches, this is a verified refactor, not a verified fix. ### The wrapper generation gate (`_manifest_wrapper_gate`) A multi-arch index is only meaningful if its children came from one run. `_manifest_wrapper_gate` (`build-runtime-manifest.sh`) reads the recorded `run-id` off each per-arch wrapper tag through `ancestry_recorded_run_id` and refuses to assemble `:latest` out of tags that span generations — the failure it exists for is an index that mixes a fresh arm64 wrapper with an amd64 one from last week, which every existence check passes. Its refusal matrix is deliberately asymmetric, and the asymmetry is the whole design: * **tags that span generations** refuse on *both* paths — building or repairing. * **a missing run-id stamp** only warns on the build path (the run is about to overwrite that tag anyway) and **refuses** on the repair path (`--manifest-only` / `--repair`), because `ancestry_run_ids_coherent` drops empty ids, so unverifiable provenance cannot rule a mixed release out. * **a wrapper older than the current android artifact** likewise refuses on the repair path only. * `--force` overrides after the warning is printed; `RUNTIME_MANIFEST_COHERENCE=0` disables the gate outright. `tests/test-manifest-wrapper-gate.sh` extracts the function with its ancestry collaborators stubbed, so what is under test is the decision rather than the registry. ### Cross-compiler multi-arch smoke `06-packaging/smoke-cross-all-arches.sh [amd64,arm64,riscv64]` runs one probe per toolchain the image is supposed to carry, and each probe is a function so a suite can drive it without a built image (`tests/test-cross-arch-smoke-probes.sh`): | Probe | Asserts | |-------|---------| | `_smoke_probe_host_compilers` | `cc` exists and is the host arch; the same for `${GCC_PREFIX}/bin/gcc` when it is present | | `_smoke_probe_cross_compilers` | for every requested arch that is not the host: `-gcc` exists and produces target objects, and `-g++ -dumpmachine` names the target | | `_smoke_probe_llvm_target_clang` | `${SMOKE_TARGET_CLANG}` (default `/usr/local/llvm-target/bin/clang`) reports a triple matching one of the requested arches | | `_smoke_probe_symlink_chain` | `cc`, `c++`, `gcc`, `g++` all resolve to something | The clang probe is the one with history. The binary has a **single** default triple, so the check is "does it match ONE of the requested arches", not one verdict per arch. The loop that answers it used to `break` after the first arch and had no failure arm at all, so a clang built for the wrong target shipped green. `_smoke_clang_match_arch ` is now that answer on its own — it prints the first arch whose uname name prefixes the triple and nothing when none does — and the caller turns an empty answer into `matches none of: `, a real `fail`. An absent clang is still skipped silently: an image without `llvm-target` is not a defect. `SMOKE_TARGET_CLANG` exists so the probe can be pointed at a clang the suite controls. It defaults to the shipped path in the script itself, so nothing in the image or the Dockerfiles needs to set it. ### The media compile-cache launcher `03-media/core/common.sh` owns `media_compiler_launcher `. It sets the named variable to the launcher a media compile should prefix its compiler with, or to the empty string: 1. when `01-core/common.sh` is loaded — which `media_common_init` guarantees, or aborts — it establishes the launcher address with `compiler_cache_launcher_env` **first** and then resolves through `compiler_cache_launcher`, so sccache is asked when sccache is the active cache. Resolving before the address is set is how a stage ends up talking to a server that is not there; 2. otherwise, `ccache` when it is on `PATH` and `USE_CCACHE` is truthy. `build-ffmpeg.sh` and `build-pyav.sh` each carried their own copy of that ladder, and the copies had already drifted: ffmpeg tested `USE_CCACHE` with an inline deny-list (`0|false|no|off` → off, anything else → on) while pyav used the canonical `is_truthy` (`1|true|yes|on` → on, anything else → off). They agree on every value either script documents and on the `:-true` default; they disagreed only on junk (`USE_CCACHE=y` was on for ffmpeg, off for pyav). The owner uses `is_truthy`, the predicate `platform.sh` declares canonical. **It takes an out-variable name instead of printing**, which is the one thing about it that is not obvious. `compiler_cache_launcher_env` exports the sccache server address into the shell that will run the compiles, and a `$( )` caller would run it in a subshell and discard that export the moment the substitution closes — a green build that talks to no server. The same trap is written up at [`build-cache-tiers.md`](build-cache-tiers.md#the-server-address-must-be-exported-where-the-compiles-run); folding the resolve into a printing helper is how it comes back. Step 2 is unreachable from those two scripts — `media_common_init` returns non-zero when a critical 01-core module fails to load, and both scripts run it at top level under `set -e`. It stays because the helper is the media tree's answer to the question, not ffmpeg's, and a later consumer may not run the init. `tests/test-compiler-cache-launcher.sh` covers both steps plus the totality contract: no launcher and no ccache prints nothing and returns 0, so a cacheless host degrades instead of aborting a stage.