# What `:latest` promises its consumers This page is the **contract** side of the runtime image: the properties another repository's CI lane may build on, written down so they are a promise rather than an accident of the last build. Everything here is asserted per arch by `check_consumer_contract` in `linux/scripts/06-packaging/smoke-runtime-image.sh`, against the shipped bytes, as the user the image ships. It exists because on 2026-09-04 four properties a consuming lane depended on were all broken at once in `:latest-cross`, every runtime gate stayed green, and the lane in the other repo is where they were found. A property nobody wrote down is a property nothing can keep. ## The contract Run as the image's own user (`kataglyphis`, uid 1001), with no privileged flags and no extra `-e`: | # | Promise | Consumer symptom when it breaks | |---|---|---| | 1 | `CCACHE_DIR` and `SCCACHE_DIR` point **outside** `/workspace` and are writable | the cache is written into the consumer's bind-mounted checkout, pollutes their working tree, can be swept into CI artifacts, and on a non-ext4 host mount `flatpak-builder` aborts: *"Can't initialize ccache use: Failed to set permissions of /workspace/.ccache/disabled/ccache.conf: Operation not permitted"* | | 2 | `$RUSTUP_HOME/tmp` and `$CARGO_HOME` are writable | *"could not create temp file …: Permission denied (os error 13)"* — Corrosion, cargokit and `flutter_rust_bridge_codegen` cannot run. Not workaroundable by redirecting the variable: the toolchains live in that tree, and `rustup toolchain install` (fRB asks for the `nightly` channel, the image pins a dated nightly) writes there too | | 3 | `ANDROID_HOME` and `ANDROID_SDK_ROOT` are set, `$ANDROID_HOME/platform-tools` exists, and the SDK's `cmdline-tools/latest/bin` + `platform-tools` are on `PATH` (amd64; arm64 and riscv64 ship no SDK and leave both empty, [why](#per-arch-exemptions)) | `flutter build apk` stops with *"[!] No Android SDK found"*. Under CodeQL's `database create --command=…` the exit 1 aborts before the database is finalised, so the lane reports *"bundle source directory not found: build/app/outputs/flutter-apk"* — three steps from the cause | | 4 | `java` is on `PATH` and `JAVA_HOME` names a JDK with `bin/javac` | Gradle stops the Android lane with *"JAVA_HOME is not set and no 'java' command could be found in your PATH"*, and `flutter doctor` reports *"No Java Development Kit (JDK) found"* | | 5 | `appimagetool` is READABLE by the image user, not merely executable | it is an AppImage and reads `/proc/self/exe` for its own squashfs offset, so mode 711 gives *"Cannot open /proc/self/exe: Permission denied"* and produces no `.AppImage` | | 6 | Every path under `/opt/flutter` is owned by uid 1001, `packages/flutter_tools/.dart_tool` included | `flutter pub get` fails with *"Cannot open file … package_config.json (OS Error: Permission denied, errno = 13)"* | | 7 | `CHROME_EXECUTABLE` names Chrome for Testing, which renders a page headless as uid 1001, and `chromedriver` of the same version is on `PATH` (amd64, arm64; [why](#browser-tests-run-in-chrome-for-testing)) | `flutter doctor` reports *"Cannot find Chrome executable at google-chrome"*, and `flutter test --platform chrome` has no browser | | 8 | On amd64, `$ANDROID_HOME` holds the pinned emulator and one x86_64 system image with ARM translation, and `android-avd.sh` is on `PATH` ([why](#the-android-emulator-runs-on-amd64-with-kvm)) | an Android lane has no device: `adb` reports *"no devices/emulators found"* | Row 6 is the one a consumer **cannot** repair at runtime. The directory sits in a read-only overlay layer, so a non-owner can neither empty nor rename it — both were attempted and refused — and the only workaround is mounting a tmpfs with `mode=1777` over it. Rows 1–5 are merely expensive to work around, and the point of writing them down is that nobody should have to. `/workspace` is the WORKDIR and the consumer's checkout. Nothing the image generates by itself may land there; that is what makes row 1 a location assertion and not only a permission one. The compiler-cache defaults the shared library already declares (`linux/scripts/01-core/compiler-cache.sh`) are `/var/cache/ccache` and `/var/cache/sccache`, and the image ships both as `drwxrwxrwt`. An image ENV that contradicts our own library is the failure shape row 1 guards. **Neither cache directory is a `VOLUME`.** The image declares exactly one, `/workspace` (`VOLUME ["${WORKDIR}"]` in `Dockerfile.torch`, confirmed in `Config.Volumes` of all three shipped children), so a container's compiler cache lives in its writable layer and dies with it. A lane that wants the cache to survive has to mount something there itself — and that is the reason row 1 is a LOCATION assertion: a cache pointed back into the bind-mounted checkout would persist, by polluting the consumer's working tree. ### The Android lane needs a JDK `02-toolchain/android-sdk.sh` installs the full `openjdk-21-jdk` in the stage that BUILDS the SDK, but `Dockerfile.package` copies only `/opt/android-sdk` — apt had put the JDK in `/usr/lib/jvm`, which is not part of that COPY. The runtime image therefore carried a complete Android SDK and no Java at all, and the failure surfaces in Gradle rather than in Flutter, one layer below where a reader looks. The runtime image installs `JDK_PACKAGE` (`01-core/versions.env`, `openjdk-21-jdk-headless`, ~286 MB with its JRE) by name rather than through `append_available_packages`, which SKIPS what apt does not have: a silently skipped JDK would ship the same broken image the gate exists to catch. Headless is deliberate — a container build needs `javac`, not AWT. `JAVA_HOME` cannot be written literally, because Ubuntu's path carries both the version and the architecture (`java-21-openjdk-riscv64`). `anchor_java_home` resolves it once from the installed `javac` and parks the symlink `/usr/lib/jvm/default-java`, which the image ENV names. A JDK bump then needs no Dockerfile edit, and a JDK that installed no compiler fails the stage instead of shipping a JRE that Gradle cannot use. ### Executable is not usable `ensure_appimagetool` (`02-toolchain/packaging-deps.sh`) downloads into a `mktemp` file, which is `0600`, and made it executable with `chmod +x` — which adds the x bits and leaves r for the owner only, i.e. `0711`. `mv` carries that mode to `/usr/local/bin/appimagetool`, so the tool was executable by everyone and readable by root alone. That is fatal for this particular tool and invisible for most others: appimagetool is itself an AppImage, so it opens `/proc/self/exe` to find where its squashfs payload starts. As uid 1001 that open is refused and the run dies with `Failed to get fs offset for /proc/self/exe`, having produced nothing. The install site now sets `0755` explicitly rather than relying on `+x` over whatever mode the download landed with, and the contract gate asserts readability rather than presence — `command -v` finding it says nothing about whether the image's own user can run it. ## How the gate proves it One `nerdctl run`, not one per assertion. `_consumer_contract_probe` emits facts only — `WHO`, `ENV`, `WRITE`, `DIR`, `FACT` lines and a `CCPROBE_DONE` sentinel — and `_consumer_contract_verdicts` reaches every verdict on the host, so each failure path is provable from a *recorded* probe capture rather than from a doctored 30 GB image. `linux/scripts/tests/test-runtime-image-gates.sh` drives it with the capture measured in the broken 2026-09-04 image and with the fixed one. Three details are load-bearing: - **Writability is a real create + delete** of a file in the directory, not `[ -w ]`. `access(2)` answers yes for uid 0 and says nothing about a read-only layer. - **The probe must have run as the image's own `Config.User`.** As root every directory answers writable, so a probe that reports any other identity fails the gate outright instead of reporting every row green. - **No row may report nothing.** A missing fact is `NOFACT` and fails, an empty row table fails as *asserted NOTHING*, and a verdict verb no arm handles fails. A gate arm that can only ever skip is how all four defects shipped. Each red row quotes the symptom from the *consuming* repository's log, not our path, because that is the sentence someone will paste into a search. ## Per-arch exemptions `_consumer_contract_exempt` is a `:` table, same contract as `_parity_exempt`: listed means reviewed, and an arm that **stops** applying fails so the table cannot rot in place. Each arm is re-checked by **its own** probe fact, named by `_consumer_exempt_fact`; `yes` is `STALE` and names the arm for deletion, a missing fact is `NOFACT` and never a grant. Eight arms. The first two were measured on the image shipped 2026-09-05 rather than argued from the build graph, the next three on the CON50 proof images of 2026-10-01: | arm | rot fact | what the image reports | |---|---|---| | `dart-tool` | `flutter-sdk` | `/opt/flutter` exists and is **empty**, so `packages/flutter_tools/.dart_tool` is absent and the row would read as unwritable. Upstream publishes no riscv64 SDK; `check_flutter` asserts that absence instead | | `appimagetool` | `appimagetool-readable` | no `appimagetool` on `PATH` at all — `packaging-deps.sh`'s asset table covers x86_64/aarch64/armhf/i686 and refuses the rest | | `flatpak-runtimes` | `flutter-sdk` | no refs: Flathub builds the freedesktop runtimes for x86_64 and aarch64 only, and the installer skips every other arch | | `appimage-runtime` | `flutter-sdk` | no runtime: it is carved out of `appimagetool`, which riscv64 does not have | | `riscv64:chrome` | `chrome` | `CHROME_EXECUTABLE` is empty: Chrome for Testing publishes `linux64` and `linux-arm64` only | | `arm64:android-emulator` | `android-emulator` | no `$ANDROID_HOME/emulator`: Google publishes the Linux emulator for x86_64 hosts only | | `riscv64:android-emulator` | `android-emulator` | as on arm64 | | `riscv64:cargo-qa-tools` | `cargo-qa-tools` | none of the three tools: none publishes a riscv64 binary, and the riscv64 lanes cross-build on amd64 ([below](#the-cargo-qa-tools)) | **Open gap (2026-09-25):** the `flatpak-runtimes` and `appimage-runtime` arms are re-checked by `flutter-sdk`, another row's fact, because `_consumer_exempt_fact` maps every row but `appimagetool` to it. A riscv64 image that gained Flatpak runtimes or an AppImage runtime would still read `EXEMPT`, which is the rot the next paragraph describes. `flutter-owner` **was** an arm too and is gone. The same probe measures `find /opt/flutter ! -uid 1001` as **0** on riscv64, which is the row *passing*, not a row to skip: an empty tree owned by the runtime uid satisfies the promise. Exempting it also hid the defect the row exists for — a root-written SDK would have read as a documented exception on that arch. Until 2026-09-05 the `appimagetool` arm was re-checked with `FACT flutter-sdk`, i.e. by another row's fact, so a riscv64 appimagetool could never have been noticed; that is what `_consumer_exempt_fact` fixes. The Android SDK is **not** exempt anywhere. The parity table asserts the `android-sdk` prefix on every arch. The row is *skipped*, not exempted, only when the image says it ships no SDK, in `/opt/android/.android-payload-off`. Two writers put it there: - `android-sdk.sh`, when the android stage runs on a non-amd64 host. Upstream ships the NDK for `linux-x86_64` only, so there is nothing to install. - `prune-host-payload.sh`, for every arch whose `HOST_PAYLOAD` is off. That is arm64 and riscv64 by default since 2026-10-13 ([why](artifact-copy-completeness.md#the-host-payload-stays-with-the-host-that-runs-it)). Until then `/opt/android-sdk/platform-tools` was in all three shipped arches, as measured on 2026-09-04, though only amd64 can run it. A payload-off image leaves `ANDROID_HOME` and `ANDROID_SDK_ROOT` empty. The row fails one that still sets either, because a consumer would find an empty SDK behind the variable. A missing fact is still `NOFACT`. ### The Flatpak runtimes ship with the image `flatpak list --runtime` in the shipped image returned **zero refs**. Every consumer run therefore re-downloaded the whole runtime set — roughly 1.9 GB across seven refs, the single largest download in their build. Two of the seven (`Platform` and `Sdk`) were already installed here; the other five were left to each consumer run. All seven are installed now: | ref | pinned by | | --- | --- | | `org.freedesktop.Platform//` | `FLATPAK_RUNTIME_VERSION` | | `org.freedesktop.Sdk//` | `FLATPAK_RUNTIME_VERSION` | | `org.freedesktop.Platform.Locale//` | `FLATPAK_RUNTIME_VERSION` | | `org.freedesktop.Sdk.Locale//` | `FLATPAK_RUNTIME_VERSION` | | `org.freedesktop.Platform.GL.default//` | `FLATPAK_RUNTIME_VERSION` | | `org.freedesktop.Platform.GL.default//-extra` | `FLATPAK_RUNTIME_VERSION` | | `org.freedesktop.Platform.codecs-extra//-extra` | `FLATPAK_RUNTIME_VERSION` | `GL.default` appears twice on purpose: the base branch and its `extra` sibling are two separate refs, and a build that resolves one still fetches the other. The runtime is **26.08** since 2026-10-07. freedesktop-sdk 25.08 changed two names, both read from flathub's own listing and `Platform//26.08`'s metadata that day: the extra branches are spelled `-extra` (24.08 said `24.08extra`), and the `openh264` extension is gone. `Platform//26.08` declares `codecs-extra//-extra` instead, which ships FFmpeg, x264 (with its GStreamer plugin), x265 and libde265 and no openh264, so `FLATPAK_OPENH264_VERSION` went with it. A consumer flatpak that loaded the runtime's openh264 must bring its own H.264 codec or use x264. `flatpak install Platform//26.08` pulls `GL.default` (both branches), `Locale` and `codecs-extra` as related refs; the seven-ref list names them so that a failure says which one. `INSTALL_FLATPAK_RUNTIMES` defaults to **true** since 2026-09-05. Flathub builds these for `x86_64` and `aarch64` only, so on any other architecture the install is a guaranteed 404 rather than a transient failure — the installer skips those arches outright instead of retrying, and riscv64 images ship without Flatpak runtimes by construction. ### The web lane toolchain Measured in a consumer run, the Flutter **web** lane rebuilt its own tools on every invocation: `flutter_rust_bridge_codegen build-web` shells out to `wasm-pack`, and both were a from-source `cargo install` — 258 crates for `wasm-pack`, 174 for `flutter_rust_bridge_codegen` — followed by a nightly `rustup` auto-install through a path rustup itself calls deprecated. The image now installs all three ahead of time: | what | pin | why it must be here | | --- | --- | --- | | `nightly` toolchain + `rust-src` + `wasm32-unknown-unknown` | `RUST_NIGHTLY_TOOLCHAIN` (`nightly-YYYY-MM-DD`) | `flutter_rust_bridge_codegen build-web` runs wasm-pack with `RUSTUP_TOOLCHAIN=nightly` and `-Z build-std`. A **dated** toolchain is immutable, so `rustup toolchain install ` is a genuine no-op on a warm image; the floating channel is UPDATED by that command, and the update renames files out of the read-only image layer (`Invalid cross-device link`). Consumers that still name the channel get it auto-installed at runtime into the writable `RUSTUP_HOME` — works, but pays the download per run; FRB's `--wasm-pack-rustup-toolchain` names the pin instead. | | `wasm-pack` | `WASM_PACK_VERSION` | 258 crates per consumer run | | `flutter_rust_bridge_codegen` | `FLUTTER_RUST_BRIDGE_VERSION` | 174 crates per consumer run | Both crate versions live in `01-core/versions.env`. Where upstream publishes a release binary for the arch, the image takes that instead of compiling: the same two `cargo install`s cost 87 s / 113 s on amd64 but **768 s / ~1170 s on arm64** and **1813 s / 3500 s on riscv64**, which is the shape of QEMU user-mode emulation rather than a defect. `install_web_lane_prebuilt` downloads the `x86_64-` or `aarch64-unknown-linux-musl` asset and verifies it against a per-arch `*_SHA256` pin in `versions.env`, the way sccache and binaryen are already fetched. Anything else — riscv64, which upstream publishes no asset for, a missing pin, a failed or mismatching download, a tarball without the binary in it — falls back to the from-source leg ([below](#building-the-web-lane-tools-from-source)), so a consumer still gets the pinned `--locked` build rather than whatever the index resolves to that day, and nothing unverified is ever installed. **The four hashes bump with the two versions.** That leaves one open question rather than an assumption: whether a riscv64 web lane exists at all. The tools are installed uniformly because an arch-conditional image is harder to reason about than a slower one, and because "we assumed nobody uses it" is how the Android layer ended up built for the wrong ABI. `install_web_lane_toolchain` is **non-fatal for availability** — a missing nightly pin, an unpinned version and a failed `cargo install` each `WARN` and continue. The trade is deliberate: a consumer that has to build its own tools is slow, a consumer that cannot build the image at all is worse. What it does fail on — a bad knob, or a binary that claims to be good and is not — is in the next section. ### Building the web-lane tools from source riscv64 has no upstream binary, so its package stage used to compile both tools under QEMU on every chain: 960 s + 1,665 s of a 3,384 s RUN (riscv64 layer file times, 2026-09-22). That compile now has two replacements, and the old build stays one switch away, verbatim. Owner decision 2026-09-23: both options, cross the default. | Path | Where it compiles | When | | --- | --- | --- | | cross | `Dockerfile.android`, stage `web-lane-tools`: `cargo install --target ` under the hub's own `setup_linux_cross_env`, before `final` swaps the amd64-hosted cross GCC out | the target is in `WEB_LANE_TOOLS_CROSS_ARCHES` and is not the build platform's own arch | | native | the package stage, with rv64gc Rust: `cargo install --locked --version --root ` under QEMU on the amd64 host, natively on a riscv64 or arm64 build host | `WEB_LANE_TOOLS_SOURCE=native`, or when `auto` finds no usable cross artifact | | cache | the package stage's cachemount `web-lane-tools-bin-` | a native build whose exact key was built before | | legacy | the package stage, exactly as before 2026-09-23: `cargo install --locked --version ` into `CARGO_HOME` | `WEB_LANE_TOOLS_SOURCE=legacy` only | **native is not the old build.** It compiles where the old build did, with the same rv64gc Rust, but four things differ: it forces vendored static bzip2, xz and zstd (the old build linked whatever pkg-config found in the stage); it installs a bare binary copied out of a scratch `--root`, so `CARGO_HOME` has no `.crates.toml` / `.crates2.json` entry for it; a binary that fails the gate stops the build (the old build installed anything cargo produced); and it reads the cache first, so it may not compile at all (`WEB_LANE_TOOLS_CACHE=refresh` or `off` forces a compile). **legacy** is the old build, verbatim: cargo installs into `CARGO_HOME` itself (and records it there), in the stage's own environment, with no gate, no cache, no provenance line, and a failed `cargo install` only WARNs. It ignores `WEB_LANE_TOOLS_CACHE` and never reads the cross artifact. amd64 and arm64 are untouched: they install upstream's sha-pinned musl binary first and reach this code only if that download fails. **Picking a path.** Export `WEB_LANE_TOOLS_SOURCE` before `build-cross-chain.sh` or the runtime helpers: - `auto` (default) takes the cross artifact when its manifest says `status=ok`, it was built for this exact key, and it passes the gate. Anything else builds natively, with a `WARN` that ends `the cross fast path was not taken`. - `cross` requires the artifact: absent, `failed` or another key is an ERROR. A `skipped` manifest is not, because the producer was never asked to build that arch; the package builds natively. - `native` never reads the artifact. It still reads the cache, so add `WEB_LANE_TOOLS_CACHE=refresh` for a guaranteed fresh compile. - `legacy` is the build the package stage ran before 2026-09-23, unchanged (above). Use it to rule the new paths out, or when a consumer needs cargo's install record. `WEB_LANE_TOOLS_CACHE` is `on` (default), `refresh` (never read an entry, store the new build) or `off` (neither). `RUNTIME_NO_CACHE=1` does not empty a cachemount; `refresh` is its equivalent here. `WEB_LANE_TOOLS_CROSS_ARCHES` (default `riscv64`, a comma list, or `none`) is the arches android cross-builds for. A build host whose own arch is the target — the riscv64 X100, an arm64 Jetson on `CROSS_BUILD_PLATFORM=linux/arm64` — always gets `skipped: native-build-platform`, so `auto` builds natively there with nothing set. All three are Dockerfile ARGs that `lib-orchestrator.sh` forwards only when they are set, so leaving them unset moves no cache key. They are deliberately not in `versions.env`: a line there re-keys the whole chain from base. **The key.** Eight fields: schema, tool, version, target triple, rustc release, RUSTFLAGS, the C environment and `--locked`. Who built it (`built_by=cross:amd64`, `native:riscv64`) is recorded, never keyed. The cross key carries cross-env.sh's RVV flags (`-C target-feature=+v,+zvl128b`; a test pins the copy); the native key carries none, as the old build never set any. Both force vendored static bzip2, xz and zstd, because under `PKG_CONFIG_ALLOW_CROSS` those `-sys` crates link the build host's library ([failure-modes.md](failure-modes.md#a-cross-built-rust-tool-links-the-build-hosts-libbz2)). `legacy` has no key: it caches nothing. **The gate** (`wlt_assert_binary`) reads the staged bytes before anything is installed: ELF64, the arch's machine, lp64d on riscv64, glibc's own loader as `PT_INTERP`, `NEEDED` only from libc's family, no `GLIBC_` version above the image's glibc, and `--version` printing exactly ` `. No readelf is a refusal, not a skip. Artifact and cache entries must also match their recorded sha256. **Fatal or not.** Availability still only WARNs: a failed `cargo install`, a producer that could not build (it records `status=failed` and android stays green), an unusable artifact under `auto`, no `rustc -V` release under `auto` or `native`. These fail the package stage: a bad knob value, checked before anything else, so a typo stops amd64 and arm64 too; a binary that claims to be good and is not — a `status=ok` artifact failing its sha or its gate, in `auto` and `cross` alike, or a native build cargo reported as a success that fails the gate; an image whose glibc `getconf` cannot report, or an `install` into `CARGO_HOME/bin` that fails; and `cross` with no usable artifact, including no `rustc -V` release to key an expected one by. A cache entry that fails its checks is deleted with a `WARN` and rebuilt. `legacy` fails on nothing but a bad knob. **Provenance.** Each cross, cache or native install appends `tool= version= source=cross|cache|native sha256= key=` to `/usr/local/share/web-lane-tools/provenance`. All three install a bare binary with no `.crates.toml`, which is what amd64 and arm64 already ship from upstream: a consumer's own `cargo install ` then stops on `binary already exists` unless it passes `--force`. `legacy` writes no provenance line, and cargo records its install, as before. **Not covered.** Behaviour beyond `--version`: no automated riscv64 `build-web` consumer exists. A binary and manifest forged consistently by someone who can write the build host's cachemounts, the same trust boundary as the cargo, uv and apt mounts. The Jetson → riscv64 cross leg is untested; it records `failed` and falls back. The cross build ran in a probe (about 73 s at 32 vCPU, 2026-09-22), never yet in a chain. ### The AppImage runtime ships with the tool Every AppImage begins with a small ELF **runtime** that `appimagetool` prepends to the payload. When that runtime is not already on disk, `appimagetool` fetches it from GitHub on each run — so a consumer's packaging step depends on GitHub being up at build time, and fails offline. The runtime is staged at image-build time instead, and it is **not downloaded**. Upstream publishes it only under the moving `continuous` tag, which is the exact mutable-asset trap that already broke `appimagetool` itself once (a `continuous` re-upload changed the bytes under a pinned SHA256, and `download_verified_file` reported a tamper-shaped "checksum mismatch" that was only upstream drift). Since every AppImage *starts* with the runtime, and `appimagetool` is itself an AppImage pinned to an immutable versioned tag with a recorded SHA256, the runtime is taken out of the tool's own bytes, up to where its squashfs payload starts. It is therefore pinned transitively and arch-correct by construction — no second download, no second pin to keep in step. The offset is found by reading the file for the squashfs superblock, never by running the tool: QEMU user-mode cannot self-mount a foreign-arch AppImage, and `appimagetool --appimage-offset` gave `Exec format error` on aarch64 (2026-09-06). It is written to two places, as `runtime-`: | path | why | | --- | --- | | `/etc/skel/.local/share/appimagekit/` | the runtime user is created later, and inherits `/etc/skel` | | `${HOME}/.local/share/appimagekit/` | the build user that runs the packaging step now | `ensure_appimagetool_runtime` is a no-op, not a failure, when `appimagetool` is absent (riscv64 has no upstream build) or when no squashfs superblock is found in it: a missing runtime costs a consumer one download, and is never worth failing a toolchain stage over. ### What changes with the next `:latest` These are in source and not yet in a published image. Drop the workaround once your lane runs on the next `:latest`. | What | Before | After | | --- | --- | --- | | A window on arm64/riscv64 Vulkan (CON41) | the loader lists no `VK_KHR_{xcb,xlib,wayland}_surface`, so a windowed test aborts and only amd64 renders under `xvfb-run` | all three arches list them; the image smoke fails an arch that does not ([why](vulkan-foreign-arch-sdk.md#the-loader-carries-the-window-systems)) | | clang-tidy on a compile database naming `/usr/bin/clang++` (CON39) | selects the distro GCC 16; BeschleunigerBallett passes `--extra-arg=--gcc-toolchain=${GCC_PREFIX}` | `/usr/bin` carries the cfg pair too, so it selects `${GCC_PREFIX}` ([`linux-cross-builds.md` § Clang cross wrappers](linux-cross-builds.md#clang-cross-wrappers)) | | An atheris source build (CON38) | `Failed to find libFuzzer`: its `find_libfuzzer.sh` probes only `lib/linux/libclang_rt.fuzzer_no_main-.a` | the fuzzer, asan and ubsan archives are linked under that name as well; riscv64 still needs `LIBFUZZER_LIB`, since atheris' probe knows no riscv64 | | lavapipe on arm64/riscv64 (CON44) | 4-lane subgroups: a draw that builds an acceleration structure SEGVs; BeschleunigerBallett exports `LP_NATIVE_VECTOR_WIDTH=256` itself | the image sets `LP_NATIVE_VECTOR_WIDTH=256` on every arch, and the smoke fails a lavapipe whose `subgroupSize` is not 8 ([why](failure-modes.md#lavapipe-segfaults-building-an-acceleration-structure-on-arm64)) | | A C host that loads GStreamer or libcamera first | `libunwind.so.8` from `libgstreamer-1.0`/`libcamera-base` turns a C++ exception through `std::call_once` into a SIGSEGV | both build without libunwind ([why](failure-modes.md#an-exception-through-stdcall_once-segfaults-in-libunwind)) | | `gst-inspect-1.0 -b` on amd64 (CON47) | lists `libgstvalidatessim.so` | lists nothing; the SSIM plugin still works under `gst-validate-1.0` ([why](failure-modes.md#the-core-registry-blacklists-libgstvalidatessimso)) | | A browser for `flutter test --platform chrome` (CON50) | none; `flutter doctor` reports *"Cannot find Chrome executable"* | Chrome for Testing and its chromedriver on amd64 and arm64, `CHROME_EXECUTABLE` set ([how](#browser-tests-run-in-chrome-for-testing)) | | An Android device on amd64 (CON50) | none; Android tests ran on the x64 VM only | the emulator and an x86_64 system image with ARM translation, booted by `android-avd.sh` on `/dev/kvm` ([how](#the-android-emulator-runs-on-amd64-with-kvm)) | | The Android SDK on arm64 and riscv64 (CON96) | `/opt/android-sdk`, `/opt/android` and the builder's x86_64 rustup ship in both, 3.41 GB of layers no arm64 or riscv64 host can run; `ANDROID_HOME` points at x86_64 tools | both trees empty, with `/opt/android/.android-payload-off`; `ANDROID_HOME`, `ANDROID_SDK_ROOT` and the six Android roots empty, nothing Android on `PATH`. Build Android from the amd64 image, as before. `HOST_PAYLOAD_ARCHES=amd64,arm64,riscv64` builds the old shape ([why](artifact-copy-completeness.md#the-host-payload-stays-with-the-host-that-runs-it)) | | `VULKAN_SDK` and the variables `setup-env.sh` derives from it, after the entrypoint (CON48) | `/opt/vulkan//`; a riscv64 lane under QEMU in the amd64 image overrides `VULKAN_SDK` and `VK_ADD_LAYER_PATH` (`riscv64_cross_env`) | `/opt/vulkan/active` on every arch; the boot smoke fails an arch dir ([why](failure-modes.md#vulkan-env-names-an-arch-specific-sdk-dir)) | ## The Android SDK roots are advertised `Dockerfile.android` advertises where each Android payload lives; `Dockerfile.package` COPYs the payload those names point at but, until 2026-09-05, never re-declared the names. A consumer that found `/opt/android` in the runtime image therefore still had no way to be told where anything inside it was, and every Android lane hardcoded the paths or re-downloaded the SDKs. The runtime image now re-declares all six. They are empty on arm64 and riscv64, which ship no Android payload ([why](artifact-copy-completeness.md#the-host-payload-stays-with-the-host-that-runs-it)): | variable | value | | --- | --- | | `GSTREAMER_ROOT_ANDROID` | `/opt/android/gstreamer` | | `ONNXRUNTIME_ROOT_ANDROID` | `/opt/android/onnxruntime` | | `LITERT_ROOT_ANDROID` | `/opt/android/litert` | | `OPENCV_ROOT_ANDROID` | `/opt/android/opencv` | | `IREE_ROOT_ANDROID` | `/opt/android/iree` | | `OPENCV_ANDROID_JNI_DIR` | `/opt/android/opencv/sdk/native/jni` | Two shapes here are deliberate and a tidy-up would break both. Each name gets its **own** `ENV` instruction, because the env-knob owner scan reads only the first name of an instruction — collapsing them into one backslash-continued `ENV`, the way `Dockerfile.android` writes them, would leave five of the six with no recorded owner. And the OpenCV key is `OPENCV_ROOT_ANDROID`, never a bare `OpenCV_DIR`: that is the name `find_package(OpenCV)` reads, so pointing it at the Android SDK would hijack every **Linux** OpenCV consumer in the same image. These are paths, not versions, so they are outside the advertised-version-key gate (`verify_advertised_keys.py`); what holds them is the runtime smoke's path checks. ## Two things worth knowing before you configure a lane - `:latest` is a proper multi-arch index (it was called `:latest-cross` until 2026-09-22; that name is retired). An arm64 runner gets arm64 binaries; there is no longer any reason to pin `-amd64` and no `rustc: 1: ELF: not found` to work around. - **The image ships Flutter at `/opt/flutter`**, the default of the hub's `flutter_lane_prepare_env` (`05-frameworks/flutter/lane-prologue.sh`), which installs nothing: a lane that points it at `/workspace/flutter` stops with `no Flutter SDK at /workspace/flutter`, and a lane with its own installer re-downloads the SDK every run for nothing. `sccache` and `appimagetool` are on `PATH` as well. ## What changes with the image after CON11 In source since 2026-09-26; shipped in `:latest` on 2026-09-29 (index `sha256:696642b2…`, BACKLOG CON37). Each item retires a workaround a consumer carries today; drop it once your lane runs on that image. | What | Before | After | | --- | --- | --- | | A bare `clang`/`clang++` | selects the distro GCC 16, so linking a library the image's GCC built fails (`std::format` symbols, `GLIBCXX_3.4.36`) | `-clang{,++}.cfg` beside the compiler select `${GCC_PREFIX}`; a `--target` build loads neither | | `clang-tidy`, `llvm-profdata`, `llvm-cov`, `llvm-symbolizer`, `ld.lld`, … on `PATH` | LLVM 21, which cannot read what LLVM 23 wrote | clang's own, linked into `/usr/local/bin`. `clang-format` and `llvm-config` stayed 21 until CON71 moved every LLVM tool to the pin (§ Every LLVM tool is the pinned release) | | `-fsanitize=fuzzer` with the image's clang | no libFuzzer runtime | built with compiler-rt (without a private libc++) | | `VIRTUAL_ENV`, `UV_PYTHON` | point at the root-owned `/opt/venv`, over an activated venv | empty in the ENV and unset by the entrypoint; `/opt/venv/bin` stays first on `PATH` | | A Vulkan device | none (`nvidia_icd.json` only) | lavapipe, a CPU device (`mesa-vulkan-drivers`) | | `perf`, `jq`, `Xvfb`, gperftools' `libprofiler` | absent | installed (`linux-perf` carries `perf` on 26.04) | | A caller's `LD_LIBRARY_PATH`/`GST_PLUGIN_PATH` | the entrypoint put `/opt/libcamera` ahead of it, and a host `libstdc++` in it shadowed GCC's | the image's libcamera goes after it and `${GCC_PREFIX}`'s runtime before it, so a Raspberry Pi host-libcamera run needs no `--entrypoint` | | The distro GStreamer 1.28 runtime | installed beside `/opt/gstreamer` | dropped on amd64; kept on arm64/riscv64, where `libgstgtk4.so` needs Ubuntu's GTK 4 | | Android API 37 | missing although pinned (a stale SDK cache) | `platforms/android-37.0` and `build-tools/37.0.0` | | TVM on amd64 | `import tvm` works, but its compiler library needs an LLVM the image lacks, so TVM loads runtime-only and compiles nothing | linked against `/usr/local/llvm-target`'s LLVM (X86 codegen, as arm64's is AArch64); the image smoke compiles one PrimFunc | ## The Android SDK roots are advertised `Dockerfile.android` sets `GSTREAMER_ROOT_ANDROID`, `ONNXRUNTIME_ROOT_ANDROID`, `LITERT_ROOT_ANDROID`, `OPENCV_ROOT_ANDROID` and `IREE_ROOT_ANDROID`, but that is the *build* stage. `Dockerfile.package` COPYs the payload those names point at and used to stop there, so a consumer of the shipped image found `/opt/android` populated and no name telling it what was where. The reported symptom: ``` CMake Error at CMakeLists.txt:18 (message): GSTREAMER_ROOT_ANDROID must be set ``` All five are re-declared in the runtime image, with `OPENCV_ANDROID_JNI_DIR` beside them, one `ENV` instruction each — the env-knob owner scan reads the first name of an instruction only, so a single multi-name `ENV` would leave five of the six unowned. There is deliberately **no** bare `OpenCV_DIR`. That is the name `find_package(OpenCV)` resolves, and pointing it at the Android SDK would hijack every *Linux* OpenCV consumer in the same image. `OPENCV_ANDROID_JNI_DIR` names the Android tree instead, to be passed explicitly: ```bash cmake -DOpenCV_DIR="${OPENCV_ANDROID_JNI_DIR}" ... ``` ## Every LLVM tool is the pinned release Owner decision 2026-10-06 (BACKLOG CON71): every LLVM tool a lane can name is the pinned LLVM, `LLVM_RELEASE` on Linux and `LLVM_WINDOWS_VERSION` on Windows (23.1.3 today). That includes `clang-format` and `llvm-config`, which stayed LLVM 21 on Linux until then (CON15). **Linux.** Ubuntu's packages put about a hundred unversioned LLVM names in `/usr/bin`, all LLVM 21. The package stage's `wire_pinned_llvm_tools` diverts each of them with `dpkg-divert`: - a name the pinned LLVM has points into `/usr/local/llvm-target/bin`; - a name it lacks leaves `PATH`. The originals sit in `/usr/lib/distro-llvm-names`. - `bugpoint` is the only permanent gap: LLVM 23 removed it. - The `-NN` names (`clang-tidy-21`, `scan-build-21`, ...) still reach LLVM 21. They ask for it by name. The distro rustc's `rust-*` links are left alone too. An `apt-get install` of a distro LLVM package later does not undo the diversion. The pinned build carries lldb (without Python, Lua, libedit or curses scripting) and the LLVM utilities (`FileCheck`, `not`, `yaml2obj`, `llvm-tblgen`), so no name falls back to 21. `validate-compilers.sh smoke` checks two things and fails the image on either: - 26 tools report `LLVM_RELEASE`, lldb and `FileCheck` included; - no unversioned `/usr/bin` name resolves into a distro LLVM. **Windows.** `C:\llvm-patched\bin` is first on `PATH` once the entrypoint has run, ahead of VsDevCmd's MSVC directory (its `llvm-symbolizer` is a 23.0.0git fork) and `C:\runtime\iree\bin` (IREE's own `clang`, `llvm-link` and `FileCheck`, 23.0.0git). The patched LLVM installs the utilities, so `FileCheck` is the pinned one too. `clangd` and `lldb` come from scoop's official 23.1.3 release. `Test-Toolchain.ps1` checks the base stage's tools against `LLVM_WINDOWS_VERSION`, and `Test-Container.ps1` checks 21 tools in the finished image against clang-cl's release. **What it changes for a consumer:** - **A Linux format gate now grades with clang-format 23, as Windows always did.** Files formatted with 21 can come back changed. Re-run the formatter once and commit the result. - **`llvm-config` names an LLVM built for the image's own arch only** (CON35). A build that wants another backend from it needs its own LLVM. - **`scan-build-21` is still LLVM 21's analyzer.** Call `scan-build` for the pinned one. ## What the image stages so a run does not Seven of the contract rows are not about permissions at all. They ask whether a thing is *present*, because the alternative is that every consumer run fetches or rebuilds it. Measured in one consumer's build on 2026-09-05, before the fix (the `cargo-qa-tools` and `free-threaded-python` rows since 2026-10-06, the last two since 2026-10-09): | Row | Absent means | | --- | --- | | `flatpak-runtimes` | `flatpak list --runtime` returns **0 refs**; seven `org.freedesktop` refs, ~1.9 GB, re-downloaded per run per arch | | `appimage-runtime` | `appimagetool` refetches `runtime-` from GitHub, so packaging hangs on GitHub being reachable | | `web-lane-tools` | `wasm-pack` (258 crates) and `flutter_rust_bridge_codegen` (174) are `cargo install`ed from source in every run | | `cargo-qa-tools` | `cargo-audit`, `cargo-deny` and `cargo-tarpaulin` are `cargo install`ed from crates.io before every OxidANT security and coverage step | | `free-threaded-python` | every `3.14t` leg has uv download a free-threaded CPython first, its patch version unpinned | | `lint-tools` | `shellcheck` and `hadolint` are not on `PATH`, so OrchestrANT's coding bench grades every bash and Dockerfile answer with `[shellcheck SKIPPED: not on PATH]` | | `uv-cache-seed` | the riscv64 Python lane's `uv sync` of the `test` extra builds numpy, matplotlib, contourpy, pillow, line-profiler, psutil and pyyaml under QEMU: 108 min of a 6 h job | The first three share one verdict function; `cargo-qa-tools` and `free-threaded-python` have their own, because they compare versions (and `free-threaded-python` also where the interpreter came from). The cost of each is written down once, in `_consumer_contract_symptom`, which is also what the failure message prints. ### The Flatpak runtimes ship with the image `install_flatpak_runtime` had existed for months behind `INSTALL_FLATPAK_RUNTIMES`, which defaulted to **false** — so the capability was there and switched off, and it covered two of the seven refs. It now defaults to true and installs all seven, `Platform.GL.default` twice because the base branch and its `extra` sibling are separate refs. Flathub builds these for x86_64 and aarch64 only. On any other arch the install is a guaranteed 404 rather than a flake, so the function returns early and the row is exempt on riscv64. ### The AppImage runtime ships with the tool `appimagetool` embeds a type-2 runtime into every AppImage it builds, and fetches it at build time if it is not on disk. Upstream publishes that runtime **only** under the moving `continuous` tag — the exact mutable-asset trap that made `appimagetool` itself move to a pinned version tag (TS1, 2026-08-15), so it cannot be SHA-pinned. It is not downloaded. Every AppImage *begins* with that runtime, and `appimagetool` is already SHA-pinned, so `ensure_appimagetool_runtime` finds the squashfs superblock in the tool's file and copies the bytes before it out. Pinned transitively, correct by construction for whatever arch the tool is. It lands in `/etc/skel` as well as root's home, so the runtime user created later inherits it. ### The web-lane toolchain `flutter_rust_bridge_codegen build-web` shells out to `wasm-pack ... -Z build-std`, which resolves the nightly **channel** unless FRB's `--wasm-pack-rustup-toolchain` names the pin, so rustup auto-installed one per run through a path it calls deprecated. The package stage installs the dated pin `RUST_NIGHTLY_TOOLCHAIN`, not the channel ([why](#the-web-lane-toolchain)), with `rust-src` and `wasm32-unknown-unknown`, plus both binaries at pinned versions, on a cargo registry cachemount. `WASM_PACK_VERSION` and `FLUTTER_RUST_BRIDGE_VERSION` are advertised as image ENV and compared against what the binaries report, which is also the proof that they are installed. A slow consumer beats an image that cannot be built, so only a defect fails the stage; riscv64's from-source leg is [above](#building-the-web-lane-tools-from-source). ### The cargo QA tools The package stage installs `cargo-audit`, `cargo-deny` and `cargo-tarpaulin` at `CARGO_AUDIT_VERSION`, `CARGO_DENY_VERSION` and `CARGO_TARPAULIN_VERSION` on amd64 and arm64 (CON65). Each is its upstream release binary, verified against a `CARGO_*_LINUX_{X86_64,AARCH64}_SHA256` pin; `cargo-audit`'s aarch64 build is a glibc one, the rest are musl. A missing pin or a failed download stops the stage: there is no source fallback, which would cost hundreds of crates per tool under QEMU. - The pins are read from `versions.env` inside the stage, not forwarded as build-args, so they are not advertised ENV. The `cargo-qa-tools` contract row compares what each binary reports with the pins instead. - **riscv64 ships none of them.** None publishes a riscv64 binary, and no riscv64 lane runs them: those lanes cross-build on the amd64 image. The row is exempt there and fails as `STALE` if one ever appears. - `cargo_security_checks.sh` and `cargo_coverage.sh` call `cargo_install_pinned` (`_cargo_wrapper.sh`). It uses the binary already on `PATH` when it reports the pin, and `cargo install`s it otherwise. A plain `cargo install` would fail on these: `binary already exists`, since cargo did not install them. **So a consumer's hub pin must include this before it builds on an image that ships them.** ### shellcheck and hadolint The torch stage puts both on `PATH` in `/usr/local/bin` on every arch (CON83), at the `SHELLCHECK_VERSION` and `HADOLINT_VERSION` pins in `tool-pins.env`, the same pins the hub's lint gates bootstrap. `install-lint-tools.sh` takes each release binary through `download_verified_install` against its per-arch SHA256, and fails the stage when a tool does not report its pin. - **shellcheck** publishes a riscv64 build, pinned as `SHELLCHECK_LINUX_RISCV64_SHA256`. `lint-shell.sh` bootstraps the same asset on a riscv64 host. - **hadolint publishes none for riscv64.** The `hadolint-build` stage of `Dockerfile.torch` builds it from the release tag's source (`HADOLINT_SOURCE_SHA256`) with Ubuntu's GHC 9.10.3, the compiler its cabal file is tested with. The tag's `cabal.project` carries upstream's `allow-newer: time`; the Hackage release alone does not resolve. `HADOLINT_HACKAGE_INDEX_STATE` pins every dependency to the index of the bump day. The stage starts from the pinned Ubuntu, not the package image, so an app or chain change never re-keys it; on amd64 and arm64 it is an empty `/out`. `build-hadolint.sh` fails a binary that does not report the pin or does not flag a Dockerfile it must flag, and caches it in the cabal cache mount keyed by those three pins, so a chain rebuilds it only when one of them moves. The build is long: measured 2026-10-09 under QEMU on the 32-core host, 18221 s for about 105 packages, after 469 s of apt and 138 s of `cabal update`. BuildKit runs it in parallel with the torch stage; the lint-tool install in the final stage waits for it. The binary links libgmp, libffi and libnuma, which the riscv64 image carries. - The pins live in `tool-pins.env`, not `versions.env`: only the torch stage reads them, so a lint-tool bump rebuilds the torch stage, never the compiler. The `lint-tools` contract row compares what both binaries report with the pins, on every arch. ### The riscv64 uv cache seed PyPI has no cp314 riscv64 wheel for numpy, matplotlib, contourpy, pillow, line-profiler, psutil or pyyaml. OrchestrANT's riscv64 lane therefore built all of them under QEMU in every run's `uv sync --extra test`: 108 min of a 6 h job, numpy alone 107 min (2026-10-07). The image ships the result instead (CON83). - **The contract is a uv cache, not a wheel store.** `PYTHON_UV_CACHE_SEED` (`/opt/uv-cache-seed`) is a uv cache directory holding only the wheels uv built from source for the app lock's `test` extra. A `--find-links` store does not work here: `uv sync --locked` installs what the lock names, by URL and hash, and a local wheel is neither. uv looks a built sdist up in its cache by that same source, so a cache entry is what a locked sync can use. `uv sync --no-build` cannot use one either: it refuses every sdist, cached or not (measured 2026-10-09). - **Consumers copy it, they do not point at it.** uv writes its cache, and the seed is root-owned. `ci_tests.sh` and `ci-wheel-smoke.sh` call `uv_cache_seed_restore` (`02-toolchain/python/ci-common.sh`), which copies the seed into `uv cache dir` when its record says `seeded yes` for this arch; anything else is a log line, never an error. Another consumer runs the same copy: `cp -R "$PYTHON_UV_CACHE_SEED/." "$(uv cache dir)/"`. - **How it is built.** `06-packaging/uv-cache-seed.sh build`, in the torch stage's last layers, fetches the app at `APP_REF` (the shipped tree has no `.git`, and its riscv64 `pyproject.toml` is edited), and runs ci_tests' own sync, `uv sync --locked --dev --extra test`, without the project. The `uv-seed-` cache mount keeps the builds between chains. The seed is a copy of that cache after `uv cache prune --ci`, which drops every download and unpacked sdist and keeps the built wheels, and after removing every version the sync did not install. - **How it is proved.** At build, a fresh copy syncs the same lock into a fresh venv, and the stage fails when uv prints a `Built` line or adds a wheel under `sdists-v*/`. `seed-record.txt` names the arch, the app commit, the extras, uv and Python, every built wheel, and the proof. `uv-cache-seed.sh verify` checks that record and that every listed wheel is in the cache; the `uv-cache-seed` contract row runs it. On riscv64 it wants `seeded for riscv64`, elsewhere a `not seeded` record. - **What it costs.** Measured 2026-10-09 in `:latest`'s riscv64 child under QEMU on the 32-core build host, for OrchestrANT 3bdf80a7: 3172 s cold, building 8 wheels (the 7 above and the pure-Python sdist `antlr4-python3-runtime`), and 89 s warm, sync, copy and proof included. The seed is 21 MB; the work cache 671 MB. A chain pays the cold build once, then only for a package whose locked version moved. - **What it does not cover.** Only the `test` extra of the app the image builds (OrchestrANT). Another consumer's lock hits the seed only where it resolves the same versions from PyPI. A uv bump re-keys the cache layout, and the next chain rebuilds the seed with it; a consumer on an older image keeps a seed its newer uv may not read. ### The free-threaded Python The image ships the free-threaded twin of `PYTHON_VERSION` (`3.14.8t` today) on every arch, riscv64 included (CON66), so a `3.14t` leg downloads no interpreter. Since 2026-10-11 the hub's Python legs refuse to: a missing one fails the leg ([`python-ci.md` § Free-threaded and GIL legs](python-ci.md#free-threaded-and-gil-legs-in-one-container)). - **It is built from source in the toolchain stage**, from the same SHA256-pinned tarball as the GIL build (owner decision 2026-10-07). Until then the package stage installed uv's python-build-standalone download. `build_python.sh` builds each of its `PYTHON_VARIANTS` (`gil,freethreaded`) in turn. The second one is `--disable-gil`, with PGO and LTO on the build arch and LTO alone for each cross arch, where PGO would have to run the foreign interpreter. The cost lands in the toolchain stage, and every stage after it rebuilds when the pass changes. Measured on a 32-core host on 2026-10-07: 6.5 minutes natively, 4.7 of them the PGO profile run, and under 2 minutes per cross arch. The shipped trees are 138 MB (amd64), 120 MB (arm64) and 233 MB (riscv64). - **It lives in `/opt/python-freethreaded`**, with `/usr/local/bin/python3.14t` linked to it, so it never shadows the GIL `/usr/local`: `python3`, `pip3` and `python3.pc` stay the GIL build's. A `3.14t` request finds it on `PATH`. A plain `3.14` request never takes it ([`python-ci.md` § Free-threaded and GIL legs](python-ci.md#free-threaded-and-gil-legs-in-one-container)). The binary finds `libpython3.14t.so` through its own RUNPATH, set with `LDFLAGS_NODIST` so that it stays out of sysconfig: a `cp3XYt` wheel built against it inherits none. - **Every artifact stages one tree per arch**, at `/opt/python-cross-ft//opt/python-freethreaded`, the build arch's included and in either build mode, so `Dockerfile.package` takes the target arch's with one COPY. - A cross arch's tree is a real `make altinstall`, with its bytecode, `python3.14t-config` and config dir. The GIL cross trees are copies of the build tree. - It keeps `_ctypes`, which the GIL cross trees build without (`ac_cv_header_ffi_h=no`), because it ships as the runtime interpreter. - No tree carries the static `libpython3.14t.a` (`--without-static-libpython`): its fat LTO objects were 155 MB on amd64 and 435 MB on riscv64, and nothing links them. - No staged tree carries the test suite. The cross trees are built `--disable-test-modules`, and the build arch's stage drops `lib/python3.14t/test` to match. - `pip` is in the build arch's tree only; the cross builds run `--without-ensurepip`. - **The build stages keep it too**, for the `cp3XYt` wheel builds: toolchain, media and android carry the build arch's interpreter at `/opt/python-freethreaded` and `/usr/local/bin/python3.14t`, and each `CROSS_TARGETS` arch's tree under `/opt/python-cross-ft/`. The media stage builds its twins with it ([§ The free-threaded wheels](#the-free-threaded-wheels)). - **Each stage checks it.** - `build_python.sh` fails a cross tree without `_ssl`, `_hashlib`, `_sqlite3`, `zlib`, `_bz2`, `_lzma` or `_ctypes`. - `smoke-toolchain.sh` runs the twin and checks every staged arch's tree. - The package stage stops when the COPY'd interpreter is missing, does not report `PYTHON_VERSION` with the GIL off, or cannot import ssl, sqlite3, ctypes, zlib, lzma and bz2. - The `free-threaded-python` contract row checks the same on the shipped image, and that `sysconfig`'s prefix is `/opt/python-freethreaded`, which tells the source build from a python-build-standalone one. ### The free-threaded wheels Every wheel the media stage builds for a package whose own code declares free-threading support also ships as a proved free-threaded (`cp3XYt`) twin, in a store of its own: `/opt/wheels-free-threaded` (owner request 2026-10-07; no version in the name since 2026-10-12, so a Python upgrade renames nothing). That holds for the cross-built arches too, arm64 and riscv64, since 2026-10-09 in source (CON75, CON79 1b). The GIL wheels are built exactly as before. - **Which packages.** One table decides, `linux/scripts/03-media/free-threaded-twins.txt`, and both lanes read that file: `ft_wheel_table` here, `Get-FreeThreadedTwinTable` on Windows, so their verdicts cannot differ. Each row names the `versions.env` pin its evidence was read at, and `test-free-threaded-wheels.sh` fails when that pin moves, so a bump re-reads the evidence. The media RUNs that build or store a twin mount it beside `free-threaded-wheels.sh`, and a missing table stops them rather than reading every package as unclassified. | Package | Verdict | Evidence, at the pinned tag | | --- | --- | --- | | `onnxruntime` (every flavour) | twin | `PYBIND11_MODULE(..., py::mod_gil_not_used())`, pybind11 v3.0.2 | | `av` (PyAV) | twin | `setup.py`: `"freethreading_compatible": True` | | `apache-tvm-ffi` | twin | `core.pyx`: `# cython: freethreading_compatible = True` | | `iree-base-runtime` | twin | `nanobind_add_module(... FREE_THREADED ...)` | | `iree-base-compiler` | twin | MLIR's `nanobind_add_module(... FREE_THREADED ...)` | | `torch` | twin while `FT_TORCH_TWIN=1` | `torch/csrc/Module.cpp`: `PyUnstable_Module_SetGIL(module, Py_MOD_GIL_NOT_USED)`; the chain builds torch on riscv64 only | | `numpy` | twin while `FT_TORCH_TWIN=1` | `multiarraymodule.c`: `{Py_mod_gil, Py_MOD_GIL_NOT_USED}`; no chain build ships a numpy wheel, so the row only classifies one | | `apache-tvm` | none needed | `wheel.py-api = "py3"`: its one `py3` wheel installs on `3.14t` | | `torchvision` | none needed | no CPython extension module of its own | | `onnxruntime-genai`, `ai-edge-litert`, `hailort` | GIL only | no free-threading marker | | libcamera's pycamera, OpenCV's `cv2` | GIL only | no marker, and they ship in the tree, not as wheels | - **A knob row.** `twin:` names a `versions.env` switch: the row is a twin only while that key is `1`. `FT_TORCH_TWIN=1` is the default, and `Dockerfile.media` takes it in the app-wheelhouse RUN, which builds the twin, and in the final RUN, whose store check expects it. Windows reads such a row as no twin. - **How a twin is built.** Each build runs a second pass on a `3.14t` venv whose build executors are pinned to the GIL build venv's own versions, in the GIL pass's tree, so only what sees Python rebuilds. Measured in `:latest` on 32 cores (2026-10-07): | Package | Twin pass, cold compiler cache | Warm | What the twin pass rebuilds | | --- | --- | --- | --- | | ONNX Runtime CPU (GIL pass: 760 s at `-j11`) | 97 s | 11-40 s | 539 of 3206 objects, every target whose compile line carries the Python and NumPy include dirs; of the wheel's four libraries only `onnxruntime_pybind11_state.so` differs | | ONNX Runtime CUDA, `:latest-nvidia` (GIL RUN: 12 672 s, 578 CUDA objects) | 192 s | 32 s | no CUDA object; the C++ targets that see Python, from the compiler cache | | ONNX Runtime MIGraphX, `:latest-rocm` (GIL RUN: 1311 s, 590 HIP objects) | | 57 s | no HIP object; 106 C++ objects | | PyAV | 28 s | 14 s | all 50 extensions, re-Cythonized because the fresh venv's Cython includes are newer: all of PyAV sees Python | | apache-tvm-ffi | 29 s | | all 34 steps: scikit-build-core's reconfigure rebuilds the small C++ core too | | IREE compiler and runtime (GIL pass: 55 s configure, 2334 s build at `-j16`) | 25 s | | 58 of 5452 objects, the bindings alone (below) | The GPU rows are chain runs, not the 32-core measurement: nvidia 20261010-125556 (first) and 20261010-202726 (warm), rocm 20261009-152508. Each GPU build hands the pass its own `BUILD_ARGS` and build dir, and the pass appends only `Python_EXECUTABLE`, as the last define. No GPU script passes a Python include dir, so FindPython re-resolves the `3.14t` headers, NumPy include and SOABI in the warm cache (CMake 4.4.4, checked on a scratch project). A carried GIL `Python_INCLUDE_DIR` would survive that reconfigure, which is the riscv64 torch twin's bug below. `test-free-threaded-wheels.sh` holds both GPU flavours to that. The IREE row is from a cold compiler cache, the full in-tree LLVM, and a settled GIL tree. The runtime-only tree, before the settle, took 130 s at `-j4` and recompiled 464 of its 723 objects. - **Why the GIL tree configures twice.** IREE's first configure leaves `IREE_HAVE_LIBBACKTRACE` unset and a second one sets it ([linux-cross-builds.md § IREE](linux-cross-builds.md#iree-linux-lane)). On an unsettled compiler tree the twin's reconfigure would change 1272 compile lines, 757 of them compiler sources, and relink the 339 MB `libIREECompiler.so`. The GIL pass therefore configures twice (17 s more), and the twin's reconfigure (11 s) then changes only what sees Python. - **What the twin pass rebuilds.** 27 IREE compiler binding objects, 20 runtime binding objects and 11 MLIR Python objects. It then relinks nanobind's free-threaded libraries, `libMLIRPythonSupport-mlir.so` and the 15 modules. - **What differs from the GIL wheels.** The extension modules, `libMLIRPythonSupport-mlir.so`, and nanobind's library (`libnanobind-mlir.so` in the GIL wheel, `libnanobind-ft-mlir.so` in the twin), plus `WHEEL` and `RECORD`. Everything else is byte-identical, including `libIREECompiler.so` and the ten runtime tools. - **How a cross twin is built.** The amd64 build host's own `3.14t` (`/opt/python-freethreaded`) runs the build tooling, and `ft_target_resolve` reads the target's tree, which the toolchain stages at `/opt/python-cross-ft/` (CON66). That gives the target's headers, its `libpython3.14t.so`, its `EXT_SUFFIX` (`.cpython-314t-riscv64-linux-gnu.so`, read from the tree's `_sysconfigdata_t_*.py` without running it) and the wheel platform (`linux_riscv64`). `ft_target_env` then points the build's sysconfig at a private copy of that file, the way the GIL cross pass points it at the target's GIL one: - PyAV compiles against the target headers with that suffix. - tvm-ffi's scikit-build-core reruns in the cross pass's build dir. - IREE reconfigures its runtime tree with the target `3.14t` include dir and library. - ORT passes `Python_INCLUDE_DIR` to `build.sh`. - The riscv64 torch reruns its warm scikit-build tree with the target `3.14t` on the command line, where the toolchain file's cache entries cannot reach. Its `CMAKE_ARGS` are built afresh rather than appended to the GIL pass's: that pass's `-DPYTHON_INCLUDE_DIR` heads `torch_python`'s include path, so a twin that kept it compiled against the GIL `pyconfig.h`. That twin's `libtorch_python.so` was byte-identical to the GIL wheel's, and it segfaulted in `dlopen` under the target `3.14t` (2026-10-10). The fixed pass takes about 240 s warm, and its `libtorch_python.so` imports `_Py_DecRefShared` and `PyUnstable_Module_SetGIL`. A cross build that is missing the target tree fails its twin, rather than skipping it. - **How a twin is proved.** Before it is stored, `ft_soabi_gate` wants a `cp3XY-cp3XYt` name whose platform tag names the suffix's machine, and the free-threaded SOABI on every version-tagged module. A cross build gates against the target's suffix. Natively, a fresh `3.14t` venv takes the wheel alone, where `free-threaded-wheel.py prove` loads every compiled module and fails when the GIL comes back on. A cross twin is unpacked instead and proved by the target's own `3.14t` under qemu-user (`ft_prove_wheel_on_target`): `-L` names the root holding the target's loader, and the target tree's `lib` leads `LD_LIBRARY_PATH`. A bare `.so` counts as a module only when it exports `PyInit_`, so ORT's bundled `libonnxruntime_providers_shared.so` is not loaded as one. Before a module loads, its enclosing package is imported. A package whose body cannot run in a `--no-deps` venv is registered unrun instead, as an import in progress leaves it: `torch._C`'s init imports `torch`, whose body needs `typing_extensions`. numpy's test modules need the real `numpy`, which is why the import is tried first. A proof never fails with an empty reason. `faulthandler` prints a segfault's stack, the verdict names the exit status or signal, and `FT_PROVE_TRACE` names the module that was loading. A twin that fails any of it fails its RUN; inside TVM's best-effort build that withdraws TVM, as any failure there does. - **How the stores stay apart.** `collect-artifacts.sh` moves the twins into `FT_WHEELS_DIR` only, and the ORT twin gets its own manifest, `ort-provenance-free-threaded.sha256`. `verify-wheels.sh` is ABI-exact on both stores: a `cp3XYt` wheel or module in `/opt/wheels` fails it, and so does anything but `cp3XY-cp3XYt` in the twin store. The final RUN then repairs and verifies the twins with `--free-threaded`, and `free-threaded-store.sh` proves them again on the repaired bytes. A cross build proves them under qemu-user again. A build, native or cross, fails there when the store is not exactly one twin per GIL wheel the table expects a twin of, or when a GIL wheel has no row. It writes `free-threaded-store.txt`: the build mode, each twin, one `want ` line per family it expects, and one line per package without one, with its reason. - **What ships.** `Dockerfile.torch` copies `/opt/wheels-free-threaded` into `:latest`, late, so a new twin re-keys no venv layer, and the export delivery seals it with `/opt/wheels`. Nothing installs a twin: an X.Yt venv takes them with `uv pip install /opt/wheels-free-threaded/`. The image names the store in `PYTHON_WHEELS_FREE_THREADED`, as Windows' does since CON91 (its older images set `PYTHON_WHEELS_CP314T`, which every reader still takes), and `uv_reconcile_chain_ort` gives an X.Yt venv's ONNX Runtime from it ([`python-ci.md` § Trap 3](python-ci.md#trap-3--onnx-runtime-comes-from-the-chain-not-pypi)). - **How the shipped image proves it.** The runtime smoke's `FT-STORE` gate (`06-packaging/check-free-threaded-wheels.sh`) wants the store, its record and that variable on every arch. On every arch, native or cross-built, the store holds exactly the twin families of `/opt/venv` that the record wants, the installed ORT flavour's own twin included. Each twin loads every compiled module on `/usr/local/bin/python3.14t` with the GIL off, under QEMU on a foreign arch. A `/opt/venv` package whose family the record does not want gets an `OK` line naming it: PyPI's torch on amd64 and arm64, and PyPI's IREE compiler on a cross arch, whose chain builds the IREE runtime only. The GPU images pass it with both ORT twins: `:latest-nvidia` (2026-10-11, index sha256:810f36b3) proves `onnxruntime_gpu` and `onnxruntime_dnnl`, `:latest-rocm` (2026-10-09, index sha256:a5d68fbe) `onnxruntime_migraphx` and `onnxruntime_dnnl`, and the ORT census grades each twin's members against the flavour's `ort-provenance-free-threaded.sha256`. - **Not covered yet.** The cross twins are in source since 2026-10-09 and unproved until a chain publishes them. Proved so far without a chain: PyAV 19.0.1's twin, cross-built against the published riscv64 image's FFmpeg and `3.14t` tree. All 50 modules loaded with the GIL off on riscv64 under QEMU, through `FT-STORE`'s own probe and verdict. A minimal C module went through `ft_store_twin` the same way, and its `Py_mod_gil`-less control was refused. Windows builds no twin of a knob row. The IREE compiler twin was proved locally in `:latest` amd64 on 2026-10-07, but not yet in a chain: - `prove` loaded its 14 compiled modules with the GIL off. - A 3.14t venv parsed MLIR in-process, compiled an `add` function to a VMFB and ran it on `local-task` from 8 threads. - `iree-compile` and `iree-run-module` ran from the twins. ## The Windows image ships lavapipe The Windows amd64 image and the arm64 bundle carry Mesa's lavapipe — a CPU Vulkan device, the only software one that also runs the ray-tracing goldens — with the target arch's Khronos loader and `vulkaninfo.exe`, all SHA256-pinned, in `C:\runtime\lavapipe` (CON50). The image sets `LP_NATIVE_VECTOR_WIDTH=256`, because Mesa 26.2's BVH sort needs 8-lane subgroups and arm64's native 128 bits give 4. The amd64 image registers the ICD in HKLM at build time — a container process is elevated, and the loader ignores `VK_DRIVER_FILES` there. The arm64 bundle cannot register it while it is packed, so the device needs the one-time HKLM registration the bundle README names; `Test-Arm64Bundle.ps1` runs `vulkaninfo --summary` and fails a device without the llvmpipe device. `:winamd64` carries the device since the 2026-10-04 republish (the smoke lists llvmpipe); the `:winarm64` bundle is published and its device gate lists llvmpipe (12/12 on the Snapdragon X). ## The ort crate links the chain ONNX Runtime Both published images set the environment of the Rust `ort` / `ort-sys` crates, so a Rust build inside them links the chain ONNX Runtime and can never fetch pyke's prebuilt one (G3 of the [ONNX Runtime single-source rule](onnxruntime-single-source.md)). | Variable | Windows (`windows/Dockerfile`, final) | Linux (`linux/Dockerfile.package`, every variant) | |---|---|---| | `ORT_LIB_LOCATION` | `$ONNX_ROOT\lib` (holds `onnxruntime.lib`) | `${ONNXRUNTIME_OUTPUT_DIR}/lib` = `/usr/local/lib/onnxruntime-cpu/lib` | | `ORT_PREFER_DYNAMIC_LINK` | `1` | `1` | | `ORT_SKIP_DOWNLOAD` | `1` | `1` | | `ORT_DYLIB_PATH` | `$ONNX_ROOT\bin\onnxruntime.dll` | `${ONNXRUNTIME_OUTPUT_DIR}/lib/libonnxruntime.so` | Why these four, read from `ort-sys` 2.0.0-rc.13 (the checksum OxidANT's `Cargo.lock` pins) and `ort` 2.0.0-rc.13: - `build/main.rs:47-68` reads `ORT_LIB_PATH`, then `ORT_LIB_LOCATION`, before the download branch. With `ORT_PREFER_DYNAMIC_LINK` set to `1` or `true` the build script emits `rustc-link-lib=onnxruntime` from that directory; without it, ort-sys tries a static link the chain does not ship. - The first SET one of `CARGO_NET_OFFLINE`, `ORT_SKIP_DOWNLOAD` and `ORT_OFFLINE` decides, and only exactly `1` or `true` skips the download. An empty value is set, so an empty `CARGO_NET_OFFLINE` re-arms the download, and an empty `ORT_LIB_PATH` still outranks `ORT_LIB_LOCATION`. - `ort`'s load-dynamic opens `ORT_DYLIB_PATH`, otherwise a bare file name, which on Windows reaches `System32\onnxruntime.dll` (Windows ML 1.17) before PATH. - `ort`'s default features include `download-binaries`. **What it means for a consumer.** - An `ort-sys` build in the image links the chain import library, so a shipped bundle must carry the chain DLL or `.so` beside the app. - Keep `ORT_LIB_PATH` unset and never export a falsy `CARGO_NET_OFFLINE`. - On the Linux GPU variants the variables name the CPU chain build, which is what `ld.so` resolves. A GPU consumer points both at `onnxruntime-gpu/lib`, which is chain-built too. On `:winarm64` they name the aarch64 chain ORT. - `ORT_DYLIB_PATH` is baked in, so inside the image a program that honours it loads the image's copy, not one staged beside its exe. A bundle smoke run inside the image should unset it to exercise the staged copy. **How it is proven on the shipped image.** Windows: Test-Container section 19, `Get-OrtCrateEnvFinding`, on every lane. Linux: the contract row `ort-crate-env`. Its probe reports `ORT_LIB_PATH` and `CARGO_NET_OFFLINE` as `` when they are not set, and the row passes only with `ORT_LIB_PATH` unset and `CARGO_NET_OFFLINE` unset, `1` or `true`; a missing line reads as set-but-empty and fails. Both compare values untrimmed, as ort-sys does. The static half is `verify-critical-fixes.sh` fix11. Not covered: Rust builds outside the images, and the live in-image `cargo build -vv` of a scratch ort-sys crate, which is still to run at the next image build. ### The chain ONNX Runtime wheels - **Linux:** `/opt/onnxruntime-wheels`, advertised as `ORT_CHAIN_WHEEL_DIR`, holds exactly the ORT and GenAI wheels `/opt/venv` was installed from. `setup-torch-venv.sh stage_chain_ort_wheels` proves the venv against it at build time. The census that proof runs ships at `/opt/scripts/03-media/final/ort-venv-census.py`; its command line and output lines are a contract that `python_uv.sh` and `setup-torch-venv.sh` read. The free-threaded twins of those flavours sit in `/opt/wheels-free-threaded` (`PYTHON_WHEELS_FREE_THREADED`). - **Windows:** the same wheels sit in `C:\runtime\wheels` (`PYTHON_WHEELS`). The census ships at `C:\temp\scripts\ort-venv-census.py`, beside the module copy whose `Sync-UvChainOnnxRuntime` runs it; `Build-TorchApp.ps1` embeds its own copy. - **Who gets them:** a consumer on the hub's Python CI has its venvs reconciled onto them inside our images ([`python-ci.md` § Trap 3](python-ci.md#trap-3--onnx-runtime-comes-from-the-chain-not-pypi)). Anyone else calls `uv_reconcile_chain_ort ` or `Sync-UvChainOnnxRuntime -VenvPath `. ### Prove your bundle ships the image's ONNX Runtime Run the ORT census (G6) inside the image you built in, after staging the runtime DLLs or `.so` files. The reference is that image's chain ORT. - Windows: `Import-Module C:\temp\scripts\modules\WindowsOrtProvenance.Common.psm1`, then `if (-not (Test-OrtProvenanceTree -Root )) { exit 1 }`. `-PassThru` returns the census object. - Linux: `bash third_party/ANTfrastructure/linux/scripts/06-packaging/check-ort-provenance.sh `. Tree mode models a client loader. A DLL's imports resolve from the directory of the exe that loads it, then System32 (assumed to hold Windows ML's `onnxruntime.dll`), then PATH, never from the DLL's own directory; a `.pyd` also searches its own directory first. So ship `onnxruntime.dll` beside the exe, not beside a plugin. A bundle fails as STALE with an older chain DLL, FOREIGN with a NuGet, PyPI or pyke ORT (statically linked ones included), and UNRESOLVED when an importer has no app-local chain copy or a Linux `.so` has no `$ORIGIN` RUNPATH to one. ## Browser tests run in Chrome for Testing `flutter test --platform chrome` needs a browser, and Ubuntu 26.04 ships Chromium as a snap only. The image ships Google's Chrome for Testing instead (CON50), the build Playwright also downloads, from Google's own bucket: | what | where | pin in `versions.env` | | --- | --- | --- | | Chrome for Testing (Stable) | `/opt/chrome-for-testing/chrome`, started by `/usr/local/bin/chrome` | `CHROME_FOR_TESTING_VERSION`, a SHA256 per arch | | chromedriver of the same version | `/usr/local/bin/chromedriver` | `CHROMEDRIVER_LINUX64_SHA256`, `CHROMEDRIVER_LINUX_ARM64_SHA256` | `CHROME_EXECUTABLE=/usr/local/bin/chrome` on amd64 and arm64. It is **empty on riscv64**, where Google publishes no build, so a lane runs the browser leg only when the variable is set: ```bash if [ -n "${CHROME_EXECUTABLE:-}" ]; then flutter test --platform chrome else echo "no browser in this image on $(uname -m); browser tests skipped" fi ``` A lane that runs the hub's Dart gate gets this by passing `--test-platform chrome` to `05-frameworks/flutter/flutter_checks.sh` (since 2026-10-05). Where the variable is empty it runs the tests on the VM instead, so the gate never passes on zero tests. A test file that reads the checkout with `dart:io` cannot run in a browser. Mark it `@TestOn('vm')` (or one test `testOn: 'vm'`), and the browser leg skips it while the VM leg still runs it. Measured on OmniAccelerANT (2026-10-01, the proof image below): 37 tests pass in Chrome, including all four of the native plugin's. `pinned_artefacts_test.dart` and `settings_asset_paths_test.dart` fail there only for `dart:io` (*"Unsupported operation: Platform._script"*, *"_Namespace"*), and one `webrtc_settings_test.dart` case asserts the off-web behaviour. `06-packaging/install-chrome-for-testing.sh` is the package stage's last RUN, after the emulator, so a Chrome bump re-runs one layer of about 0.45 GB (+0.2 GB compressed per arch). It installs the zip's `deb.deps` under their Ubuntu 26.04 names (only `fonts-liberation` was new), checks every ELF with `ldd`, compares both binaries with the pin, and renders a JavaScript page headless. The runtime smoke's `chrome` row repeats the last two as uid 1001. ### Why the wrapper passes `--no-sandbox` and `--no-zygote` `/usr/local/bin/chrome` is `exec /opt/chrome-for-testing/chrome/chrome --no-sandbox --no-zygote --disable-dev-shm-usage "$@"`. Point `CHROME_EXECUTABLE` at it, never at the binary. - **Sandbox.** Chrome sandboxes its renderers with user namespaces, or with a setuid-root helper. The image ships no setuid helper. Whether a container may create user namespaces is the host's call: Docker's default seccomp profile, the AppArmor restriction of Ubuntu 23.10+ hosts and qemu-user can each refuse them. Measured 2026-10-01 as uid 1001: the bare binary started on amd64 under Rancher Desktop, and on arm64 under qemu-user it died with *"No usable sandbox! If you are running on Ubuntu 23.10+ …"*. The container is the isolation boundary and a test browser loads only the consumer's own pages, so the wrapper turns the sandbox off everywhere. Flutter passes `--no-sandbox` itself only when headless; the wrapper also covers `flutter run -d chrome` under Xvfb and chromedriver. - **Zygote.** Under qemu-user the zygote's children die with *"qemu: uncaught target signal 5 (Trace/breakpoint trap)"*, and `--dump-dom` prints nothing. The package stage builds arm64 under QEMU, so its own headless check failed until the wrapper passed `--no-zygote` (a page then took 21 s). Without a sandbox the zygote buys nothing on native hardware either. - **`/dev/shm`.** A container's defaults to 64 MB, which a renderer can exhaust; `--disable-dev-shm-usage` moves that to `/tmp`. ## The Android emulator runs on amd64 with KVM The amd64 image ships the Android emulator and one x86_64 system image with ARM translation, so an arm64-v8a APK installs and runs (CON50). Both come from Google's zips, each SHA256-pinned, and their SHA-1 equals Google's repository XML: | what | where | pin in `versions.env` | | --- | --- | --- | | emulator | `$ANDROID_HOME/emulator` | `ANDROID_EMULATOR_VERSION`, `ANDROID_EMULATOR_BUILD` | | `system-images;android-;google_apis;x86_64` | `$ANDROID_HOME/system-images/android-/google_apis/x86_64` | `ANDROID_EMULATOR_API`, `ANDROID_EMULATOR_SYSIMG_REVISION` | | `android-avd.sh create\|start\|stop [name]` | `/usr/local/bin` | — | - **amd64 only.** Google publishes the Linux emulator for x86_64 hosts only, and GitHub's arm64 runners have no KVM. arm64 and riscv64 record that in `/opt/android/.android-emulator-off`. It adds 4.66 GB to the amd64 image (+2.1 GB compressed); the system image is 3.5 GB of it. - **API 35, not 30.** API 30's translator (`ndk_translation` 0.2.2) stops OmniAccelerANT's release APK right after the Flutter engine starts: *"ndk_translation: Undefined instruction 0x7ee1b800"* (scalar `FCVTZU`), then SIGILL. API 35's translator runs the same APK. `test-chrome-and-emulator.sh` refuses an older pin. A lane needs an x86_64 host with KVM, and the device passed in: ```bash docker run --device /dev/kvm --group-add "$(stat -c %g /dev/kvm)" \ ghcr.io/kataglyphis/kataglyphis_beschleuniger:latest bash -lc ' android-avd.sh start # creates the AVD once, boots headless, waits for sys.boot_completed adb install --abi arm64-v8a build/app/outputs/flutter-apk/app-release.apk android-avd.sh stop' ``` - **The KVM device.** uid 1001 is not in the device's group, hence `--group-add`; without it `android-avd.sh start` stops with *"/dev/kvm is not usable by uid 1001"*. On a GitHub-hosted x64 runner, make the device usable first: `echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules`, then `sudo udevadm control --reload-rules && sudo udevadm trigger --name-match=kvm`. - **The AVD.** It lives in `$ANDROID_AVD_HOME` (default `~/.android/avd`), because `$ANDROID_HOME` is root-owned. `android-avd.sh` writes the two ini files itself: `avdmanager` does not recognise an emulator installed from its zip (*"\"emulator\" package must be installed!"*). `AVD_PORT`, `AVD_BOOT_TIMEOUT`, `AVD_RAM_MB`, `AVD_CORES` and `AVD_DATA_SIZE` tune it. - **`--abi arm64-v8a`.** An APK that also carries an `x86_64` slice is otherwise installed as x86_64, and a library built for arm64 only is then missing. - **Measured 2026-10-01** under Rancher Desktop with KVM, in the proof image: boot in 25 s, `ro.product.cpu.abilist=x86_64,arm64-v8a`, and OmniAccelerANT's release APK installed with `primaryCpuAbi=arm64-v8a` and stayed up. - **What the image cannot prove.** The build host has no KVM, so the runtime smoke never boots it. Its `android-emulator` row checks the parts as uid 1001: the emulator runs, both versions equal the pins, and `android-avd.sh` is on `PATH`. ## DeepStream in `:latest-nvidia` `:latest-nvidia` (amd64) carries NVIDIA DeepStream 9.1 (CON42). What a consumer can rely on was measured on the published index `sha256:810f36b3…` on 2026-10-12, on an RTX 2080. How the tree is built and gated, and every pipeline that ran: [`linux-accelerator-images.md` § DeepStream](linux-accelerator-images.md#deepstream-nvidia-variant). | What | Where, or what to do | | --- | --- | | DeepStream root | `/opt/nvidia/deepstream/deepstream`, a link to the versioned `/opt/nvidia/deepstream/deepstream-9.1`; the SDK headers are in its `sources/includes` | | GStreamer plugins | the link `/opt/gstreamer/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream` puts them on the image's own plugin path. They registered with `GST_PLUGIN_PATH` unset, replaced or appended to, so a consumer sets nothing | | `nvstreammux` | `ENV USE_NEW_NVSTREAMMUX=yes` selects the new mux, the only one that runs here. It has no `width`/`height`; a source with several surfaces per frame, such as `nvdewarper`, needs `num-surfaces-per-frame` | | TensorRT | DeepStream's TensorRT 10.16.1, on the loader path through `/etc/ld.so.conf.d/010-deepstream-tensorrt.conf`: `libnvinfer.so.10` resolves to `/opt/nvidia/deepstream/tensorrt-10.16.1.11/lib`. The image has no TensorRT 11 | | Helper libraries | `libcustom2d_preprocess.so`, `libpostprocess_impl.so` and `libcustom_videoimpl.so` live in `…/deepstream/lib/`, not in `lib/gst-plugins/`. NVIDIA configs that name the latter need editing | | Sample data | not in the image. NVIDIA's `deepstream-sample-data_9.1.0.deb` installs into the tree with `dpkg -i`, as root | **A GPU host.** Every pipeline ran with the GPU handed in as a CDI device, on `nvidia-container-toolkit` 1.20.1 and driver 595.58.03, the minimum NVIDIA names for DeepStream 9.1; `--gpus all` exposed the GPU as well. The host setup, rootless included, is linked from [§ What needs a GPU host](linux-accelerator-images.md#what-needs-a-gpu-host). The toolkit may load the image's CUDA 13.4 compat driver in place of the host's, which is why `nvidia-smi` inside reports CUDA 13.4. DeepStream ran on either. **How it is proven.** The runtime smoke's `check_deepstream` registers the elements with CUDA's driver stubs on every nvidia image. Everything else needs a GPU, which no lane has, so it was measured by hand. Three traps: - **No GPU, no DeepStream.** Without the driver's `libcuda.so.1` the plugins cannot load, and the GStreamer registry blacklists them. Reused once a GPU is there, that registry still hides them: 30 files stayed blacklisted. The image ships no registry; never carry `~/.cache/gstreamer-1.0` from a GPU-less run into a GPU one. - **A root-owned tree.** The runtime user (uid 1001) ran the decode, inference, tracker and demux pipelines and built its own engine, but `nvinfer` cannot store an engine next to a model in the root-owned DeepStream tree. It logs `failed to serialize cude engine` and builds the engine again on every start. Give `model-engine-file` a writable path. - **Two elements do not work here:** the legacy `nvstreammux` and `nvblender`. Both are NVIDIA's prebuilt code, and both read a buffer's mapping after releasing it, which GStreamer 1.28 and newer turn into a NULL pointer. `nvblender` also stalls with two inputs, on GStreamer 1.24 too.