What :latest promises its consumers

This page is the contract side of the runtime image: the properties another repository’s CI lane may build on, written down so they are a promise rather than an accident of the last build. Everything here is asserted per arch by check_consumer_contract in linux/scripts/06-packaging/smoke-runtime-image.sh, against the shipped bytes, as the user the image ships.

It exists because on 2026-09-04 four properties a consuming lane depended on were all broken at once in :latest-cross, every runtime gate stayed green, and the lane in the other repo is where they were found. A property nobody wrote down is a property nothing can keep.

The contract

Run as the image’s own user (kataglyphis, uid 1001), with no privileged flags and no extra -e:

#	Promise	Consumer symptom when it breaks
1	CCACHE_DIR and SCCACHE_DIR point outside /workspace and are writable	the cache is written into the consumer’s bind-mounted checkout, pollutes their working tree, can be swept into CI artifacts, and on a non-ext4 host mount flatpak-builder aborts: “Can’t initialize ccache use: Failed to set permissions of /workspace/.ccache/disabled/ccache.conf: Operation not permitted”
2	$RUSTUP_HOME/tmp and $CARGO_HOME are writable	“could not create temp file …: Permission denied (os error 13)” — Corrosion, cargokit and flutter_rust_bridge_codegen cannot run. Not workaroundable by redirecting the variable: the toolchains live in that tree, and rustup toolchain install (fRB asks for the nightly channel, the image pins a dated nightly) writes there too
3	ANDROID_HOME and ANDROID_SDK_ROOT are set, $ANDROID_HOME/platform-tools exists, and the SDK’s cmdline-tools/latest/bin + platform-tools are on PATH (amd64; arm64 and riscv64 ship no SDK and leave both empty, why)	flutter build apk stops with “[!] No Android SDK found”. Under CodeQL’s database create --command=… the exit 1 aborts before the database is finalised, so the lane reports “bundle source directory not found: build/app/outputs/flutter-apk” — three steps from the cause
4	java is on PATH and JAVA_HOME names a JDK with bin/javac	Gradle stops the Android lane with “JAVA_HOME is not set and no ‘java’ command could be found in your PATH”, and flutter doctor reports “No Java Development Kit (JDK) found”
5	appimagetool is READABLE by the image user, not merely executable	it is an AppImage and reads /proc/self/exe for its own squashfs offset, so mode 711 gives “Cannot open /proc/self/exe: Permission denied” and produces no .AppImage
6	Every path under /opt/flutter is owned by uid 1001, packages/flutter_tools/.dart_tool included	flutter pub get fails with “Cannot open file … package_config.json (OS Error: Permission denied, errno = 13)”
7	CHROME_EXECUTABLE names Chrome for Testing, which renders a page headless as uid 1001, and chromedriver of the same version is on PATH (amd64, arm64; why)	flutter doctor reports “Cannot find Chrome executable at google-chrome”, and flutter test --platform chrome has no browser
8	On amd64, $ANDROID_HOME holds the pinned emulator and one x86_64 system image with ARM translation, and android-avd.sh is on PATH (why)	an Android lane has no device: adb reports “no devices/emulators found”

Row 6 is the one a consumer cannot repair at runtime. The directory sits in a read-only overlay layer, so a non-owner can neither empty nor rename it — both were attempted and refused — and the only workaround is mounting a tmpfs with mode=1777 over it. Rows 1–5 are merely expensive to work around, and the point of writing them down is that nobody should have to.

/workspace is the WORKDIR and the consumer’s checkout. Nothing the image generates by itself may land there; that is what makes row 1 a location assertion and not only a permission one.

The compiler-cache defaults the shared library already declares (linux/scripts/01-core/compiler-cache.sh) are /var/cache/ccache and /var/cache/sccache, and the image ships both as drwxrwxrwt. An image ENV that contradicts our own library is the failure shape row 1 guards.

Neither cache directory is a VOLUME. The image declares exactly one, /workspace (VOLUME ["${WORKDIR}"] in Dockerfile.torch, confirmed in Config.Volumes of all three shipped children), so a container’s compiler cache lives in its writable layer and dies with it. A lane that wants the cache to survive has to mount something there itself — and that is the reason row 1 is a LOCATION assertion: a cache pointed back into the bind-mounted checkout would persist, by polluting the consumer’s working tree.

The Android lane needs a JDK

02-toolchain/android-sdk.sh installs the full openjdk-21-jdk in the stage that BUILDS the SDK, but Dockerfile.package copies only /opt/android-sdk — apt had put the JDK in /usr/lib/jvm, which is not part of that COPY. The runtime image therefore carried a complete Android SDK and no Java at all, and the failure surfaces in Gradle rather than in Flutter, one layer below where a reader looks.

The runtime image installs JDK_PACKAGE (01-core/versions.env, openjdk-21-jdk-headless, ~286 MB with its JRE) by name rather than through append_available_packages, which SKIPS what apt does not have: a silently skipped JDK would ship the same broken image the gate exists to catch. Headless is deliberate — a container build needs javac, not AWT.

JAVA_HOME cannot be written literally, because Ubuntu’s path carries both the version and the architecture (java-21-openjdk-riscv64). anchor_java_home resolves it once from the installed javac and parks the symlink /usr/lib/jvm/default-java, which the image ENV names. A JDK bump then needs no Dockerfile edit, and a JDK that installed no compiler fails the stage instead of shipping a JRE that Gradle cannot use.

Executable is not usable

ensure_appimagetool (02-toolchain/packaging-deps.sh) downloads into a mktemp file, which is 0600, and made it executable with chmod +x — which adds the x bits and leaves r for the owner only, i.e. 0711. mv carries that mode to /usr/local/bin/appimagetool, so the tool was executable by everyone and readable by root alone.

That is fatal for this particular tool and invisible for most others: appimagetool is itself an AppImage, so it opens /proc/self/exe to find where its squashfs payload starts. As uid 1001 that open is refused and the run dies with Failed to get fs offset for /proc/self/exe, having produced nothing.

The install site now sets 0755 explicitly rather than relying on +x over whatever mode the download landed with, and the contract gate asserts readability rather than presence — command -v finding it says nothing about whether the image’s own user can run it.

How the gate proves it

One nerdctl run, not one per assertion. _consumer_contract_probe emits facts only — WHO, ENV, WRITE, DIR, FACT lines and a CCPROBE_DONE sentinel — and _consumer_contract_verdicts reaches every verdict on the host, so each failure path is provable from a recorded probe capture rather than from a doctored 30 GB image. linux/scripts/tests/test-runtime-image-gates.sh drives it with the capture measured in the broken 2026-09-04 image and with the fixed one.

Three details are load-bearing:

• Writability is a real create + delete of a file in the directory, not [ -w ]. access(2) answers yes for uid 0 and says nothing about a read-only layer.

• The probe must have run as the image’s own Config.User. As root every directory answers writable, so a probe that reports any other identity fails the gate outright instead of reporting every row green.

• No row may report nothing. A missing fact is NOFACT and fails, an empty row table fails as asserted NOTHING, and a verdict verb no arm handles fails. A gate arm that can only ever skip is how all four defects shipped.

Each red row quotes the symptom from the consuming repository’s log, not our path, because that is the sentence someone will paste into a search.

Per-arch exemptions

_consumer_contract_exempt is a <arch>:<row> table, same contract as _parity_exempt: listed means reviewed, and an arm that stops applying fails so the table cannot rot in place. Each arm is re-checked by its own probe fact, named by _consumer_exempt_fact; yes is STALE and names the arm for deletion, a missing fact is NOFACT and never a grant.

Eight arms. The first two were measured on the image shipped 2026-09-05 rather than argued from the build graph, the next three on the CON50 proof images of 2026-10-01:

arm	rot fact	what the image reports
dart-tool	flutter-sdk	/opt/flutter exists and is empty, so packages/flutter_tools/.dart_tool is absent and the row would read as unwritable. Upstream publishes no riscv64 SDK; check_flutter asserts that absence instead
appimagetool	appimagetool-readable	no appimagetool on PATH at all — packaging-deps.sh’s asset table covers x86_64/aarch64/armhf/i686 and refuses the rest
flatpak-runtimes	flutter-sdk	no refs: Flathub builds the freedesktop runtimes for x86_64 and aarch64 only, and the installer skips every other arch
appimage-runtime	flutter-sdk	no runtime: it is carved out of appimagetool, which riscv64 does not have
riscv64:chrome	chrome	CHROME_EXECUTABLE is empty: Chrome for Testing publishes linux64 and linux-arm64 only
arm64:android-emulator	android-emulator	no $ANDROID_HOME/emulator: Google publishes the Linux emulator for x86_64 hosts only
riscv64:android-emulator	android-emulator	as on arm64
riscv64:cargo-qa-tools	cargo-qa-tools	none of the three tools: none publishes a riscv64 binary, and the riscv64 lanes cross-build on amd64 (below)

Open gap (2026-09-25): the flatpak-runtimes and appimage-runtime arms are re-checked by flutter-sdk, another row’s fact, because _consumer_exempt_fact maps every row but appimagetool to it. A riscv64 image that gained Flatpak runtimes or an AppImage runtime would still read EXEMPT, which is the rot the next paragraph describes.

flutter-owner was an arm too and is gone. The same probe measures find /opt/flutter ! -uid 1001 as 0 on riscv64, which is the row passing, not a row to skip: an empty tree owned by the runtime uid satisfies the promise. Exempting it also hid the defect the row exists for — a root-written SDK would have read as a documented exception on that arch. Until 2026-09-05 the appimagetool arm was re-checked with FACT flutter-sdk, i.e. by another row’s fact, so a riscv64 appimagetool could never have been noticed; that is what _consumer_exempt_fact fixes.

The Android SDK is not exempt anywhere. The parity table asserts the android-sdk prefix on every arch. The row is skipped, not exempted, only when the image says it ships no SDK, in /opt/android/.android-payload-off. Two writers put it there:

• android-sdk.sh, when the android stage runs on a non-amd64 host. Upstream ships the NDK for linux-x86_64 only, so there is nothing to install.

• prune-host-payload.sh, for every arch whose HOST_PAYLOAD is off. That is arm64 and riscv64 by default since 2026-10-13 (why). Until then /opt/android-sdk/platform-tools was in all three shipped arches, as measured on 2026-09-04, though only amd64 can run it.

A payload-off image leaves ANDROID_HOME and ANDROID_SDK_ROOT empty. The row fails one that still sets either, because a consumer would find an empty SDK behind the variable. A missing fact is still NOFACT.

The Flatpak runtimes ship with the image

flatpak list --runtime in the shipped image returned zero refs. Every consumer run therefore re-downloaded the whole runtime set — roughly 1.9 GB across seven refs, the single largest download in their build.

Two of the seven (Platform and Sdk) were already installed here; the other five were left to each consumer run. All seven are installed now:

ref	pinned by
org.freedesktop.Platform//<ver>	FLATPAK_RUNTIME_VERSION
org.freedesktop.Sdk//<ver>	FLATPAK_RUNTIME_VERSION
org.freedesktop.Platform.Locale//<ver>	FLATPAK_RUNTIME_VERSION
org.freedesktop.Sdk.Locale//<ver>	FLATPAK_RUNTIME_VERSION
org.freedesktop.Platform.GL.default//<ver>	FLATPAK_RUNTIME_VERSION
org.freedesktop.Platform.GL.default//<ver>-extra	FLATPAK_RUNTIME_VERSION
org.freedesktop.Platform.codecs-extra//<ver>-extra	FLATPAK_RUNTIME_VERSION

GL.default appears twice on purpose: the base branch and its extra sibling are two separate refs, and a build that resolves one still fetches the other.

The runtime is 26.08 since 2026-10-07. freedesktop-sdk 25.08 changed two names, both read from flathub’s own listing and Platform//26.08’s metadata that day: the extra branches are spelled <ver>-extra (24.08 said 24.08extra), and the openh264 extension is gone. Platform//26.08 declares codecs-extra//<ver>-extra instead, which ships FFmpeg, x264 (with its GStreamer plugin), x265 and libde265 and no openh264, so FLATPAK_OPENH264_VERSION went with it. A consumer flatpak that loaded the runtime’s openh264 must bring its own H.264 codec or use x264. flatpak install Platform//26.08 pulls GL.default (both branches), Locale and codecs-extra as related refs; the seven-ref list names them so that a failure says which one.

INSTALL_FLATPAK_RUNTIMES defaults to true since 2026-09-05. Flathub builds these for x86_64 and aarch64 only, so on any other architecture the install is a guaranteed 404 rather than a transient failure — the installer skips those arches outright instead of retrying, and riscv64 images ship without Flatpak runtimes by construction.

The web lane toolchain

Measured in a consumer run, the Flutter web lane rebuilt its own tools on every invocation: flutter_rust_bridge_codegen build-web shells out to wasm-pack, and both were a from-source cargo install — 258 crates for wasm-pack, 174 for flutter_rust_bridge_codegen — followed by a nightly rustup auto-install through a path rustup itself calls deprecated.

The image now installs all three ahead of time:

what	pin	why it must be here
nightly toolchain + rust-src + wasm32-unknown-unknown	RUST_NIGHTLY_TOOLCHAIN (nightly-YYYY-MM-DD)	flutter_rust_bridge_codegen build-web runs wasm-pack with RUSTUP_TOOLCHAIN=nightly and -Z build-std. A dated toolchain is immutable, so rustup toolchain install <pin> is a genuine no-op on a warm image; the floating channel is UPDATED by that command, and the update renames files out of the read-only image layer (Invalid cross-device link). Consumers that still name the channel get it auto-installed at runtime into the writable RUSTUP_HOME — works, but pays the download per run; FRB’s --wasm-pack-rustup-toolchain names the pin instead.
wasm-pack	WASM_PACK_VERSION	258 crates per consumer run
flutter_rust_bridge_codegen	FLUTTER_RUST_BRIDGE_VERSION	174 crates per consumer run

Both crate versions live in 01-core/versions.env. Where upstream publishes a release binary for the arch, the image takes that instead of compiling: the same two cargo installs cost 87 s / 113 s on amd64 but 768 s / ~1170 s on arm64 and 1813 s / 3500 s on riscv64, which is the shape of QEMU user-mode emulation rather than a defect. install_web_lane_prebuilt downloads the x86_64- or aarch64-unknown-linux-musl asset and verifies it against a per-arch *_SHA256 pin in versions.env, the way sccache and binaryen are already fetched. Anything else — riscv64, which upstream publishes no asset for, a missing pin, a failed or mismatching download, a tarball without the binary in it — falls back to the from-source leg (below), so a consumer still gets the pinned --locked build rather than whatever the index resolves to that day, and nothing unverified is ever installed. The four hashes bump with the two versions.

That leaves one open question rather than an assumption: whether a riscv64 web lane exists at all. The tools are installed uniformly because an arch-conditional image is harder to reason about than a slower one, and because “we assumed nobody uses it” is how the Android layer ended up built for the wrong ABI.

install_web_lane_toolchain is non-fatal for availability — a missing nightly pin, an unpinned version and a failed cargo install each WARN and continue. The trade is deliberate: a consumer that has to build its own tools is slow, a consumer that cannot build the image at all is worse. What it does fail on — a bad knob, or a binary that claims to be good and is not — is in the next section.

Building the web-lane tools from source

riscv64 has no upstream binary, so its package stage used to compile both tools under QEMU on every chain: 960 s + 1,665 s of a 3,384 s RUN (riscv64 layer file times, 2026-09-22). That compile now has two replacements, and the old build stays one switch away, verbatim. Owner decision 2026-09-23: both options, cross the default.

Path	Where it compiles	When
cross	Dockerfile.android, stage web-lane-tools: cargo install --target <triple> under the hub’s own setup_linux_cross_env, before final swaps the amd64-hosted cross GCC out	the target is in WEB_LANE_TOOLS_CROSS_ARCHES and is not the build platform’s own arch
native	the package stage, with rv64gc Rust: cargo install --locked <tool> --version <pin> --root <scratch> under QEMU on the amd64 host, natively on a riscv64 or arm64 build host	WEB_LANE_TOOLS_SOURCE=native, or when auto finds no usable cross artifact
cache	the package stage’s cachemount web-lane-tools-bin-<arch>	a native build whose exact key was built before
legacy	the package stage, exactly as before 2026-09-23: cargo install --locked <tool> --version <pin> into CARGO_HOME	WEB_LANE_TOOLS_SOURCE=legacy only

native is not the old build. It compiles where the old build did, with the same rv64gc Rust, but four things differ: it forces vendored static bzip2, xz and zstd (the old build linked whatever pkg-config found in the stage); it installs a bare binary copied out of a scratch --root, so CARGO_HOME has no .crates.toml / .crates2.json entry for it; a binary that fails the gate stops the build (the old build installed anything cargo produced); and it reads the cache first, so it may not compile at all (WEB_LANE_TOOLS_CACHE=refresh or off forces a compile). legacy is the old build, verbatim: cargo installs into CARGO_HOME itself (and records it there), in the stage’s own environment, with no gate, no cache, no provenance line, and a failed cargo install only WARNs. It ignores WEB_LANE_TOOLS_CACHE and never reads the cross artifact.

amd64 and arm64 are untouched: they install upstream’s sha-pinned musl binary first and reach this code only if that download fails.

Picking a path. Export WEB_LANE_TOOLS_SOURCE before build-cross-chain.sh or the runtime helpers:

• auto (default) takes the cross artifact when its manifest says status=ok, it was built for this exact key, and it passes the gate. Anything else builds natively, with a WARN that ends the cross fast path was not taken.

• cross requires the artifact: absent, failed or another key is an ERROR. A skipped manifest is not, because the producer was never asked to build that arch; the package builds natively.

• native never reads the artifact. It still reads the cache, so add WEB_LANE_TOOLS_CACHE=refresh for a guaranteed fresh compile.

• legacy is the build the package stage ran before 2026-09-23, unchanged (above). Use it to rule the new paths out, or when a consumer needs cargo’s install record.

WEB_LANE_TOOLS_CACHE is on (default), refresh (never read an entry, store the new build) or off (neither). RUNTIME_NO_CACHE=1 does not empty a cachemount; refresh is its equivalent here. WEB_LANE_TOOLS_CROSS_ARCHES (default riscv64, a comma list, or none) is the arches android cross-builds for. A build host whose own arch is the target — the riscv64 X100, an arm64 Jetson on CROSS_BUILD_PLATFORM=linux/arm64 — always gets skipped: native-build-platform, so auto builds natively there with nothing set.

All three are Dockerfile ARGs that lib-orchestrator.sh forwards only when they are set, so leaving them unset moves no cache key. They are deliberately not in versions.env: a line there re-keys the whole chain from base.

The key. Eight fields: schema, tool, version, target triple, rustc release, RUSTFLAGS, the C environment and --locked. Who built it (built_by=cross:amd64, native:riscv64) is recorded, never keyed. The cross key carries cross-env.sh’s RVV flags (-C target-feature=+v,+zvl128b; a test pins the copy); the native key carries none, as the old build never set any. Both force vendored static bzip2, xz and zstd, because under PKG_CONFIG_ALLOW_CROSS those -sys crates link the build host’s library (failure-modes.md). legacy has no key: it caches nothing.

The gate (wlt_assert_binary) reads the staged bytes before anything is installed: ELF64, the arch’s machine, lp64d on riscv64, glibc’s own loader as PT_INTERP, NEEDED only from libc’s family, no GLIBC_ version above the image’s glibc, and --version printing exactly <tool> <pin>. No readelf is a refusal, not a skip. Artifact and cache entries must also match their recorded sha256.

Fatal or not. Availability still only WARNs: a failed cargo install, a producer that could not build (it records status=failed and android stays green), an unusable artifact under auto, no rustc -V release under auto or native. These fail the package stage: a bad knob value, checked before anything else, so a typo stops amd64 and arm64 too; a binary that claims to be good and is not — a status=ok artifact failing its sha or its gate, in auto and cross alike, or a native build cargo reported as a success that fails the gate; an image whose glibc getconf cannot report, or an install into CARGO_HOME/bin that fails; and cross with no usable artifact, including no rustc -V release to key an expected one by. A cache entry that fails its checks is deleted with a WARN and rebuilt. legacy fails on nothing but a bad knob.

Provenance. Each cross, cache or native install appends tool= version= source=cross|cache|native sha256= key= to /usr/local/share/web-lane-tools/provenance. All three install a bare binary with no .crates.toml, which is what amd64 and arm64 already ship from upstream: a consumer’s own cargo install <tool> then stops on binary already exists unless it passes --force. legacy writes no provenance line, and cargo records its install, as before.

Not covered. Behaviour beyond --version: no automated riscv64 build-web consumer exists. A binary and manifest forged consistently by someone who can write the build host’s cachemounts, the same trust boundary as the cargo, uv and apt mounts. The Jetson → riscv64 cross leg is untested; it records failed and falls back. The cross build ran in a probe (about 73 s at 32 vCPU, 2026-09-22), never yet in a chain.

The AppImage runtime ships with the tool

Every AppImage begins with a small ELF runtime that appimagetool prepends to the payload. When that runtime is not already on disk, appimagetool fetches it from GitHub on each run — so a consumer’s packaging step depends on GitHub being up at build time, and fails offline.

The runtime is staged at image-build time instead, and it is not downloaded. Upstream publishes it only under the moving continuous tag, which is the exact mutable-asset trap that already broke appimagetool itself once (a continuous re-upload changed the bytes under a pinned SHA256, and download_verified_file reported a tamper-shaped “checksum mismatch” that was only upstream drift). Since every AppImage starts with the runtime, and appimagetool is itself an AppImage pinned to an immutable versioned tag with a recorded SHA256, the runtime is taken out of the tool’s own bytes, up to where its squashfs payload starts. It is therefore pinned transitively and arch-correct by construction — no second download, no second pin to keep in step.

The offset is found by reading the file for the squashfs superblock, never by running the tool: QEMU user-mode cannot self-mount a foreign-arch AppImage, and appimagetool --appimage-offset gave Exec format error on aarch64 (2026-09-06).

It is written to two places, as runtime-<uname -m>:

path	why
/etc/skel/.local/share/appimagekit/	the runtime user is created later, and inherits /etc/skel
${HOME}/.local/share/appimagekit/	the build user that runs the packaging step now

ensure_appimagetool_runtime is a no-op, not a failure, when appimagetool is absent (riscv64 has no upstream build) or when no squashfs superblock is found in it: a missing runtime costs a consumer one download, and is never worth failing a toolchain stage over.

What changes with the next :latest

These are in source and not yet in a published image. Drop the workaround once your lane runs on the next :latest.

What	Before	After
A window on arm64/riscv64 Vulkan (CON41)	the loader lists no VK_KHR_{xcb,xlib,wayland}_surface, so a windowed test aborts and only amd64 renders under xvfb-run	all three arches list them; the image smoke fails an arch that does not (why)
clang-tidy on a compile database naming /usr/bin/clang++ (CON39)	selects the distro GCC 16; BeschleunigerBallett passes --extra-arg=--gcc-toolchain=${GCC_PREFIX}	/usr/bin carries the cfg pair too, so it selects ${GCC_PREFIX} (linux-cross-builds.md § Clang cross wrappers)
An atheris source build (CON38)	Failed to find libFuzzer: its find_libfuzzer.sh probes only lib/linux/libclang_rt.fuzzer_no_main-<arch>.a	the fuzzer, asan and ubsan archives are linked under that name as well; riscv64 still needs LIBFUZZER_LIB, since atheris’ probe knows no riscv64
lavapipe on arm64/riscv64 (CON44)	4-lane subgroups: a draw that builds an acceleration structure SEGVs; BeschleunigerBallett exports LP_NATIVE_VECTOR_WIDTH=256 itself	the image sets LP_NATIVE_VECTOR_WIDTH=256 on every arch, and the smoke fails a lavapipe whose subgroupSize is not 8 (why)
A C host that loads GStreamer or libcamera first	libunwind.so.8 from libgstreamer-1.0/libcamera-base turns a C++ exception through std::call_once into a SIGSEGV	both build without libunwind (why)
gst-inspect-1.0 -b on amd64 (CON47)	lists libgstvalidatessim.so	lists nothing; the SSIM plugin still works under gst-validate-1.0 (why)
A browser for flutter test --platform chrome (CON50)	none; flutter doctor reports “Cannot find Chrome executable”	Chrome for Testing and its chromedriver on amd64 and arm64, CHROME_EXECUTABLE set (how)
An Android device on amd64 (CON50)	none; Android tests ran on the x64 VM only	the emulator and an x86_64 system image with ARM translation, booted by android-avd.sh on /dev/kvm (how)
The Android SDK on arm64 and riscv64 (CON96)	/opt/android-sdk, /opt/android and the builder’s x86_64 rustup ship in both, 3.41 GB of layers no arm64 or riscv64 host can run; ANDROID_HOME points at x86_64 tools	both trees empty, with /opt/android/.android-payload-off; ANDROID_HOME, ANDROID_SDK_ROOT and the six Android roots empty, nothing Android on PATH. Build Android from the amd64 image, as before. HOST_PAYLOAD_ARCHES=amd64,arm64,riscv64 builds the old shape (why)
VULKAN_SDK and the variables setup-env.sh derives from it, after the entrypoint (CON48)	/opt/vulkan/<version>/<arch>; a riscv64 lane under QEMU in the amd64 image overrides VULKAN_SDK and VK_ADD_LAYER_PATH (riscv64_cross_env)	/opt/vulkan/active on every arch; the boot smoke fails an arch dir (why)

The Android SDK roots are advertised

Dockerfile.android advertises where each Android payload lives; Dockerfile.package COPYs the payload those names point at but, until 2026-09-05, never re-declared the names. A consumer that found /opt/android in the runtime image therefore still had no way to be told where anything inside it was, and every Android lane hardcoded the paths or re-downloaded the SDKs. The runtime image now re-declares all six. They are empty on arm64 and riscv64, which ship no Android payload (why):

variable	value
GSTREAMER_ROOT_ANDROID	/opt/android/gstreamer
ONNXRUNTIME_ROOT_ANDROID	/opt/android/onnxruntime
LITERT_ROOT_ANDROID	/opt/android/litert
OPENCV_ROOT_ANDROID	/opt/android/opencv
IREE_ROOT_ANDROID	/opt/android/iree
OPENCV_ANDROID_JNI_DIR	/opt/android/opencv/sdk/native/jni

Two shapes here are deliberate and a tidy-up would break both. Each name gets its own ENV instruction, because the env-knob owner scan reads only the first name of an instruction — collapsing them into one backslash-continued ENV, the way Dockerfile.android writes them, would leave five of the six with no recorded owner. And the OpenCV key is OPENCV_ROOT_ANDROID, never a bare OpenCV_DIR: that is the name find_package(OpenCV) reads, so pointing it at the Android SDK would hijack every Linux OpenCV consumer in the same image.

These are paths, not versions, so they are outside the advertised-version-key gate (verify_advertised_keys.py); what holds them is the runtime smoke’s path checks.

Two things worth knowing before you configure a lane

• :latest is a proper multi-arch index (it was called :latest-cross until 2026-09-22; that name is retired). An arm64 runner gets arm64 binaries; there is no longer any reason to pin -amd64 and no rustc: 1: ELF: not found to work around.

• The image ships Flutter at /opt/flutter, the default of the hub’s flutter_lane_prepare_env (05-frameworks/flutter/lane-prologue.sh), which installs nothing: a lane that points it at /workspace/flutter stops with no Flutter SDK at /workspace/flutter, and a lane with its own installer re-downloads the SDK every run for nothing. sccache and appimagetool are on PATH as well.

What changes with the image after CON11

In source since 2026-09-26; shipped in :latest on 2026-09-29 (index sha256:696642b2…, BACKLOG CON37). Each item retires a workaround a consumer carries today; drop it once your lane runs on that image.

What	Before	After
A bare clang/clang++	selects the distro GCC 16, so linking a library the image’s GCC built fails (std::format symbols, GLIBCXX_3.4.36)	<native-triple>-clang{,++}.cfg beside the compiler select ${GCC_PREFIX}; a --target build loads neither
clang-tidy, llvm-profdata, llvm-cov, llvm-symbolizer, ld.lld, … on PATH	LLVM 21, which cannot read what LLVM 23 wrote	clang’s own, linked into /usr/local/bin. clang-format and llvm-config stayed 21 until CON71 moved every LLVM tool to the pin (§ Every LLVM tool is the pinned release)
-fsanitize=fuzzer with the image’s clang	no libFuzzer runtime	built with compiler-rt (without a private libc++)
VIRTUAL_ENV, UV_PYTHON	point at the root-owned /opt/venv, over an activated venv	empty in the ENV and unset by the entrypoint; /opt/venv/bin stays first on PATH
A Vulkan device	none (nvidia_icd.json only)	lavapipe, a CPU device (mesa-vulkan-drivers)
perf, jq, Xvfb, gperftools’ libprofiler	absent	installed (linux-perf carries perf on 26.04)
A caller’s LD_LIBRARY_PATH/GST_PLUGIN_PATH	the entrypoint put /opt/libcamera ahead of it, and a host libstdc++ in it shadowed GCC’s	the image’s libcamera goes after it and ${GCC_PREFIX}’s runtime before it, so a Raspberry Pi host-libcamera run needs no --entrypoint
The distro GStreamer 1.28 runtime	installed beside /opt/gstreamer	dropped on amd64; kept on arm64/riscv64, where libgstgtk4.so needs Ubuntu’s GTK 4
Android API 37	missing although pinned (a stale SDK cache)	platforms/android-37.0 and build-tools/37.0.0
TVM on amd64	import tvm works, but its compiler library needs an LLVM the image lacks, so TVM loads runtime-only and compiles nothing	linked against /usr/local/llvm-target’s LLVM (X86 codegen, as arm64’s is AArch64); the image smoke compiles one PrimFunc

The Android SDK roots are advertised

Dockerfile.android sets GSTREAMER_ROOT_ANDROID, ONNXRUNTIME_ROOT_ANDROID, LITERT_ROOT_ANDROID, OPENCV_ROOT_ANDROID and IREE_ROOT_ANDROID, but that is the build stage. Dockerfile.package COPYs the payload those names point at and used to stop there, so a consumer of the shipped image found /opt/android populated and no name telling it what was where. The reported symptom:

CMake Error at CMakeLists.txt:18 (message):
  GSTREAMER_ROOT_ANDROID must be set

All five are re-declared in the runtime image, with OPENCV_ANDROID_JNI_DIR beside them, one ENV instruction each — the env-knob owner scan reads the first name of an instruction only, so a single multi-name ENV would leave five of the six unowned.

There is deliberately no bare OpenCV_DIR. That is the name find_package(OpenCV) resolves, and pointing it at the Android SDK would hijack every Linux OpenCV consumer in the same image. OPENCV_ANDROID_JNI_DIR names the Android tree instead, to be passed explicitly:

cmake -DOpenCV_DIR="${OPENCV_ANDROID_JNI_DIR}" ...

Every LLVM tool is the pinned release

Owner decision 2026-10-06 (BACKLOG CON71): every LLVM tool a lane can name is the pinned LLVM, LLVM_RELEASE on Linux and LLVM_WINDOWS_VERSION on Windows (23.1.3 today). That includes clang-format and llvm-config, which stayed LLVM 21 on Linux until then (CON15).

Linux. Ubuntu’s packages put about a hundred unversioned LLVM names in /usr/bin, all LLVM 21. The package stage’s wire_pinned_llvm_tools diverts each of them with dpkg-divert:

• a name the pinned LLVM has points into /usr/local/llvm-target/bin;

• a name it lacks leaves PATH. The originals sit in /usr/lib/distro-llvm-names.

• bugpoint is the only permanent gap: LLVM 23 removed it.

• The -NN names (clang-tidy-21, scan-build-21, …) still reach LLVM 21. They ask for it by name. The distro rustc’s rust-* links are left alone too.

An apt-get install of a distro LLVM package later does not undo the diversion. The pinned build carries lldb (without Python, Lua, libedit or curses scripting) and the LLVM utilities (FileCheck, not, yaml2obj, llvm-tblgen), so no name falls back to 21. validate-compilers.sh smoke checks two things and fails the image on either:

• 26 tools report LLVM_RELEASE, lldb and FileCheck included;

• no unversioned /usr/bin name resolves into a distro LLVM.

Windows. C:\llvm-patched\bin is first on PATH once the entrypoint has run, ahead of VsDevCmd’s MSVC directory (its llvm-symbolizer is a 23.0.0git fork) and C:\runtime\iree\bin (IREE’s own clang, llvm-link and FileCheck, 23.0.0git). The patched LLVM installs the utilities, so FileCheck is the pinned one too. clangd and lldb come from scoop’s official 23.1.3 release. Test-Toolchain.ps1 checks the base stage’s tools against LLVM_WINDOWS_VERSION, and Test-Container.ps1 checks 21 tools in the finished image against clang-cl’s release.

What it changes for a consumer:

• A Linux format gate now grades with clang-format 23, as Windows always did. Files formatted with 21 can come back changed. Re-run the formatter once and commit the result.

• llvm-config names an LLVM built for the image’s own arch only (CON35). A build that wants another backend from it needs its own LLVM.

• scan-build-21 is still LLVM 21’s analyzer. Call scan-build for the pinned one.

What the image stages so a run does not

Seven of the contract rows are not about permissions at all. They ask whether a thing is present, because the alternative is that every consumer run fetches or rebuilds it. Measured in one consumer’s build on 2026-09-05, before the fix (the cargo-qa-tools and free-threaded-python rows since 2026-10-06, the last two since 2026-10-09):

Row	Absent means
flatpak-runtimes	flatpak list --runtime returns 0 refs; seven org.freedesktop refs, ~1.9 GB, re-downloaded per run per arch
appimage-runtime	appimagetool refetches runtime-<arch> from GitHub, so packaging hangs on GitHub being reachable
web-lane-tools	wasm-pack (258 crates) and flutter_rust_bridge_codegen (174) are cargo installed from source in every run
cargo-qa-tools	cargo-audit, cargo-deny and cargo-tarpaulin are cargo installed from crates.io before every OxidANT security and coverage step
free-threaded-python	every 3.14t leg has uv download a free-threaded CPython first, its patch version unpinned
lint-tools	shellcheck and hadolint are not on PATH, so OrchestrANT’s coding bench grades every bash and Dockerfile answer with [shellcheck SKIPPED: not on PATH]
uv-cache-seed	the riscv64 Python lane’s uv sync of the test extra builds numpy, matplotlib, contourpy, pillow, line-profiler, psutil and pyyaml under QEMU: 108 min of a 6 h job

The first three share one verdict function; cargo-qa-tools and free-threaded-python have their own, because they compare versions (and free-threaded-python also where the interpreter came from). The cost of each is written down once, in _consumer_contract_symptom, which is also what the failure message prints.

The Flatpak runtimes ship with the image

install_flatpak_runtime had existed for months behind INSTALL_FLATPAK_RUNTIMES, which defaulted to false — so the capability was there and switched off, and it covered two of the seven refs. It now defaults to true and installs all seven, Platform.GL.default twice because the base branch and its extra sibling are separate refs.

Flathub builds these for x86_64 and aarch64 only. On any other arch the install is a guaranteed 404 rather than a flake, so the function returns early and the row is exempt on riscv64.

The AppImage runtime ships with the tool

appimagetool embeds a type-2 runtime into every AppImage it builds, and fetches it at build time if it is not on disk. Upstream publishes that runtime only under the moving continuous tag — the exact mutable-asset trap that made appimagetool itself move to a pinned version tag (TS1, 2026-08-15), so it cannot be SHA-pinned.

It is not downloaded. Every AppImage begins with that runtime, and appimagetool is already SHA-pinned, so ensure_appimagetool_runtime finds the squashfs superblock in the tool’s file and copies the bytes before it out. Pinned transitively, correct by construction for whatever arch the tool is. It lands in /etc/skel as well as root’s home, so the runtime user created later inherits it.

The web-lane toolchain

flutter_rust_bridge_codegen build-web shells out to wasm-pack ... -Z build-std, which resolves the nightly channel unless FRB’s --wasm-pack-rustup-toolchain names the pin, so rustup auto-installed one per run through a path it calls deprecated. The package stage installs the dated pin RUST_NIGHTLY_TOOLCHAIN, not the channel (why), with rust-src and wasm32-unknown-unknown, plus both binaries at pinned versions, on a cargo registry cachemount.

WASM_PACK_VERSION and FLUTTER_RUST_BRIDGE_VERSION are advertised as image ENV and compared against what the binaries report, which is also the proof that they are installed. A slow consumer beats an image that cannot be built, so only a defect fails the stage; riscv64’s from-source leg is above.

The cargo QA tools

The package stage installs cargo-audit, cargo-deny and cargo-tarpaulin at CARGO_AUDIT_VERSION, CARGO_DENY_VERSION and CARGO_TARPAULIN_VERSION on amd64 and arm64 (CON65). Each is its upstream release binary, verified against a CARGO_*_LINUX_{X86_64,AARCH64}_SHA256 pin; cargo-audit’s aarch64 build is a glibc one, the rest are musl. A missing pin or a failed download stops the stage: there is no source fallback, which would cost hundreds of crates per tool under QEMU.

• The pins are read from versions.env inside the stage, not forwarded as build-args, so they are not advertised ENV. The cargo-qa-tools contract row compares what each binary reports with the pins instead.

• riscv64 ships none of them. None publishes a riscv64 binary, and no riscv64 lane runs them: those lanes cross-build on the amd64 image. The row is exempt there and fails as STALE if one ever appears.

• cargo_security_checks.sh and cargo_coverage.sh call cargo_install_pinned (_cargo_wrapper.sh). It uses the binary already on PATH when it reports the pin, and cargo installs it otherwise. A plain cargo install would fail on these: binary already exists, since cargo did not install them. So a consumer’s hub pin must include this before it builds on an image that ships them.

shellcheck and hadolint

The torch stage puts both on PATH in /usr/local/bin on every arch (CON83), at the SHELLCHECK_VERSION and HADOLINT_VERSION pins in tool-pins.env, the same pins the hub’s lint gates bootstrap. install-lint-tools.sh takes each release binary through download_verified_install against its per-arch SHA256, and fails the stage when a tool does not report its pin.

• shellcheck publishes a riscv64 build, pinned as SHELLCHECK_LINUX_RISCV64_SHA256. lint-shell.sh bootstraps the same asset on a riscv64 host.

• hadolint publishes none for riscv64. The hadolint-build stage of Dockerfile.torch builds it from the release tag’s source (HADOLINT_SOURCE_SHA256) with Ubuntu’s GHC 9.10.3, the compiler its cabal file is tested with. The tag’s cabal.project carries upstream’s allow-newer: time; the Hackage release alone does not resolve. HADOLINT_HACKAGE_INDEX_STATE pins every dependency to the index of the bump day. The stage starts from the pinned Ubuntu, not the package image, so an app or chain change never re-keys it; on amd64 and arm64 it is an empty /out. build-hadolint.sh fails a binary that does not report the pin or does not flag a Dockerfile it must flag, and caches it in the cabal cache mount keyed by those three pins, so a chain rebuilds it only when one of them moves. The build is long: measured 2026-10-09 under QEMU on the 32-core host, 18221 s for about 105 packages, after 469 s of apt and 138 s of cabal update. BuildKit runs it in parallel with the torch stage; the lint-tool install in the final stage waits for it. The binary links libgmp, libffi and libnuma, which the riscv64 image carries.

• The pins live in tool-pins.env, not versions.env: only the torch stage reads them, so a lint-tool bump rebuilds the torch stage, never the compiler.

The lint-tools contract row compares what both binaries report with the pins, on every arch.

The riscv64 uv cache seed

PyPI has no cp314 riscv64 wheel for numpy, matplotlib, contourpy, pillow, line-profiler, psutil or pyyaml. OrchestrANT’s riscv64 lane therefore built all of them under QEMU in every run’s uv sync --extra test: 108 min of a 6 h job, numpy alone 107 min (2026-10-07). The image ships the result instead (CON83).

• The contract is a uv cache, not a wheel store. PYTHON_UV_CACHE_SEED (/opt/uv-cache-seed) is a uv cache directory holding only the wheels uv built from source for the app lock’s test extra. A --find-links store does not work here: uv sync --locked installs what the lock names, by URL and hash, and a local wheel is neither. uv looks a built sdist up in its cache by that same source, so a cache entry is what a locked sync can use. uv sync --no-build cannot use one either: it refuses every sdist, cached or not (measured 2026-10-09).

• Consumers copy it, they do not point at it. uv writes its cache, and the seed is root-owned. ci_tests.sh and ci-wheel-smoke.sh call uv_cache_seed_restore (02-toolchain/python/ci-common.sh), which copies the seed into uv cache dir when its record says seeded yes for this arch; anything else is a log line, never an error. Another consumer runs the same copy: cp -R "$PYTHON_UV_CACHE_SEED/." "$(uv cache dir)/".

• How it is built. 06-packaging/uv-cache-seed.sh build, in the torch stage’s last layers, fetches the app at APP_REF (the shipped tree has no .git, and its riscv64 pyproject.toml is edited), and runs ci_tests’ own sync, uv sync --locked --dev --extra test, without the project. The uv-seed-<arch> cache mount keeps the builds between chains. The seed is a copy of that cache after uv cache prune --ci, which drops every download and unpacked sdist and keeps the built wheels, and after removing every version the sync did not install.

• How it is proved. At build, a fresh copy syncs the same lock into a fresh venv, and the stage fails when uv prints a Built line or adds a wheel under sdists-v*/. seed-record.txt names the arch, the app commit, the extras, uv and Python, every built wheel, and the proof. uv-cache-seed.sh verify checks that record and that every listed wheel is in the cache; the uv-cache-seed contract row runs it. On riscv64 it wants seeded for riscv64, elsewhere a not seeded record.

• What it costs. Measured 2026-10-09 in :latest’s riscv64 child under QEMU on the 32-core build host, for OrchestrANT 3bdf80a7: 3172 s cold, building 8 wheels (the 7 above and the pure-Python sdist antlr4-python3-runtime), and 89 s warm, sync, copy and proof included. The seed is 21 MB; the work cache 671 MB. A chain pays the cold build once, then only for a package whose locked version moved.

• What it does not cover. Only the test extra of the app the image builds (OrchestrANT). Another consumer’s lock hits the seed only where it resolves the same versions from PyPI. A uv bump re-keys the cache layout, and the next chain rebuilds the seed with it; a consumer on an older image keeps a seed its newer uv may not read.

The free-threaded Python

The image ships the free-threaded twin of PYTHON_VERSION (3.14.8t today) on every arch, riscv64 included (CON66), so a 3.14t leg downloads no interpreter. Since 2026-10-11 the hub’s Python legs refuse to: a missing one fails the leg (python-ci.md § Free-threaded and GIL legs).

• It is built from source in the toolchain stage, from the same SHA256-pinned tarball as the GIL build (owner decision 2026-10-07). Until then the package stage installed uv’s python-build-standalone download. build_python.sh builds each of its PYTHON_VARIANTS (gil,freethreaded) in turn. The second one is --disable-gil, with PGO and LTO on the build arch and LTO alone for each cross arch, where PGO would have to run the foreign interpreter. The cost lands in the toolchain stage, and every stage after it rebuilds when the pass changes. Measured on a 32-core host on 2026-10-07: 6.5 minutes natively, 4.7 of them the PGO profile run, and under 2 minutes per cross arch. The shipped trees are 138 MB (amd64), 120 MB (arm64) and 233 MB (riscv64).

• It lives in /opt/python-freethreaded, with /usr/local/bin/python3.14t linked to it, so it never shadows the GIL /usr/local: python3, pip3 and python3.pc stay the GIL build’s. A 3.14t request finds it on PATH. A plain 3.14 request never takes it (python-ci.md § Free-threaded and GIL legs). The binary finds libpython3.14t.so through its own RUNPATH, set with LDFLAGS_NODIST so that it stays out of sysconfig: a cp3XYt wheel built against it inherits none.

• Every artifact stages one tree per arch, at /opt/python-cross-ft/<arch>/opt/python-freethreaded, the build arch’s included and in either build mode, so Dockerfile.package takes the target arch’s with one COPY.

  • A cross arch’s tree is a real make altinstall, with its bytecode, python3.14t-config and config dir. The GIL cross trees are copies of the build tree.

  • It keeps _ctypes, which the GIL cross trees build without (ac_cv_header_ffi_h=no), because it ships as the runtime interpreter.

  • No tree carries the static libpython3.14t.a (--without-static-libpython): its fat LTO objects were 155 MB on amd64 and 435 MB on riscv64, and nothing links them.

  • No staged tree carries the test suite. The cross trees are built --disable-test-modules, and the build arch’s stage drops lib/python3.14t/test to match.

  • pip is in the build arch’s tree only; the cross builds run --without-ensurepip.

• The build stages keep it too, for the cp3XYt wheel builds: toolchain, media and android carry the build arch’s interpreter at /opt/python-freethreaded and /usr/local/bin/python3.14t, and each CROSS_TARGETS arch’s tree under /opt/python-cross-ft/. The media stage builds its twins with it (§ The free-threaded wheels).

• Each stage checks it.

  • build_python.sh fails a cross tree without _ssl, _hashlib, _sqlite3, zlib, _bz2, _lzma or _ctypes.

  • smoke-toolchain.sh runs the twin and checks every staged arch’s tree.

  • The package stage stops when the COPY’d interpreter is missing, does not report PYTHON_VERSION with the GIL off, or cannot import ssl, sqlite3, ctypes, zlib, lzma and bz2.

  • The free-threaded-python contract row checks the same on the shipped image, and that sysconfig’s prefix is /opt/python-freethreaded, which tells the source build from a python-build-standalone one.

The free-threaded wheels

Every wheel the media stage builds for a package whose own code declares free-threading support also ships as a proved free-threaded (cp3XYt) twin, in a store of its own: /opt/wheels-free-threaded (owner request 2026-10-07; no version in the name since 2026-10-12, so a Python upgrade renames nothing). That holds for the cross-built arches too, arm64 and riscv64, since 2026-10-09 in source (CON75, CON79 1b). The GIL wheels are built exactly as before.

• Which packages. One table decides, linux/scripts/03-media/free-threaded-twins.txt, and both lanes read that file: ft_wheel_table here, Get-FreeThreadedTwinTable on Windows, so their verdicts cannot differ. Each row names the versions.env pin its evidence was read at, and test-free-threaded-wheels.sh fails when that pin moves, so a bump re-reads the evidence. The media RUNs that build or store a twin mount it beside free-threaded-wheels.sh, and a missing table stops them rather than reading every package as unclassified.

  Package	Verdict	Evidence, at the pinned tag
  onnxruntime (every flavour)	twin	PYBIND11_MODULE(..., py::mod_gil_not_used()), pybind11 v3.0.2
  av (PyAV)	twin	setup.py: "freethreading_compatible": True
  apache-tvm-ffi	twin	core.pyx: # cython: freethreading_compatible = True
  iree-base-runtime	twin	nanobind_add_module(... FREE_THREADED ...)
  iree-base-compiler	twin	MLIR’s nanobind_add_module(... FREE_THREADED ...)
  torch	twin while FT_TORCH_TWIN=1	torch/csrc/Module.cpp: PyUnstable_Module_SetGIL(module, Py_MOD_GIL_NOT_USED); the chain builds torch on riscv64 only
  numpy	twin while FT_TORCH_TWIN=1	multiarraymodule.c: {Py_mod_gil, Py_MOD_GIL_NOT_USED}; no chain build ships a numpy wheel, so the row only classifies one
  apache-tvm	none needed	wheel.py-api = "py3": its one py3 wheel installs on 3.14t
  torchvision	none needed	no CPython extension module of its own
  onnxruntime-genai, ai-edge-litert, hailort	GIL only	no free-threading marker
  libcamera’s pycamera, OpenCV’s cv2	GIL only	no marker, and they ship in the tree, not as wheels

• A knob row. twin:<KNOB> names a versions.env switch: the row is a twin only while that key is 1. FT_TORCH_TWIN=1 is the default, and Dockerfile.media takes it in the app-wheelhouse RUN, which builds the twin, and in the final RUN, whose store check expects it. Windows reads such a row as no twin.

• How a twin is built. Each build runs a second pass on a 3.14t venv whose build executors are pinned to the GIL build venv’s own versions, in the GIL pass’s tree, so only what sees Python rebuilds. Measured in :latest on 32 cores (2026-10-07):

  Package	Twin pass, cold compiler cache	Warm	What the twin pass rebuilds
  ONNX Runtime CPU (GIL pass: 760 s at -j11)	97 s	11-40 s	539 of 3206 objects, every target whose compile line carries the Python and NumPy include dirs; of the wheel’s four libraries only onnxruntime_pybind11_state.so differs
  ONNX Runtime CUDA, :latest-nvidia (GIL RUN: 12 672 s, 578 CUDA objects)	192 s	32 s	no CUDA object; the C++ targets that see Python, from the compiler cache
  ONNX Runtime MIGraphX, :latest-rocm (GIL RUN: 1311 s, 590 HIP objects)		57 s	no HIP object; 106 C++ objects
  PyAV	28 s	14 s	all 50 extensions, re-Cythonized because the fresh venv’s Cython includes are newer: all of PyAV sees Python
  apache-tvm-ffi	29 s		all 34 steps: scikit-build-core’s reconfigure rebuilds the small C++ core too
  IREE compiler and runtime (GIL pass: 55 s configure, 2334 s build at -j16)	25 s		58 of 5452 objects, the bindings alone (below)

  The GPU rows are chain runs, not the 32-core measurement: nvidia 20261010-125556 (first) and 20261010-202726 (warm), rocm 20261009-152508. Each GPU build hands the pass its own BUILD_ARGS and build dir, and the pass appends only Python_EXECUTABLE, as the last define. No GPU script passes a Python include dir, so FindPython re-resolves the 3.14t headers, NumPy include and SOABI in the warm cache (CMake 4.4.4, checked on a scratch project). A carried GIL Python_INCLUDE_DIR would survive that reconfigure, which is the riscv64 torch twin’s bug below. test-free-threaded-wheels.sh holds both GPU flavours to that. The IREE row is from a cold compiler cache, the full in-tree LLVM, and a settled GIL tree. The runtime-only tree, before the settle, took 130 s at -j4 and recompiled 464 of its 723 objects.

  • Why the GIL tree configures twice. IREE’s first configure leaves IREE_HAVE_LIBBACKTRACE unset and a second one sets it (linux-cross-builds.md § IREE). On an unsettled compiler tree the twin’s reconfigure would change 1272 compile lines, 757 of them compiler sources, and relink the 339 MB libIREECompiler.so. The GIL pass therefore configures twice (17 s more), and the twin’s reconfigure (11 s) then changes only what sees Python.

  • What the twin pass rebuilds. 27 IREE compiler binding objects, 20 runtime binding objects and 11 MLIR Python objects. It then relinks nanobind’s free-threaded libraries, libMLIRPythonSupport-mlir.so and the 15 modules.

  • What differs from the GIL wheels. The extension modules, libMLIRPythonSupport-mlir.so, and nanobind’s library (libnanobind-mlir.so in the GIL wheel, libnanobind-ft-mlir.so in the twin), plus WHEEL and RECORD. Everything else is byte-identical, including libIREECompiler.so and the ten runtime tools.

• How a cross twin is built. The amd64 build host’s own 3.14t (/opt/python-freethreaded) runs the build tooling, and ft_target_resolve reads the target’s tree, which the toolchain stages at /opt/python-cross-ft/<arch> (CON66). That gives the target’s headers, its libpython3.14t.so, its EXT_SUFFIX (.cpython-314t-riscv64-linux-gnu.so, read from the tree’s _sysconfigdata_t_*.py without running it) and the wheel platform (linux_riscv64). ft_target_env then points the build’s sysconfig at a private copy of that file, the way the GIL cross pass points it at the target’s GIL one:

  • PyAV compiles against the target headers with that suffix.

  • tvm-ffi’s scikit-build-core reruns in the cross pass’s build dir.

  • IREE reconfigures its runtime tree with the target 3.14t include dir and library.

  • ORT passes Python_INCLUDE_DIR to build.sh.

  • The riscv64 torch reruns its warm scikit-build tree with the target 3.14t on the command line, where the toolchain file’s cache entries cannot reach. Its CMAKE_ARGS are built afresh rather than appended to the GIL pass’s: that pass’s -DPYTHON_INCLUDE_DIR heads torch_python’s include path, so a twin that kept it compiled against the GIL pyconfig.h. That twin’s libtorch_python.so was byte-identical to the GIL wheel’s, and it segfaulted in dlopen under the target 3.14t (2026-10-10). The fixed pass takes about 240 s warm, and its libtorch_python.so imports _Py_DecRefShared and PyUnstable_Module_SetGIL.

  A cross build that is missing the target tree fails its twin, rather than skipping it.

• How a twin is proved. Before it is stored, ft_soabi_gate wants a cp3XY-cp3XYt name whose platform tag names the suffix’s machine, and the free-threaded SOABI on every version-tagged module. A cross build gates against the target’s suffix. Natively, a fresh 3.14t venv takes the wheel alone, where free-threaded-wheel.py prove loads every compiled module and fails when the GIL comes back on. A cross twin is unpacked instead and proved by the target’s own 3.14t under qemu-user (ft_prove_wheel_on_target): -L names the root holding the target’s loader, and the target tree’s lib leads LD_LIBRARY_PATH. A bare .so counts as a module only when it exports PyInit_<name>, so ORT’s bundled libonnxruntime_providers_shared.so is not loaded as one. Before a module loads, its enclosing package is imported. A package whose body cannot run in a --no-deps venv is registered unrun instead, as an import in progress leaves it: torch._C’s init imports torch, whose body needs typing_extensions. numpy’s test modules need the real numpy, which is why the import is tried first. A proof never fails with an empty reason. faulthandler prints a segfault’s stack, the verdict names the exit status or signal, and FT_PROVE_TRACE names the module that was loading. A twin that fails any of it fails its RUN; inside TVM’s best-effort build that withdraws TVM, as any failure there does.

• How the stores stay apart. collect-artifacts.sh moves the twins into FT_WHEELS_DIR only, and the ORT twin gets its own manifest, ort-provenance-free-threaded.sha256. verify-wheels.sh is ABI-exact on both stores: a cp3XYt wheel or module in /opt/wheels fails it, and so does anything but cp3XY-cp3XYt in the twin store. The final RUN then repairs and verifies the twins with --free-threaded, and free-threaded-store.sh proves them again on the repaired bytes. A cross build proves them under qemu-user again. A build, native or cross, fails there when the store is not exactly one twin per GIL wheel the table expects a twin of, or when a GIL wheel has no row. It writes free-threaded-store.txt: the build mode, each twin, one want <family> line per family it expects, and one line per package without one, with its reason.

• What ships. Dockerfile.torch copies /opt/wheels-free-threaded into :latest, late, so a new twin re-keys no venv layer, and the export delivery seals it with /opt/wheels. Nothing installs a twin: an X.Yt venv takes them with uv pip install /opt/wheels-free-threaded/<wheel>. The image names the store in PYTHON_WHEELS_FREE_THREADED, as Windows’ does since CON91 (its older images set PYTHON_WHEELS_CP314T, which every reader still takes), and uv_reconcile_chain_ort gives an X.Yt venv’s ONNX Runtime from it (python-ci.md § Trap 3).

• How the shipped image proves it. The runtime smoke’s FT-STORE gate (06-packaging/check-free-threaded-wheels.sh) wants the store, its record and that variable on every arch. On every arch, native or cross-built, the store holds exactly the twin families of /opt/venv that the record wants, the installed ORT flavour’s own twin included. Each twin loads every compiled module on /usr/local/bin/python3.14t with the GIL off, under QEMU on a foreign arch. A /opt/venv package whose family the record does not want gets an OK line naming it: PyPI’s torch on amd64 and arm64, and PyPI’s IREE compiler on a cross arch, whose chain builds the IREE runtime only. The GPU images pass it with both ORT twins: :latest-nvidia (2026-10-11, index sha256:810f36b3) proves onnxruntime_gpu and onnxruntime_dnnl, :latest-rocm (2026-10-09, index sha256:a5d68fbe) onnxruntime_migraphx and onnxruntime_dnnl, and the ORT census grades each twin’s members against the flavour’s ort-provenance-free-threaded.sha256.

• Not covered yet. The cross twins are in source since 2026-10-09 and unproved until a chain publishes them. Proved so far without a chain: PyAV 19.0.1’s twin, cross-built against the published riscv64 image’s FFmpeg and 3.14t tree. All 50 modules loaded with the GIL off on riscv64 under QEMU, through FT-STORE’s own probe and verdict. A minimal C module went through ft_store_twin the same way, and its Py_mod_gil-less control was refused. Windows builds no twin of a knob row. The IREE compiler twin was proved locally in :latest amd64 on 2026-10-07, but not yet in a chain:

  • prove loaded its 14 compiled modules with the GIL off.

  • A 3.14t venv parsed MLIR in-process, compiled an add function to a VMFB and ran it on local-task from 8 threads.

  • iree-compile and iree-run-module ran from the twins.

The Windows image ships lavapipe

The Windows amd64 image and the arm64 bundle carry Mesa’s lavapipe — a CPU Vulkan device, the only software one that also runs the ray-tracing goldens — with the target arch’s Khronos loader and vulkaninfo.exe, all SHA256-pinned, in C:\runtime\lavapipe (CON50). The image sets LP_NATIVE_VECTOR_WIDTH=256, because Mesa 26.2’s BVH sort needs 8-lane subgroups and arm64’s native 128 bits give 4. The amd64 image registers the ICD in HKLM at build time — a container process is elevated, and the loader ignores VK_DRIVER_FILES there. The arm64 bundle cannot register it while it is packed, so the device needs the one-time HKLM registration the bundle README names; Test-Arm64Bundle.ps1 runs vulkaninfo --summary and fails a device without the llvmpipe device. :winamd64 carries the device since the 2026-10-04 republish (the smoke lists llvmpipe); the :winarm64 bundle is published and its device gate lists llvmpipe (12/12 on the Snapdragon X).

The ort crate links the chain ONNX Runtime

Both published images set the environment of the Rust ort / ort-sys crates, so a Rust build inside them links the chain ONNX Runtime and can never fetch pyke’s prebuilt one (G3 of the ONNX Runtime single-source rule).

Variable	Windows (windows/Dockerfile, final)	Linux (linux/Dockerfile.package, every variant)
ORT_LIB_LOCATION	$ONNX_ROOT\lib (holds onnxruntime.lib)	${ONNXRUNTIME_OUTPUT_DIR}/lib = /usr/local/lib/onnxruntime-cpu/lib
ORT_PREFER_DYNAMIC_LINK	1	1
ORT_SKIP_DOWNLOAD	1	1
ORT_DYLIB_PATH	$ONNX_ROOT\bin\onnxruntime.dll	${ONNXRUNTIME_OUTPUT_DIR}/lib/libonnxruntime.so

Why these four, read from ort-sys 2.0.0-rc.13 (the checksum OxidANT’s Cargo.lock pins) and ort 2.0.0-rc.13:

• build/main.rs:47-68 reads ORT_LIB_PATH, then ORT_LIB_LOCATION, before the download branch. With ORT_PREFER_DYNAMIC_LINK set to 1 or true the build script emits rustc-link-lib=onnxruntime from that directory; without it, ort-sys tries a static link the chain does not ship.

• The first SET one of CARGO_NET_OFFLINE, ORT_SKIP_DOWNLOAD and ORT_OFFLINE decides, and only exactly 1 or true skips the download. An empty value is set, so an empty CARGO_NET_OFFLINE re-arms the download, and an empty ORT_LIB_PATH still outranks ORT_LIB_LOCATION.

• ort’s load-dynamic opens ORT_DYLIB_PATH, otherwise a bare file name, which on Windows reaches System32\onnxruntime.dll (Windows ML 1.17) before PATH.

• ort’s default features include download-binaries.

What it means for a consumer.

• An ort-sys build in the image links the chain import library, so a shipped bundle must carry the chain DLL or .so beside the app.

• Keep ORT_LIB_PATH unset and never export a falsy CARGO_NET_OFFLINE.

• On the Linux GPU variants the variables name the CPU chain build, which is what ld.so resolves. A GPU consumer points both at onnxruntime-gpu/lib, which is chain-built too. On :winarm64 they name the aarch64 chain ORT.

• ORT_DYLIB_PATH is baked in, so inside the image a program that honours it loads the image’s copy, not one staged beside its exe. A bundle smoke run inside the image should unset it to exercise the staged copy.

How it is proven on the shipped image. Windows: Test-Container section 19, Get-OrtCrateEnvFinding, on every lane. Linux: the contract row ort-crate-env. Its probe reports ORT_LIB_PATH and CARGO_NET_OFFLINE as <unset> when they are not set, and the row passes only with ORT_LIB_PATH unset and CARGO_NET_OFFLINE unset, 1 or true; a missing line reads as set-but-empty and fails. Both compare values untrimmed, as ort-sys does. The static half is verify-critical-fixes.sh fix11. Not covered: Rust builds outside the images, and the live in-image cargo build -vv of a scratch ort-sys crate, which is still to run at the next image build.

The chain ONNX Runtime wheels

• Linux: /opt/onnxruntime-wheels, advertised as ORT_CHAIN_WHEEL_DIR, holds exactly the ORT and GenAI wheels /opt/venv was installed from. setup-torch-venv.sh stage_chain_ort_wheels proves the venv against it at build time. The census that proof runs ships at /opt/scripts/03-media/final/ort-venv-census.py; its command line and output lines are a contract that python_uv.sh and setup-torch-venv.sh read. The free-threaded twins of those flavours sit in /opt/wheels-free-threaded (PYTHON_WHEELS_FREE_THREADED).

• Windows: the same wheels sit in C:\runtime\wheels (PYTHON_WHEELS). The census ships at C:\temp\scripts\ort-venv-census.py, beside the module copy whose Sync-UvChainOnnxRuntime runs it; Build-TorchApp.ps1 embeds its own copy.

• Who gets them: a consumer on the hub’s Python CI has its venvs reconciled onto them inside our images (python-ci.md § Trap 3). Anyone else calls uv_reconcile_chain_ort <venv> or Sync-UvChainOnnxRuntime -VenvPath <venv>.

Prove your bundle ships the image’s ONNX Runtime

Run the ORT census (G6) inside the image you built in, after staging the runtime DLLs or .so files. The reference is that image’s chain ORT.

• Windows: Import-Module C:\temp\scripts\modules\WindowsOrtProvenance.Common.psm1, then if (-not (Test-OrtProvenanceTree -Root <bundle>)) { exit 1 }. -PassThru returns the census object.

• Linux: bash third_party/ANTfrastructure/linux/scripts/06-packaging/check-ort-provenance.sh <bundle>.

Tree mode models a client loader. A DLL’s imports resolve from the directory of the exe that loads it, then System32 (assumed to hold Windows ML’s onnxruntime.dll), then PATH, never from the DLL’s own directory; a .pyd also searches its own directory first. So ship onnxruntime.dll beside the exe, not beside a plugin. A bundle fails as STALE with an older chain DLL, FOREIGN with a NuGet, PyPI or pyke ORT (statically linked ones included), and UNRESOLVED when an importer has no app-local chain copy or a Linux .so has no $ORIGIN RUNPATH to one.

Browser tests run in Chrome for Testing

flutter test --platform chrome needs a browser, and Ubuntu 26.04 ships Chromium as a snap only. The image ships Google’s Chrome for Testing instead (CON50), the build Playwright also downloads, from Google’s own bucket:

what	where	pin in versions.env
Chrome for Testing (Stable)	/opt/chrome-for-testing/chrome, started by /usr/local/bin/chrome	CHROME_FOR_TESTING_VERSION, a SHA256 per arch
chromedriver of the same version	/usr/local/bin/chromedriver	CHROMEDRIVER_LINUX64_SHA256, CHROMEDRIVER_LINUX_ARM64_SHA256

CHROME_EXECUTABLE=/usr/local/bin/chrome on amd64 and arm64. It is empty on riscv64, where Google publishes no build, so a lane runs the browser leg only when the variable is set:

if [ -n "${CHROME_EXECUTABLE:-}" ]; then
  flutter test --platform chrome
else
  echo "no browser in this image on $(uname -m); browser tests skipped"
fi

A lane that runs the hub’s Dart gate gets this by passing --test-platform chrome to 05-frameworks/flutter/flutter_checks.sh (since 2026-10-05). Where the variable is empty it runs the tests on the VM instead, so the gate never passes on zero tests.

A test file that reads the checkout with dart:io cannot run in a browser. Mark it @TestOn('vm') (or one test testOn: 'vm'), and the browser leg skips it while the VM leg still runs it. Measured on OmniAccelerANT (2026-10-01, the proof image below): 37 tests pass in Chrome, including all four of the native plugin’s. pinned_artefacts_test.dart and settings_asset_paths_test.dart fail there only for dart:io (“Unsupported operation: Platform._script”, “_Namespace”), and one webrtc_settings_test.dart case asserts the off-web behaviour.

06-packaging/install-chrome-for-testing.sh is the package stage’s last RUN, after the emulator, so a Chrome bump re-runs one layer of about 0.45 GB (+0.2 GB compressed per arch). It installs the zip’s deb.deps under their Ubuntu 26.04 names (only fonts-liberation was new), checks every ELF with ldd, compares both binaries with the pin, and renders a JavaScript page headless. The runtime smoke’s chrome row repeats the last two as uid 1001.

Why the wrapper passes --no-sandbox and --no-zygote

/usr/local/bin/chrome is exec /opt/chrome-for-testing/chrome/chrome --no-sandbox --no-zygote --disable-dev-shm-usage "$@". Point CHROME_EXECUTABLE at it, never at the binary.

• Sandbox. Chrome sandboxes its renderers with user namespaces, or with a setuid-root helper. The image ships no setuid helper. Whether a container may create user namespaces is the host’s call: Docker’s default seccomp profile, the AppArmor restriction of Ubuntu 23.10+ hosts and qemu-user can each refuse them. Measured 2026-10-01 as uid 1001: the bare binary started on amd64 under Rancher Desktop, and on arm64 under qemu-user it died with “No usable sandbox! If you are running on Ubuntu 23.10+ …”. The container is the isolation boundary and a test browser loads only the consumer’s own pages, so the wrapper turns the sandbox off everywhere. Flutter passes --no-sandbox itself only when headless; the wrapper also covers flutter run -d chrome under Xvfb and chromedriver.

• Zygote. Under qemu-user the zygote’s children die with “qemu: uncaught target signal 5 (Trace/breakpoint trap)”, and --dump-dom prints nothing. The package stage builds arm64 under QEMU, so its own headless check failed until the wrapper passed --no-zygote (a page then took 21 s). Without a sandbox the zygote buys nothing on native hardware either.

• /dev/shm. A container’s defaults to 64 MB, which a renderer can exhaust; --disable-dev-shm-usage moves that to /tmp.

The Android emulator runs on amd64 with KVM

The amd64 image ships the Android emulator and one x86_64 system image with ARM translation, so an arm64-v8a APK installs and runs (CON50). Both come from Google’s zips, each SHA256-pinned, and their SHA-1 equals Google’s repository XML:

what	where	pin in versions.env
emulator	$ANDROID_HOME/emulator	ANDROID_EMULATOR_VERSION, ANDROID_EMULATOR_BUILD
system-images;android-<API>;google_apis;x86_64	$ANDROID_HOME/system-images/android-<API>/google_apis/x86_64	ANDROID_EMULATOR_API, ANDROID_EMULATOR_SYSIMG_REVISION
android-avd.sh create|start|stop [name]	/usr/local/bin	—

• amd64 only. Google publishes the Linux emulator for x86_64 hosts only, and GitHub’s arm64 runners have no KVM. arm64 and riscv64 record that in /opt/android/.android-emulator-off. It adds 4.66 GB to the amd64 image (+2.1 GB compressed); the system image is 3.5 GB of it.

• API 35, not 30. API 30’s translator (ndk_translation 0.2.2) stops OmniAccelerANT’s release APK right after the Flutter engine starts: “ndk_translation: Undefined instruction 0x7ee1b800” (scalar FCVTZU), then SIGILL. API 35’s translator runs the same APK. test-chrome-and-emulator.sh refuses an older pin.

A lane needs an x86_64 host with KVM, and the device passed in:

docker run --device /dev/kvm --group-add "$(stat -c %g /dev/kvm)" \
  ghcr.io/kataglyphis/kataglyphis_beschleuniger:latest bash -lc '
    android-avd.sh start          # creates the AVD once, boots headless, waits for sys.boot_completed
    adb install --abi arm64-v8a build/app/outputs/flutter-apk/app-release.apk
    android-avd.sh stop'

• The KVM device. uid 1001 is not in the device’s group, hence --group-add; without it android-avd.sh start stops with “/dev/kvm is not usable by uid 1001”. On a GitHub-hosted x64 runner, make the device usable first: echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules, then sudo udevadm control --reload-rules && sudo udevadm trigger --name-match=kvm.

• The AVD. It lives in $ANDROID_AVD_HOME (default ~/.android/avd), because $ANDROID_HOME is root-owned. android-avd.sh writes the two ini files itself: avdmanager does not recognise an emulator installed from its zip (“”emulator” package must be installed!”). AVD_PORT, AVD_BOOT_TIMEOUT, AVD_RAM_MB, AVD_CORES and AVD_DATA_SIZE tune it.

• --abi arm64-v8a. An APK that also carries an x86_64 slice is otherwise installed as x86_64, and a library built for arm64 only is then missing.

• Measured 2026-10-01 under Rancher Desktop with KVM, in the proof image: boot in 25 s, ro.product.cpu.abilist=x86_64,arm64-v8a, and OmniAccelerANT’s release APK installed with primaryCpuAbi=arm64-v8a and stayed up.

• What the image cannot prove. The build host has no KVM, so the runtime smoke never boots it. Its android-emulator row checks the parts as uid 1001: the emulator runs, both versions equal the pins, and android-avd.sh is on PATH.

DeepStream in :latest-nvidia

:latest-nvidia (amd64) carries NVIDIA DeepStream 9.1 (CON42). What a consumer can rely on was measured on the published index sha256:810f36b3… on 2026-10-12, on an RTX 2080. How the tree is built and gated, and every pipeline that ran: linux-accelerator-images.md § DeepStream.

What	Where, or what to do
DeepStream root	/opt/nvidia/deepstream/deepstream, a link to the versioned /opt/nvidia/deepstream/deepstream-9.1; the SDK headers are in its sources/includes
GStreamer plugins	the link /opt/gstreamer/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream puts them on the image’s own plugin path. They registered with GST_PLUGIN_PATH unset, replaced or appended to, so a consumer sets nothing
nvstreammux	ENV USE_NEW_NVSTREAMMUX=yes selects the new mux, the only one that runs here. It has no width/height; a source with several surfaces per frame, such as nvdewarper, needs num-surfaces-per-frame
TensorRT	DeepStream’s TensorRT 10.16.1, on the loader path through /etc/ld.so.conf.d/010-deepstream-tensorrt.conf: libnvinfer.so.10 resolves to /opt/nvidia/deepstream/tensorrt-10.16.1.11/lib. The image has no TensorRT 11
Helper libraries	libcustom2d_preprocess.so, libpostprocess_impl.so and libcustom_videoimpl.so live in …/deepstream/lib/, not in lib/gst-plugins/. NVIDIA configs that name the latter need editing
Sample data	not in the image. NVIDIA’s deepstream-sample-data_9.1.0.deb installs into the tree with dpkg -i, as root

A GPU host. Every pipeline ran with the GPU handed in as a CDI device, on nvidia-container-toolkit 1.20.1 and driver 595.58.03, the minimum NVIDIA names for DeepStream 9.1; --gpus all exposed the GPU as well. The host setup, rootless included, is linked from § What needs a GPU host. The toolkit may load the image’s CUDA 13.4 compat driver in place of the host’s, which is why nvidia-smi inside reports CUDA 13.4. DeepStream ran on either.

How it is proven. The runtime smoke’s check_deepstream registers the elements with CUDA’s driver stubs on every nvidia image. Everything else needs a GPU, which no lane has, so it was measured by hand.

Three traps:

• No GPU, no DeepStream. Without the driver’s libcuda.so.1 the plugins cannot load, and the GStreamer registry blacklists them. Reused once a GPU is there, that registry still hides them: 30 files stayed blacklisted. The image ships no registry; never carry ~/.cache/gstreamer-1.0 from a GPU-less run into a GPU one.

• A root-owned tree. The runtime user (uid 1001) ran the decode, inference, tracker and demux pipelines and built its own engine, but nvinfer cannot store an engine next to a model in the root-owned DeepStream tree. It logs failed to serialize cude engine and builds the engine again on every start. Give model-engine-file a writable path.

• Two elements do not work here: the legacy nvstreammux and nvblender. Both are NVIDIA’s prebuilt code, and both read a buffer’s mapping after releasing it, which GStreamer 1.28 and newer turn into a NULL pointer. nvblender also stalls with two inputs, on GStreamer 1.24 too.