Artifact-copy completeness

Artifact-copy completeness#

The failure class#

The cross chain builds heavy components (ONNX Runtime, OpenCV, FFmpeg, GStreamer, Flutter, the toolchains, …) in Dockerfile.sdk / .media / .android, each hard-checked in place — e.g. test -x /opt/flutter/bin/flutter. The final runtime image is a fresh stage: Dockerfile.package’s package-image is FROM ${BASE_IMAGE}, not FROM the build stages. Every artifact must therefore be carried across that FROM boundary by an explicit COPY --link --from=artifact-source <path> <path>.

Nothing used to assert that carry happened. So an artifact could be built, verified, and then silently dropped at the boundary because no one added its COPY line. This has happened at least three times:

  • Flutter (2026-09-03) — built and test -x’d in Dockerfile.sdk, absent from :latest-cross. The user reported it missing.

  • ArmNN / ACL (F7, 2026-09-01) — cross-compiled and verified on every arm64 chain, reaching no image, until the COPY was added.

The test -x in the build stage does not catch it: that stage genuinely has the file. The runtime image is a different stage that simply never received it.

The gate#

verify-artifact-copy-parity.sh (preflight slug artifact-parity) has two halves:

  1. Consistency — every COPY --from=artifact-source has src == dst, unless the pair is a documented relocation (ALLOWED_RELOCATIONS). This half only sees paths that ARE copied.

  2. Completeness — the set of copied paths must equal linux/scripts/runtime-artifacts.manifest. A manifest path that is not copied fails (the drop bug); a copied path with no manifest line fails (a stray or renamed artifact). This is the half that would have caught Flutter.

The manifest lists each artifact-source path that must reach the runtime image, in the symbolic form used in the COPY line (a ${VAR} is compared as text), with a one-line reason.

Adding a component that ships in the runtime image#

  1. Build and hard-check it in the sdk/media/android stage.

  2. Add COPY --link --from=artifact-source <path> <path> to the package-image stage of Dockerfile.package.

  3. Add <path> | <reason> to runtime-artifacts.manifest.

Miss step 2 or 3 and artifact-parity fails before the build ever runs. If the component is deliberately build-only (never shipped), simply do not add it to the manifest and do not COPY it — the gate stays quiet.

Per-arch empty artifacts#

Some components exist on only some arches (Flutter and ArmNN/ACL are not built on riscv64). The build stage creates an empty dir on the other arches (mkdir -p /opt/flutter), so the single unconditional COPY is safe everywhere — it copies an empty dir where the component is absent. The runtime smoke then asserts the honest per-arch truth (see below).

Scope: the COPY boundary only#

The gate audits the COPY --from=artifact-source mechanism, because that is where Flutter and ArmNN were dropped. Some components reach the runtime image by other paths and are deliberately NOT in the manifest:

  • cmake — inherited from ${BASE_IMAGE} (installed in Dockerfile.base).

  • /opt/python and the /opt/venv runtime venv — built in the package stage by setup-package-image.sh (install_staged_target_python, create_runtime_venv), not copied from artifact-source.

A pre-rebuild audit on 2026-09-03 listed every /opt/* in the media image and cross-checked it against the manifest: the only artifact-source component missing was Flutter (now fixed). The build-only trees (*-cross, *-native-*, llvm-cross, wheels, cross-bin) are correctly absent. So for components that ship via a NON-COPY mechanism, the runtime smoke is the backstop — the static gate cannot see them, by design.

Bootstrapping Flutter in the package stage#

Being COPY’d is not enough for Flutter. The upstream tarball is the framework checkout only: bin/cache is empty, and the first flutter invocation downloads the Dart SDK for uname -m and compiles the flutter_tools snapshot into it — both arch-specific. Three facts shape where that has to happen:

  • The sdk stage runs on the amd64 host for every arch. Anything flutter cached there would be the x86-64 Dart SDK, stamped as current (engine-dart-sdk.stamp) and therefore never replaced on arm64. Both images shipped on 2026-09-03 carried a 716 MB SDK with an EMPTY cache precisely because the sdk stage’s best-effort flutter --version had been failing (git “dubious ownership”: the tar’s files are uid 1000, the build runs as root) behind a WARNING — the one time a masked failure was the lucky outcome. setup-flutter.sh now ships the checkout bare on purpose, and the Dockerfile.sdk cache-restore path strips bin/cache as well.

  • setup-flutter.sh is a build-stage script and returns early if the requested version is already bootstrapped. Calling it from a running container used to undo the package stage: it re-extracted the tarball over /opt/flutter and stripped bin/cache, so the very next flutter call re-downloaded the Dart SDK — 227 MB and a flutter_tools rebuild on every run of every lane. Measured 2026-09-05 in OmniAccelerANT, whose Linux, Android and web lanes all called it from setup_flutter_sdk. The guard reads frameworkVersion from bin/cache/flutter.version.json, which exists only once the SDK is bootstrapped, so a fresh /opt still installs and still ships bare.

  • Only the target-arch package stage can bootstrap. setup-package-image.sh bootstrap_flutter_sdk registers /opt/flutter as a git safe.directory, runs flutter --suppress-analytics --version, fails the stage with flutter’s own output if that fails, prints the version line, and assert_elf_archs bin/cache/dart-sdk/bin/dart against dpkg --print-architecture. Under QEMU on arm64 this costs about 3 minutes (measured 2026-09-03: Dart SDK download + snapshot compile); amd64 ~30 s; riscv64 has no flutter and the function is a no-op.

  • The runtime user must own the tree — all of it. Dockerfile.torch runs the image as kataglyphis (uid 1001) and flutter writes into $FLUTTER_ROOT on every call. With a root-owned SDK the runtime user got engine.stamp.tmp: Permission denied. So Dockerfile.package COPYs /opt/flutter with --chown=${RUNTIME_UID}:${RUNTIME_UID} (a global ARG RUNTIME_UID=1001, free at copy time — a later chown -R would copy the 716 MB into a new layer) and the bootstrap hands over what root wrote.

    Handing over bin/cache alone was not enough, and the 2026-09-04 image proved it: flutter --version ALSO runs git fetch and a dart pub get for the flutter_tools snapshot, so 37 further paths — the fetched pack, FETCH_HEAD, refs/remotes, logs/refs, and packages/flutter_tools/.dart_tool — were left root-owned. The last three broke flutter pub get for every consumer with package_config.json (OS Error: Permission denied), and that one is unfixable from outside: the directory sits in a read-only overlay layer, so a non-owner can neither chown, empty nor rename it. The handover is therefore hand_root_created_paths_to_runtime_user /opt/flutter — the same find ! -user … -exec chown -h the rust trees go through, with one owner rather than two copies — in the SAME RUN that created those paths: it selects only what root owns, which is exactly what this layer already carries (bin/cache 687 MB + 8.3 MB of git/.dart_tool measured on the shipped amd64 image), so the metadata rewrite copies nothing up. A chown -R /opt/flutter would instead copy the COPY --chown’d ~700 MB of framework checkout into the RUN layer on every arch. -h so a symlink is retargeted rather than its target.

    Running the bootstrap AS uid 1001 instead — nothing created as root, no handover at all — was considered and rejected: flutter --version downloads the Dart SDK and runs pub get, so it needs a $HOME and PUB_CACHE for that uid inside a root RUN (setpriv/su plus env), and git config --system is root-only. It would trade a measured, zero-byte find for a change nothing here can validate without a full chain. The find also covers whatever a FUTURE root command in this stage leaves behind, which running one command as the user would not.

The runtime uid is a contract#

Dockerfile.package chowns /opt/flutter to ARG RUNTIME_UID at COPY time, but the user that reads it is created much later, in Dockerfile.torch’s useradd. Those two numbers are in different files and nothing joined them: useradd without -u takes the first free uid, which is 1001 only as long as the base image happens to have no other non-system user. A base that gains one would move kataglyphis to 1002 and hand Flutter’s bin/cache to a uid that does not exist — the same Permission denied the chown exists to prevent, reappearing from an unrelated change.

Dockerfile.torch therefore declares its own ARG RUNTIME_UID=1001, passes it to useradd -u, and asserts id -u kataglyphis afterwards, so a base image that already owns that uid fails the build instead of shipping a mismatch. Both defaults must stay equal; test-setup-package-image.sh compares them.

The rust toolchain must be writable by the runtime user#

Flutter is not the only tree the runtime user writes into. rustup writes $RUSTUP_HOME/tmp/ on every invocation and downloads/, update-hashes/, toolchains/ plus settings.toml on rustup toolchain install; cargo creates $CARGO_HOME/registry/ and git/. The 2026-09-03 images shipped both trees root-owned, so at uid 1001 rustup died on could not create temp file /usr/local/rustup/tmp/...: Permission denied — which takes Corrosion, cargokit and flutter_rust_bridge_codegen with it, and cannot be worked around by repointing RUSTUP_HOME, because that directory is where the toolchains live. Consumers were passing -e CARGO_HOME=... to get half of it back.

Two paths put the toolchain in the runtime image and both had to be fixed where the ownership is decided:

  • The COPY (Dockerfile.package, amd64’s toolchain is already the right arch) carries --chown=${RUNTIME_UID}:${RUNTIME_UID}, like /opt/flutter’s.

  • The native re-install ensure_native_rust_toolchain performs on arm64/riscv64 runs as root inside the package RUN and re-creates both trees root-owned, so main calls hand_root_created_paths_to_runtime_user on them afterwards — after wire_cargo_symlinks, whose apt cargo-cbuild/ cargo-cinstall fallback links are root-created too.

hand_root_created_paths_to_runtime_user is find <paths> ! -user ${RUNTIME_UID} -exec chown -h ..., not chown -R, and that distinction is the whole cost argument. A chown -R in a new RUN rewrites metadata on all 5261 files of /usr/local/rustup (2.0 GB) and /usr/local/cargo (173 MB), forcing overlayfs to copy every one of them up into that layer: ~2.2 GB added to each of the three shipped images. Run inside the RUN that created the files the chown is free — they are already in that layer — and ! -user makes it a no-op stat walk on the arch where the COPY already did the job. Modes are never touched, so the trees stay owner-writable, not world-writable.

The shipped trees must carry the image’s own arch#

artifact-source is the builder’s image. A tree that was cross-built for the target carries target-arch ELF; a tree that was merely installed on the host carries x86-64, and the unconditional COPY ships it into the arm64/riscv64 runtime image unchanged. Two members of that class shipped for months — the 2 GB rustup whose rustc exited 127 on every foreign image, and Flutter’s cached Dart SDK, which executes natively on the build host and so passes any run-only check. Both are fixed, and each has its own gate. What was missing is the general audit: nothing read the ELF machine of what a foreign image actually carries, so a third member would ship the same way.

smoke-runtime-image.sh check_manifest_tree_arch closes that. It is manifest-driven — every path in runtime-artifacts.manifest, resolved the way the image sees it — and it reads bytes, not behaviour:

  • ${VAR} in a manifest path is resolved from the environment, else from Dockerfile.package’s own ARG VAR=default. A token that resolves from neither fails; it would otherwise scan nothing and say nothing.

  • The manifest carries the COPY source path, so _rt_tree_probe_path maps it to where the gate looks in the image. Two arms, both documented relocations listed in verify-artifact-copy-parity.sh’s ALLOWED_RELOCATIONS (the suite fails if the two owners of that fact stop agreeing): /opt/llvm-target → /usr/local/llvm-target, and the target arch’s free-threaded CPython, /opt/python-cross-ft/${TARGET_ARCH:-${TARGETARCH}}/opt/python-freethreaded → /opt/python-freethreaded (2026-10-07). The relocation is applied before the unresolved-token check, because the resolver cannot expand a ${VAR:-...}. The old /opt/vulkan → /opt/vulkan/active arm is gone: it existed to look past a builder-arch SDK that no longer ships. See below.

  • One in-image scanner reads the ELF header of every object under those trees and aggregates (tree, machine) → count. Header reads in a single process, never a readelf exec per file, which under QEMU would cost minutes; the walk is sorted so a capped scan picks the same files on every run, and it says so when the cap is reached.

  • Any object whose machine is not this image’s is fatal, and the message names the tree and an example path. Exit status is not evidence: without the TREESCAN_DONE sentinel the gate fails rather than passing on empty output, and a scan that saw no tree at all is reported as the vacuous pass it is.

What is exempt, and why the arm names the tree#

_RT_TREE_ARCH_EXEMPT holds two trees. /opt/android-sdk is one linux-x86_64 SDK, copied unchanged into every image whose HOST_PAYLOAD is on (amd64 by default, below), so its arch says nothing about this image either way. /opt/android carries the Android target’s ABI (ANDROID_TARGET_ABI, default arm64-v8a), not the image’s, so check_android_abi judges it against the ABI the image advertises instead (linux-cross-builds.md § The Android ABI is a target, not the build host). The arm names the tree, never an arch — so a newly host-installed tree fails by default instead of inheriting somebody’s exemption. A tree that is present but holds no ELF at all (a per-arch empty dir) is a note, not a pass: presence is the ARCH-PARITY table’s assertion, and absence of the directory itself is fatal here.

What the exemptions are worth#

Every exemption above was reasoned from the build graph when it was written. On 2026-09-05 each was re-read from the three shipped :latest-cross-<arch> images with the gate’s own scanner, and the reasons did not all survive.

exemption

measured, per image

verdict

/opt/android-sdk

the same 40 644-file tree in all three: 582 X86-64, 417 Intel-80386, 414 ARM, 414 AArch64, 33 RISC-V

KEEP. Without it arm64 and riscv64 red on 1 446 foreign objects that are the SDK’s host toolchain and device libs

/opt/android

amd64 37 X86-64, arm64 37 AArch64, riscv64 34 RISC-V — and nothing else

DELETED. The reason (“device .so, arch says nothing”) is false: arch_android_abi_for maps each build arch to the ABI with the SAME ELF machine (amd64→x86_64, arm64→arm64-v8a, riscv64→riscv64), and each image carries exactly one ABI directory. The gate asserted it for free — until ANDROID_TARGET_ABI stopped following the build arch; /opt/android is exempt again, and check_android_abi asserts it (above)

/opt/vulkan probed at /opt/vulkan/active

active/ is 122 files, 3 objects, all target. The rest of /opt/vulkan on arm64/riscv64 was 1 317 X86-64 + 2 Intel-80386 objects across 33 926 files — 1.8 GB of x86_64/ plus 3.9 GB of source/

DELETED 2026-09-05. The narrowing was hiding a packaging defect, not a legitimate host tree. Both subtrees are pruned before the COPY now and the gate asserts the WHOLE tree on all three arches — the section below

the cross-payload filter (/lib/rustlib/, /lib/clang/, /lib/gcc/, gcc-*/​<triple>/)

on amd64 it hides exactly the target payload: rustup’s AArch64+RISC-V libstd, gcc’s per-triple crt objects, clang’s i386 multilib runtimes. On the foreign images it hides only own-arch objects

KEEP. It also does not hide the five X86-64 libLLVM/libclang in /usr/local/llvm-target on either foreign arch, which is how the frozen count of 5 was confirmed exact

Two further facts the same scan settled, both of which had never been checked on a real image: all 15 manifest trees exist as directories on all three arches (so no MISSING verdict is latent), and no tree comes near the 20 000-file RT_TREE_CAP — the largest asserted one is /opt/flutter at 17 523 candidate files on amd64/arm64, 88 % of the cap. A Flutter bump could cross it; that is a CAPPED failure, not a silent pass.

The Vulkan tree ships only what the image runs#

Measured on the shipped :latest-cross-<arch> bytes, 2026-09-05, and re-measured after the fix by running the prune inside a throwaway container over the same bytes.

/opt/vulkan held 6 010 149 323 B on arm64 and 6 037 710 004 B on riscv64 across 33 926 files — 19.5 % / 20.3 % of those two images. Of that, exactly 129 files (60 101 090 B / 87 611 162 B) were the target prefix the image runs. The other two subtrees:

subtree

bytes

what it is

who reaches it in the runtime image

<ver>/x86_64/

1 858 995 156

the LunarG tarball’s own prefix — vulkaninfo, glslc, the validation layers, all x86-64

nobody. setup-env.sh derives VULKAN_SDK from uname -m, so a foreign image resolves <ver>/aarch64; PATH, LD_LIBRARY_PATH and active follow it

<ver>/source/

4 091 053 077

the checkouts + build trees ./vulkansdk makes, holding 1 256 of the 1 317 X86-64 objects and both Intel-80386 ones

nobody, and nobody after the SDK stage either: _build_vulkan_targets is its only consumer

Both are removed at their source rather than filtered at the gate:

  • _vulkan_prune_sdk_sources (02-toolchain/vulkan.sh) drops <ver>/source in the same RUN that created it, so no layer from the SDK stage onward carries it. It runs only on the cross path — an amd64 install never builds and has no source/.

  • prune-vulkan-host-sdk.sh runs in Dockerfile.package’s artifact-source stage, ahead of COPY --from=artifact-source /opt/vulkan, and drops <ver>/x86_64 when — and only when — a real (non-symlink) <ver>/<own-arch>/lib/libvulkan.so.1 exists. On amd64 the two are the same directory and nothing is dropped. When a cross lane’s loader failed to build, the host prefix is KEPT and the run says so loudly: the image then reds the tree-arch gate, which is the honest outcome for an image with no Vulkan of its own.

Measured on the shipped bytes with the real script: /opt/vulkan goes from 6 010 149 323 to 60 101 090 bytes on arm64 (33 926 → 129 files) and from 6 037 710 004 to 87 879 250 on riscv64, with VULKAN_SDK, the active link, glslangValidator and a vkEnumerateInstanceVersion call through the entrypoint all still resolving to the target prefix afterwards. amd64 is byte-identical.

What the gate now asserts. The /opt/vulkan → /opt/vulkan/active narrowing is gone, so check_manifest_tree_arch walks the whole tree: 3 objects on each foreign image, 61 on amd64, every one the image’s own machine. Before the prune the same walk reported 1 317 X86-64 + 2 Intel-80386, which is exactly the failure the narrowing was suppressing. A second, run-based assertion backs it: check_vulkan_loader reads /proc/self/maps after CDLL("libvulkan.so.1") and fails unless the loader that answered lives under /opt/vulkan. Ubuntu’s multiarch libvulkan.so.1 is in every image, so a prune that took the prefix the image actually runs would otherwise have passed a load-only check on the distro copy. On the three shipped images today that path is <ver>/aarch64/lib, <ver>/riscv64/lib and — via setup-env.sh — <ver>/x86_64/lib/VulkanLoader/lib.

The host payload stays with the host that runs it#

Read from the published :latest of 2026-10-11 (index b52f8c34…): its arm64 and riscv64 children each carried three layers byte-identical to amd64’s, 3.41 GB compressed of the arm64 child’s 10.11 GB and of riscv64’s 8.51 GB. None of it runs on those arches:

tree (layer, compressed)

why an arm64 or riscv64 image cannot use it

/opt/android-sdk (1 816 MB)

adb, aapt2 and the NDK’s only prebuilt toolchain are linux-x86_64 (why)

/opt/android (850 MB)

arm64-v8a libraries for an Android app, which only that NDK links

/usr/local/rustup (745 MB)

the cross builder’s x86_64 toolchain, deleted again one layer later by ensure_native_rust_toolchain (why)

prune-host-payload.sh runs in Dockerfile.package’s artifact-source stage, ahead of the four COPYs, like the Vulkan prune above. With HOST_PAYLOAD=off it:

  • empties /opt/android-sdk and /opt/android, and writes /opt/android/.android-payload-off, the marker smoke-android.sh, the emulator installer and the contract gate already read. A marker the android stage wrote on a non-amd64 build host is kept as it is.

  • empties /usr/local/rustup, and /usr/local/cargo but its registry, when the toolchain is not the target’s. That is ensure_native_rust_toolchain’s own rule, one stage earlier, so no shipped file changes: the package stage installs the native toolchain over an empty tree, as it did over the deleted one. A native artifact’s toolchain, and amd64’s, stays.

The directories stay, so the COPYs, this page’s manifest and the ARCH-PARITY table are unchanged. The package stage sets ANDROID_HOME, ANDROID_SDK_ROOT and the six Android root variables to empty and appends nothing to PATH, the way CHROME_EXECUTABLE is empty on riscv64. ANDROID_TARGET_ABI stays: it is a pin, not a path.

runtime_host_payload (01-core/runtime-build-fns.sh) passes HOST_PAYLOAD per arch. It is on for the arches in HOST_PAYLOAD_ARCHES, and off for the rest and for every artifact that no amd64 host built, since the android stage installs the SDK only there:

HOST_PAYLOAD_ARCHES

amd64

arm64, riscv64

unset (amd64)

as before

empty trees, the marker, empty variables

amd64,arm64,riscv64

as before

the images as published until 2026-10-13

none

no SDK either

as the default

The chain and both runtime orchestrators refuse any other value (exit 2) before they build. Measured over the real trees on 2026-10-13, with the amd64 :latest running the script for an arm64 target: 2 193 MB of rustup, 230 MB of cargo, 8 167 MB of SDK (the emulator included, which only amd64 installs) and 3 490 MB of /opt/android emptied in 9.6 s. The arm64 and riscv64 pulls lose the three layers: at the ~51 MB/s (compressed) the arm64 pulls unpack at, about 67 s of every arm64 job’s pull, an estimate until a chain publishes it.

What holds it: test-host-payload.sh (the prune on fixture trees, its place in the Dockerfile, the per-arch resolver), the contract row android-home, which skips a payload-off image and fails one that still sets ANDROID_HOME, and check_android_abi, which skips it and fails one whose /opt/android still carries objects.

The llvm-target prefix fills what it needs, and nothing else#

/opt/llvm-target is the one tree that is repaired rather than merely copied, and until 2026-09-05 both repairs were written as a glob over the builder’s multiarch directory — every libLLVM*.so.2* / libclang*.so.2* in /usr/lib/x86_64-linux-gnu, copied in whenever lib/<soname> was absent.

The repair exists for a real defect: on amd64 the prefix is an apt LLVM tree whose lib/ holds dev symlinks into the multiarch dir, so the runtime sonames never entered the prefix and the shipped clang bound to whatever ambient libLLVM the consuming image carried (the 2026-08-11 franken-toolchain incident). What was wrong was the selection rule. A glob copies whatever Ubuntu happens to have installed, and the builder carries llvm-20 and llvm-21 alongside the pinned release.

Measured in the 2026-09-05 images, that shipped five x86-64 libraries into both foreign runtime images — libLLVM.so.20.1, libLLVM.so.21.1, libclang-21.so.21, libclang-23.so.23, libclang-cpp.so.21.1, 392 MiB — because the guard tests the filename, and an arm64/riscv64 source build never produces a soname named after Ubuntu’s llvm-20/21. They were a closed island: the only objects in either image that DT_NEEDED any of them were the copied libs themselves, and the four real consumers (libgallium, libOSMesa, doxygen, the llvm-21 tools) resolve through /usr/lib/<triplet> — ld.so rejects the wrong e_machine and falls through, so the copies were never loaded, only carried. The same glob put 358 MiB of the same Ubuntu libs into the amd64 prefix, where the tree-arch gate structurally cannot see them because they are the image’s own arch.

The rule is now demand-driven: _llvm_target_fill_needed in 02-toolchain/materialize-llvm-target.sh reads the DT_NEEDED of the prefix’s own bin/ and lib/*.so*, copies only the LLVM-family sonames that do not resolve inside lib/, and repeats until a round adds nothing (a filled lib brings its own NEEDED). -e is false for a dangling symlink, which is exactly the apt dev-link case, so the entry is removed before the copy. Its correctness is not a matter of reading: the NEEDED walk immediately below is a hard gate — every LLVM-family NEEDED of every shipped object must resolve inside the prefix, and every other NEEDED (libc, libstdc++, zlib) against the base image’s ldconfig cache, or the sdk stage fails. The cache is captured once into a variable and matched with case, because ldconfig -p | grep -q dies of SIGPIPE under pipefail.

The package stage no longer keeps a second copy of that logic. repair_llvm_target_sonames was written as belt-and-braces for pre-2026-08-12 sdk artifacts, and its own note said it should be removed once a rebuild showed it copying nothing; the rebuild showed it copying nothing on amd64 and only the five wrong-arch libs on the foreign arches, so the copy half is gone. What remains — publish_llvm_target_ld_path in 06-packaging/copy-media-payloads.sh — is load-bearing and stays: libtvm_compiler.so carries a DT_NEEDED on libLLVM.so.<ver> that resolves through /etc/ld.so.conf.d/000-llvm-target.conf and nothing else, so dropping it makes import tvm die.

What only a real build could tell you — and did#

The table used to be reasoned from the build graph alone. The 2026-09-04 run and the 2026-09-05 re-read of the shipped images answered it: /opt/gcc-* is target-native (swap-native-gcc.sh rm -rfs the cross tree and copies the Canadian-cross native one over it in the android stage); /opt/llvm-target, /opt/armnn and /opt/acl are cross-built for the target; and the two members that could not be ruled out statically — /opt/flutter (bin/cache engine artifacts) and /usr/local/rustup (downloads/ leftovers) — measure clean: flutter is 19 objects of the image’s own machine on amd64/arm64 and empty on riscv64, rustup is 16/16/48 own-machine objects with every foreign one inside lib/rustlib/<triple>. What the run did catch was the gate’s own false positives (cross payload, now filtered) and the five builder-arch LLVM libraries that are backlog HT3. If a tree turns out to carry a legitimate builder-arch helper the gate names the tree and an example path, and the fix is one arm with a written reason — never a loosened comparison.

The end-to-end backstop#

smoke-runtime-image.sh check_flutter runs against the shipped image as the image user, with --network none: on amd64/arm64 flutter --version must report FLUTTER_VERSION from versions.env, and the cached dart-sdk/bin/dart must have the target arch’s ELF machine (an x86-64 dart in the arm64 image would execute natively on this host and pass a run-only check); on riscv64 flutter must be honestly absent. The static gate proves the COPY is wired; the smoke proves the result actually runs in the image, offline. Both are needed — the gate cannot execute the binary, and the smoke needs a built image.

Running is not the claim, though — the 2026-09-04 image passed this gate green while flutter pub get was broken for every consumer. So the same probe also asks whether the SDK is USABLE by that user: bin/cache and packages/flutter_tools/.dart_tool must be writable, and find /opt/flutter ! -user "$(id -u)" must come back empty. Both are test/stat questions, not a pub get, because the smoke runs with --network none; each has its own verdict line, so the message names the consumer symptom rather than a path.

Guards#

  • tests/test-artifact-copy-completeness.sh — fixtures for both failure directions plus regression guards that the real manifest and Dockerfile agree.

  • tests/test-setup-package-image.sh — bootstrap_flutter_sdk off-target: the no-op, the hard failure with flutter’s output, the arch check, the handover of every root-created path (bin/cache, .git/FETCH_HEAD, the flutter_tools .dart_tool) and — on a tree the runtime user already owns — no chown at all, which is the size rule as an assertion.

  • tests/test-setup-package-image.sh — hand_root_created_paths_to_runtime_user over a real tree with chown stubbed on PATH: the rustup/cargo pair root re-installed is handed over (tmp/, toolchains/, the apt cargo-cbuild link), a tree the COPY already owns draws no chown at all, main calls it after both root writers, and both /usr/local COPYs carry --chown. Mutations rust.copy-chown, rust.toolchain-handover, rust.handover-only-root-owned.

  • tests/test-runtime-image-gates.sh — check_flutter against recorded image output: offline, the permission-denied shape, a pin mismatch, a builder-arch Dart SDK, an unwritable .dart_tool, a root-owned leftover, and the correct shape.

  • tests/test-runtime-image-gates.sh — check_manifest_tree_arch: the real scanner against a fixture tree of hand-written ELF headers, the verdict function on a builder-arch tree in both directions, the missing-sentinel path, and the two host-side cross-checks (every manifest path resolves; the relocation and the exemption list still agree with their other owners).

  • tests/test-llvm-target-prefix.sh — _llvm_target_fill_needed over a fixture prefix: the Ubuntu llvm-20/21 libs nobody needs are NOT copied, a dangling dev symlink IS replaced by the real soname, a lib pulled in by a lib is reached, the loop terminates, the package stage no longer copies anything into the prefix, and the loader-path half is still written. Plus _RT_TREE_ARCH_FROZEN empty and both of its gate arms.

  • Mutations artifact-parity.completeness-check, flutter.bootstrap-hard-fail, flutter.bootstrap-cache-handover, flutter.bootstrap-whole-tree-chown, flutter.smoke-offline, flutter.smoke-dart-arch, flutter.smoke-sdk-writable, flutter.smoke-root-owned, probe.tree-arch-mismatch, probe.tree-arch-machine-word, probe.tree-arch-scan-sentinel, probe.tree-arch-arg-default, probe.tree-arch-relocation, llvm-target.fill-is-needed-driven, llvm-target.fill-replaces-dangling, llvm-target.fill-reaches-fixpoint, llvm-target.no-package-stage-copy, llvm-target.ld-path-published, tree-arch.frozen-table-empty, tree-arch.frozen-key-is-per-arch.

The membership test must not shell out#

2026-09-09: this gate failed roughly one run in ten on an unchanged tree, naming a different artifact each time and in either direction — /opt/ffmpeg, /opt/acl, /opt/armnn, /opt/flutter, ${ONNXRUNTIME_OUTPUT_DIR}. It blocked a commit whose diff touched neither Dockerfile.package nor the manifest.

Instrumenting the failure settled it: on a red run the two sets are byte-for-byte identical, 15 against 15, and the needle that was reported absent is plainly in the haystack. The bug was the test itself,

printf '%s\n' "${haystack}" | grep -qxF -- "${p}"     # DO NOT

-q makes the matcher exit on its first hit while a pipe is still feeding it, and the exit status that comes back from ugrep (this host’s grep) is not reliably 0 in that race. 2250 isolated repetitions did not reproduce it; only the gate’s own two-direction loop did, at ~1 in 300 comparisons.

The replacement needs no subprocess at all, and is both exact-line and literal:

case ${_hay_padded} in                 # _hay_padded="${_NL}${haystack}${_NL}"
  *"${_NL}${p}${_NL}"*) ;;             # present
  *) ...report the miss... ;;
esac

200 consecutive runs green afterwards, and both directions still redden when a manifest row or a COPY is removed — the fix must not be allowed to make the gate quiet instead of correct.

The general rule this earns: a set-membership test in a gate loop is a shell case, not a grep -q on a pipe. The same spelling appears elsewhere in this repo; each one is a coin flip waiting to be blamed on the tree.