# Build watch list — the 2026-09-05 wave, read against a running chain

Follow this while the chain runs. Every row names the **exact** line that means the
change worked and the **exact** line that means it did not. Grouped by stage, in the
order a chain reaches them, so it can be read top-to-bottom live.

The wave that produced this list closed eleven backlog entries on **static** proof
only. A first rebuild attempt already found two build-killing bugs that a full green
battery had passed (HEAD `e109f5ad`), so treat every "expected" below as a
hypothesis the chain is testing, not a fact.

Chain in flight when this page was written: `chain-status.json` run
`20260905-120554-7b7a0d4e`, `sdk..runtime`, `amd64,arm64,riscv64`.

**Re-groomed 2026-09-07 for the Vulkan rows only.** That chain finished; the wave
after it wired all four components this page had listed as known or expected
failures. Those rows now say the opposite of what they said, because an
`unavailable` line for any of the four is a regression. VK2 closed on 2026-09-09
with 20/20 components on both foreign arches
([archive](refactoring-backlog-archive-2026-09-09.md)). The rest of the page is
the 2026-09-05 text, with log strings kept matching the code.

## Read this first: the three that abort a run

If the chain dies early, it is almost certainly one of these. Each fails **fast**, at
its own stage, rather than shipping — which is the design, but it is still a dead run.

| # | grep for | stage | what it means |
|---|---|---|---|
| 1 | `media_compiler_launcher: command not found` | media | the image's `03-media/core/common.sh` is an **older layer** than `build-ffmpeg.sh`/`build-pyav.sh`, which now call the new helper unqualified under `set -e`. A cache-layering problem, not a wiring one — all three files ship in the same commit |
| 2 | any path printed after `still holds link(s) that resolve to nothing` | sdk (all 3 arches) | `_llvm_target_repair_links` ends in `find <prefix> -xtype l` and exits 1 on any survivor. It has never run on any arch. The likely cause is a **materialised directory carrying a broken link of its own** — the one path the loop cannot reach |
| 3 | `ERROR` from `prune-vulkan-host-sdk.sh` | package (`artifact-source`) | a NEW `RUN` with two bind mounts. Fails before any COPY, so it costs minutes, not hours |

## SDK stage

### Vulkan: the source tree dies where it is born (HT5)

| verdict | line |
|---|---|
| PASS | `Vulkan cross-targets <arch>: N/N component(s) built` **immediately followed by** `Pruning the Vulkan SDK build tree at /opt/vulkan/<ver>/source` |
| FAIL | the prune line appearing **before** the component verdict — that order deletes the loader/SPIRV-Tools/glslang sources out from under the build |
| FAIL | no prune line at all on arm64/riscv64 — the foreign lanes then carry 4.09 GB of `./vulkansdk` build tree into every downstream image |
| n/a | amd64 prints neither; it never takes the cross path and has no `source/` |

### Vulkan: fifteen components cross-built for the first time (VK1)

**The arm64 lane of this run has already answered this**, and the answer is backlog
**VK2**: eleven of fifteen built, including the three that matter for building an
application — `glslc`, `vulkaninfo` and the validation layers. Use that as the
baseline for the riscv64 lane rather than the pre-run guesses.

| verdict | line |
|---|---|
| PASS (measured on arm64) | `Cross-building <label> for <arch>` with no following `unavailable` line, for eleven of: `vulkan-headers`, `spirv-headers`, `vulkan-utility-libraries`, `volk`, `vma`, `spirv-cross`, `spirv-reflect`, `shaderc`, `vulkan-tools`, `vulkan-extensionlayer`, `vulkan-validationlayers` |
| FAIL (was KNOWN FAIL before VK2) | `vulkan-profiles unavailable` — `jsoncpp` and `valijson` are rows of their own in `_VK_TARGET_COMPONENTS` since `c59799b4`, ordered ahead of it, so `find_package(valijson)` has somewhere to look |
| FAIL (was KNOWN FAIL before VK2) | `gfxreconstruct unavailable` — and the old diagnosis on this row was wrong. The `:${arch}` packages were already unpacked; `find_library` could not see them because nothing passed `CMAKE_LIBRARY_ARCHITECTURE`, which `_cross_build_sdk_component` now does for every row. The genuinely missing half was GL, added to `install_optional_target_packages` |
| FAIL (was EXPECTED FAIL before VK2) | `slang unavailable` — and NOT for the reason this row gave. It is not host LLVM `tblgen`; slang cross-compiles its own generators and then cannot run them, so `SLANG_GENERATORS_PATH` points at the host `./vulkansdk` run's copies. `tblgen` is the `dx*` family's problem; `dxc` has been a row since VK4 (2026-09-09), fed the host run's `llvm-tblgen` the same way ([host tools](vulkan-foreign-arch-sdk.md#components-that-need-a-host-tool)) |
| FAIL (was EXPECTED FAIL before VK2) | `vulkancapsviewer unavailable` — `qt6-base-dev:${arch}` is in the optional set with `QT_HOST_PATH=/usr` and a sysroot `CMAKE_PREFIX_PATH` |
| FAIL | `shaderc: source missing at …/shaderc/src; skipping` — the checkout IS one level down and `_vulkan_target_src` is supposed to find it. It built on arm64, so this would be a regression |
| FAIL | any `Cross-building` line naming a `-B TMP/<label>` path for a component whose source is absent — the missing-source guard has stopped skipping |

**Do not "fix" a Vulkan row while the chain is running.** That was the 2026-09-05
instruction and it still holds: editing `vulkan.sh` mid-run ships two arches from
different sources, which is the drift this repo has been bitten by before. The four
routes above are already in the tree; what the chain owes is a verdict on them, and
`<arch>/bin` carrying what `x86_64/bin` does is what closed VK2 (2026-09-09).

### LLVM target prefix (HT4)

| verdict | line |
|---|---|
| PASS | the stage completes with no output at all from `_llvm_target_repair_links` on arm64/riscv64 (both measured 0 dangling today, so **any** output there is news) |
| PASS | amd64 prints `amd64 /opt/llvm-target NEEDED walk clean` and **not** `is NOT self-contained` |
| FAIL | `still holds link(s) that resolve to nothing` plus a path list — see #2 above |
| FAIL | the walk aborting on an unresolved `liblldb` soname. `liblldb*` joined the LLVM family this wave, so an apt set that ships the lldb binaries without the lib now stops the stage where it used to fall through |

### sccache — the YB verdict, and it is finally greppable (YB)

The launcher used to print its `[server=]` field **only when sccache failed**, so a
run where the fix worked carried no evidence. Both address setters now print it on the
healthy path, identically spelled. One grep covers the whole chain:

```
grep -o '\[server=[^]]*\]' <log> | sort | uniq -c
```

| verdict | line |
|---|---|
| PASS | every occurrence reads `[server=/tmp/sccache-<uid>.sock]` |
| FAIL | a **single** `[server=tcp:4226]` anywhere — the regression, unfixed |
| context | the two lines are `[INFO] Using sccache with SCCACHE_DIR=… (cap …) [server=…]` (`01-core/common.sh`) and `[INFO] sccache enabled: SCCACHE_DIR=…, CACHE_SIZE=… [server=…]` (`01-core/compiler-cache.sh`) |

Needs a **compile-heavy** chain. `--only runtime` emits none of these lines. Also
worth reading deliberately: `sccache --show-stats` at the START of each step should
report **0** compile requests, not another container's hundreds.

## Media stage

| what | PASS | FAIL |
|---|---|---|
| the ffmpeg/pyav launcher owner (F3) | `Using sccache` / a `--cc=<launcher>` in the ffmpeg configure line, and a **non-zero sccache hit count on the second arch** | `media_compiler_launcher: command not found` — see #1 above |
| TFLite for the GStreamer monorepo (CL6) | the monorepo configures with TFLite found on **arm64 AND riscv64**. riscv64 is the one to watch, as always | TFLite missing on either. First thing to check is whether the parent still calls all three helpers in order — `_gst_tflite_probe_flags`, `_gst_tflite_fix_pc`, `_gst_tflite_symlink_for_meson` |
| the media stage's cross assertion (CL7.3) | nothing new printed | `media_common_init: critical module(s) did not load` — the `03-media` fallback clone was deleted, so a caller without cross-env now stops here instead of limping |

## Package / runtime stage

### The Vulkan host prefix (HT5)

| verdict | line |
|---|---|
| PASS | the `artifact-source` prune runs and exits 0 before the `/opt/vulkan` COPY |
| PASS, deliberately | on amd64 the prune is a **no-op** — there `x86_64` IS the prefix the image runs |
| PASS, deliberately | on a lane whose cross Vulkan loader failed to build, the prefix is **KEPT** and the script says so loudly. The tree-arch gate will then fail that image on 61 X86-64 objects. That is the honest verdict for an image with no Vulkan of its own, and it is a **new way for a chain to go red** |
| FAIL | the script deleting anything on amd64 — that takes `VULKAN_SDK`, the loader and every host tool with it |

### The host payload (CON96, new 2026-10-13)

| verdict | line |
|---|---|
| PASS | amd64: `host-payload: HOST_PAYLOAD=on for amd64; the Android SDK, its payloads and the Rust toolchain are copied as built` |
| PASS | arm64, riscv64: `host-payload: emptying /usr/local/rustup and /usr/local/cargo but its registry (… -- <arch> runs <triple>, …)`, then `host-payload: emptying /opt/android-sdk and /opt/android (HOST_PAYLOAD=off for <arch>)`; later in the package RUN, `Rust toolchain in /usr/local/rustup is not <triple>: -- reinstalling natively` |
| FAIL | any `host-payload: emptying` on amd64 — the Android lane loses its SDK; the smoke's `check_android_abi` fails such an image too |
| FAIL | the smoke's `BAD android-home … says the image ships no SDK`: the ENV and the pruned trees disagree |
| on the bytes | `nerdctl run --platform linux/arm64 <image> sh -c 'du -sh /opt/android-sdk /opt/android; echo "${ANDROID_HOME:-empty}"'` prints `4.0K`, about `8.0K` (the `.android-payload-off` marker, plus `.android-emulator-off` where the emulator installer wrote one) and `empty`; the arm64 child's compressed size falls by about 3.4 GB against 10.11 GB |

### absl, the defect the corrected probe found (CL1)

| verdict | line |
|---|---|
| FAIL | `copy-media-payloads: optional payload missing: /artifact-src/usr/local/include/absl` on **any** arch — the artifact image did not carry absl and the fix is inert there |
| PASS | no such line, on all three |

### Packaging (new this wave, none of it ever run)

| what | PASS | FAIL |
|---|---|---|
| AppImage runtime | `Staged AppImage runtime-<arch> (<n> bytes) from …`, once per arch | `no squashfs superblock in <tool>; runtime-<arch> not staged` — a WARN, non-fatal, but the runtime is not staged and consumers go back to fetching it from GitHub |
| Flatpak runtimes | seven refs installed on amd64/arm64; **skipped outright** on riscv64 (Flathub builds x86_64 and aarch64 only) | a 404 retry loop on riscv64 means the arch guard stopped working |
| web-lane toolchain | `OK: nightly-<date> installed with rust-src + wasm32-unknown-unknown` (the `RUST_NIGHTLY_TOOLCHAIN` pin) and two `OK: <crate> <version> installed` | `WARN: nightly-<date> is unavailable` — non-fatal, but the web lane then auto-installs a nightly per consumer run, which is the cost this exists to remove |
| web-lane tools from source (riscv64, new 2026-09-23) | android `web-lane-tools`: `OK: web-lane producer: <crate> <version> built for riscv64gc-unknown-linux-gnu in <n> s`, or `cache HIT`; amd64/arm64 android: `skipped: not in WEB_LANE_TOOLS_CROSS_ARCHES` / `native-build-platform`. riscv64 package: `installed from the cross-built artifact`, twice | `WARN: web-lane <crate>: … the cross fast path was not taken` — non-fatal, but that RUN pays the QEMU compile (~44 min) once per key; the line carries the producer's reason. Any `ERROR: web-lane` stops the package stage on purpose ([contract](consumer-image-contract.md#building-the-web-lane-tools-from-source)) |

## On the shipped bytes — after the run, not from the log

This is the half the 2026-09-04 chain never reached, and the half where "verify the
shipped BYTES, never the push" applies.

### Sizes (CC1 has asked for this baseline twice)

Today's reading is **30.37 / 30.84 / 29.69 GB** (amd64/arm64/riscv64), verified
against `nerdctl images`. HT5 removes ~5.95 GB from each foreign image; VK1 adds an
unknown amount back by cross-building fifteen more components. **Read the two
together or neither number means anything.** Record all three.

| expected | `/opt/vulkan` ~60 MB on arm64 and ~88 MB on riscv64 instead of 6.0 GB; amd64 byte-identical; the `cross-android-*` artifact images ~3.9 GB lighter on the two foreign lanes |

### Probes to run, read-only

```
# HT4 — was 12 / 0 / 0
find /usr/local/llvm-target -xtype l                      # must be 0 on all three
/usr/local/llvm-target/bin/lldb --version                 # works (fails today)
python -c 'import lldb'                                   # imports (fails today)
readlink /usr/local/llvm-target/lib/libclang.so           # = libclang-23.so.23
test -f /usr/local/llvm-target/include/llvm/IR/IRBuilder.h
ls /usr/local/llvm-target/lib/libc++*                     # must be EMPTY
# the unstartable-binary count: 3-of-142 -> 0-of-142

# HT5 — the gate was SHARPENED, not loosened
#   check_manifest_tree_arch now walks the WHOLE /opt/vulkan tree
#   expect: "OK /opt/vulkan: N ELF object(s), all AArch64"  (all X86-64 on amd64)
#   a FAIL naming 1 317 X86-64 objects means the prune did not run
#   check_vulkan_loader now asserts WHERE the loader loaded from:
#     "loads from /opt/vulkan/<ver>/<arch>/lib/..."     = PASS
#     a FAIL naming /usr/lib/<triple>                   = the prune took the prefix
#                                                         the image actually runs
#     a WARN naming no path                             = /proc/self/maps unreadable,
#                                                         not a defect

# CC1 — a REGRESSION watch, not a discovery: all three read 0 today
find /usr/local/rustup /usr/local/cargo ! -user 1001      # must stay 0
find /opt/flutter -user root                              # must stay 0
nerdctl image inspect … | jq .Config.Env                  # the ENV strings must be BAKED

# CL1 / absl — 1 FAIL today, must be 0
nerdctl run --rm --platform linux/<arch> -v <repo>:/repo:ro --entrypoint bash \
  ghcr.io/kataglyphis/kataglyphis_beschleuniger:latest-<arch> \
  /repo/linux/scripts/06-packaging/smoke-critical-fixes.sh
#   expect: PASS absl/types/span.h found in /usr/local/include

# AB1 — 420 objects, every one X86-64, on today's bytes
#   check_android_abi must report the payload matching the advertised ABI,
#   reading .a MEMBERS as well as .so files
```

### What only the runtime smoke can say

`check_consumer_contract` asserts 9 rows and has **only ever run on amd64** — the
2026-09-04 chain died before arm64 and riscv64. Same for the advertised-key gate's 16
`OK`s with zero UNSET/UNREAD. Two green foreign verdicts here are new information.

## Known-good noise — do not chase these

* `VK_LAYER_PATH` is empty in every running image and always has been. The entrypoint
  sources LunarG's `setup-env.sh`, which UNSETS it and exports `VK_ADD_LAYER_PATH`
  instead. Not touched this wave; backlog **R1** (closed 2026-09-07,
  [archive](refactoring-backlog-archive-2026-09-07.md)) fixed the value a consumer
  that does not source it gets
  ([`vulkan-foreign-arch-sdk.md`](vulkan-foreign-arch-sdk.md#vk_layer_path-pointed-at-a-directory-that-has-never-existed)).
* The lib-dynload audit WARNs for five optional modules (`_zstd`, `readline`,
  `_curses`, `_uuid`, `_decimal`). Information on `optional` rows — but any of them on
  **amd64** is a genuine finding. `readline` is a `required` row in
  `01-core/cpython-dev-packages.sh` now, so its WARN is a finding on every arch.
  `_ctypes` warns on the GIL cross arches by design (`build_python.sh` sets
  `ac_cv_header_ffi_h=no` there). The free-threaded cross trees build it, and a
  missing one fails their audit.
* `slang` and `vulkancapsviewer` failing to cross-configure was expected until
  2026-09-07 and is **not** any more — both have a route now, so either one failing
  is a finding, not noise.