Linux Host Setup (GPU, drivers, runtime, boot)

Contents

Linux Host Setup (GPU, drivers, runtime, boot)#

Goal: take a fresh Ubuntu/Debian host to a state where it can build and run this repo’s Linux images with GPU acceleration. This is the Linux counterpart to Fresh Windows Host Bring-Up.

Scope is the host, not the images. What goes inside a container is owned by Linux Build Basics and Linux Accelerator Images; running Linux containers from a Windows host is Rancher Desktop.

Every phase ends with a verify command. Run them — a driver that installed without error but did not load is the common failure here, and it looks identical to success until the first build.

Phase A — GPU drivers#

A1. NVIDIA driver#

sudo apt update
sudo apt upgrade -y
sudo apt install -y build-essential linux-headers-$(uname -r)

sudo ubuntu-drivers devices        # list what this GPU supports
sudo ubuntu-drivers install        # let the tool pick
# ...or pin a specific branch:
# sudo apt install -y nvidia-driver-590

Reboot afterwards. The kernel module is only loaded on a fresh boot.

Verify:

nvidia-smi

A2. CUDA toolkit#

ubuntu-drivers does not install the toolkit — nvcc comes separately.

sudo add-apt-repository -y multiverse
sudo apt update
sudo apt install -y cuda-toolkit

CUDA does not put itself on PATH. Add a profile drop-in so every login shell and every build script sees it:

sudo tee /etc/profile.d/cuda-path.sh >/dev/null <<'PROFILE'
export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
PROFILE

Verify (in a new shell):

nvcc --version

A3. Purge and recover a broken NVIDIA install#

A half-upgraded driver stack is the usual cause of nvidia-smi failing after an apt upgrade. Do not fight it incrementally — purge and reinstall.

First see what was pulled in manually, so you know what to put back:

sudo apt-mark showmanual | grep -E 'nvidia|cuda'

Then remove everything, including the repo definitions:

sudo apt-get remove --purge '^nvidia-.*' 'libnvidia-*' 'linux-modules-nvidia-*' -y
sudo apt purge -y 'cuda-*' 'nsight-*'
sudo apt autoremove --purge -y
sudo apt autoclean

sudo rm -f /etc/apt/sources.list.d/cuda*.list
sudo rm -f /etc/apt/sources.list.d/nvidia*.list
sudo rm -f /usr/share/keyrings/cuda-archive-keyring.gpg
sudo dpkg -P cuda-keyring 2>/dev/null || true

sudo dpkg --configure -a
sudo apt --fix-broken install -y
sudo apt update
sudo reboot

Then start again at A1.

A4. AMD#

AMD GPUs need no proprietary driver package for the ROCm lane — the kernel driver is in-tree. What is sometimes missing is firmware:

sudo apt install -y linux-firmware

Verify:

rocm-smi

The prerequisites list AGENTS.md carried#

Moved out of AGENTS.md on 2026-09-15 (owner decision D10), unedited except for this heading and the relative links. The RULES stayed there; this is the reference behind them.

  • nerdctl with BuildKit backend

  • QEMU/binfmt — and --install-service is NOT enough on its own. The unit it writes used After=/Wants= only, which order STARTUP and do not propagate a restart; combined with Type=oneshot + RemainAfterExit=yes, systemd holds the unit permanently satisfied while its effect — a registration inside the rootlesskit namespace — dies with every containerd restart. Measured 2026-08-27 on the dev host: unit last ran 2026-08-09, containerd restarted 2026-08-26, and the runtime stage then failed BOTH foreign arches with an empty BuildKit error. The template now sets PartOf=containerd.service (re-runs the unit on containerd restart) plus wait_for_namespace() + Restart=on-failure (afefdfc) to win the cold-boot race: After= orders only the unit start, so on a fresh boot the binfmt unit fired before containerd-rootless had unshared and written its child_pid, dying with cat: …/child_pid: No such file. wait_for_namespace() polls for the pid file and a joinable namespace before registering; Restart=on-failure is belt-and-braces. A daemon restart and a cold boot no longer silently strip foreign-arch emulation. On this rootless host the privileged tonistiigi/binfmt --install container does NOT work (wrong namespace); use:

    linux/scripts/setup-rootless-binfmt.sh --arches arm64,riscv64 --install-service

    --install-service installs a systemd –user unit so re-registration is automatic. Without registration, foreign-arch execs fail with exec format error — including wrong-arch NATIVE tool sub-builds inside “no-emulation” cross stages (the IREE tblgen failure mode).

  • Registry access (GHCR) for pushing intermediate and final images

  • Disk space: far more than the ~50GB+ this line used to say. The chain’s disk preflight (_chain_disk_preflight in build-cross-chain.sh) budgets 60 GB per arch for a run from base (40 GB from media on), plus ~120 GB for the runtime lane (CROSS_RUNTIME_LANE_GB), and warns when free space is below the sum; the runtime lane refuses to start without its share

  • Python 3 for digest resolution (registry-digest.py)

Phase B — Container runtime host config#

B1. Docker without sudo#

sudo usermod -aG docker $USER

Log out and back in — group membership is only picked up on a new session.

B2. Make the NVIDIA runtime the default#

Without this, every GPU container needs an explicit --runtime nvidia, and Compose files that omit it silently run on CPU.

sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json >/dev/null <<'DAEMON'
{
  "default-runtime": "nvidia",
  "runtimes": {
    "nvidia": {
      "path": "/usr/bin/nvidia-container-runtime",
      "runtimeArgs": []
    }
  }
}
DAEMON

sudo systemctl daemon-reload
sudo systemctl restart docker

Verify:

docker info | grep -i 'default runtime'

B2b. A GPU container on a Jetson, with rootless nerdctl#

Measured on a Jetson AGX Orin (L4T R39, nvidia-container-toolkit 1.20.1, 2026-09-22): the arm64 GPU wrapper ran PyTorch, the ONNX Runtime CUDA EP, OpenCV CUDA and an nvcc -arch=sm_87 kernel on the GPU with this call:

nerdctl --cdi-spec-dirs "$HOME/.config/cdi" run --rm \
  --runtime "$HOME/.local/bin/crun" \
  --annotation run.oci.keep_original_groups=1 \
  --device nvidia.com/gpu=all <image> \
  python -c "import torch; print(torch.cuda.get_device_name(0))"

Each flag answers one failure, in the order they appear:

Symptom

Cause

Fix

unresolvable CDI devices nvidia.com/gpu=all

The toolkit writes its spec to /var/run/cdi, which rootlesskit’s own /run hides from the daemon.

Copy it: mkdir -p ~/.config/cdi && cp /var/run/cdi/nvidia.yaml ~/.config/cdi/, and pass --cdi-spec-dirs. Copy again after a driver update, which regenerates the spec.

NvRmMemInitNvmap failed: error Permission denied, then No CUDA GPUs are available

/dev/nvmap and /dev/nvhost-* belong to the host group video. A rootless container process cannot carry a host group, and runc cannot keep it.

crun with the run.oci.keep_original_groups=1 annotation keeps the caller’s groups.

OCI runtime create failed: unknown version specified

Ubuntu’s crun (1.14) predates the OCI spec containerd 2.x writes.

Install the static release binary into ~/.local/bin and check its SHA256 against the release’s digest field (1.29.1 worked).

linux/jetson-webcam/ wraps this call around a USB-camera detection demo.

The official PyTorch cu130 wheels warn that they do not support compute capability 8.7 (Orin). A matmul and the elementwise kernels it needed still ran correctly, but a kernel that exists only as SASS for other architectures will fail. The libraries this repo builds (ORT, OpenCV, TVM) carry native sm_87 code.

B2c. A GPU container on an x86 desktop, with rootless nerdctl#

Set up 2026-10-01 on the build host: an RTX 2080, driver 595.58.03 (nvidia-driver-open, DKMS), Secure Boot on, nvidia-container-toolkit 1.20.1, nerdctl 2.4.0. The DeepStream GPU gate ran this way (linux-accelerator-images.md § The GPU run).

  1. Secure Boot: enroll the DKMS key before anything else. Until then the module never loads: the kernel logs Loading of module with unavailable key is rejected at every boot and nvidia-smi finds no driver. As root, mokutil --import /var/lib/shim-signed/mok/MOK.der, reboot, and enroll it in MOK Manager.

  2. The NVIDIA Container Toolkit, from NVIDIA’s apt repository; Ubuntu does not package it.

  3. Generate the CDI spec: sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml.

  4. Hand it to rootless nerdctl. /etc/cdi is not visible to rootless nerdctl, so copy the spec into the user’s own directory and name that directory in nerdctl’s config:

    mkdir -p ~/.config/cdi && cp /etc/cdi/nvidia.yaml ~/.config/cdi/
# ~/.config/nerdctl/nerdctl.toml
cdi_spec_dirs = ["/home/<user>/.config/cdi", "/etc/cdi", "/var/run/cdi"]

  5. Regenerate and copy again after every driver update. The spec lists the driver’s library files by version, so an old spec mounts files that no longer exist.

Verify:

nerdctl run --rm --device nvidia.com/gpu=all \
  ghcr.io/kataglyphis/kataglyphis_beschleuniger:latest-amd64 nvidia-smi

No --runtime or group annotation is needed here, unlike on the Jetson (B2b): the desktop device nodes are world-accessible, and the image’s runtime user (uid 1001) ran a DeepStream pipeline on the GPU as well.

B3. Cap container log growth#

A long build or a chatty service will fill the disk with JSON logs. Merge these keys into the same daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Restart Docker afterwards. Disk exhaustion mid-build surfaces as no space left on device deep inside a compile step — see Build resource monitoring.

B3b. Install or upgrade nerdctl-full#

nerdctl-full ships nerdctl together with the whole stack it drives — containerd, BuildKit (buildkitd + buildctl), runc, the CNI plugins, the snapshotters and the rootless helpers. They are version-matched, so on this host upgrading buildkitd means upgrading the bundle; there is no separate buildkit package to bump.

bash linux/host-config/install-nerdctl-full.sh                    # dry run: version delta + plan
NERDCTL_INSTALL_CONFIRM=1 bash linux/host-config/install-nerdctl-full.sh
bash linux/host-config/install-nerdctl-full.sh --rollback         # restore the backup

Knobs: NERDCTL_VERSION=x.y.z pins a release (default: latest), NERDCTL_PREFIX (default /usr/local, or $HOME/.local in rootless mode), NERDCTL_ROOTLESS=1|0 forces the prefix mode instead of auto-detecting it — see B3c, NERDCTL_BACKUP_DIR, NERDCTL_FORCE=1 (re-install the version already present), NERDCTL_SKIP_CACHE_CENSUS=1 (proceed without a cache baseline when buildkitd is down).

This host runs rootless AND rootful side by side, and that changes the procedure. The systemd --user units are the rootless stack the chain builds with; but a rootful containerd.service and buildkit.service also run, and they execute the same /usr/local/bin binaries plus unit files the bundle ships in /usr/local/lib/systemd/system. Extracting the bundle replaces all of that underneath them. Measured on this host: GNU tar 1.35 and cp -a both unlink-and-recreate rather than failing with ETXTBSY, so there is no error at all — the root daemons simply keep executing the now-deleted old inode, and because both units are Restart=always the real version jump then happens unattended at some later moment. The script therefore refuses unless you choose:

NERDCTL_INCLUDE_ROOTFUL=1 ...   # stop, upgrade and restart them too — no skew
NERDCTL_IGNORE_ROOTFUL=1  ...   # rootless only; accept the documented skew

A dry run always shows the plan and the choice — only the actual install is blocked.

What the script guarantees, and why each guard exists:

  • It refuses while a build runs. Extracting over live binaries mid-chain kills the run; chains here regularly last 10+ hours.

  • It verifies the release SHA256 against the published SHA256SUMS before touching anything, and refuses if the checksum is missing.

  • It backs up exactly the bin/ binaries the bundle ships, plus the /usr/local/lib/systemd/system unit files it rewrites, so --rollback puts nerdctl, buildkitd/buildctl, containerd and runc back and restores the rootful unit definitions — it stops the root services first, copies the units back and runs a system daemon-reload, so a rollback does not strand them on the new definitions. Know its scope: the bundle also ships libexec/ (CNI) and share/, and those stay at the newly installed version. That is harmless for the build path, but a rollback is not a full downgrade — for that, re-run with NERDCTL_VERSION=<previous>.

  • It counts BuildKit cache-mount records before and after. Compile caches (ccache/sccache/uv/cargo/cerbero) live in ~/.local/share/buildkit, not in /usr/local, so an upgrade must not change that number — see prune-safe.sh for why those records are the thing worth protecting.

  • It proves the stack came back up (nerdctl images, buildctl du, service active) instead of assuming, and tells you the rollback command if it did not.

Why this was done, and what it bought — BuildKit v0.31.0 introduced a daemon crash, “concurrent map iteration and map write” (moby/buildkit#6915), that reproduces under concurrent builds, which is exactly how this chain drives it with three arch lanes at once. The fix ships in BuildKit v0.31.2, bundled by nerdctl-full 2.3.5. Two caveats, so nobody upgrades expecting the wrong thing: it does not cure BKD1 — the buildkitd session rot (export hangs, no active session, lost layer blobs) has no upstream fix and is still cured by stopping the chain and restarting the service — and the parallel-build cache-miss fix (moby/buildkit#6954) landed in v0.32.0, which no nerdctl-full release ships yet.

Performed on this host, 2026-08-26 (the reference run, with what was actually proven afterwards rather than assumed):

before

after

nerdctl on disk

2.3.4

2.3.5

buildctl on disk

v0.31.1

v0.31.2

buildkitd daemon reports

v0.31.1

v0.31.2

containerd Server Version

v2.3.2

v2.3.3

cache-mount records

51

51 (unchanged)

buildkitd workers

—

1

Run as NERDCTL_INCLUDE_ROOTFUL=1 NERDCTL_INSTALL_CONFIRM=1, so the rootful pair was stopped and restarted with the rootless one. Verified afterwards: the rootful daemons came back on NEW pids (1982/1983 → 30591/30592, i.e. they are not stranded on the deleted old inodes), no daemon’s /proc/<pid>/exe reads (deleted), and both root units report NeedDaemonReload=no — the system daemon-reload took, so nothing flips over unattended later.

Two gotchas worth repeating, both hit during that run:

  • The env vars must be on one line with the bash call. Split across two lines they are set in the parent shell, never exported, and the script runs as a plain dry run that changes nothing — which is exactly what happened on the first attempt.

  • Installing into /usr/local needs sudo (that tree is root-owned), so that mode cannot be driven from a non-interactive session. This used to read as an unconditional rule — it is not. A $HOME/.local install needs no sudo at all and is the way to update a host whose sudo prompts for a password; see B3c.

Afterwards run linux/host-config/verify-host-config.sh and linux/scripts/preflight.sh before starting the next chain — the daemon restart is also the moment the staged buildkitd.toml gcpolicy takes effect.

B3c. Install rootless, into $HOME/.local (no sudo)#

On a host whose stack is rootless-only, install the bundle into the user’s own prefix. Nothing about this needs root, so it is the only mode that can update a machine where sudo prompts for a password. The distinction B3b’s last gotcha used to blur: rootless describes both how the daemons run and where the bundle lives.

NERDCTL_INSTALL_CONFIRM=1 bash linux/host-config/install-nerdctl-full.sh

The mode is auto-detected — no knob needed. The script reads the ExecStart of the live systemd --user units and installs into whatever prefix they already name, because those units are the only authority on what the host actually runs. NERDCTL_ROOTLESS=1 forces it on a host being converted for the first time; NERDCTL_ROOTLESS=0 forces the /usr/local path back.

The units are rewritten for you, and that is the load-bearing half. Extracting the bundle is not a prefix change on its own: the units keep the absolute ExecStart they were generated with, so without the rewrite the daemons keep launching the old prefix and the install moves nothing. The script repoints ~/.config/systemd/user/{containerd,buildkit}.service and prepends ${PREFIX}/bin to their Environment=PATH (containerd-rootless.sh resolves rootlesskit, slirp4netns and containerd through PATH), then reloads and restarts. The pre-image goes into ${NERDCTL_BACKUP_DIR}/systemd-user, so --rollback restores the units as well as the binaries.

The CNI plugins are the half nobody verifies. Rootless nerdctl resolves them under its own default — $HOME/.local/libexec/cni — not under /usr/local. A /usr/local install therefore leaves a rootless host with no plugins at all, and the symptom is a container-networking error that never mentions the install. Same class as B6: the build looks correctly configured and the networking simply is not there. The script now counts them and fails the run at zero.

Performed on this host, 2026-09-08 (aarch64, Snapdragon X, WSL2):

before

after

prefix

/usr/local

$HOME/.local

nerdctl / containerd / buildkit

2.3.5 / v2.3.3 / v0.31.2

unchanged

CNI plugins where nerdctl looks

0

18

sudo needed

yes

no

The binaries were byte-identical — sha256 matched across nerdctl, containerd, buildkitd, rootlesskit, runc and containerd-rootless.sh, because /usr/local already held the same bundle. This was a relocation, not a version change, which is why the script proves the daemons now execute from ${PREFIX}/bin instead of asserting that their reported version moved.

A drop-in ExecStart beats the unit file’s. linux/host-config/buildkit.service-override.conf therefore carries @NERDCTL_PREFIX@ rather than a literal path; apply-host-config.sh and verify-host-config.sh substitute the live prefix before installing or diffing. Without that, applying host config after a rootless install silently reverted buildkitd to /usr/local — invisible until a build failed.

NERDCTL_INCLUDE_ROOTFUL / NERDCTL_IGNORE_ROOTFUL are moot here: a $HOME/.local extraction cannot reach root daemons, so the script skips the rootful-coexistence fork entirely in this mode. Afterwards, run the same verify-host-config.sh + preflight.sh hand-off B3b ends with.

B4. Verify a nerdctl-full install#

The nerdctl-full tarball bundles containerd, BuildKit, runc, snapshotters and the rootless helpers. A partial extraction leaves some of them off PATH, and the symptom is a confusing failure much later — a build that cannot find buildctl, or rootless mode silently unavailable. Check all of them at once:

#!/usr/bin/env bash
binaries=(
    nerdctl containerd runc buildkitd buildctl
    containerd-stargz-grpc ctd-decoder
    slirp4netns bypass4netns
    fuse-overlayfs containerd-fuse-overlayfs-grpc
    tini buildg rootlesskit gomodjail
)

for cmd in "${binaries[@]}"; do
    if command -v "$cmd" >/dev/null 2>&1; then
        printf '\nOK   %s: ' "$cmd"
        "$cmd" --version 2>/dev/null || "$cmd" version 2>/dev/null \
            || echo "(no standard version flag)"
    else
        printf '\nMISS %s (not on PATH)\n' "$cmd"
    fi
done

command -v alone passes on a stale copy in the other prefix while the daemons run from the intended one, and the list above contains no CNI plugin at all — the exact gap that stayed invisible until 2026-09-08. Assert both:

prefix="$(sed -n 's/^ExecStart="\{0,1\}\([^"[:space:]]*\)\/bin\/.*/\1/p' \
           ~/.config/systemd/user/containerd.service | head -1)"
echo "units name: ${prefix}"
echo "PATH gives: $(command -v nerdctl)"        # must be ${prefix}/bin/nerdctl
cni="$(nerdctl info --format '{{.CNIPath}}')"
echo "CNI: $(find "${cni}" -maxdepth 1 -type f ! -name 'LICENSE' ! -name 'README.md' | wc -l) plugins in ${cni}"
for p in $(pgrep -f 'containerd$|buildkitd'); do readlink -f "/proc/$p/exe"; done

Expected on a healthy rootless host: PATH and the units agree, 18 CNI plugins, and every daemon exe under ${prefix}/bin.

B5. TLS handshake failures inside containers#

A container that cannot resolve names — TLS handshake timeout on an image pull, or apt failing to reach a mirror — is usually the host handing it an unreachable resolver.

# temporary: overwrite the resolver and restart the runtime
echo -e "nameserver 1.1.1.1\nnameserver 8.8.8.8" | sudo tee /etc/resolv.conf
sudo systemctl restart containerd

On a systemd-resolved host, /etc/resolv.conf is a managed symlink and the edit above will be reverted. Make it stick in /etc/systemd/resolved.conf instead:

[Resolve]
DNS=8.8.8.8 8.8.4.4
FallbackDNS=1.1.1.1

sudo systemctl restart systemd-resolved

B6. UFW silently breaks container networking#

If the firewall is enabled, containers can lose outbound networking entirely — image pulls hang, apt inside a build reaches nothing. UFW defaults to dropping forwarded traffic, which is exactly what a container bridge relies on.

sudo nano /etc/default/ufw
# DEFAULT_FORWARD_POLICY="ACCEPT"
sudo ufw reload

This is the Linux counterpart to the Windows lane’s CNI .conf requirement — in both cases the build looks correctly configured and the RUN steps simply have no network. Check this before debugging a mirror or a DNS resolver.

B7. Reclaiming disk without losing the compile caches#

Moved here from AGENTS.md on 2026-09-15: the rule lives there, the measurements live here.

linux/host-config/prune-safe.sh, never nerdctl builder prune -f. The -f prune deletes type==exec.cachemount records — ccache, sccache, uv, cargo, the LLVM source tree — together with the cheap-to-regenerate layer cache. Measured on this host: 207 GB of cachemount against 4.9 GB of layer cache, so the destructive half is 97% of what a -f prune takes, and one run paid about 1.5–2 h of cold LLVM rebuilds for a few GB.

prune-safe.sh prunes type==regular only, through buildctl --filter (nerdctl has no such flag), takes PRUNE_KEEP_GB and DRY_RUN, and proves cachemount survival by counting the records before and after. PRUNE_KEEP_GB is GB of layer cache: the script adds the cache mounts and local sources to the --keep-storage it passes, because that flag bounds the whole store (build-cache-tiers.md § 3.2.1).

A cache id with more than one record is listed after the cache mounts. With sharing=shared, BuildKit looks a cache id up without waiting: when the record it would reuse is locked, for instance mid-release by a parallel stage, it creates a second one. From then on it reuses the lowest record id it can lock (its bolt index order), and the other is touched only in the next such race. On 2026-10-01 six ids carried 52 GB of surplus that way (/uv-cache-riscv64 alone 25.9 GB). The Dockerfiles’ mount options were uniform; the race is BuildKit’s. PRUNE_DUP_CACHEMOUNTS=1 removes the surplus records, and refuses while any build holds a record. Through wave 4: 12 invocations, ~1 TB reclaimed, 0 cachemount losses.

Mid-run, use PRUNE_KEEP_GB>=100. Smaller budgets evict the in-flight lanes’ fresh layers and buy recompile churn; the kata-buildcache media slugs, rewritten every round, are the better lever when the store runs lean.

The lever ORDER mid-run:

  1. prune-safe.sh.

  2. nerdctl rmi of specific already-PUSHED tags — it refuses in-use ones, so this is safe.

  3. The regenerable cache-export directory ~/.cache/kata-buildcache. This is a DIFFERENT store, which prune-safe.sh cannot reach: it grew 62 → 110 GB inside one session and caused a disk emergency.

nerdctl system prune and nerdctl builder prune are not on this list at all. system prune is worse again mid-run, because “unused” means not-container-referenced, so it deletes TAGGED cross-stage locals too — on 2026-08-18 the cross-media-* tags vanished mid-run and only the registry-digest-pinned handoffs survived, via a re-pull that cost ~25 minutes.

B8. ghcr registry hygiene#

Also moved from AGENTS.md on 2026-09-15.

Two tools over ghcr-common.sh: ghcr-prune-package.sh deletes UNTAGGED versions, ghcr-delete-tags.sh deletes NAMED tags from an explicit list and never guesses what is legacy. Both dry-run by default (GHCR_PRUNE_CONFIRM=1 / GHCR_DELETE_TAGS_CONFIRM=1 perform), and both take the PAT from a docker login or GHCR_TOKEN.

The shared Accept header is a safety property, not tidiness. Listing the index media types makes a multi-arch tag resolve to its INDEX, so its children are visible; omitting them collapses the tag to a single platform manifest, and an incomplete keep-set is exactly how a prune tool deletes something it should not.

Never “delete all untagged” by hand. The per-arch entries of a multi-arch index are themselves untagged manifests, and a chain that is pushing creates untagged manifests seconds before it tags them. The keep-set is: every tag, plus every index CHILD resolved live (abort on any unresolvable tag), plus everything younger than KEEP_DAYS. The sanity gate checks digests, not tag names: every tag’s live digest must be readable and in the keep-set. Two tags on one image are fine. The tag-count gate it replaced refused one on 2026-09-27, during the :latest-cross rename.

Runs: 604 versions deleted on 2026-08-24; then on 2026-08-27, 23 untagged plus 47 tags with 0 failures — 81 → 34 tags, 204 → 134 versions — with :latest-cross verified 3/3 children HTTP 200 afterwards and a running build untouched. Six long-dangling legacy tags (android, compiler, latest, media, sdk, torch, whose children had 404’d since before either tool existed) were deleted the same day by operator decision. :latest is therefore gone and will not come back by itself: the cross lane’s orchestrators tag cross-* and latest-cross, and build-runtime-manifest.sh publishes only the cross manifest, so the native lane has no build path any more. (Superseded 2026-09-22: the cross manifest was renamed from :latest-cross to :latest, so :latest is live again as the default multi-arch index.)

Phase C — Performance mode#

Build hosts ship in a power-saving governor by default, which costs real wall-clock on long compiles.

C1. CPU frequency governor#

Check what is active:

cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

Expect one of powersave, schedutil, or performance.

Set it for the current boot — either form works:

echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
# ...or, if cpupower is installed:
sudo cpupower frequency-set -g performance

Make it survive a reboot:

sudo tee /etc/systemd/system/cpu-performance.service >/dev/null <<'UNIT'
[Unit]
Description=Set CPU to performance mode
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/bin/bash -c 'echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor'
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
UNIT

sudo systemctl daemon-reload
sudo systemctl enable --now cpu-performance.service

C2. AMD GPU performance level#

sudo rocm-smi --setperflevel high            # all devices
sudo rocm-smi --setperflevel high --device 0 # one device

Inspect:

rocm-smi --showclkfrq      # current clock frequencies
rocm-smi --showperflevel   # current performance level
rocm-smi --showprofile     # available performance profiles

C3. Stop the host suspending mid-build#

A machine that suspends during a multi-hour cross chain kills the run. Desktop installs enable idle suspend by default, so this is a required step on any host that builds unattended — not an optimization.

In /etc/systemd/logind.conf:

HandleSuspendKey=ignore
HandleHibernateKey=ignore
HandleLidSwitch=ignore
HandleLidSwitchExternalPower=ignore
HandleLidSwitchDocked=ignore

sudo systemctl restart systemd-logind

logind only covers keys and the lid. Block the sleep targets themselves as well, or something else can still trigger a suspend:

sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target

And belt-and-braces in /etc/systemd/sleep.conf:

[Sleep]
AllowSuspend=no
AllowHibernation=no
AllowSuspendThenHibernate=no
AllowHybridSleep=no

Verify:

systemctl status sleep.target      # should read: masked

On laptop-class hardware also check that TLP is not overriding the governor set in C1: systemctl status tlp.#

Phase D — Host toolchain#

D1. A newer GCC than the distro ships#

# only if add-apt-repository is missing:
sudo apt install --reinstall -y software-properties-common \
  python3-software-properties python3-launchpadlib

v=16   # the major you need; it must be newer than the distro's own
sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
sudo apt update
sudo apt install -y "gcc-${v}" "g++-${v}"

Register the alternatives and pick one:

sudo update-alternatives --install /usr/bin/gcc gcc "/usr/bin/gcc-${v}" 60
sudo update-alternatives --install /usr/bin/g++ g++ "/usr/bin/g++-${v}" 60
sudo update-alternatives --config gcc
sudo update-alternatives --config g++

Verify:

gcc --version && g++ --version

The image toolchain is pinned separately in linux/scripts/01-core/versions.env and built by linux/Dockerfile.toolchain. This section is only about the host compiler — for building a kernel module or a driver out-of-tree.

D2. Flutter on ARM hosts#

After installing the SDK on an ARM host, delete the cache folder inside the SDK directory. It ships x86_64 artifacts that are not replaced automatically, and the failure surfaces much later as an unrelated-looking tool crash.

D3. Multiple clang versions side by side#

Same mechanism as GCC above, for clang on the host (the images carry their own, source-built at LLVM_RELEASE):

v=20   # an installed clang-<major>
sudo update-alternatives --install /usr/bin/clang   clang   "/usr/bin/clang-${v}"   100
sudo update-alternatives --install /usr/bin/clang++ clang++ "/usr/bin/clang++-${v}" 100
sudo update-alternatives --config clang
sudo update-alternatives --config clang++

Set both, not just clang — a mismatched clang/clang++ pair produces link errors that read like a missing standard library.

D4. Python CLI tools that build from source on riscv64#

On amd64 and arm64, uv tool install <something> is one command because PyPI has wheels. On riscv64 there are no wheels for the packages these CLIs depend on, so uv silently switches to building them from source — and then needs a Rust toolchain, a C toolchain, OpenSSL headers and the CPython headers that nothing installed for you. The errors name a compiler or a linker, never the missing -dev package, which is why this section exists.

Ready-made for the Mistral Vibe CLI, including both workarounds below:

bash linux/host-config/install-mistral-vibe-riscv64.sh

The four things that make it work, in case you are installing something else:

1. The native dependencies. cryptography links against OpenSSL and libffi; a python*-dev is needed by anything with a C extension. Install the -dev packages, not just the openssl binary — pkg-config --modversion openssl is the check that actually matters:

sudo apt install -y build-essential pkg-config libffi-dev libssl-dev openssl \
  "python$(python3 -c 'import sys; print("%d.%d" % sys.version_info[:2])')-dev"
pkg-config --cflags --libs openssl   # empty output => libssl-dev is missing

2. The Rust target. This host’s Rust is set up for the native RVA23 profile, so cargo must be pointed at that triple rather than the generic riscv64gc-unknown-linux-gnu:

export CARGO_BUILD_TARGET=riscv64a23-unknown-linux-gnu
export RUSTFLAGS="-C target-cpu=native"
rustc -vV | grep '^host:'

3. Drop the dependencies that do not know RVA23 yet. textual-speedups pins a target-lexicon old enough that it rejects riscv64a23-unknown-linux-gnu before compiling anything. It is a pure optional accelerator for Textual’s rendering, so deleting the dependency line costs TUI performance and nothing else:

sed -i '/textual-speedups/d' pyproject.toml

That is also why the install goes through a git clone plus uv tool install . rather than uv tool install mistral-vibe — you need the source tree in order to edit pyproject.toml before the resolver reaches it.

4. Clear the cache between attempts. A source build that failed leaves a half-built entry behind, and the retry then fails identically for a cause you already fixed:

uv cache clean cryptography
uv cache clean textual-speedups

Finally, uv and the tools it installs live in ~/.local/bin, which is not on a login PATH by default — source "$HOME/.local/bin/env" (or uv tool update-shell) before deciding an install failed.

When a new dependency starts failing this way, the triage order is: is it a wheel or a source build (uv prints Building …), does it need a C library (-dev package) or Rust (target-lexicon / unknown-target errors), and is it optional? An optional accelerator gets deleted from pyproject.toml; a required one needs the target support upstream.

D5. The host tools preflight needs and never asked for#

preflight.sh fails on a correctly-set-up Linux host without these. uv, shellcheck and uidmap were undeclared until 2026-09-15. All but uidmap are user-scope and need no sudo. Two tools this list used to lead with, pwsh and pytest, are no longer preflight prerequisites.

PowerShell is no longer needed by preflight. The shared-config slug used to shell out to shared/config/Sync-SharedConfig.ps1, and without pwsh it failed with pwsh: command not found, which read exactly like config drift. It now runs the bash twin, shared/config/sync-shared-config.sh, because no hub Linux image ships pwsh. Install pwsh on a Linux host only for the Windows-side Pester suites and linter:

# NOTE the arch in the asset name: linux-x64 on amd64, linux-arm64 on an ARM
# host. This snippet said linux-x64 unconditionally until 2026-09-15, which on
# a Jetson installs an x86-64 pwsh that cannot exec.
curl -fsSL -o /tmp/ps.tar.gz \
  "https://github.com/PowerShell/PowerShell/releases/download/v7.6.5/powershell-7.6.5-linux-$(case "$(uname -m)" in aarch64|arm64) echo arm64 ;; *) echo x64 ;; esac).tar.gz"
mkdir -p ~/.local/powershell && tar -xzf /tmp/ps.tar.gz -C ~/.local/powershell
chmod +x ~/.local/powershell/pwsh && ln -sfn ~/.local/powershell/pwsh ~/.local/bin/pwsh
pwsh -NoProfile -Command '$PSVersionTable.PSVersion'

pytest is no longer needed by preflight either. The mutations slug needed it while about a quarter of the manifest drove python3 -m pytest over the linux/llm-stack/ suites; without it each of those reported baseline test already fails unmutated (vacuous bite). Those entries left with the benchmark lab (2026-09-12) and the NAS census (2026-09-15), and no manifest entry runs pytest today. The serving-shape tests in linux/llm-stack/tests/ still use it (llm-stack-serving.yml installs it in a venv). To run them on the host, whose python is PEP 668 externally-managed, the flag is not optional:

pip3 install --user --break-system-packages pytest

--user writes to ~/.local/lib/python3.*/site-packages and touches no system package; --break-system-packages only waives the PEP 668 guard.

uv, for three gate suites at once. test-cmake-format, test-python-lint-gate and the cmake-format / ruff gates all bootstrap their tool through uv/uvx. Without it all three fail with Required tool not found: uv, which names the missing tool but is easy to read as a broken gate rather than a missing prerequisite:

# installs uv + uvx to ~/.local/bin; the installer is downloaded to a file and
# checked against UV_INSTALL_SH_SHA256 (versions.env), never piped into sh
bash -c 'source linux/scripts/01-core/python_uv.sh && uv_ensure_installed'

shellcheck on PATH, for test-shellcheck-warnings. Note the asymmetry: the gate (lint-shell.sh) bootstraps its own SHA-pinned binary and needs nothing on PATH, but that suite asserts command -v shellcheck and fails without it. The bootstrap’s cache lives under $TMPDIR, so copy it somewhere durable:

cp "$(bash linux/scripts/lint-shell.sh --print-bin)" ~/.local/bin/shellcheck

uidmap — the one that does need sudo, and the only hard blocker for a rootless stack. newuidmap/newgidmap must be setuid-root from the distro; the nerdctl-full bundle ships everything else (slirp4netns, fuse-overlayfs) but cannot ship these. Without them containerd-rootless-setuptool.sh install fails and B3c cannot proceed:

sudo apt install -y uidmap
grep "^$(whoami):" /etc/subuid /etc/subgid   # ranges must exist (usually preallocated)

Phase E — Package sources and automatic updates#

E2. Unattended upgrades#

sudo apt install -y unattended-upgrades apt-listchanges
sudo dpkg-reconfigure --priority=low unattended-upgrades

Non-interactively, /etc/apt/apt.conf.d/20auto-upgrades:

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";

Which origins are allowed is set in /etc/apt/apt.conf.d/50unattended-upgrades. Security-only is the conservative default:

Unattended-Upgrade::Allowed-Origins {
        "${distro_id}:${distro_codename}-security";
        // "${distro_id}:${distro_codename}-updates";
};

Excluding a package is the important part on a build host. An unattended Docker upgrade is how a working container runtime breaks overnight — see the Jetson case in Linux Accelerator Images § Edge accelerators:

Unattended-Upgrade::Package-Blacklist {
    "docker-ce";
    "docker-ce-cli";
    "containerd.io";
};

Other options worth setting deliberately rather than by default:

Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "02:00";
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Mail "root";
Unattended-Upgrade::MailOnlyOnError "true";

Unattended-Upgrade::Mail needs a working MTA on the host, or the notification is silently dropped. msmtp is the least-effort option — it relays to an existing mailbox rather than running a mail server:

sudo apt install -y mailutils msmtp msmtp-mta

Configure it per-user in ~/.msmtprc — create this without sudo, and give it mode 600 or msmtp refuses to use it:

defaults
auth           on
tls            on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile        ~/.msmtp.log

account        mymail
host           smtp.example.com
port           587
from           builder@example.com
user           builder@example.com
password       <APP_PASSWORD>

account default : mymail

chmod 600 ~/.msmtprc
echo "test" | mail -s "Test from $(hostname)" you@example.com

For a provider with 2FA, this needs an app password, not the account password. The file holds that credential in plaintext, so it belongs in the build user’s home on the host, never in a repo.

Automatic reboots interrupt long builds — leave it off on a machine that runs the chain unattended.

If the host uses kernel livepatching (Canonical Livepatch) or a vendor kernel, read that vendor’s guidance first — unattended-upgrades may still install kernel packages without rebooting into them, so uname -r and the installed package drift apart silently.

Dry-run and check it is actually firing:

sudo unattended-upgrades --dry-run --debug
systemctl list-timers --all | grep apt      # apt-daily / apt-daily-upgrade
sudo journalctl -u unattended-upgrades --no-pager

Logs also land in /var/log/unattended-upgrades/unattended-upgrades.log.

E3. Turning automatic updates off entirely#

On a build host that must not change underneath a running chain:

sudo systemctl stop    unattended-upgrades.service
sudo systemctl disable unattended-upgrades.service
sudo systemctl mask    unattended-upgrades.service

sudo systemctl stop    apt-daily.timer apt-daily-upgrade.timer
sudo systemctl disable apt-daily.timer apt-daily-upgrade.timer
sudo systemctl mask    apt-daily.timer apt-daily-upgrade.timer

Masking, not just disabling — a dependency can re-enable a merely disabled unit.

E4. Pinning a package to a specific source or version#

Package-Blacklist in E2 stops a package being upgraded. Pinning is the other half: it forces which source a package comes from, which is what you want when the distro’s version is wrong for the host (an older driver branch, a vendor repo, a PPA).

sudo tee /etc/apt/preferences.d/my-pin >/dev/null <<'PIN'
Package: somepackage*
Pin: release o=LP-PPA-somevendor
Pin-Priority: 1001
PIN

A priority above 1000 permits a downgrade to that source, which is the point — anything at or below 1000 will not pull a package backwards. To exclude a source entirely, pin it negative:

sudo tee /etc/apt/preferences.d/no-distro-version >/dev/null <<'PIN'
Package: somepackage*
Pin: release o=Ubuntu
Pin-Priority: -1
PIN

Always verify which candidate actually won — pinning fails quietly:

sudo apt update
apt-cache policy somepackage

The trap: APT preferences require each field on its own line. A single-line echo "Package: x Pin: ... Pin-Priority: ..." writes a file APT parses as garbage and ignores, with no error. The symptom is a pin that simply does nothing, which reads as “pinning doesn’t work here”.

If the pinned package comes from a PPA and you want unattended-upgrades to keep it current, that origin has to be allowed explicitly:

echo 'Unattended-Upgrade::Allowed-Origins:: "LP-PPA-somevendor:${distro_codename}";' \
  | sudo tee /etc/apt/apt.conf.d/51unattended-upgrades-somevendor

Troubleshooting#

Reading the journal#

# errors in a date window
journalctl -p err --since "2025-04-21" --until "2025-04-22"

# search the last week of messages by text
journalctl --since "1 week ago" --grep "Error message"

# kernel ring buffer from the PREVIOUS boot — for anything that hard-locked
journalctl -k -b -1 | grep -i thermal

The -b -1 form is the one that matters after a machine wedges: the current boot’s log will not contain the cause.

nvidia-smi reports a device it cannot address#

Unable to determine the device handle for GPU1: 0000:28:00.0: Unknown Error

Seen on a dual-GPU host: the PSU could not supply both cards. It is a power fault, not a driver fault — reinstalling the driver will not help. Run one GPU, or move to a PSU that covers both.

Boot drops to a GRUB prompt#

A grub> shell means GRUB loaded its modules but could not find grub.cfg; grub rescue> means it could not even load normal.mod. Usual causes are partition renumbering or UUID changes after a resize or clone, an overwritten bootloader after installing another OS, or deleted /boot/grub/*.

Boot once from the prompt:

grub> ls                                  # find the partition holding /boot/grub
grub> set root=(hd0,msdos1)
grub> set prefix=(hd0,msdos1)/boot/grub
grub> insmod normal
grub> normal

Then fix it permanently. Identify the disk, not the partition:

lsblk -d -o NAME,SIZE,MODEL
sudo grub-install /dev/sda        # or /dev/nvme0n1
sudo update-grub

Check that /etc/default/grub has GRUB_TIMEOUT greater than zero, and verify the UEFI boot order lists the Linux entry before Windows:

sudo efibootmgr -v

Fast Boot can skip keypress detection and Secure Boot can block unsigned modules — disable both while diagnosing.

If the manual steps do not get you booting, Boot-Repair automates most of them and is worth trying before a reinstall:

sudo add-apt-repository -y ppa:yannubuntu/boot-repair
sudo apt update
sudo apt install -y boot-repair
boot-repair          # choose "Recommended repair"

Choosing the default boot entry#

# what entries exist
grep -E "^menuentry|^submenu" /boot/grub/grub.cfg

Set a permanent default in /etc/default/grub (GRUB_DEFAULT=0, or a name such as GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 6.8.0-45-generic"), then run sudo update-grub.

Remember the last selection instead:

sudo sed -i 's/^GRUB_DEFAULT=.*/GRUB_DEFAULT=saved/' /etc/default/grub
sudo sed -i 's/^#\?GRUB_SAVEDEFAULT=.*/GRUB_SAVEDEFAULT=true/' /etc/default/grub
sudo update-grub

Boot into one entry exactly once:

sudo grub-reboot "Windows Boot Manager (on /dev/sda2)"
sudo reboot

Or set the next boot via UEFI, which does not need a working GRUB config:

sudo efibootmgr -v
sudo efibootmgr -n 0003
sudo reboot

From a rescue system with the root filesystem mounted, grubenv can be written directly:

sudo grub-editenv /mnt/nvme0root/boot/grub/grubenv set next_entry=9
sudo grub-editenv /mnt/nvme0root/boot/grub/grubenv list

Finding what filled the disk#

no space left on device from a build step is a host problem, not a build problem. Locate the consumer before pruning anything:

du -sh .                                        # this directory
sudo du -ah --max-depth 2 . | sort -h           # what is under it
sudo du -ah / --max-depth=2 | sort -hr | head -20   # worst offenders, whole disk

On a WSL host, exclude the Windows mounts or the scan walks the entire C: drive and reports it as Linux usage:

sudo du -ah / --max-depth=2 --exclude='/mnt/*' | sort -hr | head -20

ncdu is the interactive form and usually faster to act on:

sudo apt install -y ncdu
ncdu /

Container images and the build cache are the usual answer. Reclaim those through linux/host-config/prune-safe.sh, which is written to spare the compile-cache mounts — a plain docker system prune throws away the ccache/sccache tiers and the next build pays for it.

/tmp is full during a build#

/tmp is often a tmpfs sized to a fraction of RAM, and a large build can exhaust it. Resize it for the current boot:

free -h                                  # check there is RAM to give
sudo mount -o remount,size=8G /tmp

To clear it without removing the directory or losing its permissions:

sudo find /tmp -mindepth 1 -maxdepth 1 -exec rm -rf {} +
sudo chmod 1777 /tmp                     # sticky bit must be restored

Never rm -rf /tmp itself — recreating it without mode 1777 breaks every service that writes there.

Contents