# Fresh Windows Host Bring-Up (BK lane + repo gates)

Goal: take a **fresh Windows 11 machine** to (1) a green
`.\windows\Build-Buildkit.ps1 -Gpu` on the preferred BuildKit/containerd lane
and (2) working repo gates (lint / tests / preflight) — using only this
checklist and the sections it links. Every step names its shell
(**[admin]** / **[non-admin]**) and ends with a verify command. Deep rationale
lives in [Windows Build Image](windows-builds.md); this page is the ordered
path through it, not a replacement.

> **Check yourself against the machine, not against this page.**
> `windows/scripts/host/Test-HostSetup.ps1` asserts most host-side claims below
> and prints a fix for each failure. It does not cover Phase B's gate tooling,
> C2's step-log env or buildkitd's `--debug` flag, and it checks two of C4's
> exclusions (2026-09-25); the driver refuses a launch without the step-log env
> itself. Run it **first** (to see
> what a fresh box still needs), **last** (to confirm bring-up), and after any
> host change:
>
> ```pwsh
> pwsh -File windows\scripts\host\Test-HostSetup.ps1 -SccacheEndpoint http://<lan-ip>:5000
> ```
>
> It needs no admin (Defender exclusions are reported UNKNOWN rather than
> skipped, so their absence cannot look like success) and exits 1 on any
> failure.
>
> **`pwsh`, never `powershell`, and that includes the ADMIN window.** Every
> script here carries `#requires -Version 7.0`; under Windows PowerShell 5.1
> they refuse with a `#requires` message and change nothing. That refusal is
> easy to miss: wrapped in a command that collects only the pipeline stream it
> looks exactly like "the script ran and did nothing" — it cost two round trips
> on 2026-08-08 against `Set-BuildkitdGcpolicy.ps1`, whose effect
> (`reservedSpace` in the deployed toml) then silently stayed at the old value.
> **Verify the effect, not the exit.**
>
> Do not hardcode a pwsh path. On this host pwsh 7 is the **MSIX/Store** build
> under `C:\Program Files\WindowsApps\Microsoft.PowerShell_<version>_x64__…\`,
> not the MSI at `C:\Program Files\PowerShell\7\` — the path carries the
> version and moves on every update. Resolve it with `(Get-Command pwsh).Source`
> (app-execution aliases occasionally fail to resolve under elevation; fall back
> to a recursive search of `WindowsApps` for `Microsoft.PowerShell_*`). **This page and that script are two views of one contract — change
> them together.** The reason it exists: until 2026-08-07 the CNI section here
> handed fresh hosts a `.conf` template that silently broke the entire nerdctl
> lane, and it read as authoritative for days. Prose cannot be executed; that
> is the whole failure mode this guards.

Phases:

- **A** — one-time host provisioning **[admin]**
- **B** — repo checkout + gate tooling **[non-admin]**
- **C** — build-service tuning **[admin]** (debug flags, log limit, GC policy, Defender, sccache/dufs, TIME_WAIT)
- **D** — per-boot / per-run checks
- **E** — first build + verification
- **R** — recovering after a Stevedore reinstall / repair **[admin]**

> **Fast path for Phase A5 + C: `Install-NewHost.ps1`.** Once the interactive
> steps are done (A1 Stevedore+reboot, A2 docker-users + a new shell, A3
> services, B0 SSH/B1 Git + repo), a single elevated run of
> `windows\scripts\host\Install-NewHost.ps1` does the *entire* scriptable half — CNI
> `.conflist` authored from the **live** `vEthernet (nat)` subnet (magic
> constants removed: it derives `network/prefix` + gateway at runtime), then
> `Set-ContainerdConfig.ps1` (debug flags, teardown env var, Defender
> exclusions, `.conf` derive), `Set-BuildkitdGcpolicy.ps1` + the step-log
> env var, the patched runhcs shim (built from the pinned hcsshim fork commit,
> § R1, if no `-ShimPath` is given — Go installed via scoop as needed — then
> deployed with `TEARDOWN_TIMEOUT=5m`), and dufs (scooped if missing, started
> serving the cache dir, ONLOGON task registered, machine
> `SCCACHE_WEBDAV_ENDPOINT` set to the host's LAN IP; § C5 has the
> session-independent task that replaces the ONLOGON one), and the host's
> `TcpTimedWaitDelay` of 30 s (§ C6; it takes effect after a reboot).
>
> ```pwsh
> pwsh -File windows\scripts\host\Install-NewHost.ps1 -ReportOnly   # plan first (safe, non-admin)
> pwsh -File windows\scripts\host\Install-NewHost.ps1               # admin - bring the host to green
> pwsh -File windows\scripts\host\Install-NewHost.ps1 -ShimPath C:\src\hcsshim\containerd-shim-runhcs-v1.exe
> ```
>
> It is idempotent and refuses to run while a build is live (unless `-Force`).
> The rest of A5/C below is what the script does by hand — read it to
> understand, run the script to execute.

> **⚠️ FIRST CHECK on any Windows host doing container builds.**
> Layer commit/finalize failing as `hcsshim::ActivateLayer 0x20` (BuildKit) or
> `mkdir \?\Volume{<GUID>}\C:.` (docker classic) is **an ENABLED AMD RDNA4
> dGPU locking freshly-written container layers** — not a Defender, disk,
> driver-install or OS-corruption problem, and it survives `-NoCache`, service
> restarts, a full store reset and a reboot (upstream docker/for-win#14977,
> A/B-proven on an RX 9070 XT host on 2026-08-10).
>
> Order of operations:
>
> 1. `pwsh -File windows\scripts\diagnostics\Test-BuildCopy.ps1 -Heavy` —
>    only a `-Heavy`-green verdict counts; the light lanes can pass while
>    RUN-layer finalize is broken.
> 2. RDNA4 present? Elevated `Set-Rdna4Gpu.ps1 -Disable` → re-probe
>    `-Heavy` → build → re-enable. `Build-Buildkit.ps1` enforces this via its
>    `Assert-NoActiveRdna4Gpu` preflight.
> 3. **After ANY red finalize, REBOOT before testing anything else** — a
>    failed finalize wedges hcs state and falsifies every later experiment.
>    This is why early A/B rounds contradicted each other.
>
> The full A/B history, the list of things falsified along the way, and the
> superseded 2026-08-09 "faulty Adrenaline install" verdict are owned by
> [`windows-build-lanes.md`](windows-build-lanes.md#rdna4-dgpu-layer-lock-ab-history-and-diagnostics).
> The Linux cross lane and every repo gate are unaffected.

---

## Phase A — One-time host provisioning [admin]

### A1. Install Stevedore, reboot

```pwsh
winget install stevedore     # or: choco install stevedore; custom dir: --custom="INSTALLDIR=D:\Stevedore"
```

**Reboot** afterwards — this enables the Windows Containers feature. Details
and the custom-INSTALLDIR path substitution: [Windows Build Image](windows-builds.md)
§ Prerequisites.

> Path quirk: Stevedore puts `docker.exe`, `buildctl.exe`, `nerdctl.exe` and
> `containerd.exe` under `<INSTALLDIR>\bin\`, but `dockerd.exe` and
> `buildkitd.exe` at the install root. When in doubt, trust the service
> ImagePath in the registry, not a hand-typed path.

Verify:

```pwsh
Get-Service stevedore, containerd, buildkitd          # all three must exist
& "$env:ProgramFiles\Stevedore\bin\docker.exe" version
```

### A2. docker-users group membership

The installer normally adds the installing user; the group grant is what makes
**non-admin** builds possible (dockerd's and buildkitd's named pipes are ACL'd
to `docker-users`; containerd's pipe stays admin-only by upstream design).

```pwsh
net localgroup docker-users $env:USERNAME /add        # only if missing
```

Log out/in after a group change. Verify (non-admin shell):

```pwsh
whoami /groups | Select-String docker-users           # must match
& "$env:ProgramFiles\Stevedore\bin\buildctl.exe" --addr npipe:////./pipe/buildkitd debug workers
```

### A3. Services running, reboot-safe

```pwsh
Get-Service stevedore, containerd, buildkitd | Set-Service -StartupType AutomaticDelayedStart
Start-Service stevedore, containerd, buildkitd
```

Verify: `Get-Service stevedore, containerd, buildkitd` → all `Running`.

### A4. Conditional dockerd fixes (only on the matching symptom)

- **`stevedore` service won't start / 1053 timeout** → a stale Docker Desktop
  `C:\ProgramData\docker\config\daemon.json` conflicts with the service's
  `--host` flags. Remove/rename it:
  [`windows-stevedore-and-docker.md`](windows-stevedore-and-docker.md) § Stevedore Setup Fixes, Fix 1.
- **`docker build` fails `runtime "com.docker.hcsshim.v1" binary not installed`**
  → apply Fix 2 (re-register with `--default-runtime=io.containerd.runhcs.v1`).
  Note: current Stevedore releases may not need this — the reference host runs
  WITHOUT the flag; apply it only when the symptom appears.

Verify: `& "$env:ProgramFiles\Stevedore\bin\docker.exe" info` succeeds.

### A5. CNI nat conf (required — RUN steps have NO network without it)

`nat.exe` already ships in `C:\Program Files\containerd\cni\bin`; only the
conf is missing on a fresh host. Install it as a **`.conflist`** (plugin-list
form), NOT a bare `.conf` — see the format note below — using the live
adapter's subnet (derived below, as in
[`windows-build-lanes.md`](windows-build-lanes.md) § Getting it going, step 2). The
`ipam.subnet`/`GW` values MUST match the live `vEthernet (nat)` adapter
(`ipconfig`), and dockerd restarts can silently re-create that network on a
new subnet (the driver's preflight fail-fasts on drift with the exact fix).

```javascript
// C:\Program Files\containerd\cni\conf\0-containerd-nat.conflist
{
    "cniVersion": "0.3.0",
    "name": "nat",
    "plugins": [
        {
            "type": "nat",
            "master": "Ethernet",
            "ipam": {
                "subnet": "<subnet of the vEthernet (nat) adapter>",   // DERIVE, don't copy: see below
                "routes": [ { "GW": "<the adapter's own IP>" } ]
            },
            "capabilities": { "portMappings": true, "dns": true }
        }
    ]
}
```

**No magic subnets — derive them.** Every example number shipped in these docs
(`172.31.32.0/20`, etc.) was a snapshot of ONE host and went stale; the only
correct values are the live adapter's. `Install-NewHost.ps1` derives them
automatically; to do it by hand:

```pwsh
$n = Get-NetIPAddress -AddressFamily IPv4 | Where-Object { $_.InterfaceAlias -eq 'vEthernet (nat)' }
$n.IPAddress, $n.PrefixLength     # adapter IP + prefix -> GW + subnet (e.g. 172.21.32.1 / 20 -> subnet 172.21.32.0/20)
```

The `ipam.subnet`/`GW` MUST match that live `vEthernet (nat)` adapter
(`ipconfig`), and dockerd restarts can silently re-create that network on a
new subnet (the driver's preflight fail-fasts on drift with the exact fix).

**Install BOTH forms — conf AND conflist (corrected 2026-08-07, same day, after
the conflist-only state cost a launched chain).** One network, two files: the
`.conflist` above, and the single-plugin `.conf` derived from it:

- **`0-containerd-nat.conflist`** is required by **nerdctl**; **`.conf`** is
  required by **buildkitd**. Both files must exist, and neither client fails in
  a way that names the missing one — nerdctl panics on an index, buildkitd
  silently gives RUN steps no network adapter at all. Never "convert" one into
  the other in place. The two failure signatures, the debug-log evidence and
  the day this cost a launched chain:
  [`windows-build-invariants.md`](windows-build-invariants.md#the-cni-nat-config-must-exist-as-both-conf-and-conflist).

The earlier claim here that "containerd and BuildKit read either form" was
wrong. The 2026-08-07 conversion fixed nerdctl and silently killed the buildctl
lane; it went unnoticed because no chain build ran in between. Keep both files,
and **edit only the `.conflist`**: `Set-ContainerdConfig.ps1` (§ C1) rewrites the
`.conf` from it whenever the two differ
([`windows-build-invariants.md`](windows-build-invariants.md#the-cni-conf-is-derived-from-the-conflist-not-hand-edited)),
then `Restart-Service buildkitd -Force` picks it up. `Build-Buildkit.ps1`
fail-fasts on a missing `.conf` (`Get-CniConfFormIssue`), and
`Test-HostSetup.ps1` fails when the two disagree on the subnet.

Verify:

```pwsh
Test-Path 'C:\Program Files\containerd\cni\conf\0-containerd-nat.conflist'  # True (nerdctl)
Test-Path 'C:\Program Files\containerd\cni\conf\0-containerd-nat.conf'      # True (buildkitd)
ipconfig | Select-String -Context 0,4 'vEthernet \(nat\)'                   # subnet matches the conf
```

The drift guard reads either name (`Get-CniNatSubnetDrift` checks `.conflist`
then `.conf`) — note its contract is "file absent = nothing to judge", so a
conf under any OTHER name turns the guard into a silent no-op.

**Verify nerdctl works too** (ADMIN shell — nerdctl opens containerd's pipe,
which is Administrator-only; `buildctl` stays non-admin because `buildkitd`
has `--group docker-users` and containerd has no equivalent). This is the
fastest confirmation that the conflist is correct, because nerdctl is the
component that is picky about it. It needs a built `bk-windows-base`, so on a
fresh host run it after the first base build (Phase E):

```pwsh
nerdctl --namespace buildkit run --rm --network nat `
    docker.io/local/kataglyphis:bk-windows-base cmd /c ipconfig
```

Expect an IPv4 address inside the conf's subnet with the nat gateway. Two
warnings are normal and harmless (`default network named "nat" does not have an
internal nerdctl ID`, and a `failed to remove hosts file` on exit). If instead
you get `panic: runtime error: index out of range [0] with length 0`, the config
is still in bare-`.conf` form.

If `nerdctl` is "not recognized": `C:\Program Files\Stevedore\bin` is on the
MACHINE path, so only shells opened AFTER the Stevedore install see it — open a
new window rather than editing `$env:Path`.

Full recipe set (interactive shell into an image, `nerdctl build`, the
`ENTRYPOINT` trap, zombie cleanup):
[`windows-build-lanes.md`](windows-build-lanes.md) § nerdctl lane.

---

## Phase B — Repo checkout + gate tooling [non-admin]

### B0. SSH agent — one passphrase, then never again

B1's clone URL is `git@github.com:`. A key that cannot *sign* therefore blocks
the clone and every push. (The one submodule, `third_party/DocumANTation`, is
fetched over HTTPS and needs no key.) Windows makes this easy to get wrong in a
way that looks exactly like a rejected key.

**Two OpenSSH installations, two agents, and PATH picks the wrong one.**

| Command | Resolves to | Talks to |
|---|---|---|
| `ssh-add` in **PowerShell** | `C:\Program Files\Git\usr\bin\ssh-add.exe` | MSYS agent via `$SSH_AUTH_SOCK` — normally not running |
| `ssh-add` in **Git Bash** | `/usr/bin/ssh-add` | the same MSYS agent |
| `C:\Windows\System32\OpenSSH\ssh-add.exe` | — | the **`ssh-agent` Windows service** — the only one that survives a reboot |

Git for Windows puts `usr\bin` ahead of `System32\OpenSSH` on PATH, so a bare
`ssh-add` reaches the MSYS agent **in both shells**, including PowerShell. Keys
loaded there die with the shell. The Windows service (`Automatic`, key store
DPAPI-encrypted under `HKCU\Software\OpenSSH\Agent\Keys`, readable only by
SYSTEM) reloads its keys on every boot. **Always use full paths here, never the
bare command name.**

One-time setup (**[non-admin]** except where marked), PowerShell:

```pwsh
$SSHADD = "$env:WINDIR\System32\OpenSSH\ssh-add.exe"
$SSHEXE = "$env:WINDIR\System32\OpenSSH\ssh.exe"

Set-Service ssh-agent -StartupType Automatic   # [admin]; usually already set
Start-Service ssh-agent                        # [admin]; no-op if Running

& $SSHADD "$env:USERPROFILE\.ssh\id_ed25519"   # asks for the passphrase ONCE, ever
& $SSHADD -l                                   # must list that key's fingerprint

git config --global core.sshCommand "C:/Windows/System32/OpenSSH/ssh.exe"
& $SSHEXE -T git@github.com                    # "Hi <user>! You've successfully authenticated"
```

The `core.sshCommand` line is the half that is easy to omit and without which
nothing else helps: Git Bash otherwise runs its own `/usr/bin/ssh`, which cannot
reach the Windows service at all (MSYS unix socket vs Windows named pipe). It
then falls back to reading the key file directly and needs a passphrase it has
no TTY to ask for.

**What failure looks like.** Two unrelated causes print the same misleading last
line, `Permission denied (publickey)`. Read the two lines above it:

```
debug1: Server accepts key: .../id_ed25519 ED25519 SHA256:…
debug1: No more authentication methods to try.
```

GitHub **accepted** the key; ssh could not produce the signature because the key
is passphrase-protected and the shell has no TTY. This is an agent problem, not
a key-registration problem — do **not** re-upload the key. Confirm what the
account actually holds, without needing any token scope:

```bash
curl -s https://github.com/<user>.keys | ssh-keygen -lf -   # fingerprints of every key on the account
ssh-keygen -lf ~/.ssh/id_ed25519.pub                        # the local one; look for it in that list
```

```
sign_and_send_pubkey: signing failed for ED25519 "…" from agent: agent refused operation
```

A stale or unusable entry in the Windows agent: the stored key can no longer
sign. Do **not** reach for `ssh-add -D` first.

> **`-D` can destroy a key that exists nowhere else.** The Windows agent keeps
> its own encrypted copy of the private key, and `ssh-add` has no export — a key
> whose file was deleted after it was added survives *only* inside the agent,
> and `-D` erases it with no way back. On the 2026-09-03 host the agent's single
> identity was exactly that: fingerprint `8UtaDNOh…`, with no matching file
> anywhere under the profile (`~/.ssh` held only an unrelated `id_ed25519`).
> Before purging anything, check that every fingerprint the agent lists still
> has a file behind it:
>
> ```pwsh
> & $SSHADD -l                                        # what the agent holds
> Get-ChildItem "$env:USERPROFILE\.ssh\*.pub" | ForEach-Object { ssh-keygen -lf $_ }
> ```
>
> A fingerprint in the first list with no counterpart in the second is
> file-less. Leave it alone.

The fix needs no deletion. Add your working key **alongside** the broken one —
ssh offers each agent identity in turn, so one that refuses no longer ends the
attempt once a usable key follows it:

```pwsh
$SSHADD = "$env:WINDIR\System32\OpenSSH\ssh-add.exe"   # repeated: full path, not the MSYS one
& $SSHADD "$env:USERPROFILE\.ssh\id_ed25519"
```

Purge only when every fingerprint in `ssh-add -l` maps to a `.pub` on disk, so
everything dropped can be re-added.

Both symptoms appeared on one host on 2026-09-03, and the first was initially
misdiagnosed as "the key is not on the GitHub account" — it was on it the whole
time. The `.keys` check above settles that question in one command and should be
the first thing run, before touching anything on GitHub.

**While SSH is still broken, work is not blocked.** If `gh auth status` shows an
HTTPS login, rewrite the URL per command and leave the config untouched:

```bash
git -c 'url.https://github.com/.insteadOf=git@github.com:' fetch --all
git -c 'url.https://github.com/.insteadOf=git@github.com:' push origin HEAD
```

Per command, not `--global`: a persistent rewrite silently masks the SSH problem
and then surprises anyone who clones with an unrewritten URL.

Verify: `& $SSHADD -l` lists your key; `git ls-remote git@github.com:<org>/<repo>`
succeeds from a **non-interactive** shell with no prompt; and both still hold
after a reboot with no re-entry of the passphrase.

### B1. Git for Windows + clone

Install Git for Windows (`winget install Git.Git`). The gates run under **Git
Bash** (`C:\Program Files\Git\bin\bash.exe`) — do NOT rely on a bare `bash` on
PATH, which on many hosts resolves to `System32\bash.exe` (WSL).

```pwsh
git clone --recurse-submodules git@github.com:Kataglyphis/ANTfrastructure.git
cd ANTfrastructure
git config core.hooksPath linux/host-config/git-hooks   # = `make hooks`; pre-commit runs a fast subset of CI's gates
git config core.longpaths true                          # deep vendored trees; host LongPathsEnabled=1 recommended too
```

- The `third_party/DocumANTation` submodule (Sphinx theme) is **optional for
  building images** — it is only needed for `cd docs && make html`. A plain
  clone builds the Windows chain fine.
- **Line endings:** `.gitattributes` pins the load-bearing files, so
  `core.autocrlf=true` (the Windows default, and what the reference host uses)
  is tolerated. The preflight `crlf-guard` check catches any `*.sh` that went
  CRLF in the working tree; after editing media `.psm1`/`.ps1` files confirm
  `git diff` shows only your change, not a whole-file EOL flip
  ([`windows-build-invariants.md`](windows-build-invariants.md#preserve-committed-line-endings-when-editing-a-copyd-psm1ps1)).

Verify: `git config core.hooksPath` prints `linux/host-config/git-hooks`;
`git status` is clean. The hook set and what each hook runs: AGENTS.md
§ Validation.

### B2. PowerShell 7 (pwsh)

Everything on this lane requires pwsh 7 (`#requires -Version 7.0` in every
script but two deliberate 5.1 exceptions; owner policy — see AGENTS.md).

```pwsh
winget install Microsoft.PowerShell
```

Verify: `pwsh -NoProfile -c '$PSVersionTable.PSVersion'` → 7.x.

### B3. PSScriptAnalyzer + Pester (the PowerShell gates)

CI parity pins (`.github/workflows/windows-x64.yml`): PSScriptAnalyzer
**1.25.0**, Pester **>= 5.7**. `Invoke-Tests.ps1` FAILS (never silently
skips) when Pester >= 5 is missing.

```pwsh
Install-Module PSScriptAnalyzer -RequiredVersion 1.25.0 -Scope CurrentUser -Force
Install-Module Pester -MinimumVersion 5.7 -Scope CurrentUser -Force -SkipPublisherCheck
```

Verify:

```pwsh
Get-Module -ListAvailable PSScriptAnalyzer, Pester | Select-Object Name, Version
```

### B4. Python for preflight/sync — dodge the Windows Store stub

On a fresh Win11, `python3` on PATH is the **Microsoft Store stub** (opens the
Store instead of running). `preflight.sh` provides the escape hatch
`PREFLIGHT_PYTHON`; the simplest working setup is uv:

```pwsh
scoop install uv          # or: winget install astral-sh.uv (install scoop first: https://scoop.sh)
uv python install 3.14    # puts python3.14.exe into %USERPROFILE%\.local\bin
```

Verify:

```pwsh
uv run --no-project python -V                              # Python 3.x
& "$env:USERPROFILE\.local\bin\python3.14.exe" -V          # if installed via uv python install
```

`PREFLIGHT_PYTHON` is **not** preflight-only, and it is not a `python3` alias
either: it may be a whole command line (`uv run --no-project python`), so every
consumer expands it unquoted. One file answers the question for the standalone
gates and the hooks — `linux/scripts/01-core/python-probe.sh`, whose
`preflight_python_require` probes the value with `-c pass` and fails with this
exact hint rather than dying inside a Python step. `preflight.sh` probes its own
candidate list instead (uv's `~/.local/bin/python3.14.exe` among them) and
exports what it finds to every check it runs. Export it once in the shell you
run gates from and these all honour it: `preflight.sh`, `run-lint-gates.sh`
(including the `--ratchets` gates), `lint-workflows.sh`, `lint-python.sh` and
the repo's own `pre-commit` / `pre-push` hooks. A bare `python3` left in any of
them is a bug, not a style choice — on this host it resolves to the Store stub.

### B5. shellcheck / hadolint / actionlint — nothing to install

The preflight lint gates **auto-bootstrap** these: a PATH copy is used when
present (shellcheck's only when it is the pinned version), otherwise the pinned
release from `versions.env` is downloaded once into a version-keyed cache dir
and SHA256-verified (`lint-shell.sh`, `lint-dockerfiles.sh`,
`lint-workflows.sh`). First run needs network; a failed bootstrap fails the
gate loudly (no silent skip).

### B6. Run all three gates

```pwsh
pwsh -File windows/scripts/Invoke-Lint.ps1 -FailOnAnalyzer # parse gate + PSScriptAnalyzer, findings fatal as in CI
pwsh -File windows/scripts/tests/Invoke-Tests.ps1          # harness + Pester suites
```

```bash
# Git Bash (not WSL bash):
PREFLIGHT_PYTHON="uv run --no-project python" bash linux/scripts/preflight.sh
```

Verify: all three exit 0. Preflight's `KNOWN_SLUGS` array is the single
source of the no-build gate list (don't enumerate it in docs — enumerations
drift; `tests/test-preflight-slugs.sh` keeps the array honest) — see
AGENTS.md § Validation.

---

## Phase C — Build-service tuning [admin]

Do C1–C3 in this order: C1 and C2 are plain registry edits, C3
(`Set-BuildkitdGcpolicy.ps1`) preserves the existing flags, adds
`--config`, and performs the one buildkitd restart. **Never restart buildkitd
while a build is solving.**

### C1. Permanent debug flags on containerd + buildkitd (owner policy)

Debug logging stays PERMANENTLY ON on build hosts, so the next snapshotter
incident carries its evidence immediately (owner decision 2026-08-04; if the
log grows huge, truncate it — never disable the flags). Recipe and rationale:
[`windows-build-lanes.md`](windows-build-lanes.md), "How to capture the debug
evidence again".

**Use the script — it is the source of truth for the containerd side:**

```pwsh
pwsh -File windows\scripts\host\Set-ContainerdConfig.ps1 -ReportOnly   # inspect, no admin needed
pwsh -File windows\scripts\host\Set-ContainerdConfig.ps1               # admin; restarts containerd
```

containerd runs with **no `config.toml`** here — every setting lives in the
service's `ImagePath`/`Environment` registry values, which is why it needs a
script to be reproducible at all (buildkitd has `buildkitd.toml` +
`Set-BuildkitdGcpolicy.ps1`; this is the missing counterpart, added
2026-08-07). It owns four things a fresh host must have: the debug flags
below, `CONTAINERD_SHIM_RUNHCS_V1_TEARDOWN_TIMEOUT` (the runhcs shim inherits
the SERVICE environment — a shim built from the upstream patch keeps its 30 s
defaults and silently reverts to the `ExportLayer 0x3` defect without it), the
load-bearing Defender exclusions on the three `C:\ProgramData` stores, and the
CNI `.conf`, which it derives from the `.conflist` (§ A5). Never run it while a
build is solving.

The manual equivalent, if you want to see what it does — set via registry,
because `sc.exe` quoting mangles these in PowerShell:

```pwsh
Set-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\containerd' -Name ImagePath `
  -Value '"C:\Program Files\Stevedore\bin\containerd.exe" --run-service --service-name containerd --log-level debug --log-file C:\ProgramData\containerd\containerd-debug.log'
# buildkitd: insert --debug before --run-service in ITS ImagePath (exe path may be
# the install ROOT, not bin\ — read the current value first and keep every flag):
Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\buildkitd' | Select-Object -Expand ImagePath
```

Restart containerd (`Restart-Service containerd -Force`, then
`Start-Service buildkitd, stevedore` — both depend on it). Verify (reference
host state):

```pwsh
(Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\containerd').ImagePath   # contains --log-level debug --log-file ...
(Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\buildkitd').ImagePath    # contains --debug
```

### C2. Disable buildkitd's per-step log limit (REQUIRED for compile stages)

Without this, heavy steps deadlock silently at the 2 MiB clip
([`windows-build-lanes.md`](windows-build-lanes.md) § Getting it going, step 4).
Since 2026-08-10 this is ENFORCED: `Build-Buildkit.ps1`'s
`Assert-BuildkitdStepLogEnv` preflight refuses to launch while the value is
missing (a Stevedore repair once wiped it silently); `-SkipStepLogGate` is
the documented one-launch escape when no admin is at hand:

```pwsh
Set-ItemProperty -Path HKLM:\SYSTEM\CurrentControlSet\Services\buildkitd `
  -Name Environment -Type MultiString `
  -Value @('BUILDKIT_STEP_LOG_MAX_SIZE=-1','BUILDKIT_STEP_LOG_MAX_SPEED=-1')
```

(The restart comes with C3.) Verify after C3:

```pwsh
(Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\buildkitd').Environment
```

### C3. Deploy the GC policy + build-history cap (REQUIRED — not optional tuning)

Skipping this cost this project twice: buildkitd's computed defaults evicted
the multi-hour VS Build Tools layer between two runs, and unlimited build
history pinned a 414 GB store at `Reclaimable: 0B`. The repo policy
(`windows/buildkitd.toml`: three GC tiers, `reservedSpace = 150GB`, plus the
`[history] maxAge/maxEntries` cap) is deployed by:

```pwsh
pwsh -File windows\scripts\host\Set-BuildkitdGcpolicy.ps1    # admin; refuses while a build runs
```

**Sizing on a different disk:** the toml's literals are sized for the
reference host's C:. Reproduce the INVARIANTS, not the numbers —
`reservedSpace` must exceed the fresh chain spine (~120–150 GB; rule of thumb
20–25 % of the disk, floor 150 GB), `maxUsedSpace` ≈ 1.5× reservedSpace,
`minFreeSpace` a trigger well above the ~25 GB danger band (the toml's main
tier uses 60 GB; 30 GB left GC no runway on 2026-08-07), `[history]` unchanged
everywhere. **Re-run the apply script
after every repo-side toml change** — deploy is a copy, nothing syncs
automatically (this host ran a stale copy for hours after the `[history]`
section landed).

Full story: [`windows-build-lanes.md`](windows-build-lanes.md#store-gc)
§ Store GC. Verify:

```pwsh
& "$env:ProgramFiles\Stevedore\bin\buildctl.exe" debug workers -v | Select-String 'Reserved space|Maximum used|Minimum free'
# WATCH THE UNITS AND THE LABELS -- both differ from the toml, and both have
# already fooled a reader (2026-08-26):
#   * the toml takes GiB, buildctl prints GB. reservedSpace = "150GB" in the file
#     shows up as `Reserved space: 161.0612736GB`. Grepping for "150GB" finds
#     NOTHING on a perfectly deployed policy.
#   * the keys are `reservedSpace`/`maxUsedSpace` in the toml but
#     `Reserved space:`/`Maximum used space:` in the output -- a grep for the
#     toml spelling matches only the tier-0 `Filters:` line.
# Expected (2026-08-26): rules 1 and 2 both at Reserved space 161.06GB, Maximum
# used space 697.93GB and 751.62GB. What FAILURE looks like: the computed
# defaults, maxUsedSpace ~100GB / minFreeSpace ~187GB, i.e. no policy loaded.
```

### C4. Windows Defender exclusions — LOAD-BEARING, not hygiene

These exclusions are load-bearing: without them the realtime scanner races
container churn and finalize/export operations flake constantly (the
hcs-temp sharing-violation family). They also tamed — but did NOT cure — the
`ExportLayer 0x3` heavy-churn finalize defect (TVM-class finalizes became
reliable with them; OpenCV-class still tripped it). The cure is the patched
runhcs shim (§ R1), which retired the warm/materialize workaround on
2026-08-06 — [`windows-build-lanes.md`](windows-build-lanes.md#defect-solved).
Skipping this step on a new machine makes builds flaky across the board.
`windows\scripts\host\Sync-DefenderExclusions.ps1` (elevated) owns the full
set: it prints what is excluded, adds what is missing — more paths and
processes than the core below — and prints the result. The core by hand (the
[`windows-build-lanes.md`](windows-build-lanes.md) § Getting it going step-3
list plus the process exclusions added 2026-08-05 after the hcs-temp
sharing-violation flake family):

```pwsh
Add-MpPreference -ExclusionPath "C:\ProgramData\containerd"
Add-MpPreference -ExclusionPath "C:\ProgramData\buildkitd"
Add-MpPreference -ExclusionPath "C:\ProgramData\nerdctl"
Add-MpPreference -ExclusionPath "C:\temp"
Add-MpPreference -ExclusionProcess "buildkitd.exe"
Add-MpPreference -ExclusionProcess "containerd.exe"
Add-MpPreference -ExclusionProcess "dockerd.exe"
```

Also exclude your repo checkout directory (antivirus file locks during
FetchContent/cargo are a documented failure family — see the note at the top
of [Windows Build Image](windows-builds.md)). Verify (admin — non-admin
reads print "N/A"):

```pwsh
Get-MpPreference | Select-Object -Expand ExclusionPath
Get-MpPreference | Select-Object -Expand ExclusionProcess
```

### C5. sccache / dufs WebDAV server (REQUIRED for the media stages)

This server is the compile cache (sccache WebDAV backend), and the driver also
stages files on it under `preseed/` (the Vulkan SDK installer and the media
memory budget) — **without it the BK media solves fail fast**
(`Assert-SccacheEndpoint`). The warm/materialize handoff tars (`bkhandoff/`)
that also rode it were retired with that workaround on 2026-08-06; only the
rollback scripts still use them. `Install-NewHost.ps1` automates all of it
(scoop install if missing, cache dir, start, ONLOGON task, machine-level
endpoint env with the host's LAN IP — never localhost). By hand (non-admin,
except the machine-env line):

```pwsh
scoop install dufs
mkdir C:\sccache-cache
dufs C:\sccache-cache -A -p 5000                 # keep it running

# endpoint = the host's LAN IP (ipconfig), NEVER localhost — containers must reach it:
[Environment]::SetEnvironmentVariable('SCCACHE_WEBDAV_ENDPOINT','http://<host-LAN-IP>:5000','Machine')  # admin; or pass -SccacheEndpoint per run
```

**dufs does NOT survive reboots.** Either make it logon-persistent once —

```pwsh
schtasks /Create /TN dufs-sccache /TR "\"%USERPROFILE%\scoop\shims\dufs.exe\" C:\sccache-cache -A -p 5000" /SC ONLOGON
```

— or, better, make it independent of any logon session. An ONLOGON task dies
with its session, and every sccache write then fails open, with an empty cache
as the only symptom (2026-08-11). Elevated,
`pwsh -File windows\scripts\host\Install-DufsService.ps1 -ServeDir C:\sccache-cache`
stops running dufs instances, unregisters every other dufs task and registers
`dufs-sccache-l2`: ONSTART, as SYSTEM, restarted on failure, port 5000 by
default. Without `-ServeDir` it serves `%USERPROFILE%\sccache-cache`, and it
refuses a directory that does not exist. Restarting dufs by hand after every
reboot works too; the Phase-D check below is what catches a forgotten one.
Verify:

```pwsh
(Invoke-WebRequest http://<host-LAN-IP>:5000 -Method Head -UseBasicParsing).StatusCode   # 200
[Environment]::GetEnvironmentVariable('SCCACHE_WEBDAV_ENDPOINT','Machine')               # set (new shells inherit)
```

### C6. `TcpTimedWaitDelay` = 30 s (REQUIRED for a chain without idle tails)

A container's teardown waits until every TIME_WAIT it left has expired, and Windows keeps one for
120 s by default. sccache's client closes first after every compile and every `--show-stats`, so
**every RUN that started an sccache server reached DONE about 120 s after its last output line**:
16 to 27 min per chain (`windows-refactor-backlog.md` #179). The stack is the host's. Nothing
inside the container shortens it, and the registry value inside the container changes nothing.
`Install-NewHost.ps1` sets the host value; by hand, elevated:

```pwsh
Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters TcpTimedWaitDelay 30 -Type DWord
Restart-Computer    # the stack reads the value at boot; until then the old 120 s stays live
```

30 is the lowest value Windows accepts. `Test-HostSetup.ps1` checks the registry value, which
cannot show whether the host has rebooted since. The chain shows it: measured in the from-base chain
of 2026-10-14 (`bk-20261014-cpu`), the sccache RUNs (ONNX Runtime 1.31, FFmpeg) reached DONE 29.9 s
and 30.2 s after their last line, against 114 to 124 s before. A tail of ~120 s after an sccache
RUN means the value is missing or the reboot is still due.

### C7. The shared S3 level (optional: compile hits from your other build hosts)

C5's dufs is this host's own cache, level 0. A second level, an S3 bucket every build host
shares, makes one host's compiles hits on the others
([`windows-build-resources.md` § Two cache levels](windows-build-resources.md#two-cache-levels)).
Needs the VPN to the bucket's server (its runbook:
[`linux/sccache-s3/README.md`](../linux/sccache-s3/README.md)). Non-admin:

```pwsh
pwsh -File windows\scripts\host\Set-SccacheS3Level.ps1 -Endpoint http://<s3-server-lan-ip>:3900 -AccessKeyId <key id>
pwsh -File windows\scripts\host\Test-HostSetup.ps1           # [PASS] sccache S3 level reachable
```

The script asks for the secret without echo and stores the five `KATAGLYPHIS_SCCACHE_S3_*`
user variables. `-ReportOnly` shows them and `-Clear` removes them.

- **Install s5cmd first** (`scoop install s5cmd`, 2.3.0, the runbook's pin). With it the script
  tries the key against the bucket and stores nothing the server refuses. Without it the key is
  stored unchecked.
- **Do not copy the command after the key.** That replaces the key in the clipboard, and the
  paste at the prompt is the command (2026-10-15). The script now refuses a secret with
  whitespace, so that mistake stops there.
- **Seeding.** The first host may seed the bucket with its own cache:
  `Copy-SccacheCacheToS3.ps1 -DryRun`, then without `-DryRun`.

---


## Phase D — Per-boot / per-run checks

Run these before every chain launch (30 seconds; each one has cost a real run):

1. **dufs up + endpoint reachable?**
   `(Invoke-WebRequest $env:SCCACHE_WEBDAV_ENDPOINT -Method Head -UseBasicParsing).StatusCode`
   → 200. A reboot kills a manually-started dufs (cost a run on 2026-08-04).
2. **Services running?** `Get-Service stevedore, containerd, buildkitd` → all
   `Running`.
3. **Disk headroom ≥ 40 GB free — now gated automatically.**
   `Build-Buildkit.ps1` refuses to start below the floor (`Assert-DiskHeadroom`;
   override with `-SkipHostChecks`, raise/lower with `-MinFreeGb`), and it
   also verifies the patched runhcs shim is still
   installed (`Assert-ShimPatch` — a Stevedore update silently restores the
   stock binary, and the first heavy media finalize then dies with
   `ExportLayer 0x3` hours into the run). Both were manual checks here until
   2026-08-07; on 2026-08-06 a chain ran 2.5 h and died of a disk shortage
   disguised as a missing `ninja`.
   The disk gate checks **every drive the build uses** — C: (the layer stores)
   plus the repo checkout's drive, which on a VHDX-backed checkout has its own
   exhaustion mode that a C:-only check cannot see
   ([`windows-build-lanes.md`](windows-build-lanes.md#vhdx-backed-checkouts) § VHDX-backed checkouts).
   The shim gate compares the live binary's **SHA256** against the hash
   `Publish-ShimPatch.ps1` recorded when it installed the patch (state file:
   `C:\ProgramData\kataglyphis\shim-patch.json`); a Stevedore update overwriting
   the binary is then an unambiguous hard failure. Hosts that have not yet
   re-run the deploy script fall back to the older file-size heuristic and get a
   warning telling them so — **run `Publish-ShimPatch.ps1` once to record the
   hash.** `-ReportOnly` prints the recorded hash and whether it still matches.
   `(Get-PSDrive C).Free / 1GB` — below ~25 GB free, hcsshim gets "weird"
   *before* an honest disk-full error (`ExportLayer 0x3`/`0x70`, spawn
   flakes). Reclaim levers, non-admin first: `buildctl prune --free-storage <MB>`
   (see the target trap below), then `buildctl prune-histories`,
   `buildctl prune --free-storage <MB>`, `docker image prune -f`; the full
   playbook is [`windows-build-lanes.md`](windows-build-lanes.md#store-gc)
   § Store GC. Two traps
   that make the levers look broken: `--free-storage` is a **minimum-free
   target**, so it deletes nothing once the disk is already above it (ask for
   more free space than the disk has to drain everything unpinned), and a
   **superseded lineage** of stage tags can pin whole duplicate copies of the
   base spine — the biggest single reclaim measured on this host (266 GB).
   **If the checkout or the store lives on a dynamically-expanding VHDX**,
   the store levers cannot see the biggest pool: dead blocks in the VHDX
   itself (270 GB physical for 16 GB of data on the reference host). Check
   it — the report costs nothing and stops nothing:

   ```pwsh
   pwsh -File windows\scripts\host\Optimize-HostVhdx.ps1 -VhdxPath <your.vhdx> -ReportOnly   # admin
   ```

   Without `-ReportOnly` it stops the build services, compacts and restores
   the disk — **admin, and never while a build solves.** Read the ReFS
   caveat in
   [`windows-build-lanes.md`](windows-build-lanes.md#vhdx-backed-checkouts)
   § VHDX-backed checkouts first: on ReFS guests compaction reclaims ~nothing,
   and the reclaim that does work is `Update-HostVhdx.ps1`, which rebuilds
   the disk around its live data. Run its `-CopyOnly` phase whenever you like
   — it touches nothing live — but the swap detaches the volume, so nothing
   may hold a handle on it: no shell sitting in the checkout, no editor, no
   agent session. Losing that volume mid-session is how a working session
   died on 2026-08-06.
4. **CNI subnet drift** — if dockerd/the host restarted since the last run,
   expect it; `Build-Buildkit.ps1`'s preflight fail-fasts with the exact fix
   (see Phase A5).
5. **Debug log size** — `C:\ProgramData\containerd\containerd-debug.log`
   grows unbounded; if it is huge, truncate (admin:
   `Clear-Content C:\ProgramData\containerd\containerd-debug.log`), never
   disable the flags.

---

## Phase E — First build + verification

### E1. Optional inputs, expectations

- **TensorRT (GPU lane, optional):** drop the NVIDIA EULA zip into
  `windows\downloads\` if you have one, with its hash in `versions.env`'s
  `TENSORRT_ZIP_SHA256` (a mismatch fails the build); **without a zip the build
  skips TensorRT gracefully** — the zip-less state is the normal state of the
  reference host's GPU lane
  ([`windows-builds.md`](windows-builds.md) § TensorRT setup (GPU lane, optional)).
  Do not wait on it.
- **Cost:** cold full chain ≈ 5–6 h (≈ 2.5 h in the media fan-out); hot
  rebuild of the whole BK chain ≈ 44 min. Parallelism is memory-bound —
  ~35–45 % average CPU during compiles is the expected signature, not a
  fault ([`windows-build-resources.md`](windows-build-resources.md) § Maximum resource
  envelope; 32 CPU / 39 GB is the verified max on the 64 GB reference host).
- **Logs:** per-stage under `out\windows-build-logs\`.
- **Every solve is direct:** the warm/materialize two-step that once routed
  around the `ExportLayer 0x3` snapshotter defect was retired on 2026-08-06,
  when the patched shim (§ R1) fixed the defect at its root. The
  `(warm solve, no output)` label a normal log still prints belongs to the
  smoke and publish gates, which export nothing on purpose.
- **Transient retries are automatic:** the driver retries the known flake
  families (ActivateLayer 0x20, hcs-temp finalize/export). On any OTHER weird
  hcsshim failure: check free disk first
  ([`failure-modes.md`](failure-modes.md#exportlayer-0x3-spawn-flakes-exportlayer-0x70--disk-exhaustion-in-costume)).

### E2. Launch (non-admin)

```pwsh
# endpoint via machine env (C5) or explicitly:
.\windows\Build-Buildkit.ps1 -Gpu                                    # full chain
.\windows\Build-Buildkit.ps1 -Gpu -SccacheEndpoint http://<LAN-IP>:5000
.\windows\Build-Buildkit.ps1 -Gpu -FinalTar out\bk-winamd64.tar      # + docker-loadable tar
```

The driver's preflight verifies buildkitd reachability and CNI subnet before
solving. Single-stage iteration: `-Stages toolchain` etc.

### E3. Verify the images exist (admin — containerd's pipe)

```pwsh
& "$env:ProgramFiles\Stevedore\bin\nerdctl.exe" --namespace buildkit images
# expect docker.io/local/kataglyphis:bk-winamd64 and the bk-windows-* stage tags
```

### E4. Smoke test — on the GPU lane ALWAYS pass -ExpectGpu

The chain already ran this suite: the driver's smoke gate runs it after every
`final` stage (amd64 since 2026-08-14), and `-Gpu` passes `-ExpectGpu` itself
and raises the floor to 198 passed. Run it by hand only to re-check an
existing image. Without `-ExpectGpu`, a broken CUDA env is silently SKIPPED
instead of failed ([`windows-builds.md`](windows-builds.md) § Smoke Testing).
Two routes:

```pwsh
# (a) via docker after a -FinalTar export (loads as local/kataglyphis:winamd64):
& "$env:ProgramFiles\Stevedore\bin\docker.exe" load -i out\bk-winamd64.tar
& "$env:ProgramFiles\Stevedore\bin\docker.exe" run --memory 48g --rm --isolation process `
  local/kataglyphis:winamd64 pwsh -File C:\temp\scripts\Test-Container.ps1 -ExpectGpu

# (b) from the containerd store (admin shell). No trailing command: nerdctl
# appends it to the ENTRYPOINT and exits 255. The entrypoint loads the VS
# environment the suite checks, then opens pwsh; run the suite at that prompt.
& "$env:ProgramFiles\Stevedore\bin\nerdctl.exe" --namespace buildkit run --rm -it `
  docker.io/local/kataglyphis:bk-winamd64
#   PS> & C:\temp\scripts\Test-Container.ps1 -ExpectGpu
```

Expected: the current baseline in [`windows-builds.md`](windows-builds.md)
§ Smoke Testing (2026-09-22, GPU lane: 236 passed / 0 failed / 0 skipped). A
higher count is growth, not a regression.

### E5. Publish (optional, non-admin)

```pwsh
& "$env:ProgramFiles\Stevedore\bin\docker.exe" login ghcr.io      # once, same shell
.\windows\Build-Buildkit.ps1 -PushRef ghcr.io/kataglyphis/kataglyphis_beschleuniger:winamd64
.\windows\Build-Buildkit.ps1 -Variant nvidia -PushRef ghcr.io/kataglyphis/kataglyphis_beschleuniger:winamd64-nvidia
.\windows\Build-Buildkit.ps1 -Variant rocm -PushRef ghcr.io/kataglyphis/kataglyphis_beschleuniger:winamd64-rocm
```

Each variant pushes only to its own tag (`-Gpu` is `-Variant nvidia`); a variant run that
names another lane's tag stops before it builds.

---

## Classic-lane fallback — there is none any more

`windows\build.ps1` (docker-classic, Hyper-V run+commit) was **retired on
2026-08-26** and **deleted on 2026-08-31** — `Build-Buildkit.ps1` is the only
driver. It could not build `base`, and its `merge` target could not pass the
smoke gate — the reasoning is in
[Windows build lanes](windows-build-lanes.md) § The classic lane was retired.

So if the BK lane is unavailable, the fix is to repair the BK host setup (Phases
A–C here, then § Phase R for a Stevedore reinstall), **not** to switch drivers.

## Phase R — Recovering after a Stevedore reinstall / repair [admin]

> Measured 2026-08-21, on a host where Stevedore had just been reinstalled.
> A reinstall is NOT a no-op for this repo: it silently takes out four things,
> and each of them fails LATER, in a way that does not name the reinstall.
> Work through this before starting a chain on a freshly repaired host.

| What the reinstall takes | How it fails later | Fix |
|---|---|---|
| The **patched runhcs shim** | `Assert-ShimPatch` refuses the BK lane at preflight (and without the gate: `hcsshim::ExportLayer 0x3` on a heavy media layer, after the compile is paid for) | Rebuild — see below. There is **no local rollback**: `.exe.orig` and every `.exe.bak-*` are stock too |
| The **buildkitd service `Environment`** | `Assert-BuildkitdStepLogEnv` refuses the BK lane; ungated, the 2 MiB step-log clip buries build verdicts | § C2, or the registry Multi-String + `Restart-Service buildkitd` |
| The **dufs `dufs-sccache-l2` task** *and* its `%USERPROFILE%\sccache-cache` serve directory | `Assert-SccacheEndpoint` fails the media stages at preflight, on an endpoint that never comes up | Re-create the serve directory FIRST, then § C5's `Install-DufsService.ps1 -NoPrompt`. Without the directory the script refuses to run (`serve dir not found`) and the endpoint stays dead |
| Nothing — but note the **containerd content store** can also be left inconsistent (e.g. by killing a `docker build` mid-pull) | `failed to resolve source metadata ... blob sha256:<config> ... blob not found` at `Dockerfile.base` | R2 below — and note a plain `pull` CANNOT fix it, see the warning there |
| — likewise the **windows snapshotter** | `failed to create scratch layer: failed to open ...\io.containerd.snapshotter.v1.windows\snapshots\<n>\blank.vhdx: The system cannot find the path specified` at the first `RUN` | R4 — this one has no surgical fix; reset the stores |

### R1. Rebuild + deploy the patched shim

`Publish-ShimPatch.ps1` installs a binary; it does not build one. Go is
required (`scoop install go`):

```powershell
git clone --filter=blob:none https://github.com/Kataglyphis/hcsshim.git D:\src\hcsshim
cd D:\src\hcsshim
git checkout feature/configurable-teardown-timeout   # the fork branch: upstream main + PR #2855
go build -o containerd-shim-runhcs-v1.exe .\cmd\containerd-shim-runhcs-v1
```

An existing clone after the branch was rebased: `git fetch origin`, then
`git checkout -B feature/configurable-teardown-timeout origin/feature/configurable-teardown-timeout`.
A plain `git pull` stops on the diverged history. `Install-NewHost.ps1` does
not follow the branch head: it builds the pinned commit (`$forkPin`,
`04a9bdaa` since 2026-10-14) and refuses a tree without the teardown knob.

**It skips the shim only while the live binary's SHA256 equals the record's**
(`Test-RecordedShimLive`, since 2026-10-06). Before that, the record file alone
made it skip, and that is exactly what a Stevedore update leaves behind. On
2026-10-06 a host whose 2026-09-21 update had put the stock shim (25 975 296 B)
back still reported `recorded patched shim already installed - skipping`, while
`Build-Buildkit.ps1`'s `Assert-ShimPatch` refused the same binary.

~15-21 s. The 2026-10-14 build of `04a9bdaa` (upstream `main` `bd97a054`
merged in) was **25 971 200 bytes** with Go 1.27.2 (the 2026-09-22 build of
`5e9df53c`: 25 890 304 with Go 1.27.1; the 2026-09-01 build of `19251429`:
25 998 336 with Go 1.27.0). Stock measured 23 279 616 until Stevedore's
2026-09-21 update and 25 975 296 after it — the exact size drifts with the Go release and the branch
head, which is why the gate keys on the SHA256 that `Publish-ShimPatch.ps1`
records at install time and treats the size table only as a fallback. Then,
elevated — **the `-ServiceEnvironment` part is NOT optional**: this build keeps
stock 30 s defaults until the knob is set, and the value must be a Go duration
string (`5m`, never a bare `300` — unparseable silently means stock):

```powershell
pwsh -File windows\scripts\host\Publish-ShimPatch.ps1 `
  -ShimPath D:\src\hcsshim\containerd-shim-runhcs-v1.exe `
  -ServiceEnvironment CONTAINERD_SHIM_RUNHCS_V1_TEARDOWN_TIMEOUT=5m
```

Verify with `-ReportOnly`: the gate hash must match the live binary and the
env line must show the knob. (The pre-2026-09-01 recipe — `microsoft/hcsshim`
@ `81e2e01` + `local-45min-deployed.patch` — is retired: under the
lost-notification regression the fixed 45 min taxed every RUN with 2841.2 s;
see `docs/failure-modes.md` § "Every RUN step reports DONE 2841.2s".)

**The other `Publish-ShimPatch.ps1` modes:**

| Goal | Command |
|---|---|
| See what is installed and which backups exist; changes nothing | `-ReportOnly` |
| The patched shim is already installed and only the gate bookkeeping is missing — records the hash in place, no services touched, no elevation | `-RecordCurrent` |
| Deploy an upstream-shaped build, which needs the env var to do anything | `-ShimPath C:\src\shim.exe -ServiceEnvironment CONTAINERD_SHIM_RUNHCS_V1_TEARDOWN_TIMEOUT=5m` (never the retired `45m`: it taxes every RUN) |
| Put the stock binary back — this CLEARS the gate record, because restoring stock must never teach the gate that stock is acceptable | `-Restore .orig` |

**Verifying the result is behavioural, not log-based.** The shim logs its
effective timeout at Debug level only, which does not reach containerd's log on
a default setup — a quiet log is *not* proof the deployment worked. Run a
filesystem-heavy container (the OpenCV canary in
[`windows-build-lanes.md`](windows-build-lanes.md#defect-solved)) and confirm it finalizes and exports
without `0x3`. A disposable canary snapshot is the right thing to risk; a chain
run is not.

**The timing tells a working shim apart from a broken one.** Count from a RUN's last output line to
its `DONE`. The fork build `04a9bdaa` with the 5m knob, deployed 2026-10-14, closed every RUN without an
sccache server 0.9 to 3.3 s after its last line in the from-base chain `bk-20261014-cpu`. Visual
Studio's install was among them, at 199.9 s of output and `DONE 202.3s`. About 180 s means the knob
is gone from the containerd service, about 450 s means the defect with the knob, and about 2841 s means
the 45 min build (`failure-modes.md`, the recurrence playbook). An sccache RUN adds the host's
TIME_WAIT on top, which is § C6, not the shim.

Run it from an admin shell, and **never while a build is running** — the service
stop kills every in-flight solve, and the binary cannot be replaced while a shim
process holds it. The script refuses on both unless `-Force` is passed.

### R2. Re-pull the pinned base into the buildkit namespace

The BK lane solves every stage with `--opt image-resolve-mode=local` — correct
for stage handoff, but it also forbids buildkit from fetching the PUBLIC pinned
base from mcr. On an empty or damaged content store the base stage therefore
cannot bootstrap itself. Elevated (containerd's pipe is admin-only), with the
digest the driver passes, `WINDOWS_BASE_DIGEST` in
`linux/scripts/01-core/versions.env` (`windows/Dockerfile.base`'s ARG default
mirrors it):

```powershell
& "$env:ProgramFiles\Stevedore\bin\nerdctl.exe" --namespace buildkit pull `
    mcr.microsoft.com/windows/servercore:ltsc2025@sha256:<WINDOWS_BASE_DIGEST>
```

> **A plain `pull` cannot repair a dangling content record.** Measured
> 2026-08-21: the pull printed `index ... already exists`,
> `manifest ... already exists`, `config ... already exists` and then died on
> the missing file — containerd trusts its metadata and never re-fetches what
> it believes it has. Delete the record first, then pull:
>
> ```powershell
> & "$env:ProgramFiles\Stevedore\bin\ctr.exe" --namespace buildkit content delete sha256:<config-digest>
> ```
>
> The config digest is the one named in the error. After the delete the same
> pull ran to `config-... complete` / `Completed pull`. Note also that on this
> host `ctr`/`nerdctl` reached containerd **non-elevated** — the "containerd's
> pipe is admin-only" rule is not absolute here; check rather than assume.

### R3. There is no classic lane to reach for — it was retired 2026-08-26

`build.ps1` was **deleted on 2026-08-31**, and the two reasons it went are
structural — restoring it from git history buys nothing. It could not bootstrap a
chain: the `windows/Dockerfile.*` use BuildKit-only `RUN --mount=type=bind` for
their script closures (twelve then, fifteen of seventeen on 2026-09-25) and
`build.ps1` never set `DOCKER_BUILDKIT`, so the legacy
builder died at `Dockerfile.base` step 8 with *"the --mount option requires
BuildKit"*. And even given a chain, its `merge` target skipped the OpenCV GStreamer
plugin, so the smoke gate that ends the run hard-failed on `cv2.CAP_GSTREAMER`. Use
`Build-Buildkit.ps1`; full reasoning in
[windows-build-lanes.md](windows-build-lanes.md) § The classic lane was retired.

### R4. When surgical repair is the wrong tool: reset the stores

Two independent inconsistencies in one store (a content blob gone while the
metadata claimed it, then the snapshotter missing `blank.vhdx`) mean the store
is not worth repairing piece by piece. `Reset-ContainerStores.ps1` (elevated)
RENAMES `C:\ProgramData\{containerd,buildkitd,Docker}` aside as `.bak-<stamp>`
— reversible, nothing is deleted — restarts the three services and re-applies
the GC policy.

Right after a reinstall this costs almost nothing: check first with
`buildctl du` (it was ~123 MB on 2026-08-21). The caveat is that dockerd's
store is parked too, so any non-repo docker images need re-pulling.

**Always follow a reset with R2's re-seed pull.** A reset leaves the content
store EMPTY, and `image-resolve-mode=local` means the BK lane cannot fetch the
public base itself — without the re-seed the very next build fails at
`Dockerfile.base` with the same `blob not found` class of error, which reads
like the reset did not work.

Afterwards the `.bak-<stamp>` husks are exactly what
`windows/scripts/host/Clear-DiskSpace.ps1` is allow-listed to reclaim.

---

## Appendix — host odds and ends

Small host-level settings that are not part of the ordered bring-up above but
come up often enough to be worth writing down.

### Defender performance mode (Dev Drive)

Complements the exclusions applied in the phases above. Performance mode defers
scanning of trusted Dev Drive content instead of excluding it outright, so you
keep protection while losing the build-time cost:

```powershell
Set-MpPreference -PerformanceModeStatus Enabled
```

See Microsoft's
[antivirus performance mode](https://learn.microsoft.com/en-us/defender-endpoint/microsoft-defender-endpoint-antivirus-performance-mode)
reference. A slow build with no obvious cause is worth checking here first —
realtime scanning of a build tree is the usual answer.

### Running scripts in a fresh shell

The repo's scripts are signed for nothing and the default policy blocks them.
Scope the bypass to the session rather than the machine:

```powershell
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
```

### Picking up a PATH change without logging out

After installing a toolchain, existing shells keep the old environment. Rebuild
`PATH` in place instead of opening a new session:

```powershell
$env:Path = [System.Environment]::GetEnvironmentVariable("Path", "Machine") + ";" +
            [System.Environment]::GetEnvironmentVariable("Path", "User")
```

### Updating Visual Studio non-interactively

```powershell
& 'C:\Program Files (x86)\Microsoft Visual Studio\Installer\setup.exe' `
    update --passive --norestart `
    --installPath 'C:\Program Files\Microsoft Visual Studio\18\Community'
```

`18` is VS 2026, the family's toolchain (`VISUAL_STUDIO_VERSION` in
`versions.env`); pass the path of the edition you have installed. Needs an
elevated shell. Pair with `winget upgrade --all --silent
--accept-source-agreements --accept-package-agreements --include-unknown` for
everything else.

To force a **specific** version of a pinned SDK — the usual case when a gate
requires an exact toolchain — winget needs both flags:

```powershell
winget install --id KhronosGroup.VulkanSDK --version <VULKAN_VERSION> --force   # the pin in linux/scripts/01-core/versions.env
```

### Opening a port for a service running in WSL

```powershell
New-NetFirewallRule -DisplayName "WSL Port 4173" -Direction Inbound `
    -Protocol TCP -LocalPort 4173 -Action Allow
```

### Dual-boot: boot into the other OS exactly once

The Windows counterpart to
[`grub-reboot`](linux-host-setup.md#choosing-the-default-boot-entry) — useful
when a host serves both lanes and you want the next boot only:

```bat
bcdedit /enum
bcdedit /bootsequence {identifier}
```

`/bootsequence` applies to the next boot only and does not change the default.

### Removing a driver that will not uninstall

Occasionally relevant on a GPU host where a stale driver blocks memory
integrity / core isolation:

```bat
pnputil /enum-drivers
pnputil /delete-driver oemXX.inf /uninstall /force
```

Elevated Command Prompt, then reboot. Identify `oemXX.inf` from the enum output
before deleting anything.