# Windows Container Build Performance Findings from making a large C++23 project build fast inside the Windows toolchain image. Everything here was **measured**, including the approaches that failed β€” the dead ends are documented deliberately so nobody spends an afternoon rediscovering them. Reference project: a ~690-object C++23 **modules** engine (Vulkan, CMake + ninja + clang-cl, Rust bridged in via corrosion/cxx), built on a Dev Drive host. Concrete script: `BeschleunigerBallett/scripts/windows/Build-Windows-Container.ps1`. ## Result | Approach | Outcome | | --- | --- | | **Reusable container + tar-pipe** | βœ… **9.6 s** ninja / **44 s** wall, no-change | | Reusable container + bind mount | βœ… works; 32.7 s ninja / 159 s wall β€” slower *here* | | Streaming the build tree in and out | 🟑 ~230 s (worked, but moved ~17 GB per build) | | sccache with a persistent named volume | ❌ 0.00 % hit rate, 0 bytes stored | | Named volume mounted as the build directory | ❌ CMake cannot configure inside it | | Fresh container per build (starting point) | 352–484 s | Cold builds, for reference: 327.9 s (tar-pipe, fresh container) and 318.1 s (bind mount). The transport barely matters cold β€” it is the *incremental* case where the difference is decisive. ## Transports: how to set up both Getting sources into the container and artifacts back out has two working implementations. **Both are supported and worth keeping** β€” which one wins depends on the host, and the answer here was the opposite of what we expected. Keep them behind a switch and measure rather than assume. ### Which one should I use? | | tar-pipe | bind mount | | --- | --- | --- | | Host setup needed | none | elevated `fsutil` + reboot (Dev Drive only) | | Where the build tree lives | inside the container | on the host volume | | Incremental cost | copies sources each build (~35 s here) | none | | Per-file I/O cost | container-local, fast | crosses the filter, slow | | **Measured here (no-change)** | **9.6 s ninja / 44 s wall** | 32.7 s ninja / 159 s wall | | Survives `docker rm` | ❌ tree is lost | βœ… tree is on the host | Rules of thumb: - **Large build tree, incremental edit-build loop β†’ tar-pipe.** You pay a fixed bulk copy once per build instead of filtered I/O on every one of ~1000 targets. This is why it wins on the reference project. - **Small tree, or a host where the transport dominates β†’ bind mount.** The fixed copy cost stops being amortised and removing it wins. - **Non-Dev-Drive volume β†’ measure again.** The penalty below is specific to a Dev Drive with filters allow-listed; an ordinary NTFS volume does not behave the same way. - **You want the build tree to survive the container β†’ bind mount.** ### Transport A β€” tar-pipe (no host setup) Works everywhere, including Dev Drive hosts with default settings, because it never asks the filesystem to attach anything. ```pwsh # in: sources only, excluding .git and build trees tar -cf - --exclude .git -C $repoRoot . | docker exec -i $c tar -xf - -C C:\ws # out: only what the host runs, selected by EXCLUSION (tar does not expand globs) docker exec $c tar -cf - --exclude "*/CMakeFiles" --exclude "*.obj" -C C:\ws build-x | tar -xf - -C $repoRoot ``` Requirements: none. Caveats: the deletion hazard (safety rail 2 below), and `tar` aborting a whole transfer on one over-long path (see Gotchas). **`--exclude` matches at every depth.** A pattern that names a root directory (`build`) also drops every directory of that name anywhere in the tree β€” including `third_party/ANTinfrastructure/windows/scripts/build`, which consumers need. No pattern form anchors: `./build`, `[.]/build` and `build/` all behave the same. A caller that must drop such a root directory passes `-InboundItems` (the top-level entries to stream, computed by omitting it) instead of a pattern. ### Transport B β€” bind mount (Dev Drive needs setup) A Dev Drive refuses bind mounts by default β€” the minifilter cannot attach ("Der Dateisystem-Minifilter kann nicht an das Entwicklervolume angefΓΌgt werden"). Allow-list the two filters, **elevated**: ```pwsh fsutil devdrv setFiltersAllowed /volume D: "bindFlt,wcifs" ``` Four things that cost time here: 1. **The filter list is ONE quoted argument.** `bindFlt, wcifs` unquoted is parsed as two arguments and fails with a bare syntax dump. 2. **It needs a reboot.** The setting persists immediately, but the filters only attach when the volume is dismounted. You will see `Fehler 5: Zugriff verweigert` β€” that is the *dismount* failing because the volume is in use, not the setting failing. `/f` force-dismounts instead; do not use it on the volume holding your repo. 3. **`query` shows "allowed" before "attached".** After the reboot the filters are listed as allowed but still not attached β€” they attach *on demand*, the first time something actually requests a bind mount. Do not read that as failure; probe instead. 4. **Omitting `/volume` sets it machine-wide.** Scope it to the volume you mean. Verify β€” allow-list, then an actual mount: ```pwsh fsutil devdrv query D: # expect: bindFlt, wcifs under "allowed" docker run --rm --isolation process ` --mount "type=bind,source=$repoRoot,target=C:\ws" ` --entrypoint cmd $image /c "dir C:\ws\CMakePresets.json" ``` Revert (a Dev Drive is fast *because* filters do not attach β€” allow-listing them slows general I/O on that volume, not just container builds): ```pwsh fsutil devdrv clearFiltersAllowed /volume D: # + reboot ``` #### Mount both transports at the SAME in-container path If you support both β€” and you should β€” mount at the path the tar-pipe already uses. CMake bakes absolute paths into `CMakeCache.txt` and refuses to reuse a cache generated elsewhere: ``` CMake Error: The source "C:/ws-mnt/CMakeLists.txt" does not match the source "C:/ws/CMakeLists.txt" used to generate cache. ``` Without a shared path, every switch between transports forces a cold rebuild and the two are not really interchangeable. One path (here `C:\ws`) fixes it. The path must still be **absent from the image**: mounting over a directory baked in (`C:\workspace`, until the image after 2026-09-26) fails at `CreateComputeSystem` when the host OS build differs from the image base build. Verify with `docker run --rm --entrypoint cmd $image /c "if exist C:\ws (echo BAKED IN)"`. #### Why the bind mount lost here With a bind mount the build tree lives on the Dev Drive, so every ninja stat and every object write crosses the `bindFlt` filter from inside the container. Copying sources in bulk once is cheaper than paying filtered I/O across ~1000 targets β€” **ninja alone tripled**, 9.6 s β†’ 32.7 s on an identical tree. Repeated to rule out a first-run artifact. Cold builds are near-identical (327.9 s vs 318.1 s), which is the tell: the penalty is per-file-operation, not per-byte, so it only shows up once compilation stops dominating. ## What works: reuse one container Create a **single long-lived container** instead of one per build. The build tree never leaves it, so ninja's dependency graph, C++23 module BMIs and object files are all still there next time. ```pwsh # reuse if running; start if stopped; recreate if the image changed $state = docker inspect -f '{{.State.Running}}|{{.Image}}' $name 2>$null ``` - **Inbound:** stream sources only. `tar` preserves mtimes, so ninja rebuilds exactly what changed. - **Outbound:** stream back only what the host needs to *run* β€” `*.exe`, `*.dll`, `*.pdb`, `compile_commands.json`, logs. Copying the whole build tree back is unnecessary once the container keeps its own copy. ### Safety rails this needs Reuse trades isolation for speed, so guard it: 1. **Compare the container's image ID against the referenced image** and recreate on mismatch β€” otherwise a rebuilt toolchain image is silently ignored and you keep building against the old one. 2. **Provide an explicit reset switch** (`-FreshContainer`, or `docker rm -f `). Sources are overwritten in place and never pruned, so a file **deleted** on the host still exists inside the container. This is not theoretical - measured 2026-07-19. A probe test file was added, built and observed to run; the file was then deleted on the host and the project rebuilt. **The test still ran**, and the `.cpp` was still present inside the container. Consequences worth internalising: tests keep passing against deleted code, and a deletion that breaks the build passes locally and fails in CI, where nothing is reused. If you want reuse without this hazard, prune the source tree inside the container before streaming (keeping the build directory and logs) rather than extracting over it. Sources re-stream in seconds; only the build tree is worth preserving. Get the exclusion pattern right first - a wrong one deletes the build tree on every build and silently undoes the whole optimisation. `Invoke-ContainerBuild` runs this prune on every reuse (`Remove-StaleContainerSources`, below). It removes top-level directories only, so a deleted top-level *file* still needs `-FreshContainer`. 3. Keep the build root: if the build script wipes its build directory before configuring, gate that behaviour behind an env var (this project uses `KATAGLYPHIS_KEEP_BUILD_ROOT`) or reuse buys nothing. 4. **Verify the artifacts exist and arrived.** A zero exit code proves neither. Both halves failed silently in the reference project: - A build was **cut off before linking**, reported success, and produced no test executable at all. The log simply stopped partway through the step count (`[982/1022]`) with no summary and no error. - The outbound `tar` selected files by **glob** β€” which `tar` does not expand. It copied nothing, reported success, and looked correct only because stale artifacts from an earlier build were already on the host. Select by `--exclude` instead. Hours went into diagnosing a "transport bug" that was really a truncated build, because a green build was taken as evidence that binaries existed. After the transfer, list the executables **inside** the container and assert each one reached the host; fail the build if the container produced none, or if any did not arrive. Check **existence, not timestamps**: on a no-change incremental build the linker does not run, so the executables are legitimately older than the current run. A freshness check would false-fail exactly when the cache is working best. ## What does not work ### sccache's cache directory on a Windows container volume An sccache server pointed at `SCCACHE_DIR` on a mounted volume reports a healthy server, compiles fine, and stores **nothing**: measured 2026-07-20, **66 write errors out of 66 misses** β€” every single write failing. The volume was the cause. Running the server by hand with `SCCACHE_LOG=trace` showed every `DiskCache::put_raw` dying with **os error 3** ("The system cannot find the path specified") while the mount was in place. PowerShell **in the same container** wrote the same paths β€” including `\\?\`-prefixed ones β€” without error, so it is not a permissions or path-shape problem from the container's side; it is specific to how the sccache server writes (tempfile + rename) on a `wcifs` volume. Pointing `SCCACHE_DIR` at a container-local directory made the very next compile pair go **miss β†’ HIT with zero write errors**, which is the A/B that settles it. The cache dying with the container is **fine**: builds run in the persistent reusable container (above), so the cache lives exactly as long as the thing that uses it β€” and a volume that took 100% write errors persisted nothing anyway. **Why this stayed invisible:** without `SCCACHE_ERROR_LOG` + `SCCACHE_LOG`, sccache reports the write-error count and discards the reason, so "0 bytes cached" reads as a hit-rate problem rather than a volume one. The reusable-library helper (`Get-SccacheContainerEnv`, `windows/scripts/modules/WindowsContainerBuild.Reuse.psm1`) sets both and defaults `SCCACHE_DIR` to `C:\sccache-local` for this reason. `SCCACHE_ERROR_LOG` must NOT live under `SCCACHE_DIR`: the server opens the log before it creates the cache directory and dies when the log's parent is missing, which surfaces three indirections later as `Timed out waiting for server startup` from every wrapped tool β€” with `RUSTC_WRAPPER=sccache` that poisons even `cargo tree`, reported by corrosion as a missing `cxxbridge-cmd`. Not to be confused with the module-build section below, which is about what sccache caches, not whether it can write. This one is the disk backend never receiving a byte. ### sccache on a C++23 modules build Rewritten 2026-09-29 from real builds of BeschleunigerBallett and AccelerANTgine in `:winamd64`. This section used to say sccache "caches nothing" here and is "harmless to leave wired up". The first claim was wrong. The second was the dangerous one: - **The released sccache (0.18.0, the image's) with clang-cl** caches the importers and every non-module unit (Ballett, debug + release, warm: 1170 of 1474 requests hit, 470 MB stored). It refuses the interface units, because it reads `-x c++-module` as a second input ("multiple input files"). And it hashes an importer **without** the BMIs it imports: after an interface edit a warm cache hands back the old object. Reproduced on both projects (Ballett's test ran the old constant, AccelerANTgine's exe printed the old version). In a reused container whose cache stays warm across edits, empty `SCCACHE_DIR` after an interface change, or configure through `Invoke-CmakeConfigureAndBuild -DisableSccache`, which since 2026-09-29 also overrides a preset's `COMPILER_CACHE`. - **With mozilla/sccache#2876** (clang-cl's module flags), interface units cache with their BMI and importers hash every BMI they import, so an interface edit misses exactly its importers. Ballett's warm clean-tree rebuild fell from 409.7 s to 187.1 s. Not in the image until that PR ships. - **On Linux**, clang's CMake module maps already cache correctly with the released sccache ([`build-cache-tiers.md` Β§ Keeping the compiler cache across CI runs](build-cache-tiers.md#keeping-the-compiler-cache-across-ci-runs)). GCC's `-fmodules-ts` units are cached by neither sccache nor ccache. The numbers this section used to show (907 requests, 0 bytes stored) match no measured lane. They most likely came from the volume write failure above. ### A named volume as the build directory Mounting a Docker named volume at the build directory looks like the natural fix. CMake cannot configure inside one: ``` CMakeTestCXXCompiler.cmake:71 (message) ninja: error: loading 'build.ninja': The system cannot find the file specified. ``` Reproduces with a **freshly created** volume, so it is not stale state. Windows container volumes are filter-driver backed and do not behave like an ordinary directory for the operations CMake's compiler test performs. ## The image does not fit on C:, and that is the default On a GitHub-hosted `windows-2025` runner, Docker keeps its data-root under the **system** drive (`C:\ProgramData\Docker`). That drive is the small one. Measured 2026-08-11 at job start: ``` DriveLetter FreeGB SizeGB FileSystemLabel C 33.00 149.40 Windows D 146.60 150.00 Temporary Storage ``` The `winamd64` image needs **~54 GB** to import. It does not fit in 33 GB, and the failure is expensive rather than obvious: `docker pull` grinds for tens of minutes and then dies with `hcsshim::ImportLayer ... not enough space on the disk (0x70)`. The historical workaround was `cleanup-disk-space`, which deletes Visual Studio and the tool caches to claw C: up to ~71 GB. That works, costs minutes, is destructive β€” and leaves a 150 GB drive sitting at 146.6 GB free. **Point the data-root at the big drive instead**, with the [`set-docker-data-root`](../.github/actions/README.md) action, *before* the pull: ```yaml - name: 'Put the Docker data-root on D:' uses: Kataglyphis/ANTfrastructure/.github/actions/set-docker-data-root@develop with: data-root: 'D:\docker' ``` [`prepare-windows-container-host`](../.github/actions/README.md#prepare-windows-container-host) already runs this step before its own pull (input `data-root`, default `D:\docker`; an empty value skips it), and so does every lane built on the reusable [`container-ci-windows.yml`](../.github/workflows/container-ci-windows.yml). Call the action yourself only in a job that pulls without that prologue. Measured on the same job, across the whole run: | Point in the job | C: free | D: free | |---|---|---| | start | 33.00 | 146.60 | | after the data-root move | 33.00 | 146.60 | | after `cleanup-disk-space` | 71.20 | 146.60 | | **after `docker pull`** | **71.20** | **108.10** | D: absorbs **38.5 GB**; C: does not move at all, even though 71 GB were free there by then. The daemon confirms it independently: `Docker data-root: D:\docker`. Things worth knowing before you copy this: - **Order is not negotiable.** Run it before the pull. Changing the data-root makes images under the old one invisible, so doing it afterwards throws away a pull you already paid for. - **It merges into an existing `daemon.json`.** The runners ship one β€” on the measured job it contained `{"hosts":["npipe://"]}`. Overwriting that would have taken the daemon's named-pipe listener with it. - **D: is not the same size everywhere.** Two runners on the same day reported 150 GB and 220 GB. The action's `required-free-gb` checks the *target* drive rather than assuming, so a wrong drive letter fails in seconds with a clear message instead of during the import. - **D: is the ephemeral temp disk on hosted runners** β€” wiped between jobs, which is right for CI and wrong for a self-hosted machine you expect to keep a layer cache on. Point `data-root` somewhere persistent there. - **`cleanup-disk-space` is not what makes the image fit.** The short-path clone is on D:, and the build runs in a container whose files are under the data-root and the D:\ws bind mount. What still used C: was hcsshim's staging, next section. ## The prologue on hosted runners Until 2026-10-13 every Windows lane spent 23-34 min in `prepare-windows-container-host`, about half its job time: - **The cleanup** (`cleanup-disk-space`): 9.0-16.5 min, run before the pull. It frees C: from about 29 to 68 GB, while the data-root and the image already sit on D:, and its Docker prune reclaims 0 B. - **The pull**: 13.4-15.6 min for the 16.7 GB `:winamd64`. About 1.5 min of that is download; the rest is hcsshim importing one layer after another. **The pull does not use C: at all once the data-root is on D:.** hcsshim writes each layer to a staging directory before importing it, and the expectation was that the SYSTEM docker service stages under `C:\Windows\SystemTemp`. It does not here. Measured with a 5-second sampler in run 38425248553, with no cleanup before the pull: C: stayed between 31.71 and 31.92 GB free for the whole pull, while D: dropped and recovered by up to 12.8 GB at a time as each large layer was staged and removed. `D:\docker\tmp` held up to 49 entries during every pull of run 38455306377: the staging follows the data-root. Setting the service's `SystemTemp` to `D:\docker-tmp` changed nothing (pulls of 904 and 920 s against 874 s without it), so the prologue has no input for it. The cleanup is therefore not needed for the pull. `prepare-windows-container-host`'s `cleanup` input says whether and when it runs, with the old paths kept: | `cleanup` | What runs | |---|---| | `hide` (the default since 2026-10-14) | No deletion; every `PATH` entry under a directory the cleanup deletes leaves `PATH` for the later steps | | `after-pull` (the default on 2026-10-13) | The pull, then the cleanup as a detached process; `run-in-windows-container` waits for it after its container run | | `before` | The cleanup, then the pull: the old order | | `off` | No cleanup, `PATH` untouched | Measured on `windows-2025`, the same image, one job per mode and run (`ci-host-speed-probe.yml`; prologue = everything before the container step): | Run | `before` | `after-pull` | `off` | |---|---|---|---| | 38425248553 | 1761 s (cleanup 786, pull 932) | 974 s (pull 920) | 927 s (pull 874) | | 38455306377 | 1630 s (cleanup 652, pull 927) | 1043 s (pull 895) | 1183 s (pull 1137) | - **The saving is the cleanup**: 9.8-13.1 min per Windows job. What is left of the prologue is the pull plus about a minute; the pull alone varies by 4 min between runners. - **The background cleanup costs the container little.** It took 731 s beside a fixed container workload that ran 43 s, against 46 s with no cleanup at all. A real lane's first container step runs far longer than the cleanup, so the wait after it is about 0; the probe's short workload waited 483 s. - **It is detached through WMI** (`Win32_Process.Create`). A `Start-Process` child with redirected output died seconds after its step ended (run 38455303000); one without redirection, and the WMI one, outlived it. If WMI fails, the cleanup runs in the foreground with a warning. **`after-pull` slowed the builds it ran beside.** Its first consumer runs (2026-10-13) waited 0-181 s for the cleanup, but their container builds ran 23-54 % longer: - OxidANT x64 1087 β†’ 1334 s; - AccelerANTgine x64 1420 β†’ 1682 s, and its arm64 cross 1001 β†’ 1546 s; - OmniAccelerANT's clang build was unchanged (1086 β†’ 1048 s). The probe's synthetic workload did not reproduce that, and a probe whose first container step is short waits for the cleanup instead. Run 38473550649, from the prologue's start to the end of two container steps: | `before` | `after-pull` | `after-pull`, idle priority | `off` | |---|---|---|---| | 2400 s (cleanup 672) | 2259 s (waited 387) | 2060 s (waited 293) | 1718 s, C: never below 28.87 GB free | **`hide` is `off` with the host steps' view of `PATH` kept.** The cleanup frees C:, and a container lane writes little there: the image, the container's scratch layer and the bind-mounted tree are on D:. What it also did was take the runner's toolchains (Strawberry, LLVM, CMake, the tool cache) off `PATH`. A host step that runs a built binary with the inherited `PATH` would otherwise load a DLL from them that the product failed to bundle: OmniAccelerANT's `Test-LaunchSmoke.ps1` and AccelerANTgine's `Start-Windows.ps1` both search it. `hide` writes the shortened `PATH` to `GITHUB_ENV`, and the cleanup's own two lists are the roots, so the two modes cannot drift apart (`DiskCleanup.Hide.Tests.ps1`). A job that builds on the host and needs those toolchains passes `cleanup: off`. **Linux has the same shape at a smaller scale.** `prepare-linux-ci-host` ran the full jlumbroso cleanup before a pull whose unpack is serial and disk-bound, although the runner starts with 91 GB (x64) or 112 GB (arm64) free against an image that needs about 49 GB. Since 2026-10-13 it logs in, starts the pull in the background, runs the cleanup and the checkout beside it and waits for the pull last (`pull-first`), and the cleanup keeps only its quick part (`free-disk-space: fast`: .NET, Haskell, swap). Run 38425248553: | Runner | old order, full cleanup | pull first, full | pull first, fast | |---|---|---|---| | `ubuntu-26.04` | 444 s (cleanup 122, pull 321) | 331 s | 311 s, 58.1 GB free after | | `ubuntu-26.04-arm` | 323 s (cleanup 64, pull 257) | 269 s | 229 s, 85.6 GB free after | The full cleanup left 78.4 (x64) and 90.6 GB (arm64) free after the pull. A lane whose build trees need more than the fast tier leaves passes `free-disk-space: 'true'`, as the riscv64 lanes do while they pull the riscv64 child as a second image (`container-ci-riscv64.yml` with `sysroot-mode: export`, the riscv64 row of `python-ci-linux.yml`). `pull-first: 'false'` restores the old order. `.github/workflows/ci-host-speed-probe.yml` times every combination side by side on the hosted runners and prints one `TIMING` line per job, with the lowest free space its 5-second sampler saw. ## Gotchas that cost real time - **Windows path limit inside containers.** Deeply nested paths (here: Rust `cxxbridge` output under `cargo/`) fail to extract: ``` ...out/cxxbridge/include/.../native_only.rs: Can't create '\\?\C:\ws\...': Invalid argument tar: Error exit delayed from previous errors ``` **One such failure aborts the entire tar transfer**, so a single deep path silently turns a full transfer into a partial one. Exclude those subtrees. - **Dev Drive rejects bind mounts.** The filesystem minifilter cannot attach ("Der Dateisystem-Minifilter kann nicht an das Entwicklervolume angefΓΌgt werden"), which forces the tar-pipe transport in the first place. Allow- listing the filters lifts the restriction β€” but measured slower here; see [Why the bind mount lost here](#why-the-bind-mount-lost-here) above before reaching for it. - **Containers survive successful builds** (`wcifs` teardown lock). A lingering container makes it look like a build is still running. Compare the newest build-summary timestamp against the container start time before assuming, and reap stale containers by name. - **`docker exec` bypasses the image entrypoint.** When driving a long-lived container with `docker exec` (rather than `docker run`), the entrypoint that sets up the VS developer environment and the clang-cl ASAN runtime DLL directory never runs. Invoke it explicitly: ```pwsh docker exec -w C:\ws $container cmd /S /C C:\temp\scripts\entrypoint.cmd @buildArgs ``` Symptoms if you forget: compilers "not found", or ASAN binaries failing to start because `clang_rt.asan*.dll` is not on `PATH`. - **Mount over a fresh path.** Mounting onto a directory baked into the image (e.g. `C:\workspace`, until the image after 2026-09-26) fails at `CreateComputeSystem` when the host OS build differs from the image base build. Use a path absent from the image β€” and if you support both transports, make it the *same* path the tar-pipe extracts to (`C:\ws`), not a separate one, or CMake rejects the cache on every switch. See [Transports](#transports-how-to-set-up-both). ## Reusable implementation The pattern is implemented here so consumers do not copy it: `windows/scripts/modules/WindowsContainerBuild.Reuse.psm1`. Import it via the consumer's module resolver. **`Invoke-ContainerBuild` is the whole pattern in one call:** tar-pipe into the reusable container by default, `-UseBindMount` for the bind-mount transport (probed first; the tar-pipe takes over when the mount does not attach), `-FreshContainer` for the reset switch of rail 2, and `-VerifyDirs` for the delivery check of rail 4. It also hands the container this host's sccache remote tier ([`windows-build-resources.md`](windows-build-resources.md#the-build-hosts-remote-tier-at-run-time)), and the shared S3 level per `docker exec`, by name only ([`windows-build-resources.md` Β§ Two cache levels](windows-build-resources.md#two-cache-levels)). The BeschleunigerBallett driver named at the top of this page is built on it. Its parts are exported for a caller that needs only some of them: `Get-ReusableBuildContainer`, `Copy-IntoBuildContainer`, `Copy-FromBuildContainer`, `Initialize-ContainerPwsh`, `Remove-StaleContainerSources`, `Test-BuildArtifactsDelivered`, `Remove-BuildContainerSafe` and `Wait-ContainerExit`, and the plumbing under them: `Get-SccacheContainerEnv`, `Get-ContainerEnvArgs`, `Resolve-ContainerBuildCommand`, `Resolve-DockerExe`, `Get-ContainerIsolationArgs` and `Test-ContainerBindMount`. `Initialize-ContainerPwsh`, `Remove-StaleContainerSources` and `Test-BuildArtifactsDelivered` are the safety rails of the reusable-container pattern as functions: ensure pwsh exists in the container, prune stale sources on reuse (tar never deletes), and verify every built executable actually reached the host before trusting a green build. `Remove-BuildContainerSafe` removes a container while tolerating the wcifs teardown lock. `Wait-ContainerExit` is the fourth rail, and it points the other way: the docker CLI intermittently drops its pipe mid-run on this host while the container keeps building, so a non-zero *client* exit can be a failure that did not happen. It polls `{{.State.Status}}` until the container is no longer running and returns `{{.State.ExitCode}}` β€” the container's verdict, not the client's β€” with a bounded timeout and a named error for every way the wait can go wrong (vanished container, unreachable daemon, a container that never started, a state it does not recognise β€” unknown or empty states fail CLOSED rather than being read as "finished"). It requires a run that is **named and not `--rm`**: with `--rm` the daemon removes the container the instant it exits, taking the exit code with it. That is why the bind-mount transport names its run and removes it itself β€” but only on success. Every failure thrown out of the run/wait block points the operator at `docker logs -bindmount`, so a failed or timed-out run KEEPS its container (a warning says so, with the removal command); deleting it in a `finally` would destroy the evidence the error message just promised. The pre-removal before the run has two refusals, both there to stop a false green. A leftover under the name that is still *running* (a concurrent build of the same tree, or a runaway a timed-out wait kept) is never force-removed β€” the build throws instead. And when `docker rm -f` is blocked by the wcifs teardown lock (the removal returns but the container survives), the run falls back to a unique `-bindmount-<6 hex>` name, exactly like `Get-ReusableBuildContainer -Fresh` does: running against the held name would make `docker run` fail 125 with a name conflict, and the wait would then read the STALE container's exit code β€” a build that never ran, reported green. One layer down, `Wait-ContainerExit`'s inspect helper takes its value ONLY from stdout: docker prints client notices (deprecations, context warnings) on stderr *before* the field value, so a first-line pick over a merged stream returns the notice as the "value" β€” a clean container read as a non-numeric exit code, or a still-running one as finished. stderr is kept separately for classifying failures (missing container vs unreachable daemon). A red client exit gets one extra `docker inspect` to distinguish "docker run failed before any container existed" (unresolvable image, refused mount β€” the client's code is the whole story, nothing to wait on) from a dropped pipe. That probe runs ONLY on the failure path: after a clean run the container exists by definition, and the happy path must not grow a docker call β€” the no-change build is 44 s wall, and every client round-trip on this host costs seconds. `Wait-ContainerExit` does **not** apply to the reusable container's `docker exec` builds β€” that container's main process is a 7-day `ping`, so its state is `running` no matter what the build did. ## Applying this elsewhere The pattern generalises to any large project built in a Windows container: 1. One reusable container, keyed by name, recreated when the image changes. 2. Sources in, artifacts out β€” never the intermediate build tree. 3. An explicit "start clean" switch, because reuse means staleness. 4. Exclude deep paths from tar transfers before they abort silently.