Windows cross builds (arm64)

Contents

Windows cross builds (arm64)#

The Windows twin of linux-cross-builds.md. It covers the aarch64-pc-windows-msvc target lane: why it is shaped the way it is, what it can and cannot produce, and which gates keep it honest.

Status — re-measured 2026-09-02 after the #158-wave follow-ups: BOTH lanes green (bk-20260828-171914 still supplies the gate numbers below; #135’s patched toolchain is the default, so the amd64 workarounds are gone; #167 moved the arm64 smoke floors 66→69 and the measured run stays 97/0/15). The current tree — module-closure refactor (#134), forced clang-cl 23.1.0, per-TU AArch64 codegen workarounds (#135) — built end to end and reached RUNTIME PARITY with :winamd64. Every gate below hit its target on the current HEAD. Nothing the lane produces had been executed until the consumer cross lanes loaded some of it on 2026-09-25 (below) — wheels ship staged, not installed, and every verdict of this lane is a static check.

The amd64 lane’s TVM-vs-LLVM-23.1.0 blocker is fixed and REBUILT (since 2026-09-02 both lanes are green; TVM_COMMIT=994e0216, upstream main, with the guards). See docs/windows-refactor-backlog.md #134.

Same media and inference surface: GStreamer with an identical plugin set (200 linked plugin DLLs, all six contract plugins incl. webrtc/nice, plus gst-ptp-helper; the seven entries CON28 added on 2026-10-05 run on an arm64 device since 2026-10-11, § WebRTC on the cross lane), ONNX Runtime + GenAI with DirectML, OpenCV 5 (NEON dispatch: NEON_DOTPROD NEON_FP16 NEON_BF16), FFmpeg with NEON asm + PyAV, LiteRT with the tflite plugin, and the TVM/IREE runtimes together with their python packages — 6 wheels, the same count as amd64.

KleidiAI (2026-09-28): ORT’s MLAS builds with onnxruntime_USE_KLEIDIAI=ON on arm64. CMake’s default is OFF and only upstream’s build.py turns it on, so bundles before this date shipped without it. It is runtime-dispatched, so the armv8-a baseline is unchanged, and it routes MatMulNBits (the GenAI hot path), plus SGEMM/conv on SME CPUs, through KleidiAI. Build-OnnxFromSource.ps1 fails configure if ORT drops the option or build.ninja has no kleidiai target. Upstream’s Windows path does not assemble under Ninja as shipped. Its ASM_MARMASM rule hands the .S kernels raw to armasm64, which has no C preprocessor and rejects the /arch:armv8.2 KleidiAI adds. So the script fetches ORT’s own KleidiAI tarball (deps.txt, SHA1-checked) and sets the rule, right after KleidiAI’s enable_language, to clang-cl /P /EP /TC /U__clang__ then armasm64. ORT gets that tree as FETCHCONTENT_SOURCE_DIR_KLEIDIAI. Proven in :winamd64: 178/178 objects, SME2 kernels in kleidiai.lib. Not available at the pinned versions, checked against their sources:

  • XNNPACK in LiteRT v2.3.0 (d89ef666) forces XNNPACK_ENABLE_KLEIDIAI OFF on Windows with a plain SET.

  • OpenCV 5.0.0 offers WITH_KLEIDICV only for ANDROID OR UNIX.

  • Arm NN v26.07 has no Windows support, and ACL v53.3.0 calls Windows builds experimental and native-only.

  • llama.cpp is not built from source here.

Gate (arm64 bk-20260828-171914)

Result

amd64 (run 8, blocked at TVM)

PE arch gate over C:\runtime + host site-packages

992 inspected / 0 violations

1134 / 0

Import walk (-ImportWalk, unpacks staged wheels)

606 walked / 0 unresolved (3 allowlisted, 6 device-OS)

report-only

Target python deps

12 wheels / 0 unresolved requirement edges

installed natively

Mandatory GStreamer plugins

6 / 6

6 / 6

Smoke

97 passed / 0 failed / 15 skipped (floors then 69/20, now 76/20)

222 / 0 / 0

Absent by construction, each named inside the bundle (ABSENT-ON-ARM64.txt / COMPILER-ABSENT-ON-ARM64.txt): the TVM and IREE compilers and iree.compiler — they need an LLVM cross-built for aarch64-windows, with no upstream precedent (PyPI ships win_amd64 only); LiteRT-LM, whose active Bazel path has no windows-arm64 config and default-links an x86_64-only prebuilt (upstream’s CMake path with its constraint-provider stub is an unattempted port, backlog #133(d)); classic TensorRT (x64-only; TensorRT-RTX is the arm64 successor and is not wired — CUDA/cuDNN are cross-built since #176); and the torch app stage, because uv sync must execute the target interpreter. The optional gdkpixbuf GStreamer plugin is absent on arm64 only since 2026-09-26: gdk-pixbuf runs glib-compile-resources at build time, and a cross build has only the target’s (amd64 lost it to a missing rst2man until Get-GstGdkPixbufMesonArgs turned the man pages off). Anything needing cargo-cbuild is absent on BOTH lanes, so it is not a parity gap.

Barely verified — the asymmetry that outranks every number above: until 2026-09-25 no arm64 binary produced by this repo had been executed, anywhere. Its wheels ship staged, not installed; the smoke gate runs only host-toolchain and static sections (1-6, 14-16, 19 arch-filtered, 23, 25), and sections 14/15 compile for the target and assert the produced PE machine rather than run anything. A green build proves the code compiles and links for the target, and nothing more. Execution needs a windows-11-arm runner, and two kinds of job use one. The consumer apps’ cross lanes run their own products there since 2026-09-25 (§ Consumer cross lanes). The hub’s windows-arm64.yml runs Test-Arm64Bundle.ps1 over the published bundle since 2026-10-11 (§ Verification). There the bundle’s tools and Python execute, two GStreamer pipelines run (the WebRTC loopback is one), and since 2026-10-12 an ONNX Runtime session computes on the CPU EP. No runner has a camera or a GPU, so neither a capture pipeline nor a GPU EP has run on arm64 (BACKLOG.md CON30).

How the lane got here, run by run (#116, #128, #131, #133 narratives, including the three meson build-only-subproject defects and the seven fix-and-rerun cycles of the runtime-python work): windows-backlog-archive-2026-08-26.md. Open items: windows-refactor-backlog.md.

The constraint everything follows from#

There is no arm64 Windows container base image, and there is no arm64 Windows Server. mcr.microsoft.com/windows/servercore and nanoserver are published for amd64 only; Microsoft’s tracking issue (Windows-Containers#586) closed without one.

So the question “build an arm64 Windows image” has no answer. What this lane does instead:

  • The build host and container stay windows/amd64.

  • Only the compile target changes, via clang-cl --target=aarch64-pc-windows-msvc + lld-link.

  • The product is a versioned artifact bundle (import libs, DLLs, headers), consumed on real ARM64 Windows hardware — not a runnable container.

Two corollaries worth stating plainly, because both have already been guessed wrong:

  1. Never tag or push the output with --platform windows/arm64. The image carrying the payload is a windows/amd64 image; an arm64 platform descriptor on it produces a manifest nothing can run, and someone will pull it onto a Windows-on-ARM box and file a bug.

  2. arm64 binaries cannot execute on the build host. Windows x64 has no ARM64 emulation (only the reverse). Every “run it and see” smoke in Test-Container.ps1 is unavailable here, which is why the static gates below carry so much weight.

Why clang-cl makes this cheap#

The whole Windows media chain already builds with Ninja + clang-cl + lld-link and explicit -DCMAKE_C(XX)_COMPILER= — there is no Visual Studio generator anywhere in the tree (CMAKE_GENERATOR_PLATFORM and -A ARM64 have zero occurrences). clang-cl cross-targets aarch64 natively, so this lane is a target-triple change, not a toolchain replacement.

The rule is absolute: clang-cl for every compile, lld-link for every link, plus llvm-lib, llvm-rc, llvm-mt, llvm-readobj. On arm64 that costs nothing extra for the C/C++ half; the one place the amd64 lane deliberately falls back to cl.exe — as nvcc’s host compiler, since nvcc rejects clang-cl — has its cross twin since #176: the arm64 CUDA payload is compiled by the same x64 nvcc driving Hostx64\arm64\cl.exe with --use-local-env (see § CUDA / cuDNN / TensorRT).

The MSVC ARM64 component is still required, but not for its compiler. clang-cl targets the MSVC ABI, so an aarch64 link needs Microsoft’s ARM64 CRT and import libraries (VC\Tools\MSVC\<ver>\lib\arm64), which ship only with Microsoft.VisualStudio.Component.VC.Tools.ARM64. Its Hostx64\arm64\cl.exe rides along unused. This is why Install-Vs.ps1 and Test-Toolchain.ps1 assert the library directories rather than the compiler binary — a cl.exe probe would pass even when the libraries, the part that is actually load-bearing, are absent.

Vulkan needs an optional component#

LunarG ships the aarch64 import libraries inside the x64 Windows SDK, but as an optional component the default install does not select:

com.lunarg.vulkan.arm64   ARM64 binaries for cross compiling on Windows x86_64
Lib-ARM64 / Bin-ARM64     for cross compiling from Intel based development environments

scoop installs the SDK with the manifest’s default component set, so a stock base image has no $VULKAN_SDK\Lib-ARM64. Install-ScoopTools.ps1 adds it explicitly through the SDK’s Qt Installer Framework maintenancetool.exe.

That step warns, it does not fail — deliberately. Every arm64 prerequisite in the base (Vulkan Lib-ARM64, the MSVC lib\arm64 CRT, the Windows SDK um\arm64 libs) is checked warn-only, because base is shared by both lanes: a hard failure over an arm64-only prerequisite would block the amd64 build too, and the maintenancetool invocation has never yet been executed against a real base image. An unverified installer call must not be able to kill the chain’s most expensive layer.

Set WINDOWS_ARM64_STRICT=1 to turn them into hard gates — the same opt-in shape as CUDA_STACK_STRICT=1 on the Linux side. All of them except one (this sentence said “all” until 2026-08-24, disagreeing with AGENTS.md): Install-Vs.ps1’s MSVC lib\arm64 check runs in a RUN that sits above the ARG WINDOWS_ARM64_STRICT declaration in Dockerfile.base and therefore never sees the flag — that one check stays warn-only regardless. Use STRICT once the arm64 lane is real; it is the flag that says “this image claims a complete arm64 toolchain”.

It is plumbed as a real ARG in Dockerfile.base and forwarded as a script parameter. That matters: a bare $env: read is unreachable from docker build, and buildctl silently discards --build-arg for undeclared ARG names — an earlier version of this gate had an escape hatch that could not actually be set from anywhere.

Cache discipline: the base image is shared#

base / sdk / toolchain install host tooling (VS, LLVM, scoop, vcpkg, rust, host CPython). None of it is target-specific, so one base serves both lanes and the arch fan-out starts at media:

base ─ sdk ─ toolchain ──┬─ media(amd64) ─ torch ─ final  → :winamd64  (image)
   (shared, x64 host)    └─ media(arm64) ───────────────  → winarm64 artifact bundle

Consequently WINDOWS_TARGET_ARCH is never declared in Dockerfile.base or .toolchain-builder. On the CPU lanes it first appears in Dockerfile.media-builder’s common stage. This is the file’s own documented ARG discipline (Dockerfile.base — “every RUN after an ARG declaration keys its cache on the ARG’s value”): declaring the arch ARG in base would re-pay the VS Build Tools install — the chain’s most expensive layer — on every lane switch. The GPU lane is the one exception: since #176 (2026-09-20) Dockerfile.nvidia declares it, because the sdk stage stages the arm64 CUDA payload on the cross lane. So on the GPU lane the CUDA layer, and everything built FROM it, is keyed on the target arch, while both arches still write the same unsuffixed bk-windows-sdk / bk-windows-toolchain tags ($script:NoSuffixTags in Build-Buildkit.ps1).

For the same reason WindowsTargetArch.Common.psm1 is COPY’d into base below the VS layer, in the same group as Test-Toolchain.ps1 — its only consumer there. The host provisioning scripts (Install-ScoopTools.ps1, Install-Vcpkg.ps1) spell their few arm64 facts inline and import just the three modules that precede the VS layer. Placing the module higher invalidated scoop, vcpkg and the ~30-minute rust layer on every edit to a file whose whole point is that adding a target is a one-line table edit.

The one-time cost is a single base rebuild for the VS ARM64 component + the Vulkan component. Batch them, and see the sequencing warning below.

An ARG does not cross a FROM boundary, so every stage whose FROM is a ${...} image reference and that RUNs a build script redeclares ARG WINDOWS_TARGET_ARCH; without it the stage silently builds amd64 on top of an arm64 parent. The arch is never a versions.env key either: Import-Versions.ps1 would overwrite the build-arg in inherited stages. TargetArch.Common.Tests.ps1 gates both.

Where arch facts live#

windows/scripts/modules/WindowsTargetArch.Common.psm1 is the single source of truth — the Windows twin of linux/scripts/01-core/arch-mapping.sh. It maps, per target: clang triple, VsDevCmd -arch, PE machine type, vcpkg triplet, MSVC lib/bin subdirs, Vulkan Lib-ARM64, CPython -p platform and PCbuild\<dir>, wheel tag, rust triple, NuGet RID, ffmpeg --arch, lib /machine, SIMD flag sets, and the CMake cross arguments.

Two behaviours are deliberate and tested:

  • An unknown arch throws. Silently degrading to amd64 would emit an x64 build labelled arm64, which no downstream gate would catch.

  • Get-CMakeCrossArgs returns nothing for amd64, so the existing lane’s configure command line is provably unchanged. (Since 2026-08-24 the cross args also cover the ASM language — CMAKE_ASM_COMPILER_TARGET / CMAKE_ASM_FLAGS_INIT; see the LiteRT machinery section for the failure that forced it.)

It is re-exported through WindowsSourceBuild.Common.psm1 on the same terms as WindowsNative.Common.psm1, so build scripts get it with their usual import. Ship it in every Dockerfile COPY list that carries the module set — an incomplete COPY list is a known failure mode, guarded here by a throwing stub.

SIMD: the failure that hides inside a green build#

The kernel SIMD flags were never an ordinary flag helper’s output. Build-OnnxFromSource.ps1 injects them per-TU into build.ninja post-configure — via Get-WindowsTargetKernelSimdFlags -Arch; the old Get-WindowsX86Avx512Flags compat shim was deleted on 2026-08-26 — onto exactly the MLAS kernels matched by Get-MlasKernelTuPattern. Globally-enabled AVX-512 was field-proven to crash protoc and onnxruntime.dll’s static initializers with STATUS_ILLEGAL_INSTRUCTION; entirely without the flags those TUs fail to compile. Per-TU is the only correct answer, because the kernels are runtime-dispatched.

On aarch64 that x86 pattern matches nothing — and a patch that matches nothing succeeds. The build would go green with MLAS’s NEON/dotprod/i8mm kernels compiled without their features: unoptimised at best, absent at worst, and undetectable from the build host.

So the pattern is arch-parameterized (Get-MlasKernelTuPattern) alongside a minimum match count (Get-MlasKernelTuMinimum), and Build-OnnxFromSource.ps1 throws when the tagged-TU count falls below that floor. The floor is the actual guard; the pattern alone is not — a warning there would have preserved exactly the failure mode this exists to prevent.

And on 2026-08-24 the amd64 lane demonstrated the same failure with the floor present but too low. The x86 pattern (qgemm_kernel_amx|intrinsics[\\/]avx512) predated ONNX Runtime v1.29.0, which keeps six more AVX-512 kernel TUs directly in mlas/lib/ (q4gemm_avx512.cpp, the sqnbitgemm_kernel_avx512* family, qkv_quant_kernel_avx512vnni.cpp). The stale pattern matched 5 lines, the floor was 4, so 5 >= 4 sailed through — and five TUs then failed to compile (always_inline function '_mm512_set1_ps' requires target feature 'avx512f'). The aarch64 pattern had been re-measured against v1.29.0 on 2026-08-23; the amd64 one had not, and it was the amd64 regression run for the cross-lane changes that surfaced it. Fix: pattern extended with a \.cpp-anchored third alternative (the anchor makes it structurally impossible to tag a MASM FLAGS line — -match is case-insensitive and lib/amd64/ is full of *KernelAvx512*.asm), floor raised 4→8. The floor rule this hardens: a floor only earns its keep if the previous broken state would trip it — the old pattern’s 5 matches now fail an 8-floor loudly. Verified: 11 TUs tagged, all five previously failing TUs compiled, full media-core green on both lanes.

Note the asymmetry in the baseline flags: amd64 enables a broad SSE/AVX2 set globally, arm64 enables nothing globally. AArch64 already mandates NEON, and its optional features (dotprod/i8mm/SVE) fault on hardware that lacks them — the same class of failure as AVX-512. Optional AArch64 features belong only on dispatched kernels.

Vulkan on lavapipe#

The image and the bundle carry Mesa’s lavapipe; what ships and why: consumer-image-contract.md § The Windows image ships lavapipe (CON50). For this lane, Test-Arm64Bundle.ps1’s device step (18 steps, floor 18 since 2026-10-12) is the gate that proves the bundle’s device, and :winamd64 carries it since the 2026-10-04 republish; :winarm64 followed the same day and its device gate passed 12/12.

The Vulkan validation layer#

The merge stage stages VkLayer_khronos_validation.dll and its JSON in C:\runtime\vulkan-layers on both arches (CON64), so a consumer validates its GPU tests from one path:

  • amd64 copies the layer from the image’s Vulkan SDK ($env:VULKAN_SDK\Bin, VULKAN_VERSION).

  • arm64 builds it, because no SDK the container can install has one. The x64 SDK’s Bin-ARM64 carries the crash-diagnostic and gfxreconstruct layers but not this one, and LunarG’s ARM64 SDK installer is itself an ARM64 executable. Build-VulkanValidationLayers.ps1 clones Khronos Vulkan-ValidationLayers at vulkan-sdk-<VULKAN_VERSION>, builds the dependencies its scripts\known_good.json pins (Vulkan-Headers, Vulkan-Utility-Libraries, SPIRV-Headers, SPIRV-Tools, mimalloc; tests-only repos skipped), then the layer, all through Invoke-CmakeConfigure with the arm64 cross args.

The tag pins the whole set, as Linux’s vulkan.sh does. Copying the JSON beside the DLL is what the manifest’s .\VkLayer_khronos_validation.dll needs. An elevated loader ignores VK_ADD_LAYER_PATH, so a consumer registers the JSON under HKLM\SOFTWARE\Khronos\Vulkan\ExplicitLayers for its run (the bundle README shows the command). Test-Arm64Bundle.ps1 forces the layer into a vulkaninfo instance, which loads the DLL. Every bundle runs that step since the published :winarm64 of 2026-10-10 passed it (BeschleunigerBallett run 38100828628).

Verification#

With nothing runnable on the build host, verification is layered:

Gate

Where

What it proves

Test-Toolchain.ps1 arm64 section

base image

clang-cl emits aarch64 objects; MSVC/SDK/Vulkan arm64 libraries present

Test-TargetArch.ps1

any staged tree

every shipped .dll/.exe (optionally .lib) has PE machine 0xAA64, with a minimum inspected floor; with -ImportWalk every wheel’s native members too, and every module of a cp3XYt wheel imports python3XYt.dll and no GIL runtime

TargetArch.Common.Tests.ps1

Invoke-Tests.ps1

the arch table, the amd64 byte-identity guarantee, and the MLAS pattern behaviour

Test-Arm64Bundle.ps1

an arm64 device, or the hub’s windows-arm64.yml

the bundle’s tools and Python execute (HailoRT, a GStreamer pipeline, IREE, the offline wheel install, the ORT providers, an ORT session over the smoke suite’s 63-byte Identity model on the CPU EP, with DirectML’s outcome printed only), the shipped aarch64 ASan and OpenMP runtimes are real ARM64 PEs, the cp313 torch stack is in the wheel store, the pinned cpXY pytest stack installs offline from it, python-freethreaded\python3XYt.exe imports its stdlib extensions with sys._is_gil_enabled() False, every cp3XYt twin in wheels-free-threaded (wheels-cp314t in a bundle exported before CON91) installs alone, offline, into a venv of that interpreter and loads with the GIL off (free-threaded-wheel.py prove), vulkaninfo --summary lists the bundle’s lavapipe (llvmpipe), the validation layer loads, and a webrtcsink -> webrtcsrc loopback decodes 60 frames over DTLS-SRTP (CON63)

This repo’s own lane has no native execution gate, so Test-Arm64Bundle.ps1 is the device half: every step is exit-code-checked and the run must pass -MinPassed, so a device that ran nothing cannot look green. The first device run (2026-10-03, Snapdragon X, the published :winarm64 of that day) passed all eleven steps; the lavapipe step (2026-10-04, CON50) made it twelve — the bundle executes, not just loads. The hub’s own windows-arm64.yml runs the gate daily and on a push to its files: a windows-2025 job packs the published image’s C:\runtime with Export-Arm64Bundle.ps1, and a windows-11-arm job extracts and runs it. Consumer lanes hosted it from 2026-10-03 to 2026-10-11, until a WebRTC defect turned BeschleunigerBallett’s renderer lane red and skipped its GPU suites over a component the renderer never uses.

Test-TargetArch.ps1 is the Windows twin of the Linux lane’s ELF check in validate-media-runtime.sh. Three design points, each learned from a gate that could not fail:

  • A minimum inspected count. A mis-pathed or empty tree otherwise passes green with zero files checked — the failure Dockerfile.smoke-gate’s MIN_PASSED floor exists for. Allowlisting everything therefore also fails, because it drives the inspected count to zero.

  • .lib archives are not PE files. A naive bytes[0x3C] walk over a COFF archive reads whatever sits at that offset and can compare equal by accident. Archives are decoded from their first member header, including the short-import header form.

  • The host-tool allowlist is printed. An over-broad allowlist is itself a defect, and the only way to notice is to see what it swallowed.

Usage:

# strict: every binary must be arm64, at least 20 inspected
windows\scripts\build\Test-TargetArch.ps1 -Path C:\runtime -Arch arm64 -MinInspected 20

# permit genuinely host-arch build tools that never ship to the target
windows\scripts\build\Test-TargetArch.ps1 -Path C:\runtime -Arch arm64 `
    -HostToolPattern 'protoc\.exe|flatc\.exe|\\_deps\\'

# on an arm64 device, from a bundle zip: eighteen steps, and the default floor is all of them
windows\scripts\build\Test-Arm64Bundle.ps1 -ZipPath C:\temp\winarm64-bundle.zip

Free native validation is available: this repo is public, so GitHub’s windows-11-arm runners cost nothing. They are Windows 11 client, not Server Core — a caveat to state rather than a problem to solve, since no Server Core arm64 exists.

Consumer cross lanes (container-ci-windows.yml)#

The consumers build FOR this bundle’s target in CI: windows-arm64-cross.yml in OxidANT, AccelerANTgine and BeschleunigerBallett (owner decision 2026-09-25). OmniAccelerANT’s windows-arm64.yml (2026-09-27) is a hybrid: Flutter cannot cross-build windows-arm64 from an x64 host (flutter/flutter#62597, open), so its natives cross-build here and a second job builds the Flutter app natively on windows-11-arm against them. Its vendored Cargokit gained windows-arm64 and a switch that takes a prebuilt library for that. OrchestrANT’s windows-arm64-cross.yml (2026-10-01) is the Python one: Invoke-CiPackaging.ps1 -TargetArch arm64 lays out an app bundle and its packages, and the run job starts the bundle and the unpacked zip (docs/python-app-bundles.md § Packages).

Each caller is thin. It names its build script, the script’s arguments and the directory the product lands in. The hub’s reusable container-ci-windows.yml owns everything else, and it is the same file an x64 lane uses with target-arch: amd64:

  1. prepare-windows-container-host with target-arch: arm64 pulls the arm64 bundle, the action’s image-arm64 default. verify_ci_image_refs.py holds that default to CI_IMAGE_WINDOWS_ARM64_TAG, so no workflow names the tag.

  2. run-in-windows-container runs the build script in that image, under C:\ws. Its entrypoint loads the arm64-targeting VS environment, so the rules of this page apply to the consumer’s build: a bare clang-cl still targets x64 and needs --target.

  3. The arch gate: Test-TargetArch.ps1 -Arch arm64 -ImportWalk over the product, inside the same image, from the caller’s own hub submodule. An x64 object in the tree fails, and so does an import that a clean device cannot resolve (the CRT has to ship with the product).

  4. The product is uploaded.

  5. With a run-command, a second job on windows-11-arm downloads the product and runs that command in it natively. The hub’s bundle device gate (§ Verification) is the other place an arm64 binary from the family executes.

  6. With a test-artifact-dir and a test-command (CON43, 2026-10-01), the build also stages test binaries. They get the same arch gate as the product, because Windows on Arm runs x64 code transparently and an amd64 test would pass there by emulation. They are uploaded as <artifact-name>-tests, and the second job runs test-command in them after run-command.

    • The verdict is a printed line, TESTS: passed=<n> failed=<n> skipped=<n>. The job fails without it, on any failure, and when nothing passed, so an empty run cannot read as green. The counts go to the job summary.

    • A test-results-path upload takes the reports out. Without it the results a device generates (coverage, junit, html, md) exist only in the job log: the -tests artifact is uploaded from the build job, before the device runs. One relative path or glob against the test working directory, uploaded as <artifact-name>-test-results (OrchestrANT’s arm64 pytest is the first consumer).

    • windows/scripts/build/Invoke-StagedTests.ps1 prints that line. It is self-contained, since the runner has no checkout. A consumer’s build copies it and a tests.json into the test directory. Each entry names one binary and how to count it: gtest from googletest’s [  PASSED  ]/[  FAILED  ]/[  SKIPPED ] summary, cargo from libtest’s test result: lines, pytest from pytest’s final == N passed … == line (errors count as failures, xfails as skips), exitcode as one test. A failing test is counted, not thrown, so the line always prints. A missing binary or an unreadable summary is an error. For a binary that died before its summary, that error carries the exit code with its NTSTATUS name, e.g. exit -1073741819 (0xC0000005 STATUS_ACCESS_VIOLATION). When the runner’s event log has a Windows Error Reporting record for that binary, the faulting module, offset and exception code come with it, read from the record’s data, not its localized message. OxidANT’s arm64 WARP crash (run 36889467167) left only running 39 tests before this.

    • A test tree with no binaries sets test-arch-gate-min-inspected: 0. That fits a Python suite that the product’s own python.exe runs. The gate still refuses any non-arm64 PE file in it, but needs none, and runs no import walk, since its DLLs live in the product tree.

    • A lane is renamed Windows arm64 · cross build + test only once its tests gate.

  7. The image’s runtime bundle is not a consumer lane’s to gate (2026-10-11). The hub’s windows-arm64.yml packs it (C:\runtime, ~290 MB zipped, by Export-Arm64Bundle.ps1) beside Test-Arm64Bundle.ps1, free-threaded-wheel.py and the two GStreamer test modules the WebRTC step imports, and runs the gate on windows-11-arm. A consumer gates what its product uses, such as BeschleunigerBallett’s GPU suites with the validation layer -StageTests stages.

Three inputs serve the x64 lanes that moved onto the same file (the family’s next sharing step, 2026-09-25: OxidANT’s windows-x64.yml first, then AccelerANTgine’s, and BeschleunigerBallett’s on 2026-09-26):

  • version-file resolves the CI version on the host with the hub’s version_util.sh and hands VERSION and MSIX_VERSION to the container. It may name a VERSION.txt or a Cargo.toml, whose [workspace.package] version it reads.

  • host-command runs on the runner host after the build, in the workspace the container built into. It is for what Server Core cannot run because it lacks opengl32.dll: OxidANT’s WebGPU renderer tests, and a run of the x64 product.

  • The CONTAINER_SECRET_ENV secret carries the caller’s own secrets into the container: one KEY=value per line, composed from the caller’s secrets. Each value is masked, and the pairs reach docker as an --env-file in RUNNER_TEMP, so none enters a command line, a step output or the job environment. BeschleunigerBallett’s WebDAV credentials and signing password go this way.

  • compiler-cache-key (2026-09-29) keeps the build’s compiler cache across runs: build-cache-tiers.md § Keeping the compiler cache across CI runs.

host-command and run-command both run with $PSNativeCommandUseErrorActionPreference, so a native command that exits non-zero on any line fails the lane. Before that, only the exit code of the last line counted.

All three lanes went green on their first CI runs, 2026-09-25:

Consumer

Binaries graded

What runs on windows-11-arm

OxidANT

12

kataglyphis_cli.exe --help and stats

AccelerANTgine

22

AccelerANTgine.exe, which prints its version once the chain ONNX Runtime and GStreamer have loaded

BeschleunigerBallett

8

GraphicsEngine.exe --version

The runner is a GPU-less Windows 11 client. It has opengl32.dll, which Server Core lacks, but no Vulkan loader. BeschleunigerBallett’s engine imports vulkan-1.dll, which a device gets from its GPU driver. Its run job therefore borrows the Khronos loader from LunarG’s arm64 runtime, pinned by SHA256, for that one run. The product never ships the loader.

The product folder carries its own DLL closure. A clean arm64 device has no VC++ redist and none of C:\runtime, so step 3’s import walk fails any import the folder does not hold. Copy-PeImportClosure (WindowsCrossBundle.Common) fills it. It walks the static and delay-load imports of the binaries it is given, transitively, the set the walk grades. It copies each name it finds in the search directories, the first directory winning, and throws on a closure DLL of the wrong machine. A DLL loaded by name at run time (ONNX Runtime under load-dynamic, a GStreamer plugin) is in neither import table, so the build passes it in as a seed. Names found in no search directory are left to the device, and the walk grades them.

OxidANT and AccelerANTgine take the search directories from Get-ProductDllSearchPath (each in its scripts/windows/Build-Windows.ps1), so both look in the same order:

  1. $env:ONNX_ROOT\bin, so an ORT-family import resolves to the chain build;

  2. C:\runtime\bin, the media stack;

  3. VCToolsRedistDir\<x64|arm64>\Microsoft.VC*.CRT, the VC++ runtime, app-local.

BeschleunigerBallett’s cross bundle passes -SearchDirectory C:\runtime\bin to Copy-PeImportClosure itself (its scripts/windows/Build-Windows.ps1, checked 2026-09-25). Its VC++ runtime comes from the cmake --install it bundles, through CPackCommon.cmake’s InstallRequiredSystemLibraries.

x64 packages carry the same closure (owner decision 2026-09-25). A clean x64 PC lacks the media stack and may lack the VC++ runtime too. A CMake consumer whose packages come from CPack fills the closure into a directory it names at configure time as KATAGLYPHIS_PACKAGE_DLL_DIR. CPackCommon.cmake installs that directory’s DLLs beside the executables (kataglyphis_install_package_dlls), so the MSI, the ZIP, the NSIS installer and a cmake --install bundle all ship them. On 2026-09-25 AccelerANTgine sets it; BeschleunigerBallett’s build does not yet.

A CMake consumer names the target at configure time. Get-CrossConfigureArgs (WindowsCrossBundle.Common) returns nothing on amd64, so an x64 configure line does not change. On arm64 it returns Get-CMakeCrossArgs, which sets the triple and CMAKE_SYSTEM_NAME/CMAKE_SYSTEM_PROCESSOR, so vendored try_run checks become try_compile. Two more follow only on request, because each otherwise follows the HOST’s pointer size:

  • -Corrosion adds Rust_CARGO_TARGET;

  • -Vulkan adds Vulkan_LIBRARY from the SDK’s Lib-ARM64.

Get-WindowsPackageArch (same module) spells the target the way Windows packaging does, x64 or arm64: an AppxManifest’s ProcessorArchitecture, wix build -arch and the VC++ redist directory.

The hub’s cmake/ reads CMAKE_SYSTEM_PROCESSOR too. Hardening.cmake links /CETCOMPAT on x64 only, because an ARM64 link refuses it (test-cmake-windows-arch.sh). CPackCommon.cmake stamps the MSI arm64 when the package name says aarch64. It also installs the VC++ runtime without vcruntime140_1.dll: the arm64 redist folder carries that file as ARM64EC, an x64-machine PE that only x64 and ARM64EC code imports, and the arch gate refuses it (measured in AccelerANTgine’s first cross build).

A cross lane builds Release, Profile and - since 2026-10-03 - the Debug (ASan) configuration: VS 2026’s toolset carries the aarch64 runtime DLL and lib/arm64 the import libs, and Sanitizers.cmake picks dir and lib names from CMAKE_SYSTEM_PROCESSOR. The Debug suite is a /MDd build, so the run job supplies the debug CRT and ucrtbased.dll from its own VS and Windows SDK - neither is redistributable. Anything that runs a test, a benchmark or a PGO training pass cannot run on the amd64 host: that is what the run job is for.

Sequencing: rebuild base twice, on purpose#

Install-Vs.ps1 deliberately does not SHA-pin the VS bootstrapper (the installer refreshes within a channel every few weeks; the hash is logged for provenance instead). Adding the ARM64 component therefore also pulls whatever newer VS servicing build is current — and any regression it carries will look exactly like “the arm64 change broke the build”.

Rebuild the base with no change at all first, confirm it is green, and only then add the components. It is the single highest-value sequencing decision in this work.

Upstream portability fixes this lane carries#

None of these are configuration mistakes — they are code paths that only compile on ARM, so no amount of amd64 testing could have surfaced them. Each is scoped to the cross branch so the amd64 command line stays byte-identical. All were measured on 2026-08-23 against LLVM 22.1.8.

Of the OpenCV patches, four are live and one is dead (audited 2026-08-24): the mlasi.h intrinsic remap below never compiles, because patch 003’s WIN32-gated return() removes MLAS from the OpenCV build before that header is ever reached. Its row stays as a record of a real collision, flagged dead where it appears.

/D_USE_MATH_DEFINES (OpenCV)#

hal/carotene, OpenCV’s ARM NEON HAL, is compiled only for ARM targets. It uses M_PI, which the C standard does not mandate and the MSVC CRT withholds unless this macro is defined — carotene assumes POSIX. Symptom: phase.cpp(121,5): use of undeclared identifier 'M_PI'.

softfloat.cpp typedef → macro (OpenCV)#

Genuine upstream bug, see below.

WITH_IPP=OFF, BUILD_IPP_IW=OFF (OpenCV)#

IPP is Intel’s x86-only primitives library; no AArch64 build exists. OpenCV still resolved and unpacked 3rdparty/ippicv/ippicv_win, putting its headers on every core TU’s include path, and the staged .lib is x64 COFF that lld-link would reject against an arm64 image.

USE_DML=ON (ONNX Runtime, and GenAI since #118), WITH_DIRECTML=ON (OpenCV) — all three initially OFF on this lane#

The original justification was wrong and is retracted: the nuget does ship an arm64 import library. Microsoft.AI.DirectML 1.15.4 contains bin/arm64-win/DirectML.lib, a COFF import archive whose machine field is 0xAA64. The real defect was a case mismatch inside ONNX Runtime’s own CMake: cmake/external/dml.cmake declares the download’s outputs with a lower-case bin/arm64-win, while cmake/onnxruntime_providers_dml.cmake composes its consumer paths as bin/${onnxruntime_target_platform}-win — and onnxruntime_target_platform is the verbatim, upper-case ARM64. The two spellings never meet, so the arm64 lane failed with bin/ARM64-win/DirectML.lib ... missing and no known rule to make it, and that was misread as “no arm64 package”. A cross-scoped inline patch lower-cases the redist directory once (string(TOLOWER … onnxruntime_dml_redist_platform)) and routes both consumers through it. The sequencing hold has since cleared: as of #118 (2026-08-24) GenAI builds USE_DML=ON on both lanes and stages D3D12Core.dll through a target-derived filter, and OpenCV’s WITH_DIRECTML is ON on both lanes — it feeds contrib G-API’s ONNX DirectML EP, not cv::dnn. That EP was a stub that throws until 2026-09-23, when OpenCV started building against the chain ORT; its opencv_gapi500.dll now delay-imports dxcore.dll, a new edge for this lane’s import walk that servercore:ltsc2025 resolves (onnxruntime-single-source.md).

USE_CUDA=OFF forced by target, not host (GenAI, and every CUDA consumer)#

Get-GpuEnvironment probes the x64 build host. On a GPU-equipped host it answers “yes” and would switch nvcc on for an aarch64 target. The decision therefore belongs to the target, never to a host GPU probe — and since #176 the cross lane enables CUDA when the IMAGE carries the arm64 payload (Test-CudaWindowsArm64Payload), which Install-Cuda.ps1 -TargetArch arm64 stages. A cross image without it stays CPU + DirectML.

Python bindings ON everywhere since #120 step 2 (2026-08-24 evening): ORT wheel, GenAI wheel, cv2, PyAV — built for the target, staged, never imported#

Every wheel links the target CPython (C:\runtime\python, #120 step 1) while the host interpreter runs the build — Get-TargetBuildPython returns exactly that split (.Exe host, .Include arch-neutral, .Lib/.LibDir target) and Get-SourceBuildPython stays host-pinned. Wheels go through Invoke-PythonWheelBuild -CrossStage (one call for both lanes: on cross it appends the target --plat-name, stages, and hands the wheel to Assert-WheelTargetArch (PE machine and EXT_SUFFIX name tag of every native member); cv2 gets a static gate on cv2.cp314-win_arm64.pyd in the target site-packages. The sitecustomize shim pins EXT_SUFFIX to the target on this lane (the name a target interpreter will import) while get_platform() stays host (what pip resolves downloads with). See § “#120 step 2” below for the three findings this took.

-mllvm -aarch64-enable-compress-jump-tables=false (OpenCV)#

An LLVM AArch64 codegen limitation, not a bug in any of the affected libraries. Switch-heavy TUs overflow a one-byte compressed jump-table entry. REPLACED on LLVM 23.1.0 (2026-08-26) by -Xclang -target-feature -Xclang +force-32bit-jump-tables, the subtarget feature the pass itself consults. It disables the compression pass exactly as this flag does — byte-identical output, verified 2026-08-27 — and was preferred only because a target feature is a supported spelling where -mllvm is a debug knob. The separate branch-range failure in median_blur.dispatch.cpp was handled per-TU with /Ob1 and by no jump-table setting at all. Both workarounds were REMOVED on 2026-08-29, when the patched toolchain below became the default (Build-OpencvFromSource.ps1 carries neither). Heading kept as a live anchor target; see below and failure-modes.md § AArch64 cross compile aborts. ROOT CAUSE, corrected 2026-08-28 — the two failures are ONE defect, not two. An earlier version of this paragraph claimed they were unrelated and that a toolchain move to LLVM main would retire /Ob1; both halves were wrong, and the correction is recorded rather than deleted because the wrong story was acted on. AsmPrinter emits a NOP after an EH_LABEL under async EH (/EHa, which OpenCV passes) while getInstSizeInBytes reports EH_LABEL as a zero-size meta-instruction, so every MIR-level block-size estimate is 4 bytes short per label. The two consumers of that estimate then each pick an encoding the assembler rejects: AArch64CompressJumpTables (value evaluated as <N>) and BranchRelaxation (fixup value out of range). That is the under-counted instruction this paragraph used to say was unidentified. Fixed by windows/scripts/patches/llvm/001-aarch64-ehlabel-size.patch (+ 002 for SEH pseudos) on the pinned LLVM (23.1.0 then, 23.1.1 since 2026-09-18; both patches still apply), filed upstream as llvm#219275 (merged 2026-09-16, but not in llvmorg-23.1.1) and llvm#219276 (still open). Claimed but unlogged: a NINJA_KEEP_GOING=1 census is recorded as having built all 1,869 objects green with BOTH OPENCV_NO_JUMPTABLE_WORKAROUND=1 and OPENCV_NO_OB1_WORKAROUND=1 on a compiler containing no llvm#202716 — but it ran by hand in the container and left no log, and no file in out/windows-build-logs/ mentions either knob. Both knobs went with the workarounds, so it can no longer be re-run as it was. #202716 remains a real upstream defect either way; the evidence that it is not this lane’s cause is the patch set that DID fix the lane, not this census. There is no stock-toolchain fallback any more: Dockerfile.toolchain-builder ships ARG BUILD_PATCHED_LLVM=1 (patched is the DEFAULT since #135), and -StockLlvm builds with the stock scoop clang-cl, which still has the bug and is for patch debugging only (failure-modes.md § AArch64 cross compile aborts). Full evidence: windows-refactor-backlog.md, backlog item #135.

MLAS skip re-gated on WIN32 alone (OpenCV, patch 003)#

The existing Windows skip was gated on WIN32 AND _MLAS_REQUIRES_ASM, and upstream derives that flag from MLAS_X86_64 / MLAS_ARM64 / … — whose detection does not fire for a CMAKE_SYSTEM_PROCESSOR=ARM64 cross configure. The skip silently did nothing on the arm64 lane, MLAS built a C++-only subset whose objects still referenced the GAS-only assembly kernels, and it failed at link with undefined symbol: MlasGemvFloatKernel / MlasHGemmSupported. amd64 is unchanged — _MLAS_REQUIRES_ASM is TRUE there, so both forms of the condition fire identically, and that lane already skipped MLAS.

mlasi.h MSVC intrinsic remap guarded by !defined(__clang__) (OpenCV)#

Bundled MLAS assumes “_M_ARM64 implies the MSVC compiler” and remaps two ACLE reduction intrinsics onto MSVC’s private spellings: #define vmaxvq_f32(src) neon_fmaxv(src). clang-cl defines _M_ARM64 too, but implements the ACLE names and has no neon_fmaxv at all. The upstream #ifndef vmaxvq_f32 guard does not help — clang provides it as a function, not a macro, so the guard is true and MLAS shadows the real intrinsic. The condition being corrected is which compiler, not which architecture, so each #define is wrapped rather than deleted. Dead code (audited 2026-08-24): patch 003’s WIN32 return() fires first, so OpenCV never compiles MLAS — or this header — on either lane; the patch is retained as documentation of the collision, not as a live fix.

have_sse/have_sse2 gated on cpu_family (gst-plugins-base)#

Upstream bug. Its MSVC branch assumes “not x86_64” means “x86 32-bit” and never considers ARM64, so aarch64 falls into the else and gets sse_args = '/arch:SSE'. It then decides purely on cc.has_argument(sse_args) — and clang-cl accepts /arch:SSE for an aarch64 target completely silently, so the x86 SSE resampler sources are compiled for ARM and die in mmintrin.h. have_sse41 already carries the cpu_family guard; the fix just extends it to its two siblings. Nothing is fixable on the meson side: ClangClCompiler.has_arguments already appends -Werror=unknown-argument, -Werror=unknown-warning-option and -Werror=unused-command-line-argument.

Vulkan lib dir follows host_machine (gst-plugins-bad)#

Upstream bug, same class: the wrong machine is asked. vulkan/meson.build picks join_paths(vulkan_root, 'Lib') when build_machine.cpu_family() == 'x86_64' — the machine doing the compiling, which is x86_64 here no matter the target. Because the directory is passed explicitly via cc.find_library('vulkan-1', dirs: …), no LIB ordering can override it. meson itself already gets this right (VulkanDependencySystem maps build x86_64 + host aarch64 → Lib-ARM64); gst-plugins-bad simply computes the path by hand instead of using it.

-FIio.h → assembly-safe shim (GStreamer)#

meson hands c_args to .S files too, so a force-included C header lands in an assembly translation unit and is parsed as instructions (vadefs.h: unrecognized instruction mnemonic — typedef char* va_list;). The shim wraps the include in #ifndef __ASSEMBLER__, which clang defines only for .S. Beyond openh264 this matters for dav1d, libvpx and x264, which all ship aarch64 .S as well — fixing the flag beats disabling one subproject at a time.

--as=clang --target=aarch64-pc-windows-msvc (FFmpeg)#

Assembly is ENABLED, and this row used to say the opposite. Until 2026-08-24 the lane passed --disable-asm on the assumption that aarch64 GAS needed gas-preprocessor.pl driving armasm64. It does not: clang’s integrated assembler accepts FFmpeg’s aarch64 GAS directly, so pointing --as at clang with the target triple is the whole fix. The precondition that makes it safe on a host which cannot execute aarch64 is that --enable-cross-compile disables configure’s runtime probes — configure only ever assembles its test fragments. Measured: NEON enabled yes, and 99 distinct aarch64/*.o objects built through real AS steps. gas-preprocessor.pl + armasm64 remains the documented fallback if a future FFmpeg lands a .S file clang rejects; reaching for a different toolchain does not. amd64 is untouched by this row because the flag lives in a cross-only branch. When this row was written amd64 had no assembler path of its own (--disable-x86asm was appended unconditionally; nasm served only GStreamer’s openh264); backlog #119 closed that asymmetry the same day by enabling nasm-assembled x86asm on the amd64 lane, while the cross lane keeps --disable-x86asm explicitly — it is an x86-only knob.

The OpenCV softfloat / NEON collision#

Worth spelling out, because the fix looks like a stylistic nit and is not.

modules/core/src/softfloat.cpp:163 does, inside namespace cv:

typedef softfloat  float32_t;
typedef softdouble float64_t;

intrin_neon.hpp also lives in namespace cv, so unqualified lookup for float32_t finds cv::float32_t before the ::float32_t (= float) that clang’s arm_neon.h declares. Since clang 16 the NEON lane accessors are macros of the form

__ret = __builtin_bit_cast(float32_t, __builtin_neon_vgetq_lane_f32(__s0, __p1));

and cv::softfloat has a user-provided copy constructor, so it is not trivially copyable — exactly what __builtin_bit_cast requires. Every v_extract_n instantiated in that translation unit fails with '__builtin_bit_cast' destination type must be trivially copyable. x86 is unaffected because the SSE intrinsics never mention float32_t, which is why this stayed invisible on the amd64 lane.

The fix converts the two typedefs into object-like macros, and that distinction is the mechanism: intrin_neon.hpp is textually preprocessed at #include "precomp.hpp" on line 66, long before line 163, so the template bodies keep a bare float32_t token that a later #define can no longer rewrite. Removing the typedef leaves nothing named cv::float32_t, so that token resolves to ::float32_t as clang intends — while every use after line 163, i.e. all of the ported Berkeley SoftFloat code, still expands to softfloat exactly as the typedef did. The file has no #include after line 163, so the macro cannot leak into a header.

The patch carries a throwing drift assertion rather than a warning: a silent no-op here resurfaces as a wall of confusing bit_cast errors deep inside arm_neon.h, which is precisely the failure mode worth spending an explicit throw on.

AArch64 compressed jump tables#

Switch-heavy translation units abort the compile with a diagnostic that names no source file:

generated_message_reflection.cc: error: value evaluated as 284 is out of range.
descriptor.cc:                   error: value evaluated as 262 is out of range.
grfmt_tiff.cpp:                  error: value evaluated as 256 / 272 is out of range.
gapi serialization.cpp:          error: value evaluated as 259 is out of range.

AArch64AsmPrinter::emitJumpTableImpl emits a compressed jump-table entry as (LBB - Base) >> 2 through MCObjectStreamer::emitValueImpl with Size = 1, and that is the only producer of this message in LLVM. Every value measured lands just past 255 * 4 = 1020 bytes of table span:

Reported

× 4

vs. 1020

256

1024

just over

259

1036

just over

262

1048

just over

272

1088

just over

284

1136

just over

Under LLVM 22 the fix was one arm64-only flag that keeps full /O2 — uncompressed jump tables are larger, not slower:

-mllvm -aarch64-enable-compress-jump-tables=false

SUPERSEDED on LLVM 23.1.0 (2026-08-26); this note CORRECTED 2026-08-27. The flag above is no longer set; the lane then passed -Xclang -target-feature -Xclang +force-32bit-jump-tables instead, which reaches the same end state by a supported spelling, and since 2026-08-29 it passes neither: the patched toolchain fixes the size estimate both overflows came from. The measurement table above still describes a real mechanism — it is simply no longer suppressed.

Two claims this note used to make were disproved on 2026-08-27, and both had cost runs: that the feature and the -mllvm flag differ in what they leave enabled, and that the difference explained a second failure in OpenCV’s largest dispatch TUs. It did not — that failure is a branch-range defect with no jump table in it, fixed per-TU with /Ob1. failure-modes.md § AArch64 cross compile aborts owns the whole account: both diagnostics, what each flag actually does to the pass, the four “do NOT”s and the local-reproduction recipe. Read it there rather than trusting a summary here.

This corrects an earlier, wrong diagnosis recorded here. The first explanation blamed an 8-bit Code Words field in the .xdata unwind record, and the workaround was a per-TU /Od pass over build.ninja. That story was refuted from primary source: MCWin64EH.cpp reports the Code-Words ceiling as report_fatal_error("SEH unwind data splitting is only implemented for large functions ...") — a different message entirely, and the string is out of range does not appear in that file at all.

The jump-table explanation also accounts for something the unwind story never did. Escalating /Ob2 → /Ob1 → /Ob0 kept moving which TU overflowed instead of fixing it, and only /Od cleared every one. That is because AArch64CompressJumpTables is added only when getOptLevel() != None: /Od disabled the pass as a side effect of turning optimisation off wholesale, while the /Ob* levels merely reshuffled codegen. The right conclusion from that measurement was “something only /Od switches off”, not “something inlining drives”.

The observation that every offender was serialisation, parsing or decoding code was correct but incidental: the common thread is big switch statements, not when the code runs.

The meson cross file: --target belongs in [binaries]#

The single most load-bearing line in the GStreamer cross setup is easy to get wrong, and getting it wrong produces a green configure followed by machine-type conflicts everywhere:

[binaries]
c   = ['sccache', 'clang-cl', '--target=aarch64-pc-windows-msvc']
cpp = ['sccache', 'clang-cl', '--target=aarch64-pc-windows-msvc']

[host_machine] cpu_family does not drive /MACHINE — that was the wrong assumption, and it only steers the meson.build tree’s own arch branches. What actually decides is the triple the compiler reports. Verified against meson 1.12.0:

Where

What it does

compilers/detect.py:439

runs <exelist> --version and parses ^Target: (.*?)-

compilers/mixins/visualstudio.py:114

elif 'aarch64' in target: self.machine = 'arm64'

compilers/mixins/visualstudio.py:123

self.linker.machine = self.machine — the dynamic linker inherits it

compilers/detect.py:224

VisualStudioLinker(linker, env, getattr(compiler, 'machine', None)) — and so does the archiver

With --target only in -Dc_args, clang-cl --version reports the host triple, meson canonicalises that to x64, and every static archive and DLL is built /MACHINE:x64 while the objects are aarch64:

libgnulib.a.p/asnprintf.c.obj: file machine type arm64 conflicts with library machine type x64

Note meson skips /MACHINE entirely when the machine is unknown (ClangClDynamicLinker.get_output_args: “If we’re being driven indirectly by clang just skip /MACHINE, as clang’s target triple will handle the machine selection”), so a missing /MACHINE is fine — a wrong one is not.

The duplicate --target in -Dc_args/-Dcpp_args stays deliberately: those keep the compile flags correct even for subprojects that rebuild the command line, and the command line beats a machine file’s [built-in options], which is why the triple cannot live there.

The target CPython is built from source (#120 step 1)#

Since 2026-08-24 the media-core stage builds CPython itself for the target — PCbuild\build.bat -e -p ARM64 under the repo’s ClangCL props plus /p:PreferredToolArchitecture=x64 — and stages 2864 files to C:\runtime\python: the interpreter, python314.lib, headers and the stdlib. The whole build took ~91 s including the externals fetch, and python.exe’s PE machine (0xAA64) is verified in-stage, not left for the merge gate to find.

This answers #114’s Phase-0 Q1 positively: VS 18 does ship the ClangCL PlatformToolset for ARM64. The proof is pythoncore’s own warning — “Toolset ClangCL is not used for official builds” — firing, which requires the toolset to have resolved.

One host-arch artifact still got in, and the extended arch gate earned its keep catching it: vcruntime140_1.dll has no ARM64 edition by design — it exists solely to carry the x64 FH4 exception helpers, and ARM64 keeps everything in vcruntime140.dll. MSBuild’s host-blind redist copy dropped the x64 one into the ARM64 output; the gate flagged it as the single violation in a 932-binary run, and the stage now self-polices every staged binary’s PE machine and drops exactly that file, under a tightly-guarded rule rather than a pattern that could grow.

What this deliberately did not include at first: the consumers. They followed the same evening as step 2.

A free-threaded twin since 2026-10-07 (CON74). The same stage builds the checkout again with --disable-gil and stages it into C:\runtime\python-freethreaded through the same staging function: windows-builds.md § The free-threaded CPython. Its python3XYt.lib links the cp3XYt twins of the chain wheels in C:\runtime\wheels-free-threaded (CON79): windows-builds.md § The arm64 cross twins.

From 3.15 the free-threaded build also writes a python.exe and a pythonw.exe. CPython’s CopyFreethreadedBinary target then duplicates each to its X.Yt name. The staging drops the plain names only when they are byte-identical to that copy, because the tree must not answer a GIL request for python.exe. Any other python.exe still stops the stage. The first 3.15 arm64 chain (2026-10-15) stopped there before the staging learned this.

The Python consumers are built for the target (#120 step 2)#

Since 2026-08-24 evening (arm64 run 3) the four consumers build against the target CPython: onnxruntime-1.29.0-cp314-cp314-win_arm64.whl (4 native members), onnxruntime_genai_directml-0.15.2-cp314-cp314-win_arm64.whl (3), av-18.1.0-cp314-cp314-win_arm64.whl (49) — all members 0xAA64, all staged in C:\runtime\wheels — and cv2.cp314-win_arm64.pyd installed into C:\runtime\python\Lib\site-packages. None of it is installed into, or imported by, any interpreter in the build: the only interpreter that can run here is x64.

The split that makes it correct. One interpreter runs the builds (host, x64 — pybind11 and numpy probes execute it, bdist_wheel zips with it); a different one is linked (target — python314.lib from PCbuild\arm64). Get-TargetBuildPython encodes that as .Exe (host), .Include (arch-neutral headers), .Lib/.LibDir (target) with an .Available guard, and every consumer passes exactly those. On amd64 the accessor collapses to the host values, so the native lane is byte-identical.

Three findings from the two failed runs before the green one — each now pinned in code or test:

  1. ORT’s CMake reads Python_*, not Python3_*. The Python3_EXECUTABLE/INCLUDE_DIR/LIBRARY hints passed for months were ignored on every lane — CMake printed “Manually-specified variables were not used by the project” each time — and amd64 only worked because FindPython auto-detected the host interpreter and its numpy. Cross had no luck to fall back on: Target "onnxruntime_pybind11_state" links to Python::NumPy but the target was not found. The script now passes Python_* plus an explicit Python_NumPy_INCLUDE_DIR probed by running the host interpreter. GenAI needs both spellings: Python_* for its own find_package(Python), and the legacy PYTHON_* trio for its vendored pybind11 in classic mode (FindPythonLibsNew.cmake: Python libraries not found without it). SourceBuild.FindPythonPrefix.Tests.ps1 pins the spellings case-sensitively (-cmatch — PowerShell’s -match would call Python_ and PYTHON_ the same name).

  2. if (Test-WindowsCrossTarget -and -not $tpy.Available) is parsed in command mode. A condition whose first token is a command name makes -and, -not and $tpy.Available arguments of that command; no error, the branch fires. Run 2 reported “6. python wheel 0s” with the bindings ON. Fix: a variable first ($onnxCross -and …). Probed and confirmed on the host before fixing.

  3. The host-pinned sitecustomize shim stamped the host EXT_SUFFIX on target modules. cv2.cp314-win_amd64.pyd, machine 0xAA64 — right bytes, a name no arm64 interpreter will ever import (_imp.extension_suffixes() there is .cp314-win_arm64.pyd / .pyd). The shim now sets sysconfig.get_config_vars()['EXT_SUFFIX'] to the target suffix on the cross lane and leaves get_platform() host-pinned, because those are two different facts: the first names what this interpreter builds, the second what pip may download for it. Verified standalone under a host python (EXT_SUFFIX = .cp314-win_arm64.pyd, get_platform = win-amd64, import suffixes untouched) before the run, and Assert-WheelTargetArch + the cv2 gate now check the name tag as well as the machine field.

Two more facts worth keeping: PyAV is a setuptools extension, so it is compiled by cl.exe via setuptools’ x86_arm64 vcvars spec (build_ext --plat-name win-arm64 selects it) — the documented PyAV-shaped hole in the clang-cl rule, on both lanes; and the -CrossStage staging path is the cross lane’s replacement for Install-StagedPythonWheel/Test-PythonImport, which stay mandatory on amd64 (the same switch takes that native path there).

A consumer’s test runner, pinned (CON67, 2026-10-06)#

The wheel store also carries the pinned cpXY pytest stack a consumer runs its suite with on the device. That is pytest 9.1.1, pytest-cov 7.1.0, pytest-benchmark 5.2.3, pytest-html 4.2.0, pytest-md 0.2.0, pytest-md-report 0.8.0 and requests 2.34.2, with their dependency closure: the versions OrchestrANT’s lock runs on x64. Before, OrchestrANT’s Stage-Arm64Tests.ps1 fetched them from PyPI at cross-build time, unpinned.

  • Copy-Arm64TorchWheels.ps1 downloads each wheel against its PYTEST_WINDOWS_ARM64_*_SHA256 pin in versions.env.

  • The three native ones (charset-normalizer, coverage, MarkupSafe) also pass Assert-WheelTargetArch.

  • jinja2 and setuptools are the torch stack’s files, so the stack has no pins of its own for them.

  • Test-Arm64Bundle.ps1 installs the stack offline into the bundle’s interpreter and imports it, in a bundle that carries it.

Cross machinery the LiteRT branch added#

Plain LiteRT (#115) went green on 2026-08-24 — the merge now carries its 146 staged libs, including an aarch64 tensorflowlite_c.lib. The fixes that got it there generalize; each is recorded with its one-line “why”.

Two host tools, both from pinned sources. Cross-compiling LiteRT needs a host flatc and a host protoc. flatc is built natively from the same vendored tree (the flatbuffers-flatc target, with a per-call -TargetArch host override at the choke point) — because generated code and schema compiler must come from the same flatbuffers version. protoc is the 21.9 GitHub release zip (since pinned as LITERT_TFLITE_PROTOC_VERSION in versions.env), its version derived from the vendored protobuf commit (90b73ac3 = C++ runtime 3.21.9) — not the LM lane’s PROTOC_VERSION (31.1 then, 36.1 today), whose generated code includes google/protobuf/runtime_version.h, a header 3.21.9 does not ship. The vendored runtime picks the protoc family; nothing else may.

XNNPACK gets the MLAS-class per-TU treatment (Build-LitertFromSource.ps1). 569 C microkernel TUs are tagged per feature family in build.ninja post-configure — families completed against upstream’s PROD_* source lists: scalar fp16arith, neonfp16, neonfp16arith, neondot, neondotfp16arith, neonbf16, neoni8mm, neoni8mmbf16; SME skipped — with a floor of 100. The 335 hand-written aarch64 .S kernels instead get a full-union in-source directive, .arch armv8.2-a+fp16+dotprod+i8mm+bf16, floor 10. The asymmetry is the point: an assembler only validates — it never emits an instruction the source does not contain, so the union is byte-neutral for asm — while a compiler may auto-vectorize, so C must stay per-family. Same reason MLAS is per-TU: these kernels are runtime-dispatched.

The ASM language now gets the target triple — chain-wide. Get-CMakeCrossArgs (WindowsTargetArch.Common.psm1) also sets CMAKE_ASM_COMPILER_TARGET and CMAKE_ASM_FLAGS_INIT. Before that, any project enabling the ASM language assembled with the x64 default target (brackets expression not supported on this target), and an aarch64 -march handed to that x86-targeting driver was misread as a CPU name (unknown target CPU 'armv8.2-a+fp16') — diagnostics that sent the first analysis chasing a driver gap that does not exist. One choke-point line fixes every ASM-enabling project at once; amd64 is untouched because its cross args stay empty.

The tflite plugin is presence-driven, and the cross gate now proves plugin-ness. GStreamer’s configure gets -Dgst-plugins-bad:tflite=enabled when LiteRT is staged — never auto, because auto degrades a missing dependency into a silently absent mandatory plugin. The hardened cross plugin gate walks each plugin’s dependency tree (dumpbin) and asserts the per-plugin export marker gst_plugin_<name>_get_desc — because a DLL with the right name is not yet a plugin. Measured recalibration (2026-08-24): modern GStreamer (per-plugin registration since 1.14) exports gst_plugin_<name>_get_desc + _register, not the legacy gst_plugin_desc this gate first asserted — that first version failed all four plugins, including three amd64-proven ones, and was recalibrated from the dumped export tables.

Iteration honesty. The litert+merge green took eight iterations with named fixes. Two were self-inflicted (a PowerShell plus-sign-outside-parameter binding bug; the wrong legacy export-symbol assumption above); the rest were genuine cross gaps — per-TU kernel features, the ASM triple, the host flatc, the host protoc and its version family, and the vcruntime redist copy from the CPython section.

The merge stage on arm64#

The merge fans three branch trees into one install prefix, builds GStreamer from source (~4965 targets on this lane), and then runs the PE architecture gate — over C:\runtime and the host CPython’s C:\temp\cpython\Lib\site-packages, with -IncludeArchives. Measured 2026-08-24: 931 binaries inspected, 0 violations, the 58 host .pyds reported as allowlist skips. That is the whole-image statement backlog #117 asked for, and it resolves #117’s gate-scope half; the other half — “is C:\temp payload?” — is settled by #120: the shipped interpreter on arm64 is C:\runtime\python, and the host CPython stays build tooling.

Since 2026-08-24 (#115) media-litert is a real branch on this lane — plain LiteRT cross-builds; see the machinery section above. Inside it, the LiteRT-LM stage self-skips (its two real Bazel blockers are recorded in the exclusion table) and stages the empty litert-lm stand-in tree itself — Build-LitertAll.ps1’s skip path — so the merge’s unconditional COPY still finds every path it expects. No branch is absent any more (2026-08-24 evening): media-tvm — which also carries IREE — cross-builds runtime-only (#116, see its own section), so the merge fans in three real images on both lanes. History of this paragraph, kept because it was wrong twice: the first version said TVM’s codegen “cannot emit aarch64” (false — the LLVM target list is this repo’s own array), the second said the branch was blocked on executing llvm-config (true for the compiler, irrelevant for the runtime the lane now ships).

Dockerfile.media-merge-builder copies from both with unconditional COPY --from=media-litert / --from=media-tvm, and a Dockerfile cannot make a COPY conditional. Until 2026-08-24 evening a media-branch-absent stand-in stage stood in for whatever the lane could not build (first both branches, then only media-tvm), providing the expected paths empty plus an ABSENT-ON-ARM64.txt marker in each. That stage is retired (#116 made media-tvm real, runtime-only — see the next section): $script:MergeRequiredBranches is the same three-branch list on both lanes, and the surviving convention is that a branch which cannot build a component ships its own empty, marker-carrying tree — Build-LitertAll.ps1 does exactly that for LiteRT-LM, and Build-TvmFromSource.ps1 / Build-IreeFromSource.ps1 drop COMPILER-ABSENT-ON-ARM64.txt next to their runtime installs. The merge’s pointer contract is unchanged: TVM_LIBRARY_PATH / IREE_BIN must resolve on both lanes, and smoke section 19 asserts it (the first arm64 smoke run found exactly that pair dangling).

TVM and IREE cross-build runtime-only (#116, 2026-08-24)#

The blocker recorded for media-tvm was real but narrower than “the branch”: USE_LLVM=<path> makes TVM execute llvm-config at configure time and link target-arch LLVM libraries into tvm_compiler.dll, and IREE’s compiler needs its in-tree LLVM for the target too. Neither is needed to run compiled modules. So the cross lane builds the runtimes and names the compilers absent:

Component

arm64 cross build

Shipped

Named ABSENT

TVM

USE_LLVM=OFF, TVM_BUILD_PYTHON_MODULE=OFF, ninja tvm_runtime only (the -Targets parameter added to Invoke-NinjaBuildWithRetry for this), manual stage: lib\tvm_runtime*.dll/.lib, lib\tvm_ffi*.dll/.lib, include\{tvm,dlpack,dmlc}; Vulkan lib dir arch-aware (Get-VulkanLibDirName)

C:\runtime\lib\tvm\{lib,include} — every DLL PE-gated in-stage

tvm_compiler.dll, the tvm/tvm_ffi python packages (COMPILER-ABSENT-ON-ARM64.txt)

IREE

Two configures of the same tree, upstream’s documented recipe: (A) a native x64 runtime-only build into build-host\, installed, asserted to contain iree-flatcc-cli.exe + iree-c-embed-data.exe; (B) the target configure with IREE_HOST_BIN_DIR=<A>/install/bin, IREE_BUILD_COMPILER=OFF, python OFF, Vulkan HAL ON

C:\runtime\iree\{bin,lib,include} — every .exe/.dll under bin\ PE-gated in-stage (a host tool leaking into the target install is the failure this catches)

iree-compile.exe, iree.compiler / iree.runtime wheels (COMPILER-ABSENT-ON-ARM64.txt)

The functional compile+run gate (iree-compile → iree-run-module, abs(-5)=5) runs only on amd64; on arm64 it is replaced by the static PE gate, and the execution proof stays owed to a native host like every other arm64 signal.

Four findings from the first runs (3–6), each fixed in code and, where it is a rule, in a test:

  • TVM 0.26 header layout: dmlc-core is gone, dlpack lives inside the tvm-ffi submodule’s own 3rdparty, and c_runtime_api.h no longer exists (its C surface is tvm\ffi\c_api.h). The copy is layout-searched and asserts tvm\runtime\c_backend_api.h, tvm\runtime\device_api.h, tvm\ffi\c_api.h, dlpack\dlpack.h.

  • The host pass needs the HOST’s LIB, not just the host target. VsDevCmd -arch=arm64 leaves LIB on the ARM64 CRT, CMake invokes lld-link directly (no clang-driver auto-detection), and the first try-compile died with msvcrtd.lib(exe_main.obj): machine type arm64 conflicts with x64. Invoke-WithHostArchLibraryEnvironment rewrites the arch segment of every LIB/LIBPATH entry for the block and restores it (SourceBuild.HostArchLibEnv.Tests.ps1). Exactly the “necessary but not sufficient” the choke point’s comment predicted for -TargetArch (Get-WindowsHostArch).

  • IREE 3.x host tool names: iree-c-embed-data.exe, not generate_embed_data.

  • Upstream gap — IREE_HOST_BIN_DIR on a Windows host: IREE composes ${IREE_HOST_BIN_DIR}/iree-flatcc-cli without .exe, and ninja wants that exact file (missing and no known rule to make it). The script stages a suffix-less twin beside each host .exe; CreateProcess appends .exe for an extension-less full path, so both names run the same bytes. Draft issue: out/upstream-issue-iree-host-bin-dir-exe.md.

  • IREE’s arm_64 ukernels need per-TU feature flags under clang-cl — the third instance of the MLAS/XNNPACK class. Upstream hands mmt4d_arm_64_{fullfp16,fp16fml,bf16,dotprod,i8mm}.c their -march=armv8.2-a+<feat> via iree_select_compiler_opts(CLANG_OR_GCC …); clang-cl is classified as MSVC there, the flags vanish, and the kernels die with always_inline function 'vfmaq_f16' requires target feature 'fullfp16'. mmt4d_arm_64_fp16fml.c additionally uses the bare GNU asm keyword, which MS compat turns off. Post-configure build.ninja tagging appends /clang:-march=armv8.2-a+<feat> to exactly those five TUs (floor 5; the pre-fix state tags 0) and -Dasm=__asm__ to every arm_64 ukernel TU.

  • Upstream bug — ARM64 matches MATCHES 64: hal/local/elf/CMakeLists.txt adds the x86-64 MASM trampoline object (arch/x86_64_msvc.obj) whenever MSVC_C_ARCHITECTURE_ID MATCHES 64, so the ARM64 archive got an x64 object (file machine type x64 conflicts with library machine type arm64). Same “MSVC implies x86” class as gst-plugins-base’s have_sse. Inline patch to an exact x64 match, on both lanes, verified after applying. Prepared as a PR, not posted: windows/upstream/iree-elf-arch-x64-match/.

  • C99 inline linkage in one ukernel: with everything compiled, the tools failed to link on iree_uk_mmt4d_tile_s8s4s32_1x8x16_arm_64_i8mm — the single non-static C inline definition in the arm_64 set (checked file by file), whose address the entry point takes; C99 inline semantics emit no external symbol for it. -fgnu89-inline on that TU did not help under clang-cl (run 10); the definition is inline-patched to a plain external function (inline buys nothing for a function used by address), verified post-patch. (How upstream links this on Linux/clang was not verified — a question, not a claim.) Prepared as a PR, not posted: windows/upstream/iree-ukernel-i8mm-linkage/.

First full run: 2026-08-24 evening (arm64 run 11) — see the status banner for the result.

QNN execution provider (#121, 2026-08-24; SDK staged + multi-framework 2026-08-29)#

The Qualcomm AI Engine Direct SDK is login-gated, so the wiring is opt-in by staging a zip: drop it into windows/qnn-sdk/ (git-ignored except its README; bind-mounted into the media-core onnx, genai, litert, and tvm RUNs at C:\temp\qnn-sdk), optionally pin it with QNN_SDK_ZIP_SHA256 in versions.env. Resolve-QnnSdk (in WindowsSourceBuild.Common.psm1) extracts it, anchors the SDK root on include\QNN\QnnInterface.h, asserts lib\<arch>\QnnCpu.dll exists for the target, and checks version compatibility against the ORT build (QNN_OP_STFT canary — falls back to QNN-off if the SDK is too old). Copy-QnnRuntime stages the per-arch backend DLLs (QnnHtp*.dll, QnnCpu.dll, QnnSystem.dll) plus hexagon-v* skel dirs beside each framework’s install.

The QNN SDK gives a build-time flag to ONE framework (corrected 2026-08-31, backlog #154 — the other three rows record what was wrong):

Framework

CMake flag

What it enables

ONNX Runtime

onnxruntime_USE_QNN=ON

ORT QNN execution provider — NPU inference for ONNX models

ONNX Runtime GenAI

(inherits from ORT)

QNN runtime DLLs staged beside GenAI install

LiteRT

(none — see #154)

Nothing. TFLITE_ENABLE_QNN was invented; the real switch is LITERT_ENABLE_QUALCOMM in the litert/ tree, which this lane does not configure

TVM

(none — see #154)

Nothing. TVM has no QNN option at all; its qnn is the Quantized-Neural-Network dialect, and its Snapdragon path is the separate Hexagon SDK

IREE

(none — see #154)

Nothing. IREE has never had a Qualcomm backend

No zip = QNN off with one notice on every framework. A version-mismatch (SDK too old for the framework version) also falls back to QNN-off gracefully. The SDK staged on this host is QAIRT 2.44.0.260225 (QNN API 2.33.0), which carries QNN_OP_STFT, QNN_OP_RANDOM_UNIFORM_LIKE, and QNN_OP_SCATTER_ELEMENTS_REDUCTION_MAX — all required by ORT 1.29. The QNN EP built with it on 2026-08-31 (§ QNN staging below). The verification ceiling is DirectML’s: a green build proves the right bytes ship, never NPU execution.

The mandatory GStreamer plugin contract demands all four plugins on BOTH lanes again (six since #128, 2026-08-25, when webrtc and nice joined). The UnavailableOn.arm64 entry that dropped tflite while LiteRT was a stand-in was deleted on 2026-08-24 (#115) — the plugin is mandatory once more, and the green run proves it present. The arch-filter mechanism stays where it always was: in Get-RequiredGstPlugin -Arch, ONE place, because the contract has three consumers (build gate, smoke test, healthcheck) and them disagreeing is the documented 2026-07-11 regression that shipped an image without plugins. With no entry carrying an arch key any more, the filter is provably a no-op on both lanes: every entry comes back in the same order.

Verification cannot mean “run it”#

Eight of the nineteen blockers found in the merge/final audit were the same class: code that executes arm64 binaries on the x64 build host. The GStreamer post-install gate runs gst-inspect-1.0.exe; the smoke suite — 23 sections, ~100 assertions — is dominated by payload execution: LoadLibraryW over every shipped DLL, compile-and-run native probes, ffmpeg -version and its kin. None of those are checks that fail on arm64 — they are checks that cannot exist there, all failing for the same uninformative reason.

So on the cross lane:

  • The GStreamer gate asserts the plugin DLL was produced, walks its dependency tree with dumpbin, and asserts the per-plugin export marker gst_plugin_<name>_get_desc (see the machinery section for its measured recalibration) — and still says explicitly (cross lane - load probe impossible on an x64 host) why it stops there. Whether it is the right machine is Test-TargetArch.ps1’s job.

  • Since 2026-08-24 the smoke gate’s host-toolchain sections (1-6, 7 on the GPU lane, 14-16, and 19, arch-filtered: TORCH_APP_DIR is dropped; since then also the static sections 23 and 25) run on this lane against their own floors — measured 127 passed / 0 failed / 15 skipped on the 2026-09-21 green run (97 before the CUDA/Hailo sections landed) — and the payload sections are skipped as sections, reported NOT APPLICABLE, not “passed”. (Until then the whole gate was NOT APPLICABLE.) Sections 14/15 were not “unchanged, they just run”: the final image bakes VSDEVCMD_ARCH=arm64, so a bare clang-cl — x64 default target — fought the ARM64 environment libraries, measured as the first arm64 smoke run’s 90/7/13. They now compile for the target and assert the produced PE machine instead of running the probe; ASAN is skipped (LLVM’s win-x64 package ships no aarch64-windows ASAN runtime), and the arm64 floor for section 14 is the measured 2.

  • meson install runs with --destdir so meson skips post-install scripts that would have to run target binaries — see the DESTDIR section below.

  • The shipped image’s HEALTHCHECK skips payload execution. windows/Dockerfile declares it unconditionally (a Dockerfile cannot branch on an ARG) and the same file produces :winarm64, so without this the bundle would sit permanently unhealthy, retrying failing checks every five minutes forever. Test-Health.ps1 reads the baked WINDOWS_TARGET_ARCH; since 2026-08-24 it still runs its four host-tool checks on arm64 and skips only payload execution, reporting that the payload is a cross-compiled artifact bundle, deliberately not runnable.

The amd64 smoke floors are deliberately never lowered for arm64. Since 2026-08-24 the arm64 lane carries its own floor column — a third column in the floor table, sized for the host-toolchain sections it actually runs (floors 76/20; the green 97/0/15 was measured under 69/20, before sections 19 and 25 grew) — and the amd64 numbers stay untouched. A shared, reduced -SmokeMinPassed would leave a number that a later amd64 change could quietly be measured against — which is exactly how the gate documented at backlog #44 became decorative once before.

Two silent failure modes the audit caught#

Both would have produced a green result:

  1. compiler-rt was selected arch-blind. The old code took Select-Object -First 1 over every *builtins*.lib and handed the path straight to lld-link. LLVM ships one per target, and on the cross lane the x86_64 one sorts first — so an aarch64 image would have been linked against the host’s runtime. The selection is now filtered by target, and when no match exists the lane links nothing rather than the wrong thing. See the next section: the gap turned out to be real, and it is now filled. The mirror-image bug then bit amd64 (2026-08-24): the first fix filtered only the cross branch, on the rationale that this “keeps the amd64 selection exactly what it is today” — written while the x86_64 lib was the only one installed. Once the base started shipping clang_rt.builtins-aarch64.lib (the #113 ride), amd64’s alphabetical -First 1 flipped to aarch64, and the first amd64 merge on that base died linking gstreamer-1.0-0.dll with machine type arm64 conflicts with x64 — caught by the full-chain amd64 regression run, exactly what it exists for. The pick is now target-filtered on BOTH lanes; a selection that depends on what happens to be installed is not a selection.

  2. The arch gate’s coverage floor could silently disable itself. It arrives as -MinInspected ([int]$env:ARCH_GATE_MIN_INSPECTED), and [int]$null is 0, which switches the floor off entirely — so a dropped build-arg would turn the gate into a clean pass over whatever it happened to find. “No floor” is now an explicit -AllowEmptyTree opt-in, and the driver passes a real floor (the Dockerfile default of 10 is far below what a complete bundle contains, and could not detect losing a whole component): 100 on the cross lane at first, today 580 on arm64 and 650 on amd64 (Build-Buildkit.ps1, measured counts minus headroom).

aarch64 compiler-rt is a base prerequisite#

scoop’s main/llvm is the x64 Windows release, and LLVM ships compiler-rt for the host architecture only: the install contains clang_rt.builtins-x86_64.lib and nothing else. Test-Arm64Prereqs.ps1 Q5 has checked for the aarch64 counterpart since this lane was designed, and reported [FAIL] the whole time.

That is not cosmetic. clang lowers 128-bit integer arithmetic to compiler-rt libcalls on aarch64, so the first component that does 64×64→128 math fails at link, not compile:

lld-link: error: undefined symbol: __udivti3
>>> referenced by gstutils.c:670 (gst_util_uint64_scale)

ONNX Runtime, FFmpeg and OpenCV happen not to need it. GStreamer does — which is why the gap stayed invisible until the merge stage.

Install-ScoopTools.ps1 now fetches the official clang+llvm-<ver>-aarch64-pc-windows-msvc.tar.xz, extracts only clang_rt.builtins-aarch64.lib (280 KB out of a 704 MB archive), drops it beside the host library, and deletes the archive. Measured on the build host: 0.8 min to download, 0.4 min to extract.

Three details worth keeping:

  • The destination is derived from the host library’s own directory, never hardcoded. That is by construction the directory clang and every consumer already search, so no discovery logic has to learn a new path.

  • It installs unconditionally, like the MSVC ARM64 toolset and the Vulkan ARM64 component. Gating it on an arch ARG would re-pay the chain’s most expensive layers on every lane switch. It sits after Install-Vs.ps1 in Dockerfile.base, so adding it does not invalidate the VS layer.

  • The URL must encode the + as %2B. That is the canonical browser_download_url the GitHub release API returns, and the unencoded form 404s. Note also that GitHub refuses HEAD on release assets, so a failed HEAD is not evidence the asset is missing — verify with a ranged GET instead.

The patched-LLVM toolchain (#135) does not carry it — GStreamer self-heals#

BUILD_PATCHED_LLVM=1 (the default since 2026-08-29) builds clang/LLVM from source, and that build emits compiler-rt builtins for the host arch only (-DLLVM_ENABLE_RUNTIMES=compiler-rt), so C:\llvm-patched\lib\clang holds clang_rt.builtins-x86_64.lib and nothing else. The arm64 GStreamer link then fails exactly as above — found on the 2026-08-30 arm64 cross run (merge stage, __udivti3 undefined linking gstreamer-1.0-0.dll), with the script’s own warning naming the cause.

Rather than rebuild the whole chain to add the lib to the toolchain layer (the media branches derive FROM bk-windows-toolchain, so one added layer re-pays ~2 h of media compiles), the GStreamer merge stage self-heals: Build-GstreamerFromSource.ps1 § 5d, on the cross lane only, mines clang_rt.builtins-aarch64.lib from the official release archive (same URL/recipe as Install-ScoopTools.ps1) next to the x86_64 lib, then re-runs its candidate search. The existing warn-and-link-without policy stays for the case the fetch fails. Regression: SourceBuild.GstreamerCompilerRt.Tests.ps1. The toolchain-level fix has landed since (heading kept as an anchor): Build-LlvmFromSource.ps1 stages clang_rt.builtins-aarch64.lib into C:\llvm-patched from the same release archive (Install-AArch64CompilerRt, pinned by LLVM_WINDOWS_AARCH64_RT_SHA256), and re-checks a cached tree from before 2026-08-31 under -SkipIfPresent. The merge-stage self-heal stays as the fail-open fallback.

opus NEON intrinsics stay DISABLED on the cross lane (enablement reverted 2026-08-31)#

The speculative cross-lane enablement (-Dopus:intrinsics=enabled, added 2026-08-30) was proven broken twice once the compiler-rt fix let the GStreamer build actually reach opus. Under clang-cl aarch64, the RTCD path (default) applies -mfpu=neon — an ARM32-only flag clang-cl rejects for aarch64 (unsupported option '-mfpu='), and the RTCD CPU probe celt/arm/armcpu.c uses MSVC’s __emit intrinsic, which clang-cl does not implement. The lane is back on the 2026-08-26 proven shape: -Dopus:intrinsics=disabled on BOTH lanes (the scalar opus codec is fully functional).

The working enablement recipe for a future dedicated test window: -Dopus:intrinsics=enabled -Dopus:rtcd=disabled — with RTCD off, a clang-cl aarch64 build presumes SIMD (opus_can_presume_simd = true), so no -mfpu= args are applied and the __emit-based arm_armcpu.c is not compiled. The tradeoff to verify on a real device: dotprod and NEON are presumed unconditionally, so the image is Snapdragon-class-only (all current Windows-on-ARM devices qualify, but it is not a universally-safe default). Re-enable only with a smoke-tested device run. Tracked in docs/windows-refactor-backlog.md #135 follow-up.

The staged QNN runtime also extended the merge gate’s knowledge: the QAIRT HTP stub DLLs (QnnHtpV*Stub.dll, calculator*.dll — staged beside every framework by Copy-QnnRuntime) import libcdsprpc.dll/libadsprpc.dll, Qualcomm’s FastRPC ADSP/CDSP drivers. Those are device-OS libraries: present in every Windows-on-Snapdragon image, never in the SDK zip and never on the Server Core reference host — so Test-TargetArch.ps1’s ClientOsPattern now allows them, and the arch gate reports the QNN payload as inspected PE + resolved imports instead of 63 phantom unresolved edges.

WebRTC on the cross lane (CON28, 2026-10-05)#

The plugin contract is the same thirteen entries on both lanes, so the arm64 bundle needs dtls, srtp, sctp, rtpmanager, openh264, rswebrtc and rsrtp as well. The cross gate proves each statically, as before: the DLL by its exact name gst<name>.dll, its dependency tree, and the gst_plugin_<name>_get_desc export, which the gst-plugins-rs DLLs carry too (gst_plugin_rswebrtc_get_desc, checked on the amd64 build). The two Windows WebRTC defects and their fixes reach this lane like this:

  • libffi has the same zeroed type table on aarch64: aarch64/ffi.c and the four common files include the same ffi.h. -fcommon arrives through the same -Dc_args string, and /FORCE:MULTIPLE is gone from the host link args here too. The build machine’s libffi needs it separately: meson puts build.subprojects/libffi/src/ffi-7.dll into the default all target and links it with the native file’s linker (measured 2026-10-05 with a minimal cross configure of the libffi port in a :winamd64 container), and the native file never had /FORCE:MULTIPLE, so it now carries c_args = ['-fcommon']. Assert-LibffiTypeExport reads the aarch64 ffi-7.dll statically; size_t is 8 bytes on both lanes, so the expected descriptors are the same.

  • DTLS is arch-neutral C, so the 17d22abe89 hunk applies unchanged, and the aarch64 OpenSSL staging below was already in place.

  • gst-plugins-rs cross-builds with --target aarch64-pc-windows-msvc, PKG_CONFIG_ALLOW_CROSS=1 against the target prefix’s .pc files, lld-link as the target linker, and clang-cl --target=aarch64-pc-windows-msvc with llvm-lib for the C and assembly that ring compiles. Build scripts and proc macros stay host binaries, linked by rustc’s own MSVC lookup.

Proved on the device on 2026-10-11 (CON63). The 2026-10-10 merge built all seven, and the aarch64 ffi-7.dll passed Assert-LibffiTypeExport. Then Test-Arm64Bundle.ps1 ran its WebRTC step on windows-11-arm with the bundle’s gst-launch-1.0.exe: webrtcsink to webrtcsrc, 60 frames decoded over DTLS-SRTP (hub run 38170783445). On arm64 webrtcsink needs about 0.8 s to settle on mfh264enc, so the loopback waits for the producer to register before the consumer lists producers.

aarch64 OpenSSL is a base prerequisite too#

scoop installs one architecture per app, and that is the host’s — so the image carries lib\VC\x64\MD\libcrypto.lib and nothing else. Four GStreamer targets link OpenSSL and all four failed identically: ext/hls (HTTP Live Streaming), ext/dtls (WebRTC), ext/aes, and glib-networking’s OpenSSL TLS backend.

Install-ScoopTools.ps1 now installs the arm64 build beside the x64 one, using the same upstream artifact and the same SHA256 that scoop’s own openssl manifest pins for its arm64 entry. Three things about that step are load-bearing:

  • Extract, never run. The manifest is marked "innosetup": true, which is exactly how scoop installs it — with innounp. Running the installer silently was tried first and is what not to do: the whole step took 11.8 s including the 218 MB download, exited 0, and produced no files at all. A silent no-op that looks like success is the failure mode this repo exists to gate against, and it was only caught because the step searches for libcrypto.lib afterwards instead of trusting the exit code.

  • Declare the dependency, don’t inherit it from order. innounp reaches this image only as a side effect of scoop installing some innosetup package, and scoop install main/openssl runs later in the same file (measured: this block at 349 s, openssl at 406 s). The block installs main/innounp explicitly so it is position-independent.

  • Find the layout, never compose it. innounp extracts InnoSetup payloads under a literal {app} directory, so the real paths are …\{app}\lib\VC\arm64\MD\libcrypto.lib and …\{app}\include\openssl\opensslv.h. Both the lib dir and the include dir are located by searching for a known file; a Join-Path $root 'include' would silently point at nothing.

  • Read the URL and hash, never pin them. slproweb deletes an installer once its successor ships. A literal Win64ARMOpenSSL-4_0_2.exe answered 404 on 2026-09-30, and a base built that day came out without aarch64 OpenSSL, because this step only warns. The step now reads architecture.arm64 from buckets\main\bucket\openssl.json, the manifest the x64 install resolves as well. The two architectures therefore move together, and the hash is still checked.

  • Take it from the LAN preseed first. slproweb throttles each connection to ~20 KB/s. The x64 installer (251 MB) ran 3.6 h as one stream and then failed. Build-Buildkit.ps1 downloads both installers host-side in 32 parallel ranges (Save-ParallelRangeDownload, about 1 MB/s in total), checks their SHA256 against the same manifest, and puts them on the WebDAV under preseed/. This step fetches from there first and falls back to slproweb. The x64 installer lands in scoop’s cache under scoop’s own name, like the Vulkan SDK.

The package ships no .pc files, so the GStreamer build authors libcrypto, libssl and openssl with Write-PkgConfigFile — the same helper OpenCV and ONNX Runtime already need for the same reason — and puts that directory first on PKG_CONFIG_PATH so it wins over the image’s x64 openssl.pc.

meson install needs DESTDIR on the cross lane#

The last thing between a fully compiled GStreamer and an installed one is a post-install script:

ERROR: Failed to run install script gio-querymodules: Executable was not found
ERROR: Install scripts failed to run

gio-querymodules indexes GIO modules, and doing that means executing an aarch64 binary on this x64 host — the same class as the gst-inspect gate and the smoke test. meson already has the escape hatch, and it is keyed on DESTDIR (minstall.py:709-713, :729-731):

if not destdir and len(failing_scripts) > 0:  raise MesonException('Install scripts failed to run')
if destdir and (isinstance(i, InstallScriptFailure) or i.skip_if_destdir):
    self.log('Skipping custom install script because DESTDIR is set')

DESTDIR is meson’s “this is a staged/packaging install, do not try to run target binaries” signal.

--destdir C:\ is chosen so the files land exactly where they always did, with no staging tree to move afterwards. From mesonbuild/scripts/__init__.py:

def destdir_join(d1, d2): return str(PurePath(d1, *PurePath(d2).parts[1:]))

PureWindowsPath('C:\runtime\bin').parts is ('C:\','runtime','bin'), so parts[1:] drops the drive and PurePath('C:\', 'runtime', 'bin') is once again C:\runtime\bin — byte-identical to the non-DESTDIR path. amd64 keeps the plain invocation: there every install script can run, and skipping gio-querymodules would ship an unindexed GIO module directory.

Three resilience fixes the merge stage needed#

Neither is arm64-specific; both were found by the arm64 work and apply to both lanes.

win-pkgconfig is the one subproject that fetches with no fallback. Its download-binary.py has a single MIRROR_URL, one urlopen, and zero retries. When gstreamer.freedesktop.org returned HTTP Error 503, win-flex-bison fell back to GitHub and nasm to nasm.us — and this alone killed the merge, four separate runs. The same script opens with if os.path.isfile(dest_path) and sha256 matches: sys.exit(0), so the archive is pre-placed in phase 5 and verified against the same hash meson checks. Version and hash are parsed out of the subproject’s meson.build rather than hardcoded.

Retries were the wrong answer, and that took three failures to see. Four attempts with exponential backoff still lost to a sustained outage — backoff helps against a blip, not against a server that is down for minutes. The fix is the one the repo already uses for the Vulkan SDK (“Preseed the … exe onto the LAN webdav so containers never pull it from the vendor”): the archive is fetched from $SCCACHE_WEBDAV_ENDPOINT/preseed/ first, with upstream only as fallback — and whichever source works, it is PUT back to the preseed path, so the first successful run immunises every later one. Every step is fail-open: a preseed miss is not an error, and meson still has its own attempt. This must never become the thing that breaks a build.

A failed download wears a deterministic error’s costume. meson setup’s retry logic short-circuits on meson.build:LINE:COL: ERROR, correctly — a real configure error repeats identically and a retry costs a full wrap re-download. But a download failure is reported in exactly that form, so the classifier now also looks for a network signature (HTTP Error …, Failed to download, URLError, timeouts, refused connections) and retries when it finds one, without weakening the short-circuit for genuine errors.

TensorRT staging, as AGENTS.md carried it#

Moved out of AGENTS.md on 2026-09-15 (owner decision D10), unedited except for this heading and the relative links. The RULES stayed there; this is the reference behind them.

TensorRT is not downloaded automatically — it needs an accepted NVIDIA EULA. docs/windows-builds.md § TensorRT setup owns the staging procedure, the current/ rationale and the reasoning behind each rule below. The rules themselves:

  • OWNER DIRECTIVE: always take the NEWEST release. Never resolve a pin-vs-zip mismatch by lowering TENSORRT_VERSION — stage a newer zip and delete the superseded one.

  • Set TENSORRT_ZIP_SHA256 in versions.env for the new zip. A stale hash failing the build loudly is intended.

  • TENSORRT_VERSION must never derive a filesystem path or the runtime PATH; the tree is resolved from disk and normalised to current.

  • No zip staged is the NORMAL state of this host’s GPU lane — do NOT re-harden the graceful skip into a fail-fast. That was tried on 2026-08-04 and reverted the next day.

  • A PRESENT zip fails CLOSED: a half-extracted tree is a build failure, while an absent one is supported.

QNN staging on Windows, as AGENTS.md carried it#

Moved out of AGENTS.md on 2026-09-15 (owner decision D10), unedited except for this heading and the relative links. The RULES stayed there; this is the reference behind them.

The Qualcomm QAIRT SDK is login-gated (Qualcomm developer account + EULA). Stage the Windows zip in windows/qnn-sdk/ (git-ignored except its README); pin it with QNN_SDK_ZIP_SHA256 in versions.env. When staged, Resolve-QnnSdk extracts it and enables the QNN EP in ONE framework: ONNX Runtime (onnxruntime_USE_QNN=ON). ONNX GenAI inherits it at runtime through the ORT it links. LiteRT, TVM and IREE were passing INVENTED flags that CMake dropped silently while printing a success banner — removed 2026-08-31, see backlog #154. Copy-QnnRuntime still stages the per-arch backend DLLs beside all five installs, but only ORT loads them. No zip = QNN off with one notice on every framework. A version-mismatch (SDK too old for the framework) also falls back to QNN-off gracefully. The SDK is bind-mounted into the onnx, genai, litert, and tvm RUN stages at C:\temp\qnn-sdk. Full details: docs/windows-cross-builds.md § QNN. Windows #121 BUILD-TIME PATH PROVEN 2026-08-31 (staged QAIRT 2.44.0.260225, full :winarm64 chain: QNN EP ON with the aarch64-windows-msvc backend set, runtime staged beside all five frameworks, arch gate 1168/0, smoke 97/0/15); runtime execution still needs a Snapdragon host.

The Linux ARM64 lane mirrors this (backlog QNN-LINUX, PROVEN 2026-08-30 on a staged QAIRT v2.49.0.260730 zip — arm64 media build GREEN): stage the Linux AArch64 SDK zip in linux/qnn-sdk/ (different SDK — lib/aarch64-oe-linux-gcc11.2/, not aarch64-windows-msvc), pin with QNN_SDK_LINUX_ZIP_SHA256 in versions.env. ORT-only today; framework fan-out to GenAI/LiteRT/TVM/IREE is OPEN. No zip = QNN off. Mechanism (the QNN_ARCH_ABI override, stage_qnn_runtime, the bind-mount): see docs/linux-cross-builds.md (QNN EP, in the toggles section) and docs/refactoring-backlog.md A2. QNN-LINUX.

What this lane cannot produce#

Components with no arm64 story, and what stands in their place.

CUDA / cuDNN / TensorRT#

WIRED — phase 1 landed 2026-09-20 (#176). The toolkit and the ORT CUDA EP now build for Windows arm64 on this x64 host; only the runtime half needs an arm64 device. What changed, and what it replaced, in order:

  • Two earlier reasons recorded here were wrong. (1) The blanket “no Windows-on-ARM support” claim was retracted 2026-08-24, and the 421 MB cuDNN figure was never re-fetched: the cuDNN 9.25.0.15 windows-arm64 archive existed at that pin (ranged GET: HTTP 200, ~90 MB re-measured 2026-08-28 — see backlog #122, CLOSED), lib/arm64 inside. (2) “CUDA has no windows-arm64 redist or installer” was true on 2026-08-28 and is false at 13.4.2: NVIDIA ships cuda_13.4.2_windows_arm64_network.exe / cuda_13.4.2_windows_arm64.exe and per-component windows-arm64 redist archives, and its Windows-on-Arm Porting Guide documents x64→ARM64 cross-compilation (vcvarsall x64_arm64 + nvcc --use-local-env, %CUDA_PATH%\lib\arm64).

  • The payload: Install-Cuda.ps1 -TargetArch arm64 installs the x64 toolkit as before (headers + nvcc, the host tools) and stages the arm64 redist components (cuda_cudart, libcublas, libcufft, libcurand, libnvjitlink) into the same root as lib\arm64 / bin\arm64; cuDNN comes from cudnn-windows-arm64-<pin>_cuda13.4-archive.zip. All SHA-pinned in versions.env from redistrib_13.4.2.json / redistrib_9.26.0.json. Dockerfile.nvidia takes WINDOWS_TARGET_ARCH and appends bin\arm64 AFTER the x64 dirs (cudart64_13.dll exists under both arches). Since 2026-10-03 the payload also carries cuda_nvrtc and cuda_cupti (13.4.92, SHA-pinned) for consumers that compile kernels at run time or profile; cuda_nvtx’s windows-arm64 archive is header-only, so there is nothing to stage for it.

  • The build: Get-NvccCudaCmakeArgs drives Hostx64\arm64\cl.exe + --use-local-env on the cross lane, and Get-CudnnLibraryDir picks lib\arm64. Build-OnnxFromSource.ps1 enables USE_CUDA on cross only when Test-CudaWindowsArm64Payload finds lib\arm64\cudart.lib + cudadevrt.lib — a positive payload signal, never a host GPU probe. Result (2026-09-20, bk-20260920-031232): onnxruntime_providers_cuda.dll 0xAA64 in the win_arm64 wheel, 2065-object ORT build, arch gate 1011/0.

  • Phase 2 landed the same day: OpenCV, GenAI and TVM build their CUDA paths for arm64 too. OpenCV’s CUDA modules link NPP/cusolver/cusparse, so the arm64 payload gained libnpp, libcusolver and libcusparse; its one x86 intrinsic (_mm_popcnt_u64 in cudafilters’ wavelet_matrix_2d.cuh) is patched to a software popcount (patches/opencv_contrib/002-arm64-cudafilters-popcount.patch, probe-proven in out/probe-arm64-popcount). GenAI produces onnxruntime-genai-cuda.dll 0xAA64. TVM’s legacy FindCUDA hardcodes lib\x64 on WIN32, so the cross branch names CUDA_CUDART_LIBRARY/CUDA_CUBLAS_LIBRARY/CUDA_CUDA_LIBRARY (lib\arm64), CUDA_HOST_COMPILER (the Hostx64\arm64 cl) and CUDA_NVCC_FLAGS=--use-local-env. Result (2026-09-20, bk-20260920-203631): smoke 120/0/15, arch gate 1047/0, OpenCV’s opencv_cuda*500.dll installed under lib\opencv5\arm64\vc18\bin.

  • Still OFF: classic TensorRT (x64-only; TensorRT-RTX is the arm64 successor and is not wired). DirectML stays ON beside CUDA, as on amd64.

  • The smoke runs its host-toolchain §7 on this lane (nvcc is x64) with the cuDNN run-probe replaced by a link + PE-machine assert (Assert-NativeLinkRun -CrossLinkOnly), because the arm64 DLL cannot execute here. The payload sections still skip in-suite; the payload’s presence and arch are gated by the merge-stage wheel/arch checks and the whole-image arch gate instead.

  • What no gate here proves: that the arm64 CUDA EP runs. That needs an RTX Spark / N1x-class device.

DirectML#

No longer excluded — the “packaging gap” verdict recorded until 2026-08-23 was mistaken and is retracted. That entry claimed the nuget ships no arm64 import library. Byte inspection refutes it: Microsoft.AI.DirectML 1.15.4 contains bin/arm64-win/DirectML.lib, machine 0xAA64. The failure was an upper/lower-case mismatch when ONNX Runtime composes the redist path from onnxruntime_target_platform; see the portability table above. ONNX Runtime is therefore configured USE_DML=ON on both lanes as of backlog #113, and that build has now run (2026-08-24): the DML EP compiled and linked (onnxruntime_providers_dml.lib at ninja step 1116/1118), DirectML.dll was staged into C:\runtime\lib\onnxruntime-source\bin, and the merge-stage arch gate reported 390 binaries inspected, 0 violations with no allowlist skips (the whole-image scan has since grown to 931/0 with the litert branch and the target CPython aboard) — since the gate’s root is C:\runtime, .dll is in its extension set and the merge copies that whole tree, the shipped DirectML is necessarily arm64. #118 landed 2026-08-24: GenAI builds USE_DML=ON on both lanes (staging D3D12Core.dll through a target-derived filter) and OpenCV’s WITH_DIRECTML is ON on both lanes, feeding contrib G-API’s ONNX DirectML EP rather than cv::dnn. The limit worth stating plainly: this proves the right bytes ship, not that the EP runs — nothing arm64 executes on this x64 host, so only the windows-11-arm CI job can show that. Microsoft’s Snapdragon guidance still points at the QNN provider for NPU work, which would pull in the Qualcomm AI Engine SDK this stack does not integrate; that part is unchanged.

LiteRT-LM#

Blocked upstream — and the 2026-08-23 correction recorded here over-corrected (corrected again 2026-08-24). That correction called the prebuilt libGemmaModelConstraintProvider.lib “optional… never the obstacle”. On the active Bazel path that is wrong: gemma3_data_processor puts the x86_64-only prebuilt in the default Windows dependency graph, severable only via the litert_lm_fst_constraints_disabled config_setting (model_data_processor/BUILD:26-33). Both blockers are real: no windows-arm64 configuration in .bazelrc AND the prebuilt in the default graph (the upstream stub belongs to the non-active CMake path). Plain LiteRT (not -LM) has neither problem — pure CMake, no prebuilt — and #115 is done (2026-08-24): it cross-builds; media-litert is a real branch on this lane and the tflite GStreamer plugin is back among the mandatory four. The host-tool requirement was met with a natively built flatc from the same vendored tree plus a pinned host protoc 21.9 (see the LiteRT machinery section). Only LiteRT-LM stays excluded; its stage self-skips and stages the empty litert-lm stand-in tree for the merge.

Flutter#

Not cross-compilable. windows-arm64 needs a native arm64 host — native engine builds landed on beta/stable in March 2026 (flutter/flutter#176385) — but cross-compiling from an x64 host is still not upstream: flutter/flutter#62597, open, tracks arm64 as a target. Until 2026-09-27 this cited #179777, which only asked when arm64 builds reach stable and is closed. OmniAccelerANT builds its Flutter part on windows-11-arm instead (§ Consumer cross lanes).

PyTorch / the torch app stage#

Still dropped — the binding constraint is the cp314 pin, re-checked 2026-10-03. download.pytorch.org publishes win_arm64 +cpu wheels for cp311-cp313 (torch 2.14.1+cpu matches PYTORCH_VERSION; nightly is at 2.15.0.dev) and none for cp314 on any channel (stable, test, nightly). uv sync proper must execute the target interpreter, so the stage stays dropped. What ships instead: the merge stage stages the cp313 stack (torch, torchvision, pillow, MarkupSafe and the pure-python first-touch deps) into the bundle’s wheel store, SHA-pinned, for a device that brings its own cp313 interpreter.

Everything in that table is a product gap to document, not an engineering problem to route around. Where a coverage floor can encode it (CUDA sections in the smoke floors), encode it, so it can never be silently “fixed” by a skip.