Unverified Commit e44c5f19 authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

ci: check that Turbopack compiles for wasm (#97588)

### What?

Adds a CI job that checks Turbopack compiles for
`wasm32-wasip1-threads`, so wasm portability
regressions are caught rather than rediscovered.

### Why?

Everything under `#[cfg(target_family = "wasm")]` is invisible to host
builds **and** to host clippy —
it is only checked when you deliberately build for the target. Four
separate defects in this stack were
caught only that way (a wrong build-script condition, two bad imports,
and a value that compiled but was
wrong). Without a gate, the next one lands unnoticed.

This also re-enables coverage that had been off for ~2 years: the old
`test-next-napi-bindings-wasi` job
was disabled with `if: false` pending napi-rs/napi-rs#2009, which closed
in April 2024.

### How?

Modelled on `rust-check` via `build_reusable.yml` (`needsRust`,
`skipInstallBuild`, `skipNativeBuild`),
so it does not pay for a JS build. Beyond `rustup target add` it needs
two things:

- **a WASI C toolchain**, because `lzzzz` (LZ4, via `turbo-persistence`)
and `zstd-sys` have C build
scripts. The SDK build is selected from `$RUNNER_ARCH` —
`build_reusable.yml` defaults to an arm64
runner, and an x86_64 clang fails there with `Exec format error` — with
a pinned sha256 per arch,
  unpacked under `$RUNNER_TEMP` so the workspace stays clean.
- **emnapi**, because `next-napi-bindings`' build script calls
`napi_build::setup()`, whose wasi path
panics without `EMNAPI_LINK_DIR` — so it is required even for `cargo
check`. It is installed into a
scratch directory rather than the root `package.json`, because this job
runs with `skipInstallBuild`
  and therefore never runs `pnpm install`.

Two things worth recording, both of which cost a CI round trip to find:

- pnpm is invoked from the repo root with `--dir`, not by `cd`-ing into
the scratch directory: corepack
resolves the pnpm version from the nearest `package.json`, and outside
the repo it picks the latest
  pnpm (11.x), which cannot run on the pinned Node 20
  (`ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING`).
- the job is added to `tests-pass`, whose `needs:` list is what actually
blocks a PR — a job that runs
but is absent from that list looks like coverage while blocking nothing.

The `emnapi@2.0.0-alpha.4` pin is deliberate and is the one fragility
here: the archive must define
`emnapi_create_env` / `emnapi_delete_env`, which exists only in emnapi
v2, still a prerelease. Move to
the stable release once it ships.


<!-- fleet b6d0486f-97c7-42a7-bdaf-3490774cdec3 -->
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent 73a26bfc
...@@ -621,26 +621,25 @@ jobs: ...@@ -621,26 +621,25 @@ jobs:
stepName: 'test-next-swc-wasm' stepName: 'test-next-swc-wasm'
secrets: inherit secrets: inherit
#[NOTE] currently this only checks building wasi target
test-next-napi-bindings-wasi: test-next-napi-bindings-wasi:
name: test next-swc wasi name: test next-swc wasi
needs: ['optimize-ci', 'changes', 'build-next'] needs: ['optimize-ci', 'changes', 'build-next']
# TODO: Re-enable this when https://github.com/napi-rs/napi-rs/issues/2009 is addressed. if: ${{ needs.optimize-ci.outputs.skip == 'false' && needs.changes.outputs.docs-only == 'false' }}
# Specifically, the `platform` value is now `threads` in
# https://github.com/napi-rs/napi-rs/blob/e4ad4767efaf093fdff3dc768856f6100a6e3f72/cli/src/api/build.ts#L530
if: false
# if: ${{ needs.changes.outputs.docs-only == 'false' }}
permissions: permissions:
contents: read contents: read
id-token: write id-token: write
uses: ./.github/workflows/build_reusable.yml uses: ./.github/workflows/build_reusable.yml
with: with:
needsRust: 'yes'
skipInstallBuild: 'yes'
skipNativeBuild: 'yes' skipNativeBuild: 'yes'
skipNativeInstall: 'yes' skipNativeInstall: 'yes'
afterBuild: | afterBuild: |
rustup target add wasm32-wasip1-threads set -euo pipefail
pnpm dlx turbo@${TURBO_VERSION} run build-native-wasi ${TURBO_ARGS}
source scripts/setup-wasi-env.sh
cargo check -p next-napi-bindings --target wasm32-wasip1-threads
stepName: 'test-next-napi-bindings-wasi' stepName: 'test-next-napi-bindings-wasi'
secrets: inherit secrets: inherit
...@@ -1209,6 +1208,7 @@ jobs: ...@@ -1209,6 +1208,7 @@ jobs:
- rust-check - rust-check
- rustdoc-check - rustdoc-check
- test-next-swc-wasm - test-next-swc-wasm
- test-next-napi-bindings-wasi
- test-turbopack-dev - test-turbopack-dev
- test-new-tests-dev - test-new-tests-dev
- test-new-tests-start - test-new-tests-start
......
# Building Turbopack for WebAssembly
Turbopack's napi bindings (`crates/next-napi-bindings`) can be compiled for
`wasm32-wasip1-threads`. This is the target that would let Turbopack run where no native binding
exists — unusual CPU architectures and operating systems we do not publish artifacts for.
> This is distinct from the `@next/swc-wasm-*` packages, which are built with `wasm-pack` from
> `crates/wasm` for `wasm32-unknown-unknown` and contain SWC only, not Turbopack.
## Prerequisites
Beyond the usual Rust toolchain, the target needs a WASI clang and sysroot (several dependencies have
C build scripts) and emnapi (napi's build script links against it). `scripts/setup-wasi-env.sh`
provisions both.
## Building on Linux
The setup script supports Linux systems directly. Source it, then build as usual:
```sh
source scripts/setup-wasi-env.sh
cargo check -p next-napi-bindings --target wasm32-wasip1-threads
```
It must be sourced from **Bash** because it uses `BASH_SOURCE`; alternative shells such as zsh should
invoke Bash explicitly:
```sh
bash -c 'source scripts/setup-wasi-env.sh && cargo check -p next-napi-bindings --target wasm32-wasip1-threads'
```
The script exports environment variables into the calling shell, which a subprocess cannot do.
Running it directly prints an error and does nothing.
The script downloads the wasi-sdk matching your host architecture, verifies it against a pinned
sha256, installs emnapi, and exports the cross-compilation variables (`WASI_SDK_PATH`,
`EMNAPI_LINK_DIR`, and the `*_wasm32_wasip1_threads` compiler variables). Downloads are cached under
`~/.cache/next-wasi-toolchain`, so re-sourcing in a new shell is fast. Set `WASI_SETUP_CACHE_DIR` to
move the cache.
CI sources the same script, so local and CI builds cannot drift apart.
## Building with Docker
On macOS, Windows, or any other Docker-capable host, use the Linux builder image:
```sh
docker build -t next-wasi-builder -f scripts/wasi-builder.Dockerfile .
docker run --rm -it \
-v "$PWD:/workspace" \
-v next-wasi-cache:/root/.cache/next-wasi-toolchain \
-w /workspace \
next-wasi-builder
```
Inside the container, use the same commands as CI:
```sh
source scripts/setup-wasi-env.sh
cargo check -p next-napi-bindings --target wasm32-wasip1-threads
```
The named volume preserves the wasi-sdk and emnapi downloads between runs. Mount a second volume at
`/workspace/target` if you also want to preserve Rust build artifacts without writing them to the host
checkout.
## Running tests
Wasm test binaries cannot be run directly: `turbo-tasks` gathers its task registries at link time and
needs the embedder to supply an `env.read_custom_section` import. `scripts/wasi-test-host/` is a Node
host that provides it, along with WASI preview1 and thread spawning.
Sourcing the setup script points Cargo's runner at that host, so tests work with the normal command:
```sh
source scripts/setup-wasi-env.sh
cargo test -p turbo-tasks --lib --target wasm32-wasip1-threads
```
Some tests are skipped on wasm; each carries a reason describing the specific platform limitation
(no unwinding, no mmap, and so on).
### Linting
Use `--lib --tests` rather than `--all-targets` when running clippy for wasm. `--all-targets` includes
benchmark targets, which depend on `criterion` (and so rayon) and cannot build for WASI:
```sh
cargo clippy -p turbo-tasks --lib --tests --target wasm32-wasip1-threads -- -D warnings
```
## Known limitations
- **emnapi v2 is a prerelease.** `napi-build` needs the `emnapi_create_env` / `emnapi_delete_env`
exports, which exist only in v2, so the script pins `emnapi@2.0.0-alpha.4`. Move to the stable
release once it ships.
- **A full `napi build` is not wired up yet**, for the same reason — CI currently compiles and runs
tests rather than producing a publishable artifact.
- **The JS side cannot load the artifact yet.** It needs a loader that supplies
`env.read_custom_section` and runs the module's initialization, which does not exist.
- Some features are unavailable on wasm and report an error when configured: SWC wasm plugins, and
anything requiring the child-process pool.
#!/usr/bin/env bash
# Provisions the toolchain needed to build `next-napi-bindings` for `wasm32-wasip1-threads`.
#
# This script must be SOURCED, not executed — it exports environment variables into the calling
# shell, which a subprocess cannot do:
#
# source scripts/setup-wasi-env.sh
# cargo check -p next-napi-bindings --target wasm32-wasip1-threads
#
# See contributing/core/building-wasm.md for the full workflow, including running tests.
#
# Downloads are cached in $WASI_SETUP_CACHE_DIR (defaults to $RUNNER_TEMP on CI, otherwise
# ~/.cache/next-wasi-toolchain), so re-sourcing is cheap. The cache deliberately lives outside the
# repository: the emnapi install would otherwise be picked up as a pnpm workspace project and dirty
# pnpm-lock.yaml.
if [ "${BASH_SOURCE[0]}" = "$0" ]; then
echo "error: setup-wasi-env.sh must be sourced, not executed:" >&2
echo " source scripts/setup-wasi-env.sh" >&2
exit 1
fi
setup_wasi_env() {
# No `set -e` here: this function runs in the caller's shell, and changing its shell options (or
# exiting on error) would affect an interactive session. Each step is checked explicitly instead.
local repo_root
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" || return 1
local cache_dir="${WASI_SETUP_CACHE_DIR:-${RUNNER_TEMP:-${XDG_CACHE_HOME:-$HOME/.cache}/next-wasi-toolchain}}"
mkdir -p "$cache_dir" || return 1
rustup target add wasm32-wasip1-threads || return 1
# `lzzzz` (LZ4, via turbo-persistence) and `zstd-sys` have C build scripts, so a WASI clang and
# sysroot are required on top of the Rust target. The SDK build must match the host running the
# compiler; an x86_64 clang on arm64 fails with "Exec format error".
local wasi_sdk_version=33.0 wasi_sdk_arch wasi_sdk_sha256
case "$(uname -m)" in
x86_64 | amd64)
wasi_sdk_arch=x86_64
wasi_sdk_sha256=0ba8b5bfaeb2adf3f29bab5841d76cf5318ab8e1642ea195f88baba1abd47bce
;;
arm64 | aarch64)
wasi_sdk_arch=arm64
wasi_sdk_sha256=4f98ee738c7abb45c81a94d1461fc53cc569d1cd01498951c8184d841a027844
;;
*)
echo "error: unsupported host architecture for the WASI SDK: $(uname -m)" >&2
return 1
;;
esac
local sdk_dir="$cache_dir/wasi-sdk-${wasi_sdk_version}-${wasi_sdk_arch}-linux"
if [ ! -x "$sdk_dir/bin/clang" ]; then
local tar="wasi-sdk-${wasi_sdk_version}-${wasi_sdk_arch}-linux.tar.gz"
curl --retry 3 --fail --location --silent --show-error --output "$cache_dir/$tar" \
"https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${wasi_sdk_version%%.*}/${tar}" || return 1
echo "${wasi_sdk_sha256} $cache_dir/${tar}" | sha256sum --check --strict || return 1
tar xzf "$cache_dir/$tar" -C "$cache_dir" || return 1
fi
export WASI_SDK_PATH="$sdk_dir"
export CC_wasm32_wasip1_threads="$sdk_dir/bin/clang"
export CXX_wasm32_wasip1_threads="$sdk_dir/bin/clang++"
export AR_wasm32_wasip1_threads="$sdk_dir/bin/llvm-ar"
export CFLAGS_wasm32_wasip1_threads="--target=wasm32-wasip1-threads --sysroot=$sdk_dir/share/wasi-sysroot"
export CXXFLAGS_wasm32_wasip1_threads="$CFLAGS_wasm32_wasip1_threads"
# next-napi-bindings' build script calls napi_build::setup(), whose wasi path links against
# emnapi and panics without EMNAPI_LINK_DIR — needed even for `cargo check`.
#
# The archive must define emnapi_create_env / emnapi_delete_env, which only exist in emnapi v2.
# That is still a prerelease, hence the exact alpha pin; move to the stable release once it ships.
# The archives are wasm objects, so they are arch-independent.
local emnapi_version=2.0.0-alpha.4
local emnapi_dir="$cache_dir/emnapi"
if [ ! -d "$emnapi_dir/node_modules/emnapi/lib/wasm32-wasip1-threads" ]; then
mkdir -p "$emnapi_dir" || return 1
printf '{\n "name": "emnapi-scratch",\n "private": true\n}\n' > "$emnapi_dir/package.json" || return 1
# pnpm is invoked from the repository root with --dir rather than by cd'ing into the scratch
# directory: corepack resolves the pnpm version from the nearest package.json, and outside the
# repo there is none, so it would pick the latest pnpm (11.x), which cannot run on the pinned
# Node 20 (ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING).
(cd "$repo_root" && pnpm --dir "$emnapi_dir" add "emnapi@${emnapi_version}") || return 1
fi
export EMNAPI_LINK_DIR="$emnapi_dir/node_modules/emnapi/lib/wasm32-wasip1-threads"
# Runs wasm test binaries under the Node WASI host, which supplies the `env.read_custom_section`
# import that turbo-tasks' link-time registries need.
if [ -f "$repo_root/scripts/wasi-test-host/run.mjs" ]; then
export CARGO_TARGET_WASM32_WASIP1_THREADS_RUNNER="node $repo_root/scripts/wasi-test-host/run.mjs"
fi
}
setup_wasi_env
setup_wasi_env_status=$?
unset -f setup_wasi_env
if [ "$setup_wasi_env_status" -ne 0 ]; then
echo "error: setup-wasi-env.sh failed" >&2
fi
unset setup_wasi_env_status
# Linux environment for building and testing Next.js' wasm32-wasip1-threads bindings from any host
# that can run Docker.
#
# Build from the repository root:
# docker build -t next-wasi-builder -f scripts/wasi-builder.Dockerfile .
#
# Run with the repository mounted at /workspace:
# docker run --rm -it -v "$PWD:/workspace" -w /workspace next-wasi-builder
FROM node:20-trixie
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
bash \
ca-certificates \
curl \
git \
build-essential \
&& rm -rf /var/lib/apt/lists/*
RUN corepack enable && \
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --no-update-default-toolchain
ENV PATH="/root/.cargo/bin:${PATH}"
WORKDIR /workspace
CMD ["bash"]
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment