Unverified Commit 5e08d7d4 authored by vercel-fleet-prod[bot]'s avatar vercel-fleet-prod[bot] Committed by GitHub

feat(turbopack): initialize the runtime on wasm

Run next-napi-bindings module initialization once across shared WASI instances, size Tokio from Node availableParallelism, register napi-rs wasm exports, and keep worker instances from recursively running process initialization. Link the artifact in CI with compiler and linker warnings denied.
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>
parent cf61700c
......@@ -649,12 +649,11 @@ jobs:
stepName: 'test-next-swc-wasm'
secrets: inherit
# Guards that Turbopack keeps compiling for wasm, so the wasm fallback Next.js loads when the
# native bindings are unavailable can eventually support Turbopack and not just SWC.
# Guards that Turbopack keeps compiling and linking for wasm, so the wasm fallback Next.js loads
# when the native bindings are unavailable can eventually support Turbopack and not just SWC.
# See https://github.com/vercel/next.js/issues/96960.
#
# This compiles the bindings and runs the wasm test suites with the pinned emnapi v2 prerelease.
# A linked build is gated by the first stack layer that provides every required runtime hook.
# setup-wasi-env.sh pins the exact emnapi v2 prerelease needed for linking.
test-next-napi-bindings-wasi:
name: test next-swc wasi
needs: ['optimize-ci', 'changes', 'build-next']
......@@ -675,6 +674,7 @@ jobs:
# builds so the two cannot drift; see contributing/core/building-wasm.md.
source scripts/setup-wasi-env.sh
# swc's plugin host is gated off for wasm (see turbopack-ecmascript-plugins and
# next-napi-bindings), so nothing needs patching here: the wasm32 plugin path that does not
# compile is simply never built.
......
......@@ -19,14 +19,14 @@ The setup script supports Linux systems directly. Source it, then build as usual
```sh
source scripts/setup-wasi-env.sh
cargo check -p next-napi-bindings --target wasm32-wasip1-threads
cargo build -p next-napi-bindings --target wasm32-wasip1-threads
```
It must be sourced from **Bash** because it uses `BASH_SOURCE`; alternative shells such as zsh should
invoke Bash explicitly:
```sh
bash -c 'source scripts/setup-wasi-env.sh && cargo check -p next-napi-bindings --target wasm32-wasip1-threads'
bash -c 'source scripts/setup-wasi-env.sh && cargo build -p next-napi-bindings --target wasm32-wasip1-threads'
```
The script exports environment variables into the calling shell, which a subprocess cannot do.
......@@ -38,7 +38,9 @@ sha256, installs emnapi, and exports the cross-compilation variables (`WASI_SDK_
`~/.cache/next-wasi-toolchain`, so re-sourcing in a new shell is fast. Set `WASI_SETUP_CACHE_DIR` to
move the cache.
CI sources the same script, so local and CI builds cannot drift apart.
CI sources the same script, so local and CI builds cannot drift apart. The Node host passes its
effective CPU allowance into WASI, which sizes the bindings' multi-threaded Tokio runtime; set
`TURBO_TASKS_AVAILABLE_PARALLELISM` before invoking the host to override it.
## Building with Docker
......@@ -57,7 +59,7 @@ Inside the container, use the same commands as CI:
```sh
source scripts/setup-wasi-env.sh
cargo check -p next-napi-bindings --target wasm32-wasip1-threads
cargo build -p next-napi-bindings --target wasm32-wasip1-threads
```
The named volume preserves the wasi-sdk and emnapi downloads between runs. Mount a second volume at
......@@ -94,9 +96,9 @@ cargo clippy -p turbo-tasks --lib --tests --target wasm32-wasip1-threads -- -D w
- **emnapi v2 is a prerelease.** `napi-build` needs the `emnapi_create_env` / `emnapi_delete_env`
exports, which exist only in v2, so the script pins `emnapi@2.0.0-alpha.4`. Move to the stable
release once it ships.
- **A full `napi build` is not wired up yet**, for the same reason — CI currently compiles and runs
tests rather than producing a publishable artifact.
- **The JS side cannot load the artifact yet.** It needs a loader that supplies
`env.read_custom_section` and runs the module's initialization, which does not exist.
- **A publishable N-API wasm package is not wired up yet.** CI links the raw wasm artifact, but does
not produce an `@next/swc-wasm-wasi` package.
- **The JS loader is not connected to the SWC fallback yet.** The loader hooks and module
initialization exist, but no published package consumes them.
- Some features are unavailable on wasm and report an error when configured: SWC wasm plugins, and
anything requiring the child-process pool.
......@@ -63,9 +63,30 @@ static ALLOC: turbo_tasks_malloc::TurboMalloc = turbo_tasks_malloc::TurboMalloc;
#[global_allocator]
static ALLOC: dhat::Alloc = dhat::Alloc;
#[cfg(not(target_arch = "wasm32"))]
#[cfg(target_arch = "wasm32")]
static WASM_MODULE_INIT_STARTED: std::sync::atomic::AtomicBool =
std::sync::atomic::AtomicBool::new(false);
#[napi_derive::module_init]
fn init() {
// Every WASI thread has its own module instance, and `WASI.initialize()` runs this constructor
// in each instance. The instances share linear memory, so this atomic lets the main
// instance create the process-wide Tokio runtime while worker instances skip recursive
// runtime construction. Workers must not wait for initialization to finish: the runtime
// builder is itself waiting for those workers to start.
#[cfg(target_arch = "wasm32")]
if WASM_MODULE_INIT_STARTED
.compare_exchange(
false,
true,
std::sync::atomic::Ordering::Relaxed,
std::sync::atomic::Ordering::Relaxed,
)
.is_err()
{
return;
}
use std::{
cell::RefCell,
panic::{set_hook, take_hook},
......
......@@ -10,7 +10,9 @@ const {
createWasiEnvironment,
createWasiImportObject,
initializeWasiThread,
instantiateWasiNapiModule,
nextThreadId,
registerNapiExports,
WASI_MEMORY_INITIAL_PAGES,
WASI_MEMORY_MAXIMUM_PAGES,
} = loader
......@@ -129,6 +131,18 @@ test('rejects duplicate custom sections instead of picking one', async () => {
assert.throws(() => read(71, nameLength, 512, 8), /is ambiguous/)
})
test('registers every napi-rs wasm export before N-API initialization', () => {
const calls = []
registerNapiExports({
exports: {
memory: {},
__napi_register__first_0: () => calls.push('first'),
__napi_register__second_1: () => calls.push('second'),
},
})
assert.deepEqual(calls, ['first', 'second'])
})
test('aliases @emnapi/core napi imports into napi-sys env imports', async () => {
const bytes = buildModule({ customSections: [] })
const module = await WebAssembly.compile(bytes)
......@@ -164,22 +178,59 @@ test('aliases @emnapi/core napi imports into napi-sys env imports', async () =>
assert.equal(imports.wasi_snapshot_preview1.fd_write, wasiFdWrite)
})
test('allocates thread ids atomically from shared state', () => {
const ids = new Int32Array(new SharedArrayBuffer(4))
assert.equal(nextThreadId(ids), 1)
assert.equal(nextThreadId(ids), 2)
assert.throws(() => nextThreadId(new Int32Array(1)), /shared Int32Array/)
test('passes Node parallelism to the main WASI instance', async () => {
const wasiOptions = []
class FakeWasi {
constructor(options) {
wasiOptions.push(options)
}
getImportObject() {
return {}
}
initialize() {}
}
const initCalls = []
await instantiateWasiNapiModule({
bytes: buildModule({
initial: WASI_MEMORY_INITIAL_PAGES,
maximum: WASI_MEMORY_MAXIMUM_PAGES,
}),
napiModule: {
imports: {},
init: (options) => initCalls.push(options),
},
env: { KEEP: 'yes', TURBO_TASKS_AVAILABLE_PARALLELISM: '2' },
onThreadError: () => {},
workerPath: '/loader-worker.js',
WasiClass: FakeWasi,
})
assert.equal(wasiOptions.length, 1)
assert.deepEqual(wasiOptions[0].env, {
TURBO_TASKS_AVAILABLE_PARALLELISM: '2',
KEEP: 'yes',
})
assert.equal(initCalls.length, 1)
})
test('uses Node parallelism by default and preserves unrelated env', () => {
const detected = Number(
createWasiEnvironment({}).TURBO_TASKS_AVAILABLE_PARALLELISM
)
assert.equal(Number.isInteger(detected) && detected > 0, true)
assert.deepEqual(createWasiEnvironment({ KEEP: 'yes' }, 6), {
TURBO_TASKS_AVAILABLE_PARALLELISM: '6',
KEEP: 'yes',
})
assert.deepEqual(
createWasiEnvironment({ TURBO_TASKS_AVAILABLE_PARALLELISM: '2' }, 6),
{ TURBO_TASKS_AVAILABLE_PARALLELISM: '2' }
)
})
test('allocates thread ids atomically from shared state', () => {
const ids = new Int32Array(new SharedArrayBuffer(4))
assert.equal(nextThreadId(ids), 1)
assert.equal(nextThreadId(ids), 2)
assert.throws(() => nextThreadId(new Int32Array(1)), /shared Int32Array/)
})
test('transfers one compiled module through the shared thread manager', async () => {
......@@ -242,7 +293,7 @@ test('transfers one compiled module through the shared thread manager', async ()
threadIds: ids,
workerData: {
args: ['test.wasm'],
env: { KEEP: 'yes' },
env: createWasiEnvironment({ KEEP: 'yes' }, 6),
preopens: {},
workerPath: '/loader-worker.js',
napiModuleSpecifier: '/emnapi-provider.js',
......@@ -271,7 +322,7 @@ test('transfers one compiled module through the shared thread manager', async ()
assert.equal(workers[0].unreferenced, true)
})
test('spawned threads initialize WASI without _start before entering the thread', () => {
test('spawned threads initialize WASI without process entry points', () => {
const calls = []
const wasi = {
initialize(instance) {
......@@ -281,12 +332,14 @@ test('spawned threads initialize WASI without _start before entering the thread'
const instance = {
exports: {
_start: () => calls.push(['unexpected start']),
_initialize: () => calls.push(['unexpected initialize']),
wasi_thread_start: (id, arg) => calls.push(['thread start', id, arg]),
},
}
initializeWasiThread(wasi, instance)(7, 11)
assert.equal('_start' in calls[0][1], false)
assert.equal('_initialize' in calls[0][1], false)
assert.deepEqual(calls[1], ['thread start', 7, 11])
assert.throws(
() => initializeWasiThread(wasi, { exports: {} }),
......
......@@ -8,11 +8,11 @@
* compiled copy so the production loader and test runner cannot drift.
*
* This is groundwork only — nothing here is wired into the SWC-only wasm fallback in `./index.ts`,
* and `module_init` (absent on wasm) and packaging are deliberately out of scope, so loading
* Turbopack from wasm does not work yet.
* and packaging is deliberately out of scope, so loading Turbopack from wasm does not work yet.
*/
import path from 'node:path'
import { WASI } from 'node:wasi'
import {
createImportedMemory,
......@@ -51,6 +51,14 @@ export type WasiLike = {
initialize(instance: WebAssembly.Instance): void
}
type WasiConstructor = new (options: {
version: 'preview1'
args: string[]
env: Record<string, string>
preopens: Record<string, string>
returnOnExit: true
}) => WasiLike
export type WasiThreadWorkerData = RuntimeWorkerData & {
args: string[]
env: Record<string, string>
......@@ -92,11 +100,24 @@ export function createWasiImportObject(options: {
}
}
/**
* Bind WASI for a worker without running the command-only `_start` export.
/** Populate napi-rs's wasm-side export registry before `napiModule.init()` consumes it. */
export function registerNapiExports(instance: WebAssembly.Instance) {
for (const [name, register] of Object.entries(instance.exports)) {
if (name.startsWith('__napi_register__')) {
if (typeof register !== 'function') {
throw new TypeError(`${name} must be a WebAssembly function`)
}
register()
}
}
}
/** Bind WASI for a worker without rerunning process initialization.
*
* `WASI#initialize` rejects a command exporting `_start`, which must run exactly once on the main
* thread. `ThreadMessageHandler` enters the bound instance through `wasi_thread_start` instead.
* `WASI#initialize` rejects a command's `_start` and invokes a reactor's `_initialize`; both entry
* points belong to the main instance. A spawned thread hides both while binding WASI, then
* `ThreadMessageHandler` enters through `wasi_thread_start`. This is especially important now that
* `_initialize` constructs the process-wide Tokio runtime.
*/
export function initializeWasiThread(
wasi: WasiLike,
......@@ -108,6 +129,7 @@ export function initializeWasiThread(
}
const threadExports = { ...instance.exports }
delete threadExports._start
delete threadExports._initialize
wasi.initialize({ exports: threadExports } as WebAssembly.Instance)
return threadStart
}
......@@ -115,7 +137,6 @@ export function initializeWasiThread(
export async function instantiateWasiNapiModule(options: {
bytes: Uint8Array
napiModule: NapiModuleLike
wasi: WasiLike
args?: string[]
env?: Record<string, string>
preopens?: Record<string, string>
......@@ -123,16 +144,25 @@ export async function instantiateWasiNapiModule(options: {
workerPath?: string
napiModuleSpecifier?: string
wasiThreadsModuleSpecifier?: string
WasiClass?: WasiConstructor
}) {
const { bytes, napiModule, wasi } = options
const { bytes, napiModule } = options
const args = options.args ?? []
const env = options.env ?? {}
const env = createWasiEnvironment(options.env)
const preopens = options.preopens ?? {}
const workerPath =
options.workerPath ?? path.join(__dirname, 'wasi-loader-worker.js')
const napiModuleSpecifier = options.napiModuleSpecifier ?? '@emnapi/core'
const wasiThreadsModuleSpecifier =
options.wasiThreadsModuleSpecifier ?? '@emnapi/wasi-threads'
const WasiClass = options.WasiClass ?? WASI
const wasi = new WasiClass({
version: 'preview1',
args,
env,
preopens,
returnOnExit: true,
})
const module = await WebAssembly.compile(Uint8Array.from(bytes))
const memory = createImportedMemory()
......@@ -163,6 +193,7 @@ export async function instantiateWasiNapiModule(options: {
})
const instance = await WebAssembly.instantiate(module, imports)
wasi.initialize(instance)
registerNapiExports(instance)
napiModule.init({ instance, module, memory })
return { instance, module, memory, threadIds }
}
......@@ -5,7 +5,7 @@
# shell, which a subprocess cannot do:
#
# source scripts/setup-wasi-env.sh
# cargo check -p next-napi-bindings --target wasm32-wasip1-threads
# cargo build -p next-napi-bindings --target wasm32-wasip1-threads
#
# See contributing/core/building-wasm.md for the full workflow, including running tests.
#
......
......@@ -2,7 +2,8 @@
* Test-host access to the production-owned WASI support.
*
* The CI job builds `packages/next` before running Rust tests, so importing its compiled CommonJS
* output keeps the memory contract and link-section handling identical to the production loader.
* output keeps environment setup, the memory contract, and link-section handling identical to the
* production loader.
*/
import wasiRuntime from '../../packages/next/dist/build/swc/wasi-runtime.js'
......@@ -10,6 +11,7 @@ import wasiRuntime from '../../packages/next/dist/build/swc/wasi-runtime.js'
export const {
createImportedMemory,
createReadCustomSection,
createWasiEnvironment,
WASI_MEMORY_INITIAL_PAGES,
WASI_MEMORY_MAXIMUM_PAGES,
} = wasiRuntime
......@@ -18,9 +20,17 @@ export const {
export const WASI_TEST_TEMP_DIR = '/tmp'
/** Build the environment and preopens shared by the main instance and all pthread instances. */
export function createWasiTestEnvironment(env, cwd, hostTempDir) {
export function createWasiTestEnvironment(
env,
cwd,
hostTempDir,
detectedParallelism
) {
return {
env: { ...env, TMPDIR: WASI_TEST_TEMP_DIR },
env: createWasiEnvironment(
{ ...env, TMPDIR: WASI_TEST_TEMP_DIR },
detectedParallelism
),
preopens: { '/': cwd, [WASI_TEST_TEMP_DIR]: hostTempDir },
}
}
......@@ -49,8 +49,11 @@ const handler = new ThreadMessageHandler({
wasi: { 'thread-spawn': threadSpawn },
})
// Hide process constructors before binding WASI. `ThreadMessageHandler` enters only through
// `wasi_thread_start`, so workers neither rerun `_start` nor create another Tokio runtime.
const threadExports = { ...instance.exports }
delete threadExports._start
delete threadExports._initialize
wasi.initialize({ exports: threadExports })
return { module: wasmModule, instance }
},
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment