Unverified Commit 5e08d7d4 authored by vercel-fleet-prod[bot]'s avatar vercel-fleet-prod[bot] Committed by GitHub

feat(turbopack): initialize the runtime on wasm

Run next-napi-bindings module initialization once across shared WASI instances, size Tokio from Node availableParallelism, register napi-rs wasm exports, and keep worker instances from recursively running process initialization. Link the artifact in CI with compiler and linker warnings denied.
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>
parent cf61700c
...@@ -649,12 +649,11 @@ jobs: ...@@ -649,12 +649,11 @@ jobs:
stepName: 'test-next-swc-wasm' stepName: 'test-next-swc-wasm'
secrets: inherit secrets: inherit
# Guards that Turbopack keeps compiling for wasm, so the wasm fallback Next.js loads when the # Guards that Turbopack keeps compiling and linking for wasm, so the wasm fallback Next.js loads
# native bindings are unavailable can eventually support Turbopack and not just SWC. # when the native bindings are unavailable can eventually support Turbopack and not just SWC.
# See https://github.com/vercel/next.js/issues/96960. # See https://github.com/vercel/next.js/issues/96960.
# #
# This compiles the bindings and runs the wasm test suites with the pinned emnapi v2 prerelease. # setup-wasi-env.sh pins the exact emnapi v2 prerelease needed for linking.
# A linked build is gated by the first stack layer that provides every required runtime hook.
test-next-napi-bindings-wasi: test-next-napi-bindings-wasi:
name: test next-swc wasi name: test next-swc wasi
needs: ['optimize-ci', 'changes', 'build-next'] needs: ['optimize-ci', 'changes', 'build-next']
...@@ -675,6 +674,7 @@ jobs: ...@@ -675,6 +674,7 @@ jobs:
# builds so the two cannot drift; see contributing/core/building-wasm.md. # builds so the two cannot drift; see contributing/core/building-wasm.md.
source scripts/setup-wasi-env.sh source scripts/setup-wasi-env.sh
# swc's plugin host is gated off for wasm (see turbopack-ecmascript-plugins and # swc's plugin host is gated off for wasm (see turbopack-ecmascript-plugins and
# next-napi-bindings), so nothing needs patching here: the wasm32 plugin path that does not # next-napi-bindings), so nothing needs patching here: the wasm32 plugin path that does not
# compile is simply never built. # compile is simply never built.
......
...@@ -19,14 +19,14 @@ The setup script supports Linux systems directly. Source it, then build as usual ...@@ -19,14 +19,14 @@ The setup script supports Linux systems directly. Source it, then build as usual
```sh ```sh
source scripts/setup-wasi-env.sh source scripts/setup-wasi-env.sh
cargo check -p next-napi-bindings --target wasm32-wasip1-threads cargo build -p next-napi-bindings --target wasm32-wasip1-threads
``` ```
It must be sourced from **Bash** because it uses `BASH_SOURCE`; alternative shells such as zsh should It must be sourced from **Bash** because it uses `BASH_SOURCE`; alternative shells such as zsh should
invoke Bash explicitly: invoke Bash explicitly:
```sh ```sh
bash -c 'source scripts/setup-wasi-env.sh && cargo check -p next-napi-bindings --target wasm32-wasip1-threads' bash -c 'source scripts/setup-wasi-env.sh && cargo build -p next-napi-bindings --target wasm32-wasip1-threads'
``` ```
The script exports environment variables into the calling shell, which a subprocess cannot do. The script exports environment variables into the calling shell, which a subprocess cannot do.
...@@ -38,7 +38,9 @@ sha256, installs emnapi, and exports the cross-compilation variables (`WASI_SDK_ ...@@ -38,7 +38,9 @@ sha256, installs emnapi, and exports the cross-compilation variables (`WASI_SDK_
`~/.cache/next-wasi-toolchain`, so re-sourcing in a new shell is fast. Set `WASI_SETUP_CACHE_DIR` to `~/.cache/next-wasi-toolchain`, so re-sourcing in a new shell is fast. Set `WASI_SETUP_CACHE_DIR` to
move the cache. move the cache.
CI sources the same script, so local and CI builds cannot drift apart. CI sources the same script, so local and CI builds cannot drift apart. The Node host passes its
effective CPU allowance into WASI, which sizes the bindings' multi-threaded Tokio runtime; set
`TURBO_TASKS_AVAILABLE_PARALLELISM` before invoking the host to override it.
## Building with Docker ## Building with Docker
...@@ -57,7 +59,7 @@ Inside the container, use the same commands as CI: ...@@ -57,7 +59,7 @@ Inside the container, use the same commands as CI:
```sh ```sh
source scripts/setup-wasi-env.sh source scripts/setup-wasi-env.sh
cargo check -p next-napi-bindings --target wasm32-wasip1-threads cargo build -p next-napi-bindings --target wasm32-wasip1-threads
``` ```
The named volume preserves the wasi-sdk and emnapi downloads between runs. Mount a second volume at The named volume preserves the wasi-sdk and emnapi downloads between runs. Mount a second volume at
...@@ -94,9 +96,9 @@ cargo clippy -p turbo-tasks --lib --tests --target wasm32-wasip1-threads -- -D w ...@@ -94,9 +96,9 @@ cargo clippy -p turbo-tasks --lib --tests --target wasm32-wasip1-threads -- -D w
- **emnapi v2 is a prerelease.** `napi-build` needs the `emnapi_create_env` / `emnapi_delete_env` - **emnapi v2 is a prerelease.** `napi-build` needs the `emnapi_create_env` / `emnapi_delete_env`
exports, which exist only in v2, so the script pins `emnapi@2.0.0-alpha.4`. Move to the stable exports, which exist only in v2, so the script pins `emnapi@2.0.0-alpha.4`. Move to the stable
release once it ships. release once it ships.
- **A full `napi build` is not wired up yet**, for the same reason — CI currently compiles and runs - **A publishable N-API wasm package is not wired up yet.** CI links the raw wasm artifact, but does
tests rather than producing a publishable artifact. not produce an `@next/swc-wasm-wasi` package.
- **The JS side cannot load the artifact yet.** It needs a loader that supplies - **The JS loader is not connected to the SWC fallback yet.** The loader hooks and module
`env.read_custom_section` and runs the module's initialization, which does not exist. initialization exist, but no published package consumes them.
- Some features are unavailable on wasm and report an error when configured: SWC wasm plugins, and - Some features are unavailable on wasm and report an error when configured: SWC wasm plugins, and
anything requiring the child-process pool. anything requiring the child-process pool.
...@@ -63,9 +63,30 @@ static ALLOC: turbo_tasks_malloc::TurboMalloc = turbo_tasks_malloc::TurboMalloc; ...@@ -63,9 +63,30 @@ static ALLOC: turbo_tasks_malloc::TurboMalloc = turbo_tasks_malloc::TurboMalloc;
#[global_allocator] #[global_allocator]
static ALLOC: dhat::Alloc = dhat::Alloc; static ALLOC: dhat::Alloc = dhat::Alloc;
#[cfg(not(target_arch = "wasm32"))] #[cfg(target_arch = "wasm32")]
static WASM_MODULE_INIT_STARTED: std::sync::atomic::AtomicBool =
std::sync::atomic::AtomicBool::new(false);
#[napi_derive::module_init] #[napi_derive::module_init]
fn init() { fn init() {
// Every WASI thread has its own module instance, and `WASI.initialize()` runs this constructor
// in each instance. The instances share linear memory, so this atomic lets the main
// instance create the process-wide Tokio runtime while worker instances skip recursive
// runtime construction. Workers must not wait for initialization to finish: the runtime
// builder is itself waiting for those workers to start.
#[cfg(target_arch = "wasm32")]
if WASM_MODULE_INIT_STARTED
.compare_exchange(
false,
true,
std::sync::atomic::Ordering::Relaxed,
std::sync::atomic::Ordering::Relaxed,
)
.is_err()
{
return;
}
use std::{ use std::{
cell::RefCell, cell::RefCell,
panic::{set_hook, take_hook}, panic::{set_hook, take_hook},
......
...@@ -10,7 +10,9 @@ const { ...@@ -10,7 +10,9 @@ const {
createWasiEnvironment, createWasiEnvironment,
createWasiImportObject, createWasiImportObject,
initializeWasiThread, initializeWasiThread,
instantiateWasiNapiModule,
nextThreadId, nextThreadId,
registerNapiExports,
WASI_MEMORY_INITIAL_PAGES, WASI_MEMORY_INITIAL_PAGES,
WASI_MEMORY_MAXIMUM_PAGES, WASI_MEMORY_MAXIMUM_PAGES,
} = loader } = loader
...@@ -129,6 +131,18 @@ test('rejects duplicate custom sections instead of picking one', async () => { ...@@ -129,6 +131,18 @@ test('rejects duplicate custom sections instead of picking one', async () => {
assert.throws(() => read(71, nameLength, 512, 8), /is ambiguous/) assert.throws(() => read(71, nameLength, 512, 8), /is ambiguous/)
}) })
test('registers every napi-rs wasm export before N-API initialization', () => {
const calls = []
registerNapiExports({
exports: {
memory: {},
__napi_register__first_0: () => calls.push('first'),
__napi_register__second_1: () => calls.push('second'),
},
})
assert.deepEqual(calls, ['first', 'second'])
})
test('aliases @emnapi/core napi imports into napi-sys env imports', async () => { test('aliases @emnapi/core napi imports into napi-sys env imports', async () => {
const bytes = buildModule({ customSections: [] }) const bytes = buildModule({ customSections: [] })
const module = await WebAssembly.compile(bytes) const module = await WebAssembly.compile(bytes)
...@@ -164,22 +178,59 @@ test('aliases @emnapi/core napi imports into napi-sys env imports', async () => ...@@ -164,22 +178,59 @@ test('aliases @emnapi/core napi imports into napi-sys env imports', async () =>
assert.equal(imports.wasi_snapshot_preview1.fd_write, wasiFdWrite) assert.equal(imports.wasi_snapshot_preview1.fd_write, wasiFdWrite)
}) })
test('allocates thread ids atomically from shared state', () => { test('passes Node parallelism to the main WASI instance', async () => {
const ids = new Int32Array(new SharedArrayBuffer(4)) const wasiOptions = []
assert.equal(nextThreadId(ids), 1) class FakeWasi {
assert.equal(nextThreadId(ids), 2) constructor(options) {
assert.throws(() => nextThreadId(new Int32Array(1)), /shared Int32Array/) wasiOptions.push(options)
}
getImportObject() {
return {}
}
initialize() {}
}
const initCalls = []
await instantiateWasiNapiModule({
bytes: buildModule({
initial: WASI_MEMORY_INITIAL_PAGES,
maximum: WASI_MEMORY_MAXIMUM_PAGES,
}),
napiModule: {
imports: {},
init: (options) => initCalls.push(options),
},
env: { KEEP: 'yes', TURBO_TASKS_AVAILABLE_PARALLELISM: '2' },
onThreadError: () => {},
workerPath: '/loader-worker.js',
WasiClass: FakeWasi,
})
assert.equal(wasiOptions.length, 1)
assert.deepEqual(wasiOptions[0].env, {
TURBO_TASKS_AVAILABLE_PARALLELISM: '2',
KEEP: 'yes',
})
assert.equal(initCalls.length, 1)
}) })
test('uses Node parallelism by default and preserves unrelated env', () => { test('uses Node parallelism by default and preserves unrelated env', () => {
const detected = Number(
createWasiEnvironment({}).TURBO_TASKS_AVAILABLE_PARALLELISM
)
assert.equal(Number.isInteger(detected) && detected > 0, true)
assert.deepEqual(createWasiEnvironment({ KEEP: 'yes' }, 6), { assert.deepEqual(createWasiEnvironment({ KEEP: 'yes' }, 6), {
TURBO_TASKS_AVAILABLE_PARALLELISM: '6', TURBO_TASKS_AVAILABLE_PARALLELISM: '6',
KEEP: 'yes', KEEP: 'yes',
}) })
assert.deepEqual( })
createWasiEnvironment({ TURBO_TASKS_AVAILABLE_PARALLELISM: '2' }, 6),
{ TURBO_TASKS_AVAILABLE_PARALLELISM: '2' } test('allocates thread ids atomically from shared state', () => {
) const ids = new Int32Array(new SharedArrayBuffer(4))
assert.equal(nextThreadId(ids), 1)
assert.equal(nextThreadId(ids), 2)
assert.throws(() => nextThreadId(new Int32Array(1)), /shared Int32Array/)
}) })
test('transfers one compiled module through the shared thread manager', async () => { test('transfers one compiled module through the shared thread manager', async () => {
...@@ -242,7 +293,7 @@ test('transfers one compiled module through the shared thread manager', async () ...@@ -242,7 +293,7 @@ test('transfers one compiled module through the shared thread manager', async ()
threadIds: ids, threadIds: ids,
workerData: { workerData: {
args: ['test.wasm'], args: ['test.wasm'],
env: { KEEP: 'yes' }, env: createWasiEnvironment({ KEEP: 'yes' }, 6),
preopens: {}, preopens: {},
workerPath: '/loader-worker.js', workerPath: '/loader-worker.js',
napiModuleSpecifier: '/emnapi-provider.js', napiModuleSpecifier: '/emnapi-provider.js',
...@@ -271,7 +322,7 @@ test('transfers one compiled module through the shared thread manager', async () ...@@ -271,7 +322,7 @@ test('transfers one compiled module through the shared thread manager', async ()
assert.equal(workers[0].unreferenced, true) assert.equal(workers[0].unreferenced, true)
}) })
test('spawned threads initialize WASI without _start before entering the thread', () => { test('spawned threads initialize WASI without process entry points', () => {
const calls = [] const calls = []
const wasi = { const wasi = {
initialize(instance) { initialize(instance) {
...@@ -281,12 +332,14 @@ test('spawned threads initialize WASI without _start before entering the thread' ...@@ -281,12 +332,14 @@ test('spawned threads initialize WASI without _start before entering the thread'
const instance = { const instance = {
exports: { exports: {
_start: () => calls.push(['unexpected start']), _start: () => calls.push(['unexpected start']),
_initialize: () => calls.push(['unexpected initialize']),
wasi_thread_start: (id, arg) => calls.push(['thread start', id, arg]), wasi_thread_start: (id, arg) => calls.push(['thread start', id, arg]),
}, },
} }
initializeWasiThread(wasi, instance)(7, 11) initializeWasiThread(wasi, instance)(7, 11)
assert.equal('_start' in calls[0][1], false) assert.equal('_start' in calls[0][1], false)
assert.equal('_initialize' in calls[0][1], false)
assert.deepEqual(calls[1], ['thread start', 7, 11]) assert.deepEqual(calls[1], ['thread start', 7, 11])
assert.throws( assert.throws(
() => initializeWasiThread(wasi, { exports: {} }), () => initializeWasiThread(wasi, { exports: {} }),
......
...@@ -8,11 +8,11 @@ ...@@ -8,11 +8,11 @@
* compiled copy so the production loader and test runner cannot drift. * compiled copy so the production loader and test runner cannot drift.
* *
* This is groundwork only — nothing here is wired into the SWC-only wasm fallback in `./index.ts`, * This is groundwork only — nothing here is wired into the SWC-only wasm fallback in `./index.ts`,
* and `module_init` (absent on wasm) and packaging are deliberately out of scope, so loading * and packaging is deliberately out of scope, so loading Turbopack from wasm does not work yet.
* Turbopack from wasm does not work yet.
*/ */
import path from 'node:path' import path from 'node:path'
import { WASI } from 'node:wasi'
import { import {
createImportedMemory, createImportedMemory,
...@@ -51,6 +51,14 @@ export type WasiLike = { ...@@ -51,6 +51,14 @@ export type WasiLike = {
initialize(instance: WebAssembly.Instance): void initialize(instance: WebAssembly.Instance): void
} }
type WasiConstructor = new (options: {
version: 'preview1'
args: string[]
env: Record<string, string>
preopens: Record<string, string>
returnOnExit: true
}) => WasiLike
export type WasiThreadWorkerData = RuntimeWorkerData & { export type WasiThreadWorkerData = RuntimeWorkerData & {
args: string[] args: string[]
env: Record<string, string> env: Record<string, string>
...@@ -92,11 +100,24 @@ export function createWasiImportObject(options: { ...@@ -92,11 +100,24 @@ export function createWasiImportObject(options: {
} }
} }
/** /** Populate napi-rs's wasm-side export registry before `napiModule.init()` consumes it. */
* Bind WASI for a worker without running the command-only `_start` export. export function registerNapiExports(instance: WebAssembly.Instance) {
for (const [name, register] of Object.entries(instance.exports)) {
if (name.startsWith('__napi_register__')) {
if (typeof register !== 'function') {
throw new TypeError(`${name} must be a WebAssembly function`)
}
register()
}
}
}
/** Bind WASI for a worker without rerunning process initialization.
* *
* `WASI#initialize` rejects a command exporting `_start`, which must run exactly once on the main * `WASI#initialize` rejects a command's `_start` and invokes a reactor's `_initialize`; both entry
* thread. `ThreadMessageHandler` enters the bound instance through `wasi_thread_start` instead. * points belong to the main instance. A spawned thread hides both while binding WASI, then
* `ThreadMessageHandler` enters through `wasi_thread_start`. This is especially important now that
* `_initialize` constructs the process-wide Tokio runtime.
*/ */
export function initializeWasiThread( export function initializeWasiThread(
wasi: WasiLike, wasi: WasiLike,
...@@ -108,6 +129,7 @@ export function initializeWasiThread( ...@@ -108,6 +129,7 @@ export function initializeWasiThread(
} }
const threadExports = { ...instance.exports } const threadExports = { ...instance.exports }
delete threadExports._start delete threadExports._start
delete threadExports._initialize
wasi.initialize({ exports: threadExports } as WebAssembly.Instance) wasi.initialize({ exports: threadExports } as WebAssembly.Instance)
return threadStart return threadStart
} }
...@@ -115,7 +137,6 @@ export function initializeWasiThread( ...@@ -115,7 +137,6 @@ export function initializeWasiThread(
export async function instantiateWasiNapiModule(options: { export async function instantiateWasiNapiModule(options: {
bytes: Uint8Array bytes: Uint8Array
napiModule: NapiModuleLike napiModule: NapiModuleLike
wasi: WasiLike
args?: string[] args?: string[]
env?: Record<string, string> env?: Record<string, string>
preopens?: Record<string, string> preopens?: Record<string, string>
...@@ -123,16 +144,25 @@ export async function instantiateWasiNapiModule(options: { ...@@ -123,16 +144,25 @@ export async function instantiateWasiNapiModule(options: {
workerPath?: string workerPath?: string
napiModuleSpecifier?: string napiModuleSpecifier?: string
wasiThreadsModuleSpecifier?: string wasiThreadsModuleSpecifier?: string
WasiClass?: WasiConstructor
}) { }) {
const { bytes, napiModule, wasi } = options const { bytes, napiModule } = options
const args = options.args ?? [] const args = options.args ?? []
const env = options.env ?? {} const env = createWasiEnvironment(options.env)
const preopens = options.preopens ?? {} const preopens = options.preopens ?? {}
const workerPath = const workerPath =
options.workerPath ?? path.join(__dirname, 'wasi-loader-worker.js') options.workerPath ?? path.join(__dirname, 'wasi-loader-worker.js')
const napiModuleSpecifier = options.napiModuleSpecifier ?? '@emnapi/core' const napiModuleSpecifier = options.napiModuleSpecifier ?? '@emnapi/core'
const wasiThreadsModuleSpecifier = const wasiThreadsModuleSpecifier =
options.wasiThreadsModuleSpecifier ?? '@emnapi/wasi-threads' options.wasiThreadsModuleSpecifier ?? '@emnapi/wasi-threads'
const WasiClass = options.WasiClass ?? WASI
const wasi = new WasiClass({
version: 'preview1',
args,
env,
preopens,
returnOnExit: true,
})
const module = await WebAssembly.compile(Uint8Array.from(bytes)) const module = await WebAssembly.compile(Uint8Array.from(bytes))
const memory = createImportedMemory() const memory = createImportedMemory()
...@@ -163,6 +193,7 @@ export async function instantiateWasiNapiModule(options: { ...@@ -163,6 +193,7 @@ export async function instantiateWasiNapiModule(options: {
}) })
const instance = await WebAssembly.instantiate(module, imports) const instance = await WebAssembly.instantiate(module, imports)
wasi.initialize(instance) wasi.initialize(instance)
registerNapiExports(instance)
napiModule.init({ instance, module, memory }) napiModule.init({ instance, module, memory })
return { instance, module, memory, threadIds } return { instance, module, memory, threadIds }
} }
...@@ -5,7 +5,7 @@ ...@@ -5,7 +5,7 @@
# shell, which a subprocess cannot do: # shell, which a subprocess cannot do:
# #
# source scripts/setup-wasi-env.sh # source scripts/setup-wasi-env.sh
# cargo check -p next-napi-bindings --target wasm32-wasip1-threads # cargo build -p next-napi-bindings --target wasm32-wasip1-threads
# #
# See contributing/core/building-wasm.md for the full workflow, including running tests. # See contributing/core/building-wasm.md for the full workflow, including running tests.
# #
......
...@@ -2,7 +2,8 @@ ...@@ -2,7 +2,8 @@
* Test-host access to the production-owned WASI support. * Test-host access to the production-owned WASI support.
* *
* The CI job builds `packages/next` before running Rust tests, so importing its compiled CommonJS * The CI job builds `packages/next` before running Rust tests, so importing its compiled CommonJS
* output keeps the memory contract and link-section handling identical to the production loader. * output keeps environment setup, the memory contract, and link-section handling identical to the
* production loader.
*/ */
import wasiRuntime from '../../packages/next/dist/build/swc/wasi-runtime.js' import wasiRuntime from '../../packages/next/dist/build/swc/wasi-runtime.js'
...@@ -10,6 +11,7 @@ import wasiRuntime from '../../packages/next/dist/build/swc/wasi-runtime.js' ...@@ -10,6 +11,7 @@ import wasiRuntime from '../../packages/next/dist/build/swc/wasi-runtime.js'
export const { export const {
createImportedMemory, createImportedMemory,
createReadCustomSection, createReadCustomSection,
createWasiEnvironment,
WASI_MEMORY_INITIAL_PAGES, WASI_MEMORY_INITIAL_PAGES,
WASI_MEMORY_MAXIMUM_PAGES, WASI_MEMORY_MAXIMUM_PAGES,
} = wasiRuntime } = wasiRuntime
...@@ -18,9 +20,17 @@ export const { ...@@ -18,9 +20,17 @@ export const {
export const WASI_TEST_TEMP_DIR = '/tmp' export const WASI_TEST_TEMP_DIR = '/tmp'
/** Build the environment and preopens shared by the main instance and all pthread instances. */ /** Build the environment and preopens shared by the main instance and all pthread instances. */
export function createWasiTestEnvironment(env, cwd, hostTempDir) { export function createWasiTestEnvironment(
env,
cwd,
hostTempDir,
detectedParallelism
) {
return { return {
env: { ...env, TMPDIR: WASI_TEST_TEMP_DIR }, env: createWasiEnvironment(
{ ...env, TMPDIR: WASI_TEST_TEMP_DIR },
detectedParallelism
),
preopens: { '/': cwd, [WASI_TEST_TEMP_DIR]: hostTempDir }, preopens: { '/': cwd, [WASI_TEST_TEMP_DIR]: hostTempDir },
} }
} }
...@@ -49,8 +49,11 @@ const handler = new ThreadMessageHandler({ ...@@ -49,8 +49,11 @@ const handler = new ThreadMessageHandler({
wasi: { 'thread-spawn': threadSpawn }, wasi: { 'thread-spawn': threadSpawn },
}) })
// Hide process constructors before binding WASI. `ThreadMessageHandler` enters only through
// `wasi_thread_start`, so workers neither rerun `_start` nor create another Tokio runtime.
const threadExports = { ...instance.exports } const threadExports = { ...instance.exports }
delete threadExports._start delete threadExports._start
delete threadExports._initialize
wasi.initialize({ exports: threadExports }) wasi.initialize({ exports: threadExports })
return { module: wasmModule, instance } return { module: wasmModule, instance }
}, },
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment