Unverified Commit a41486df authored by vercel-fleet-prod[bot]'s avatar vercel-fleet-prod[bot] Committed by GitHub

wasi-test-host: share fd table across threads

Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
parent f58c038c
...@@ -685,6 +685,9 @@ jobs: ...@@ -685,6 +685,9 @@ jobs:
# compile is simply never built. # compile is simply never built.
cargo build -p next-napi-bindings --target wasm32-wasip1-threads cargo build -p next-napi-bindings --target wasm32-wasip1-threads
# The host's own tests (e.g. the fd table shared between threads); no wasm build needed.
node --test scripts/wasi-test-host/*.test.mjs
# Run the crates whose tests can execute on wasm. This needs a host that supplies the # Run the crates whose tests can execute on wasm. This needs a host that supplies the
# `env.read_custom_section` import turbo-tasks' link-time registries depend on; no stock # `env.read_custom_section` import turbo-tasks' link-time registries depend on; no stock
# runtime provides it. See scripts/wasi-test-host/README.md, which also lists the platform # runtime provides it. See scripts/wasi-test-host/README.md, which also lists the platform
......
...@@ -23,6 +23,11 @@ The other missing host feature is ...@@ -23,6 +23,11 @@ The other missing host feature is
memory and atomics, but Node's `WASI#getImportObject()` exposes only `wasi_snapshot_preview1`, not the memory and atomics, but Node's `WASI#getImportObject()` exposes only `wasi_snapshot_preview1`, not the
proposal's `wasi` namespace. `@emnapi/wasi-threads` bridges that import to Node Workers. proposal's `wasi` namespace. `@emnapi/wasi-threads` bridges that import to Node Workers.
Threads also need one file descriptor table. `node:wasi` keeps its fd table inside each `WASI`
instance, and every thread is a separate instance, so a file opened on one thread would be a bad
descriptor (or a different file) on all the others. The host therefore implements the filesystem
imports itself over a table shared by every thread; see [Files](#files).
The same hook will be needed by the Next.js wasm loader for Turbopack itself, so the The same hook will be needed by the Next.js wasm loader for Turbopack itself, so the
`read_custom_section` implementation here is a first cut of that work rather than test-only `read_custom_section` implementation here is a first cut of that work rather than test-only
scaffolding. scaffolding.
...@@ -36,19 +41,27 @@ cargo test -p turbo-tasks --lib --target wasm32-wasip1-threads ...@@ -36,19 +41,27 @@ cargo test -p turbo-tasks --lib --target wasm32-wasip1-threads
cargo test -p turbo-tasks-backend --lib --target wasm32-wasip1-threads cargo test -p turbo-tasks-backend --lib --target wasm32-wasip1-threads
``` ```
The host's own tests run without a wasm build:
```sh
node --test scripts/wasi-test-host/*.test.mjs
```
Building for that target also needs a WASI C toolchain, because `lz4-sys` and `zstd-sys` have C build Building for that target also needs a WASI C toolchain, because `lz4-sys` and `zstd-sys` have C build
scripts — see the `test-next-napi-bindings-wasi` job in `.github/workflows/build_and_test.yml` for the scripts — see the `test-next-napi-bindings-wasi` job in `.github/workflows/build_and_test.yml` for the
wasi-sdk setup. wasi-sdk setup.
## What it provides ## What it provides
| Import | Source | | Import | Source |
| -------------------------- | ----------------------------------------------------------------------------------------- | | --------------------------------------- | ----------------------------------------------------------------------------------------- |
| `wasi_snapshot_preview1.*` | `node:wasi` | | `wasi_snapshot_preview1.fd_*`, `path_*` | `node:fs` over an fd table shared by all threads (`fs.mjs`) |
| `wasi.thread-spawn` | `@emnapi/wasi-threads` over `node:worker_threads` (`spawn.mjs`) | | `wasi_snapshot_preview1.poll_oneoff` | clock-only: `node:wasi`; on fds: the shared table (`fs.mjs`) |
| `env.memory` | a shared `WebAssembly.Memory` matching the explicit linker limits in `.cargo/config.toml` | | other `wasi_snapshot_preview1.*` | `node:wasi`, one instance per thread |
| `env.read_custom_section` | `WebAssembly.Module.customSections` (`lib.mjs`) | | `wasi.thread-spawn` | `@emnapi/wasi-threads` over `node:worker_threads` (`spawn.mjs`) |
| filesystem | working directory at `/`; the system `os.tmpdir()` at `/tmp`, also exported as `TMPDIR` | | `env.memory` | a shared `WebAssembly.Memory` matching the explicit linker limits in `.cargo/config.toml` |
| `env.read_custom_section` | `WebAssembly.Module.customSections` (`lib.mjs`) |
| filesystem | working directory at `/`; the system `os.tmpdir()` at `/tmp`, also exported as `TMPDIR` |
`@emnapi/wasi-threads` owns Worker lifecycle, compiled-module transfer, load/start ordering, and `@emnapi/wasi-threads` owns Worker lifecycle, compiled-module transfer, load/start ordering, and
cleanup. A small local adapter remains because Rust imports the original one-argument cleanup. A small local adapter remains because Rust imports the original one-argument
...@@ -77,10 +90,55 @@ make the module usable. `registry::tests::registries_are_populated` guards this ...@@ -77,10 +90,55 @@ make the module usable. `registry::tests::registries_are_populated` guards this
WASI preview1's `std::env::temp_dir()` is unsupported, so Rust tests cannot use WASI preview1's `std::env::temp_dir()` is unsupported, so Rust tests cannot use
`tempfile::tempdir()` directly. The runner pre-opens the system's actual temporary directory as `tempfile::tempdir()` directly. The runner pre-opens the system's actual temporary directory as
`/tmp` for the main instance and every pthread, and exports that guest path through `TMPDIR`. `/tmp` for the whole process (see [Files](#files)), and exports that guest path through `TMPDIR`.
`turbo-tasks-backend`'s `test_temp_dir()` helper reads it explicitly, keeping fixtures out of the `turbo-tasks-backend`'s `test_temp_dir()` helper reads it explicitly, keeping fixtures out of the
checkout and Cargo `target` tree. checkout and Cargo `target` tree.
### Files
Rust programs pass open files between threads all the time: Tokio tasks move between workers, and
`turbo-persistence` creates, syncs and reads files from parallel helper jobs. With `node:wasi`'s
per-instance fd table, that fails (`Bad file descriptor`) or, worse, an fd number names a different
file on another thread.
Host OS fds, unlike that table, are process-wide across Workers. So `fs.mjs` implements the `fd_*`
and `path_*` imports on synchronous `node:fs` calls, and keeps the guest fd table — host fd, file
type, flags, rights, offset, and for directories their host path — in a `SharedArrayBuffer` that
every thread maps:
- **I/O stays concurrent.** Each call runs on the calling thread. Only fd allocation, `fd_close` and
`fd_renumber` take a short lock on the table, never across a syscall, and lookups are lock-free
(a per-entry sequence counter makes sure a lookup never sees half of a replaced entry). A thread
blocked in a read does not hold up file I/O on any other thread.
- **fds belong to the process, not the thread.** The main thread opens stdio and the preopens once.
Workers are created with `trackUnmanagedFds: false`, so a thread exiting does not close host fds
that other threads still use; a guest fd stays valid until the guest closes it.
- **The file offset is shared** across threads, like a real fd's. Node has no `lseek`, so the offset
of a seekable fd lives in the table, and the calls that use it (`fd_read`, `fd_write`, `fd_seek`,
`fd_tell`) hold that fd's own I/O lock across their syscall. Concurrent reads or writes through one
fd therefore never see the same bytes twice or overwrite each other. The lock is per fd: other
fds, positioned I/O (`fd_pread`/`fd_pwrite`, which `turbo-persistence` uses for SST reads),
syncs, stats, path calls and non-seekable fds (pipes, FIFOs, stdio) never wait for it.
- **Paths stay inside the directory fd.** Components resolve in order, like the OS, so `link/..` is
the parent of the link's target. Absolute paths, `..` above the base, and symlinks (in any
component) that lead out of it fail with `ENOTCAPABLE`. A directory fd resolves through its host
path, so it remembers the directory's `dev`/`ino` and fails with `ENOENT` once that path names
something else, such as after a rename. Resolution is not atomic with the syscall that follows; this
is a test host, not a sandbox.
- **What `node:fs` cannot do is an errno, not a fallback.** Rights are reported but not enforced, so
`fd_fdstat_set_rights` only accepts the current rights. `fd_fdstat_set_flags` only accepts the
current flags. `fd_allocate` cannot reserve space, and offsets beyond `Number.MAX_SAFE_INTEGER` are
rejected rather than rounded. All of these return `ENOTSUP`, `EINVAL` or `EOVERFLOW`. `poll_oneoff`
reports regular files and directories as ready and other fd types as `ENOTSUP`. Nothing falls back
to `node:wasi`'s private table.
`fs.test.mjs` covers these properties, including:
- a thread blocked on a FIFO read not stopping another from opening, writing, syncing and closing a
file;
- several threads reading or writing records through one fd;
- lookups racing `fd_close`/`fd_renumber`.
### Threads ### Threads
Each thread is a Worker running the same module over the same shared memory, entered through the Each thread is a Worker running the same module over the same shared memory, entered through the
......
This diff is collapsed.
This diff is collapsed.
...@@ -14,6 +14,11 @@ import { readFileSync } from 'node:fs' ...@@ -14,6 +14,11 @@ import { readFileSync } from 'node:fs'
import { tmpdir } from 'node:os' import { tmpdir } from 'node:os'
import process from 'node:process' import process from 'node:process'
import {
createPreview1Imports,
createSharedFdTableBuffer,
initializeSharedFdTable,
} from './fs.mjs'
import { import {
createImportedMemory, createImportedMemory,
createReadCustomSection, createReadCustomSection,
...@@ -44,7 +49,8 @@ const wasi = new WASI({ ...@@ -44,7 +49,8 @@ const wasi = new WASI({
version: 'preview1', version: 'preview1',
args, args,
env, env,
// The checkout remains available at `/`; `/tmp` maps to the host's actual temp directory. // Filesystem imports come from the shared fd table (./fs.mjs), which opens these same preopens:
// the checkout at `/`, and `/tmp` mapped to the host's actual temp directory.
preopens, preopens,
returnOnExit: true, returnOnExit: true,
}) })
...@@ -55,10 +61,18 @@ const wasi = new WASI({ ...@@ -55,10 +61,18 @@ const wasi = new WASI({
*/ */
const threadIds = new Int32Array(new SharedArrayBuffer(4)) const threadIds = new Int32Array(new SharedArrayBuffer(4))
/**
* The guest fd table, shared by every thread so that a file opened on one thread is usable on all of
* them. It is populated once, here, before any thread exists. See ./fs.mjs.
*/
const fdTable = createSharedFdTableBuffer()
initializeSharedFdTable(fdTable, preopens)
const { threadSpawn } = createThreadRuntime({ const { threadSpawn } = createThreadRuntime({
module, module,
memory, memory,
threadIds, threadIds,
fdTable,
args, args,
env, env,
preopens, preopens,
...@@ -71,7 +85,11 @@ const { threadSpawn } = createThreadRuntime({ ...@@ -71,7 +85,11 @@ const { threadSpawn } = createThreadRuntime({
}) })
const instance = await WebAssembly.instantiate(module, { const instance = await WebAssembly.instantiate(module, {
...wasi.getImportObject(), wasi_snapshot_preview1: createPreview1Imports({
buffer: fdTable,
memory,
wasi,
}),
env: { env: {
memory, memory,
read_custom_section: createReadCustomSection(module, memory), read_custom_section: createReadCustomSection(module, memory),
......
...@@ -52,13 +52,17 @@ export function nextThreadId(threadIds) { ...@@ -52,13 +52,17 @@ export function nextThreadId(threadIds) {
* main wasm thread is parked in `Atomics.wait`. All managers allocate IDs from the same shared * main wasm thread is parked in `Atomics.wait`. All managers allocate IDs from the same shared
* counter. * counter.
* *
* @param {{ module: WebAssembly.Module, memory: WebAssembly.Memory, threadIds: Int32Array, args: string[], env: Record<string, string>, preopens: Record<string, string>, onError: (error: Error, threadId: number | undefined) => void, workerPath?: string, WorkerClass?: typeof Worker, ThreadManagerClass?: typeof ThreadManager }} context * Every Worker shares `fdTable` (see ./fs.mjs) and is created with `trackUnmanagedFds: false`: host
* fds opened by one thread belong to the whole process, so a thread exiting must not close them.
*
* @param {{ module: WebAssembly.Module, memory: WebAssembly.Memory, threadIds: Int32Array, fdTable: SharedArrayBuffer, args: string[], env: Record<string, string>, preopens: Record<string, string>, onError: (error: Error, threadId: number | undefined) => void, workerPath?: string, WorkerClass?: typeof Worker, ThreadManagerClass?: typeof ThreadManager }} context
*/ */
export function createThreadRuntime(context) { export function createThreadRuntime(context) {
const { const {
module, module,
memory, memory,
threadIds, threadIds,
fdTable,
args, args,
env, env,
preopens, preopens,
...@@ -82,7 +86,10 @@ export function createThreadRuntime(context) { ...@@ -82,7 +86,10 @@ export function createThreadRuntime(context) {
reuseWorker: false, reuseWorker: false,
onCreateWorker: () => onCreateWorker: () =>
new WorkerClass(workerPath, { new WorkerClass(workerPath, {
workerData: { threadIds, args, env, preopens }, workerData: { threadIds, fdTable, args, env, preopens },
// Guest fds outlive the thread that opened them; Node would otherwise close the host fds
// a Worker opened when it exits.
trackUnmanagedFds: false,
// Inherit stdio so panics and test output from threads reach the terminal. // Inherit stdio so panics and test output from threads reach the terminal.
stdout: false, stdout: false,
stderr: false, stderr: false,
......
...@@ -10,10 +10,11 @@ import process from 'node:process' ...@@ -10,10 +10,11 @@ import process from 'node:process'
import { WASI } from 'node:wasi' import { WASI } from 'node:wasi'
import { parentPort, workerData } from 'node:worker_threads' import { parentPort, workerData } from 'node:worker_threads'
import { createPreview1Imports } from './fs.mjs'
import { createReadCustomSection } from './lib.mjs' import { createReadCustomSection } from './lib.mjs'
import { createThreadRuntime } from './spawn.mjs' import { createThreadRuntime } from './spawn.mjs'
const { threadIds, args, env, preopens } = workerData const { threadIds, fdTable, args, env, preopens } = workerData
const handler = new ThreadMessageHandler({ const handler = new ThreadMessageHandler({
postMessage: (message) => parentPort.postMessage(message), postMessage: (message) => parentPort.postMessage(message),
...@@ -36,6 +37,7 @@ const handler = new ThreadMessageHandler({ ...@@ -36,6 +37,7 @@ const handler = new ThreadMessageHandler({
module: wasmModule, module: wasmModule,
memory: wasmMemory, memory: wasmMemory,
threadIds, threadIds,
fdTable,
args, args,
env, env,
preopens, preopens,
...@@ -47,7 +49,12 @@ const handler = new ThreadMessageHandler({ ...@@ -47,7 +49,12 @@ const handler = new ThreadMessageHandler({
}) })
const instance = await WebAssembly.instantiate(wasmModule, { const instance = await WebAssembly.instantiate(wasmModule, {
...wasi.getImportObject(), // Files go through the fd table shared with every other thread; the rest stays local.
wasi_snapshot_preview1: createPreview1Imports({
buffer: fdTable,
memory: wasmMemory,
wasi,
}),
env: { env: {
memory: wasmMemory, memory: wasmMemory,
read_custom_section: createReadCustomSection(wasmModule, wasmMemory), read_custom_section: createReadCustomSection(wasmModule, wasmMemory),
......
...@@ -714,9 +714,7 @@ mod tests { ...@@ -714,9 +714,7 @@ mod tests {
/// The production batch lookup preserves request order, duplicates and missing entries on /// The production batch lookup preserves request order, duplicates and missing entries on
/// both sides of the parallel cutoff. /// both sides of the parallel cutoff.
// Default worker count like the other database tests here: under the WASI test host, file #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
// descriptors are per thread, so committing SST files from extra runtime workers fails.
#[tokio::test(flavor = "multi_thread")]
async fn test_batch_lookup_data_roundtrip_at_parallel_threshold() -> Result<()> { async fn test_batch_lookup_data_roundtrip_at_parallel_threshold() -> Result<()> {
let tempdir = test_temp_dir()?; let tempdir = test_temp_dir()?;
let db = TurboKeyValueDatabase::new(tempdir.path().to_path_buf(), TEST_STORAGE_OPTIONS)?; let db = TurboKeyValueDatabase::new(tempdir.path().to_path_buf(), TEST_STORAGE_OPTIONS)?;
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment