Turbopack: use memchr3 for source position scanning and count astral chars as…
Turbopack: use memchr3 for source position scanning and count astral chars as two UTF-16 columns (#100919) ## Summary Turbopack's sectioned source maps compute each section's `offset` with `SourcePos::update`. It used to decode the generated code one UTF-8 character at a time. That loop was slow, and it counted a 4-byte (non-BMP) character such as `🦀 ` as one column even though source-map columns are UTF-16 code units, where it takes two. So any section that started after an astral character on the same line got an offset column that was too small. `SourcePos::update` now works like the line index in `next-code-frame`'s highlighter: - `memchr3` finds line-terminator candidates (`\n`, `\r`, and the `0xE2` lead byte of U+2028/U+2029). The JS terminator semantics are kept: `\r\n` counts as one terminator, and only `E2 80 A8` / `E2 80 A9` count as line/paragraph separators. - Only the text after the last terminator affects the column, so only that segment is counted. A small `utf16_len(&[u8])` helper counts its UTF-16 units byte by byte: continuation bytes count 0, 4-byte lead bytes count 2, everything else counts 1. This is exact for UTF-8, safe on arbitrary bytes, and needs no `unsafe` or new dependency. A TODO points at switching to `simd-utf16-len` once it gains a byte-slice API (SyMind/simd-utf16-len#22). `memchr` was already a workspace dependency; `turbopack-core` now uses it too. `Code::generate_source_map_ref` calls `SourcePos::update` once per rope chunk, so its docs and `SourcePos::update`'s docs now state the contract this relies on: each chunk must be self-contained text, and a UTF-8 sequence or a `\r\n` must not be split across chunks or mapping positions. Splitting one is a bug in the code that built the rope, and can produce wrong offsets. It is not memory-unsafe. Text built from `&str` already meets this contract in practice. Documenting it covers the concern that chunk boundaries could split terminators, without adding scanner state. Tests cover: - ASCII, BMP and astral characters; - `\n`, `\r`, `\r\n`, U+2028/U+2029, and neighbouring `0xE2` characters that are not separators; - incremental updates; - `utf16_len` against `str::encode_utf16`; - a `CodeBuilder` test with an astral character before a mapped section on the same line. ## Verification - `cargo test -p turbopack-core --lib` - A scanner-only micro-benchmark (synthetic ~1 MiB inputs, a noisy machine, not an end-to-end build) showed several-times higher throughput on newline-dense and minified ASCII. Text dense in U+20xx punctuation sees little benefit. <!-- NEXT_JS_LLM --> <!-- fleet 8a1a11e5-9ce4-4f06-b7cb-50485f329e62 --> --------- Co-authored-by:vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com> Co-authored-by:
Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
Showing
Please
register
or
sign in
to comment