Unverified Commit ee39f8f4 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

turbo-rcstr: replace `rcstr!` macro_rules with proc macro (#93551)

### What?

Replaces the `rcstr!` macro in `turbo-rcstr` with a proc-macro
implementation in a new `turbo-rcstr-macros` crate. The proc macro
inspects the literal at expansion time and emits only the relevant arm —
no dead `else` branch.

### Why?

The previous `macro_rules!` macro emitted both an inline arm and a
static + `inventory::submit!` arm at every call site, expecting `const`
evaluation to discard one. `inventory::submit!` adds `#[used]` to its
constructor, so the dead arm shipped in the binary even when const eval
determined the live arm was the inline one.

Across the workspace there are ~852 inlinable `rcstr!` literals (≤ 7
bytes); each previously emitted a wasted `PrehashedString` static, an
inventory node, and a startup `ctor`.

### How?

A `#[proc_macro] rcstr` in the new `turbo-rcstr-macros` crate decides at
expansion time:

- inlinable literal → `inline_atom(lit).unwrap()`. No static, no
inventory entry, no dead branch.
- non-inlinable literal → static + `inventory::submit!` + `from_static`,
no inline arm.
- constant identifiers / `concat!(…)` / non-literal inputs → fall back
to the original both-branches expansion so const eval picks the arm. ~11
such call sites.

`turbo-rcstr`'s `atom_size_64` / `atom_size_128` features forward to the
proc-macro crate so the threshold is exact for every build configuration
without a runtime const branch.

The proc macro deliberately avoids `syn`, `quote`, and `proc-macro2`. It
pattern-matches on `proc_macro::TokenTree` directly, asks the compiler
for the literal's unescaped value via `Literal::str_value` (gated on the
unstable `proc_macro_value` feature), and emits the chosen expansion by
parsing a string template via `TokenStream::from_str`. The proc-macro
crate compiles in ~0.7 s with zero third-party dependencies.

### Compile time

Clean rebuild of `turbopack-core` in release mode (143 `rcstr!` call
sites). Dependencies cached; only `turbopack-core` itself rebuilt each
iteration. `darwin-aarch64`.

|                    |   n | min      | mean     | median   | stdev    |
| ------------------ | ---:| --------:| --------:| --------:| --------:|
| canary             | 7   | 27.11 s  | 28.00 s  | 28.12 s  | 0.61 s   |
| `rcstr_inventory`  | 7   | 26.60 s  | 27.99 s  | 27.96 s  | 0.80 s   |
| **delta** | | | **−0.01 s (−0.06 %)** | −0.16 s | |

Within noise. The proc-macro per-invocation overhead is offset by
reduced emitted-token volume (~40 % less code across all `rcstr!` sites
— the 852 inlinable literals collapse from ~80 tokens to ~6 tokens
each), so the compiler does materially less parsing and type-checking on
the macro output.

### Binary size

Release build of `next-napi-bindings` cdylib on `darwin-aarch64`:

| | canary | `rcstr_inventory` | delta |
| ---------------------------- | ------------------------------- |
------------------------------- | ------------------------------ |
| Release `.dylib` (raw) | 123,699,312 B (117.97 MiB) | 123,349,424 B
(117.64 MiB) | **−349,888 B (−0.28 %)** |
| Stripped (`strip -x`) | 83,502,264 B (79.63 MiB) | 83,434,984 B (79.57
MiB) | **−67,280 B (−0.08 %)** |
| Stripped + `gzip -9` | 28,967,762 B (27.63 MiB) | 28,958,198 B (27.62
MiB) | **−9,564 B (−0.03 %)** |

The raw release build shrinks by ~342 KiB — the 852 dead `static
PrehashedString` + inventory node + ctor entries no longer being emitted
for inlinable call sites. Most of that compresses well (dead data is
stripped and gzip handles the rest), so the user-facing impact in the
gzipped npm tarball is ~9 KiB.

<!-- NEXT_JS_LLM_PR -->
parent 225be6af
...@@ -9807,10 +9807,15 @@ dependencies = [ ...@@ -9807,10 +9807,15 @@ dependencies = [
"smallvec", "smallvec",
"triomphe 0.1.12", "triomphe 0.1.12",
"turbo-bincode", "turbo-bincode",
"turbo-rcstr-macros",
"turbo-tasks-hash", "turbo-tasks-hash",
"unty", "unty",
] ]
[[package]]
name = "turbo-rcstr-macros"
version = "0.1.0"
[[package]] [[package]]
name = "turbo-tasks" name = "turbo-tasks"
version = "0.1.0" version = "0.1.0"
......
[package]
name = "turbo-rcstr-macros"
version = "0.1.0"
edition = "2024"
license = "MIT"
[lib]
proc-macro = true
bench = false
[features]
# Mirror the threshold-affecting features of `turbo-rcstr`. The consuming
# crate routes them via `turbo-rcstr/atom_size_*` -> `turbo-rcstr-macros/atom_size_*`
# so the proc macro can compute the correct `MAX_INLINE_LEN` at expansion time
# instead of leaving an ambiguous range that needs a runtime const branch.
atom_size_64 = []
atom_size_128 = []
[lints]
workspace = true
[dependencies]
# Intentionally minimal: the only dependency is the standard `proc_macro`
# API. Dropping `syn` and `quote` removes ~hundreds of milliseconds of
# per-invocation overhead and a large slice of the proc-macro crate's
# build time. See the crate-level docs for the reasoning.
//! Proc macro implementation of `rcstr!`.
//!
//! The implementation deliberately avoids `syn` and `quote`. The macro is
//! invoked thousands of times across the workspace, so per-invocation cost
//! matters: we pattern-match on `proc_macro::TokenTree` directly to
//! identify a single string-literal token, ask the compiler for its
//! unescaped value via [`Literal::str_value`] (gated by the unstable
//! `proc_macro_value` feature), and emit the chosen expansion by parsing
//! a string template via `TokenStream::from_str`.
#![feature(proc_macro_value)]
use std::str::FromStr;
use proc_macro::{Literal, TokenStream, TokenTree};
/// `MAX_INLINE_LEN` for the active `turbo-rcstr` configuration. Mirrors
/// [`turbo_rcstr::tagged_value::MAX_INLINE_LEN`]
const MAX_INLINE_LEN: usize = if cfg!(feature = "atom_size_128") {
15
} else {
7
};
#[proc_macro]
pub fn rcstr(input: TokenStream) -> TokenStream {
// Fast path: input is a single string-literal token whose unescaped
// length we can determine cheaply. Otherwise (multi-token expressions
// like `concat!(...)`, identifiers, empty input, non-string literals,
// escape-bearing literals) defer to the const-branch expansion so
// const evaluation picks the arm at compile time.
//
// `input.clone()` is cheap — a `TokenStream` is an opaque handle into
// the proc-macro server's storage rather than an owned tree of tokens
// — so cloning here lets us consume one copy in `classify_literal`
// while keeping the original around for the fallback path.
{
let source = if let Some((lit, len)) = classify_literal(input.clone()) {
if len <= MAX_INLINE_LEN {
format!("::turbo_rcstr::inline_atom({lit}).unwrap()")
} else {
format!(
"{{ static RCSTR_STORAGE: ::turbo_rcstr::PrehashedString = \
::turbo_rcstr::make_const_prehashed_string({lit}); const RCSTR: \
::turbo_rcstr::RcStr = ::turbo_rcstr::from_static(&RCSTR_STORAGE); \
::turbo_rcstr::__rcstr_inventory_submit!( \
::turbo_rcstr::StaticRcStr(&RCSTR_STORAGE) ); RCSTR }}",
)
}
} else {
format!(
"{{ const TEXT: &str = {input}; if ::turbo_rcstr::is_atom_inlineable(TEXT) {{ \
::turbo_rcstr::inline_atom(TEXT).unwrap() }} else {{ static RCSTR_STORAGE: \
::turbo_rcstr::PrehashedString = \
::turbo_rcstr::make_const_prehashed_string(TEXT); const RCSTR: \
::turbo_rcstr::RcStr = ::turbo_rcstr::from_static(&RCSTR_STORAGE); \
::turbo_rcstr::__rcstr_inventory_submit!( \
::turbo_rcstr::StaticRcStr(&RCSTR_STORAGE) ); RCSTR }} }}",
)
};
TokenStream::from_str(&source).expect("emitted source parses")
}
}
/// If `input` is a single string-literal token, return the literal and
/// its unescaped length. Returns `None` for non-literal inputs, multi-
/// token inputs, or non-string literals (numeric, byte string, char,
/// etc.) so the caller falls back to the const-branch expansion.
///
/// [`Literal::str_value`] resolves all escape sequences (regular strings,
/// raw strings, unicode escapes) and reports an error for non-string
/// literals — exactly the inspection we want.
fn classify_literal(input: TokenStream) -> Option<(Literal, usize)> {
let mut iter = input.into_iter();
let TokenTree::Literal(lit) = iter.next()? else {
return None;
};
if iter.next().is_some() {
return None;
}
let value = lit.str_value().ok()?;
Some((lit, value.len()))
}
...@@ -5,8 +5,8 @@ edition = "2024" ...@@ -5,8 +5,8 @@ edition = "2024"
license = "MIT" license = "MIT"
[features] [features]
atom_size_64 = [] atom_size_64 = ["turbo-rcstr-macros/atom_size_64"]
atom_size_128 = [] atom_size_128 = ["turbo-rcstr-macros/atom_size_128"]
napi = ["dep:napi"] napi = ["dep:napi"]
[dependencies] [dependencies]
...@@ -21,6 +21,7 @@ bytes-str = { workspace = true } ...@@ -21,6 +21,7 @@ bytes-str = { workspace = true }
inventory = { workspace = true } inventory = { workspace = true }
smallvec = { workspace = true } smallvec = { workspace = true }
turbo-bincode = { workspace = true } turbo-bincode = { workspace = true }
turbo-rcstr-macros = { path = "../turbo-rcstr-macros" }
unty = { workspace = true } unty = { workspace = true }
[target.'cfg(not(target_family = "wasm"))'.dependencies] [target.'cfg(not(target_family = "wasm"))'.dependencies]
......
// Allow the `rcstr!` proc macro's emitted `::turbo_rcstr::...` paths to
// resolve when used inside this crate's own source (e.g. tests, doctests).
extern crate self as turbo_rcstr;
use std::{ use std::{
borrow::{Borrow, Cow}, borrow::{Borrow, Cow},
collections::HashMap, collections::HashMap,
...@@ -53,7 +57,7 @@ mod tagged_value; ...@@ -53,7 +57,7 @@ mod tagged_value;
/// `RcStr::from(...)`, or the `rcstr!` macro. /// `RcStr::from(...)`, or the `rcstr!` macro.
/// ///
/// ``` /// ```
/// # use turbo_rcstr::RcStr; /// # use turbo_rcstr::{RcStr, rcstr};
/// # /// #
/// let s = "foo"; /// let s = "foo";
/// let rc_s1: RcStr = s.into(); /// let rc_s1: RcStr = s.into();
...@@ -517,6 +521,20 @@ pub struct StaticRcStr(pub &'static PrehashedString); ...@@ -517,6 +521,20 @@ pub struct StaticRcStr(pub &'static PrehashedString);
inventory::collect!(StaticRcStr); inventory::collect!(StaticRcStr);
/// Forwarder around [`inventory::submit!`] that lets the `rcstr!` proc macro
/// emit a single path it can rely on, without depending on whether
/// `turbo_rcstr::inventory` is reachable as a macro path in the call site
/// crate. Macros emitted from a proc macro lose access to the proc macro
/// crate's deps, so the submission has to bounce through this declarative
/// macro defined where `inventory::submit!` is in scope.
#[doc(hidden)]
#[macro_export]
macro_rules! __rcstr_inventory_submit {
($value:expr) => {
$crate::inventory::submit!($value);
};
}
/// Read-only lookup table mapping precomputed hash -> static PrehashedString. /// Read-only lookup table mapping precomputed hash -> static PrehashedString.
/// Built once on first access from all `rcstr!` constants collected by `inventory`. /// Built once on first access from all `rcstr!` constants collected by `inventory`.
/// ///
...@@ -529,6 +547,13 @@ static STATIC_TABLE: LazyLock< ...@@ -529,6 +547,13 @@ static STATIC_TABLE: LazyLock<
let mut map: HashMap<u64, SmallVec<[&'static PrehashedString; 1]>, FxBuildHasher> = let mut map: HashMap<u64, SmallVec<[&'static PrehashedString; 1]>, FxBuildHasher> =
HashMap::with_hasher(FxBuildHasher); HashMap::with_hasher(FxBuildHasher);
for StaticRcStr(phs) in inventory::iter::<StaticRcStr> { for StaticRcStr(phs) in inventory::iter::<StaticRcStr> {
if phs.value.as_str().len() <= MAX_INLINE_LEN {
// This is rare, but possible if our macro cannot determine the length of the string at
// macro time we may end up with a wasted PrehashedString submitted to inventory.
// Just skip it
continue;
}
let entries = map.entry(phs.hash).or_default(); let entries = map.entry(phs.hash).or_default();
// Deduplicate: skip if an entry with the same string content exists // Deduplicate: skip if an entry with the same string content exists
// Mostly linkers will merge static strings but this isn't guaranteed so we cannot just rely // Mostly linkers will merge static strings but this isn't guaranteed so we cannot just rely
...@@ -545,30 +570,9 @@ static STATIC_TABLE: LazyLock< ...@@ -545,30 +570,9 @@ static STATIC_TABLE: LazyLock<
}); });
/// Create an rcstr from a string literal. /// Create an rcstr from a string literal.
/// Allocates the RcStr inline when possible, otherwise uses a static `PrehashedString`. In either /// Allocates the RcStr inline when possible, otherwise uses a static `PrehashedString`. In
/// case this is a compile time constant /// either case this is a compile time constant
#[macro_export] pub use turbo_rcstr_macros::rcstr;
macro_rules! rcstr {
($s:expr) => {{
let text = $s;
// This condition can be compile time evaluated and inlined.
if $crate::is_atom_inlineable(text) {
$crate::inline_atom(text).unwrap()
} else {
const fn get_rcstr() -> $crate::RcStr {
// Allocate static storage for the PrehashedString
static RCSTR_STORAGE: $crate::PrehashedString =
$crate::make_const_prehashed_string($s);
// Register with inventory so deserialization can find this static
$crate::inventory::submit!($crate::StaticRcStr(&RCSTR_STORAGE));
// This basically just tags a bit onto the raw pointer and wraps it in an RcStr
// should be fast enough to do every time.
$crate::from_static(&RCSTR_STORAGE)
}
get_rcstr()
}
}};
}
/// noop /// noop
impl ShrinkToFit for RcStr { impl ShrinkToFit for RcStr {
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment