Unverified Commit 4425414a authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

turbo-tasks-malloc: report memory from mimalloc (#97761)

### What?

`turbo-tasks-malloc` keeps a process-wide `ALLOCATED` atomic to answer
`memory_usage()`. To optimize performance thread local buffers were
maintained but this lead to code size issues. So instead we rely on
mimalloc apis saving 1.6M of binary size


### Why not ask the OS?

That was the obvious alternative, and it does not work well. The natural
per-platform figures are not the same quantity: macOS `phys_footprint`
and Windows `PrivateUsage` account for compressed and swapped pages, but
Linux `VmRSS` does not — so under swap or zram the Linux number can read
flat, or fall, while real consumption climbs. Closing that gap means
`/proc/self/smaps_rollup` (`Pss + SwapPss`), which is not the cheap
one-line read it looks like, it takes on the order of [100ms per
4GB](https://github.com/ncabatoff/process-exporter/issues/246), and it
would contend with other allocations in our process.

### How?

`memory_usage()` reads `current_commit` from `mi_process_info`. We fall
back to the `ALLOCATED` atomic only when `custom_allocator` is off.


### Behaviour changes

**It does not track frees in lock step.** mimalloc reuses and purges
pages on its own schedule or when eviction calls `collect`

### Size

Rust inlines `#[global_allocator]` methods into every allocation site,
so the buffer arithmetic and the branch guarding it were duplicated
across the whole binary. Dropping them from the default build is where
the win comes from.

`libnext_napi_bindings.dylib`, release profile as shipped (thin LTO,
`codegen-units = 1`).

| Section | canary | this PR | Δ |
|---|---:|---:|---:|
| `__text` | 64,833,760 | 63,319,776 | **−1,513,984** |
| `__eh_frame` | 8,966,904 | 8,748,464 | −218,440 |
| `__gcc_except_tab` | 2,234,428 | 2,229,484 | −4,944 |
| `__unwind_info` | 1,489,072 | 1,484,644 | −4,428 |
| **Total** | | | **−1,741,796 = −1.66 MiB** |

### Performance

`cargo bench -p turbo-tasks-malloc --bench allocation`, macOS aarch64,
revisions interleaved across 3 rounds so drift hits both arms equally.
Figures are the paired per-round deltas.

| benchmark | canary | this PR | mean Δ | per-round |
|---|---:|---:|---:|---|
| `alloc_dealloc` | 6.65 ns | 6.45 ns | **−2.9%** | −3.7% .. −2.4% |
| `alloc_realloc_dealloc` | 18.62 ns | 16.76 ns | **−10.0%** | −10.2% ..
−9.7% |

Removing the counter takes the buffer arithmetic and its guarding branch
out of every allocation, and a realloc pays that twice — which is
roughly the shape of the result. That said, the absolute numbers are
small (a couple of hundred picoseconds on `alloc_dealloc`) and this
benchmark is noisy, so treat the direction as the signal rather than the
magnitude.
parent 9c626ac9
......@@ -129,8 +129,11 @@ impl EvictionControl {
evict
}
/// Call after completing an eviction cycle. Seeds the memory floor with the
/// post-eviction usage; later cycles lower it further as memory settles.
/// Call after completing an eviction cycle and after mimalloc is cleaned up
/// with [`TurboMalloc::collect`], since the freed memory is only reflected
/// in [`TurboMalloc::memory_usage`] once it has been. Seeds the memory floor
/// with the post-eviction usage; later cycles lower it further as memory
/// settles.
pub(crate) fn record_eviction(&mut self) {
self.memory_floor = Some(TurboMalloc::memory_usage());
}
......
......@@ -3044,8 +3044,6 @@ impl TurboTasksBackend {
// memory so racing with execution is as likely to save time as
// cost it.
self.storage.evict_after_snapshot(background_span.id());
// Sample the post-eviction floor as the new baseline.
eviction_control.record_eviction();
true
} else {
false
......@@ -3098,6 +3096,12 @@ impl TurboTasksBackend {
{
TurboMalloc::collect(true);
}
// Sample the new baseline after the collect above, which is what
// makes the evicted memory show up in `memory_usage`.
if ran_eviction {
eviction_control.record_eviction();
}
}
}
}
......
......@@ -7,7 +7,7 @@ use std::{
ops::{Add, AddAssign},
};
use self::counter::{add, flush, get, remove, update};
use self::counter::{add, flush, remove, update};
#[derive(Default, Clone, Debug)]
pub struct AllocationInfo {
......@@ -85,16 +85,52 @@ impl AllocationCounters {
pub struct TurboMalloc;
impl TurboMalloc {
/// Returns the current amount of live memory (bytes allocated minus freed)
/// tracked across all threads.
/// Returns the bytes the allocator currently has committed from the OS.
///
/// For efficiency reasons every thread only synchronizes with this counter after ~100K bytes of
/// allocations or deallocations. So this could be off by as much as 100K*number of thread in
/// either direction.
/// This is the allocator's own accounting, not a per-OS query, so it means the same thing on
/// every platform. It counts what mimalloc has taken from the OS, which includes allocator
/// overhead and fragmentation, and excludes anything mimalloc did not hand out — the binary,
/// mmap'd files, and any memory allocated by the embedding process. It is a measure of what
/// this allocator holds, not of the process's total footprint.
///
/// It does not track frees in lock step. mimalloc reuses and purges pages on its own
/// schedule, so the figure lags a burst of frees, and memory abandoned by threads that have
/// since exited is only reclaimed by a forcing [`Self::collect`].
///
/// Without the `custom_allocator` feature this is a process-wide counter of live bytes
/// (allocations minus deallocations), maintained by [`self::counter`]. That figure is
/// approximate: threads buffer their updates, so it can be off by up to a fixed amount per
/// thread in either direction.
pub fn memory_usage() -> usize {
get()
#[cfg(all(feature = "custom_allocator", not(target_family = "wasm")))]
{
// `current_commit` is a relaxed atomic load, but `mi_process_info` also calls
// `_mi_prim_process_info`, which is a `getrusage` (plus a `task_info` on macOS). All
// eight out-params are optional, so ask only for the one we use.
let mut current_commit = 0usize;
// Safety: every out-param is either null or a valid `usize` we own.
unsafe {
libmimalloc_sys::mi_process_info(
std::ptr::null_mut(),
std::ptr::null_mut(),
std::ptr::null_mut(),
std::ptr::null_mut(),
std::ptr::null_mut(),
&mut current_commit,
std::ptr::null_mut(),
std::ptr::null_mut(),
);
}
current_commit
}
#[cfg(not(all(feature = "custom_allocator", not(target_family = "wasm"))))]
{
self::counter::get()
}
}
/// Clears the calling thread's allocation counters. Call this when a thread is about to stop,
/// so a thread that reuses its slot does not inherit the previous totals.
pub fn thread_stop() {
flush();
}
......@@ -205,6 +241,40 @@ unsafe impl GlobalAlloc for TurboMalloc {
mod tests {
use super::TurboMalloc;
// `memory_usage` reports what *this* allocator has committed, so the test binary has to
// actually route its allocations through it. Without this the `vec!` below goes to the
// system allocator and mimalloc's counter never moves.
#[global_allocator]
static ALLOC: TurboMalloc = TurboMalloc;
/// Also guards against the counter silently becoming unavailable. mimalloc's `committed`
/// stat is maintained even at `MI_STAT 0` (which is what a release build compiles, since
/// `build.rs` sets `MI_DEBUG=0`) because the `mi_os_stat_*` macros are not gated on
/// `MI_STAT` — an internal detail rather than a documented guarantee, so a
/// `libmimalloc-sys` bump could zero it out. If that happens, this fails.
#[test]
fn memory_usage_is_reported_and_tracks_a_large_allocation() {
let before = TurboMalloc::memory_usage();
assert!(before > 0, "a running process has live memory");
// Large enough to dwarf whatever else the test process does concurrently, and written to
// so the pages are actually committed.
const SIZE: usize = 256 * 1024 * 1024;
let mut buffer = vec![0u8; SIZE];
for chunk in buffer.chunks_mut(4096) {
chunk[0] = 1;
}
std::hint::black_box(&buffer);
let after = TurboMalloc::memory_usage();
assert!(
after >= before + SIZE / 2,
"expected a rise of at least {} bytes, got {before} -> {after}",
SIZE / 2
);
drop(buffer);
}
#[test]
fn memory_pressure_is_in_range() {
let value = TurboMalloc::memory_pressure();
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment