trace-server: move Span's derived half into lazy-allocated side arrays

tl;dr size_of::<Span>() 328 -> 192
Not pre-computing things also improves ingestion speed: 269->280MB/s

Five per-span values are pure functions of the trace. They can be
computed on demand and discarded on invalidation.
 * end
 * total_time
 * corrected_self_time
 * corrected_total_time
 * subtree totals bundle

By preallocating these, we add 112 bytes to every span regardless of whether
they're requested.

`max_depth` moved to an AtomicU8 with a sentinel value for "not set"

`extra` moves to a locked hashmap of boxes. `SpanRef::extra()` still
retrusn a SpanExtra ref. The box it points to won't move while it's
stored in the map.

The timestamps move out, and are allocated on first touch. Slots are
AtomicU64s rather than OnceLocks; we don't need a mutex because the
value is the result of a pure function. In the rare case two threads
race, they'll still end up with the same result.

The `MEMORY_REPORT=warm` env var can force every derived value to be
derived before reporting. This is less about performance of the
server, and more to be able to estimate the peak allocation when
the client is actually requesting data.
parent 78871311
...@@ -17,7 +17,10 @@ use std::{ ...@@ -17,7 +17,10 @@ use std::{
/// Number of elements per chunk. Power of two so `idx / CHUNK_SIZE` and /// Number of elements per chunk. Power of two so `idx / CHUNK_SIZE` and
/// `idx % CHUNK_SIZE` compile to a shift and a mask. /// `idx % CHUNK_SIZE` compile to a shift and a mask.
const CHUNK_SIZE: usize = 1 << 16; ///
/// Public because [`crate::cold`] keeps arrays parallel to these chunks and must
/// agree on the boundary.
pub const CHUNK_SIZE: usize = 1 << 16;
/// Returns the chunk index and intra-chunk offset for an element index. /// Returns the chunk index and intra-chunk offset for an element index.
#[inline] #[inline]
......
//! Lazily-allocated storage for the derived half of a span.
//!
//! Five per-span values — `end`, `total_time`, `corrected_self_time`,
//! `corrected_total_time`, and the subtree totals bundle — are pure functions of
//! the trace, computed on demand and thrown away when a span is invalidated.
//! Kept as fields on `Span` they cost 112 bytes on *every* span whether or not
//! anything ever asked for them, which at tens of millions of spans is over a
//! gigabyte that a headless ingest never reads.
//!
//! So they live here instead, in arrays parallel to the span array's 64K chunks,
//! allocated per chunk on first touch. During a read with no client attached
//! nothing allocates at all.
//!
//! # Why atomics and not `OnceLock`
//!
//! Every value here is a pure function of a store held under a read lock, so two
//! threads racing to compute one produce the same answer and write the same
//! bits. That is a weaker contract than `OnceLock` provides, and it is all that
//! is needed: a relaxed load, and a relaxed store of a value someone else may
//! have just stored identically. The cost of the race is duplicated work, never
//! a wrong answer. In exchange the per-value guard word disappears — which was
//! half the bytes.
//!
//! This is only sound while the invariant holds. **If a future change makes any
//! of these values depend on mutable state reachable through `&self`, this
//! scheme breaks.**
//!
//! # Encoding
//!
//! Zero means "not computed", so a freshly zeroed allocation is a valid empty
//! cache — which is the second reason for this design. Chunks come from
//! [`Box::new_zeroed_slice`], so the allocator hands back calloc'd pages and the
//! kernel faults them in only as they are written. Touching three spans in a
//! chunk commits three pages, not the whole 2 MiB.
//!
//! Timestamps are therefore stored biased by one. One timestamp collides:
//! `Timestamp::MAX`, which is the root span's `self_end` (see `store::new_root_span`)
//! and hence its `end`. Biased it wraps back to zero, so the root's `end` reads
//! as "not computed" and is recalculated on every call. That is one span out of
//! millions, and the alternative — a separate validity word per value — costs
//! more than the recomputation ever will.
//!
//! The totals bundle needs no bias: `span_count` counts the span itself, so a
//! computed bundle always has `span_count >= 1` and zero is unambiguous.
use std::sync::{
OnceLock,
atomic::{AtomicU64, Ordering},
};
use crate::{chunked_vec::CHUNK_SIZE, span::SpanTotals, timestamp::Timestamp};
/// Which of a span's four cached timestamps a slot holds.
#[derive(Clone, Copy)]
pub enum TimeSlot {
End = 0,
TotalTime = 1,
CorrectedSelfTime = 2,
CorrectedTotalTime = 3,
}
const TIMES_PER_SPAN: usize = 4;
/// `allocations`, `deallocations`, `persistent_allocations`, `allocation_count`,
/// `span_count` — in that order, matching [`SpanTotals`].
const TOTALS_PER_SPAN: usize = 5;
const TOTALS_SPAN_COUNT: usize = 4;
/// Allocate `len` zeroed slots without writing to them, so the pages stay
/// untouched until a span in this chunk is actually cached.
fn zeroed(len: usize) -> Box<[AtomicU64]> {
// SAFETY: `AtomicU64` has the same size and bit validity as `u64`, and an
// all-zero `u64` is a valid value, so a zeroed allocation is a slice of
// initialized `AtomicU64`s holding 0.
unsafe { Box::new_zeroed_slice(len).assume_init() }
}
#[derive(Default)]
struct ColdChunk {
times: OnceLock<Box<[AtomicU64]>>,
totals: OnceLock<Box<[AtomicU64]>>,
}
/// The derived-value arrays for the whole store, one entry per span chunk.
#[derive(Default)]
pub struct ColdStore {
chunks: Vec<ColdChunk>,
}
impl ColdStore {
pub fn new() -> Self {
Self::default()
}
pub fn clear(&mut self) {
self.chunks.clear();
}
/// Make room for `span_index`. Called from span creation, which holds
/// `&mut Store`, so no reader can observe the `Vec` reallocating.
pub fn reserve_for(&mut self, span_index: usize) {
let needed = span_index / CHUNK_SIZE + 1;
if self.chunks.len() < needed {
self.chunks.resize_with(needed, ColdChunk::default);
}
}
fn split(span_index: usize) -> (usize, usize) {
(span_index / CHUNK_SIZE, span_index % CHUNK_SIZE)
}
/// One cached timestamp slot. Allocates this chunk's array on first touch.
pub fn time(&self, span_index: usize, slot: TimeSlot) -> &AtomicU64 {
let (chunk, offset) = Self::split(span_index);
let times = self.chunks[chunk]
.times
.get_or_init(|| zeroed(TIMES_PER_SPAN * CHUNK_SIZE));
&times[offset * TIMES_PER_SPAN + slot as usize]
}
fn totals_slots(&self, span_index: usize) -> &[AtomicU64] {
let (chunk, offset) = Self::split(span_index);
let totals = self.chunks[chunk]
.totals
.get_or_init(|| zeroed(TOTALS_PER_SPAN * CHUNK_SIZE));
&totals[offset * TOTALS_PER_SPAN..(offset + 1) * TOTALS_PER_SPAN]
}
pub fn totals(&self, span_index: usize) -> Option<SpanTotals> {
let s = self.totals_slots(span_index);
// `span_count` is the validity marker: a computed bundle counts at least
// the span itself.
if s[TOTALS_SPAN_COUNT].load(Ordering::Relaxed) == 0 {
return None;
}
Some(SpanTotals {
allocations: s[0].load(Ordering::Relaxed),
deallocations: s[1].load(Ordering::Relaxed),
persistent_allocations: s[2].load(Ordering::Relaxed),
allocation_count: s[3].load(Ordering::Relaxed),
span_count: s[4].load(Ordering::Relaxed),
})
}
pub fn set_totals(&self, span_index: usize, totals: &SpanTotals) {
let s = self.totals_slots(span_index);
debug_assert!(
totals.span_count >= 1,
"span_count is the validity marker and must count the span itself"
);
// Store `span_count` last: it is what makes the bundle readable, so the
// other four are already in place by the time any reader accepts it.
s[0].store(totals.allocations, Ordering::Relaxed);
s[1].store(totals.deallocations, Ordering::Relaxed);
s[2].store(totals.persistent_allocations, Ordering::Relaxed);
s[3].store(totals.allocation_count, Ordering::Relaxed);
s[TOTALS_SPAN_COUNT].store(totals.span_count, Ordering::Release);
}
/// Drop every cached value for one span. Takes `&mut self`, so `get_mut`
/// avoids the atomic loads and — more usefully — a chunk that was never
/// allocated short-circuits to nothing at all. That is why invalidation
/// during a headless ingest costs a pointer check per ancestor rather than
/// clearing 112 bytes.
pub fn invalidate(&mut self, span_index: usize) {
let (chunk, offset) = Self::split(span_index);
let Some(chunk) = self.chunks.get_mut(chunk) else {
return;
};
if let Some(times) = chunk.times.get_mut() {
for slot in &mut times[offset * TIMES_PER_SPAN..(offset + 1) * TIMES_PER_SPAN] {
*slot.get_mut() = 0;
}
}
if let Some(totals) = chunk.totals.get_mut() {
for slot in &mut totals[offset * TOTALS_PER_SPAN..(offset + 1) * TOTALS_PER_SPAN] {
*slot.get_mut() = 0;
}
}
}
/// Bytes committed so far. Counts whole arrays, so it is an upper bound on
/// resident memory when only part of a chunk has been touched.
pub fn allocated_bytes(&self) -> usize {
self.chunks
.iter()
.map(|chunk| {
let times = chunk.times.get().map_or(0, |t| t.len());
let totals = chunk.totals.get().map_or(0, |t| t.len());
(times + totals) * std::mem::size_of::<AtomicU64>()
})
.sum()
}
/// How many chunks have materialized each array, for the memory report.
pub fn materialized_chunks(&self) -> (usize, usize) {
let times = self
.chunks
.iter()
.filter(|c| c.times.get().is_some())
.count();
let totals = self
.chunks
.iter()
.filter(|c| c.totals.get().is_some())
.count();
(times, totals)
}
}
/// Read a biased timestamp slot. `None` means "not computed".
pub fn load_time(slot: &AtomicU64) -> Option<Timestamp> {
match slot.load(Ordering::Relaxed) {
0 => None,
biased => Some(Timestamp::from_value(biased - 1)),
}
}
/// Write a timestamp slot, biased by one so zero stays free as "not computed".
///
/// `Timestamp::MAX` wraps to zero and so never caches; see the module docs.
pub fn store_time(slot: &AtomicU64, value: Timestamp) {
slot.store((*value).wrapping_add(1), Ordering::Relaxed);
}
...@@ -18,6 +18,7 @@ use self::{ ...@@ -18,6 +18,7 @@ use self::{
mod bottom_up; mod bottom_up;
mod chunked_vec; mod chunked_vec;
mod cold;
#[cfg(test)] #[cfg(test)]
mod golden; mod golden;
mod lazy_sorted_vec; mod lazy_sorted_vec;
......
...@@ -13,6 +13,7 @@ use self::{reader::TraceReader, server::serve, store_container::StoreContainer}; ...@@ -13,6 +13,7 @@ use self::{reader::TraceReader, server::serve, store_container::StoreContainer};
mod bottom_up; mod bottom_up;
mod chunked_vec; mod chunked_vec;
mod cold;
mod lazy_sorted_vec; mod lazy_sorted_vec;
mod memory_report; mod memory_report;
mod reader; mod reader;
......
...@@ -50,6 +50,12 @@ pub struct MemoryReport { ...@@ -50,6 +50,12 @@ pub struct MemoryReport {
pub names_populated: usize, pub names_populated: usize,
pub extra_populated: usize, pub extra_populated: usize,
/// Bytes committed by the lazily-allocated derived-value arrays, and how
/// many chunks have materialized each of the two arrays. Zero across the
/// board during a headless ingest; that is the point of the design.
pub cold_bytes: usize,
pub cold_chunks: (usize, usize),
/// `TurboMalloc::memory_usage()` at the same instant, for comparison. /// `TurboMalloc::memory_usage()` at the same instant, for comparison.
pub allocator_live_bytes: usize, pub allocator_live_bytes: usize,
} }
...@@ -63,6 +69,7 @@ impl MemoryReport { ...@@ -63,6 +69,7 @@ impl MemoryReport {
+ self.arg_heap_bytes + self.arg_heap_bytes
+ self.self_time_bytes + self.self_time_bytes
+ self.memory_sample_bytes + self.memory_sample_bytes
+ self.cold_bytes
} }
} }
...@@ -125,6 +132,17 @@ impl fmt::Display for MemoryReport { ...@@ -125,6 +132,17 @@ impl fmt::Display for MemoryReport {
Mb(self.memory_sample_bytes), Mb(self.memory_sample_bytes),
per_span(self.memory_sample_bytes), per_span(self.memory_sample_bytes),
)?; )?;
writeln!(
f,
" {:<22} {} {:>8.1} B/span times {}/{} chunks, totals {}/{} chunks",
"derived (cold)",
Mb(self.cold_bytes),
per_span(self.cold_bytes),
self.cold_chunks.0,
self.spans.div_ceil(1 << 16),
self.cold_chunks.1,
self.spans.div_ceil(1 << 16),
)?;
writeln!( writeln!(
f, f,
" {:<22} {} {:>8.1} B/span", " {:<22} {} {:>8.1} B/span",
......
...@@ -367,9 +367,15 @@ impl TraceReader { ...@@ -367,9 +367,15 @@ impl TraceReader {
/// write lock and walks every span, so it is gated behind the env var /// write lock and walks every span, so it is gated behind the env var
/// rather than being always-on. /// rather than being always-on.
fn print_memory_report(&self) { fn print_memory_report(&self) {
if env::var("MEMORY_REPORT").is_ok() { let Ok(mode) = env::var("MEMORY_REPORT") else {
println!("{}", self.store.write().memory_report()); return;
};
if mode == "warm" {
// Compute everything a client rendering the whole trace would, so
// the report shows the serving peak and not just the ingest peak.
self.store.read().warm_all_derived();
} }
println!("{}", self.store.write().memory_report());
} }
fn wait_for_more_data( fn wait_for_more_data(
......
use std::{ use std::{
num::{NonZeroU64, NonZeroUsize}, num::{NonZeroU64, NonZeroUsize},
sync::{Arc, OnceLock}, sync::{Arc, OnceLock, atomic::AtomicU8},
}; };
use hashbrown::HashMap; use hashbrown::HashMap;
...@@ -11,6 +11,10 @@ use crate::{lazy_sorted_vec::LazySortedVec, timestamp::Timestamp}; ...@@ -11,6 +11,10 @@ use crate::{lazy_sorted_vec::LazySortedVec, timestamp::Timestamp};
pub type SpanIndex = NonZeroUsize; pub type SpanIndex = NonZeroUsize;
/// Sentinel in [`Span::max_depth`] meaning "not yet computed". Real values are
/// bounded by `store::CUT_OFF_DEPTH` (80), so this cannot collide.
pub const MAX_DEPTH_UNSET: u8 = u8::MAX;
/// Storage for `Span::args` ~32% of spans have <=1 arg (typically just the /// Storage for `Span::args` ~32% of spans have <=1 arg (typically just the
/// `name` key for `turbo_tasks::function` spans), so inlining one entry /// `name` key for `turbo_tasks::function` spans), so inlining one entry
/// avoids a heap allocation in this common case. /// avoids a heap allocation in this common case.
...@@ -32,27 +36,44 @@ pub struct Span { ...@@ -32,27 +36,44 @@ pub struct Span {
pub events: LazySortedVec<SpanEvent>, pub events: LazySortedVec<SpanEvent>,
pub is_complete: bool, pub is_complete: bool,
/// Height of the subtree below this span, lazily computed and cached here
/// rather than inside [`SpanTotals`].
///
/// It lives in the hot record because it is the one derived value the viewer
/// asks for about *every* span before drawing anything
/// (`viewer.rs`: `root_spans.par_iter()` forcing `max_depth()`). While it was
/// bundled with the subtree totals, that single call materialized the whole
/// totals cache for the entire trace, which would defeat any attempt to keep
/// the derived half lazily allocated.
///
/// [`MAX_DEPTH_UNSET`] means "not computed". Values fit a `u8` because
/// `store::CUT_OFF_DEPTH` caps tree height at 80, and this byte is free —
/// it shares padding with `depth` and `is_complete`.
pub max_depth: AtomicU8,
// These values are computed automatically: // These values are computed automatically:
pub self_allocations: u64, pub self_allocations: u64,
pub self_allocation_count: u64, pub self_allocation_count: u64,
pub self_deallocations: u64, pub self_deallocations: u64,
pub self_deallocation_count: u64, pub self_deallocation_count: u64,
// These values are computed when accessed (and maybe deleted during writing). // The derived half — subtree totals and the four lazily-computed timestamps —
// Bundling the subtree totals into a single OnceLock pays a small cost on // is not stored here at all. See [`crate::cold`]: it lives in arrays parallel
// partial reads in exchange for a much-reduced lock count per Span. // to the span chunks, allocated on first touch, so a headless ingest never
pub totals: OnceLock<SpanTotals>, // pays for it.
pub time_data: SpanTimeData, pub time_data: SpanTimeData,
pub extra: OnceLock<Box<SpanExtra>>,
/// Lazy first-touch via `OnceLock`, but inline rather than boxed: ~96% of /// Lazy first-touch via `OnceLock`, but inline rather than boxed: ~96% of
/// spans get names populated after browsing, never invalidated, so the box /// spans get names populated after browsing, never invalidated, so the box
/// indirection is pure overhead. /// indirection is pure overhead.
pub names: OnceLock<SpanNames>, pub names: OnceLock<SpanNames>,
} }
#[derive(Default)] /// Subtree aggregates, filled together by one `OnceLock` so a partial read
/// pays a single lock rather than one per field.
///
/// Deliberately does *not* include the subtree height; see [`Span::max_depth`].
#[derive(Default, Clone, Copy)]
pub struct SpanTotals { pub struct SpanTotals {
pub max_depth: u32,
pub allocations: u64, pub allocations: u64,
pub deallocations: u64, pub deallocations: u64,
pub persistent_allocations: u64, pub persistent_allocations: u64,
...@@ -60,6 +81,8 @@ pub struct SpanTotals { ...@@ -60,6 +81,8 @@ pub struct SpanTotals {
pub span_count: u64, pub span_count: u64,
} }
/// The timing values the reader writes. The derived ones (`end`, `total_time`
/// and the two corrected times) are in [`crate::cold`].
#[derive(Default)] #[derive(Default)]
pub struct SpanTimeData { pub struct SpanTimeData {
// These values won't change after creation: // These values won't change after creation:
...@@ -70,12 +93,6 @@ pub struct SpanTimeData { ...@@ -70,12 +93,6 @@ pub struct SpanTimeData {
// These values are computed automatically: // These values are computed automatically:
pub self_time: Timestamp, pub self_time: Timestamp,
// These values are computed when accessed (and maybe deleted during writing):
pub end: OnceLock<Timestamp>,
pub total_time: OnceLock<Timestamp>,
pub corrected_self_time: OnceLock<Timestamp>,
pub corrected_total_time: OnceLock<Timestamp>,
} }
#[derive(Default)] #[derive(Default)]
...@@ -97,10 +114,6 @@ pub struct SpanNames { ...@@ -97,10 +114,6 @@ pub struct SpanNames {
} }
impl Span { impl Span {
pub fn extra(&self) -> &SpanExtra {
self.extra.get_or_init(Default::default)
}
pub fn names(&self) -> &SpanNames { pub fn names(&self) -> &SpanNames {
self.names.get_or_init(|| self.compute_names()) self.names.get_or_init(|| self.compute_names())
} }
...@@ -216,7 +229,7 @@ const _: () = assert!(std::mem::size_of::<SpanEvent>() == 16); ...@@ -216,7 +229,7 @@ const _: () = assert!(std::mem::size_of::<SpanEvent>() == 16);
/// a field addition a compile error rather than a code-review catch; lower it as /// a field addition a compile error rather than a code-review catch; lower it as
/// the hot/cold split lands, and never raise it without a note saying what the /// the hot/cold split lands, and never raise it without a note saying what the
/// bytes bought. /// bytes bought.
const _: () = assert!(std::mem::size_of::<Span>() <= 328); const _: () = assert!(std::mem::size_of::<Span>() == 192);
impl SpanEvent { impl SpanEvent {
/// Constructs a self-time event from start and end timestamps. Returns /// Constructs a self-time event from start and end timestamps. Returns
......
...@@ -3,6 +3,7 @@ use std::{ ...@@ -3,6 +3,7 @@ use std::{
collections::VecDeque, collections::VecDeque,
fmt::{Debug, Formatter}, fmt::{Debug, Formatter},
num::NonZeroU64, num::NonZeroU64,
sync::atomic::Ordering,
vec, vec,
}; };
...@@ -14,9 +15,10 @@ use turbo_rcstr::{RcStr, rcstr}; ...@@ -14,9 +15,10 @@ use turbo_rcstr::{RcStr, rcstr};
use crate::{ use crate::{
FxIndexMap, FxIndexMap,
bottom_up::build_bottom_up_graph, bottom_up::build_bottom_up_graph,
cold::{TimeSlot, load_time, store_time},
span::{ span::{
Span, SpanEvent, SpanEventKind, SpanExtra, SpanGraphEvent, SpanIndex, SpanName, SpanNames, MAX_DEPTH_UNSET, Span, SpanEvent, SpanEventKind, SpanExtra, SpanGraphEvent, SpanIndex,
SpanTimeData, SpanTotals, SpanName, SpanNames, SpanTimeData, SpanTotals,
}, },
span_bottom_up_ref::SpanBottomUpRef, span_bottom_up_ref::SpanBottomUpRef,
span_graph_ref::{SpanGraphEventRef, SpanGraphRef, event_map_to_list}, span_graph_ref::{SpanGraphEventRef, SpanGraphRef, event_map_to_list},
...@@ -60,24 +62,32 @@ impl<'a> SpanRef<'a> { ...@@ -60,24 +62,32 @@ impl<'a> SpanRef<'a> {
} }
pub fn extra(&self) -> &'a SpanExtra { pub fn extra(&self) -> &'a SpanExtra {
self.span.extra() self.store.extra(self.index)
} }
pub fn names(&self) -> &'a SpanNames { pub fn names(&self) -> &'a SpanNames {
self.span.names() self.span.names()
} }
/// Cached in [`crate::cold`] rather than on the span.
///
/// The root's `end` is `Timestamp::MAX`, which the biased encoding cannot
/// represent, so it recomputes on every call. One span, and cheap next to the
/// bytes the encoding saves — see the `cold` module docs.
pub fn end(&self) -> Timestamp { pub fn end(&self) -> Timestamp {
let time_data = self.time_data(); let slot = self.store.cold.time(self.index, TimeSlot::End);
*time_data.end.get_or_init(|| { if let Some(cached) = load_time(slot) {
max( return cached;
time_data.self_end, }
self.children() let computed = max(
.map(|child| child.end()) self.span.time_data.self_end,
.max() self.children()
.unwrap_or_default(), .map(|child| child.end())
) .max()
}) .unwrap_or_default(),
);
store_time(slot, computed);
computed
} }
pub fn is_complete(&self) -> bool { pub fn is_complete(&self) -> bool {
...@@ -177,13 +187,18 @@ impl<'a> SpanRef<'a> { ...@@ -177,13 +187,18 @@ impl<'a> SpanRef<'a> {
} }
pub fn total_time(&self) -> Timestamp { pub fn total_time(&self) -> Timestamp {
*self.time_data().total_time.get_or_init(|| { let slot = self.store.cold.time(self.index, TimeSlot::TotalTime);
self.children() if let Some(cached) = load_time(slot) {
.map(|child| child.total_time()) return cached;
.reduce(|a, b| a + b) }
.unwrap_or_default() let computed = self
+ self.self_time() .children()
}) .map(|child| child.total_time())
.reduce(|a, b| a + b)
.unwrap_or_default()
+ self.self_time();
store_time(slot, computed);
computed
} }
/// Compute (or fetch) the bundled subtree totals. All six totals share a /// Compute (or fetch) the bundled subtree totals. All six totals share a
...@@ -191,27 +206,27 @@ impl<'a> SpanRef<'a> { ...@@ -191,27 +206,27 @@ impl<'a> SpanRef<'a> {
/// every field; subsequent calls return cached values. Children's bundles /// every field; subsequent calls return cached values. Children's bundles
/// are computed recursively, so depth-many calls happen once per subtree /// are computed recursively, so depth-many calls happen once per subtree
/// regardless of which field is queried first. /// regardless of which field is queried first.
fn totals(&self) -> &'a SpanTotals { fn totals(&self) -> SpanTotals {
self.span.totals.get_or_init(|| { if let Some(cached) = self.store.cold.totals(self.index) {
let mut t = SpanTotals { return cached;
max_depth: 0, }
allocations: self.self_allocations(), let mut t = SpanTotals {
deallocations: self.self_deallocations(), allocations: self.self_allocations(),
persistent_allocations: self.self_persistent_allocations(), deallocations: self.self_deallocations(),
allocation_count: self.self_allocation_count(), persistent_allocations: self.self_persistent_allocations(),
span_count: 1, allocation_count: self.self_allocation_count(),
}; span_count: 1,
for child in self.children() { };
let c = child.totals(); for child in self.children() {
t.max_depth = max(t.max_depth, c.max_depth + 1); let c = child.totals();
t.allocations += c.allocations; t.allocations += c.allocations;
t.deallocations += c.deallocations; t.deallocations += c.deallocations;
t.persistent_allocations += c.persistent_allocations; t.persistent_allocations += c.persistent_allocations;
t.allocation_count += c.allocation_count; t.allocation_count += c.allocation_count;
t.span_count += c.span_count; t.span_count += c.span_count;
} }
t self.store.cold.set_totals(self.index, &t);
}) t
} }
pub fn total_allocations(&self) -> u64 { pub fn total_allocations(&self) -> u64 {
...@@ -236,7 +251,11 @@ impl<'a> SpanRef<'a> { ...@@ -236,7 +251,11 @@ impl<'a> SpanRef<'a> {
pub fn corrected_self_time(&self) -> Timestamp { pub fn corrected_self_time(&self) -> Timestamp {
let store = self.store; let store = self.store;
*self.time_data().corrected_self_time.get_or_init(|| { let slot = store.cold.time(self.index, TimeSlot::CorrectedSelfTime);
if let Some(cached) = load_time(slot) {
return cached;
}
let computed = {
let mut self_time = self let mut self_time = self
.span .span
.events .events
...@@ -257,20 +276,58 @@ impl<'a> SpanRef<'a> { ...@@ -257,20 +276,58 @@ impl<'a> SpanRef<'a> {
self_time = max(self_time, Timestamp::from_value(1)); self_time = max(self_time, Timestamp::from_value(1));
} }
self_time self_time
}) };
store_time(slot, computed);
computed
} }
pub fn corrected_total_time(&self) -> Timestamp { pub fn corrected_total_time(&self) -> Timestamp {
*self.time_data().corrected_total_time.get_or_init(|| { let slot = self
self.children_par() .store
.map(|child| child.corrected_total_time()) .cold
.sum::<Timestamp>() .time(self.index, TimeSlot::CorrectedTotalTime);
+ self.corrected_self_time() if let Some(cached) = load_time(slot) {
}) return cached;
}
let computed = self
.children_par()
.map(|child| child.corrected_total_time())
.sum::<Timestamp>()
+ self.corrected_self_time();
store_time(slot, computed);
computed
} }
/// Height of the subtree below this span; 0 for a leaf.
///
/// Cached in the hot record rather than in [`SpanTotals`], so the viewer's
/// "force `max_depth()` on every root span" warm-up walks one byte per span
/// instead of materializing the whole subtree-totals cache. Recursion is
/// bounded by `CUT_OFF_DEPTH`.
///
/// Two threads racing here compute the same value and store the same byte,
/// so a plain relaxed load/store is enough — no `OnceLock` needed.
pub fn max_depth(&self) -> u32 { pub fn max_depth(&self) -> u32 {
self.totals().max_depth let cached = self.span.max_depth.load(Ordering::Relaxed);
if cached != MAX_DEPTH_UNSET {
return cached as u32;
}
let computed = self
.children()
.map(|child| child.max_depth() + 1)
.max()
.unwrap_or(0);
debug_assert!(
computed < MAX_DEPTH_UNSET as u32,
"subtree height {computed} does not fit the max_depth byte; CUT_OFF_DEPTH should have \
bounded it"
);
// Clamp rather than wrap if that bound is ever raised past 254.
self.span.max_depth.store(
computed.min(MAX_DEPTH_UNSET as u32 - 1) as u8,
Ordering::Relaxed,
);
computed
} }
pub fn graph(&self) -> impl Iterator<Item = SpanGraphEventRef<'a>> + '_ { pub fn graph(&self) -> impl Iterator<Item = SpanGraphEventRef<'a>> + '_ {
......
...@@ -2,18 +2,22 @@ use std::{ ...@@ -2,18 +2,22 @@ use std::{
cmp::{max, min}, cmp::{max, min},
env, env,
num::NonZeroUsize, num::NonZeroUsize,
sync::{OnceLock, atomic::AtomicU64}, sync::{
OnceLock, RwLock,
atomic::{AtomicU8, AtomicU64, Ordering},
},
}; };
use rustc_hash::FxHashSet; use rustc_hash::{FxHashMap, FxHashSet};
use turbo_rcstr::{RcStr, rcstr}; use turbo_rcstr::{RcStr, rcstr};
use turbo_tasks_malloc::TurboMalloc; use turbo_tasks_malloc::TurboMalloc;
use crate::{ use crate::{
chunked_vec::ChunkedVec, chunked_vec::ChunkedVec,
cold::ColdStore,
memory_report::MemoryReport, memory_report::MemoryReport,
self_time_tree::SelfTimeTree, self_time_tree::SelfTimeTree,
span::{Span, SpanArgs, SpanEvent, SpanIndex, SpanTimeData}, span::{MAX_DEPTH_UNSET, Span, SpanArgs, SpanEvent, SpanExtra, SpanIndex, SpanTimeData},
span_ref::SpanRef, span_ref::SpanRef,
timestamp::Timestamp, timestamp::Timestamp,
}; };
...@@ -37,6 +41,16 @@ const MAX_MEMORY_SAMPLES: usize = 200; ...@@ -37,6 +41,16 @@ const MAX_MEMORY_SAMPLES: usize = 200;
pub struct Store { pub struct Store {
pub(crate) spans: ChunkedVec<Span>, pub(crate) spans: ChunkedVec<Span>,
/// Per-span graph / bottom-up / search-index caches, keyed by span index.
///
/// Held here rather than as a field on `Span` because only a handful of spans
/// ever get one — the roots, and whatever the user expands — while a field
/// costs 16 bytes on every span in the trace. Boxed so entries have a stable
/// address; see [`Store::extra`].
extra: RwLock<FxHashMap<usize, Box<SpanExtra>>>,
/// The derived half of every span, lazily allocated per chunk. See
/// [`crate::cold`].
pub(crate) cold: ColdStore,
pub(crate) self_time_tree: Option<SelfTimeTree<SpanIndex>>, pub(crate) self_time_tree: Option<SelfTimeTree<SpanIndex>>,
max_self_time_lookup_time: AtomicU64, max_self_time_lookup_time: AtomicU64,
/// Global sorted list of memory samples (timestamp, memory_bytes). /// Global sorted list of memory samples (timestamp, memory_bytes).
...@@ -53,16 +67,15 @@ fn new_root_span() -> Span { ...@@ -53,16 +67,15 @@ fn new_root_span() -> Span {
args: SpanArgs::new(), args: SpanArgs::new(),
events: Default::default(), events: Default::default(),
is_complete: true, is_complete: true,
max_depth: AtomicU8::new(MAX_DEPTH_UNSET),
self_allocations: 0, self_allocations: 0,
self_allocation_count: 0, self_allocation_count: 0,
self_deallocations: 0, self_deallocations: 0,
self_deallocation_count: 0, self_deallocation_count: 0,
totals: OnceLock::new(),
time_data: SpanTimeData { time_data: SpanTimeData {
self_end: Timestamp::MAX, self_end: Timestamp::MAX,
..Default::default() ..Default::default()
}, },
extra: OnceLock::new(),
names: OnceLock::new(), names: OnceLock::new(),
} }
} }
...@@ -79,9 +92,41 @@ impl Store { ...@@ -79,9 +92,41 @@ impl Store {
.then(SelfTimeTree::new), .then(SelfTimeTree::new),
max_self_time_lookup_time: AtomicU64::new(0), max_self_time_lookup_time: AtomicU64::new(0),
memory_samples: Vec::new(), memory_samples: Vec::new(),
extra: RwLock::new(FxHashMap::default()),
cold: ColdStore::new(),
} }
} }
/// The [`SpanExtra`] for a span, creating it on first touch.
///
/// Returns a reference tied to `&self` even though the entry is created
/// behind a lock. That is what lets `SpanRef::extra` keep its signature, so
/// `span_graph_ref` and `span_bottom_up_ref` need no changes.
pub(crate) fn extra(&self, index: usize) -> &SpanExtra {
// Fast path: already present. Under a read lock so the viewer's parallel
// warm-up does not serialize on a mutex.
if let Some(extra) = self.extra.read().unwrap().get(&index) {
let ptr: *const SpanExtra = &**extra;
// SAFETY: as below.
return unsafe { &*ptr };
}
let ptr: *const SpanExtra = {
let mut map = self.extra.write().unwrap();
&**map.entry(index).or_default()
};
// SAFETY: the pointee is a `Box` on the heap, so its address is stable
// for as long as it stays in the map — rehashing moves the `Box`, not
// what it points to. Entries are removed only by
// `invalidate_outdated_spans` and `reset`, both of which take `&mut self`
// and so cannot run while the `&self` borrow this reference is tied to is
// alive.
//
// The lock is released before returning, and deliberately before any
// caller runs a `get_or_init` closure on the result: those closures
// recurse into other spans' `extra`, which would deadlock otherwise.
unsafe { &*ptr }
}
pub fn reset(&mut self) { pub fn reset(&mut self) {
self.spans = ChunkedVec::new(); self.spans = ChunkedVec::new();
self.spans.push(new_root_span()); self.spans.push(new_root_span());
...@@ -90,6 +135,8 @@ impl Store { ...@@ -90,6 +135,8 @@ impl Store {
} }
*self.max_self_time_lookup_time.get_mut() = 0; *self.max_self_time_lookup_time.get_mut() = 0;
self.memory_samples.clear(); self.memory_samples.clear();
self.extra.write().unwrap().clear();
self.cold.clear();
} }
/// Walk the store and add up where its bytes actually went. See /// Walk the store and add up where its bytes actually went. See
...@@ -110,13 +157,14 @@ impl Store { ...@@ -110,13 +157,14 @@ impl Store {
memory_sample_bytes: self.memory_samples.capacity() memory_sample_bytes: self.memory_samples.capacity()
* std::mem::size_of::<MemorySample>(), * std::mem::size_of::<MemorySample>(),
allocator_live_bytes: TurboMalloc::memory_usage(), allocator_live_bytes: TurboMalloc::memory_usage(),
extra_populated: self.extra.read().unwrap().len(),
cold_bytes: self.cold.allocated_bytes(),
cold_chunks: self.cold.materialized_chunks(),
..Default::default() ..Default::default()
}; };
for span in self.spans.iter_mut() { for span in self.spans.iter_mut() {
report.totals_populated += span.totals.get().is_some() as usize;
report.names_populated += span.names.get().is_some() as usize; report.names_populated += span.names.get().is_some() as usize;
report.extra_populated += span.extra.get().is_some() as usize;
report.args += span.args.len(); report.args += span.args.len();
if span.args.spilled() { if span.args.spilled() {
...@@ -158,6 +206,7 @@ impl Store { ...@@ -158,6 +206,7 @@ impl Store {
outdated_spans: &mut FxHashSet<SpanIndex>, outdated_spans: &mut FxHashSet<SpanIndex>,
) -> SpanIndex { ) -> SpanIndex {
let id = SpanIndex::new(self.spans.len()).unwrap(); let id = SpanIndex::new(self.spans.len()).unwrap();
self.cold.reserve_for(id.get());
let ignore_self_time = &name == "thread" || &name == "blocking"; let ignore_self_time = &name == "thread" || &name == "blocking";
self.spans.push(Span { self.spans.push(Span {
parent, parent,
...@@ -168,17 +217,16 @@ impl Store { ...@@ -168,17 +217,16 @@ impl Store {
args, args,
events: Default::default(), events: Default::default(),
is_complete: false, is_complete: false,
max_depth: AtomicU8::new(MAX_DEPTH_UNSET),
self_allocations: 0, self_allocations: 0,
self_allocation_count: 0, self_allocation_count: 0,
self_deallocations: 0, self_deallocations: 0,
self_deallocation_count: 0, self_deallocation_count: 0,
totals: OnceLock::new(),
time_data: SpanTimeData { time_data: SpanTimeData {
self_end: start, self_end: start,
ignore_self_time, ignore_self_time,
..Default::default() ..Default::default()
}, },
extra: OnceLock::new(),
names: OnceLock::new(), names: OnceLock::new(),
}); });
let mut parent = if let Some(parent) = parent { let mut parent = if let Some(parent) = parent {
...@@ -475,34 +523,54 @@ impl Store { ...@@ -475,34 +523,54 @@ impl Store {
} }
pub fn invalidate_outdated_spans(&mut self, outdated_spans: &FxHashSet<SpanId>) { pub fn invalidate_outdated_spans(&mut self, outdated_spans: &FxHashSet<SpanId>) {
fn invalidate_span(span: &mut Span) { // Events hold no cache of their own, and the derived timestamps and
span.time_data.end.take(); // totals are no longer fields, so per-span invalidation is now: reset the
span.time_data.total_time.take(); // `max_depth` byte, drop the side-table entry, and zero the cold slots.
span.time_data.corrected_self_time.take(); // During a headless ingest the cold chunks were never allocated, so that
span.time_data.corrected_total_time.take(); // last step short-circuits and this whole pass — which runs over every
// Events hold no cache of their own, so there is nothing to clear // ancestor of every touched span after each read batch — does almost
// here. That matters for more than tidiness: this function runs on // nothing.
// every ancestor of every touched span after each read batch, and it let spans = &mut self.spans;
// used to walk each of their event lists. let extra = self.extra.get_mut().unwrap();
span.totals.take(); let cold = &mut self.cold;
span.extra.take();
} // The walk carries the index rather than just the reference, because both
// side structures are keyed by it.
for id in outdated_spans.iter() { for id in outdated_spans.iter() {
let mut span = &mut self.spans[id.get()]; let mut index = id.get();
loop { loop {
invalidate_span(span); extra.remove(&index);
cold.invalidate(index);
let span = &mut spans[index];
span.max_depth.store(MAX_DEPTH_UNSET, Ordering::Relaxed);
let Some(parent) = span.parent else { let Some(parent) = span.parent else {
break; break;
}; };
if outdated_spans.contains(&parent) { if outdated_spans.contains(&parent) {
break; break;
} }
span = &mut self.spans[parent.get()]; index = parent.get();
} }
} }
invalidate_span(&mut self.spans[0]); extra.remove(&0);
cold.invalidate(0);
spans[0].max_depth.store(MAX_DEPTH_UNSET, Ordering::Relaxed);
}
/// Force every derived value, as a client rendering the whole trace would.
///
/// Diagnostic only, for `MEMORY_REPORT=warm`. Headless ingest leaves the
/// derived half entirely unallocated, so without this the report only ever
/// shows the ingest peak — but the server also has to *serve*, and that is
/// the number that decides whether it fits on a small machine.
pub fn warm_all_derived(&self) {
let root = self.root_span();
root.max_depth();
root.end();
root.total_time();
root.corrected_total_time();
root.total_span_count();
} }
pub fn root_spans(&self) -> impl Iterator<Item = SpanRef<'_>> { pub fn root_spans(&self) -> impl Iterator<Item = SpanRef<'_>> {
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment