Unverified Commit 29045997 authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

Optimize how we track data for persistence (#89370)

## Summary

Reworks how the turbo-tasks backend tracks modified tasks for persistence snapshots, reducing overhead and simplifying the snapshot lifecycle.

**Key changes:**

- **Replace `modified` DashMap with per-shard atomic counters + inline flags.** Instead of maintaining a separate `FxDashMap<TaskId, ModifiedState>` that mirrors every modification, track modifications via flags already on `TaskStorage` and use per-shard `AtomicU64` counters to skip unmodified shards during snapshot iteration. This eliminates a major source of  memory overhead.
    - The downside here is needing to scan the map.  per shard counters enable early exits but we will still need to scan entire shards.  For a large site that means scanning thousands of

- **Merge task cache writes into the snapshot pipeline.** New tasks now carry a `new_task` flag and their type hash in `SnapshotItem`, so task cache entries are written in the same batch as task data/meta — removing the separate `persisted_task_cache_log` (`Sharded<ChunkedVec<...>>`) and its associated locking.

- **Remove `local_is_partial` optimization.** The backing storage layer already short-circuits on empty databases, and new tasks eagerly set `restored` flags at allocation time, making this redundant.

- **Simplify `end_snapshot`.** Instead of a multi-pass retain/iterate/update cycle over the `modified` map, `end_snapshot` now just drains the small `snapshots` map (only tasks concurrently accessed during snapshot mode) and promotes their `modified_during_snapshot` flags.

- **Delete unused utilities.** Removes `Sharded`, `ChunkedVec` (from backing_storage), and `swap_retain` import now that they're no longer needed.

**Other cleanups:**
- `initialize_new_task` sets restored + new_task flags at allocation time for both persistent and transient tasks
- Fuzz test updated to use `active_tracking: true` and `StorageMode::ReadWrite`
- New KV storage test for batch write+flush+reopen pattern
- Minor fix: `SmallVec::into_boxed_slice()` instead of `into_vec().into_boxed_slice()`


## Build Benchmark Results

  Measured over 9 runs (1 warm-up discarded), macOS, `TURBOPACK_PERSISTENT_CACHE=1`.

  ### Cold build (`rm -rf .next/`)

  | | Time (avg) | Time (stddev) | MaxRSS (avg) |
  |---|---|---|---|
  | HEAD | 75.48s | 0.72s | 24,231 MiB |
  | This PR | 75.20s | 1.98s | 23,829 MiB |
  | **Delta** | −0.28s | — | **−402 MiB (−1.7%)** |

  ### Warm build (single file edit)

  | | Time (avg) | MaxRSS (avg) |
  |---|---|---|
  | HEAD | 23.79s | 8,764 MiB |
  | This PR | 23.67s | 8,766 MiB |
  | **Delta** | −0.12s | flat |

  Cold build time difference is within noise (< 1 stddev). The meaningful improvement is a **~400 MiB reduction in peak memory on cold builds**, consistent with the fixed overhead this PR targets. Warm builds are unaffected as expected.
parent 18430c7f
......@@ -17,14 +17,10 @@ const nextConfig = {
},
},
},
experimental: enableCaching
? {
turbopackFileSystemCacheForBuild: true,
}
: {
turbopackFileSystemCacheForDev: false,
turbopackFileSystemCacheForBuild: false,
},
experimental: {
turbopackFileSystemCacheForBuild: enableCaching,
turbopackFileSystemCacheForDev: enableCaching,
},
env: {
NEXT_PUBLIC_CONFIG_ENV: 'hello world',
},
......
......@@ -23,11 +23,9 @@ const runDev = async (config: any, shouldWaitForReady = true) => {
const port = await findPort()
const obj = { port, stdout: '', stderr: '' }
app = await launchApp(appDir, port, {
stdout: false,
onStdout(msg: string) {
obj.stdout += msg || ''
},
stderr: false,
onStderr(msg: string) {
obj.stderr += msg || ''
},
......
......@@ -13,7 +13,7 @@ impl ValueBuffer<'_> {
match self {
ValueBuffer::Borrowed(b) => b.into(),
ValueBuffer::Vec(v) => v.into_boxed_slice(),
ValueBuffer::SmallVec(sv) => sv.into_vec().into_boxed_slice(),
ValueBuffer::SmallVec(sv) => sv.into_boxed_slice(),
}
}
}
......
......@@ -16,7 +16,7 @@ libfuzzer-sys = "0.4.9"
once_cell = { workspace = true }
serde = { workspace = true }
tokio = { workspace = true, features = ["full"] }
turbo-tasks = { workspace = true }
turbo-tasks = { workspace = true, features = ["non_operation_vc_strongly_consistent"] }
turbo-tasks-backend = { workspace = true }
turbo-tasks-malloc = { workspace = true }
......
......@@ -133,16 +133,17 @@ pub fn run(data: Vec<TaskSpec>) {
struct Iteration(State<usize>);
fn actual_operation(spec: Arc<Vec<TaskSpec>>, iterations: usize) {
let tt = TurboTasks::new(turbo_tasks_backend::TurboTasksBackend::new(
turbo_tasks_backend::BackendOptions {
storage_mode: None,
small_preallocation: true,
..Default::default()
},
turbo_tasks_backend::noop_backing_storage(),
));
RUNTIME
.block_on(async {
let tt = TurboTasks::new(turbo_tasks_backend::TurboTasksBackend::new(
turbo_tasks_backend::BackendOptions {
storage_mode: Some(turbo_tasks_backend::StorageMode::ReadWrite),
small_preallocation: false,
active_tracking: true,
..Default::default()
},
turbo_tasks_backend::noop_backing_storage(),
));
for i in 0..iterations {
let spec = spec.clone();
tt.run(async move {
......
......@@ -63,7 +63,7 @@ use crate::{
storage::Storage,
storage_schema::{TaskStorage, TaskStorageAccessors},
},
backing_storage::{BackingStorage, SnapshotItem},
backing_storage::{BackingStorage, SnapshotItem, compute_task_type_hash},
data::{
ActivenessState, CellRef, CollectibleRef, CollectiblesRef, Dirtyness, InProgressCellState,
InProgressState, InProgressStateInner, OutputValue, TransientTask,
......@@ -71,13 +71,10 @@ use crate::{
error::TaskError,
utils::{
arc_or_owned::ArcOrOwned,
chunked_vec::ChunkedVec,
dash_map_drop_contents::drop_contents,
dash_map_raw_entry::{RawEntry, raw_entry},
ptr_eq_arc::PtrEqArc,
shard_amount::compute_shard_amount,
sharded::Sharded,
swap_retain,
},
};
......@@ -167,8 +164,6 @@ pub enum TurboTasksBackendJob {
pub struct TurboTasksBackend<B: BackingStorage>(Arc<TurboTasksBackendInner<B>>);
type TaskCacheLog = Sharded<ChunkedVec<(Arc<CachedTaskType>, TaskId)>>;
struct TurboTasksBackendInner<B: BackingStorage> {
options: BackendOptions,
......@@ -177,15 +172,10 @@ struct TurboTasksBackendInner<B: BackingStorage> {
persisted_task_id_factory: IdFactoryWithReuse<TaskId>,
transient_task_id_factory: IdFactoryWithReuse<TaskId>,
persisted_task_cache_log: Option<TaskCacheLog>,
task_cache: FxDashMap<Arc<CachedTaskType>, TaskId>,
storage: Storage,
/// When true, the backing_storage has data that is not in the local storage.
/// This is determined once at startup and never changes.
local_is_partial: bool,
/// Number of executing operations + Highest bit is set when snapshot is
/// requested. When that bit is set, operations should pause until the
/// snapshot is completed. When the bit is set and in progress counter
......@@ -235,10 +225,6 @@ impl<B: BackingStorage> TurboTasksBackend<B> {
impl<B: BackingStorage> TurboTasksBackendInner<B> {
pub fn new(mut options: BackendOptions, backing_storage: B) -> Self {
let shard_amount = compute_shard_amount(options.num_workers, options.small_preallocation);
let need_log = matches!(
options.storage_mode,
Some(StorageMode::ReadWrite) | Some(StorageMode::ReadWriteOnShutdown)
);
if !options.dependency_tracking {
options.active_tracking = false;
}
......@@ -257,9 +243,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
TaskId::try_from(TRANSIENT_TASK_BIT).unwrap(),
TaskId::MAX,
),
persisted_task_cache_log: need_log.then(|| Sharded::new(shard_amount)),
task_cache: FxDashMap::default(),
local_is_partial: next_task_id != TaskId::MIN,
storage: Storage::new(shard_amount, small_preallocation),
in_progress_operations: AtomicUsize::new(0),
snapshot_request: Mutex::new(SnapshotRequest::new()),
......@@ -1036,12 +1020,9 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
.map(|op| op.arc().clone())
.collect::<Vec<_>>();
}
self.storage.start_snapshot();
let mut persisted_task_cache_log = self
.persisted_task_cache_log
.as_ref()
.map(|l| l.take(|i| i))
.unwrap_or_default();
// Enter snapshot mode, which atomically reads and resets the modified count.
// Checking after start_snapshot ensures no concurrent increments can race.
let (snapshot_guard, has_modifications) = self.storage.start_snapshot();
let mut snapshot_request = self.snapshot_request.lock();
snapshot_request.snapshot_requested = false;
self.in_progress_operations
......@@ -1050,6 +1031,13 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
let snapshot_time = Instant::now();
drop(snapshot_request);
if !has_modifications {
// No tasks modified since the last snapshot — drop the guard (which
// calls end_snapshot) and skip the expensive O(N) scan.
drop(snapshot_guard);
return Some((start, false));
}
#[cfg(feature = "print_cache_item_size")]
#[derive(Default)]
struct TaskCacheStats {
......@@ -1278,35 +1266,49 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
} else {
None
};
let task_type_hash = if inner.flags.new_task() {
let task_type = inner.get_persistent_task_type().expect(
"It is not possible for a new_task to not have a persistent_task_type. Task \
creation for persistent tasks uses a single ExecutionContextImpl for \
creating the task (which sets new_task) and connect_child (which sets \
persistent_task_type) and take_snapshot waits for all operations to complete \
or suspend before we start snapshotting. So task creation will always set \
the task_type.",
);
Some(compute_task_type_hash(task_type))
} else {
None
};
SnapshotItem {
task_id,
meta,
data,
task_type_hash,
}
};
// take_snapshot already filters empty items and empty shards in parallel
let task_snapshots = self.storage.take_snapshot(&process);
swap_retain(&mut persisted_task_cache_log, |shard| !shard.is_empty());
let task_snapshots = self.storage.take_snapshot(snapshot_guard, &process);
drop(snapshot_span);
let snapshot_duration = start.elapsed();
let task_count = task_snapshots.len();
if persisted_task_cache_log.is_empty() && task_snapshots.is_empty() {
if task_snapshots.is_empty() {
// This should be impossible — if we got here, modified_count was nonzero, and every
// modification that increments the count also failed during encoding.
std::hint::cold_path();
return Some((snapshot_time, false));
}
let persist_start = Instant::now();
let _span = tracing::info_span!(parent: parent_span, "persist", reason = reason).entered();
{
if let Err(err) = self.backing_storage.save_snapshot(
suspended_operations,
persisted_task_cache_log,
task_snapshots,
) {
if let Err(err) = self
.backing_storage
.save_snapshot(suspended_operations, task_snapshots)
{
eprintln!("Persisting failed: {err:?}");
return None;
}
......@@ -1551,7 +1553,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
let (task_id, task_type) = match raw_entry(&self.task_cache, &task_type) {
RawEntry::Occupied(e) => {
// Another thread beat us to creating this task - use their task_id.
// They will handle logging to persisted_task_cache_log.
// They will handle logging the new task as modified
let task_id = *e.get();
drop(e);
self.track_cache_hit(&task_type);
......@@ -1562,12 +1564,12 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
let task_type = Arc::new(task_type);
let task_id = self.persisted_task_id_factory.get();
e.insert(task_type.clone(), task_id);
// Mark the task as new in storage.
// Do this after e.insert so we aren't holding the task_cache lock
self.storage.initialize_new_task(task_id);
// insert() consumes e, releasing the lock
self.track_cache_miss(&task_type);
is_new = true;
if let Some(log) = &self.persisted_task_cache_log {
log.lock(task_id).push((task_type.clone(), task_id));
}
(task_id, ArcOrOwned::Arc(task_type))
}
};
......@@ -1637,6 +1639,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
let task_type = Arc::new(task_type);
let task_id = self.transient_task_id_factory.get();
e.insert(task_type.clone(), task_id);
self.storage.initialize_new_task(task_id);
self.track_cache_miss(&task_type);
if is_root {
......
......@@ -1896,7 +1896,7 @@ impl AggregationUpdateQueue {
/// See detailed comments in that function, follow the STEP numbers.
fn inner_of_uppers_has_new_follower<T: TaskIdWithOptionalCount + Clone, const N: usize>(
&mut self,
ctx: &mut impl ExecuteContext,
ctx: &mut impl ExecuteContext<'_>,
new_follower_id: TaskId,
mut upper_ids: SmallVec<[T; N]>,
) {
......@@ -2618,7 +2618,7 @@ impl AggregationUpdateQueue {
/// Only used when activeness is tracked.
fn increase_active_count(
&mut self,
ctx: &mut impl ExecuteContext,
ctx: &mut impl ExecuteContext<'_>,
task_id: TaskId,
task_type: Option<Arc<CachedTaskType>>,
) {
......@@ -2637,7 +2637,7 @@ impl AggregationUpdateQueue {
if let Some(task_type) = task_type
&& !task.has_persistent_task_type()
{
let _ = task.set_persistent_task_type(task_type);
task.set_persistent_task_type(task_type);
}
let state = task.get_activeness_mut_or_insert_with(|| ActivenessState::new(task_id));
let is_new = state.is_empty();
......
......@@ -165,20 +165,15 @@ impl<'e, B: BackingStorage> ExecuteContextImpl<'e, B> {
}
}
fn should_check_backing_storage(&self) -> bool {
self.backend.should_restore() && self.backend.local_is_partial
}
fn restore_task_data(
&self,
task_id: TaskId,
category: SpecificTaskDataCategory,
) -> TaskStorage {
if !self.should_check_backing_storage() {
// If we don't need to restore, we can just return an empty storage
return TaskStorage::default();
}
let mut storage = TaskStorage::default();
if !self.backend.should_restore() {
return storage;
}
let result = self
.backend
.backing_storage
......@@ -204,8 +199,7 @@ impl<'e, B: BackingStorage> ExecuteContextImpl<'e, B> {
task_ids.len() > 1,
"Use restore_task_data_typed for single task"
);
if !self.should_check_backing_storage() {
// If we don't need to restore, we return None
if !self.backend.should_restore() {
return None;
}
let result = self
......@@ -244,13 +238,10 @@ impl<'e, B: BackingStorage> ExecuteContextImpl<'e, B> {
.into_iter()
.filter(|&(id, category)| {
if id.is_transient() {
// Transient tasks don't need DB restoration (restored flags are
// set at allocation time), so just invoke the callback directly.
if call_prepared_task_callback_for_transient_tasks {
let mut task = self.backend.storage.access_mut(id);
// TODO add is_restoring and avoid concurrent restores and duplicates tasks
// ids in `task_ids`
if !task.flags.is_restored(category) {
task.flags.set_restored(TaskDataCategory::All);
}
let task = self.backend.storage.access_mut(id);
prepared_task_callback(self, id, category, task);
}
false
......@@ -400,40 +391,43 @@ impl<'e, B: BackingStorage> ExecuteContext<'e> for ExecuteContextImpl<'e, B> {
let mut task = self.backend.storage.access_mut(task_id);
if !task.flags.is_restored(category) {
if task_id.is_transient() {
task.flags.set_restored(TaskDataCategory::All);
} else {
// Collect which categories need restoring while we have the lock
let needs_data =
category.includes_data() && !task.flags.is_restored(TaskDataCategory::Data);
let needs_meta =
category.includes_meta() && !task.flags.is_restored(TaskDataCategory::Meta);
if needs_data || needs_meta {
// Avoid holding the lock too long since this can also affect other tasks
// Drop lock once, do all I/O, then re-acquire once
drop(task);
let storage_data = needs_data
.then(|| self.restore_task_data(task_id, SpecificTaskDataCategory::Data));
let storage_meta = needs_meta
.then(|| self.restore_task_data(task_id, SpecificTaskDataCategory::Meta));
task = self.backend.storage.access_mut(task_id);
// Handle race conditions and merge
if let Some(storage) = storage_data
&& !task.flags.is_restored(TaskDataCategory::Data)
{
task.restore_from(storage, TaskDataCategory::Data);
task.flags.set_restored(TaskDataCategory::Data);
}
if let Some(storage) = storage_meta
&& !task.flags.is_restored(TaskDataCategory::Meta)
{
task.restore_from(storage, TaskDataCategory::Meta);
task.flags.set_restored(TaskDataCategory::Meta);
}
// New tasks (transient and persistent) have restored flags set at allocation
// time, so this path is only hit for persistent tasks being restored from DB.
debug_assert!(
!task.flags.new_task() && !task_id.is_transient(),
"new or transient task should already be marked restored"
);
// Collect which categories need restoring while we have the lock
let needs_data =
category.includes_data() && !task.flags.is_restored(TaskDataCategory::Data);
let needs_meta =
category.includes_meta() && !task.flags.is_restored(TaskDataCategory::Meta);
if needs_data || needs_meta {
// Avoid holding the lock too long since this can also affect other tasks
// Drop lock once, do all I/O, then re-acquire once
drop(task);
let storage_data = needs_data
.then(|| self.restore_task_data(task_id, SpecificTaskDataCategory::Data));
let storage_meta = needs_meta
.then(|| self.restore_task_data(task_id, SpecificTaskDataCategory::Meta));
task = self.backend.storage.access_mut(task_id);
// Handle race conditions and merge
if let Some(storage) = storage_data
&& !task.flags.is_restored(TaskDataCategory::Data)
{
task.restore_from(storage, TaskDataCategory::Data);
task.flags.set_restored(TaskDataCategory::Data);
}
if let Some(storage) = storage_meta
&& !task.flags.is_restored(TaskDataCategory::Meta)
{
task.restore_from(storage, TaskDataCategory::Meta);
task.flags.set_restored(TaskDataCategory::Meta);
}
}
}
......@@ -600,7 +594,7 @@ impl<'e, B: BackingStorage> ExecuteContext<'e> for ExecuteContextImpl<'e, B> {
}
fn task_by_type(&mut self, task_type: &CachedTaskType) -> Option<TaskId> {
if !self.should_check_backing_storage() {
if !self.backend.should_restore() {
return None;
}
......
......@@ -187,6 +187,11 @@ struct TaskStorageSchema {
#[field(storage = "flag", category = "transient")]
stateful: bool,
/// Whether this task is new and needs its type persisted to the task cache.
/// Set when task is created, cleared after persisting.
#[field(storage = "flag", category = "transient")]
pub new_task: bool,
// =========================================================================
// CHILDREN & AGGREGATION (meta)
// =========================================================================
......@@ -548,7 +553,8 @@ impl TaskStorage {
task_type: TransientTaskType,
should_track_activeness: bool,
) {
// Mark as fully restored since transient tasks don't need restoration from disk
// Mark as fully restored since transient tasks don't need restoration from disk,
// and as new since this task was just created.
self.flags.set_restored(TaskDataCategory::All);
// This is a root (or once) task. These tasks use the max aggregation number.
......
......@@ -5,24 +5,50 @@ use either::Either;
use smallvec::SmallVec;
use turbo_bincode::TurboBincodeBuffer;
use turbo_tasks::{TaskId, backend::CachedTaskType};
use turbo_tasks_hash::Xxh3Hash64Hasher;
use crate::{
backend::{AnyOperation, SpecificTaskDataCategory, storage_schema::TaskStorage},
utils::chunked_vec::ChunkedVec,
};
use crate::backend::{AnyOperation, SpecificTaskDataCategory, storage_schema::TaskStorage};
pub type TaskTypeHash = [u8; 8];
/// A single item yielded by the snapshot iterator during persistence.
pub struct SnapshotItem {
pub task_id: TaskId,
pub data: Option<TurboBincodeBuffer>,
/// Serialized task meta data, if modified
pub meta: Option<TurboBincodeBuffer>,
/// Serialized task data, if modified
pub data: Option<TurboBincodeBuffer>,
/// Task type for new tasks that need to be added to the task cache
pub task_type_hash: Option<TaskTypeHash>,
}
impl SnapshotItem {
pub fn is_empty(&self) -> bool {
self.meta.is_none() && self.data.is_none()
self.meta.is_none() && self.data.is_none() && self.task_type_hash.is_none()
}
}
/// Computes a deterministic 64-bit hash of a CachedTaskType for use as a TaskCache key.
///
/// This encodes the task type directly to a hasher, avoiding intermediate buffer allocation.
/// The encoding is deterministic (function IDs from registry, bincode argument encoding).
pub fn compute_task_type_hash(task_type: &CachedTaskType) -> TaskTypeHash {
let mut hasher = Xxh3Hash64Hasher::new();
task_type.hash_encode(&mut hasher);
let hash = hasher.finish();
if cfg!(feature = "verify_serialization") {
hasher = Xxh3Hash64Hasher::new();
task_type.hash_encode(&mut hasher);
let hash2 = hasher.finish();
assert_eq!(
hash, hash2,
"Hashing TaskType twice was non-deterministic: \n{:?}\ngot hashes {} != {}",
task_type, hash, hash2
);
}
hash.to_le_bytes()
}
/// Represents types accepted by [`TurboTasksBackend::new`]. Typically this is the value returned by
/// [`turbo_backing_storage`] or [`noop_backing_storage`].
///
......@@ -55,12 +81,7 @@ pub trait BackingStorageSealed: 'static + Send + Sync {
fn next_free_task_id(&self) -> Result<TaskId>;
fn uncompleted_operations(&self) -> Result<Vec<AnyOperation>>;
fn save_snapshot<I>(
&self,
operations: Vec<Arc<AnyOperation>>,
task_cache_updates: Vec<ChunkedVec<(Arc<CachedTaskType>, TaskId)>>,
snapshots: Vec<I>,
) -> Result<()>
fn save_snapshot<I>(&self, operations: Vec<Arc<AnyOperation>>, snapshots: Vec<I>) -> Result<()>
where
I: IntoIterator<Item = SnapshotItem> + Send + Sync;
/// Returns all task IDs that match the given task type (hash collision candidates).
......@@ -118,18 +139,12 @@ where
either::for_both!(self, this => this.uncompleted_operations())
}
fn save_snapshot<I>(
&self,
operations: Vec<Arc<AnyOperation>>,
task_cache_updates: Vec<ChunkedVec<(Arc<CachedTaskType>, TaskId)>>,
snapshots: Vec<I>,
) -> Result<()>
fn save_snapshot<I>(&self, operations: Vec<Arc<AnyOperation>>, snapshots: Vec<I>) -> Result<()>
where
I: IntoIterator<Item = SnapshotItem> + Send + Sync,
{
either::for_both!(self, this => this.save_snapshot(
operations,
task_cache_updates,
snapshots,
))
}
......
pub struct ChunkedVec<T> {
chunks: Vec<Vec<T>>,
}
impl<T> Default for ChunkedVec<T> {
fn default() -> Self {
Self::new()
}
}
impl<T> ChunkedVec<T> {
pub fn new() -> Self {
Self { chunks: Vec::new() }
}
pub fn len(&self) -> usize {
for (i, chunk) in self.chunks.iter().enumerate().rev() {
if !chunk.is_empty() {
let free = chunk.capacity() - chunk.len();
return cumulative_chunk_size(i) - free;
}
}
0
}
pub fn push(&mut self, item: T) {
if let Some(chunk) = self.chunks.last_mut()
&& chunk.len() < chunk.capacity()
{
chunk.push(item);
return;
}
let mut chunk = Vec::with_capacity(chunk_size(self.chunks.len()));
chunk.push(item);
self.chunks.push(chunk);
}
pub fn extend<I: IntoIterator<Item = T>>(&mut self, iter: I) {
for item in iter {
self.push(item);
}
}
pub fn iter(&self) -> impl ExactSizeIterator<Item = &T> {
ExactSizeIter {
iter: self.chunks.iter().flat_map(|chunk| chunk.iter()),
len: self.len(),
}
}
pub fn is_empty(&self) -> bool {
self.chunks.first().is_none_or(|chunk| chunk.is_empty())
}
}
impl<T> IntoIterator for ChunkedVec<T> {
type Item = T;
type IntoIter = ExactSizeIter<std::iter::Flatten<std::vec::IntoIter<Vec<T>>>>;
fn into_iter(self) -> Self::IntoIter {
let len = self.len();
ExactSizeIter {
iter: self.chunks.into_iter().flatten(),
len,
}
}
}
impl<T> Extend<T> for ChunkedVec<T> {
fn extend<I: IntoIterator<Item = T>>(&mut self, iter: I) {
for item in iter {
self.push(item);
}
}
}
fn chunk_size(chunk_index: usize) -> usize {
8 << chunk_index
}
fn cumulative_chunk_size(chunk_index: usize) -> usize {
(8 << (chunk_index + 1)) - 8
}
pub struct ExactSizeIter<I: Iterator> {
iter: I,
len: usize,
}
impl<I: Iterator> Iterator for ExactSizeIter<I> {
type Item = I::Item;
fn next(&mut self) -> Option<Self::Item> {
self.iter.next().inspect(|_| self.len -= 1)
}
fn size_hint(&self) -> (usize, Option<usize>) {
(self.len, Some(self.len))
}
}
impl<I: Iterator> ExactSizeIterator for ExactSizeIter<I> {
fn len(&self) -> usize {
self.len
}
}
#[cfg(test)]
mod tests {
use super::ChunkedVec;
#[test]
fn test_chunked_vec() {
for i in 0..1000 {
let mut vec = ChunkedVec::new();
for j in 0..i {
vec.push(j);
}
assert_eq!(vec.len(), i);
assert_eq!(
vec.iter().copied().collect::<Vec<_>>(),
(0..i).collect::<Vec<_>>()
);
assert_eq!(vec.iter().len(), i);
assert_eq!(vec.is_empty(), i == 0);
let iter = vec.into_iter();
assert_eq!(iter.len(), i);
assert_eq!(iter.collect::<Vec<_>>(), (0..i).collect::<Vec<_>>());
}
}
}
pub mod arc_or_owned;
pub mod chunked_vec;
pub mod dash_map_drop_contents;
pub mod dash_map_multi;
pub mod dash_map_raw_entry;
pub mod markdown_table;
pub mod ptr_eq_arc;
pub mod shard_amount;
pub mod sharded;
pub mod swap_retain;
pub use swap_retain::swap_retain;
......@@ -11,7 +11,9 @@ use std::thread::available_parallelism;
/// The returned number is always a power of two as this is often required
/// by sharded data structures. The maximum shard amount is capped at 1 << 16 (65536).
pub fn compute_shard_amount(num_workers: Option<usize>, small_preallocation: bool) -> usize {
let num_workers = num_workers.unwrap_or_else(|| available_parallelism().map_or(4, |v| v.get()));
let num_workers = num_workers
.unwrap_or_else(|| available_parallelism().map_or(4, |v| v.get()))
.max(4);
// Once can compute the probability of a shard collision (which leads to false sharing) using
// the birthday paradox formula. It's notable that the probability of collisions increases
......
use std::hash::{BuildHasher, BuildHasherDefault, Hash};
use parking_lot::{Mutex, MutexGuard};
use rustc_hash::FxHasher;
pub struct Sharded<T, H = BuildHasherDefault<FxHasher>> {
data: Box<[Mutex<T>]>,
hasher: H,
bitmask: u16,
}
impl<T, H> Sharded<T, H> {
pub fn new(mut shard_amount: usize) -> Self
where
T: Default,
H: Default,
{
assert!(shard_amount.is_power_of_two());
shard_amount = shard_amount.min(u16::MAX as usize + 1);
let data = (0..shard_amount)
.map(|_| Mutex::new(T::default()))
.collect::<Vec<_>>()
.into_boxed_slice();
Self {
data,
hasher: H::default(),
bitmask: (shard_amount - 1) as u16,
}
}
pub fn lock<K>(&self, key: K) -> MutexGuard<'_, T>
where
K: Hash,
H: BuildHasher,
{
let hash = self.hasher.hash_one(key);
let shard = hash as u16 & self.bitmask;
self.data[shard as usize].lock()
}
pub fn take<R>(&self, map: impl Fn(T) -> R) -> Vec<R>
where
T: Default,
{
let locked = self.data.iter().map(|m| m.lock()).collect::<Vec<_>>();
locked
.into_iter()
.map(|mut m| map(std::mem::take(&mut *m)))
.collect()
}
}
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment