Unverified Commit 4bb4e8f1 authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

turbo-bitset: make block storage pluggable (#100779)

### What?

`turbo_bitset::Bitset` can now use different storage for its blocks. A
new public `BlockStorage` trait defines what that storage must support.

- `Bitset` is now `Bitset<S: BlockStorage = Vec<usize>>`, so every
existing user keeps `Vec<usize>` storage.
- The trait is implemented for `Vec<usize>`, `SmallVec<[usize; N]>` and
`ArrayVec<usize, N>`. `turbo-bitset` now depends on `smallvec` and
`arrayvec` unconditionally.
- All `Bitset` methods and operators, plus `Encode`, `Decode` and
`ShrinkToFit`, work with any storage. Set operations require both sides
to use the same storage. The `turbo-tasks` integrations remain on the
default `Bitset`.
- The encoding doesn't depend on the storage, so bytes written with one
storage decode into any other.
- With `ArrayVec`, operations that would need more than its capacity of
`N` blocks panic. Decoding a set that doesn't fit returns a
`DecodeError`.
- `shrink_to_fit` only touches the storage when that frees capacity. An
`ArrayVec` or an inline `SmallVec` is left unchanged.

### Why?

This lets small sets use inline storage where that helps, without making
every `Bitset` user change its storage type.

### How?

- **Bounded storage:** when an operation would exceed an `ArrayVec`'s
capacity, empty blocks at either end of the storage are dropped first.
This fixes a case where an insert at the back panicked although the bits
fit, because earlier growth at the front had filled the capacity with
zero blocks.
- **`intersection`:** builds its result with `FromIterator`, so a `Vec`
keeps its exact-size allocation.
- **Inlining:** only the small methods of the non-generic `Vec` storage
are marked `#[inline]`. The generic `SmallVec` and `ArrayVec`
implementations and `Iter::next` have no annotation; removing those had
no effect on the benchmark binary (see below).
- **Tests:** the shared semantic tests run for all three storages. New
tests cover `ArrayVec` overflow and decode errors, `SmallVec`
inline/spill behaviour, `shrink_to_fit`, zero-capacity storage, and
encoding equality across storages. The benchmarks gain cases for
`SmallVec` and `ArrayVec`.

### Verification

- `cargo fmt --check -p turbo-bitset -p turbo-tasks -p turbopack-core`
- `cargo clippy -q -p turbo-bitset -p turbo-tasks -p turbopack-core -p
next-api --all-targets -- -D warnings`
- `RUSTDOCFLAGS="-D warnings" cargo doc -q --no-deps -p turbo-bitset -p
turbopack-core`
- `cargo check -q -p next-api`
- `cargo test -q -p turbo-bitset`
- `cargo test -q -p turbo-tasks --lib`
- `cargo test -q -p turbopack-core --lib`
- `cargo test -q -p turbopack-tests --test snapshot` (no snapshot
changes)
- `cargo bench -q -p turbo-bitset --bench mod -- --test
'bitset/storage_'` (216 cases)
- `cargo run -q --release -p turbo-bitset --example sizes` gives output
identical to before this change.
- Not run: a 32-bit cross-check, because the `i686-unknown-linux-gnu`
standard library is not installed locally. The test and benchmark
changes avoid assuming a 64-bit `usize`.

### Performance caveats

I compared release benchmark binaries for the previous `Vec`-only
implementation and this change. Each side was built in its own target
directory, I checked that the binary hashes differ, and the runs
alternated old/new/new/old.

- Before the final storage changes, the full comparison (336 cases)
averaged **0.992×** of the old timings.
- Those final changes switched `intersection` to `collect` and added
`#[inline]`. After them, a focused rerun of 84 previously slower cases
averaged **1.040×**:

| Group | New / old (geomean) |
| --- | ---: |
| `intersection` | 0.907 |
| `union` | 0.974 |
| `contains` | 1.049 |
| `insert` | 1.057 |
| `iter` | 1.112 |
| `from_iter` | 1.165 |

- **`iter` and `contains`:** some inputs remain slower even though their
normalized hot-loop instructions are the same in both binaries. The
cause is unresolved.
- **Empty `from_iter`:** about 3 ns becomes about 5 ns because of an
extra out-of-line call.

### `#[inline]` annotations

At `17be1d5e`, which annotated `Iter::next` and every `BlockStorage`
method, I compared builds with and without all 43 of those annotations:
full `mod` bench, 768 cases, runs alternating with/without/without/with,
separate target dirs, release profile (thin LTO, one codegen unit).

- Without any of them, the timings were **1.091×** overall (1.086× and
1.095× in the two pairs). The same binary run twice drifted 1.024× and
1.033×, but per group the same-binary drift reached about 30%, so this
is a shared-VM microbenchmark trend, not a proven whole-build effect.
- Removing only the annotation on `Iter::next`, or only the 28 on the
generic `SmallVec`/`ArrayVec` implementations, gives a bench binary
**byte-identical** to the annotated one.
- Removing the 14 on the `Vec` implementation reproduces the whole
difference. Without them, `Vec`'s `with_capacity` and `resize` wrappers
remain separate functions in the binary, even with thin LTO.
- So only the `Vec` annotations are kept. The bench binary of this PR's
tip is byte-identical to the one built at `17be1d5e`.

<!-- NEXT_JS_LLM -->


<!-- fleet b56269ed-2145-4f30-b0fe-dc9762520521 -->

---------
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarVercel Fleet <fleet@vercel.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>
parent b35fb85d
...@@ -9113,11 +9113,13 @@ dependencies = [ ...@@ -9113,11 +9113,13 @@ dependencies = [
name = "turbo-bitset" name = "turbo-bitset"
version = "0.1.0" version = "0.1.0"
dependencies = [ dependencies = [
"arrayvec 0.7.6",
"bincode", "bincode",
"codspeed-criterion-compat", "codspeed-criterion-compat",
"rand 0.10.1", "rand 0.10.1",
"rustc-hash 2.1.1", "rustc-hash 2.1.1",
"shrink-to-fit", "shrink-to-fit",
"smallvec",
"turbo-bincode", "turbo-bincode",
"turbo-tasks-hash", "turbo-tasks-hash",
] ]
......
...@@ -255,6 +255,7 @@ allsorts = { version = "0.14.0", default-features = false, features = [ ...@@ -255,6 +255,7 @@ allsorts = { version = "0.14.0", default-features = false, features = [
"flate2_rust", "flate2_rust",
] } ] }
anyhow = "1.0.100" anyhow = "1.0.100"
arrayvec = "0.7.6"
async-trait = "0.1.64" async-trait = "0.1.64"
base64 = "0.22.1" base64 = "0.22.1"
bincode = { version = "2.0.1", features = ["serde"] } bincode = { version = "2.0.1", features = ["serde"] }
......
...@@ -9,8 +9,10 @@ edition = "2024" ...@@ -9,8 +9,10 @@ edition = "2024"
workspace = true workspace = true
[dependencies] [dependencies]
arrayvec = { workspace = true }
bincode = { workspace = true } bincode = { workspace = true }
shrink-to-fit = { workspace = true } shrink-to-fit = { workspace = true }
smallvec = { workspace = true }
[dev-dependencies] [dev-dependencies]
criterion = { workspace = true } criterion = { workspace = true }
......
...@@ -10,6 +10,10 @@ ...@@ -10,6 +10,10 @@
//! The `hash` group hashes the sets with `FxHasher` through `std::hash::Hash`, and `hash_encode` //! The `hash` group hashes the sets with `FxHasher` through `std::hash::Hash`, and `hash_encode`
//! hashes a `Bitset` like a task input is hashed for the persistent cache: encoded with //! hashes a `Bitset` like a task input is hashed for the persistent cache: encoded with
//! `turbo_bincode`'s hash encoder into an `Xxh3Hash64Hasher`. //! `turbo_bincode`'s hash encoder into an `Xxh3Hash64Hasher`.
//!
//! The `storage_*` groups compare the block storages on a few operations, with the ids
//! `<storage>/<input>`: `Vec` (the default), `SmallVec4` (`SmallVec<[usize; 4]>`) and
//! `ArrayVec` (an `ArrayVec<usize, N>` with enough blocks for the largest universe).
#[path = "common/inputs.rs"] #[path = "common/inputs.rs"]
mod inputs; mod inputs;
...@@ -20,17 +24,22 @@ use std::{ ...@@ -20,17 +24,22 @@ use std::{
hint::black_box, hint::black_box,
}; };
use arrayvec::ArrayVec;
use bincode::Encode; use bincode::Encode;
use criterion::{BatchSize, BenchmarkId, Criterion, criterion_group, criterion_main}; use criterion::{
BatchSize, BenchmarkGroup, BenchmarkId, Criterion, criterion_group, criterion_main,
measurement::WallTime,
};
use rand::seq::SliceRandom; use rand::seq::SliceRandom;
use rustc_hash::FxBuildHasher; use rustc_hash::FxBuildHasher;
use smallvec::SmallVec;
use turbo_bincode::{ use turbo_bincode::{
TurboBincodeBuffer, new_hash_encoder, turbo_bincode_decode, turbo_bincode_encode, TurboBincodeBuffer, new_hash_encoder, turbo_bincode_decode, turbo_bincode_encode,
}; };
use turbo_bitset::Bitset; use turbo_bitset::{Bitset, BlockStorage};
use turbo_tasks_hash::Xxh3Hash64Hasher; use turbo_tasks_hash::Xxh3Hash64Hasher;
use self::inputs::{Input, inputs, rng}; use self::inputs::{Input, UNIVERSES, inputs, rng};
fn bitset(bits: &[u32]) -> Bitset { fn bitset(bits: &[u32]) -> Bitset {
bits.iter().copied().collect() bits.iter().copied().collect()
...@@ -141,7 +150,7 @@ fn bench_insert(c: &mut Criterion, name: &str, inputs: &[Input], order: fn(&[u32 ...@@ -141,7 +150,7 @@ fn bench_insert(c: &mut Criterion, name: &str, inputs: &[Input], order: fn(&[u32
inputs, inputs,
|i| order(&i.bits), |i| order(&i.bits),
|bits| { |bits| {
let mut set = Bitset::new(); let mut set: Bitset = Bitset::new();
for &bit in bits { for &bit in bits {
black_box(set.insert(bit)); black_box(set.insert(bit));
} }
...@@ -150,6 +159,95 @@ fn bench_insert(c: &mut Criterion, name: &str, inputs: &[Input], order: fn(&[u32 ...@@ -150,6 +159,95 @@ fn bench_insert(c: &mut Criterion, name: &str, inputs: &[Input], order: fn(&[u32
); );
} }
/// The `ArrayVec` capacity of the storage benchmarks, enough for the largest universe.
const ARRAY_BLOCKS: usize =
(UNIVERSES[UNIVERSES.len() - 1] as usize).div_ceil(usize::BITS as usize);
/// Inserting the bits of `input` one by one in a fixed random order, with the storage `S`.
fn storage_insert<S: BlockStorage>(
group: &mut BenchmarkGroup<'_, WallTime>,
storage: &str,
input: &Input,
) {
let bits = shuffled(&input.bits);
group.bench_function(BenchmarkId::new(storage, &input.name), |b| {
b.iter(|| {
let mut set = Bitset::<S>::new();
for &bit in black_box(&bits) {
black_box(set.insert(bit));
}
black_box(set);
})
});
}
/// `contains` for the probes of `input`, with the storage `S`.
fn storage_contains<S: BlockStorage>(
group: &mut BenchmarkGroup<'_, WallTime>,
storage: &str,
input: &Input,
) {
let set = input.bits.iter().copied().collect::<Bitset<S>>();
group.bench_function(BenchmarkId::new(storage, &input.name), |b| {
b.iter(|| {
for &bit in black_box(&input.probes) {
black_box(black_box(&set).contains(bit));
}
})
});
}
/// `a |= b` like the `union` group, with the storage `S`.
fn storage_union<S: BlockStorage>(
group: &mut BenchmarkGroup<'_, WallTime>,
storage: &str,
input: &Input,
) {
let a = input.bits.iter().copied().collect::<Bitset<S>>();
let b = input.other_bits.iter().copied().collect::<Bitset<S>>();
group.bench_function(BenchmarkId::new(storage, &input.name), |bencher| {
bencher.iter_batched_ref(
|| a.clone(),
|a| {
*a |= black_box(&b);
black_box(&*a);
},
BatchSize::SmallInput,
)
});
}
/// Runs a `storage_*` function for every compared storage.
macro_rules! for_each_storage {
($bench:ident, $group:expr, $input:expr) => {
$bench::<Vec<usize>>($group, "Vec", $input);
$bench::<SmallVec<[usize; 4]>>($group, "SmallVec4", $input);
$bench::<ArrayVec<usize, ARRAY_BLOCKS>>($group, "ArrayVec", $input);
};
}
/// Compares the block storages on inserting, `contains` and `|=`.
fn bench_storages(c: &mut Criterion, inputs: &[Input]) {
for input in inputs {
assert!(input.universe as usize <= ARRAY_BLOCKS * usize::BITS as usize);
}
let mut group = c.benchmark_group("bitset/storage_insert");
for input in inputs {
for_each_storage!(storage_insert, &mut group, input);
}
group.finish();
let mut group = c.benchmark_group("bitset/storage_contains");
for input in inputs {
for_each_storage!(storage_contains, &mut group, input);
}
group.finish();
let mut group = c.benchmark_group("bitset/storage_union");
for input in inputs {
for_each_storage!(storage_union, &mut group, input);
}
group.finish();
}
pub fn bench(c: &mut Criterion) { pub fn bench(c: &mut Criterion) {
let inputs = inputs(); let inputs = inputs();
validate(&inputs); validate(&inputs);
...@@ -359,6 +457,8 @@ pub fn bench(c: &mut Criterion) { ...@@ -359,6 +457,8 @@ pub fn bench(c: &mut Criterion) {
black_box(hash_encode(black_box(set))); black_box(hash_encode(black_box(set)));
}, },
); );
bench_storages(c, &inputs);
} }
/// The bits in a fixed pseudo-random order. /// The bits in a fixed pseudo-random order.
......
...@@ -27,6 +27,10 @@ ...@@ -27,6 +27,10 @@
//! //!
//! With fixed-width integers (used to hash values instead of storing them) the bitmap form is //! With fixed-width integers (used to hash values instead of storing them) the bitmap form is
//! always used. The decoder accepts both forms with any config. //! always used. The decoder accepts both forms with any config.
//!
//! The encoding doesn't depend on the [`BlockStorage`], so a set can be decoded into a different
//! storage than it was encoded from. Decoding a set that needs more blocks than the storage can
//! hold returns an error.
use bincode::{ use bincode::{
Decode, Encode, Decode, Encode,
...@@ -40,7 +44,7 @@ use bincode::{ ...@@ -40,7 +44,7 @@ use bincode::{
impl_borrow_decode, impl_borrow_decode,
}; };
use crate::{BLOCK_BITS, Bitset}; use crate::{BLOCK_BITS, Bitset, BlockStorage};
/// Number of bytes per storage block. /// Number of bytes per storage block.
const BLOCK_BYTES: usize = BLOCK_BITS / 8; const BLOCK_BYTES: usize = BLOCK_BITS / 8;
...@@ -163,7 +167,7 @@ fn write_list_items<E: Encoder>( ...@@ -163,7 +167,7 @@ fn write_list_items<E: Encoder>(
Ok(()) Ok(())
} }
impl Bitset { impl<S: BlockStorage> Bitset<S> {
/// The trimmed byte range of the set, or `None` if it is empty. /// The trimmed byte range of the set, or `None` if it is empty.
fn byte_range(&self) -> Option<ByteRange<'_>> { fn byte_range(&self) -> Option<ByteRange<'_>> {
let (first_block, blocks) = self.trimmed(); let (first_block, blocks) = self.trimmed();
...@@ -179,7 +183,7 @@ impl Bitset { ...@@ -179,7 +183,7 @@ impl Bitset {
} }
} }
impl Encode for Bitset { impl<S: BlockStorage> Encode for Bitset<S> {
fn encode<E: Encoder>(&self, encoder: &mut E) -> Result<(), EncodeError> { fn encode<E: Encoder>(&self, encoder: &mut E) -> Result<(), EncodeError> {
let Some(range) = self.byte_range() else { let Some(range) = self.byte_range() else {
return 0i64.encode(encoder); return 0i64.encode(encoder);
...@@ -203,7 +207,7 @@ impl Encode for Bitset { ...@@ -203,7 +207,7 @@ impl Encode for Bitset {
} }
} }
impl<Context> Decode<Context> for Bitset { impl<Context, S: BlockStorage> Decode<Context> for Bitset<S> {
fn decode<D: Decoder<Context = Context>>(decoder: &mut D) -> Result<Self, DecodeError> { fn decode<D: Decoder<Context = Context>>(decoder: &mut D) -> Result<Self, DecodeError> {
let len = i64::decode(decoder)?; let len = i64::decode(decoder)?;
match len { match len {
...@@ -214,39 +218,51 @@ impl<Context> Decode<Context> for Bitset { ...@@ -214,39 +218,51 @@ impl<Context> Decode<Context> for Bitset {
} }
} }
impl_borrow_decode!(Bitset); impl_borrow_decode!(Bitset<S>, S: BlockStorage);
/// Allocates the zeroed storage for the bytes `byte_offset..byte_offset + len`, claiming its size /// Allocates the zeroed storage for the bytes `byte_offset..byte_offset + len`, claiming its size
/// from the decoder's limit first. Returns the index of the first block and the storage. /// from the decoder's limit first. Returns the index of the first block and the storage.
fn allocate<D: Decoder>( fn allocate<D: Decoder, S: BlockStorage>(
decoder: &mut D, decoder: &mut D,
byte_offset: u64, byte_offset: u64,
len: u64, len: u64,
) -> Result<(usize, Vec<usize>), DecodeError> { ) -> Result<(usize, S), DecodeError> {
if len > MAX_BYTES || byte_offset + len > MAX_BYTES { if len > MAX_BYTES || byte_offset + len > MAX_BYTES {
return Err(DecodeError::Other("Bitset extends beyond the u32 range")); return Err(DecodeError::Other("Bitset extends beyond the u32 range"));
} }
let block_bytes = BLOCK_BYTES as u64; let block_bytes = BLOCK_BYTES as u64;
let first_block = byte_offset / block_bytes; let first_block = byte_offset / block_bytes;
let block_count = ((byte_offset + len).div_ceil(block_bytes) - first_block) as usize; let block_count = ((byte_offset + len).div_ceil(block_bytes) - first_block) as usize;
if block_count > S::max_len() {
return Err(DecodeError::Other("Bitset doesn't fit into its storage"));
}
decoder.claim_bytes_read(block_count * BLOCK_BYTES)?; decoder.claim_bytes_read(block_count * BLOCK_BYTES)?;
Ok((first_block as usize, vec![0; block_count])) let mut blocks = S::with_capacity(block_count);
blocks.resize(block_count);
Ok((first_block as usize, blocks))
} }
fn decode_bitmap<D: Decoder>(decoder: &mut D, len: u64) -> Result<Bitset, DecodeError> { fn decode_bitmap<D: Decoder, S: BlockStorage>(
decoder: &mut D,
len: u64,
) -> Result<Bitset<S>, DecodeError> {
let byte_offset = u32::decode(decoder)? as u64; let byte_offset = u32::decode(decoder)? as u64;
let (first_block, mut blocks) = allocate(decoder, byte_offset, len)?; let (first_block, mut blocks) = allocate::<_, S>(decoder, byte_offset, len)?;
// Read the payload straight into the storage's bytes. // Read the payload straight into the storage's bytes.
let start = (byte_offset - first_block as u64 * BLOCK_BYTES as u64) as usize; let start = (byte_offset - first_block as u64 * BLOCK_BYTES as u64) as usize;
// SAFETY: The slice covers exactly the memory of `blocks` and borrows it mutably for its let block_slice = blocks.as_mut_slice();
// SAFETY: The slice covers exactly the memory of `block_slice` and borrows it mutably for its
// lifetime. `u8` has no alignment requirements, and every byte pattern is a valid `usize`. // lifetime. `u8` has no alignment requirements, and every byte pattern is a valid `usize`.
let bytes = unsafe { let bytes = unsafe {
std::slice::from_raw_parts_mut(blocks.as_mut_ptr().cast::<u8>(), size_of_val(&*blocks)) std::slice::from_raw_parts_mut(
block_slice.as_mut_ptr().cast::<u8>(),
size_of_val(&*block_slice),
)
}; };
decoder decoder
.reader() .reader()
.read(&mut bytes[start..start + len as usize])?; .read(&mut bytes[start..start + len as usize])?;
le_blocks_to_native(&mut blocks, cfg!(target_endian = "big")); le_blocks_to_native(blocks.as_mut_slice(), cfg!(target_endian = "big"));
Ok(Bitset { Ok(Bitset {
offset: first_block, offset: first_block,
blocks, blocks,
...@@ -264,12 +280,12 @@ fn le_blocks_to_native(blocks: &mut [usize], big_endian_target: bool) { ...@@ -264,12 +280,12 @@ fn le_blocks_to_native(blocks: &mut [usize], big_endian_target: bool) {
} }
/// Sets bits in a preallocated storage, keeping the bits of the current block in a register. /// Sets bits in a preallocated storage, keeping the bits of the current block in a register.
struct ListBuilder { struct ListBuilder<S> {
/// The first bit of the storage. /// The first bit of the storage.
base: u64, base: u64,
/// The end of the bits that may be set (exclusive), relative to `base`. /// The end of the bits that may be set (exclusive), relative to `base`.
end: u64, end: u64,
blocks: Vec<usize>, blocks: S,
/// The index of the current block and its bits, not yet stored in `blocks`. /// The index of the current block and its bits, not yet stored in `blocks`.
index: usize, index: usize,
word: usize, word: usize,
...@@ -277,7 +293,7 @@ struct ListBuilder { ...@@ -277,7 +293,7 @@ struct ListBuilder {
last: u64, last: u64,
} }
impl ListBuilder { impl<S: BlockStorage> ListBuilder<S> {
/// Sets the bit `last + difference`, which must be within the storage. /// Sets the bit `last + difference`, which must be within the storage.
fn push(&mut self, difference: u64) -> Result<(), DecodeError> { fn push(&mut self, difference: u64) -> Result<(), DecodeError> {
let bit = self.last + difference; let bit = self.last + difference;
...@@ -286,7 +302,7 @@ impl ListBuilder { ...@@ -286,7 +302,7 @@ impl ListBuilder {
} }
let index = (bit / BLOCK_BITS as u64) as usize; let index = (bit / BLOCK_BITS as u64) as usize;
if index != self.index { if index != self.index {
self.blocks[self.index] |= self.word; self.blocks.as_mut_slice()[self.index] |= self.word;
self.index = index; self.index = index;
self.word = 0; self.word = 0;
} }
...@@ -295,8 +311,8 @@ impl ListBuilder { ...@@ -295,8 +311,8 @@ impl ListBuilder {
Ok(()) Ok(())
} }
fn finish(mut self) -> Vec<usize> { fn finish(mut self) -> S {
self.blocks[self.index] |= self.word; self.blocks.as_mut_slice()[self.index] |= self.word;
self.blocks self.blocks
} }
} }
...@@ -310,7 +326,10 @@ fn decode_difference<D: Decoder>(decoder: &mut D) -> Result<u64, DecodeError> { ...@@ -310,7 +326,10 @@ fn decode_difference<D: Decoder>(decoder: &mut D) -> Result<u64, DecodeError> {
Ok(difference as u64) Ok(difference as u64)
} }
fn decode_list<D: Decoder>(decoder: &mut D, count: u64) -> Result<Bitset, DecodeError> { fn decode_list<D: Decoder, S: BlockStorage>(
decoder: &mut D,
count: u64,
) -> Result<Bitset<S>, DecodeError> {
if count > 1 << 32 { if count > 1 << 32 {
return Err(DecodeError::Other( return Err(DecodeError::Other(
"Bitset list has more items than the u32 range", "Bitset list has more items than the u32 range",
...@@ -324,7 +343,7 @@ fn decode_list<D: Decoder>(decoder: &mut D, count: u64) -> Result<Bitset, Decode ...@@ -324,7 +343,7 @@ fn decode_list<D: Decoder>(decoder: &mut D, count: u64) -> Result<Bitset, Decode
} }
let first = u32::decode(decoder)? as u64; let first = u32::decode(decoder)? as u64;
let byte_offset = first / 8; let byte_offset = first / 8;
let (first_block, blocks) = allocate(decoder, byte_offset, len)?; let (first_block, blocks) = allocate::<_, S>(decoder, byte_offset, len)?;
let base = (first_block * BLOCK_BITS) as u64; let base = (first_block * BLOCK_BITS) as u64;
let mut builder = ListBuilder { let mut builder = ListBuilder {
base, base,
......
This diff is collapsed.
//! The storage of the blocks of a [`Bitset`](crate::Bitset).
use arrayvec::ArrayVec;
use smallvec::SmallVec;
/// A growable list of `usize` blocks that a [`Bitset`](crate::Bitset) stores its bits in.
///
/// Implemented for:
///
/// - `Vec<usize>` (the default): grows without a limit, and [`shrink_to_fit`] releases all spare
/// capacity.
/// - `SmallVec<[usize; N]>`: stores up to `N` blocks inline and moves to the heap beyond that, like
/// a `Vec`. [`shrink_to_fit`] moves the blocks back inline when they fit.
/// - `ArrayVec<usize, N>`: always stores the blocks inline, so it holds at most `N` blocks. Growing
/// beyond that panics, and [`shrink_to_fit`] can't release anything.
///
/// The methods follow the `Vec` methods of the same name. Every method that adds blocks panics if
/// the result would exceed [`max_len`] blocks. Collecting more blocks than that from an iterator
/// (`FromIterator`) panics the same way.
///
/// An `ArrayVec<usize, N>` always has a capacity of exactly `N` blocks, so
/// [`with_capacity`](BlockStorage::with_capacity) accepts any request up to `N` and panics above
/// it.
///
/// [`shrink_to_fit`]: BlockStorage::shrink_to_fit
/// [`max_len`]: BlockStorage::max_len
pub trait BlockStorage: Clone + FromIterator<usize> {
/// Creates an empty storage without allocating.
fn new() -> Self;
/// Creates an empty storage with capacity for at least `capacity` blocks.
///
/// The capacity is exact if the storage needs to allocate for it. A storage with a fixed
/// capacity accepts any request up to [`BlockStorage::max_len`] and panics above it.
fn with_capacity(capacity: usize) -> Self;
/// The maximum number of blocks that the storage can hold.
fn max_len() -> usize;
/// The number of blocks that the storage can hold without reallocating.
fn capacity(&self) -> usize;
/// The capacity that a storage of `len` blocks has after [`BlockStorage::shrink_to_fit`]. A
/// set only moves its blocks to shrink the storage if this is below the current capacity.
fn shrunk_capacity(len: usize) -> usize;
/// The stored blocks.
fn as_slice(&self) -> &[usize];
/// The stored blocks, mutably.
fn as_mut_slice(&mut self) -> &mut [usize];
/// Resizes the storage to `len` blocks, appending zero blocks if it grows.
fn resize(&mut self, len: usize);
/// Appends `blocks`.
fn extend_from_slice(&mut self, blocks: &[usize]);
/// Reserves capacity for exactly `additional` more blocks, if the storage allocates.
fn reserve_exact(&mut self, additional: usize);
/// Keeps the first `len` blocks, dropping the rest.
fn truncate(&mut self, len: usize);
/// Removes the first `len` blocks, moving the others to the front.
fn remove_prefix(&mut self, len: usize);
/// Removes all blocks, keeping the capacity.
fn clear(&mut self);
/// Releases as much capacity as possible.
fn shrink_to_fit(&mut self);
}
// Only the `Vec` implementation is `#[inline]`: it isn't generic, so the attribute makes its small
// wrappers available for inlining in other crates; without it, some of them stay out of line even
// with the workspace's thin LTO. The `SmallVec` and `ArrayVec` implementations are generic and
// therefore instantiated in the calling crate.
impl BlockStorage for Vec<usize> {
#[inline]
fn new() -> Self {
Vec::new()
}
#[inline]
fn with_capacity(capacity: usize) -> Self {
Vec::with_capacity(capacity)
}
#[inline]
fn max_len() -> usize {
usize::MAX
}
#[inline]
fn capacity(&self) -> usize {
self.capacity()
}
#[inline]
fn shrunk_capacity(len: usize) -> usize {
len
}
#[inline]
fn as_slice(&self) -> &[usize] {
self
}
#[inline]
fn as_mut_slice(&mut self) -> &mut [usize] {
self
}
#[inline]
fn resize(&mut self, len: usize) {
self.resize(len, 0);
}
#[inline]
fn extend_from_slice(&mut self, blocks: &[usize]) {
self.extend_from_slice(blocks);
}
#[inline]
fn reserve_exact(&mut self, additional: usize) {
self.reserve_exact(additional);
}
#[inline]
fn truncate(&mut self, len: usize) {
self.truncate(len);
}
#[inline]
fn remove_prefix(&mut self, len: usize) {
self.drain(..len);
}
#[inline]
fn clear(&mut self) {
self.clear();
}
#[inline]
fn shrink_to_fit(&mut self) {
self.shrink_to_fit();
}
}
impl<const N: usize> BlockStorage for SmallVec<[usize; N]> {
fn new() -> Self {
SmallVec::new()
}
fn with_capacity(capacity: usize) -> Self {
SmallVec::with_capacity(capacity)
}
fn max_len() -> usize {
usize::MAX
}
fn capacity(&self) -> usize {
self.capacity()
}
fn shrunk_capacity(len: usize) -> usize {
len.max(N)
}
fn as_slice(&self) -> &[usize] {
self
}
fn as_mut_slice(&mut self) -> &mut [usize] {
self
}
fn resize(&mut self, len: usize) {
self.resize(len, 0);
}
fn extend_from_slice(&mut self, blocks: &[usize]) {
self.extend_from_slice(blocks);
}
fn reserve_exact(&mut self, additional: usize) {
self.reserve_exact(additional);
}
fn truncate(&mut self, len: usize) {
self.truncate(len);
}
fn remove_prefix(&mut self, len: usize) {
self.drain(..len);
}
fn clear(&mut self) {
self.clear();
}
fn shrink_to_fit(&mut self) {
self.shrink_to_fit();
}
}
/// Panics because an `ArrayVec` with capacity `N` can't hold `len` blocks.
#[cold]
#[track_caller]
fn array_overflow<const N: usize>(len: usize) -> ! {
panic!("a Bitset storage of ArrayVec<usize, {N}> can't hold {len} blocks")
}
impl<const N: usize> BlockStorage for ArrayVec<usize, N> {
fn new() -> Self {
ArrayVec::new()
}
#[track_caller]
fn with_capacity(capacity: usize) -> Self {
if capacity > N {
array_overflow::<N>(capacity);
}
ArrayVec::new()
}
fn max_len() -> usize {
N
}
fn capacity(&self) -> usize {
N
}
fn shrunk_capacity(_len: usize) -> usize {
N
}
fn as_slice(&self) -> &[usize] {
self
}
fn as_mut_slice(&mut self) -> &mut [usize] {
self
}
#[track_caller]
fn resize(&mut self, len: usize) {
if len > N {
array_overflow::<N>(len);
}
if len <= self.len() {
self.truncate(len);
} else {
for _ in self.len()..len {
self.push(0);
}
}
}
#[track_caller]
fn extend_from_slice(&mut self, blocks: &[usize]) {
if self.try_extend_from_slice(blocks).is_err() {
array_overflow::<N>(self.len() + blocks.len());
}
}
#[track_caller]
fn reserve_exact(&mut self, additional: usize) {
if self.len() + additional > N {
array_overflow::<N>(self.len() + additional);
}
}
fn truncate(&mut self, len: usize) {
self.truncate(len);
}
fn remove_prefix(&mut self, len: usize) {
self.drain(..len);
}
fn clear(&mut self) {
self.clear();
}
fn shrink_to_fit(&mut self) {}
}
This diff is collapsed.
...@@ -285,7 +285,7 @@ fn front_and_back_growth() { ...@@ -285,7 +285,7 @@ fn front_and_back_growth() {
#[test] #[test]
fn front_growth_reuses_capacity() { fn front_growth_reuses_capacity() {
let block = usize::BITS; let block = usize::BITS;
let mut set = Bitset::new(); let mut set: Bitset = Bitset::new();
let mut expected = BTreeSet::new(); let mut expected = BTreeSet::new();
for bit in [100 * block + 3, 101 * block] { for bit in [100 * block + 3, 101 * block] {
set.insert(bit); set.insert(bit);
...@@ -529,7 +529,7 @@ fn union_with_into_empty_is_exact() { ...@@ -529,7 +529,7 @@ fn union_with_into_empty_is_exact() {
#[test] #[test]
fn union_with_empty_source_keeps_storage() { fn union_with_empty_source_keeps_storage() {
let mut removed = Bitset::new(); let mut removed: Bitset = Bitset::new();
removed.insert(5_000); removed.insert(5_000);
removed.remove(5_000); removed.remove(5_000);
let sources = [Bitset::new(), Bitset::with_len(500), removed]; let sources = [Bitset::new(), Bitset::with_len(500), removed];
...@@ -754,7 +754,7 @@ fn front_growth_beyond_the_capacity_keeps_all_bits() { ...@@ -754,7 +754,7 @@ fn front_growth_beyond_the_capacity_keeps_all_bits() {
50 * block - 1, 50 * block - 1,
), ),
] { ] {
let mut set = Bitset::new(); let mut set: Bitset = Bitset::new();
let mut expected = BTreeSet::new(); let mut expected = BTreeSet::new();
for &bit in existing { for &bit in existing {
set.insert(bit); set.insert(bit);
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment