turbo-bitset: add Bitset, a fast bitset over a bounded bit range (#100566)
Stacked on #100452. This PR only adds the crate; the PRs above use it:
#100643 (`compute_chunk_group_info`) → #100567 (`collect_graph`) →
#100568 (`compute_merged_modules`) → #100630 (persisted chunk group
sets) → #100724 (removes the FixedBitSet/Roaring comparisons from the
benchmarks).
## Summary
The module graph fixed-point traversals track set membership with
`RoaringBitmap`, which is built for compact storage, not speed. They
check "would `current |= parent` change `current`?" on every edge and
then do the union; on array containers both cost time proportional to
the number of bits.
This PR adds a new `turbo-bitset` crate with `Bitset`: a `Vec<usize>` of
blocks plus an offset counted in whole blocks.
- Bit `i` is stored at `i - offset * usize::BITS`, so a set only stores
the word range between its lowest and highest bit. Sets whose bits are
clustered at high indices stay small.
- Unions, intersections, differences, subset checks, `is_disjoint` and
`intersection_len` are word loops over the overlapping range, so their
cost is proportional to the range, not the number of bits. The subset
check reduces chunks of 4 words before branching, so it vectorizes.
- `union_with` returns whether it added a bit. It ORs the words and
collects the added bits in the same loop, so a traversal checks for a
change and applies the union in one pass instead of a subset check plus
`|=`. If the set stores no blocks (a module's first visit), it copies
`other` without its zero edge blocks, reusing a reservation or
allocating exactly the copied blocks, so the traversals need no separate
`is_zero()` check and `clone()`.
- `contains`, `insert` and `remove` check the stored range with a single
comparison. Growth is amortized at both ends: growing at the back uses
the `Vec`'s growth, and growing at the front moves the blocks once to
the end of the (reused or doubled) capacity, which leaves headroom for
further front growth. So inserting in any order, including descending,
moves each block O(log n) times. `extend` inserts the items directly.
- `Bitset::with_len(bits: u32)` stores exactly the zeroed blocks for the
bits `0..bits` at offset 0, in an allocation without spare capacity, so
inserting any of them doesn't grow the storage.
`Bitset::with_capacity(bits: u32)` reserves capacity for that many bits
without storing any block; the first insert still picks the offset.
- Equality and hashing depend only on the bits, not on the stored range.
`count_ones()` counts over the stored range (the name says it isn't
free), `is_zero()` checks for no bits. `Bitset` implements
`ShrinkToFit`.
- The storage is a plain `Vec`. A `SmallVec<[usize; INLINE]>` variant
was tried and reverted: it adds an inline-or-heap branch to every block
access, and no user needs inline blocks. Against it, `Vec` measured
0.967× overall in the microbenchmarks; unions right after a clone are
slower, because `Vec::clone` allocates exactly the stored blocks
(details in the review thread). Clones keep the default exact capacity,
to avoid extra memory per module set in the traversals.
- `to_roaring()` converts back container by container (2^16 bits at a
time) straight from the words, with the same container types (and
serialization) as inserting the bits one by one. The PRs above use it at
the boundary until #100630 removes it.
- No `#[inline]` hints: the crate relies on thin LTO. Removing the 8
hints it had (one went away with the hand-written varint writer) changed
41 of 480 benchmark cases by ≥ 5% consistently (24 faster, 17 slower;
geometric mean 0.99×), and restoring the decoder's most likely candidate
made 3 cases faster and 3 slower, so none is kept.
### Bincode encoding
`Encode`/`Decode` (in `src/encode.rs`) start with a signed length
header:
- `0`: the empty set.
- `len > 0`, the **bitmap form**: a `u32` byte offset, then `len` raw
bytes, with leading and trailing zero bytes trimmed.
- `len = -n < 0`, the **list form**: the payload length `len` the bitmap
form would have, then the first bit and `n - 1` differences (each `≥
1`), all `u32` (varints under the standard config). The bits must cover
exactly that payload range.
Which form is written:
- With varint integers (how Turbopack persists), the list form is used
iff `5 · n < 4 · len`, for `n` bits and a bitmap payload of `len` bytes.
That only needs the bit count (counting stops once it reaches the
threshold), not the sizes of the list items.
- If the bitmap is chosen, its payload is at most 1.25× the list's
items, as every item takes at least one byte.
- If the list is chosen, the bits are more than 10 apart on average, and
only differences above 250 take more than one byte; their sum is below
`8 · len`, so the differences take fewer bytes than the bitmap payload.
- The rule ignores the headers and the first item, so for sets with very
few bits the chosen form can be a few bytes (at most 13) larger than the
other.
- For a set spanning its whole range, the switch is at 10% density. The
10%-of-16,384-bits benchmark input has exactly 1,638 bits and a
2,047-byte payload, just at the threshold (5 · 1,638 = 8,190 ≥ 4 · 2,047
= 8,188), so it is written as a bitmap.
- With fixed-width integers (how task inputs are hashed), always the
bitmap form.
Integers are encoded with bincode's own `Encode` impls. The list items
are encoded with the outer encoder's config into a 1 KB stack buffer
(bincode's `SliceWriter`), which is written in chunks, a block at a
time; encoding doesn't allocate. The bitmap bytes are little-endian on
every platform: on little-endian targets they are written and read with
a single copy of the storage viewed as bytes (documented `unsafe` slice
casts), big-endian targets convert each block.
Decoding allocates the storage once (from the bitmap length in both
forms), claiming it from the decoder's byte limit first. List items are
set into a register-held block, and runs of single-byte varints are
handled in batches, still claiming 4 bytes per item like `u32::decode`.
Malformed input is rejected with a `DecodeError`: an out-of-range header
or payload, a bitmap or list past the `u32` range, a zero or overflowing
difference, a list item outside its payload range or a list that doesn't
end in its last byte, and truncated input.
## Benchmarks
All numbers in this section are criterion point estimates from `cargo
bench -p turbo-bitset` (workspace release settings, thin LTO,
`RUSTFLAGS="-C llvm-args=-x86-branches-within-32B-boundaries"`, rustc
1.101.0-nightly 2026-10-05; bincode 2.0.1, fixedbitset 0.5.7, roaring
0.11.5, codspeed-criterion-compat 4.3.0), on this PR's `Vec`-backed code
before the sizing constructors and `union_with -> bool` were added (tree
of `d7d69b85`), on a shared VM with 8 vCPUs and 16 GB.
### Microbenchmarks
Each operation compares `Bitset` with a plain `FixedBitSet` (indexed
from bit 0) and `RoaringBitmap` 0.11.5. Inputs are `empty`, `1`, `4` and
`16` random bits, plus densities that set at least 32 bits (10%, 50%,
100% of 1,024 bits; 1%, 10%, 50%, 100% of 16,384 bits; 0.1%, 1%, 10%,
50%, 100% of 65,536 bits). Every input has exactly its count, or exactly
⌊universe · density⌋ distinct bits, picked by selection sampling from a
seeded `SmallRng`, starting at bit 0. The `sizes` example uses the same
inputs (`benches/common/inputs.rs`). Results are checked for agreement
outside the measurement. Cells are Bitset / FixedBitSet / RoaringBitmap
in ns, 16,384-bit universe:
| operation | 1 bit | 16 bits | 1% | 10% | 100% |
|---|---|---|---|---|---|
| union (`a \|= b`) | 52 / 9.8 / 24 | 178 / 81 / 92 | 75 / 79 / 600 | 78
/ 78 / 5,633 | 101 / 84 / 834 |
| superset, holds | 2.3 / 3.1 / 3.8 | 58 / 90 / 32 | 63 / 89 / 263 | 62
/ 89 / 2,728 | 72 / 89 / 384 |
| intersection | 2.9 / 19 / 23 | 102 / 106 / 52 | 90 / 102 / 350 | 88 /
101 / 3,383 | 86 / 102 / 918 |
| contains (1,024 lookups) | 774 / 593 / 3,234 | 731 / 768 / 4,309 | 765
/ 749 / 9,437 | 750 / 748 / 12,535 | 766 / 747 / 2,474 |
| insert (random order) | 19 / 24 / 35 | 249 / 119 / 255 | 387 / 261 /
3,709 | 2,225 / 1,695 / 69,182 | 19,087 / 14,452 / 328,090 |
| iter | 2.3 / 57 / 8.0 | 120 / 157 / 52 | 282 / 256 / 389 | 1,337 /
1,098 / 3,832 | 15,188 / 13,820 / 44,408 |
<details>
<summary>Charts: microbenchmarks (log scale)</summary>
log10 of the time in ns (1 = 10 ns, 3 = 1 µs), 16,384-bit universe.
Exact values are in the table above.
```mermaid
xychart-beta
title "union (a |= b)"
x-axis ["1 bit", "16 bits", "1%", "10%", "100%"]
y-axis "log10(ns)"
line "Bitset" [1.72, 2.25, 1.88, 1.89, 2.0]
line "FixedBitSet" [0.99, 1.91, 1.9, 1.89, 1.92]
line "RoaringBitmap" [1.38, 1.97, 2.78, 3.75, 2.92]
```
```mermaid
xychart-beta
title "superset, holds"
x-axis ["1 bit", "16 bits", "1%", "10%", "100%"]
y-axis "log10(ns)"
line "Bitset" [0.37, 1.77, 1.8, 1.79, 1.86]
line "FixedBitSet" [0.49, 1.95, 1.95, 1.95, 1.95]
line "RoaringBitmap" [0.58, 1.51, 2.42, 3.44, 2.58]
```
```mermaid
xychart-beta
title "intersection"
x-axis ["1 bit", "16 bits", "1%", "10%", "100%"]
y-axis "log10(ns)"
line "Bitset" [0.46, 2.01, 1.96, 1.94, 1.93]
line "FixedBitSet" [1.27, 2.03, 2.01, 2.0, 2.01]
line "RoaringBitmap" [1.36, 1.72, 2.54, 3.53, 2.96]
```
```mermaid
xychart-beta
title "contains (1,024 lookups)"
x-axis ["1 bit", "16 bits", "1%", "10%", "100%"]
y-axis "log10(ns)"
line "Bitset" [2.89, 2.86, 2.88, 2.88, 2.88]
line "FixedBitSet" [2.77, 2.89, 2.87, 2.87, 2.87]
line "RoaringBitmap" [3.51, 3.63, 3.97, 4.1, 3.39]
```
```mermaid
xychart-beta
title "insert (random order)"
x-axis ["1 bit", "16 bits", "1%", "10%", "100%"]
y-axis "log10(ns)"
line "Bitset" [1.29, 2.4, 2.59, 3.35, 4.28]
line "FixedBitSet" [1.39, 2.08, 2.42, 3.23, 4.16]
line "RoaringBitmap" [1.54, 2.41, 3.57, 4.84, 5.52]
```
```mermaid
xychart-beta
title "iter"
x-axis ["1 bit", "16 bits", "1%", "10%", "100%"]
y-axis "log10(ns)"
line "Bitset" [0.36, 2.08, 2.45, 3.13, 4.18]
line "FixedBitSet" [1.76, 2.2, 2.41, 3.04, 4.14]
line "RoaringBitmap" [0.91, 1.72, 2.59, 3.58, 4.65]
```
</details>
### Ordered inserts
Building a set by inserting its bits one by one in a fixed random order,
ascending (the set only grows at the back) and descending (it only grows
at the front). FixedBitSet uses `grow_and_insert`. Bitset / FixedBitSet
/ RoaringBitmap in ns, 16,384-bit universe:
| order | 1 bit | 16 bits | 1% | 10% | 100% |
|---|---|---|---|---|---|
| random order | 19 / 24 / 35 | 249 / 119 / 255 | 387 / 261 / 3,709 |
2,225 / 1,695 / 69,182 | 19,087 / 14,452 / 328,090 |
| ascending | 20 / 25 / 35 | 337 / 362 / 241 | 971 / 1,038 / 2,879 |
3,385 / 7,760 / 29,593 | 31,185 / 71,557 / 160,400 |
| descending | 19 / 25 / 36 | 209 / 74 / 286 | 417 / 201 / 4,570 | 2,431
/ 1,666 / 82,091 | 29,920 / 27,721 / 393,320 |
Before/after A/B of this round's growth change (front growth with
headroom instead of an exact shift, and `extend` inserting items
directly instead of buffering them). Both sides use the new inputs and
benchmarks; "before" is the previous `ensure_blocks`/`Extend` on the
rebased code. Two interleaved rounds per side (before, after, after,
before), mean in ns:
| input | `insert_desc` before → after | `from_iter` before → after |
|---|---|---|
| `low-1024-16bits` | 167 → 116 (0.69×) | 70 → 146 (2.08×) |
| `low-1024-10%` | 377 → 209 (0.55×) | 277 → 292 (1.05×) |
| `low-16384-16bits` | 406 → 227 (0.56×) | 108 → 296 (2.76×) |
| `low-16384-1%` | 2,497 → 413 (0.17×) | 424 → 1,046 (2.47×) |
| `low-16384-10%` | 7,285 → 2,378 (0.33×) | 3,889 → 3,392 (0.87×) |
| `low-16384-100%` | 35,372 → 30,340 (0.86×) | 47,816 → 29,290 (0.61×) |
| `low-65536-0.1%` | 3,036 → 774 (0.25×) | 250 → 861 (3.45×) |
| `low-65536-1%` | 23,598 → 1,235 (0.05×) | 1,597 → 3,178 (1.99×) |
| `low-65536-10%` | 57,686 → 11,312 (0.20×) | 15,433 → 12,488 (0.81×) |
| `low-65536-100%` | 171,340 → 132,520 (0.77×) | 191,165 → 115,290
(0.60×) |
Descending inserts used to move every stored block on each new lowest
block, which is quadratic; they are now up to 19× faster. `from_iter` is
13–40% faster for 10% to 100% of 16,384 and 65,536 bits, but **1.8–3.5×
slower for the sparse inputs** (4 and 16 bits, 0.1% and 1%; e.g. 0.1% of
65,536 bits: 250 → 861 ns). The removed buffered `extend` computed the
whole range first and allocated once; inserting directly grows the
storage several times.
<details>
<summary>Chart: descending inserts before vs after (log scale)</summary>
log10 of the time in ns, mean of the two rounds per side. Exact values
are in the table above.
```mermaid
xychart-beta
title "insert_desc"
x-axis ["1024-16bits", "1024-10%", "16384-16bits", "16384-1%", "16384-10%", "16384-100%", "65536-0.1%", "65536-1%", "65536-10%", "65536-100%"]
y-axis "log10(ns)"
bar "before" [2.22, 2.58, 2.61, 3.4, 3.86, 4.55, 3.48, 4.37, 4.76, 5.23]
bar "after" [2.06, 2.32, 2.36, 2.62, 3.38, 4.48, 2.89, 3.09, 4.05, 5.12]
```
</details>
### Serialization
With the writer and reader Turbopack persists with (`turbo_bincode`:
`TurboBincodeWriter` into a new `TurboBincodeBuffer`,
`TurboBincodeReader`, standard config). Roaring goes through serde, as
`RoaringBitmapWrapper` persists it. Bitset / RoaringBitmap in ns:
| universe | 1 bit | 16 bits | 0.1% | 1% | 10% | 50% | 100% |
|---|---|---|---|---|---|---|---|
| 1,024: serialize | 33 / 102 | 83 / 184 | - | - | 62 / 347 | 62 / 904 |
63 / 1,467 |
| 16,384: serialize | 41 / 102 | 394 / 177 | - | 704 / 404 | 286 / 2,020
| 143 / 1,501 | 147 / 1,529 |
| 65,536: serialize | 42 / 102 | 1,316 / 164 | 1,514 / 255 | 2,726 /
1,074 | 889 / 1,435 | 278 / 2,136 | 207 / 1,445 |
| 1,024: deserialize | 57 / 137 | 67 / 153 | - | - | 53 / 208 | 54 / 627
| 53 / 1,060 |
| 16,384: deserialize | 47 / 132 | 143 / 132 | - | 332 / 267 | 91 /
1,579 | 107 / 1,148 | 89 / 1,145 |
| 65,536: deserialize | 48 / 174 | 193 / 172 | 443 / 187 | 1,078 / 833 |
150 / 1,172 | 149 / 1,141 | 153 / 1,140 |
For the bitmap-form inputs, `Bitset` is 1.6–23× faster than Roaring to
serialize and 2.4–20× faster to deserialize. The list-form inputs (4 and
16 bits, 0.1% and 1%) depend on the width of the stored range: on 1,024
bits `Bitset` is 1.7–2.3× faster both ways, but on 16,384 and 65,536
bits it is up to 11× slower to serialize (4 bits in 65,536) and up to
2.4× slower to deserialize (0.1% of 65,536) than Roaring's sorted
arrays. Both directions walk the stored word range, not just the set
bits.
<details>
<summary>Charts: serialization (log scale)</summary>
log10 of the time in ns, `TurboBincodeWriter`/`TurboBincodeReader`.
Points that set fewer than 32 bits are left out. Exact values are in the
table above.
```mermaid
xychart-beta
title "1,024: serialize"
x-axis ["1 bit", "16 bits", "10%", "50%", "100%"]
y-axis "log10(ns)"
line "Bitset" [1.52, 1.92, 1.79, 1.79, 1.8]
line "RoaringBitmap" [2.01, 2.26, 2.54, 2.96, 3.17]
```
```mermaid
xychart-beta
title "16,384: serialize"
x-axis ["1 bit", "16 bits", "1%", "10%", "50%", "100%"]
y-axis "log10(ns)"
line "Bitset" [1.62, 2.6, 2.85, 2.46, 2.16, 2.17]
line "RoaringBitmap" [2.01, 2.25, 2.61, 3.31, 3.18, 3.18]
```
```mermaid
xychart-beta
title "65,536: serialize"
x-axis ["1 bit", "16 bits", "0.1%", "1%", "10%", "50%", "100%"]
y-axis "log10(ns)"
line "Bitset" [1.62, 3.12, 3.18, 3.44, 2.95, 2.44, 2.32]
line "RoaringBitmap" [2.01, 2.21, 2.41, 3.03, 3.16, 3.33, 3.16]
```
```mermaid
xychart-beta
title "1,024: deserialize"
x-axis ["1 bit", "16 bits", "10%", "50%", "100%"]
y-axis "log10(ns)"
line "Bitset" [1.75, 1.83, 1.73, 1.73, 1.72]
line "RoaringBitmap" [2.14, 2.18, 2.32, 2.8, 3.03]
```
```mermaid
xychart-beta
title "16,384: deserialize"
x-axis ["1 bit", "16 bits", "1%", "10%", "50%", "100%"]
y-axis "log10(ns)"
line "Bitset" [1.67, 2.16, 2.52, 1.96, 2.03, 1.95]
line "RoaringBitmap" [2.12, 2.12, 2.43, 3.2, 3.06, 3.06]
```
```mermaid
xychart-beta
title "65,536: deserialize"
x-axis ["1 bit", "16 bits", "0.1%", "1%", "10%", "50%", "100%"]
y-axis "log10(ns)"
line "Bitset" [1.68, 2.29, 2.65, 3.03, 2.18, 2.17, 2.19]
line "RoaringBitmap" [2.24, 2.24, 2.27, 2.92, 3.07, 3.06, 3.06]
```
</details>
Before/after A/B of removing the decoder's batch copy (the bytes are now
read straight from `peek_read`). Both builds differ only in
`decode_list`; same scheme as above:
| input (form) | `deserialize` before → after |
|---|---|
| `low-1024-16bits` (list) | 68 → 60 (0.88×) |
| `low-16384-4bits` (list) | 95 → 86 (0.90×) |
| `low-16384-16bits` (list) | 206 → 174 (0.84×) |
| `low-16384-1%` (list) | 358 → 344 (0.96×) |
| `low-16384-10%` (bitmap) | 91 → 112 (1.24×) |
| `low-65536-16bits` (list) | 219 → 187 (0.85×) |
| `low-65536-0.1%` (list) | 493 → 450 (0.91×) |
| `low-65536-1%` (list) | 1,276 → 1,073 (0.84×) |
| `low-16384-100%` (bitmap) | 82 → 92 (1.13×) |
| `low-65536-100%` (bitmap) | 171 → 144 (0.84×) |
List-form inputs got 4–16% faster. Bitmap-form inputs, whose decoding
code didn't change, moved by up to ±24% in both rounds, so differences
of that size between these two builds are not attributable to the change
(code layout is a likely, unproven cause).
**Historical, not rerun:** before/after A/B of the previous round's
encoder change (`5 · n < 4 · len` instead of exact list sizing with
hand-written varint lengths), measured on the pre-rebase heads
`b149cf8a` → `1c212943` with the previous inputs (densities drawn per
bit, labelled in per mille). The old head got only the then-new
benchmark files and lockfile, so both sides ran the same inputs and
harness. Two interleaved rounds per side, mean in ns:
| input (form) | serialize before → after | deserialize before → after |
task-input hash before → after |
|---|---|---|---|
| `low-1024-16bits` (list) | 118 → 97 (0.82×) | 88 → 95 (1.08×) | 48 →
49 (1.01×) |
| `low-16384-16bits` (list) | 512 → 435 (0.85×) | 176 → 203 (1.15×) |
129 → 142 (1.10×) |
| `low-16384-10` (list) | 939 → 757 (0.81×) | 339 → 370 (1.09×) | 144 →
152 (1.06×) |
| `low-16384-100` (list) | 2,441 → 1,932 (0.79×) | 1,570 → 1,670 (1.06×)
| 165 → 168 (1.02×) |
| `low-65536-1` (list) | 2,027 → 1,788 (0.88×) | 430 → 513 (1.19×) | 425
→ 430 (1.01×) |
| `low-65536-10` (list) | 3,741 → 2,752 (0.74×) | 1,158 → 1,196 (1.03×)
| 447 → 452 (1.01×) |
| `low-65536-100` (bitmap) | 899 → 885 (0.98×) | 165 → 177 (1.07×) | 449
→ 448 (1.00×) |
Both encoders write identical bytes for all 24 inputs (same form and
size). Over all 72 serialization, deserialization and task-input-hash
cases the geometric mean is 0.998×: list serialization is 12–26% faster;
deserialization (decoder unchanged apart from the removed `#[inline]`
hints) is 3–19% slower; task-input hashing (always the bitmap form) is
5–18 ns slower on 9 of 24 inputs, most on dense 1,024-bit sets (45 → 63
ns), and unchanged on the rest.
<details>
<summary>Chart: serialize before vs after (linear)</summary>
Time in µs, mean of the two rounds per side. Exact values are in the
table above.
```mermaid
xychart-beta
title "serialize, list-form inputs"
x-axis ["1024-16bits", "16384-16bits", "16384-10", "16384-100", "65536-1", "65536-10"]
y-axis "µs"
bar "before" [0.118, 0.512, 0.939, 2.44, 2.03, 3.74]
bar "after" [0.0968, 0.435, 0.757, 1.93, 1.79, 2.75]
```
</details>
### Hashing
`hash`: `FxHasher` through `std::hash::Hash` (`RoaringBitmap` doesn't
implement `Hash`). `hash_encode`: like a task input is hashed for the
persistent cache, encoded with `turbo_bincode`'s hash encoder into
`Xxh3Hash64Hasher`. Bitset (Fx) / FixedBitSet (Fx) / Bitset (task-input
hash) in ns:
| universe | 1 bit | 16 bits | 0.1% | 1% | 10% | 50% | 100% |
|---|---|---|---|---|---|---|---|
| 1,024 | 3.9 / 6.9 / 55 | 8.6 / 9.6 / 49 | - | - | 9.5 / 10 / 60 | 9.5
/ 11 / 60 | 9.6 / 9.9 / 61 |
| 16,384 | 3.9 / 46 / 46 | 110 / 171 / 204 | - | 171 / 139 / 191 | 123 /
143 / 169 | 127 / 143 / 167 | 122 / 145 / 171 |
| 65,536 | 4.1 / 40 / 46 | 417 / 535 / 390 | 487 / 682 / 455 | 493 / 604
/ 440 | 802 / 585 / 614 | 499 / 587 / 464 | 503 / 586 / 549 |
<details>
<summary>Charts: hashing (log scale)</summary>
log10 of the time in ns. Exact values are in the table above.
```mermaid
xychart-beta
title "1,024: hash"
x-axis ["1 bit", "16 bits", "10%", "50%", "100%"]
y-axis "log10(ns)"
line "Bitset (FxHasher)" [0.6, 0.93, 0.98, 0.98, 0.98]
line "FixedBitSet (FxHasher)" [0.84, 0.98, 1.0, 1.02, 0.99]
line "Bitset (task-input hash)" [1.74, 1.69, 1.77, 1.77, 1.78]
```
```mermaid
xychart-beta
title "16,384: hash"
x-axis ["1 bit", "16 bits", "1%", "10%", "50%", "100%"]
y-axis "log10(ns)"
line "Bitset (FxHasher)" [0.59, 2.04, 2.23, 2.09, 2.1, 2.09]
line "FixedBitSet (FxHasher)" [1.67, 2.23, 2.14, 2.15, 2.16, 2.16]
line "Bitset (task-input hash)" [1.66, 2.31, 2.28, 2.23, 2.22, 2.23]
```
```mermaid
xychart-beta
title "65,536: hash"
x-axis ["1 bit", "16 bits", "0.1%", "1%", "10%", "50%", "100%"]
y-axis "log10(ns)"
line "Bitset (FxHasher)" [0.61, 2.62, 2.69, 2.69, 2.9, 2.7, 2.7]
line "FixedBitSet (FxHasher)" [1.6, 2.73, 2.83, 2.78, 2.77, 2.77, 2.77]
line "Bitset (task-input hash)" [1.67, 2.59, 2.66, 2.64, 2.79, 2.67, 2.74]
```
</details>
### Size: in memory and serialized
`cargo run --release -p turbo-bitset --example sizes` (prints CSV), on
the benchmark inputs. In memory is the inline struct plus the requested
heap bytes. Serialized is with `turbo_bincode`:
| input | bits | in memory: Bitset / Roaring | serialized: Bitset (form)
/ Roaring |
|---|---|---|---|
| `low-1024-empty` | 0 | 32 / 24 | 1 (empty) / 9 |
| `low-1024-1bit` | 1 | 64 / 192 | 3 (bitmap) / 19 |
| `low-1024-4bits` | 4 | 224 / 192 | 8 (list) / 25 |
| `low-1024-16bits` | 16 | 160 / 216 | 18 (list) / 49 |
| `low-1024-10%` | 102 | 160 / 440 | 131 (bitmap) / 221 |
| `low-1024-50%` | 512 | 160 / 1,208 | 132 (bitmap) / 1,043 |
| `low-1024-100%` | 1,024 | 160 / 2,232 | 132 (bitmap) / 2,067 |
| `low-16384-empty` | 0 | 32 / 24 | 1 (empty) / 9 |
| `low-16384-1bit` | 1 | 64 / 192 | 5 (bitmap) / 19 |
| `low-16384-4bits` | 4 | 1,536 / 192 | 16 (list) / 25 |
| `low-16384-16bits` | 16 | 2,272 / 216 | 42 (list) / 49 |
| `low-16384-1%` | 163 | 2,080 / 696 | 197 (list) / 345 |
| `low-16384-10%` | 1,638 | 2,080 / 4,280 | 2,051 (bitmap) / 3,295 |
| `low-16384-50%` | 8,192 | 2,080 / 8,376 | 2,052 (bitmap) / 8,211 |
| `low-16384-100%` | 16,384 | 2,080 / 8,376 | 2,052 (bitmap) / 8,211 |
| `low-65536-empty` | 0 | 32 / 24 | 1 (empty) / 9 |
| `low-65536-1bit` | 1 | 64 / 192 | 5 (bitmap) / 19 |
| `low-65536-4bits` | 4 | 6,736 / 192 | 16 (list) / 25 |
| `low-65536-16bits` | 16 | 8,832 / 216 | 50 (list) / 49 |
| `low-65536-0.1%` | 65 | 8,224 / 440 | 165 (list) / 147 |
| `low-65536-1%` | 655 | 8,224 / 2,232 | 769 (list) / 1,329 |
| `low-65536-10%` | 6,553 | 8,224 / 8,376 | 8,195 (bitmap) / 8,211 |
| `low-65536-50%` | 32,768 | 8,224 / 8,376 | 8,196 (bitmap) / 8,211 |
| `low-65536-100%` | 65,536 | 8,224 / 8,376 | 8,196 (bitmap) / 8,211 |
The encoding is smaller than Roaring's for 19 of the 21 non-empty
inputs: the exceptions are `low-65536-16bits` (50 vs 49 bytes),
`low-65536-0.1%` (165 vs 147 bytes). Both are sparse sets whose
differences need multi-byte varints while Roaring's arrays use 2 bytes
per item. In memory, `Bitset` follows its stored range, so wide sparse
sets are much larger than in Roaring.
<details>
<summary>Charts: size (log scale)</summary>
log10 of the size in bytes (2 = 100 B, 4 = 10 KB), one pair of charts
per universe; x is the number of bits or the density. Exact values are
in the table above.
```mermaid
xychart-beta
title "In memory, 1,024-bit universe"
x-axis ["1 bit", "4 bits", "16 bits", "10%", "50%", "100%"]
y-axis "log10(bytes)"
line "Bitset" [1.81, 2.35, 2.2, 2.2, 2.2, 2.2]
line "RoaringBitmap" [2.28, 2.28, 2.33, 2.64, 3.08, 3.35]
```
```mermaid
xychart-beta
title "Serialized, 1,024-bit universe"
x-axis ["1 bit", "4 bits", "16 bits", "10%", "50%", "100%"]
y-axis "log10(bytes)"
line "Bitset" [0.48, 0.9, 1.26, 2.12, 2.12, 2.12]
line "RoaringBitmap" [1.28, 1.4, 1.69, 2.34, 3.02, 3.32]
```
```mermaid
xychart-beta
title "In memory, 16,384-bit universe"
x-axis ["1 bit", "4 bits", "16 bits", "1%", "10%", "50%", "100%"]
y-axis "log10(bytes)"
line "Bitset" [1.81, 3.19, 3.36, 3.32, 3.32, 3.32, 3.32]
line "RoaringBitmap" [2.28, 2.28, 2.33, 2.84, 3.63, 3.92, 3.92]
```
```mermaid
xychart-beta
title "Serialized, 16,384-bit universe"
x-axis ["1 bit", "4 bits", "16 bits", "1%", "10%", "50%", "100%"]
y-axis "log10(bytes)"
line "Bitset" [0.7, 1.2, 1.62, 2.29, 3.31, 3.31, 3.31]
line "RoaringBitmap" [1.28, 1.4, 1.69, 2.54, 3.52, 3.91, 3.91]
```
```mermaid
xychart-beta
title "In memory, 65,536-bit universe"
x-axis ["1 bit", "4 bits", "16 bits", "0.1%", "1%", "10%", "50%", "100%"]
y-axis "log10(bytes)"
line "Bitset" [1.81, 3.83, 3.95, 3.92, 3.92, 3.92, 3.92, 3.92]
line "RoaringBitmap" [2.28, 2.28, 2.33, 2.64, 3.35, 3.92, 3.92, 3.92]
```
```mermaid
xychart-beta
title "Serialized, 65,536-bit universe"
x-axis ["1 bit", "4 bits", "16 bits", "0.1%", "1%", "10%", "50%", "100%"]
y-axis "log10(bytes)"
line "Bitset" [0.7, 1.2, 1.7, 2.22, 2.89, 3.91, 3.91, 3.91]
line "RoaringBitmap" [1.28, 1.4, 1.69, 2.17, 3.12, 3.91, 3.91, 3.91]
```
</details>
## Verification
- `cargo test -p turbo-bitset` (38 unit tests, 4 input tests):
- seeded differential tests against `BTreeSet` (sparse and dense), word
and container boundaries, front and back growth (front growth leaving
headroom at the end of the capacity, or limited by block 0; front and
back growth at once; reallocation), descending inserts of 4,096 blocks
with O(log n) relocations and reallocations, the chunked subset check
with a missing bit in every word position, `extend` with ascending,
descending, shuffled, duplicate and length-hiding inputs,
`has_bits_not_in` against `!is_subset` (padded storage and random sets),
and hashing ignoring the stored range;
- `to_roaring` matching inserted bitmaps exactly, including the
serialized bytes;
- round trips with the standard, fixed-int and big-endian configs; exact
bytes against an independent reference encoder of both forms; the form
rule on crafted sets and just below, at and above `5 · n = 4 · len`; the
size bounds against both forced forms on dense, sparse, clustered,
gap-251 and high-bit sets; the fixed-int config always writing the
bitmap form; `BorrowDecode`; malformed inputs (including payload spans
that are zero, past `u32`, too short or too long) and every truncated
prefix; decoder limits (huge spans, 4 bytes claimed per list item);
- the decoder against a straightforward per-item reference decoder on
~16,000 valid and corrupted encodings, with no limit and
64/512/4,096-byte limits: both accept and reject the same inputs, with
the same sets and bytes read;
- the single-write bitmap path writing the same bytes as the per-block
path that big-endian targets use, and the bulk-read byte-order
conversion for both target endiannesses.
- The benchmark inputs (`tests/bench_inputs.rs`): exact counts,
ascending unique in-range bits, deterministic seeds, percentage labels.
- `cargo bench -p turbo-bitset -- --test` checks every benchmark case;
`cargo clippy`, `cargo doc` with `-D warnings`, `cargo fmt --check`.
<!-- NEXT_JS_LLM -->
<!-- fleet b56269ed-2145-4f30-b0fe-dc9762520521 -->
---------
Co-authored-by:
vercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by:
Tobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by:
Vercel Fleet <fleet@vercel.com>
Co-authored-by:
Luke Sandberg <210140+lukesandberg@users.noreply.github.com>
Showing
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
Please
register
or
sign in
to comment