Unverified Commit ea56922a authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

turbo-persistence: add CRC32 block checksums (#90754)

### What?

Add a 4-byte CRC32 checksum to every block in turbo-persistence SST
files.

### Why?

Detect on-disk cache corruption early with a clear error message, rather
than silently returning wrong data or hitting confusing LZ4
decompression failures.

### How?

**New on-disk block format:**
```
[4B uncompressed_length] [4B CRC32 checksum] [block data]
```
(Previously the checksum field did not exist — header was just the 4B
uncompressed_length.)

- The checksum is computed on the **uncompressed** block data using
`crc32fast` (hardware-accelerated CRC32C on modern CPUs).
- On write, the checksum is computed before compression and stored in
the block header.
- On read, the checksum is verified after decompression (or directly for
uncompressed blocks). A mismatch returns an `anyhow` error with a
message like: `Cache corruption detected: checksum mismatch in block N
of SST file seq:M`.
- During compaction, medium values are copied as raw compressed blocks
without decompression — the checksum is carried through the
`MediumRaw`/`LazyLookupValue::Medium` path unchanged.
- A `BLOCK_HEADER_SIZE` constant (= 8) replaces magic numbers across the
write, read, and inspect code.

**Files changed:**
- `compression.rs` — `checksum_block()` helper wrapping
`crc32fast::hash()`
- `static_sorted_file_builder.rs` — write path
(`write_raw_block_to_file`, `write_block_to_file`, `close()` index
block), `EntryValue::MediumRaw` carries checksum
- `static_sorted_file.rs` — read path (`get_raw_block_slice`,
`read_block` with `verify_checksum()` helper)
- `lookup_entry.rs` — `LazyLookupValue::Medium` carries checksum for
compaction
- `bin/sst_inspect.rs` — updated block parsing offsets
- `README.md` — documented new format

**Tests:**
- All 48 existing tests pass (checksums are transparent to correct data)
- 2 new corruption tests: one corrupts a compressed block's checksum,
the other corrupts an uncompressed block's data — both verify the error
is returned
parent 82db9449
......@@ -9431,6 +9431,7 @@ dependencies = [
"bitfield",
"byteorder",
"codspeed-criterion-compat",
"crc32fast",
"dashmap 6.1.0",
"either",
"jiff",
......
......@@ -380,6 +380,7 @@ clap = { version = "4.5.2", features = ["derive"] }
concurrent-queue = "2.5.0"
console-subscriber = "0.4.1"
const_format = "0.2.30"
crc32fast = "1.4.2"
criterion = { package = "codspeed-criterion-compat", version = "4.3.0" }
crossbeam-channel = "0.5.8"
dashmap = "6.1.0"
......
......@@ -17,6 +17,7 @@ anyhow = { workspace = true }
bincode = { workspace = true }
bitfield = { workspace = true }
byteorder = { workspace = true }
crc32fast = { workspace = true }
dashmap = { workspace = true}
either = { workspace = true }
jiff = "0.2.10"
......
......@@ -41,7 +41,7 @@ Therefore there are these value types:
| Compression unit | key block | shared value block (≥ 8 kB) | dedicated value block | separate file |
| Compression unit size | ≤ 16 kB | 8 kB .. 12 kB | 4 kB .. 64 MB | > 64 MB |
| Access cost | no extra overhead | decompress shared block (~8 kB) | decompress value size | open separate file, decompress value size |
| Storage overhead | 0 | 8 B in key block + 8 B per ~8 kB in block table | 2 B in key block + 8 B in block table | 4 B in key block + 4 B in blob header |
| Storage overhead | 0 | 8 B in key block + 8 B per ~8 kB in block table | 2 B in key block + 8 B in block table | 4 B in key block + 8 B in blob header |
| Compaction | re-compressed | re-compressed | copied compressed | pointer copied |
Small value blocks are emitted once they accumulate at least `MIN_SMALL_VALUE_BLOCK_SIZE` (8 kB) of data. This means actual block sizes range from 8 kB up to 8 kB + `MAX_SMALL_VALUE_SIZE` (4 kB) = 12 kB. This provides a good balance between compression efficiency (blocks ≥ 4 kB compress well with LZ4) and access cost (only ~8–12 kB needs to be decompressed per lookup).
......@@ -78,6 +78,7 @@ The SST file contains only data without any header.
- foreach block
- 4 bytes block header (uncompressed length or sentinel)
- 4 bytes checksum (CRC32 of uncompressed block data)
- block data (compressed or uncompressed)
- foreach block
- 4 bytes end of block offset relative to start of all blocks
......@@ -89,6 +90,10 @@ Blocks can be stored compressed (LZ4) or uncompressed. The 4-byte header disting
- **Header > 0**: Block is LZ4 compressed. Header value is the uncompressed length.
- **Header = 0**: Block is stored uncompressed. Actual length is derived from block offsets.
#### Block Checksum
Each block stores a 4-byte CRC32 checksum (big-endian) computed on the **on-disk** block data (i.e. after compression). On read, the checksum is verified **before** decompression so that on-disk damage is caught before passing data to LZ4. A checksum mismatch returns an error indicating that the cached data is damaged.
#### Index Block
- 1 byte block type (0: index block)
......@@ -162,7 +167,13 @@ Future:
### Blob file
The plain value compressed with dynamic compression.
The plain value compressed with dynamic compression. Each blob file has an 8-byte header:
- 4 bytes: uncompressed length (u32 big-endian)
- 4 bytes: CRC32 checksum of the compressed data (u32 big-endian)
- remaining bytes: LZ4-compressed value data
The checksum is verified on the compressed data **before** decompression when the blob is read.
## Reading
......
......@@ -21,13 +21,13 @@ use anyhow::{Context, Result, bail};
use byteorder::{BE, ReadBytesExt};
use lzzzz::lz4::decompress;
use memmap2::Mmap;
use turbo_persistence::meta_file::MetaFile;
// Import shared constants from the crate
use turbo_persistence::static_sorted_file::{
BLOCK_TYPE_INDEX, BLOCK_TYPE_KEY_NO_HASH, BLOCK_TYPE_KEY_WITH_HASH, KEY_BLOCK_ENTRY_TYPE_BLOB,
KEY_BLOCK_ENTRY_TYPE_DELETED, KEY_BLOCK_ENTRY_TYPE_INLINE_MIN, KEY_BLOCK_ENTRY_TYPE_MEDIUM,
KEY_BLOCK_ENTRY_TYPE_SMALL,
};
use turbo_persistence::{BLOCK_HEADER_SIZE, checksum_block, meta_file::MetaFile};
/// Block size information
#[derive(Default, Debug, Clone)]
......@@ -252,9 +252,10 @@ fn analyze_sst_file(db_path: &Path, info: &SstInfo) -> Result<SstStats> {
};
let block_end = blocks_start + (&mmap[offset..offset + 4]).read_u32::<BE>()? as usize;
// Read uncompressed length and compressed data
// Read block header (uncompressed length + checksum) and block data
let uncompressed_length = (&mmap[block_start..block_start + 4]).read_u32::<BE>()?;
let compressed_data = &mmap[block_start + 4..block_end];
let expected_checksum = (&mmap[block_start + 4..block_start + 8]).read_u32::<BE>()?;
let compressed_data = &mmap[block_start + BLOCK_HEADER_SIZE..block_end];
let compressed_size = compressed_data.len() as u64;
// Determine if block was compressed (uncompressed_length > 0 means it was compressed)
......@@ -266,6 +267,19 @@ fn analyze_sst_file(db_path: &Path, info: &SstInfo) -> Result<SstStats> {
uncompressed_length as u64
};
// Verify checksum on the raw on-disk data before decompression.
let actual_checksum = checksum_block(compressed_data);
if actual_checksum != expected_checksum {
bail!(
"Cache corruption detected: checksum mismatch in block {} of {:08}.sst (expected \
{:08x}, got {:08x})",
block_index,
info.sequence_number,
expected_checksum,
actual_checksum
);
}
let decompressed = match decompress_block(compressed_data, uncompressed_length) {
Ok(data) => data,
Err(e) => {
......
......@@ -29,6 +29,11 @@ pub fn decompress_into_arc(uncompressed_length: u32, block: &[u8]) -> Result<Arc
Ok(buffer)
}
/// Computes a CRC32 checksum of a byte slice.
pub fn checksum_block(data: &[u8]) -> u32 {
crc32fast::hash(data)
}
#[tracing::instrument(level = "trace", skip_all)]
pub fn compress_into_buffer(block: &[u8], buffer: &mut Vec<u8>) -> Result<()> {
lz4::compress_to_vec(block, buffer, lz4::ACC_LEVEL_DEFAULT).context("Compression failed")?;
......
......@@ -24,7 +24,7 @@ use crate::{
DbConfig, FamilyKind, QueryKey,
arc_bytes::ArcBytes,
compaction::selector::{Compactable, get_merge_segments},
compression::decompress_into_arc,
compression::{checksum_block, decompress_into_arc},
constants::{
DATA_THRESHOLD_PER_COMPACTED_FILE, KEY_BLOCK_AVG_SIZE, KEY_BLOCK_CACHE_SIZE,
MAX_ENTRIES_PER_COMPACTED_FILE, VALUE_BLOCK_AVG_SIZE, VALUE_BLOCK_CACHE_SIZE,
......@@ -411,10 +411,23 @@ impl<S: ParallelScheduler, const FAMILIES: usize> TurboPersistence<S, FAMILIES>
mmap.advise(memmap2::Advice::DontFork)?;
#[cfg(target_os = "linux")]
mmap.advise(memmap2::Advice::Unmergeable)?;
let mut compressed = &mmap[..];
let uncompressed_length = compressed.read_u32::<BE>()?;
let mut reader = &mmap[..];
let uncompressed_length = reader.read_u32::<BE>()?;
let expected_checksum = reader.read_u32::<BE>()?;
let buffer = decompress_into_arc(uncompressed_length, compressed)?;
// Verify checksum on the compressed on-disk data before decompression.
let actual_checksum = checksum_block(reader);
if actual_checksum != expected_checksum {
bail!(
"Cache corruption detected: checksum mismatch in blob file {:08}.blob (expected \
{:08x}, got {:08x})",
seq,
expected_checksum,
actual_checksum
);
}
let buffer = decompress_into_arc(uncompressed_length, reader)?;
Ok(ArcBytes::from(buffer))
}
......
......@@ -26,6 +26,7 @@ mod write_batch;
mod tests;
pub use arc_bytes::ArcBytes;
pub use compression::checksum_block;
pub use db::{CompactConfig, MetaFileEntryInfo, MetaFileInfo, TurboPersistence};
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
......@@ -73,7 +74,7 @@ pub use static_sorted_file::{
BlockCache, BlockWeighter, SstLookupResult, StaticSortedFile, StaticSortedFileMetaData,
};
pub use static_sorted_file_builder::{
Entry, EntryValue, StreamingSstWriter, write_static_stored_file,
BLOCK_HEADER_SIZE, Entry, EntryValue, StreamingSstWriter, write_static_stored_file,
};
pub use value_buf::ValueBuffer;
pub use write_batch::WriteBatch;
......@@ -24,6 +24,7 @@ pub enum LazyLookupValue {
/// A medium sized value that is still compressed.
Medium {
uncompressed_size: u32,
checksum: u32,
block: ArcBytes,
},
}
......@@ -38,6 +39,7 @@ impl LazyLookupValue {
LazyLookupValue::Medium {
uncompressed_size,
block,
..
} => {
if *uncompressed_size == 0 {
block.len()
......@@ -114,9 +116,11 @@ impl Entry for LookupEntry {
},
LazyLookupValue::Medium {
uncompressed_size,
checksum,
block,
} => EntryValue::MediumRaw {
uncompressed_size: *uncompressed_size,
checksum: *checksum,
block: block.as_ref(),
},
}
......
......@@ -16,9 +16,10 @@ use smallvec::SmallVec;
use crate::{
QueryKey,
arc_bytes::ArcBytes,
compression::decompress_into_arc,
compression::{checksum_block, decompress_into_arc},
constants::MAX_INLINE_VALUE_SIZE,
lookup_entry::{LazyLookupValue, LookupEntry, LookupValue},
static_sorted_file_builder::BLOCK_HEADER_SIZE,
};
/// The block header for an index block.
......@@ -441,10 +442,33 @@ impl StaticSortedFile {
self.read_block(block_index)
}
/// Reads a block from the file.
/// Verifies the CRC32 checksum of on-disk block data. Returns an error on mismatch.
fn verify_checksum(&self, data: &[u8], expected: u32, block_index: u16) -> Result<()> {
let actual = checksum_block(data);
if actual != expected {
bail!(
"Cache corruption detected: checksum mismatch in block {} of {:08}.sst (expected \
{:08x}, got {:08x})",
block_index,
self.meta.sequence_number,
expected,
actual
);
}
Ok(())
}
/// Reads a block from the file, decompressing if needed, and verifies its checksum.
///
/// The checksum is verified on the raw on-disk data **before** decompression, so
/// corruption is caught before passing data to LZ4.
#[tracing::instrument(level = "info", name = "reading database block", skip_all)]
fn read_block(&self, block_index: u16) -> Result<ArcBytes> {
let (uncompressed_length, block) = self.get_raw_block_slice(block_index)?;
let (uncompressed_length, expected_checksum, block) =
self.get_raw_block_slice(block_index)?;
// Verify checksum on the raw on-disk data before decompression.
self.verify_checksum(block, expected_checksum, block_index)?;
// 0 means the block was not compressed, return the mmap-backed ArcBytes directly
if uncompressed_length == 0 {
......@@ -469,9 +493,13 @@ impl StaticSortedFile {
/// Returns `(uncompressed_length, block_data)` as an owned `ArcBytes` backed by
/// the mmap. Only use this when the block data needs to outlive the current borrow
/// (e.g. medium values stored in `LookupEntry`).
fn get_raw_block(&self, block_index: u16) -> Result<(u32, ArcBytes)> {
let (uncompressed_length, block) = self.get_raw_block_slice(block_index)?;
Ok((uncompressed_length, self.mmap_slice_to_arc_bytes(block)))
fn get_raw_block(&self, block_index: u16) -> Result<(u32, u32, ArcBytes)> {
let (uncompressed_length, checksum, block) = self.get_raw_block_slice(block_index)?;
Ok((
uncompressed_length,
checksum,
self.mmap_slice_to_arc_bytes(block),
))
}
/// Promotes a mmap subslice to an owned `ArcBytes`. This clones the `Arc<Mmap>`.
......@@ -482,7 +510,7 @@ impl StaticSortedFile {
/// Gets the raw block slice directly from the memory mapped file, without
/// cloning the `Arc<Mmap>`. The returned slice borrows from the mmap.
fn get_raw_block_slice(&self, block_index: u16) -> Result<(u32, &[u8])> {
fn get_raw_block_slice(&self, block_index: u16) -> Result<(u32, u32, &[u8])> {
#[cfg(feature = "strict_checks")]
if block_index >= self.meta.block_count {
bail!(
......@@ -526,8 +554,9 @@ impl StaticSortedFile {
}
let uncompressed_length =
u32::from_be_bytes(self.mmap[block_start..block_start + 4].try_into()?);
let block = &self.mmap[block_start + 4..block_end];
Ok((uncompressed_length, block))
let checksum = u32::from_be_bytes(self.mmap[block_start + 4..block_start + 8].try_into()?);
let block = &self.mmap[block_start + BLOCK_HEADER_SIZE..block_end];
Ok((uncompressed_length, checksum, block))
}
}
......@@ -626,9 +655,10 @@ impl StaticSortedFileIter {
let value = if ty == KEY_BLOCK_ENTRY_TYPE_MEDIUM {
let mut val = val;
let block = val.read_u16::<BE>()?;
let (uncompressed_size, block) = self.this.get_raw_block(block)?;
let (uncompressed_size, checksum, block) = self.this.get_raw_block(block)?;
LazyLookupValue::Medium {
uncompressed_size,
checksum,
block,
}
} else {
......
......@@ -11,7 +11,7 @@ use byteorder::{BE, ByteOrder, WriteBytesExt};
use turbo_bincode::turbo_bincode_encode;
use crate::{
compression::compress_into_buffer,
compression::{checksum_block, compress_into_buffer},
constants::{MAX_INLINE_VALUE_SIZE, MAX_SMALL_VALUE_SIZE, MIN_SMALL_VALUE_BLOCK_SIZE},
meta_file::{AmqfBincodeWrapper, MetaEntryFlags},
static_sorted_file::{
......@@ -21,6 +21,9 @@ use crate::{
},
};
/// Size of the per-block header on disk: 4 bytes uncompressed_size + 4 bytes CRC32 checksum.
pub const BLOCK_HEADER_SIZE: usize = 8;
/// The maximum number of entries that should go into a single key block
const MAX_KEY_BLOCK_ENTRIES: usize = MAX_KEY_BLOCK_SIZE / KEY_BLOCK_ENTRY_META_OVERHEAD;
/// The maximum bytes that should go into a single key block
......@@ -153,6 +156,8 @@ pub enum EntryValue<'l> {
/// The uncompressed size of the block data. `0` means the block is stored uncompressed
/// (and thus the size is the `len` of the block)
uncompressed_size: u32,
/// CRC32 checksum of the on-disk block data (after compression).
checksum: u32,
block: &'l [u8],
},
/// Large-sized value. They are stored in a blob file.
......@@ -210,6 +215,7 @@ fn write_raw_block_to_file(
file: &mut BufWriter<File>,
block_offsets: &mut Vec<u32>,
uncompressed_size: u32,
checksum: u32,
block: &[u8],
) -> Result<u16> {
let block_index: u16 = block_offsets
......@@ -217,7 +223,7 @@ fn write_raw_block_to_file(
.try_into()
.expect("Block index overflow");
let len: u32 = (block.len() + 4).try_into().unwrap();
let len: u32 = (block.len() + BLOCK_HEADER_SIZE).try_into().unwrap();
let offset = block_offsets
.last()
.copied()
......@@ -228,6 +234,8 @@ fn write_raw_block_to_file(
file.write_u32::<BE>(uncompressed_size)
.context("Failed to write uncompressed size")?;
file.write_u32::<BE>(checksum)
.context("Failed to write checksum")?;
file.write_all(block)
.context("Failed to write block data")?;
Ok(block_index)
......@@ -253,7 +261,16 @@ fn write_block_to_file(
(0, block)
};
let result = write_raw_block_to_file(file, block_offsets, uncompressed_size, data_to_write);
// Checksum is computed on the on-disk data (after compression).
let checksum = checksum_block(data_to_write);
let result = write_raw_block_to_file(
file,
block_offsets,
uncompressed_size,
checksum,
data_to_write,
);
compress_buffer.clear();
result
}
......@@ -500,6 +517,7 @@ impl<E: Entry> StreamingSstWriter<E> {
}
EntryValue::MediumRaw {
uncompressed_size,
checksum,
block,
} => {
// Note: tracks compressed block size (not uncompressed) unlike EntryValue::Medium.
......@@ -509,6 +527,7 @@ impl<E: Entry> StreamingSstWriter<E> {
self.file.as_mut().unwrap(),
&mut self.block_offsets,
uncompressed_size,
checksum,
block,
)
.context("Failed to write compressed value block")?;
......@@ -761,33 +780,30 @@ impl<E: Entry> StreamingSstWriter<E> {
let mut file = self.file.take().unwrap();
// Write index block directly to file (index blocks are never compressed).
// Write index block (never compressed). Buffer into a Vec first so we can
// compute the checksum, then write via the standard block helper.
let index_entry_count: u16 = (self.key_block_boundaries.len() - 1)
.try_into()
.expect("Index entries count overflow");
let index_block_size: u32 = (INDEX_BLOCK_HEADER_SIZE
+ index_entry_count as usize * INDEX_BLOCK_ENTRY_SIZE)
.try_into()
.unwrap();
// Register block offset (uncompressed_size = 0 since we store raw).
let index_block_size: usize =
INDEX_BLOCK_HEADER_SIZE + index_entry_count as usize * INDEX_BLOCK_ENTRY_SIZE;
let mut index_buf = Vec::with_capacity(index_block_size);
{
let block_len = index_block_size + 4; // +4 for the uncompressed_size header
let offset = self
.block_offsets
.last()
.copied()
.unwrap_or_default()
.checked_add(block_len)
.expect("Block offset overflow");
self.block_offsets.push(offset);
}
file.write_u32::<BE>(0)
.context("Failed to write index block header")?;
let first_block = self.key_block_boundaries[0].1;
let mut index_block = IndexBlockBuilder::new(&mut file, first_block);
for &(hash, block) in &self.key_block_boundaries[1..] {
index_block.put(hash, block);
let first_block = self.key_block_boundaries[0].1;
let mut index_block = IndexBlockBuilder::new(&mut index_buf, first_block);
for &(hash, block) in &self.key_block_boundaries[1..] {
index_block.put(hash, block);
}
}
let index_checksum = checksum_block(&index_buf);
write_raw_block_to_file(
&mut file,
&mut self.block_offsets,
0,
index_checksum,
&index_buf,
)
.context("Failed to write index block")?;
// Write block offset table
for offset in &self.block_offsets {
......@@ -1155,6 +1171,7 @@ mod tests {
TestValueKind::MediumRaw(v) => EntryValue::MediumRaw {
// uncompressed_size = 0 means the block is stored as-is (no compression).
uncompressed_size: 0,
checksum: checksum_block(v),
block: v,
},
TestValueKind::Blob(id) => EntryValue::Large { blob: *id },
......@@ -1614,4 +1631,70 @@ mod tests {
assert_lookup(&sst, &entries[0], &kc, &vc)?;
Ok(())
}
/// Flip a single byte in an SST file at the given position.
fn corrupt_sst_byte(dir: &Path, seq: u32, pos: u64) {
use std::io::{Seek, SeekFrom, Write as _};
let sst_path = dir.join(format!("{seq:08}.sst"));
let file_bytes = std::fs::read(&sst_path).unwrap();
let original = file_bytes[pos as usize];
let mut file = std::fs::OpenOptions::new()
.write(true)
.open(&sst_path)
.unwrap();
file.seek(SeekFrom::Start(pos)).unwrap();
file.write_all(&[original ^ 0xFF]).unwrap();
file.sync_all().unwrap();
}
/// Assert that looking up the first entry in a corrupted SST returns a corruption error.
fn assert_corruption_detected(
dir: &Path,
seq: u32,
meta: &StaticSortedFileBuilderMeta<'_>,
entries: &[TestEntry],
) {
let sst = open_sst(dir, seq, meta).unwrap();
let kc = make_cache();
let vc = make_cache();
match sst.lookup(entries[0].hash, &entries[0].key, &kc, &vc) {
Err(err) => {
let msg = format!("{err}");
assert!(
msg.contains("corruption"),
"Expected corruption error, got: {msg}"
);
}
Ok(_) => panic!("Expected checksum error, but lookup succeeded"),
}
}
#[test]
fn checksum_detects_corrupted_compressed_block() {
let dir = tempfile::tempdir().unwrap();
// Medium value is large enough to get its own value block, which will be compressed
let value = vec![0xCD; 8192];
let entries = vec![TestEntry::medium(b"mkey", &value)];
let meta = write_sst(dir.path(), 1, &entries, MetaEntryFlags::default()).unwrap();
// Corrupt the stored checksum of the first block (bytes 4..8).
// This guarantees a mismatch regardless of whether LZ4 decompression succeeds.
corrupt_sst_byte(dir.path(), 1, 4);
assert_corruption_detected(dir.path(), 1, &meta, &entries);
}
#[test]
fn checksum_detects_corrupted_uncompressed_block() {
let dir = tempfile::tempdir().unwrap();
// Single inline entry - the key block will be small and likely stored uncompressed
let entries = vec![TestEntry::inline(b"key1", b"val1")];
let meta = write_sst(dir.path(), 1, &entries, MetaEntryFlags::default()).unwrap();
// Corrupt a byte in the first block's data (after the 8-byte header)
corrupt_sst_byte(dir.path(), 1, BLOCK_HEADER_SIZE as u64 + 1);
assert_corruption_detected(dir.path(), 1, &meta, &entries);
}
}
......@@ -18,7 +18,7 @@ use crate::{
FamilyConfig, ValueBuffer,
collector::Collector,
collector_entry::CollectorEntry,
compression::compress_into_buffer,
compression::{checksum_block, compress_into_buffer},
constants::{MAX_MEDIUM_VALUE_SIZE, THREAD_LOCAL_SIZE_SHIFT},
key::StoreKey,
meta_file::MetaEntryFlags,
......@@ -433,11 +433,15 @@ impl<K: StoreKey + Send + Sync, S: ParallelScheduler, const FAMILIES: usize>
#[tracing::instrument(level = "trace", skip(self, value), fields(value_len = value.len()))]
fn create_blob(&self, value: &[u8]) -> Result<(u32, File)> {
let seq = self.current_sequence_number.fetch_add(1, Ordering::SeqCst) + 1;
let mut buffer = Vec::new();
buffer.write_u32::<BE>(value.len() as u32)?;
compress_into_buffer(value, &mut buffer)
let mut compressed = Vec::new();
compress_into_buffer(value, &mut compressed)
.context("Compression of value for blob file failed")?;
let mut buffer = Vec::with_capacity(8 + compressed.len());
buffer.write_u32::<BE>(value.len() as u32)?;
buffer.write_u32::<BE>(checksum_block(&compressed))?;
buffer.extend_from_slice(&compressed);
let file = self.db_path.join(format!("{seq:08}.blob"));
let mut file = File::create(&file).context("Unable to create blob file")?;
file.write_all(&buffer)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment