Unverified Commit 32893c7c authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

Turbopack: create smaller initial files to reduce memory usage (#78778)

### What?

Large SST files lead to bigger memory usage as it accumulate entries for the SST file in memory. Creating smaller SST files flushes them more frequent and reduces in flight memory. Compaction will merge these smaller files to larger SST files during compaction.
parent 576e6978
......@@ -6,13 +6,13 @@ pub const MAX_MEDIUM_VALUE_SIZE: usize = 64 * 1024 * 1024;
pub const MAX_SMALL_VALUE_SIZE: usize = 64 * 1024 - 1;
/// Maximum number of entries per SST file
pub const MAX_ENTRIES_PER_INITIAL_FILE: usize = 1024 * 1024;
pub const MAX_ENTRIES_PER_INITIAL_FILE: usize = 256 * 1024;
/// Maximum number of entries per SST file
pub const MAX_ENTRIES_PER_COMPACTED_FILE: usize = 1024 * 1024;
/// Finish file when total amount of data exceeds this
pub const DATA_THRESHOLD_PER_INITIAL_FILE: usize = 256 * 1024 * 1024;
pub const DATA_THRESHOLD_PER_INITIAL_FILE: usize = 64 * 1024 * 1024;
/// Finish file when total amount of data exceeds this
pub const DATA_THRESHOLD_PER_COMPACTED_FILE: usize = 256 * 1024 * 1024;
......
......@@ -13,8 +13,8 @@ use crate::database::{
write_batch::{BaseWriteBatch, ConcurrentWriteBatch, WriteBatch, WriteBuffer},
};
const COMPACT_MAX_COVERAGE: f32 = 10.0;
const COMPACT_MAX_MERGE_SEQUENCE: usize = 16;
const COMPACT_MAX_COVERAGE: f32 = 20.0;
const COMPACT_MAX_MERGE_SEQUENCE: usize = 64;
const COMPACT_MAX_MERGE_SIZE: usize = 512 * 1024 * 1024; // 512 MiB
pub struct TurboKeyValueDatabase {
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment