[turbopack] Make compression in SST files optional (#89309)
Some blocks in our SST files are not compressible. Notably all Index blocks and many Key blocks. * For index blocks the issue is that it is a sequence of hash->block-index pairs. The hashes are high entropy (typically only a few of the uppermost bits will be shared across all hashes) and the indexes should be compressible but based on my measurements of vercel-site this isn't true in practice. So this PR simply doesn't try. * For key blocks the same argument applies, but not quite as dramatically, we are storing high entropy hashes or small keys with similar hashes (and thus distinct values), so we shouldn't generally expect this to be very compressible. * Future work would be to mark certain _families_ as incompressible, but this is deferred for now * Value blocks on the other hand achieve high amounts of compressibility, so we should of course keep trying. This leverages heuristics from industry and avoid writing data as compressed unless it is smaller than 7/8 of the source document. This is the pattern used by LevelDB, RocksDB, and ZFS. When we detect this scenario we set the `uncompressed_size` to 0 so readers know it is uncompressed. (See README updates for a description) On the read side use the `mmap` slice directly to avoid copies, instead we make managing the slice the Operating Systems problem. This is achieved by augmenting the ArcSlice datastructure to be able to store an `Arc<Mmap>` (and renaming it since it is now officially about bytes).
Showing
Please
register
or
sign in
to comment