turbo-persistence: don't shrink the shard count before the first bottom merge (#100424)
## What Defer shrinking shard counts of turbo-persistence families until there has been a bottom merge. This reduces the maximum sizes of initial compaction jobs at the cost of slightly more SST files until that happens. The benefit is more parallelism and smaller job sizes for initial bottom runs. ## Why A 'bottom' merge is when we have performed a 'full compaction' of a single shard of a family (aka table). Families are divided into shards to accelerate lookups and reduce the size of compaction jobs. Part of the compaction strategy is to occasionally compress the full shard into a 'bottom' a set of hot and cold files that do not overlap (well hot and cold can overlap, but hots cannot overlap with hots and colds cannot with colds). Before a family has a 'bottom', its shard count can shrink. Commits compute the count too, so after a small first commit (e.g. the first snapshot in dev) the next commits already wrote files spanning all of the family's initial shards. Such files tie every shard into one component, so the first bottom merge of the family was a single job over all its files. e.g. for next-site TaskData went from 8 to 4 shards and TaskMeta from 4 to 1, doubling and quadrupling the job sizes. Now the count only grows until the family has bottom runs. Shrinking afterwards is free: the bottom runs of the former shards are disjoint, so together they are the bottom run of the combined shard and nothing is rewritten. The cost is a less than optimal shard count, and so more, smaller commit files, until the first compaction that is triggered by our normal triggers. The benefit is more parallelism in those initial bottom run jobs Also, an intermediate merge that happens takes all files of a shard without a bottom run now writes it as the shard's bottom run, since it rewrites just as much.
Showing
Please
register
or
sign in
to comment