What is compaction in distributed storage systems?

Compaction is the process of merging and reorganizing storage files to improve read performance.

In systems like Cassandra, HBase, and LSM-tree-based storage engines, writes are appended in multiple SSTable files. Over time, this leads to fragmentation and duplicate records. Compaction merges these files, removes deleted records (tombstones), and improves read efficiency. It is essential for maintaining performance in write-heavy systems.