fjall-rs/lsm-tree

enum_dispatch is the only reason lsm-tree pulls in syn etc. But writing static dispatch manually is kinda annoying...

Summary CompactionStream's MVCC GC branch tests the next-older version (peeked) against gc_seqno_threshold and then drains the whole tail of the user key. When the key's newest version (head) is above the watermark, this drops the version that the oldest open snapshot still needs to read. src/compaction/stream.rs:191 on main (also in 3.1.9 and 3.1.10): } else if peeked.key.seqno < self.gc_seqno_threshold { ... fail_iter!(self.drain_key(&head.key.user_key)); Why it is unsound In fjall, SnapshotTracker::gc() stores lowest_freed_instant = lowest_retained - 1, and that value is passed to compaction as the threshold. So with the oldest open snapshot at instant S, the threshold is W = S - 1. Reads filter item_seqno < instant (range.rs::seqno_filter), so that snapshot sees every version with seqno <= W. The newest version with seqno <= W is live and must survive. Only versions strictly older than it are dead for every reader. Take a key with versions at seqno 60, 5, 3 and a snapshot o

optimize_runs is an important function that needs to be thoroughly tested (more than it currently is), and regressions would be awful. A fuzz test should be pretty easy to implement as the function is generic, so we can easily build test data without creating a real database. As fuzz input, there are generally two operations: insert(K, Seqno) into an in-memory buffer (B-tree), and flush(), which writes the buffer to a faux "table", which holds a key range. After each flush, all tables should run through optimize_runs, and the last written version of any key should always be visible.

I bulk write large blobs (2 MB chunks) into a KV-separated keyspace. During sustained ingest the write buffer keeps climbing, well past 128 MB, and memory usage follows. It only comes back down when ingest pauses long enough for flushes to catch up. I ended up wrapping my ingest paths in a loop that polls write_buffer_size() and sleeps until it drops below a cap before writing more. Why WriteBufferManager is only a counter. Writes increment it and flush completions decrement it, but nothing ever reads it to apply backpressure. Builder::max_write_buffer_size writes its value into the config and no code reads it back, so the documented RocksDB-style cap does nothing. The backpressure that does exist is per keyspace and counts things, not bytes: An active memtable over max_memtable_size (64 MiB default) requests a rotation, but that is a try_send to the worker pool. Admission continues at full speed until a worker picks it up, and the request is silently dropped if the queue is full. Wri

Even though there is no reason to really major compact a FIFO-tree, it should still not screw it up. FIFO compaction puts all tables into L0, but major compaction moves them all into Lmax. So FIFO compaction should probably always operate on Lmax. Would allow to repair a situation like #310