Five P99 CONF Sessions Expose the Storage Trade-offs Behind High-Performance Databases

The tension between read speed, write speed, and storage efficiency defines modern database architecture. At P99 CONF, held October 21–22, speakers from leading infrastructure companies walked through how they navigate these unavoidable trade-offs. The virtual conference focuses on high-performance, low-latency systems; more than 60 sessions were recorded and remain available on demand at p99conf.io after registration.

Database designers face what researchers call the RUM conjecture: read amplification (data fetched per result), write amplification (data written per logical change), and space amplification (storage overhead relative to live data). Optimizing one dimension typically worsens another, depending on workload and configuration. The five talks below illustrate how different teams make those choices.

LSM Trees Adapt to Object Storage

Almog Gavra, co-founder at Responsive (now Databricks), framed the fundamental tension: a log structure favors fast writes but burdens reads; a sorted array reverses that burden. "I think about LSM trees as a mechanism for tuning between these two extremes," Gavra said.

Log-structured merge trees were originally designed for local disks, but their immutable file structure also suits object storage services like Amazon S3. Gavra highlighted S3's cost, durability, and availability advantages, then enumerated the penalties: objects cannot be edited in place, each GET and PUT incurs a charge, and cold reads suffer latency. A local NVMe cache miss takes roughly 150 microseconds; the same operation on S3 Standard takes 20–30 milliseconds—a two-order-of-magnitude difference. LSM batching of writes into immutable files, combined with local caching, helps keep expensive object-store reads off the critical path.

Gavra's SlateDB, an open-source LSM library built for object storage, exposes a key-value interface over S3, GCS, and R2. The tool lets developers build data systems without implementing their own LSM logic.

ScyllaDB's Read-Optimized Engine Trades Memory for Latency

Michael Litvak, senior software engineer, and Felipe Cardeneti Mendes, technical director, both at ScyllaDB, presented Logstor, an optional storage engine for read-heavy key-value workloads. Unlike a replacement for ScyllaDB's existing LSM engine, Logstor maintains an in-memory hash index and record locations on flash for each key. A read either returns a cached value or retrieves the record with a single disk I/O, bypassing the LSM search through SSTables, Bloom Filters, and indexed structures. Both engines can coexist in the same cluster using ScyllaDB's standard drivers and consistency guarantees.

In preliminary tests with 2 KB values, the read-dominant workload sustained 1.1 million reads per second with p99.9 latency under one millisecond and p99.99 latency around 1.5 milliseconds. An 80% read, 20% write workload showed similar throughput and sub-millisecond p99.9 latency. (Earlier LSM examples in the same talk used 1 KB values, so direct comparison is limited.)

The speed comes at a cost: Logstor requires a larger RAM index and heavier write work. Even small updates write the full new value to a log, leaving the old record as garbage; segment compaction reclaims the space later. Mendes positioned the engine for bounded key sets, frequent overwrites, and point lookups where extra memory is acceptable. "Optimized for time at the sacrifice of space," Mendes called the trade-off.

Valkey Breaks the Single-Thread Ceiling with I/O Threads

Madelyn Olson, principal software engineer at AWS, addressed a different bottleneck: the main execution thread. Amazon ElastiCache customers reported sudden p50 latency spikes; Olson's testing revealed that Valkey's main thread had hit its limit.

In an unpipelined GET benchmark with roughly three million keys and 650 clients, p50 latency climbed sharply near 150,000 requests per second. Valkey had I/O threads, but they remained idle while the main thread handled command execution. The team deployed lockless queues to keep network I/O and command parsing active on those threads while command execution remained single-threaded.

The result: a new ceiling of approximately 1.2 million requests per second in that workload—roughly eight times the earlier threshold. (This figure is specific to the talk's test conditions, not a general throughput guarantee.)

For horizontal scaling, Valkey 9.0 introduced atomic slot migration. The source shard retains ownership while streaming a snapshot and subsequent updates to a target; ownership transfers once the target catches up. In a test with roughly 40 GB across three to four shards, migration ran about 10 times faster without load and about five times faster under heavy GET traffic. If migration fails before handoff, the staged transfer aborts without data loss.

I/O threads can be added or removed as workload patterns shift. Olson's guidance: use horizontal scaling for baseline capacity and vertical scaling for short bursts, keeping command execution single-threaded for now. "Efficiency trumps headline numbers," she said.

Vector Search at Billion Scale Requires Hierarchical Caching

Nathan VanBenschoten, chief architect at turbopuffer, described a search index spanning 100 billion vectors—more than 200 TiB of dense vector data. The system achieves 200 ms p99 latency at over 1,000 queries per second with 92% recall in early results. Object storage serves as the durable backing store, but this high-query-rate workload pins data in local SSD caches.

Hierarchical clustering groups nearby vectors. RaBitQ binary quantization compresses a shadow index to one bit per dimension, allowing the search to narrow candidates before reranking a small subset using full-precision vectors. Uncompressed search was bandwidth-limited by vector movement from storage into memory; quantization shifts work to higher cache tiers but increases arithmetic intensity, making the system CPU-bound.

"Often when you push down one bottleneck, a new one emerges," VanBenschoten noted, pointing to more efficient vector instructions as the next optimization target.

To fit an index this large on SSDs, turbopuffer distributes it across machines. VanBenschoten credits a separated compute-and-storage design with favoring fewer, larger shards: a replacement machine can point to object storage and hydrate its local cache instead of receiving a full shard from another node. Queries fan out to shards, and a coordinator merges their top results.

TigerBeetle Builds a Custom Object Storage Client for Single-Core Transactions

Georg Kreuzmayr, software engineer at TigerBeetle, introduced "diagonal scaling": keeping hot data on local NVMe while extending the storage hierarchy to shared object storage for colder data. A failed machine does not need to reconstruct data already in object storage. Recovery time matters: 128 TB over a 100 Gbps network takes roughly three hours at line rate; if recovery exceeds the interval between failures, a cluster risks entering a failure spiral.

This design demands an object storage client compatible with TigerBeetle's single-core transaction processing. Official SDKs proved incompatible: the AWS C++ SDK's blocking getaddrinfo call would stall the transaction core. SDK thread pools, runtime allocation, and dependencies also conflicted with the project's goals.

TigerBeetle built its own multi-cloud client supporting S3, Azure, and GCP, testable with simulated I/O and injected faults. Written in Zig, the client includes custom DNS, TLS, and HTTP components, uses io_uring and zero-copy sends, and allocates working memory at startup. Kreuzmayr reports the implementation spans under 40,000 lines of code with no dependencies beyond the toolchain. These choices align with TigerStyle's three principles: safety, performance, and developer experience.

In an S3 PutObject test—8 MB requests over plain HTTP without checksums on an AWS c7gn.metal-24xlarge instance—Kreuzmayr reports uploads roughly 20 times faster than official SDKs and five times faster than a research implementation. For GetObject, throughput is about three times the SDK rate and roughly 25% higher than the research implementation. He attributes the smaller download gain to the cloud TCP receive path. These results apply to the stated test setup; the conference summary describes the broader comparison as 4–10 times faster than official GCP and AWS SDKs.

"Code stops being technical debt and ends up being an asset," Kreuzmayr said, reflecting on the upfront work required to build the client versus the payoff in testability and tuning.

The Broader Pattern

Across these five presentations, engineers identified distinct bottlenecks in storage, search, networking, and scaling. Each design optimizes for specific workloads while introducing new costs or complexity elsewhere. Over 200 additional P99 CONF talks are available on demand at p99conf.io; registration is required. P99 CONF is a ScyllaDB community event.