Managing vast quantities of small data presents a significant operational challenge at Red Canary. The company's Linux Endpoint Detection and Response sensors operate across hundreds of thousands of endpoints—virtual machines, laptops, and EC2 instances—continuously producing small telemetry payloads that capture file changes, process launches, network connections, and health status information. Uploading each individual payload to S3, where Amazon charges per PUT operation, proved prohibitively expensive as the deployment scaled.
Combining smaller payloads into larger files before pushing them to S3 reduced costs, but introduced new complications: file-size constraints, slower data visibility, and difficulty isolating specific endpoint records for troubleshooting. After testing alternatives, Red Canary found an unconventional solution: storing high volumes of small data in ScyllaDB, a write-optimized database designed for this exact workload.
Andres Medina, a staff software engineer on the LinuxEDR team, presented Red Canary's migration journey at the Monster Scale Summit, detailing the progression from S3 and Postgres through AWS Firehose to ScyllaDB, along with the technical lessons the team encountered.
Addressing the health check pipeline
Red Canary's team chose to begin their optimization efforts with health check data, treating it as a lower-risk testing ground compared to the core telemetry pipeline.

Problem 1: cost
The financial burden of S3 operations dominated the team's concerns. Medina explained, "S3 charges a flat fee per S3 PUT, which occurs for every payload upload. So if we have hundreds of thousands of endpoints, we're being charged for hundreds of thousands of S3 PUTs."

A buffering strategy using AWS Firehose and Kinesis seemed promising—combining small data streams into larger files and writing to S3 only after reaching a defined size threshold. While this did lower costs, it created unforeseen problems:

- Firehose and Kinesis enforce a strict one-megabyte-per-shard limit, preventing larger files from being processed.
- Data visibility suffered. Previously, health data reached S3 immediately; now the team had to wait for the threshold to be met.
- Locating individual endpoint records became cumbersome when many endpoint IDs were combined in large files.
Problem 2: database scalability
Postgres, which stored endpoint metadata, buckled under the load. Medina noted, "Sometimes a large customer might restart a chunk of their fleet. That means that many endpoints will try to check in at very similar times, causing a spike of writes in the database." The team had to implement request throttling to prevent sudden write bursts from crashing the database. Rather than pursuing incremental tuning, they sought a fundamentally different approach.
Pursuing a sustainable architecture
Red Canary established three core requirements for their new system:
- Capacity to sustain their large and expanding throughput
- Faster data visibility in their portal
- Reduced operational overhead
The team selected ScyllaDB Cloud using the Bring Your Own Account deployment model. Medina explained the rationale: "We installed it in our own AWS account, in a separate account from our remaining data to reduce any risk there. That way, ScyllaDB can still manage that cluster without having access to our data." The migration unfolded in two stages: first replacing Postgres, then replacing S3.
Migrating from Postgres
The Postgres replacement proved more straightforward than the S3 transition. Each sensor event appends a single row, and the web portal queries the data in time-series fashion—retrieving a customer's active endpoints over specified periods. However, the fundamental differences between Postgres and ScyllaDB required the team to address two specific challenges.
ScyllaDB does not immediately remove deleted data; instead, it leaves a timestamped marker called a tombstone. This mechanism prevents data resurrection, which could occur if a cluster node misses a deletion and later restores old data during repair operations. For Red Canary's ephemeral endpoints—which exist online only briefly—tombstones created significant problems. Medina described the issue: "Some clusters might scale up for an hour and then scale back down, which means an endpoint is alive for a very small amount of time. That meant some customers had partitions that were mostly just tombstones. When you query it, ScyllaDB has to iterate through all these tombstones, making our latency worse than it should be."
The solution was straightforward: since Red Canary did not require data deletion, they eliminated tombstones from the read path by setting a short TTL on the table and tuning tombstone garbage collection to remove them faster. This adjustment improved query latency.
The second challenge stemmed from wildly uneven data distribution across customers. Some maintained only dozens of endpoints while others operated hundreds of thousands. To prevent hot partitions in ScyllaDB, Medina's team implemented a bucketing strategy, splitting each customer's data into numbered buckets that ScyllaDB distributed across shards. When retrieving customer data, the system determines the bucket count and queries all buckets in parallel for faster results.
Replacing S3 storage


Substituting S3 with ScyllaDB presented a more complex challenge due to higher throughput and less predictable access patterns. Medina noted, "It's not a service trying to query in one way. Instead, it's developers debugging." The team could rely on one predictable pattern: developers would typically debug a single sensor at a time. This insight led them to designate sensor ID as the partition key for queries.
This choice served a strategic purpose. Medina shared, "One of our biggest reasons for wanting to do this was to see if ScyllaDB could also be used for our detection data in the future." Since health data had comparable throughput and lower operational criticality, it provided a safe testing ground for broader ScyllaDB adoption.
Events contained large arrays tracking process memory consumption at each endpoint. With thousands of processes potentially running simultaneously, rows could become quite large. The team addressed this by unnesting the arrays into an auxiliary table, storing each process's memory usage separately while maintaining the same partition for straightforward querying. Medina noted, "We can still put every row in the same partition so that our querying is still easy enough, as long as the partition doesn't get too big (too many rows)."

Batching presented the next challenge. Writing each health event individually to the database created unnecessary network round trips. Batching efficiency improves when all statements in a batch belong to the same partition, but the ScyllaDB Rust driver lacked this capability.
Customizing the Rust driver
The Rust driver did not expose prepared statements with pre-serialized values. To determine a statement's partition, developers had to serialize it once to obtain the hashed token, then serialize it again to execute it. Keeping values unserialized alongside statements created type-system complications in Rust, where collections require uniform types. Once serialized, however, values become a simple byte stream that can be stored together.
Medina described his solution: "To get around this, I forked the Rust driver to add the idea of a bound statement and then a bound batch, which is just a collection of statements. With this now in place, we can ask the bound statement what its token is without having to reserialize it, and then execute it again without reserializing it. A second benefit is that we can separate CPU work from I/O work. In Rust, you mostly use Tokio for I/O operations, and doing CPU work there can block the runtime and reduce efficiency."
The optimization allowed CPU-intensive work to run in a separate thread pool using Rayon, then return to Tokio for I/O operations. The results were dramatic. Medina reported, "When we made this change, our average time to process a file dropped from about 150 milliseconds to 50 milliseconds — a drop of about 60%. Of course, this is for our use case, so your mileage may vary."
Measuring the results

The LinuxEDR services now own the health data directly, no longer routing it through the engine. ScyllaDB stores all check-ins and health metrics, with a separate metrics database that may be consolidated later. The portal queries services for active endpoint lists, and developers can query ScyllaDB directly for debugging.
The outcomes have been substantial. Per-endpoint costs dropped significantly. Batching achieved approximately 20X write reduction relative to rows inserted—670K rows per second but only 36K writes per second thanks to batching. Portal queries for the largest customers now complete in seconds rather than minutes. Medina noted, "Each query to ScyllaDB is in the double-digit millisecond range."
Future directions
Red Canary is now exploring ScyllaDB for more critical data. Medina stated, "Since the transition has gone fairly smoothly, we're now exploring using it for data that is more critical. One of my team's main goals is to store our detection data in ScyllaDB to avoid sending data that doesn't need to go to the engine. Also, we're looking at ways to use it outside my team to replace some uses of OpenSearch, which has been giving us issues with cost, performance, and sometimes paging…no one likes being paged. Ideally, I would also like to merge the fork at some point so I don't need to rebase the driver on every new release."