When a security incident occurs on any compute platform, investigators face the same urgent question: which workload connected to that endpoint, at what time, and how much data moved? On Lambda, the challenge intensifies because thousands of microVMs run simultaneously on each bare-metal host, each lasting only hundreds of milliseconds before shutting down. Those brief windows of execution generate the only network records available for investigation, audit, and billing verification.
AWS Lambda needed a complete network ledger for every tenant workload regardless of duration or traffic volume. The company replaced an aging capture system with a purpose-built pipeline using eBPF and Rust, addressing fundamental limitations in the older architecture while maintaining compatibility with downstream systems.
The requirement: complete and trustworthy records
Every Lambda worker is a bare-metal EC2 instance containing microVMs, each an isolated Firecracker guest. These machines communicate across the network to S3, other AWS services, the public internet, and customer VPCs. A network flow log serves as the system of record for what happened to each packet, supporting investigation, incident response, audit, and workload reconstruction. The same records feed into network usage and metering services that demand absolute accuracy, and all records must persist for compliance purposes.
Two properties matter above all others. Records must be complete and correctly attributed, and capturing them must add almost no overhead to network flows or platform operations. Correct attribution means every packet and flow associates with the specific microVM and tenant that produced it. Completeness means no missed packets. Missing or misattributed records cause billing errors, observability gaps, and monitoring failures at Lambda's scale, which serves millions of requests per second. Overhead matters because every additional megabyte of RAM and microsecond of CPU consumed reduces utilization, operating margin, and the ability to serve requests under load.
Why the previous system reached its limits
Lambda's original flow logging system, inherited from the older single-tenant EC2 design, consisted of two components: a kernel-side extension that counted packets and matched them to tenants and sandboxes per flow, and a userspace daemon that read those counters, batched them into records, serialized them, and uploaded files. This approach functioned adequately for small numbers of VMs but collapsed under Lambda's density for two mutually exclusive reasons.
The first problem was rule explosion. iptables processes its rules roughly linearly for every packet, and each new microVM added more rules to the chain. A worker running a couple of thousand microVMs required well over a hundred thousand iptables rules just to maintain the record. Every packet incurred a tax proportional to host crowding and slot position. As the host became busier, packet bookkeeping slowed—exactly the wrong direction when the goal was packing more microVMs onto each host, not fewer.
The second limitation was more fundamental: the borrowed kernel module did not support IPv6. "A record that can't see half the address space isn't one you can trust, and the moment dual-stack IPv6 support was proposed for Lambda, the old approach was finished." No performance improvements could overcome this architectural gap.
Design principles for the replacement
The rewrite had three non-negotiable requirements: correct attribution, meaningful overhead reduction, and IPv6 as a first-class citizen. Dozens of internal systems already knew how to read the Amazon Ion records the old daemon produced. If the new system could emit byte-for-byte identical records, the entire capture engine could be replaced without any consumer, flow-log, or metering system noticing the swap.
To achieve correct attribution, the team implemented isolation at the microVM level. Each microVM already had its own network: a namespace with its own virtual devices. That network became the unit around which everything else organized.
The new system architecture
The replacement consists of three cooperating components, each named by its function:
ONE WORKER HOST
control plane
|
| gRPC over a Unix domain socket
v
orchestrator ....... one privileged process per host
| (loads eBPF, configures TC,
| spawns one tagger per network)
v
--- per network (one microVM) ------------------------------
eBPF capture --> ring buffer --> tagger --> Ion records
TC hooks on one per Rust,
the network's network userspace
devices
------------------------------------------------------------
|
v
same billing + flow-log pipeline as before
At the bottom sits the kernel capture layer: a set of small eBPF programs attached to the traffic-control (tc) hook on each network's relevant virtual Linux devices. They intercept packets and emit one compact event per packet into a ring buffer. These programs only observe; they contain no code paths to copy, block, drop, or rewrite packets.
In the middle is the tagger: an unprivileged userspace process written in Rust, one per network. It drains its dedicated ring buffer, rolls raw per-packet events into per-flow records, and writes them to disk in the legacy Amazon Ion format. On top is the orchestrator, one privileged process per host. It owns everything requiring elevated permissions: loading eBPF programs, wiring up traffic control, spawning and supervising the fleet of taggers. It exposes a small lifecycle API over a Unix domain socket so the control plane can create, assign, recycle, and tear down tagging as microVMs come and go.
This decoupled design allows capture in the kernel, aggregation in userspace, and one process per host orchestrating everything. Captured records go to existing downstream consumers unchanged.
Kernel-level capture without interference
The capture programs attach to the clsact qdisc in traffic control on both ingress and egress sides of each network's devices. A network spans two devices, creating four attach points per network. Traffic control provides an ideal vantage point, seeing every packet early before downstream processing. Every eBPF program reads the packet and returns the "keep going" action. No customer packets are dropped or modified, and the operation adds no meaningful latency.
For each packet, the program walks the headers—Ethernet, then IPv4 or IPv6, then TCP, UDP, or ICMP—and writes one fixed-size event into that network's BPF ring buffer. The event is intentionally small, approximately two dozen bytes for IPv4:
/* one event per packet, ~24 bytes for IPv4 */
struct flow_event {
u8 ip_version; /* 4 or 6 */
u8 protocol; /* TCP / UDP / ICMP */
u8 direction; /* ingress or egress */
u8 device_id; /* which of the network's devices */
u16 local_port; /* "local" is always the sandbox side */
u16 remote_port;
u32 flags_and_bytes; /* TCP flags in bits [31:24], byte count in [23:0] */
u32 local_addr; /* 16 bytes for IPv6 */
u32 remote_addr;
u32 received_time_ms;
};
One packet generates one event with one byte count. The kernel does not count packets; that work belongs to userspace. The flags and byte count share a single 32-bit word: eight bits of TCP flags on top, a 24-bit byte count underneath. "Doing less work per packet in the kernel is the entire point, so aggregation is somebody else's job."
The local and remote fields get normalized by direction before the event leaves the kernel. Arriving or departing, local always means the sandbox side and remote means the outside world. This normalization means userspace never reasons about direction when grouping flows, and the record reads the same way regardless of packet direction.
The eBPF program uses a reserve-then-commit order with a dedicated ring buffer. Each program asks the ring buffer for space, adds the event in place, and submits. This approach avoids copying data through a syscall, consumes minimal CPU, and requires no per-CPU bookkeeping—just a single consumer draining it in order.
Getting complex parsing and event submission logic past the eBPF verifier required careful engineering. The verifier must prove a program is safe before the kernel loads it, requiring strict memory bounds checks and instruction limits. To ensure quick verification and loading, the team made the header parser a shared subroutine so the verifier proves it once instead of re-proving it at every attach point. They bounded the IPv6 extension-header walk to a fixed number of hops so the verifier can ensure termination. Packet fragments past the first IPv4 fragment report zero ports and flags rather than garbage. Coalesced super-packets from segmentation and receive offload (GSO/GRO) have their byte counts handled correctly. A garbage port or over-counted byte would create a false log entry, so the record stays honest at the source and byte-identical with the old system.
The team does not rely on the verifier as the sole source of correctness. Because this code produces records critical to billing, compliance, and auditing systems, each eBPF program is written in C and runs through a formal model checker (CBMC) during every build. Its harnesses assert that the event struct's byte layout stays compatible with what every attached program expects. A struct that silently shifts by a byte is the kind of bug that quietly corrupts every record it touches, going unnoticed until the logs are needed.
Sizing the ring buffer from first principles
The ring buffer is the one thing the kernel producer and userspace consumer share, and its size involves real tradeoffs. Too small and events drop under a burst, creating holes in the log right when traffic matters. Too large and memory is wasted, with that waste paid on every ring buffer on the host.
Rather than guessing, the team derived the floor from each microVM's packet rate. For example, if the ceiling is 100,000 packets per second per direction and the ring drains roughly every 100 milliseconds, multiplying the peak rate by the drain interval, by the event size, and by two directions yields:
ring bytes =~ 62,500 pps x 0.1 s x ~24 bytes x 2 directions
=~ 300 KB
The ring buffer API requires a power of two, so the design defaults to 512 KiB. That is the smallest buffer that cannot overflow between drains at the guest's own maximum packet rate. Put another way, the floor ensures a guest cannot outrun the recorder, even when trying to. The size is configurable per network. In the running deployment, the system currently provisions it more generously than that floor, on the order of a couple of megabytes, while tuning the right per-workload value. The number to defend is the floor, derived from a hard system limit, not a guess.
The drain cadence has another useful property: waking a userspace process is not free, and a fleet of thousands of processes all waking constantly would thrash the CPU. The kernel decides when to bother, checking how full the ring is and only forcing a wakeup once the ring crosses about one percent full. Below that threshold, it stays quiet and lets events pile up. Userspace will not read more than once every 100 milliseconds. A quiet flow sits there until the next drain, basically free. A busy one trips that one-percent threshold and gets read almost right away. Nothing runs on a fixed timer, so neither case gets the timing wrong.
Aggregation in Rust userspace
The tagger turns raw per-packet events into per-flow records the pipeline stores. One tagger runs per network, unprivileged.
Rust was chosen for practical reasons. At this density, thousands of these processes run on a host, each holding a small amount of state that must be correct. A garbage-collected runtime would cause pause times and memory bloat under load, and a pause in the wrong place could create a gap in the record. Rust provides predictable memory and no collector, plus a compiler that refuses to build whole categories of bugs that turn into misattribution. Each tagger runs in a few hundred kilobytes of RAM against a roughly one-megabyte budget, small enough that thousands per host is practical.
Inside, a small set of cooperating tasks runs on a single-threaded async runtime. One task reads the ring, another owns the flow state, and a third writes parcels. The only work fenced onto a blocking pool is operations that genuinely block: receiving the ring descriptor and serializing Ion, since the Ion writer is not async. It reads the ring through epoll, sleeping when there is nothing to do and waking when there is.
As events arrive, the tagger drops them into a flow map keyed by device, the five-tuple, and a tenant attribution handle. That handle comes from metadata the control plane handed over when the flow was activated. Matching events accumulate bytes, packet counts, and OR'd TCP flags. Grouping happens as events are read, so the hot path stays a lookup and an add.
Attribution comes from the fact that mapping is the most important property for the record. The kernel event carries no identity and does not need to. Every network has its own dedicated ring and devices, so packets are separated long before the tagger sees them. The tagger is not pulling one tenant's packets out of some shared firehose. The stream it reads was only ever that one tenant's, because the ring and the devices feeding it belong to that tenant alone.
Once a minute, on a fixed interval lined up to the top of the second to match the old system it replaced, the tagger serializes completed flows into Amazon Ion records in exactly the schema the old daemon produced. Each file gets written to a temporary name, flushed to disk, and renamed into place. A reader sees a complete record or nothing, never a torn one. A separate flush loop, with a little random jitter at startup so thousands of processes do not all write at the same instant, drains completed flows even after a microVM has gone quiet. A workload that goes silent still leaves a finished record behind it.
Because the records are byte-compatible with the old format, the entire downstream world kept working untouched. When the two systems ran side by side, their output could be compared record for record and confirmed to agree. That is about as direct a completeness check as possible.
Least privilege enforced through file descriptors
The processes doing the actual packet work—the thousands of taggers—hold no elevated privileges. They cannot load eBPF or touch traffic control. They cannot even open the ring buffer map on their own. All of that power lives in one place: the per-host orchestrator, and even it runs with just the two capabilities it needs rather than as root.
An unprivileged tagger reads a ring buffer it is not allowed to open because the orchestrator opens it and hands the open file descriptor to the tagger over a Unix domain socket, using the kernel's SCM_RIGHTS mechanism to pass descriptors between processes. The tagger gets a ready-to-use handle to the ring and nothing else. It never had, and never needs, permission to create one. "Passing a file descriptor over a socket is a decades-old Unix feature, and it lets us keep thousands of processes powerless while concentrating privilege in one small place." The privileged surface of the whole system is one small process per host. The thousands of processes touching customer traffic are about as powerless as possible.
Lifecycle API with two-stage activation
MicroVMs come and go constantly, so the control plane needs a way to tell the orchestrator when to start and stop recording a network. It does that through gRPC APIs over a Unix domain socket, with a handful of methods: create a set of flows, activate a flow, recycle one, tear one down, plus a health check.
Starting to record is split into two calls, a heavy one and a light one, and the split is deliberate. Create is a heavy call, the expensive path: it loads and attaches the eBPF programs, configures traffic control, and spawns the tagger. Attaching to network devices takes a kernel lock that every such operation on the host contends for, so when a host stands up many networks at once, these operations are batched to keep everyone from serializing behind that one lock. Activate is the lighter call. By the time it runs, the machinery already exists, so it just hands over the customer metadata and flips the flow into steady-state recording. Its latency budget is tight: under 2 milliseconds at p90, under 10 milliseconds at p99.9, matching the baseline of the system it replaced.
An honest tradeoff: reuse versus safety
Not every decision came out clean. These are lessons for anyone building something similar.
The original design had a strict rule for recycling a network: always destroy the tagger and spawn a fresh one. From a correctness standpoint, the reasoning was airtight. A brand-new process cannot carry stale metadata from a previous tenant, so a flow from one tenant landing in another's record across a recycle becomes structurally impossible. Kill it, do not try to clean it. The team was sure that was the right call.
Reality under production workloads showed that forking and exec'ing a new process thousands of times as networks churned turned into a real source of CPU spikes at scale. The safest choice showed up as a flame graph. So the shipped system needed a new knob. A workload that reuses its networks can reuse the tagger after a recycle, trading off a little of that structural guarantee for a lot less CPU churn. A workload that wants the strict, cross-tenant-proof behavior leaves the knob off. The strict version is still considered the more correct design, but the fleet's CPU budget just did not allow it.
What the new system achieved
Start with the number that killed the old design. One host needed more than a hundred thousand firewall rules to keep the record for two thousand microVMs, and each additional microVM piled on more, taxing every packet a little further. The eBPF version swaps that linear rule walk for constant-time map lookups whose cost does not climb as the host fills up. The linear tax is gone. That puts the density target—roughly double the microVMs per host—within reach, without the burst-time gap a per-packet tax invites.
The rest of the payoff falls out of the constraints the team started with:
- IPv6 flows, invisible to the old tool, get recorded like anything else, so the log covers the whole address space instead of half.
- Each tagger lives in a few hundred kilobytes of RAM against a roughly one-megabyte budget, small enough that thousands per host is practical.
- Activating a flow into steady-state recording stays under 2 milliseconds at p90 and under 10 milliseconds at p99.9.
- The capture layer is observe-only and formally checked, and the processes touching customer traffic hold no privileges. Significant visibility was added while shrinking the trusted, privileged surface that could corrupt the record.
- The output records are byte-for-byte identical to the old format, so every downstream flow-log and metering consumer kept working with no change.
Lessons applicable beyond Lambda
Several of these principles travel well beyond Lambda. Kubernetes pods, edge runtimes, and the sandboxes people are spinning up now to run AI agents all share a common pattern: many tenants sharing a host with a need for trustworthy traffic records. The same architectural shapes hold.
First: observe from outside the hot path. The moment recording logic sits inline in packet forwarding, its cost becomes a tax on every packet, and that tax is heaviest right when the record matters most. That same pressure pushes teams to drop or sample data, and an audit trail cannot survive sampling. eBPF lets you watch from the side and emit a compact event, while everything expensive happens elsewhere.
Second: size buffers from something real. A buffer sized by an actual rate limit times an actual drain interval is a number you can defend in a review, and it is what lets you promise no dropped events under a burst. Picking 512 KB because it felt about right probably would have held most of the time, right up until some burst it was not sized for.
Third: keeping tenants apart at the point of capture is the part worth arguing hardest for. Give each tenant its own ring and its own devices, and the streams never touch, so you are labeling clean traffic instead of guessing after the fact.
Fourth: old primitives are underrated. Passing a file descriptor over a socket is a decades-old Unix feature, and it lets you keep thousands of processes powerless while concentrating privilege in one small place.
Last one, and it is the cheapest to get wrong: when you swap out an engine, keep the bolt pattern. Byte-for-byte identical output let the team replace the entire capture path with zero downstream migration, and it handed them a record-for-record way to prove the new system saw everything the old one did.
Source: The New Stack