I'm taking CSCE 430: Computer Architecture this semester at UNL, and the discussion over interconnection networks (Appendix F) and associated performance metrics made me curious how networking works under the hood. If you wanted to push networking performance to the max, how would you do it?

My initial search informed me that networking is handled in the operating system socket API, and from such most of the latency is derived. For this project I wanted to look below the socket API and see how much work could be avoided by handling a packet as soon as it reached a Linux network interface.

For the experiment, I built a small echo server using express data path (XDP) and extended Berkeley Packet Filter(eBPF) , then benchmarked it against the normal Linux networking path. The XDP program recognizes an internet control message protocol (ICMP) echo request, edits the packet in place, and sends it back out the same interface without taking the usual trip through the rest of the network stack. I also wrote a user diagram protocol (UDP) socket server and client while getting the baseline path working.

The name “kernel bypass” is a slight misnomer. XDP still runs inside the kernel, but it bypasses most of the kernel's networking stack. The packet is handled at the network driver hook, before socket lookup, protocol handling, or a user-space process gets involved.

Building a Network in One Computer

I wanted the tests to be repeatable without needing two physical computers, so I created two Linux network namespaces named cli and srv. A virtual Ethernet pair connects them: veth1 lives on the client at 10.0.0.2, and veth0 lives on the server at 10.0.0.1. From the programs' point of view these behave like two separate machines connected directly together, although everything is running on the same computer.

I wrote a management script to set up the namespaces, create the veth pair, assign the addresses, attach the XDP programs, show what is currently loaded, and remove everything between tests. I found out that an old XDP program can remain attached to an interface after the command that loaded it exits, which makes benchmark results pretty confusing if you forget to clear it.

Answering Packets Before the Network Stack

The eBPF program receives a xdp_md context containing pointers to the beginning and end of the packet. From there I manually parse the Ethernet, IPv4, and ICMP headers. Before reading each header I have to prove that it fits inside data_end. This is required by the eBPF verifier, but it is also just good packet parsing: a short or malformed packet should never turn into an out-of-bounds memory access inside the kernel.  

Packets that aren't IPv4 ICMP echo requests return XDP_PASS, which lets Linux handle them normally. For an echo request, the program:

1. Swaps the source and destination MAC addresses.
2. Swaps the source and destination IPv4 addresses.
3. Changes the ICMP type from echo request to echo reply.
4. Updates the ICMP checksum.
5. Returns XDP_TX, sending the edited packet back through the interface it arrived on. There is no new response packet and no call to send(). The request itself becomes the reply.

The checksum was the most error-prone part. Swapping the IP addresses doesn't change the sum of the IPv4 header, but changing the ICMP type certainly changes the ICMP checksum. I used bpf_csum_diff to update the checksum using the old and new type/code words rather than walking the entire packet. It is a very small piece of code, although getting a checksum wrong results in a packet that looks correct in memory and is then quietly discarded somewhere else.

I also implemented the same basic idea for UDP. That version swaps the Ethernet addresses, IP addresses, and UDP ports before returning XDP_TX. Writing both versions made the difference between a normal socket echo server and packet-level processing much more obvious. The socket server blocks in recvfrom, copies the payload through the kernel, wakes a process, and calls sendto. XDP performs a few header edits and immediately returns an action to the driver.

Loading EBPF into the Kernel

The XDP source is restricted C compiled by Clang/LLVM into eBPF bytecode in an ELF object. I used libbpf and libxdp for the user-space loader, with options to select an interface, choose native or generic XDP mode, attach a program, or unload one by its program ID.

Before it can run, the kernel verifier checks every possible path through the program. This is why the pointer bounds checks have to be explicit and why normal C patterns do not always translate directly to eBPF. It can feel overly strict when the verifier rejects a program, but the alternative is allowing arbitrary code to corrupt kernel memory every time a packet arrives. Working through those constraints gave me a much better understanding of what “safe code in the kernel” actually requires.

Benchmarking the Two Paths

I tested a baseline with no echo program attached, then repeated the same workload with the XDP program attached. The benchmark script clears the interfaces between runs, records round-trip latency with normal and flood pings, and collects per-CPU utilization with mpstat. Below are the resulting graphs.

For the normal 100-packet test, mean latency dropped from 0.0573 ms to 0.0405 ms, or about 29%. Median latency only moved from 0.040 ms to 0.036 ms, but the tail improved much more. The 95th percentile dropped from 0.104 ms to 0.058 ms, the 99th percentile from 0.333 ms to 0.116 ms, and the worst sample dropped from 1.64 ms to 0.396 ms.

The graph shows an interesting phenomenon the occurs around the tail latency. Average latency is easy to improve while still occasionally producing a very slow packet. XDP reduced the standard deviation from 0.122 ms to 0.026 ms and cut the maximum by roughly 76%. In this small test, bypassing the normal stack made the response time much more predictable, not just slightly faster.

The flood test wasn't as dramatic. Mean RTT changed from 0.0101 ms to 0.0096 ms and the 99th percentile was effectively unchanged. Since both namespaces and the virtual Ethernet pair are on the same host, the numbers shouldn't be treated like a general promise about XDP on real hardware. They show the behavior of this setup and gave me a controlled way to compare the paths.

CPU utilization result was more surprising but makes sense in hindsight. The collected averages reported about 2.2% busy CPU for the baseline and 8.0% for XDP. That doesn't support the simple claim that “fewer layers always means less CPU.” The runs are short, use virtual interfaces, and include the measurement workload, so scheduling and background activity have a large effect at these percentages. I would want longer runs, isolated CPU cores, more repetitions, and a physical XDP-capable NIC before drawing a strong CPU-efficiency conclusion.

Remarks

Having hands on experience with the networking stack gave me more intuition about how networking really works. A ping reply normally passes through several layers that are invisible from a user-space socket. With XDP I had to account for every byte of the packets myself, convince the verifier that every access was safe, repair the checksum, and then decide what should happen to the packet. The code to do this isn't all that long but building the environment around it took the most of my time.