Optimizing Local Socket Communication: avoiding tcp/ip stack(part 1)

Mar 6, 2025

Problem

When two processes on the same host communicate over TCP — think a web server talking to a local database, or a sidecar proxy sitting in front of a service — every message takes a round-trip through the full TCP/IP stack, even though the data never leaves the machine. This article shows how to use **eBPF sockops **and sk_msg programs to short-circuit that path, redirecting messages directly from one socket to another inside the kernel.

Even on loopback, a send() → recv() between two local sockets goes through:

 App A                                          App B
 send()                                         recv()
   │                                              ▲
   ▼                                              │
 TCP layer  →  IP layer  →  Netfilter  →  TCP layer  →  Socket buffer
   │                                              │
   └──────── full kernel network stack ───────────┘

Every packet is allocated, checksummed, passed through netfilter hooks, queued, and finally delivered. For local traffic this work is pure overhead — the data never touches a NIC.

The Solution

With eBPF we can attach a program that intercepts messages before they enter the TCP/IP stack and delivers them straight to the destination socket:

 App A                                          App B
 send()                                         recv()
   │                                              ▲
   ▼                                              │
 Socket  ──►  BPF sk_msg  ──►  Socket buffer ────┘
              redirect!

   (TCP/IP stack is completely bypassed)

This requires two eBPF programs working together:

4. The eBPF Programs

Both programs live in sock-redirect-ebpf/src/main.rs.

4.1 sockops — Populating the Map

The #[sock_ops] program is attached to a cgroup and fires on socket lifecycle events for every socket belonging to a process in that cgroup.

#[sock_ops]
pub fn sock_ops_prog(ctx: SockOpsContext) -> u32 {
    match try_sock_ops(&ctx) {
        Ok(ret) => ret,
        Err(_) => SK_PASS,
    }
}

fn try_sock_ops(ctx: &SockOpsContext) -> Result<u32, i64> {
    let op = unsafe { (*ctx.ops).op };

    match op {
        BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB | BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB => {
            // remote_port is in network byte order → convert to host order
            let remote_port = u32::from_be(ctx.remote_port()});

            SOCK_MAP
                .update(&remote_port, ctx.ops, BPF_ANY as u64)
                .map_err(|e| e)?;
        }
        _ => {}
    }

    Ok(SK_PASS)
}

We care about two operation types:

On both events we insert the socket into SOCK_MAP keyed by the remote port.

Byte order detail: bpf_sock_ops.remote_port is stored in network byte order (big-endian), while local_port is in host byte order. We convert with u32::from_be() so both sides of the cross-map use the same representation.

4.2 sk_msg — Redirecting Messages

The #[sk_msg] program is attached to SOCK_MAP and fires whenever sendmsg() or sendfile() is called on a socket that lives in the map.

#[sk_msg]
pub fn sk_msg_prog(ctx: SkMsgContext) -> u32 {
    match try_sk_msg(&ctx) {
        Ok(ret) => ret,
        Err(_) => SK_PASS,
    }
}

fn try_sk_msg(ctx: &SkMsgContext) -> Result<u32, i64> {
    let local_port = unsafe { (*ctx.msg).local_port };

    SOCK_MAP
        .redirect_msg(ctx, &local_port, BPF_F_INGRESS);

    Ok(SK_PASS)
}

local_port on the sender side equals the remote_port that sockops used as the key when storing the receiver’s socket.
BPF_F_INGRESS delivers the message into the target socket’s receive queue.
On error (socket not found) we return SK_PASS to fall back to the normal TCP/IP path — the connection still works, just without the optimization.

4.3 The Cross-Mapping Trick

This is the key insight that makes the whole scheme work:

Connection:  Client:54321 ◄═══════► Server:8080

sockops fires TWICE and stores by remote_port:

  ACTIVE_ESTABLISHED (client side):
    remote_port = 8080  → sock_map[8080]  = client_socket

  PASSIVE_ESTABLISHED (server side):
    remote_port = 54321 → sock_map[54321] = server_socket

sk_msg looks up by local_port:

  Client sends → local_port = 54321 → sock_map[54321] → server_socket ✓
  Server sends → local_port = 8080  → sock_map[8080]  → client_socket ✓

The sender’s local_port always matches the receiver’s entry in the map, because local_port(sender) == remote_port(receiver).

The Userspace Loader

The userspace program (sock-redirect/src/main.rs) does three things:

We create a dedicated cgroup (test.slice) so the BPF programs only affect processes we explicitly move into it.

let cgroup_path = Path::new("/sys/fs/cgroup/test.slice");
if !cgroup_path.exists() {
    fs::create_dir_all(cgroup_path)?;
}

Attach sockops to the cgroup so that on any new connection creation(connect/accet), that socket can be inserted in the sock_map.

let sockops: &mut SockOps = ebpf
    .program_mut("sock_ops_prog")
    .unwrap()
    .try_into()?;
sockops.load()?;

let cgroup_file = fs::File::open(cgroup_path)?;
sockops.attach(cgroup_file, CgroupAttachMode::Single)?;

Attach sk_msg to the map, note that sk_msg is attached to the map, not the cgroup. The kernel invokes it whenever sendmsg()/sendfile() is called on a socket inside SOCK_MAP.

let sock_map: SockHash<_, u32> = SockHash::try_from(
    ebpf.take_map("SOCK_MAP")?,
)?;
let map_fd = sock_map.fd().try_clone()?;

let sk_msg: &mut SkMsg = ebpf
    .program_mut("sk_msg_prog")
    .unwrap()
    .try_into()?;
sk_msg.load()?;
sk_msg.attach(map_fd)?;

6. Building and Running


cd sock-redirect-ebpf && cargo build --release # don't use debug bcz it will have debug symbols, and use target "bpfel-unknown-none", in config.toml

RUST_LOG=info cargo run --package sock-redirect --release

How It All Fits Together

                    ┌──────────────────────────────┐
                    │        Userspace Loader       │
                    │     (sock-redirect binary)    │
                    │                               │
                    │  1. Load eBPF ELF             │
                    │  2. Create test.slice cgroup  │
                    │  3. Attach sockops → cgroup   │
                    │  4. Attach sk_msg  → SOCK_MAP │
                    └──────────────┬───────────────┘
                                   │ loads
                    ┌──────────────▼───────────────┐
                    │           Kernel             │
                    │                              │
                    │  ┌────────────────────────┐  │
                    │  │  test.slice cgroup     │  │
                    │  │  sockops attached      │  │
                    │  │                        │  │
                    │  │  Process A  Process B  │  │
                    │  └────────┬───────────────┘  │
                    │           │                  │
                    │  ┌────────▼───────────────┐  │
                    │  │  SOCK_MAP (SockHash)   │  │
                    │  │  sk_msg attached       │  │
                    │  │                        │  │
                    │  │  port → socket_ref     │  │
                    │  │  port → socket_ref     │  │
                    │  └────────────────────────┘  │
                    │                              │
                    │  On send():                  │
                    │    sk_msg fires              │
                    │    → redirect_msg()          │
                    │    → data lands in peer's    │
                    │      recv buffer directly    │
                    │                              │
                    └──────────────────────────────┘

The beauty of this approach is that it’s transparent — applications don’t need any code changes. They use normal send()/recv() and the kernel handles the shortcut behind the scenes.

In the next article, will show how to create network namespaces and connect those two namespaces and run two different programs in those two namespaces over one client. We will showcase how their sockets communicate directly, avoiding the TCP stack cost.