How Linux Transmits Network Packets

Transmit Call Path & Memory Dynamics

#

While the top-level overview traces packets from user space to physical hardware, looking under the hood reveals the specific kernel subsystems, memory copies, and network protocol layers involved.

1. System Call Entry (sys_sendmsg)

#

When an application calls send() or sendto(), the kernel begins processing in the process context:

1. Socket Resolution: The kernel resolves the file descriptor (fd) to its corresponding struct socket and identifies the socket’s protocol-specific operations (sk_prot).
2. msghdr Construction: The kernel builds an internal struct msghdr structure containing pointers to user-space memory buffers, payload length, and destination control metadata.

2. Transport Layer: TCP Processing & Management

#

Inside the transport protocol handler (e.g., tcp_sendmsg()):

• Buffer Allocation & Memory Copy: The kernel allocates socket buffers (sk_buff) from kernel memory and copies raw bytes from user space into sk_buff payload areas via copy_from_iter().
• Write Queue Placement: The newly populated sk_buff is appended to the socket’s write queue (sk_write_queue). If memory allocation fails or queue limits (wmem_default) are reached, the call fails or blocks depending on socket configuration.
• TCP Segment Creation (Resend Buffer): Because TCP requires reliable delivery, the kernel creates a working clone of the sk_buff to pass down the stack while retaining the original buffer inside sk_write_queue until a remote ACK is received.
• Window & Control Management: The TCP state engine enforces sliding window limits, manages Congestion Notification (CCN), builds TCP segment headers, and passes cloned buffers to the Network Layer.

3. Network Layer: IP Routing & Segmentation

#

The IP layer (ip_queue_xmit()) manages packet destination resolution, header construction, and firewall filtering:

1. Route Lookup & Caching: The kernel performs a destination lookup against the routing table to determine the egress interface (Iface) and next-hop Gateway. Sockets cache previous route entries to eliminate redundant table searches on active streams.
2. IP Header Assembly: Populates source/destination IP addresses, TTL, protocol flags, and calculates IP header checksums.
3. Netfilter Hooks (iptables/nftables): Packets traverse PREROUTING and LOCAL_OUT firewall chains. Complex rule evaluation directly increases CPU overhead (sy usage).
4. IP Fragmentation & MTU Boundaries: If packet payload exceeds the Maximum Transmission Unit (MTU)—typically 1500 bytes—the IP layer splits the payload across multiple fragments:

4. The Neighbor Subsystem (ARP & MAC Layer)

#

Positioned between the Network Layer and Data Link Layer, the Neighbor Subsystem provides an abstract layer above physical addressing:

• Hardware Address Resolution: Resolves target IP addresses to hardware MAC addresses using internal ARP caches (arp_tbl).
• ARP Request Dispatch: If a target MAC address is unmapped, the subsystem queues the sk_buff and broadcasts an ARP request across the local broadcast domain.
• Header Framing: Prepends Ethernet frame headers (Destination MAC, Source MAC, EtherType) before handing the frame to the Network Device Subsystem.

5. Network Device Subsystem & Traffic Control (qdisc)

#

The packet enters device-agnostic driver queues (dev_queue_xmit()):

1. Egress Queueing (qdisc): The kernel selects a transmit queue and submits the packet through Queueing Disciplines (e.g., fq_codel, sfq) for traffic shaping and prioritization.
2. Bypassing Queues: Under low network contention, the kernel bypasses the software qdisc queue entirely and writes the frame directly down to the driver ring buffer to minimize latency.
3. Transmission Quota Loops: Processing threads poll egress queues until either all buffers flush or their CPU execution quota expires—yielding control to avoid starving other processes.

6. Memory Copies & Zero-Copy Optimization

#

Understanding memory transfers across the transmit path highlights why traditional file-serving architectures struggle under high throughput:

Standard Path vs. Zero-Copy I/O

#

flowchart LR
    subgraph Standard["Standard Path (2 CPU Copies + 2 DMA)"]
        direction LR
        D1["Disk"] -- "DMA" --> PC1["Page Cache"]
        PC1 -- "CPU Copy" --> UB["User Buffer"]
        UB -- "CPU Copy" --> SKB["Kernel SKB"]
        SKB -- "DMA" --> NIC1["NIC"]
    end

flowchart LR
    subgraph ZeroCopy["Zero-Copy Path (0 CPU Copies + 2 DMA)"]
        direction LR
        D2["Disk"] -- "DMA" --> PC2["Page Cache"]
        PC2 -- "DMA Pointer" --> NIC2["NIC"]
    end

• Standard Transmit Cost (2 CPU Copies + 2 DMA Transfers):
  1. Disk to Kernel Page Cache (DMA).
  2. Page Cache to User-Space Buffer (CPU Copy via read()).
  3. User-Space Buffer to Kernel sk_buff (CPU Copy via send()).
  4. Kernel sk_buff to NIC Ring Buffer (DMA).
• sendfile() Zero-Copy Optimization: Eliminates user-space intermediate buffers. Data moves directly from storage into kernel Page Cache via DMA, while sendfile() passes raw descriptor references to the NIC ring buffer—eliminating CPU-bound memory copying entirely.

7. Resource Monitoring: Tracking Kernel Overheads

#

Because packet transmission executes predominantly inside kernel space, standard user-space metrics (%usr) miss network bottlenecks.

# Monitor kernel CPU time and softirq overhead
top -b -n 1 | grep "%Cpu"

• sy (System CPU Usage): High sy time indicates heavy kernel execution—frequent system call entries (send()), memory allocation, and complex iptables rule evaluations.

• si (Software Interrupt Usage): High si time highlights driver packet processing overhead, NET_TX_SOFTIRQ routines, and ring buffer cleanup tasks.