How Linux Receives Network Packets

Linux Network Stack

#

1. The 5-Layer Network Model

#

Understanding how Linux processes packet flow starts with the standard TCP/IP 5-layer architecture:

flowchart TD
    subgraph L1["Application Layer"]
        UI1[Client: HTTP / FTP / SCP]
    end

    subgraph L2["Transport Layer"]
        UI2[TCP / UDP]
    end

    subgraph L3["Network Layer"]
        UI3[IP, ICMP, IGMP]
    end

    subgraph L4["Data Link Layer"]
        UI4[Network Driver]
    end

    subgraph L5["Physical Layer"]
        UI5[Network Cable, NIC]
    end

    L1 --> L2
    L2 --> L3
    L3 --> L4
    L4 --> L5

2. Network Interrupt Processing

#

Network drivers rely on hardware interrupts to notify the OS when new data arrives. However, parsing protocol headers, validating checksums, running firewall rules, and delivering packets to application sockets are compute-intensive operations.

If the kernel executed all network stack logic inside the hardware interrupt handler directly:

• CPU Core Lockup: The CPU core remains locked in interrupt context (HardIRQ) with local interrupts disabled, blocking all other execution threads on that core.
• Peripheral Latency: High-priority system tasks and input devices (mouse, keyboard, system timers) suffer extreme input lag or drop inputs entirely.
• Interrupt Starvation: Under high throughput, continuous packet arrivals lock the system in an “interrupt storm,” starving non-network kernel processes and leading to severe packet loss.

To keep system peripherals like the mouse and keyboard responsive while processing heavy network traffic, the Linux kernel splits network processing into two distinct execution phases: the Hardware Interrupt (HardIRQ Executes immediately with high priority) and the Software Interrupt (SoftIRQ, Runs asynchronously).

flowchart TD
    subgraph HW["Hardware & Driver Level"]
        A["Packet Arrives at NIC"] --> B["DMA Transfer to RX Ring Buffer"]
    end

    subgraph L1["Top Half (HardIRQ - Fast Context)"]
        B --> C["Hardware Interrupt (HardIRQ)"]
        C --> D["Disable NIC Interrupts & Schedule NAPI"]
    end

    subgraph L2["Bottom Half (SoftIRQ - Deferred Work)"]
        D --> E["NAPI Poll / SoftIRQ (NET_RX_SOFTIRQ)"]
        E --> F["Driver Allocates Socket Buffer (sk_buff)"]
        F --> G["ksoftirqd Thread Processes Packets"]
    end

    subgraph L3["Kernel Network Stack"]
        G --> H["Gro/Netfilter (iptables / nftables)"]
        H --> I["IP Protocol Layer"]
        I --> J["Transport Layer (TCP / UDP)"]
        J --> K["Socket Receive Queue (sk_receive_queue)"]
    end

    subgraph L4["User Space"]
        K --> L["Application Read syscall (read / recv / epoll)"]
    end

Linux Protocol Registration & Driver Initialization

#

During boot, the Linux kernel sets up core subsystem handlers—registering transport protocols (TCP/UDP), initializing per-CPU ksoftirqd threads, and allocating DMA ring buffers for network interface drivers.

1. Kernel Boot Setup

#

When the Linux kernel boots, it prepares the networking stack before any physical interface comes online:

• Protocol Registration: Transport layer handlers (TCP via inet_init() and UDP via udp_init()) register their protocol handlers into the kernel’s protocol table (ptype_base).
• ksoftirqd Worker Threads: The kernel spawns one ksoftirqd/X kernel thread per CPU core (e.g., ksoftirqd/0 for CPU 0, ksoftirqd/1 for CPU 1). These threads handle deferred SoftIRQ network processing asynchronously without blocking hardware interrupts.

2. NIC Driver Initialization (dev_open Sequence)

#

When an interface is brought up (e.g., via ip link set eth0 up), the kernel executes the driver’s opening hook (ndo_open()) to allocate memory and bind hardware resources:

flowchart TD
    subgraph L1["Linux Kernel"]
        A["Call dev_open() / ndo_open()"]
    end

    subgraph L2["Network Driver"]
        B["Initialize Hardware & NAPI"]
        C["Allocate Ring Buffers (RX/TX sk_buff)"]
        D["Request IRQ Line & Register Handler"]
    end

    subgraph L3["Runtime Ingress"]
        E["Process Packet on Arrival"]
    end

    A --> B
    B --> C
    C --> D
    D --> E

Inspecting Ring Buffers with ethtool

#

You can view and modify these driver ring buffer settings at runtime using ethtool:

# View maximum supported vs. current RX/TX ring buffer descriptor sizes
ethtool -g eth0

Ring Buffer and Memory Layout

#

The memory allocated for RX/TX rings consists of contiguous physical memory blocks assigned via Direct Memory Access (DMA). This continuous address space allows the driver and hardware to step through slots sequentially and wrap back around to index 0 like a circular ring.

Its physical memory address layout operates as a contiguous ring queue:

Its physical memory address layout looks like:

Benefits of the RX/TX Ring Buffer Memory Layout

• Eliminates Repeated Memory Allocation Overhead: Memory slots in the ring buffer are pre-allocated during driver initialization. Instead of dynamically allocating (kmalloc) and freeing memory for every incoming network packet, the kernel continuously reuses the fixed slots, avoiding heavy heap allocation churn.

• Prevents Memory Fragmentation: Because the buffer occupies a single contiguous physical block of address space, it eliminates external memory fragmentation that occurs when thousands of dynamic, short-lived packet buffers are constantly allocated and freed across RAM.

• Maximizes DMA Hardware Efficiency: The Network Interface Card (NIC) hardware can step through array slots sequentially via direct physical memory pointers without CPU intervention or costly virtual-to-physical address translation lookups.

• Optimizes CPU L1/L2 Cache Prefetching: Contiguous address layouts allow the CPU hardware prefetcher to anticipate upcoming memory reads, significantly reducing cache misses during high-packet-per-second (PPS) SoftIRQ processing.

create ksoftirqd Processing

#

During early system boot, the Linux kernel creates dedicated ksoftirqd/X kernel threads bound to each online CPU core (e.g., ksoftirqd/0 for CPU 0, ksoftirqd/1 for CPU 1).

process network packets

#

1. Hardware Interrupt (HardIRQ)

#

Modern Network Interface Cards (NICs) with multi-queue support split network traffic across multiple ringbuffer.

When network traffic hits the physical layer, host RAM is updated without touching the CPU—until a hardware interrupt signals that work is waiting.

Each queue can be bound to a preferred CPU core using smp_affinity. When a single core handles all hardware interrupts, it risks becoming a performance bottleneck. Adding more queues and tuning smp_affinity spreads this overhead evenly across multiple cores.

# Find IRQs assigned to your network interface
cat /proc/interrupts | grep eth0

# View the human-readable CPU core list for IRQ 42
cat /proc/irq/42/smp_affinity_list

For example, if smp_affinity_list outputs 3, routes all HardIRQs for that queue specifically to CPU3.

When incoming data arrives, the Network Interface Card (NIC) places the frame into its physical RX receive queue. The NIC’s Direct Memory Access (DMA) engine then transfers the raw frame directly into host memory (RAM) without requiring CPU intervention. Once the DMA transfer completes, the NIC issues a Hardware Interrupt (HardIRQ) to notify the CPU:

flowchart TD
    A["Incoming Data Arrives at NIC"] --> B["DMA Transfers Frame to Host Memory"]
    B --> C["NIC Triggers Hardware Interrupt (HardIRQ)"]
    D["CPU Executes Registered Interrupt Handler"] --> E["Disable NIC Interrupts & Schedule NAPI (NET_RX_SOFTIRQ)"]

    C --> D

2. Ring Buffer Drops & Overruns

#

During high-throughput bursts, if ksoftirqd or the SoftIRQ context cannot process frames fast enough, the RX ring buffer will become completely full. Once full, any new incoming frames are dropped by the NIC hardware.

• Identifying Ring Buffer Drops: In ifconfig or ip -s link output, the overrun counter tracks packets dropped due to an exhausted RX ring buffer.
• Tuning Ring Size: You can inspect and expand the maximum allowed descriptor entries using ethtool:

# View current and maximum ring buffer sizes (Pre-set and Current hardware settings)
ethtool -g eth0

# View network interface statistics (including dropped packets and errors)
ethtool -S eth0

# Increase RX / TX ring buffer size to 4096 descriptors
ethtool -G eth0 rx 4096 tx 4096

The hareware interrupt only change a varible on the poll_list that make it fast to finish.

2. SoftIRQ & ksoftirqd Processing

#

These ksoftirqd threads handle deferred, non-critical execution contexts like NET_RX_SOFTIRQ.

To preserve CPU cache locality and avoid multi-core lock contention, a CPU core only processes network packets that were placed into its local ring buffer by its corresponding hardware interrupt vector.

1. Hardware & Software Offloading: GRO, TSO, LRO, and GSO

#

To process 10Gbps+ network traffic without saturating CPU cores, modern Linux networking relies on packet aggregation offloads. Instead of traversing the entire kernel stack on a per-packet basis, multiple packets are combined into larger pseudo-buffers (up to 64KB) to drastically reduce per-packet execution overhead.

These offloading mechanisms operate across both ingress (receive) and egress (transmit) paths:

2. Packet Inspection Point (tcpdump)

#

GRO merging smaller wire frames and tcpdump sees them as single TCP packets that are much larger than the standard interface MTU. After GRO processing, the packet passes into the kernel’s network tap layer (AF_PACKET). Tools like tcpdump or tshark tap into packets at this exact point—capturing raw incoming frames after GRO reassembly, but before they pass through Netfilter firewall rules or reach the socket layer.

3. IP Layer (Netfilter & Routing)

#

At Layer 3, incoming traffic passes through Netfilter hooks and iptables/nftables rule chains. Unoptimized rule sets can severely consume CPU cycles and introduce packet delivery latency.

Once frame processing completes at the link layer, the kernel passes the packet up to the IP layer. After passing through IP-level firewall evaluation and routing logic, valid packets are handed off to transport protocols like TCP or UDP.

4. The tcpdump vs. Netfilter Asymmetry

#

A common source of confusion during network troubleshooting is the difference in hook placement between incoming (ingress) and outgoing (egress) traffic relative to AF_PACKET taps like tcpdump:

• Ingress (Incoming Traffic):
  tcpdump taps into the packet stream before Netfilter/iptables rules execute. If a firewall rule drops an incoming packet in PREROUTING or INPUT, tcpdump will still capture it.

• Egress (Outgoing Traffic):
  Netfilter rules evaluate traffic before the packet reaches the AF_PACKET tap layer. If an outbound rule drops a packet in OUTPUT or POSTROUTING, tcpdump will never see it.