How Linux Manages TCP Connection Resource Allocation

TCP Connection Memory Footprint & Kernel Allocation

#

Every TCP connection requires kernel memory to maintain its socket state, protocol control blocks, and I/O buffer queues. Because TCP connections are created and destroyed frequently, the kernel avoids allocating memory from scratch for every socket. Instead, it relies on a layered memory architecture: NUMA Nodes → Zones → Buddy System → Slab/Slub Allocator.

flowchart TD
    subgraph Layer1 [1. Hardware Topology]
        NUMA[NUMA Node
CPU + Local RAM]
    end

    subgraph Layer2 [2. Physical Memory Segmentation]
        Zone1[ZONE_DMA / DMA32]
        Zone2[ZONE_NORMAL]
    end

    subgraph Layer3 [3. Page Allocation]
        Buddy[Buddy Allocator
Manages 4KB Pages in Power-of-2 Blocks]
    end

    subgraph Layer4 [4. Object Allocation]
        SLUB[SLAB / SLUB Allocator
Pre-allocated Kernel Object Caches]
        TCPObj[TCP Objects: struct sock, sk_buff]
    end

    NUMA --> Zone1 & Zone2
    Zone2 --> Buddy
    Buddy -->|Contiguous Pages| SLUB
    SLUB -->|Slices into Object Sizes| TCPObj

How the Linux Kernel Manages Memory

#

1. NUMA Nodes (Non-Uniform Memory Access)

#

Physical memory is grouped into NUMA nodes, where memory is bound directly to specific CPU sockets. Accessing local memory connected directly to a CPU core is faster than fetching data from memory attached to a different processor socket.

2.Zone

#

Each NUMA node is divided into memory zones based on architectural hardware limits:

• ZONE_DMA / ZONE_DMA32: Reserved for direct memory access by legacy hardware devices.

• ZONE_NORMAL: Used by the kernel for standard dynamic data allocations, including TCP sockets and networking state structures.

• ZONE_HIGHMEM: Used on 32-bit architectures to address physical memory beyond the kernel’s virtual address space (obsolete on 64-bit Linux).

3. Page Management: The Buddy System

#

he core physical memory unit is a page (typically 4 KB). The Buddy System manages contiguous blocks of free pages in orders of powers of two (2^0, 2^1, 2^2 … 2^10 pages).

• When memory is requested, the Buddy System splits larger blocks into smaller “buddy” pairs.
• When memory is freed, adjacent buddies are merged back together to reduce internal and external physical memory fragmentation.

4. Object Allocation: SLAB / SLUB Allocator

#

Because allocating full 4 KB pages for small kernel structures (like a 2 KB TCP socket header) would waste memory, the kernel uses the SLAB (or modern default SLUB) allocator on top of the Buddy System:

1. Requests Pages: The SLUB allocator requests contiguous pages directly from the Buddy System.
2. Slices into Object Caches: It divides those pages into small, fixed-size slots tailored to specific kernel data structures (e.g., tcp_sock, request_sock, sk_buff).
3. Instant Reuse: When a connection closes, its memory slab is returned to the cache pool for instant reuse without needing full page allocation or deallocation cycles.

Memory Hierarchy: Split into NUMA Nodes

#

Modern server platforms use Non-Uniform Memory Access (NUMA) architecture. Instead of all CPUs accessing a centralized pool of RAM at the same speed, physical processors (sockets) and their directly attached memory modules are grouped together into local NUMA Nodes.

1. Inspect Hardware Topology (dmidecode)

#

To inspect physical CPU sockets and DIMM memory modules installed on the motherboard, use dmidecode:

# Display physical CPU processor details
sudo dmidecode -t processor

# Display installed RAM modules and slots
sudo dmidecode -t memory

2. Check NUMA Node Layout (numactl)

#

Each physical CPU socket and its directly connected local RAM form a distinct NUMA node. Accessing local memory is significantly faster than fetching data from RAM attached to a remote CPU socket across the interconnect bus (e.g., Intel UPI or AMD Infinity Fabric).

To inspect the system’s active NUMA topology and CPU-to-memory mappings:

numactl --hardware

Memory Zones Within NUMA Nodes

#

Every NUMA node is further partitioned into distinct physical address spaces called Memory Zones. These zones allow the Linux kernel to allocate pages based on hardware capabilities and usage requirements.

Standard Kernel Memory Zones

#

1. ZONE_DMA: Covers the lowest 16 MB of physical memory. Reserved for legacy devices that require Direct Memory Access within a 24-bit address space.
2. ZONE_DMA32: Covers physical memory between 16 MB and 4 GB. Used by 32-bit devices capable of 32-bit DMA addressing.
3. ZONE_NORMAL: Represents standard physical memory directly mapped by the kernel (memory above 4 GB on 64-bit systems). TCP sockets, buffer queues, and kernel network data structures are allocated here.

Each zone is composed of uniform memory blocks called Pages, which defaulted to a standard size of 4 KB on x86_64 architectures.

Inspecting Zone Configuration (/proc/zoneinfo)

#

You can view real-time per-zone memory stats, page counts, and allocation thresholds (watermarks) directly from the /proc filesystem:

cat /proc/zoneinfo

• pages free: Total unallocated 4 KB pages currently available in this zone.
• min, low, high (Watermarks): Thresholds used by the kernel background reclaim daemon (kswapd). If free pages drop below min, direct memory reclamation is triggered to free up space for incoming allocations.

Page Allocation via the Buddy System

#

Within each memory zone, unallocated physical pages are managed by the Buddy Allocator. The core tracking object is struct zone, which maintains a free_area array to manage available, contiguous memory blocks.

The free_area array contains 11 elements representing orders 0 through 10. Each order manages a linked list of free memory blocks scaled in powers of two 2 order pages, ranging from a single 4 KB page up to a 4 MB contiguous chunk:

Inspecting Buddy Allocator State (/proc/pagetypeinfo)

#

To view free page availability grouped by order and migration type (e.g., Unmovable, Reclaimable, Movable), inspect /proc/pagetypeinfo or /proc/buddyinfo:

cat /proc/pagetypeinfo

Buddy Allocation & Splitting Process

#

When a kernel component requests a 4 KB page (Order 0):

1. Direct Match: The kernel checks the Order 0 (4 KB) linked list. If a free block exists, it is allocated immediately.
2. Search Higher Orders: If Order 0 is empty, the kernel moves up to the Order 1 (8 KB) linked list.
3. Split Block: If an 8 KB$ block is found, the allocator splits it into two equal 4KB “buddy” blocks.
4. Assign & Store: One 4 KB block is returned to the requester, and the remaining 4 KB buddy block is placed into the Order 0 free list.

When pages are freed, the kernel checks if the adjacent “buddy” block is also free. If so, it merges them back into a higher-order block to prevent external memory fragmentation.

SLAB / SLUB Allocator: Small Object Management

#

While the Buddy System operates on 4 KB page boundaries, allocating a full page for smaller structures (like a 256 byte socket header) wastes significant memory due to internal fragmentation.

The SLAB Allocator (and its modern default variant, SLUB) solves this by requesting contiguous pages from the Buddy System and slicing them into fixed-size object pools called Caches (kmem_cache).

Internal Structure of a Cache (kmem_cache)

#

Each kmem_cache manages a specific kernel object type (e.g., tcp_sock, request_sock, skbuff_head). Within a cache, slabs are organized across three linked lists:

• Full List: Slabs where every object slot is currently allocated and in use.
• Partial List: Slabs containing both allocated and free object slots.
• Free List: Slabs where all object slots are empty. These can be returned to the Buddy System if memory needs to be reclaimed.

A single Slab consists of one or more contiguous 4 KB pages divided into uniform slots matching the target object size. When an object is freed, its memory slot is immediately marked available for reuse within the same cache without triggering page-level deallocations.

Inspecting Kernel Slabs

#

1. View Detailed Cache Statistics (/proc/slabinfo)

#

To inspect active slab caches, object counts, and individual object sizes:

sudo cat /proc/slabinfo

2. Real-Time Top Memory Consumers (slabtop)

#

To identify which kernel caches are consuming the most memory in real time (similar to top for user processes):

sudo slabtop -s c

TCP Connection Core Objects & Allocation Flow

#

When a application creates a socket or accepts a new TCP connection, the kernel allocates several core objects across specialized SLUB caches.

flowchart TD
    subgraph VFS [Virtual File System Representation]
        FD[User-Space File Descriptor] --> FILE[struct file
filp_cache]
        FILE --> DENTRY[struct dentry
dentry_cache]
        DENTRY --> INODE[struct inode
sock_inode_cache]
    end

    subgraph NET [Networking Layer]
        INODE --> SOCKET[struct socket
sock_inode_cache]
        SOCKET --> WQ[struct socket_wq
Wait Queue / Epoll]
        SOCKET --> INET[struct inet_sock / tcp_sock
tcp_sock cache]
    end

Key Kernel Objects for a TCP Socket

#

1. Networking Objects (socket & tcp_sock)

#

• struct socket: The BSD socket abstraction exposed to user space. It sits directly inside struct socket_alloc (allocated from sock_inode_cache).

• struct socket_wq: The socket wait queue structure. It manages process sleep/wake states and event notifications for non-blocking I/O frameworks like epoll.

• struct tcp_sock: The core TCP protocol control block. It contains sequence numbers, congestion control state, and sliding window timers. Allocated directly from the tcp_sock SLUB cache.

2. VFS File Representation Objects (file & dentry)

#

Because Linux follows the “everything is a file” paradigm, a socket must be integrated into the Virtual File System (VFS) so applications can read and write to it using file descriptors (fd):

• struct file (filp): Represents the open file object in user space. Allocated from the filp (file struct) SLUB cache.

• struct dentry: Represents the directory entry in the VFS tree. For sockets, a pseudo-dentry (e.g., socket:[12345]) is allocated from the dentry SLUB cache to link the file descriptor to the socket’s inode.

Object Allocation Flow During socket() / accept()

#

When a server creates or accepts a connection, the kernel executes the following allocation sequence:

1. Allocate Socket & Inode: Requests a struct socket_alloc block from sock_inode_cache, initializing both the struct socket and its underlying struct inode.

2. Allocate Protocol Socket: Requests a struct tcp_sock entry from the tcp_sock cache and links it to struct socket.

3. Allocate Wait Queue: Initializes socket_wq to handle asynchronous event processing (epoll_wait).

4. Bind to VFS File Descriptor:

   • Requests a struct file instance from the filp cache.

   • Requests a struct dentry instance from dentry_cache.

   • Links fd -> file -> dentry -> inode -> socket.

Inspecting Related SLUB Caches

#

You can monitor allocations across these specific socket and VFS caches in real time using slabtop or /proc/slabinfo:

# View active object counts for socket and VFS caches
sudo cat /proc/slabinfo | egrep 'tcp_sock|sock_inode_cache|dentry|filp'

TCP Core Memory Usage & Tuning

#

Understanding how much RAM a TCP socket consumes across its different connection states helps optimize servers for high concurrency.

Memory Footprint by Connection State

#

1. Established State (~3.2 KB Base Footprint)

#

When a connection completes the 3-way handshake and enters ESTABLISHED:

• Both client and server kernel sockets consume ~3.2 KB of base slab memory (for struct tcp_sock, struct socket_alloc, struct file, and struct dentry).

• Idle Connections: If no data is actively flowing, CPU usage is 0% and memory overhead remains capped at this minimal ~3.2 KB base allocation.

2. Active Data Transfer State (Dynamic Buffer Memory)

#

When data flows over a connection, the kernel allocates packet buffers (sk_buff / skb):

• Data buffers consume memory out of the send and receive window caches (tcp_wmem and tcp_rmem).

• Instant Recycling: As soon as the receiving endpoint acknowledges packet delivery via ACK, the corresponding sk_buff structure is freed back to the SLUB cache.

3. Closing States: FIN_WAIT2 & TIME_WAIT (~0.4 KB)

#

When a connection closes, the kernel replaces the heavy struct tcp_sock (~3.2 KB) with a lightweight struct inet_timewait_sock (~0.4 KB):

• Low Overhead: Thousands of sockets sitting in TIME_WAIT consume negligible RAM.
• CPU Impact: Idle TIME_WAIT sockets consume virtually zero CPU cycles unless port exhaustion occurs during dynamic port lookups.

Memory Footprint Example: 10,000 Idle Long-Lived Connections

#

For an application holding 10,000 idle TCP connections (e.g., WebSocket or HTTP keep-alive pool):

High TIME_WAIT Sockets: Risk Analysis & Kernel Tuning

#

Having thousands of TIME_WAIT sockets is not an error condition; it is a standard TCP design requirement to ensure in-flight delayed packets expire before a port pair is reused.

Relevant Kernel Parameters

#

1. net.ipv4.tcp_max_tw_buckets

#

Limits the maximum total number of sockets allowed in the TIME_WAIT state simultaneously across the system.

• If the count exceeds this limit, the kernel forcefully destroys the excess TIME_WAIT sockets and logs a warning in dmesg:

  TCP: time wait bucket table overflow

2. net.ipv4.ip_local_port_range

#

Defines the ephemeral port range available for outbound client connections (default: 32768 61000).

3. net.ipv4.tcp_tw_reuse

#

Allows the kernel to safely reuse a socket in TIME_WAIT state for new outbound client connections if the timestamp option (tcp_timestamps) proves the new packet sequence is strictly newer.

High-Load Architecture Recommendations

#

• Prefer Long-Lived Connections: Use persistent connections (HTTP Keep-Alive, gRPC, WebSockets) to avoid high connection churn rates.

• Enable tcp_tw_reuse: Set net.ipv4.tcp_tw_reuse = 1 on client machines making high-frequency outbound requests to prevent ephemeral port exhaustion.

• Scale Ephemeral Range: Expand net.ipv4.ip_local_port_range = 1024 65535 if your application frequently initiates short-lived outbound TCP sockets.