How Linux Manages TCP Connection Memory

Linux File Descriptor Limits & “Too Many Open Files”

#

Because Linux treats network sockets as files, every active socket requires an open file descriptor (fd). To safeguard system resources against exhaustion or denial-of-service, the kernel enforces limits on the maximum number of open files at both the process and system-wide levels.

Exceeding these limits causes the kernel to reject new socket creations or accept() calls with the error:

EMFILE: Too many open files

The Path from Symbol to Socket Object

#

When a process opens a socket, the kernel executes sock_alloc_file() and maps the user-space integer descriptor to the underlying VFS kernel structures:

flowchart LR
    subgraph FDTable ["Process FD Table (fdtable)"]
        fd0["fd = 0 (stdin)"]
        fd1["fd = 1 (stdout)"]
        fd2["fd = 2 (stderr)"]
        fd3["fd = 3"]
    end

    FILE["struct file
(filp_cache)"]
    DENTRY["struct dentry
(dentry_cache)"]
    INODE["struct inode / struct socket
(sock_inode_cache)"]

    fd3 --> FILE
    FILE --> DENTRY
    DENTRY --> INODE

1. Allocate Index: get_unused_fd_flags() scans the process file descriptor table (fdtable) for the lowest available integer index (fd).
2. Allocate Object: sock_alloc_file() requests a core struct file from the filp SLUB cache.
3. Bind Structures: The kernel links fd -> struct file -> struct dentry -> struct socket.

Process-Level Limits: nofile & fs.nr_open

#

During allocation, __alloc_fd() verifies that the newly requested fd index does not exceed process-level limits:

• nofile (Per-Process Limit): Set per user/process via limits.conf or ulimit -n. It defines the maximum numeric value an fd index can reach for a specific process.

• fs.nr_open (System-Wide Boundary): Defines the absolute upper ceiling for nofile across any single process on the system.

# View the current process-level file descriptor limit
ulimit -n

# View the kernel ceiling for per-process limits
cat /proc/sys/fs/nr_open

System-Wide Limit: fs.file-max

#

While nofile constrains individual processes, fs.file-max defines the total number of open file allocations allowed system-wide across all processes combined.

# View current system-wide open file allocations and limits
cat /proc/sys/fs/file-nr

Output Fields: [allocated_fds] [unused_fds] [max_fds] as:

12480    0    1048576

Rules & Pitfalls When Raising Limits

#

To safely scale a server to support 100,000+ concurrent socket connections, you must adjust all three parameters in the correct hierarchy:

Soft nofile <= Hard nofile <= fs.nr_open <= fs.file-max

Common Configuration Hazards

#

1. Soft vs. Hard Limits: A user can dynamically raise their Soft nofile up to the Hard nofile threshold. To increase the Soft limit, the Hard limit must be adjusted equal to or higher than the target value.

2. The PAM Login Trap: If Hard nofile in /etc/security/limits.conf is configured to a value greater than fs.nr_open, the pam_limits module will fail during authentication. Users (including root) will be locked out and unable to log in via SSH.

3. Persistence Requirements: Modifying /proc/sys/fs/nr_open dynamically using echo or sysctl without persisting it in /etc/sysctl.conf will cause the system to revert upon reboot, triggering login lockouts if limits.conf references the higher numbers.

Production Recipe: Scaling for 100,000 Concurrent Sockets

#

To safely configure a server to support 100,000 concurrent socket connections with adequate safety margins:

1. Update Kernel Limits (/etc/sysctl.conf)

#

Set fs.file-max and fs.nr_open with sufficient headroom above your target connection count:

# /etc/sysctl.conf
# System-wide total file allocation ceiling (includes buffer for background services)
fs.file-max = 1100000

# Per-process maximum file descriptor ceiling
fs.nr_open = 1100000

Apply the updated kernel settings immediately:

sudo sysctl -p

2. Update User Limits (/etc/security/limits.conf)

#

Set process limits ensuring Hard nofile <= fs.nr_open:

# /etc/security/limits.conf
# <domain>    <type>    <item>     <value>
*             soft      nofile     100000
*             hard      nofile     100000

Server-Side Maximum TCP Connections

#

A common misconception is that a Linux server can only support 65,535 concurrent TCP connections due to the 16-bit limit on TCP port numbers.

In reality, web servers like NGINX or Apache listen on a single static port (e.g., port 80 or 443) and can easily maintain millions of concurrent connections.

The 4-Tuple Identity

#

The Linux kernel uniquely identifies every active TCP socket using a 4-tuple:

ecause the server’s local IP address and port remain fixed, the theoretical maximum number of concurrent inbound connections is determined solely by the unique combinations of remote Client IPs and Client Ports:

2^32  (Client IPs) x  2^{16} (Client Ports) = 2^48 ~ 281 Trillion

In practice, a server is bounded not by the 16-bit port space, but by physical system limits:

1. File Descriptors: Each connection requires a VFS struct file instance (fs.file-max).
2. Kernel Memory: Memory allocated for socket buffers (sk_buff) and kernel control structures.

Monitoring Socket Memory Buffers

#

While an idle TCP connection consumes very few resources, active communication requires memory allocations for transmit and receive socket buffers (wmem and rmem).

You can check and tune these system-wide limits using sysctl:

# Check TCP receive buffer limits (min, default, max in bytes)
sysctl net.ipv4.tcp_rmem

# Check TCP send buffer limits (min, default, max in bytes)
sysctl net.ipv4.tcp_wmem

Client Outbound Limit (Single vs. Multiple Local IPs)

#

While servers can scale to millions of inbound connections, client nodes initiating outbound traffic face different constraints governed by dynamic local port allocation.

When a client process initiates a TCP connection to a single remote server IP and port, the kernel assigns a local ephemeral port from ip_local_port_range:

# Inspect ephemeral port range (default is often 32768 to 60999)
cat /proc/sys/net/ipv4/ip_local_port_range

• Single Local IP: Constrained by the ephemeral port pool size (~ 60,000 available ports). A single client IP can establish roughly 60,000 concurrent outbound connections to a single target address.
• Multiple Local IPs (IP Alias / Binding): By binding outbound sockets to distinct local secondary IP addresses (bind() before connect()), each added IP grants an additional pool of ~ 60,000 ports.

Port Reuse Across Multiple Server Ports

#

The kernel tracks active sockets using hash tables (inet_hashinfo) backed by bucket linked lists. Because TCP connection uniqueness relies on the full 4-tuple, a single client IP can reuse local source ports if the destination changes.

If the target server listens across a range of ports (e.g., binding ports 8000 through 8010 across 11 ports):

Max Outbound Connections ~ 60,000 ports x 11 server ports = 660,000 conns

Workload Profiles: Static, Dynamic, and Push Architecture

#

The total system cost of maintaining TCP connections depends heavily on the application workload profile:

1. Idle / Inactive Connections: Consume minimal kernel RAM (~3.5 KB base) and 0% CPU while waiting for events.
2. Static Web Serving (File I/O): Low CPU cost; memory overhead is predominantly managed via VFS page caching.
3. Dynamic API / Database Apps: High CPU and memory utilization driven by business logic, database query processing, and object allocation—not by the TCP socket layer itself.

Production Case Study: 5,000,000 Long-Lived Push Connections

#

Consider a push notification gateway holding 5,000,000 concurrent, persistent TCP/WebSocket connections on a single server equipped with 64 GB RAM.

Scenario Constraints

#

• Active Connections: 5,000,000 long-lived sockets.

• Traffic Pattern: Highly asymmetric (99.9% idle; average 1–2 push broadcasts per client per day).

1. Kernel Allocation Breakdown

#

2. System Feasibility Assessment

#

• Memory Capacity: 37.5 GB fits easily within the 64 GB physical memory boundary, leaving ~26.5 GB of RAM available for the user-space push daemon and operating system overhead.

• CPU Load: Negligible during steady state, as idle sockets generate no interrupt handling or poll context switches.