Skip to main content
Migration Notice
We're migrating documentation from the old portal into this one. Some things may look a little different or out of place in the meantime — we know, and we're working to get it right. If something's unclear or doesn't look right, let us know.

Agent Network — TCP Socket State Monitor

Overview​

The Network tab shows real-time, operating-system-level TCP socket state counts for each registered copy engine agent. The agent samples the values on its host and sends them to the console on every heartbeat. The heartbeat interval is about 30 seconds, plus 0 to 10 seconds of jitter for each beat.

Use this tab to diagnose connection exhaustion, port leaks, or sustained high connection counts during heavy copy workloads.

What the counters mean​

ColumnSource fieldMeaning
TCP In-UseinuseSockets in ESTABLISHED and other active states. Excludes TIME_WAIT.
TIME_WAITtwConnections waiting through the 2-MSL timeout after a graceful close.
OrphanorphanSockets with no file descriptor, awaiting teardown (FIN_WAIT1 accumulation).

TCP In-Use​

A high In-Use count during an active copy job is normal. Each S3 object transfer opens one or more HTTP connections.

High values when no jobs are running can indicate leaked connections that were not closed properly.

TIME_WAIT​

TIME_WAIT accumulates during sustained S3 copy workloads, because each per-object HTTPS request creates a short-lived TCP connection. Under load, a TIME_WAIT count of several hundred to a few thousand is expected.

It becomes a problem only when the agent exhausts the OS ephemeral port range, which is 32768–60999 by default, about 28,000 ports. If port exhaustion occurs, copy jobs start failing with EADDRNOTAVAIL ("Cannot assign requested address") errors.

Orphan​

The orphan count should be zero or near zero at rest. A persistent non-zero count indicates sockets stuck in FIN_WAIT1. This is usually caused by aborted connections that were not closed gracefully.

A growing orphan count warrants investigation.

Color coding​

CounterThresholdDisplayMeaning
TCP In-UseAbove 500BoldInformational. Not necessarily an error.
TIME_WAITAbove 200BoldInformational under load.
OrphanAbove 10RedPotential issue. Investigate if sustained.

Platform support​

TCP socket state monitoring requires Linux and a mounted /proc filesystem.

The "—" indicator in the table means the agent has not yet sent a heartbeat with TCP data. Either the agent version is older or the agent is offline.

Historical TCP socket state values are stored in ClickHouse. To view trends, open the Agent Dashboard tab and select the "TCP In-Use (Established)", "TCP TIME_WAIT", or "TCP Orphan" metric tab above the chart.

Ephemeral Ports and TIME_WAIT​

The ephemeral port pool on the agent host is not shown on this tab. It is charted on the Agent Dashboard tab.

Ephemeral port exhaustion matters because an agent-to-console call that keeps retrying under sustained failure can consume ephemeral ports faster than TIME_WAIT releases them. The agent then cannot make any outbound TCP connection, including to its own S3 source and target.

For the per-endpoint view, see the Agent Dashboard Connectivity Health card. The AG-012 alarm is defined for this condition but is not active in this release, so watch the Ephemeral Ports charts on the Agent Dashboard.

The Agent Dashboard charts​

Chart (metric tab)What it showsSource
Ephemeral Ports In UseA conservative upper bound on sockets currently bound from the ephemeral range, in any state (ESTABLISHED, TIME_WAIT, SYN_*, and so on). Equals the inuse count from /proc/net/sockstat.On every heartbeat, the agent reads /proc/net/sockstat and the configured ephemeral range from /proc/sys/net/ipv4/ip_local_port_range.
Ephemeral Ports AvailableThe count of free ports remaining in the configured ephemeral range: (rangeMax - rangeMin + 1) - inuse - tw.Derived from the configured range minus in-use minus TIME_WAIT.
TIME_WAIT SocketsTotal sockets in TIME_WAIT. This is the same value as the TIME_WAIT column on this tab, charted over time for the port-exhaustion view.The tw field of /proc/net/sockstat.

A Cannot assign requested address error (EADDRNOTAVAIL) on an agent-to-console call appears in the Last Fail Kind column of the Agent Dashboard Connectivity Health card.

info

Hosts without /proc/net/sockstat: the agent cannot compute the available ports and reports them as unknown. The console does not raise the AG-012 alarm for that agent.

What ephemeral ports are and why exhaustion is dangerous​

When a Linux process opens an outbound TCP connection without binding to a specific source port, the kernel picks one from the ephemeral port range. This is a configurable band of port numbers reserved for short-lived client-side connections.

The default range on most modern Linux distributions is 32768-60999, which is about 28,000 ports. Every outbound HTTPS request to the console, every S3 PUT, and every connection check takes one port from this pool.

When a connection closes gracefully, its port enters TIME_WAIT for about 60 seconds (the 2-MSL timeout) before it returns to the pool. This is normal TCP behavior. It prevents a delayed packet from one connection from being accepted by a later connection on the same 4-tuple.

warning

If a process retries failed connect() calls faster than TIME_WAIT can drain, ports accumulate in TIME_WAIT until the entire range is consumed. The next connect() returns EADDRNOTAVAIL ("Cannot assign requested address").

At that point every new outbound TCP attempt fails, to the console and to S3. The agent stays isolated until the TIME_WAIT backlog clears. This can take minutes if the retry loop keeps running.

Widening the ephemeral port range​

If AG-012 fires repeatedly on an agent host that does a lot of TCP work, the default 28,000-port range may be too narrow for the workload. Widen it with sysctl:

# Inspect current range
cat /proc/sys/net/ipv4/ip_local_port_range
# Default output: 32768  60999

# Widen to ~55,000 ports (10000 → 65535)
sysctl -w net.ipv4.ip_local_port_range="10000 65535"

# Make the change persistent across reboots
echo "net.ipv4.ip_local_port_range = 10000 65535" > /etc/sysctl.d/99-argus-ports.conf
sysctl --system

10000-65535 is the practical maximum on Linux. Ports below 10000 collide with well-known and registered service ports.

TIME_WAIT decay​

TIME_WAIT sockets are released after the 2-MSL timeout, which defaults to about 60 seconds on Linux. High TIME_WAIT counts during sustained copy load are expected. A value of several thousand on a busy copy agent is normal.

These values warrant attention:

  • TIME_WAIT count approaching the size of your ephemeral port range, within a few thousand.
  • TIME_WAIT growing while Ephemeral Ports Available shrinks below 1000, which triggers AG-012.
  • TIME_WAIT growing faster than 60 seconds of decay can drain it. This is a clear signal that something is opening short-lived connections in a tight loop.

Cross-references​

  • AG-012 (Agent port exhaustion) is raised when Ephemeral Ports Available drops below 1000. It clears automatically when available ports recover above 2000.
  • Agent Dashboard > Connectivity Health card shows which RPC channels (heartbeat, liveStats, copyLogCH, autoPauseRpc) are tripping their circuit breakers. The card's Last Fail Kind column shows EADDRNOTAVAIL while port exhaustion is the active failure.