Agent Fleet Management — Agent Dashboard
Overview
The Agent Dashboard provides a consolidated view of health and network metrics across all registered copy engine agents. Each metric is a time-series chart with one line per agent, so you can compare agents side by side.
Use it to identify which agents are under load, spot anomalies, and plan capacity before you schedule large sync runs. The history helps you understand utilization trends and plan scaling decisions.
The dashboard sits next to these sub-tabs in Cyber Vault > Agent Fleet Management:
- Network shows per-agent TCP socket state with current values.
- DNS Cache shows the Fleet DNS Cache Monitor.
- Live Agents
- Agent Installation
- Logging Config
Charts on the dashboard are time-series, including the ephemeral-port charts. Current-snapshot tables for sockets are on the Network tab.
Auto-Refresh
The dashboard refreshes automatically at an interval you choose. Use the Auto dropdown in the top-right toolbar.
| Option | Behavior |
|---|---|
| Auto 10s | Refreshes every 10 seconds. |
| Auto 30s | Refreshes every 30 seconds (default). |
| Auto 60s | Refreshes every 60 seconds. |
| Auto 5m | Refreshes every 5 minutes. |
| Auto Off | No automatic refresh. |
Next to the dropdown, you also see these controls:
- A countdown timer shows the time until the next refresh, for example "Refresh in 8s", or "1m 12s" when 60 s or more remain. At zero, the charts and the agent list both refresh, and the timer resets.
- A response time badge, for example "142ms", shows how long the last fetch took. It updates on every refresh, including manual ones.
- The rotate-left (↺) button triggers an immediate manual refresh and resets the countdown to the current interval.
Agent Health Trends
A row of metric tabs above the chart selects which metric is shown. The chart has one line per agent. When you hover over a point, it also shows the agent's active copy jobs at that moment.
| Group | Metric tabs |
|---|---|
| Health metrics | Host CPU, JVM Heap Used, Disk Read Latency, Disk Write Latency, JVM CPU, and CH Buffer Peak (per heartbeat) |
| Network metrics | TCP In-Use (Established), TCP TIME_WAIT, TCP Orphan, Egress (Tx), Ingress (Rx), Ephemeral Ports In Use, Ephemeral Ports Available, and TIME_WAIT Sockets |
A consistently high CPU or heap line means the agent is a bottleneck. Consider lowering its concurrency cap, giving it a larger JVM profile, or adding another agent.
Most metric tabs have their own help page. Open it with Help (?) > Help For This Page while that tab is selected. The Ephemeral Ports In Use, Ephemeral Ports Available, and TIME_WAIT Sockets tabs show this page.
Agent Table
Below the charts, you see the Connectivity Health card and the agent table. The table shows:
- Name, IP, and status
- Version
- Registration and last-seen times
- JVM profile
- Max Jobs, CH Jitter, and Step Log Jitter
- The Upgrade and Drain controls
Select agents with the checkboxes for bulk actions such as Upgrade, Cancel Upgrade, Drain, and Undrain.
Time Window
Use the time range selector to choose 1h, 6h, 24h (the default), or 7d. Zoom in on a specific incident window, or zoom out to see weekly trends.
Distribution Plan
There is no Distribution Plan button on this dashboard. The Auto Distribution Plan opens when you start several copy jobs at once, for example with Sync Group Now or a bulk resume in Copy Jobs - Live View & History.
It spreads heavy buckets across agents and uses only ONLINE agents that are not draining.
Network Bandwidth Charts
Two chart tabs in the Network group show real-time bandwidth use across the agent fleet.
| Chart tab | Metric | Meaning |
|---|---|---|
| Egress (Tx Mbps) | net_tx_mbps | Data leaving the agent host. This is the bytes transmitted across all physical network interfaces, converted to megabits per second. |
| Ingress (Rx Mbps) | net_rx_mbps | Data arriving at the agent host. This is the bytes received across all physical network interfaces, converted to megabits per second. |
During a copy job, Egress typically reflects S3 PUT traffic (data sent to the target bucket) and any data fetched back from the source. Ingress reflects data read from the source and acknowledgement traffic from the target.
How bandwidth is sampled
On each heartbeat, the agent reads /proc/net/dev, the Linux kernel interface statistics file. It sums the raw byte counters across all physical network interfaces.
Virtual and container interfaces are excluded because they carry internal traffic that is not relevant to actual WAN/LAN copy throughput. The excluded interface name patterns are lo, docker*, br-*, veth*, virbr*, tunl*, sit*, and dummy*.
The agent applies this formula to the difference between two consecutive readings:
Mbps = bytes_delta × 8 / 1_000_000 / elapsed_seconds
Results are rounded to 3 decimal places and stored in ClickHouse.
The sampler requires /proc/net/dev, so it works on Linux only.
Polling interval
Bandwidth is sampled on every agent heartbeat. The heartbeat fires at about 30 seconds plus 0 to 10 seconds of jitter. Each data point on the chart is therefore an average over about 30 to 40 seconds, not an instantaneous reading.
First heartbeat after agent start
The first bandwidth sample after an agent starts or restarts always reports 0 Mbps for both Tx and Rx. This is by design. On the first call, the sampler stores a baseline and has no earlier reading to compare against.
The second heartbeat, about 30 to 40 seconds later, produces the first real value. A single 0 data point at the start of a chart is normal.
Fleet Total line
When more than one agent has data for the selected time window, a Fleet Total line is drawn over the per-agent lines.
| Aspect | Description |
|---|---|
| What it shows | The sum of all agent values at each timestamp. At any point in time, it is the combined Egress (or Ingress) of the entire agent fleet. |
| Visual style | A thick dashed line in the text color (dark in Light mode, light in Dark mode). The chart header shows the subtitle "dashed line = fleet total". |
| When it appears | Only when two or more agents have data in the selected window. With a single agent, the Fleet Total is identical to that agent's line, so it is hidden. |
| How it is computed | In the browser, by summing the values from all per-agent series at matching timestamps. It is not stored in ClickHouse. |
Interpreting the charts
- Fleet Total tracks copy throughput. During an active sync run, the Egress Fleet Total closely mirrors total S3 PUT throughput across all agents. A flat line during a running job may indicate network saturation, a misconfigured agent, or a pause in available work.
- Uneven per-agent lines. If one agent's Egress is much higher than the others, it may be handling larger or more numerous objects. If this is undesirable, check the Auto Distribution Plan the next time you start several syncs at once.
- Gaps in the chart. Heartbeats are about 30 to 40 seconds apart, and ClickHouse keeps the data for 90 days. A gap means the agent was offline, restarting, or its heartbeat was delayed.
Why is my agent showing 0?
There are three common causes:
- The agent just started and has only one heartbeat so far.
- The agent runs on a non-Linux operating system.
- No copy jobs are currently active on that agent.
Connectivity Health
The Connectivity Health card reports the live status of every agent-to-console RPC channel for the agent selected in Connectivity Health for:.
It is the early-warning view for retry storms. Degradation shows up here before an agent goes fully offline. The AG-001 agent offline alarm fires typically 90 to 150 seconds after total silence. A breaker usually trips several minutes before that, with a specific named failure kind.
What the card shows
The card has a 4-row table, one row for each agent-to-console endpoint.
| Endpoint | What it does | When it fails |
|---|---|---|
heartbeat | The 30 s (plus or minus 10 s jitter) keep-alive. It carries agent stats, pending-job pause and resume notifications (pendingDeltas), and the connectivityHealth block itself. | This is the most critical endpoint. If this breaker opens, the agent is in serious trouble. Treat the alarm as urgent. |
liveStats | A push every 10 seconds of in-flight throughput counters for the trending UI. | Frequent but non-critical. An open breaker means only that the trending UI shows stale data. The agent keeps copying. |
copyLogCH | Per-object copy-log records written directly to ClickHouse. | Non-critical for each call, because the agent has a local SQLite spool that absorbs the loss. If this breaker stays open for long, the spool depth grows. Alarms AG-005 and AG-006 cover that case. |
autoPauseRpc | The legacy direct auto-pause call. Only agents that have not been upgraded use it. | Upgraded agents use the heartbeat path instead, so this row reads "no calls" on a fully upgraded fleet. |
State badges
Each row carries a state badge with one of three values.
| Badge | State | Meaning |
|---|---|---|
| Green | CLOSED | Normal. All recent calls succeeded. Calls flow through unconditionally. |
| Amber | HALF_OPEN | After an open period, exactly one probe call is allowed. If the probe succeeds, the breaker returns to CLOSED. If it fails, the breaker goes back to OPEN. |
| Red | OPEN | Fail-fast. The endpoint reached the failure threshold. |
With the default settings, the failure threshold is at least half of the last 10 calls failing, counted once at least 5 calls were made.
While OPEN, calls return immediately with no socket attempt. This protects the host's ephemeral port pool from being used up by a retry loop. The breaker stays OPEN for openDurationSec (60 s by default), then moves to HALF_OPEN for the next probe.
Each row also shows:
| Column | Description |
|---|---|
| Consec. Fails | How many calls in a row failed without a success in between. Resets to 0 on any success. |
| Last Failure | The relative time of the most recent failure, for example "12s ago" or "never". |
| Last Fail Kind | The failure category. See the table below. |
| Window | Successes and failures over the sliding window (10 calls by default). |
Common failure kinds
The Last Fail Kind column reports the most recent failure category for each endpoint. Each one gives a different hint.
| Kind | What it means | What to look at |
|---|---|---|
EADDRNOTAVAIL | "Cannot assign requested address". The agent host has used up its ephemeral port pool. Almost always paired with a very low Ephemeral Ports Available chart on this dashboard. Triggers AG-012. | Widen net.ipv4.ip_local_port_range on the agent host, or look for retry storms in the agent log. See the help on the Network tab. |
TIMEOUT_CONNECT | Connect-phase timeout. Either the network path to the console is broken (firewall or routing) or the console is not listening. | Verify connectivity from the agent host to the console, for example with curl against the console health endpoint. |
TIMEOUT_READ | The connection succeeded, but the console did not respond within the read deadline. The console is slow or overloaded. | Check console CPU and heap on System Monitor. Look at the console process's recent log for slow query warnings. |
HTTP_5XX | The console returned a 5xx error. This is a server-side bug or stack failure on the console. | Investigate the console error log. The breaker correctly treats this as a connectivity-class failure and backs off. |
HTTP_429 | The console is explicitly rate-limiting this agent. This should be rare. | Check whether a downstream component, such as ClickHouse, is in distress and shedding load with 429. |
CONNECT_REFUSED | TCP refused. The console process is down or the port is wrong. | Restart the console. Verify that the agent's configured console URL is correct. |
IO_ERROR | Any other underlying IOException. This is a catch-all. | Read the lastFailMessage field for the underlying exception text, truncated to 200 characters. |
4xx responses other than 429 do not count as failures. They are caller-side errors, such as a bad request or missing authentication. The breaker trips only on failure kinds where backing off helps. Counting 4xx would wrongly open the breaker on a configuration bug.
The 24-hour sparkline
Below the per-endpoint table, the card shows a 24-hour mini-sparkline for one endpoint at a time. Pick the endpoint with the buttons next to 24-hour state history.
The Y-axis has three horizontal bands, one for each breaker state:
- Top band:
OPEN. The breaker tripped and was failing fast. - Middle band:
HALF_OPEN. The breaker was probing recovery. - Bottom band:
CLOSED. Normal.
How to read the sparkline:
| Pattern | Meaning |
|---|---|
| A flat line in the bottom band | The normal, healthy picture. |
A short hop up to HALF_OPEN and back to CLOSED | Benign. A brief outage that the agent handled cleanly. |
| A square wave in the top band | The endpoint kept tripping and reopening over the period. This is typically a sustained connectivity problem. |
| A long stretch in the top band followed by recovery | The underlying problem fixed itself. |
The sparkline reads from the ClickHouse table cv_agent_connectivity_history. It is sampled on every heartbeat and kept for 7 days. For a post-incident review that goes back more than 24 hours, query this table directly. It holds up to 7 days of state transitions.
How this relates to AG-001
The AG-001 agent offline alarm fires when an agent has gone completely silent, based on the console's last-seen time for the agent.
Connectivity Health is an additional signal. It tells you something is wrong before the agent disappears entirely.
If the heartbeat endpoint breaker itself goes OPEN, the agent cannot deliver further connectivityHealth reports. The console then falls back to the last-seen time and fires AG-001 at the 90-second offline threshold. Total isolation is still covered.
Tips
If one agent consistently runs hotter than the others, check whether it was assigned disproportionately large bucket groups. The Auto Distribution Plan spreads heavy buckets across agents when you start several syncs at once.
Agent Dashboard charts are stored in ClickHouse, one sample per heartbeat. They remain available after the agent disconnects, so they are useful for post-incident review.
Bandwidth charts (Egress and Ingress) are sampled on the same cadence as all other Agent Dashboard metrics, about 30 s plus 0 to 10 s of jitter. Historical data is retained for 90 days in ClickHouse.
For agents that have not been upgraded, the Connectivity Health card reads "No connectivity health reported yet". This is expected. Those agents do not send connectivity health in their heartbeat.