Agent Fleet Management — Live Agents
Overview
Live Agents shows all currently registered copy engine agents in real time, with their connection status, current workload, and health indicators.
Use this tab to:
- Confirm that agents are online before you trigger a sync.
- Investigate an offline agent.
- Drain an agent to take it out of rotation without canceling its active jobs.
Vault agents are filtered out of this view. They are managed exclusively in Vault Operations > Vault Agents. Only COPY_ENGINE entries appear here.
The console itself is not listed. Copy jobs always run on a copy agent, never on the console.
Agent Card Fields
| Field | Description |
|---|---|
| Agent name | The display name set during registration, or renamed inline. |
| Type | COPY_ENGINE (runs copy jobs). |
| Status | ONLINE, OFFLINE, or DEGRADED. See Status values. |
| Conn | A connectivity-health badge next to the status pill. See Connectivity badge. |
| Last heartbeat | Timestamp of the most recent check-in. Staleness is shown as a relative time badge. |
| Active jobs / Max concurrent | How many jobs are running now versus the configured concurrency cap. The pill shows active/limit, for example 2/8. |
| Site | An optional grouping label for multi-site deployments. |
Status values
| Status | Meaning |
|---|---|
ONLINE | A heartbeat was received within the last 90 seconds. |
OFFLINE | No heartbeat for more than 90 seconds. See below. |
DEGRADED | The agent itself reported reduced capacity (status:"DEGRADED" in its heartbeat). Heartbeats are still arriving, but the agent has signaled that it is not fully healthy. |
When an agent goes OFFLINE, the console's agent watchdog:
- Changes the status to
OFFLINE. - Re-queues pending jobs assigned to the agent.
- Moves the agent's running jobs to
agent_pending_report. - Raises alarm
AG-001.
The yellow DRAINING pill next to the status badge reflects the separate draining flag. A draining agent stays ONLINE but is excluded from new job dispatch. See Drain Control.
Connectivity badge
The Conn badge rolls up the state of every agent-to-console RPC channel. The channels are the heartbeat, the live-stats push, the copy-log direct ClickHouse writes, and the /auto-pause endpoint. It lets you see which agents have degraded connectivity before they go fully offline.
| Badge | Meaning |
|---|---|
| Green Conn ✓ | All channels are healthy. Every per-endpoint circuit breaker is CLOSED. |
| Conn ⚠ | At least one channel is probing recovery. Any breaker is HALF_OPEN. |
| Red Conn ✕ | At least one channel is failing fast. Any breaker is OPEN. |
| Conn — | No data. Agents that have not been upgraded yet show this instead of a color. This is expected. |
Hover over the badge to see a tooltip that lists every channel with its state, and the last failure reason when the state is not CLOSED. The per-channel detail is on the agent's Connectivity Health card on the Agent Dashboard tab. See the Agent Dashboard help for the underlying state model.
Degraded agents badge in the side menu
When one or more agents report degraded connectivity, the Agent Fleet Management entry in the Cyber Vault side menu shows a number badge with the count.
- The badge is amber when channels are degraded.
- The badge is red when at least one channel is fully
OPEN, which means cut off. - The count refreshes every 30 seconds.
Click the badge to open this Live Agents list filtered to the degraded agents only. Use Show all to clear the filter.
No copy agent available
Every copy job needs at least one copy agent that is ONLINE and not draining. This applies to Run, Resume, Run Sync, Sync All, Sync Group Now, and Plan & Dispatch.
If none is available, nothing runs. The request is refused with this message:
No copy agent available — start or undrain a copy agent and try again.
A job pinned to one agent (Run or Resume with a specific agent chosen) runs only on that agent. Draining does not stop it. If that agent is offline, the request is refused with this message:
Copy agent "<name>" is offline — choose Auto or another copy agent, or start <name> and try again.
Drain Control
| Action | Description |
|---|---|
| Drain | Stops new jobs from being dispatched to this agent. Already-running jobs continue to completion. The status badge turns yellow. Use it before you reboot or upgrade an agent. |
| Undrain | Returns the agent to the active dispatch pool immediately. |
Inline Editing
Click the edit pencil on any agent row to change these settings:
- The agent name.
- The site.
- The max concurrent jobs.
- A per-agent crash timeout override.
Bulk Actions
| Action | Description |
|---|---|
| Drain All / Undrain All | Applies the drain state to every agent in the list at once. |
| Set Crash Timeout (All) | Opens a modal that applies one timeout override to all COPY_ENGINE agents at once. Leave it blank to clear all overrides back to the global default. |
| Global crash timeout pill | An amber pill that is always visible and shows the current global default. Click the pencil to edit it inline. |
The global crash timeout controls how long the console waits before it auto-fails agent_pending_report jobs on any agent that has no per-agent override. The console never moves these jobs to another agent. Resume them by hand.
Heartbeat History (vault buffering)
Copy agents assigned behind a vault agent lose direct network access to the console while the vault network is closed. A copy agent in this state does not appear offline for that whole window. It buffers its heartbeats locally. When the vault reopens, it replays the complete history to the console, with each heartbeat's original timestamp.
Every COPY_ENGINE agent card has a collapsed Heartbeat History (vault buffering) section under Job Stats. Expanding it fetches that agent's buffered and replayed heartbeat records, newest first, 20 per page.
| Column | Description |
|---|---|
| Generated At | The original timestamp when the agent produced the heartbeat, not when it was replayed to the console. After a vault cycle completes, you can use it to reconstruct when the agent started having problems, even though the console had no live connection to it at the time. |
| Metrics | The raw heartbeat payload snapshot from that moment: status, version, active job count, and health metrics. |
The section is collapsed by default and loads only when you expand it, because a long vault closure can buffer hundreds of records. An agent that has never been through a vault-closed period shows "No buffered heartbeat history."
The history is retained for 30 days on the server and pruned automatically. It is a forensic record only. It does not affect the agent's live status or job dispatch.
Tips
- An agent showing OFFLINE with active jobs means those jobs are in
agent_pending_reportstate. The console is waiting for the agent to reconnect and report outcomes. - If an agent does not come back online, drain it and let the crash timeout auto-fail its jobs. Then re-submit them manually if needed.
- If a copy agent behind a vault agent seems to have missed heartbeats during a closed period, check its Heartbeat History panel before you assume a real outage. It may simply have been buffering locally.