Agent Fleet Management
This tab shows all registered copy engine agents, their current status, and the tools for managing their availability.
Use it to monitor agent health, prepare agents for maintenance, configure crash timeout behavior, and distribute jobs across the available agents.
Vault agents are managed separately in Vault Operations > Vault Agents. They do not appear in this view.
Overview
AirGap for ObjectScale copy operations are run by copy engine agents rather than by the console itself. You can register multiple copy engine agents at the same time to distribute load across machines.
Agent Card Fields
| Field | Description |
|---|---|
| Agent Name | The registered identifier for this agent. |
| Status | The current availability state. See Status values. |
| Last Heartbeat | The timestamp of the most recent heartbeat received from the agent. A heartbeat older than the crash timeout threshold indicates that the agent may have disconnected. |
| Current Job Count | The number of jobs actively running on this agent right now. |
| Max Concurrent Jobs | The maximum number of jobs this agent runs at the same time. See Max Concurrent Jobs. |
| CH Jitter | The ClickHouse flush jitter window in seconds. The default is 60 s. See Jitter settings. |
| Step Log Jitter | The step-log jitter window in seconds. The default is 60 s. See Jitter settings. |
| Crash Timeout Override | How long, in minutes, the console waits in agent_pending_report state before it marks a job as failed when this agent stops responding unexpectedly. Blank means the global default applies. |
Status values
| Status | Meaning |
|---|---|
ONLINE | The agent is connected and sending heartbeats normally. Its most recent heartbeat was within the last 90 seconds. |
OFFLINE | The agent has not sent a heartbeat for more than 90 seconds. The console's agent watchdog sets this status. The watchdog runs every 60 s. It raises alarm AG-001 (or VA-001 for vault agents). |
DEGRADED | The agent reported reduced capacity in its heartbeat (status:"DEGRADED"). Heartbeats still arrive, but the agent signaled that it is not fully healthy. This value is reserved for future agent self-reporting. Current builds do not raise it on their own. |
When an agent goes OFFLINE:
- Pending jobs assigned to the agent are re-queued and dispatched to another available agent.
- Running jobs move to
agent_pending_reportand wait for the agent to reconnect until the crash timeout expires.
Drain mode is a separate flag (draining), not a status value. A draining agent stays ONLINE and continues to run its assigned jobs. Only new dispatch is suppressed. A yellow DRAINING pill beside the status badge reflects this flag. See Drain Mode.
A separate Conn badge next to the status shows connectivity health. It reports the circuit-breaker state of each RPC channel. It can show Conn ⚠ (probing) or a red Conn ✕ (failing fast) several minutes before the agent crosses the 90 s offline threshold. Watch this badge as an early warning. See the Agent Dashboard help for the breaker state model.
Max Concurrent Jobs
The limit is enforced in two places:
- Console side (dispatch). The dispatcher skips an agent when its running job count would exceed the configured maximum. This prevents new assignments from being sent to an agent that is already full.
- Agent side (polling). The agent also enforces the cap locally. It polls the console for newly assigned jobs every 15 seconds. If it is already running the maximum number of jobs, it defers all queued assignments until the next poll cycle in which a slot is free.
As a result, the agent does not start a job that would exceed its cap, even if the console over-assigns.
A lower value reduces concurrency right away. The agent stops accepting new work from the queue at its next 15-second poll. A higher value takes effect within one poll cycle.
The console delivers the override through the heartbeat settings channel, so changes reach the agent automatically without a restart.
Jitter settings
| Setting | Description |
|---|---|
| CH Jitter | The agent adds a random delay in the range [0, CH Jitter) before each scheduled ClickHouse INSERT (copy-log flush and metrics flush). This staggers write bursts across a fleet of agents so they do not all hit ClickHouse at the same time on the shared 30 s flush schedule. Set to 0 to disable jitter. |
| Step Log Jitter | The agent sends job step-log updates to the console every 30 s plus a random offset of up to this value. For a 300-agent fleet with the default 60 s jitter, the step-log POSTs spread over a window of about 90 s (about 3 to 4 requests per second) instead of all arriving at once. Reduce this value only if you need near-real-time step visibility on short-running jobs. |
Drain Mode
Drain mode stops the dispatcher from sending new jobs to an agent. Jobs that are already running finish normally. This is the correct way to take an agent offline for maintenance without disrupting copy runs in progress.
When drain is active, the agent card shows a yellow DRAINING badge. The Current Job Count keeps decreasing as running jobs complete. When it reaches 0, the agent is safe to stop or reboot.
| Button | Description |
|---|---|
| Drain | Enables drain mode for a single agent. |
| Undrain | Returns the agent to normal rotation after maintenance is complete. |
JVM profile changes wait for running jobs
You can assign a different JVM profile to a copy agent with the JVM Profile picker on the Live Agents tab. This restarts the agent with the new JVM flags.
The agent applies the change only when no copy job is running on it. While a job runs, the agent keeps its current profile, writes a single log message that the change is deferred, and checks again on every heartbeat. Running copies are not interrupted.
To apply a profile to a busy agent sooner, Drain it so that it gets no new jobs. Jobs started with Run on set to this agent are the exception. After its running jobs finish, the next heartbeat applies the profile and the agent restarts. Undrain the agent when it is back ONLINE.
No new job starts on the agent at the moment it restarts for a profile change or a self-upgrade. A job assigned to it at that moment is not started, and the console offers it again.
Crash Timeout
The crash timeout controls how long the console waits before it automatically fails a job whose assigned agent has gone silent. During the wait, the job sits in agent_pending_report state and is highlighted in the live queue.
| Setting | Description |
|---|---|
| Global Default | Shown as an amber pill in the action bar. It applies to all agents that do not have a per-agent override. |
| Per-Agent Override | Set inline on the agent card. Enter a value in minutes, or leave it blank to use the global default. |
| Set Crash Timeout (All) | A bulk action that applies one override value to all copy engine agents at once. It is useful after a system-wide policy change. |
For the per-agent override, use a shorter value for agents with fast restart times. Use a longer value for agents that run particularly large jobs, so that a temporary network blip does not fail an expensive run too early.
Distribution Plan
The Auto Distribution Plan is not an action on this tab. It opens when you start several copy jobs at once, for example with Sync Group Now or a bulk resume in Copy Jobs - Live View & History. It previews how the new jobs will be assigned to the available agents.
The plan uses anti-affinity logic:
- Heavy buckets (large or high change rate) are spread across agents.
- The other jobs go to the least-loaded agent.
This avoids routing too many jobs to a single agent and reduces the impact if one agent fails. Only ONLINE agents that are not in drain mode are included.
Review the plan before you confirm. You can dismiss it without starting any job.
Workflows
Reboot an agent for maintenance
- Find the agent in the registry.
- Click Drain. The badge turns yellow.
- Watch the Current Job Count drop to 0.
- Reboot or update the agent.
- When the agent reconnects and its status returns to
ONLINE, click Undrain.
Rolling agent upgrade with no downtime
- Drain agent A and wait for Current Job Count to reach 0.
- Perform the upgrade.
- Verify that the status is
ONLINE, then click Undrain. - Repeat for each agent, one at a time.
Do not drain more than one agent at a time if the remaining agents cannot handle the full job load.
Apply a JVM profile to a busy agent
- Assign the profile with the agent's JVM Profile picker on the Live Agents tab. While copy jobs run on the agent, it keeps its current profile.
- Click Drain so that the agent gets no new jobs, except jobs started with Run on set to this agent.
- Watch the Current Job Count drop to 0. At its next heartbeat, the agent restarts with the new profile.
- When the agent is
ONLINEagain, click Undrain.
Respond to an OFFLINE agent with running jobs
- Check Last Heartbeat to see how long the agent has been silent.
- If the agent is expected to reconnect, for example after a momentary network disruption, wait for the crash timeout to expire. The jobs then auto-fail and are re-queued.
- If the agent will not recover quickly, set its Crash Timeout Override to a low value. This speeds up the auto-fail and re-dispatch of its jobs.
- When the agent is confirmed dead, remove it from the registry to keep the display clean.
Spread jobs more evenly after adding a new agent
- Register the new agent and verify that it is
ONLINE. - Start the next syncs as usual, for example with Sync Group Now. The Auto Distribution Plan includes the new agent.
- Review the suggested job-to-agent assignments.
- Confirm to start the jobs across all available agents, including the new one.
Jobs already assigned to other agents are not moved.
Tips
An agent that shows ONLINE but has a Last Heartbeat more than a few minutes old may have network issues, even if it has not crossed the crash timeout threshold. Investigate before you assign it new jobs.
Setting Max Concurrent Jobs too high on a single agent can make that agent a bottleneck. Spread load across more agents before you increase individual agent limits.
Drain mode is not the same as stopping an agent. The agent process keeps running and continues to process its current jobs. Only new dispatch is blocked.
Set Crash Timeout (All) overwrites the per-agent override on every copy agent, including agents that had their own value. Leave the value blank to clear all overrides so that every agent uses the global default.
If all agents show OFFLINE at the same time, the most likely cause is a console-side connectivity issue or a network policy change, not a problem with the agents themselves.