Live Jobs
Overview
The Live Jobs tab shows all copy engine jobs currently in flight. These are jobs that are:
- Running
- Auto-paused because of a 503 throttle holdoff
- Waiting for an agent to reconnect (
agent_pending_report)
Pending (queued) jobs are not displayed here. They appear only in the Queue tab.
This is the real-time view of the copy engine pool. It is the first place to look when a sync run is not progressing as expected.
Capacity Bar
The bar at the top reads Running: N / M · X queued.
| Value | Meaning |
|---|---|
| N | The number of actively running jobs. |
| M | The total slot capacity across online agents. |
| X | The number of pending jobs waiting in the Queue tab. Pending jobs are counted separately and do not inflate the running percentage. |
The bar color shows the state of the pool:
| Color | Meaning |
|---|---|
| Green | There is headroom. |
| Amber | Running jobs are at 100% of capacity and pending jobs are waiting. |
| Red | One or more agents are in agent_pending_report. |
Pill counts show how many jobs are in each state: running, pending, auto-paused, and on offline agents. An amber Auto-Paused pill appears whenever one or more jobs are in 503 holdoff.
Per-agent summary strip
A summary strip above the filter bar shows each agent as a blue pill with the agent name and its number of active vault jobs. Jobs with no copy agent assigned yet are counted under No agent.
Click a pill to filter the list to that agent. Click it again to clear the filter.
Failure Column
The Failure column sits between Status and Copy Engine. It shows why a job failed as a badge. The cell is blank when no attribution applies.
| Badge | Meaning |
|---|---|
| Canceled | An operator canceled the job. |
| Watchdog: never started | The job stayed pending and was never picked up. |
| Watchdog: timeout | The job's agent went offline or stopped reporting. It did not reconnect and report the job's outcome within the crash timeout. The job was in agent_pending_report. |
| Watchdog: re-queued | The watchdog moved this pending job back to the queue because its assigned agent went offline. The job continues. This is not a failure. |
| Engine error | The copy agent reported an error. |
| Ransomware Event | The ransomware-detection gate blocked or canceled the job. |
| No copy agent | The job failed before it started because no copy agent could take it. Either no agent was online and undrained (Auto), or the one agent the job was pinned to was offline. The job's error says which. |
Failed jobs leave this view, so you normally see the No copy agent badge only when you look up a job with Find by ID.
Agent Column
The Agent column shows the copy agent the job is assigned to. A copy job with no agent shows —. Hover over it to see why:
- Waiting to be dispatched to a copy agent means the job is pending.
- No copy agent recorded for this job means, for example, that the job failed with No copy agent, or that its agent went offline and the watchdog cleared it.
Copy jobs run only on copy agents, never on the console. Other job types found with Find by ID, such as ObjectScale ingest, run on the console and show Console.
Job States
| State | Meaning |
|---|---|
running | The agent is actively copying objects right now. |
pending | The job is assigned to an agent but has not been acknowledged yet. It moves to running on the agent's next poll. |
auto_pause | The source or target returned an HTTP 503 "please reduce your request rate" (SlowDown or Service Unavailable). The agent suspended the job and waits a random holdoff period of 30 seconds to 5 minutes before it resumes automatically. |
agent_pending_report | The agent was running this job but disconnected before it reported the outcome. The console is waiting for the agent to reconnect. |
For auto_pause, no operator action is needed unless the job stays paused for an unusually long time. A long pause may indicate a persistent service outage rather than a temporary throttle. You can cancel an auto-paused job at any time.
For agent_pending_report, an amber section at the top groups all such jobs together. It shows a countdown to the crash timeout, when the job is failed automatically.
Filters
| Filter | Description |
|---|---|
| Search | A debounced free-text match against configuration name, bucket, device, or namespace. Disabled while ID search is active. |
| Find by ID | Looks up jobs directly by ID. See the details below. |
| Status | Narrows the list to running, pending, auto-paused, or agent_pending_report jobs. |
| Priority | Shows only CRITICAL, HIGH, NORMAL, or LOW tier jobs. |
| Agent | Shows only jobs assigned to a specific copy engine. |
Find by ID
Type one or more job IDs separated by commas, for example 17304 or 17304, 16567, 16528. Spaces around commas are ignored. Non-numeric tokens are silently discarded.
After a 300 ms debounce, the backend looks up each job directly by ID regardless of job status. The results can include running, pending, completed, or failed jobs, not just active ones.
A blue banner shows how many of the requested IDs were found. If any are missing, the jobs do not exist or have been purged.
The status, priority, and agent filters and pagination are suppressed while ID search is active. Click the × button in the field, or switch tabs, to return to normal mode.
Bulk Priority, Cancel, and Re-submit
Select rows with the checkboxes and use the bulk action bar to change priority, cancel, or re-submit (resume) the selected jobs at once. Higher-priority jobs move to the front of the agent's execution queue on its next poll.
Re-submit routes failed or canceled Full Copy, Incremental, and Batch Sync jobs through their respective resume paths.
Re-submit rules
- Re-submit is refused with the message "An active agent job already exists for this config…" while another job of the same policy is running on a live copy agent. This includes the selected job itself if it is still running normally. Cancel it first.
- Jobs stuck on an offline agent do not count. You can re-submit stuck Full Copy and Incremental jobs.
- If several stuck jobs belong to one policy, the first one restarts and the others are refused until it finishes.
- Batch Sync jobs are re-submitted as a Batch Sync resume of their remaining windows. This is refused while any job of the policy is still active, so cancel a stuck Batch Sync job first.
Select All
Select All works in two phases. The first header-checkbox click selects only the loaded rows and shows a Select all N link in the blue banner.
Click the link to fetch every matching ID from the server. The bulk action then applies to every match, including rows that are not on the current page.
Auto-Refresh
The Live Jobs tab refreshes every 30 seconds. A countdown next to the Refresh button shows the seconds until the next poll. A query-latency badge in milliseconds is displayed alongside it.
Tips
Jobs in the amber offline-agent section are failed automatically after the crash timeout. Undrain or restart the agent before the timer expires to avoid losing job history.
Use the agent filter to check whether one agent carries a disproportionate load. If so, check its configuration in Cyber Vault > Agent Fleet Management.