Copy Jobs — Live View & History

Overview

The Live View & History tab shows all copy jobs, running or in history. This includes full syncs, incremental syncs, and batch sync.

From here you can:

• Monitor job progress.
• Cancel running jobs.
• Resume failed jobs.
• Run bulk operations across multiple jobs at once.

The tab refreshes automatically. See Auto-refresh.

Table columns

Column	Description
Checkbox	Selects the row for bulk actions.
ID	The job ID. An amber ↩ #N badge shows when the job was created to resume another job, so you can see the lineage at a glance.
Actions	Cancel (×) for running or pending jobs. The job is marked canceled at once and the copy agent is told to stop. Resume (↩) for failed or canceled Full Copy, Incremental, and Batch Sync jobs.
Type	Full Copy, Incremental, or Batch Sync. Incremental jobs show a slot badge (S1, S2, …) for windows shorter than 24 hours.
Config	The vault config name that triggered the job.
Trigger	Auto means the vault agent started the sync on its schedule, or another automatic system job started it. Manual means Sync Group Now, Sync All, Run, Resume, or MCP tools.
Tags	Large and High Churn inventory badges. They reflect the current inventory state, not the state when the job ran.
Status	The raw status value. See Status values. A skipped job is displayed as Not Started.
Failure	The failure attribution. See Failure column.
Copy Engine	The copy agent the job was dispatched to. See Copy Engine column.
Progress	Objects completed compared with the total, with a progress bar.
Duration	Wall-clock time for the job.
Started	When the job began.
Details	Opens the job detail flyout. Clicking anywhere on the row does the same.

Failure column

The Failure column shows why a job failed, when an attribution applies. It is blank otherwise.

Value	Meaning
Canceled	An operator canceled the job from the console.
Watchdog: never started	The job was stuck in PENDING.
Watchdog: timeout	The job timed out.
Watchdog: re-queued	A pending job was re-queued after its agent went offline. The job continues and is not failed.
Engine error	The copy engine reported an error.
Ransomware Event	The ransomware-detection gate blocked or canceled the job. See Setup & Settings > Storage > ECS > Eyeglass Device.
No copy agent	The job was refused or failed before it started because no copy agent could take it. The job's error message says which case applies.

For No copy agent, the cause depends on how the job was assigned:

• With Auto, no copy agent was online and undrained. Start or undrain a copy agent.
• When pinned to one agent, that agent was offline. Choose Auto or another agent, or start the pinned agent.

Copy Engine column

The Copy Engine column shows the copy agent the job was dispatched to. A — means no copy agent is recorded for the job. Copy jobs run only on copy agents, never on the console.

For a job that never started, hover over the — to see the reason:

Hover text	Meaning
no copy agent could take it	The Failure column shows No copy agent. The job's error says whether no agent was available or the chosen agent was offline.
blocked by the ransomware-detection gate	The Failure column shows Ransomware Event.
canceled before it was dispatched	The job was canceled before dispatch.
waiting to be dispatched	The job is pending.
blocked before dispatch	The job is Not Started.

A job that started can also show — with the text "No copy agent recorded for this job". This happens when the job's agent went offline and the watchdog cleared the agent while it tried to reschedule the job. Check the Failure column, which usually shows a Watchdog badge.

Filters

Filter	Description
Type	Shows Full Copy, Incremental, or Batch Sync jobs only.
Status	Shows jobs with a specific status, such as running, completed, or failed.
Failure reason	An independent filter, applied together (AND) with Status, Type, and date range. It narrows to jobs with a specific failure attribution, including Ransomware Event and No copy agent. Select "All Failures" to clear it.
Action type	"Resume-type only" shows only jobs that were created as resumes of a previous job. The filter is server-side and indexed. Use it to find all historical resume attempts. It combines with the Failure reason filter. For example, "Resume-type only" plus "Ransomware Event" shows only resumed jobs that the ransomware-detection gate also blocked or canceled.
Find by ID	Comma-separated job IDs for direct lookup. It bypasses the type, status, and pagination filters. It shows how many of the requested IDs were found.
Agent pills	Click a pill to filter to that agent's jobs, client-side, within the loaded page. Jobs with no copy agent recorded are counted under No agent.

Multi-select, bulk resume, and bulk cancel

Click the checkbox on any row to select it. The header checkbox selects or deselects all visible (loaded) rows.

When rows are selected, a bulk action bar appears with the count of selected rows. It offers these actions:

• Re-submit as Resume appears if any selected jobs are failed or canceled Full Copy, Incremental, or Batch Sync (CYBER_VAULT_COPY_INCREMENTAL_MULTI) jobs.
• A red Cancel Selected button appears alongside it when one or more selected rows are running or pending. Only those two states can be canceled. The button is hidden when no eligible jobs are selected, and it shows a badge with the count of cancelable rows.

Re-submit as Resume

Batch Sync resume routes to a separate endpoint. It picks up only the remaining unsynced windows from the failed batch.

While the Incremental Sync switch is off, incremental and Batch Sync jobs are left out and listed as "skipped — incremental sync is disabled". The button is grayed out when only such jobs are selected.

1. Click Re-submit as Resume.
2. The Auto Distribution Plan dialog opens. It is the same planner used for bulk sync from Full Data Sync Management. It shows how the backend distributes the resume jobs across available copy engines.
3. Confirm. Each selected eligible job is re-submitted as a new resume job. Per-job engine assignments from the planner are honored.

Select all matching jobs

When you select the header checkbox and the result set has more rows than the loaded page, a blue banner appears with a Select all N link. Click it to select every matching job ID under the current filters. The console fetches the IDs from the server with GET /api/jobs/history/ids.

The banner then reads "All N jobs selected — actions apply to all, including rows not visible".

warning

For very large result sets, the server can return a truncated ID list. A warning is then shown, and the bulk action operates on the truncated set only.

Load more

The table starts with the 25 most recent jobs. Click Load more at the bottom to fetch additional rows in increments of 25. Changing a filter resets the list to the first 25.

Auto-refresh

The table refreshes every 3 seconds while any listed job is running or pending, and every 10 seconds otherwise.

A countdown timer next to the Refresh button shows the seconds until the next refresh. A query-latency badge, in milliseconds, is shown alongside it. The badge appears in darker, bolder text when the query takes longer than 500 ms.

A per-agent active-job summary strip sits above the filter bar. Each blue pill shows an agent's active vault copy job count. Click a pill to show only that agent's loaded jobs. Click it again to clear the filter.

Resume a single job

Click the amber ↩ button in the Actions column on any failed or canceled Full Copy, Incremental, or Batch Sync job. You can also click Resume Failed Job in the flyout of a failed job.

The resume creates a new job that picks up where the failed job stopped, using object-level checkpoints. The detail flyout shows the resume lineage: which job was resumed and which new job replaced it.

A Full Copy resume keeps the original job's copy mode: Standard, Smart Skip, or Target Skip. It also keeps the same Target Skip tiers and Tier 1.5 cutoff.

When a resume is refused or disabled

• While the group is in Vault Mode, a manual resume is refused, because the vault agent schedules that group's syncs.
• While the Incremental Sync switch is off (Cyber Vault > Incremental Sync > Settings), ↩ and Resume Failed Job are grayed out on Incremental and Batch Sync jobs. The hover text is "Enable incremental sync first". Full Copy jobs can still be resumed.

A resume is also refused with this message while a copy job for the same policy is still running on a live copy agent, for example a Sync All:

An active agent job already exists for this config. Wait for it to complete or cancel it first.

Two jobs must not copy the same bucket at once. Keep these points in mind:

• The refusal includes the job you resume, if it is still running fine. Cancel it first.
• Jobs stuck on an offline agent do not count, so stuck jobs can be re-submitted.
• If you resume several jobs of one policy together, the first starts and the others are refused until it finishes. For failed windows, use Batch Sync resume instead.
• A Batch Sync resume is refused while any job of the policy is active, including a stuck batch job. Cancel it first.
• The refused job stays as it was and can be resumed later.

Large objects continue on the same copy agent

When a job fails or is canceled while a copy agent is copying an object of 1 GiB or more, the agent keeps that object's multipart upload on the target. The job's step log shows UPLOAD KEPT. The upload is also kept when the agent restarts or crashes in the middle of the copy.

The upload is not kept in these cases:

• No part had reached the target yet.
• The upload was left by an earlier run and failed all its retries again for a reason other than the network. That upload is aborted, so the next run starts afresh.

A resume continues each kept upload from its first missing part instead of copying the object again. This works when both conditions are true:

• The resume runs on the same agent.
• The source object (ETag and size) is unchanged.

The upload keeps the part size it started with, even if the policy's Min Part Size has changed since.

Otherwise the old upload is aborted (UPLOAD DISCARDED) and the object is copied from part 1. The same applies to a re-run, the policy's next full copy, or an incremental copy of the object on that agent.

The ↩ button lets the console choose the agent (Auto). With several copy agents, the resume can land on another agent, which copies the object again.

To be sure the same agent is used, run a full copy with Copy mode: Smart Skip from Full Data Sync Management > Run. Set Run on to the agent shown in the failed job's Copy Engine column.

A continued object's Copy Log row shows this text:

resumed from part K/N (uploadId …, left by an earlier run; source MD5 not computed)

Full Copy resume

With Auto chosen, the resume goes back to the copy agent that ran the job. This applies if that agent is online, not draining, and has room. Otherwise any copy agent takes it.

The resume continues from the checkpoint of the job you resume. It never uses a checkpoint from another job of the same policy. The checkpoint is read from that agent's local store and from ClickHouse, using the newer of the two per partition.

If that job ended before it saved a checkpoint, for example a resume that failed at validate, the checkpoint of the job it resumed is used.

If the agent has no local copy of that checkpoint, usually because the resume runs on another agent, and ClickHouse cannot be read, the resume waits about a minute. It then fails with the following message instead of starting the whole bucket again. Resume again once ClickHouse is reachable.

Resume of job #N stopped: its checkpoint could not be read …

The Prefix_discovery step's details and the job summary show what the resume found:

• resumeFromJobId
• checkpointJobId
• checkpointSource
• partitionsMatched
• partitionsWithoutCheckpoint
• requeuedFailedObjects

The step log has a line such as:

RESUME of job #744: checkpoint from job #744 (ClickHouse, saved 2026-10-15 17:31 UTC) — matched 509/509 partitions, 0 start from the beginning; 719 previously failed object(s) queued for retry

The partition plan is scanned again, so a bucket that has grown can produce partitions with no checkpoint. Those partitions start from the beginning. If the job ran without Smart Skip, they skip only objects that the resumed job, or the jobs it resumed, already copied. They never skip objects copied by older jobs, in case the vault bucket was emptied in between.

Objects that failed in the resumed jobs and were never copied since are retried at the end of the walk, when the walk itself will not list them again. A resume reads up to 100,000 of them.

Batch Sync resume

Each window skips only the objects already copied by the job that last synced that window. Windows that were never synced are copied in full.

info

Known limit: if the same object changed in two windows of one batch job and that job failed between them, the later window can still skip it on resume or retry. The vault then keeps the older version. Run a Full Copy with Smart Skip on that bucket if you need to be sure.

Status values

Status	Meaning
running	The job is actively executing on a copy agent.
pending	The job is queued and waiting for an agent to pick it up.
completed	The job finished successfully.
failed	The job hit an unrecoverable error.
canceled	The job was manually canceled. Cancel is immediate. There is no intermediate "canceling" state, and the copy agent is told to stop.
agent_running	The job was dispatched to a copy agent and is running there.
agent_pending_report	The copy agent went offline, or the console restarted, while the job ran on the agent. The console waits for the agent to report. If no report arrives within the agent's crash timeout, the job is rescheduled or marked failed.
auto_pause	The job is temporarily paused. The source or target answered 503 (slow down), so the copy agent pauses all jobs on that agent for a random hold-off of 30 seconds to 5 minutes. They then carry on by themselves.
skipped (Not Started)	A pre-flight gate blocked the job before dispatch and it was never assigned to an engine. Currently the only gate is Eyeglass ransomware detection. Check the Failure column for the reason.

Job detail flyout

Clicking any row opens the slide-in Job Detail Flyout, titled Job #N Details. It provides:

• Full step logs.
• Live status updates, polled every 3 seconds while the job is running, pending, auto_pause, or agent_pending_report.
• Resume, cancel, view-log, and view-trends actions.

A completed job shows all of its steps completed.

Step logs

A Batch Sync job has three steps: validate, copy, and summary. The copy step's details show the batch totals: windows, copied, deleted, skipped, and failed.

A copy agent sends only each step's newest log line, about every 30–90 seconds. A step's log is therefore a sample of what the agent wrote:

• The copy step of a multi-window batch can show only the last window's line.
• While a job runs, its newest line can appear up to about a minute late.
• Each step's latest line is sent before the step ends.

The job's copy log has the full per-object record.

Each step's log has a Copy button next to its N log lines toggle. It copies that step's full log to the clipboard in chronological order, so you can paste it into a ticket or chat without expanding it first. A green Copied checkmark confirms the copy.

Copy event lines

A full, incremental, or Batch Sync copy job writes what happened to the network and to large uploads into its own step log. For a full copy this is the bucket_walk step. For an incremental or Batch Sync copy it is the copy step.

These lines are sent to the console one by one as they happen, while the console is reachable. They are not sampled like the step's other lines.

Network lines

WAITING FOR NETWORK: the source cannot be reached (…) — retrying for up to 15 min
NETWORK RECOVERED: the source answered again after 2m 05s

• WAITING FOR NETWORK (or the target) means requests to that side never reached the server. Causes are DNS failure, connection refused or timed out, or no route.
  • The copies wait and retry for up to 15 minutes.
  • One line is written per outage, however many objects hit it.
  • The target and each source endpoint have their own outage and their own 15 minutes. In a geo-federated job, a VDC that is down uses up its budget even while another VDC answers.
  • If the same side fails again within a minute of a reported outage, the new outage gets a line only once it has lasted a minute.
• NETWORK RECOVERED means the outage ended and the copies carried on. Any answer ends the outage, including an error answer such as 503, because the network works again.

Multipart upload lines

Line	Meaning
RESUMED: <key> from part K/N (same upload, uploadId …)	After a failed attempt, a multipart object continued its upload instead of starting again. The variant (upload left by an earlier run, …) means it continued an upload that an earlier job on this copy agent kept.
RESUMED: <key> — all N parts are already on the target, completing the upload (…)	Only the final Complete call was left.
UPLOAD KEPT: <key> — K/N parts are on the target (uploadId …); the next run continues from there (…)	The copy of an object of 1 GiB or more stopped, because its retries failed or the job was canceled. A later run on the same copy agent continues it. See Large objects continue on the same copy agent.
UPLOAD DISCARDED: <key> — uploadId … cannot be continued: …; copying from part 1	A kept upload, from an earlier attempt or run, cannot be continued. The old upload is aborted and the object is copied again with a new upload. The reason follows the colon.
ABORT FAILED: <key> — uploadId … is still open on the target with its parts: …	An upload that cannot be used any more could not be aborted. Its parts stay on the target until the upload is aborted. The line gives the upload ID. A canceled job tries each abort once, because it is stopping.
COMPLETE VERIFIED: <key> — the Complete call failed (…), but the object on the target matches the source (uploadId …)	The object counts as copied and is not copied again.
RESUME JANITOR: aborted N multipart upload(s) in <bucket> that no run continued for N days	See below.

The reason in an UPLOAD DISCARDED line can be any of these:

• The source object changed (ETag or size).
• The source sends no ETag, so a change could not be detected.
• The recorded part size does not fit the object.
• The target cannot list the upload's parts or refuses to, for example 403 without the s3:ListMultipartUploadParts permission.

The RESUME JANITOR line appears at the end of a full, incremental, or Batch Sync copy that was not canceled. The agent aborts the uploads it kept for the job's target bucket that no run continued:

• For 7 days after a full copy.
• For 35 days after an incremental or Batch Sync copy.

Incremental and Batch Sync copies copy only the objects that changed, so they never continue an upload a full copy kept. The next weekly or monthly full copy still can.

The janitor tries each upload once. It stops after 2 minutes or after 3 failed aborts in a row. An upload whose abort failed is tried again only after another 7 or 35 days.

Live Copy Summary

For copy jobs (full, incremental, and batch sync), the flyout includes a Live Copy Summary panel. It contains the items below.

Object counts

The counts are Copied, Deleted (incremental only), Skipped, Failed, Single-Part, and Multipart.

Sometimes objects are deleted at the source after they were listed, for example data the source application expires on its own schedule. In that case:

• The Skipped tile shows N deleted at source under its count.
• The line below the tiles shows Deleted at source: N.
• The job's step log lists each object as SKIPPED (deleted at source): <key>.

Nothing was left to copy for those objects. They are tried once, never retried, and counted as skipped, not failed. A job whose only issue is such objects completes instead of failing with "N objects failed to copy". A missing source bucket is still a failure.

Throughput tiles

The tiles are Transferred bytes, Elapsed seconds, MB/s, Mbps, and Peak Threads.

A large object that continued an earlier upload counts with its full size when it completes. A resumed job can therefore show more bytes, and a higher rate, than it actually sent.

Time-series charts

The charts are Throughput (MB/s), Objects/sec, and a Latency chart with average and p95 GET and PUT latencies. They use 10-second windows covering the last hour of the job.

The copy agent sends them while the job runs and saves them with the job's summary, so they stay available after the job ends. Keep these points in mind:

• A large (multipart) object counts as its parts upload. Each part's bytes go into the window the part finished in, and its GET and PUT points are the part's read and upload. The charts therefore follow a long copy while it runs.
• Parts already on the target from an earlier run are not counted. A resumed job's charts can add up to less than the Transferred tile.
• Objects/sec counts an object when it finishes.
• While a job runs, the charts show Collecting… until the first window.
• A Full Copy that ended failed keeps its charts, because the copy agent saves its summary with the failure.
• A job that was canceled, or failed before copying, and an incremental or Batch Sync job that failed save no charts. Their charts show the copy agent's last windows for about 5 minutes after the job ended, then No data.

Other panels

Panel	Description
TPS Rate Limit	Shows Allocated Budget (ops/s) and Budget Utilization % (color-coded green, amber, or red) when the job runs under the global TPS limiter.
JVM Heap	Shows Used, Max, and Heap Utilization % of the copy agent's JVM. Values are live while the job runs, then at completion. Color bands are green, amber, and red at 50% and 75%.
TCP Kernel Health	Shows the Retrans, InErrs, Resets, OutRSTs, and Fails counters. Any non-zero counter is highlighted red.
Smart Skip badge	Shown when a Full Copy was dispatched with Smart Skip enabled. For each object, the most recent copy-log record of the same config, source bucket, target bucket, and target prefix is compared with the current source ETag.
Target Skip Mode badge	Shown when the job was dispatched in Target Skip mode. It lists the tiers enabled for the job: Tier 1, Tier 1.5 with cutoff date, Tier 2, and Tier 2.5.
Target Skip per-tier breakdown	A gray sub-panel with five tiles (Tier 1, Tier 1.5, Tier 2, Tier 2.5, Copied), shown only when the job ran in Target Skip mode. The numbers come from a one-shot ClickHouse aggregate. The panel does not auto-refresh, so close and reopen the flyout to refresh it.
Resume traceability banners	"Resume of Job #N" appears at the top when the job is a resume of an earlier failure. "Superseded — resumed as Job #N" appears on the original failed job. Each banner has a link that loads the linked job into the same flyout.
Awaiting Report banner	Visible when status is agent_pending_report. It explains that the console restarted while this job was running on an agent and is waiting for the agent to reconnect.
Windows strip	For Batch Sync jobs, a strip below the live summary shows every window's status (running, done, partial, failed, or pending) with object counts per window.

Adaptive partition table

The adaptive partition table shows the per-prefix copy progress with a checkpoint key.

An amber Partition count capped at 512 banner appears above the table when the source bucket has more prefixes than the discovery cap of 512. Deeper prefixes are absorbed into their parents. Every object is still walked and copied. This is a performance optimization only.

An amber Byte-range partitioning active banner appears when the source bucket's /-delimiter scan found too few partitions, for example a flat keyspace with everything under one long shared prefix. The bucket was rescanned and split by byte value rather than directory structure. Every object is still copied, but partitioned differently in the table.

The byte-range scan skips a prefix that every key shares, for example the obj- of obj-000000001.bin. It splits the biggest parts of the bucket first. On a large flat bucket it normally fills most of the 512-partition budget, so both banners appear together.

The Prefix_discovery step shows partitionMode: byte-range or partitionMode: delimiter.

In byte-range mode, rows carry a short tag next to the prefix:

Tag	Meaning
[low-byte]	The control-character and space range.
[non-ASCII] or [non-ASCII from U+XXXX]	The non-ASCII range. It is subdivided further when the bucket has enough non-ASCII keys, so more than one [non-ASCII from U+XXXX] row can appear for the same prefix.

The step log can also show these warnings:

• WARNING: byte-range scan could not split flat bucket … is the step's last log line when even the byte-range scan cannot split a large bucket. The copy still runs but lists the whole bucket in one stream.
• WARNING: byte-range scan: N branch(es) of bucket … have keys sharing a prefix too long to skip … appears when a shared prefix hits the skip limits. A shared prefix is skipped for at most 64 characters per part of the bucket and at most 1024 characters in all. The limits keep discovery quick on buckets with many long common names. Those parts can get fewer partitions than wanted, but every object is still copied.

ECS SSH Ingest jobs

For ECS SSH Ingest jobs, the Ingest Summary card grid in the flyout is dynamic.

Card	Description
Records Decoded, Records Inserted, Inserted to CH, Ingest Rate	Always shown.
Batch Size (gray)	Shown when set. It is the resolved ingestBatchSize actually used, so operators can confirm the configured batch size is in effect.
CH Errors (red)	Shown when greater than 0. The count of ClickHouse insert errors.
CH Retries (amber)	Shown when greater than 0. The count of successful retries of ClickHouse inserts.
Dropped (filtered) and Dropped %	Gray at 0, and colored when events were dropped. See below.
Threads (gray)	Shown only when greater than 1. It is the planned total ingest parallelism: ObjectScale node count × Ingest Threads Per Node. It shows as Threads (capped at N) when the per-node value was capped.

Dropped (filtered) is the count of decoded S3 events removed before insertion because their HTTP method is not kept by Setup & Settings > Advanced Settings > Audit Event Ingestion Filter. By default PUT and DELETE are kept. GET (including bucket listings), HEAD, and POST are dropped.

Read-only traffic is filtered out on purpose. This is expected behavior, not data loss. On a read-heavy device, Records Decoded can be much larger than Records Inserted. The gap between the two equals this Dropped value.

A Log Rotation Detected banner sits above the per-file table when one or more node files were resumed from an in-progress offset. It lists each rotated file and how many records came from the active file versus the rotated tail.

A File Ingest State — Last 24 Hours table at the bottom shows all node files active or completed today. It links to the /ingest-offsets page, which shows the full history.

Tips

• Use Action type = Resume-type only combined with Status = failed to find resume attempts that themselves failed.
• The ↩ #N ID badge and the job detail flyout both show resume traceability in both directions: which job was resumed, and which new job resumed it.
• A high skipped count on incremental jobs is normal. Objects already in the vault are skipped.
• If the Alarms view shows only one CV-115 ransomware-detection alarm but you suspect more jobs were affected, filter this tab's Failure reason to Ransomware Event. Every affected job gets its own permanent row here, regardless of how many alarm events were raised for that detection.
• A job with failed objects and an overall "completed" status finished, but some objects could not be copied. Open the Copy Log from the flyout and filter to FAILED status.

Full Copy listing errors

When listing part of the bucket (a partition) fails with a temporary error, the copy agent retries that partition with backoff for up to 15 minutes. Temporary errors include an S3 500, 502, or 504, a timeout, or a DNS failure. The step log says Partition [p] LIST failed — … — retrying and later LIST recovered.

A partition that still fails is logged at once as Partition [p] LIST FAILED …. It waits for the others to finish and is listed once more from where it stopped.

If that fails too, the other partitions still finish and the summary is written. The job then fails with this message:

N of M partition(s) failed (p, …): … — resume the job to finish them

Resume the job. The resume continues those partitions from their checkpoints.

Full Copy failed objects

Objects whose copy failed during the walk are retried once every partition has finished.

• An object deleted at the source by then is skipped like any other deleted object (source deleted: …, counted under Deleted at source).
• An object the vault already holds with the same ETag and size is counted as target-skipped. An example is a write that timed out after it landed.
• Neither case counts as failed.

The last line of bucket_walk, its details, and the summary show this line:

Failed objects retried at the end of the walk: N of this walk > copied, already on the target, deleted at the source, still failed

If a partition failed, the last line names the partitions instead.

Only objects that are still failed fail the job. The Live Copy Summary counts each object once. The Copy Log's cards count rows, so an object that failed in the job's own walk and was then retried also counts under Failed there.