Copy Jobs

The Copy Jobs tab gives you a complete view of copy activity. It shows the full historical record of every job that has run. It also shows a live queue of jobs that are dispatched, pending dispatch, auto-paused, or waiting for an agent to report back.

Use it to monitor active copy runs, investigate failures, review performance trends, and manage job priorities.

info

The Copy Jobs tab has two sub-tabs.

• Live View & History is the full job table with bulk actions.
• Copy Job Active Queue Management is the real-time dispatch monitor.

Each sub-tab has its own help page. Open the help panel from either sub-tab for the most relevant content.

Overview

Copy jobs are the individual units of work run by copy agents. Each job is one copy run for one vault config. It is either a full sync of all objects in the source bucket or an incremental sync covering one changelist window.

Jobs are started in one of two ways:

• In Vault Mode, the vault agent starts them on its schedule.
• In Preparation Mode, you start them manually with Sync Group Now, Sync All, a window's buttons, or a full copy from Full Data Sync Management.

A job moves from pending dispatch, through running, to a terminal state: completed, failed, or canceled.

Copy modes

When you dispatch a full-copy job, either single or bulk, a Copy mode radio button chooses how the worker decides which objects to copy and which to skip.

Every mode produces the same target, which is a complete copy of the source. The modes differ in how much work they do to recognize content that is already present.

Mode comparison at a glance

Mode	What it skips	How it decides	I/O cost on already-present data	Primary use case
Standard	Nothing	Always copies	Highest. Re-uploads every byte.	First copy, recovery from a wiped target, distrustful re-baseline
Smart Skip	Objects whose source ETag matches a prior copy by the same config	ClickHouse comparison only	Lowest. No HEAD or GET against the source or target for skipped objects.	Resume after a partial run, periodic re-syncs, change-only catch-up
Target Skip	Objects already on the target (4-tier match)	Source listing against target listing, with optional HEAD and range-hash probes	Medium. One target listing per partition, plus a HEAD or GET for each Tier 2 or 2.5 candidate.	Migration from an ecs-sync-based deployment whose target bucket already holds the data

Standard (mode = NONE)

Standard copies every source object unconditionally. It does no skip evaluation, no listing of the target, and no ClickHouse lookups.

Choose Standard when:

• The target bucket was wiped, the data is missing, or you do not trust its contents.
• You are running the first copy of a new policy and there is nothing on the target yet.
• A previous job ended in an inconsistent state and you want a clean re-baseline.
• The bucket is small and re-copying costs less than evaluating skips.

Every object goes through the full copy path. Every successful copy writes the x-amz-meta-source-etag metadata marker on the target object. See Metadata markers written on every copy.

This is true for Standard mode and for the Tier 3 fallback in the two skip modes. The marker lets the next Target Skip job against the same target recognize these copies through Tier 2.

Smart Skip (mode = SMART_SKIP)

Smart Skip skips objects whose source ETag matches a previously recorded copy in ClickHouse (cyber_vault_copy_log). It issues no HEAD or GET against the source or target for skipped objects. It is a ClickHouse query plus a per-page in-memory comparison.

Which copies count as the baseline

Smart Skip uses the all-jobs baseline. Any prior job of the same config with status='copied', status='skipped', or status='target_skipped' is a valid baseline. Only the most recent such record for each object key is compared with the current source ETag.

Only copies made by this config to its current target count. The config name, source bucket, target bucket, and target prefix must all match.

Configs in different namespaces often copy a same-named bucket, for example a container registry, to a same-named vault bucket in their own vault namespace. A copy made by one of them never lets another skip an object.

As a result:

• Renaming a config, or changing its source bucket, target bucket, or target prefix, starts a new baseline. The next Smart Skip run copies every object once. Later runs skip again.
• Copies made by a deleted config do not count for a new config with a different name.

When to choose Smart Skip

• You are recovering from a partial run on the same source and target. Smart Skip avoids re-copying the parts that already succeeded.
• You are running a periodic re-sync to catch new or modified objects. Unchanged objects (same ETag as the baseline) are skipped. New or modified objects (different ETag, or no baseline row) are copied.
• You want the cheapest possible duplicate check. The per-object cost is a hash-map lookup, with no S3 round-trips for skipped objects.

Safety

Smart Skip compares the source listing ETag to the baseline source ETag. If the source object was re-uploaded since the baseline copy, S3 returns a different ETag and Smart Skip copies it. A different source ETag never matches a baseline ETag, so there is no false-skip path.

Skip rows in the copy log

Each skipped object writes one row to cyber_vault_copy_log with status='skipped' and an error_message of smart-skip: ETag matches baseline job #0. The #0 means the baseline is all earlier copies of this config, not one job.

An incremental sync writes smart-skip: ETag matches baseline (all-jobs). The skip row preserves the source ETag so that future jobs can use this skip as a baseline.

When the baseline lookup fails

The lookup runs once for each listing page of up to 1,000 objects. If ClickHouse rejects the lookup, times out, or cannot be reached, the objects on that page are copied instead of skipped. Nothing is lost, but the job copies more than it needs to.

The job reports the failure in these places:

• The bucket_walk step log shows the first failure with its reason, for example Smart Skip baseline lookup failed for bucket 'b' (1000 of 1000 keys unchecked): HTTP 500: ….
• The summary step log shows Smart Skip: baseline lookup failed for N page(s); M objects were copied without a skip check.
• The job summary details carry smartSkipFailedLookups (pages) and smartSkipUncheckedObjects (objects).

If a re-run of an unchanged bucket skips far fewer objects than it lists, and these counters are above zero, look at ClickHouse load, timeouts, or connectivity. Changed data is not the cause.

Target Skip (mode = TARGET_SKIP)

Target Skip compares each source object against the live target vault bucket and skips objects already present there.

Its primary use case is migration acceleration. This applies when you bring over a customer from an ecs-sync-based deployment whose target bucket already contains the data.

Choose Target Skip when:

• You are migrating from ecs-sync to AirGap for ObjectScale and the target bucket already holds most of the customer's data. Target Skip discovers that data and treats it as already copied, without re-uploading.
• You are recovering from a failed migration where the target was partially populated and you have no copy-log baseline to drive Smart Skip.
• You want to verify that the target carries the expected content. Tier 1 is a strict ETag and size comparison, so a job that target-skips most objects is evidence that the data already matches.

The 4-tier match

Tier	Type	What it does
Tier 1: strict ETag + size match	Always on, free	Skips when the source object's ETag and size match a target object at the same key.
Tier 1.5: date-trust skip	Opt-in, free	Skips when the source object was last modified before your cutoff date and the sizes match.
Tier 2: provenance marker probe	Opt-in	Issues a HEAD against the target object and checks its source-ETag marker.
Tier 2.5: range-hash content probe	Opt-in, default off	Samples three byte ranges from source and target and compares them.
Tier 3: copy	Fallback	Copies every object that no earlier tier skipped.

Tier 1: strict ETag + size match

When the source object's ETag and size match a target object at the same key, the object is skipped. The false-skip risk is negligible, because a matching ETag and size imply identical content.

Tier 1 alone handles same-scheme migrations, where the multipart part size is the same on source and target. No further probing is needed.

Tier 1.5: date-trust skip

When you provide a per-job cutoff date, an object is skipped if both of these are true:

• The source object's LastModified is strictly before the cutoff.
• The source and target sizes match.

It adds no extra requests, because both size and lastModified come from the source ListObjectsV2 response.

Correctness is attested by you. If the cutoff date is wrong, because some source bytes did change before that date, Tier 1.5 skips those objects incorrectly. Use it only when nothing was written to the target bucket before the cutoff and you can attest that no source bytes changed before the date.

Tier 2: provenance marker probe

Tier 2 issues a HEAD against the target object and checks for either of two metadata keys:

• The source-etag marker, written by every Tier 3 copy.
• The x-emc-source-etag marker, written by ecs-sync with Store Source Object Copy Markers enabled.

If either is present and matches the current source ETag, the object is skipped. See Metadata markers written on every copy.

Tier 2.5: range-hash content probe

For cross-scheme multipart objects at or above the minimum size, Tier 2.5 samples three byte windows (begin, middle, and end) from both source and target. It compares them by SHA-256 digest.

• It adds 6 × chunk MB of I/O per probed object.
• The defaults are 1 MB chunks and a 5 MB minimum object size.
• It is off by default. Use it only when the source bucket is read-only. See the Tier 2.5 risk advisory below.

Tier 3: copy

No skip matching runs at Tier 3. Any object that Tier 1, 2, or 2.5 did not skip is copied with a full source-to-target byte transfer. This is the same code path Standard mode uses for every object.

• Below the multipart threshold, the copy is a single PUT.
• Above the threshold, it is CreateMultipartUpload > UploadPart × N > CompleteMultipartUpload.

Every Tier 3 copy writes the x-amz-meta-source-etag marker. The next Target Skip job can then recognize this copy through Tier 2, even when the source and target multipart schemes differ.

Where the tiers are configured

The two opt-in probe tiers, Tier 2 and Tier 2.5, default to off in Setup & Settings > Advanced Settings > CyberVault Target Skip Mode — advanced defaults.

The dispatch popup inherits those defaults. When you select Target Skip, checkboxes appear that let you override them for the job.

Tier 1.5 has no Settings default. It is per-job only. Enable it, with a cutoff date, on every dispatch where you want it.

When to use Tier 1.5

Tier 1.5 is the cheapest skip in the cascade, with zero extra S3 requests. It trusts your cutoff date instead of verifying content, so it is safe only in specific scenarios.

Scenario	Recommended setup	Why
Migration from an ecs-sync-based deployment. Nothing was written to the target bucket before the cutoff.	Tier 1.5 on, with the cutoff set to the day before the previous product was installed. Tier 2 and Tier 2.5 are typically also on as a safety net for post-cutoff changes.	Tier 1.5 catches the bulk of the migration at zero per-object cost. Tier 2.5 still fires for objects modified after the cutoff.
Active source bucket. Users write to the source while the migration runs.	Tier 1.5 on only if nothing was written to the target bucket before the cutoff. Otherwise Tier 1.5 off. Tier 2.5 off. Rely on Tier 1, and on Tier 2 where markers exist.	You cannot safely set a cutoff unless that condition holds. Trusting LastModified would skip objects whose content changed since a cutoff you cannot verify. Tier 2.5 samples only three byte ranges, so it is for read-only sources only.
Mixed-history bucket. Some prefixes are frozen archive and others are active.	Tier 1.5 off and Tier 2.5 off.	Tier 1.5 uses a single global cutoff for each job, and per-prefix cutoffs are not available. If the cutoff is wrong for any subset, those objects are skipped incorrectly. The active prefixes also rule out Tier 2.5.
Re-running Target Skip after a previous copy by this product.	Tier 1.5 optional. Tier 2 on is usually sufficient.	After an object is copied, the target carries the x-amz-meta-source-etag marker. Tier 2 recognizes it and skips without a date cutoff.
Compliance-sensitive migration where false-skip risk must be as low as possible.	Tier 1.5 off and Tier 2.5 off. Use Tier 1 and Tier 2 only, or Standard mode.	Tier 1.5 trusts one date with no content verification, and Tier 2.5 checks only sampled byte ranges. Tier 1 (ETag and size) and Tier 2 (provenance marker) are the only tiers with no sampling or date trust.

Tier 1.5 risk advisory

Tier 1.5 skips an object when all of these are true:

1. The object exists on the target bucket.
2. The source size equals the target size.
3. The source's S3-managed LastModified is strictly before your cutoff date (UTC end-of-day).

warning

If a source object was modified before the cutoff in a way that did not update S3's LastModified, Tier 1.5 skips it even though its content changed. Examples are a non-standard ObjectScale administrative override or a storage backend that violates the S3 specification.

The probability is effectively zero on stock S3 and ObjectScale deployments, but it is not zero in heavily customized environments.

If you cannot confidently attest to the cutoff, leave Tier 1.5 disabled. On a read-only source you can use Tier 2.5 instead. It is slower, with 6 ranged GETs per probed object, and it checks only the sampled byte ranges.

The cutoff date format is ISO YYYY-MM-DD. The backend converts it to UTC end-of-day (23:59:59). The UI date picker enforces max = today, so future dates are blocked in the UI and in the backend (HTTP 400 on a direct API call).

Tier 2.5 risk advisory

Tier 2.5 detects only changes that lie inside the sampled ranges. With the default 1 MB chunks at three positions, it reads 3 MB of each object.

An isolated 1 KB change in a 1 GB object is detected only about 0.3% of the time. A same-size object with a small change is therefore usually skipped.

warning

Use Tier 2.5 only for read-only sources, such as a quiesced migration. Never use it for a source that is being written to. To be sure, run a Standard copy.

Step log forensics

Target Skip jobs emit one line for each partition in this form:

Target Skip Mode: partition=<prefix> source=<N> target=<M> tier1=<a> tier1.5=<d> tier2=<b> tier2.5=<c> toCopy=<L>

They also emit a final Target Skip Mode: TOTAL … summary.

Each skipped row in cyber_vault_copy_log carries status='target_skipped' and an error_message that identifies the tier that made the call:

• target-skip: tier1 strict ETag+size
• target-skip: tier1.5 date-trust (cutoff=YYYY-MM-DD, sourceLM=YYYY-MM-DD)
• target-skip: tier2 source-etag match (this product's own marker matched)
• target-skip: tier2 x-emc-source-etag match (the ecs-sync marker matched)
• target-skip: tier2.5 range-hash (begin+middle+end)

All Target Skip logic is read-only against the source and target buckets. Skipped objects produce one row in cyber_vault_copy_log (ClickHouse). Nothing changes in PostgreSQL.

Reading the flyout: top tiles and per-tier sub-panel

The Copy Summary card in the Job Detail flyout has two sections that come from different sources.

Top-row tiles

The tiles are Copied, Skipped, Failed, Single-Part, and Multipart. They update in near-real time from the copy agent's live statistics, which are pushed about every 10 seconds.

All Target Skip tiers (1, 1.5, 2, and 2.5) increment the same unified Skipped counter, together with Smart Skip matches. The top tile does not tell you which tier produced each skip. That detail is only in the ClickHouse row's error_message.

Per-tier breakdown sub-panel

The Target Skip Mode — per-tier breakdown sub-panel appears only when the job ran in TARGET_SKIP mode. It has five tiles: Tier 1, Tier 1.5, Tier 2, Tier 2.5, and Copied (Tier 3).

The numbers come from a one-time ClickHouse aggregate (GET /api/cyber-vault/copy-log/{jobId}/target-skip-summary) that runs when the flyout opens. It does not refresh while the job runs. To refresh it, close and reopen the flyout. The top tiles keep updating live regardless.

For this reason, a long-running Target Skip job shows the Skipped count climbing smoothly in the top tile while the per-tier breakdown lags until you reopen the flyout. The tier classification is a forensic read, not a live counter. It needs per-row text matching that is too expensive to recompute every few seconds.

Metadata markers written on every copy

Every successful Tier 3 copy writes one user-metadata header on the target object. This applies in any mode: Standard, the Tier 3 fallback in Smart Skip, or the Tier 3 fallback in Target Skip.

Wire header	Value	Written by	Used by
x-amz-meta-source-etag	The source object's ETag at the moment of copy. This is the S3 GetObject response ETag, the same value the source's ListObjectsV2 returns.	The copy agent, on every Tier 3 copy	Future Target Skip Tier 2 probes

The marker is the source object's ETag, not the target object's ETag. It records what the source looked like when it was copied.

When a later Target Skip job lists the source bucket and finds the same object with the same ETag, it takes these steps:

1. It sends a HEAD to the target.
2. It reads the marker and sees that it matches the current source ETag.
3. It skips the copy through Tier 2 (source-etag match).

Marker key compatibility

Tier 2 accepts two metadata key names:

Marker key	Source	Written when	Meaning
source-etag (wire form x-amz-meta-source-etag)	This product's copy agent	Every Tier 3 copy	This product copied the object, and the source had ETag X.
x-emc-source-etag (wire form x-amz-meta-x-emc-source-etag)	Previous product (ecs-sync)	When ecs-sync ran with Store Source Object Copy Markers enabled	ecs-sync copied the object, and the source had ETag X.

The Tier 2 probe reads both keys. It prefers this product's key and falls back to the ecs-sync key. The per-row error_message names the key that matched.

This means:

• A target bucket migrated from ecs-sync is Tier 2 skipped through x-emc-source-etag on the first Target Skip job. The Tier 3 fallback writes the new source-etag marker for any object that did not Tier 2 skip or that failed validation. It replaces the ecs-sync marker.
• Only objects copied in Tier 3 get the source-etag marker. Objects skipped by Tiers 1, 1.5, and 2 are not rewritten.
• Later Target Skip runs use Tier 1 for unchanged objects. They use a Tier 2 source-etag match for objects where the cross-scheme multipart ETag scheme produces different ETags but the content matches.

What gets overwritten when a re-copy happens

S3 PutObject and CompleteMultipartUpload replace all user metadata atomically. A Tier 3 re-copy of an object wipes any earlier metadata, such as an ecs-sync marker, manual markers, or no marker. It writes the source-etag marker with the source ETag at the moment of the new copy.

This is intentional. The marker always reflects the most recent copy.

Live View & History Sub-tab

This sub-tab is a searchable table of all completed and currently running jobs. The full column list is on the Live View & History help page.

Column	Description
Config	The vault config that generated the job.
Type	Full Copy (all objects in the source bucket), Incremental (objects changed in one changelist window), or Batch Sync (several windows in one job).
Status	The terminal or current state of the job, shown as the raw status value. See the table below.
Started	Timestamp when the job began executing on an agent.
Duration	Elapsed time from job start to completion or failure.
Progress	Job steps completed versus total steps, for example 2/4. Object, byte, and error counts are in the job detail flyout (Job #N Details) and the Copy Log.

Status values

Status	Meaning
running	Actively executing on a copy agent.
pending	Waiting to be dispatched to a copy agent.
completed	The job finished. An Incremental or Batch Sync job that copied some objects but not others still ends completed. Check its Copy Log. A Full Copy with failed objects ends failed.
failed	The job did not complete. A Full Copy also ends failed when any object failed. See the job detail for the error.
cancelled	An operator stopped the job. Cancel takes effect at once: the console marks the job cancelled and tells the copy agent to stop.
auto_pause	Paused because the source or target answered 503 or the connection to it failed. See the Live View & History help page.
agent_running, agent_pending_report	See the Live View & History help page.
Not Started (skipped)	Blocked before dispatch. The Failure column gives the reason.

Row Actions

Action	Description
Details	The › button in the Details column, or a click anywhere on the row. Opens the Job #N Details flyout with the full job metadata, agent assignment, error messages, and object-level statistics. The next two actions are at the bottom of that flyout.
View Detailed Copy Log	Opens the per-object copy log page for the job. Each row is one object with its copy status, source key, size, and any error message. Use it to find exactly which objects failed.
View Long-term Trends	Opens the Job Trends page for the job, showing objects per second and MB per second over the job's duration. Use it to see whether a slow job was uniformly slow or hit a bottleneck at a point in time.
Resume	The ↩ button in the Actions column, or Resume Failed Job in the flyout of a failed job. See below.

Resume

Resume is available on failed and cancelled Full Copy, Incremental, and Batch Sync jobs. It creates a new job for the same vault config and window, using the current credentials and settings. The original failed job is not modified.

A resume behaves as follows:

• A Full Copy resume keeps the original job's copy mode (Standard, Smart Skip, or Target Skip), with the same Target Skip tiers and Tier 1.5 cutoff.
• It picks up where the failed job stopped instead of starting over. See Smart Skip and Resume — data coverage.
• A large object (1 GiB or more) that the failed job was copying continues its multipart upload from the first missing part. This happens when the new job runs on the same copy agent.
• A Batch Sync job is resumed as a Batch Sync resume that picks up only its remaining windows.
• While the group is in Vault Mode, a manual Resume is refused, because the vault agent schedules that group's syncs.

After the resume completes, check the new job's Copy Log to confirm that all expected objects were transferred.

Smart Skip and Resume — data coverage

Resume uses two mechanisms to avoid redundant work:

• A key-based checkpoint (startAfter) resumes the S3 listing from the last object the previous job processed.
• A completed-prefix skip list bypasses any prefix marked as finished in the previous run.

A resumed job does not see objects that were added to the source bucket after the original job ran, in two situations:

• Objects in a completed prefix. If new data was written to a prefix that the original job had already finished, that prefix is in the skip list and is not listed again. The new objects are invisible to the resumed job.
• Objects before the checkpoint key. If new data was written at a key that sorts before the startAfter checkpoint position, the S3 listing does not walk back to include it.

To catch new or modified data that arrived after a partial run, dispatch a new full-copy job with Smart Skip enabled instead of resuming. A fresh job lists the complete bucket from the start and evaluates every object's ETag against the baseline.

• New objects (no matching ETag in the baseline) are copied.
• Unchanged objects (matching ETag) are skipped.

This guarantees full coverage regardless of prefix completion state or object position in the key space.

tip

Use Resume when you know the source bucket did not change between the failure and the re-run, which means the source is static. Use a fresh Smart Skip job when the source bucket is actively written to, or when you are not certain whether data was added after the partial run.

Large-object uploads

Copy agents also keep the work done inside a large object. A job can fail or be canceled, or its agent can restart or crash, while an object of 1 GiB or more is being copied. If at least one part is on the target, the agent keeps that object's multipart upload.

A later job on the same agent that copies the object continues the upload from its first missing part. That job can be the Resume, a fresh Smart Skip job, the policy's next full copy, or an incremental copy.

• The upload continues with the part size it started with, if the source object (ETag and size) is unchanged. Otherwise the agent aborts the old upload and copies the object from part 1.
• An upload left by an earlier run that fails all its retries again, for a reason other than the network, is aborted. The next run starts afresh.
• A job on another copy agent copies the object again from the start.
• An upload that no job continued is aborted by the bucket's next full copy on that agent after 7 days, or by its next incremental or Batch Sync copy after 35 days.

See Copy Log > Resumed copies for the details.

Filters

Filter	Description
Status	Shows only jobs in a specific state, for example only failed jobs.
Type	Filters by Full Copy, Incremental, or Batch Sync.
Failure reason, Action type, Find by ID	See the Live View & History help page.

Copy Job Active Queue Management Sub-tab

This sub-tab is a real-time view of all jobs that are not yet in a terminal state. It is split into two panels.

Live Jobs Panel

This panel shows jobs that are actively executing, with the statuses running, auto_pause, and agent_pending_report. Jobs in pending (queued) state do not appear here. They are in the Queue Panel.

Job kind	What you see
Running jobs	The name of the assigned agent and a live elapsed-time counter.
Auto Pause jobs	See below.
agent_pending_report jobs	The assigned agent stopped sending heartbeats, or the console restarted while the job ran on the agent. These jobs are grouped in an amber collapsible section. Each shows a countdown to auto-fail, based on the crash timeout for the assigned agent.

A copy agent pauses when the source or target gives a 503 (slow down) answer, or when the connection to it fails (reset or refused). In that case the agent:

• Pauses all jobs on that agent for a random hold-off of 30 seconds to 5 minutes.
• Starts no new job in the meantime.
• Lets the jobs carry on by themselves afterward.

The paused jobs stay visible here and can be canceled.

When a pending job moves to running, a brief green flash plays on its row to confirm dispatch. It shows that the dispatcher is working through the queue.

Queue Panel

This panel shows only jobs in pending state. They are assigned to an agent but waiting for an available execution slot. Use it to focus on what is waiting, without the noise of running jobs.

Capacity Bar

The header shows Running: N / M (pct%).

• N is the number of actively executing jobs.
• M is the total slot capacity across all online agents.
• Queued jobs waiting for a slot are shown separately as · X queued. They do not count toward the running total.

The bar turns amber when running is at 100% of capacity and jobs are still waiting in the queue.

Priority Tiers

Jobs are dispatched in priority order. When multiple jobs are pending, higher priority jobs are dispatched before lower priority jobs.

Tier	Use for
CRITICAL (40)	Highest priority. Buckets with the tightest RPO requirements.
HIGH (30)	Important buckets that are not time-critical.
NORMAL (20)	The default for all jobs unless overridden.
LOW (10)	Lowest priority. Bulk archival or low-urgency configs.

To adjust tiers, multi-select pending jobs in the active queue and use the Change Priority bulk action.

Workflows

Investigating a failed job

1. On the Live View & History sub-tab, filter Status to failed.
2. Click the failed job row to open the job detail.
3. Read the error message in the detail panel. Common causes are agent lost connectivity, vault device unavailable, and source credential expired.
4. Read the copy event lines in the job's step log. See the table below.
5. If the error was at the object level, click View Detailed Copy Log to see which objects failed.
6. After you resolve the underlying cause, click Resume (↩) on the failed job row, or Resume Failed Job in the flyout. A confirmation banner shows the new job number when the resume is accepted.

For step 4, look in the bucket_walk step of a full copy, or the copy step of an incremental or Batch Sync copy.

Log line	Meaning
WAITING FOR NETWORK with no NETWORK RECOVERED after it	The source or target did not answer again during the job. After 15 minutes of retries, the objects that needed it failed. Check name resolution and routing from the copy agent.
UPLOAD KEPT	Lists large objects whose upload the agent kept for the next run, with how many parts are already on the target.
UPLOAD DISCARDED	Names a kept upload that could not be continued, and why.
ABORT FAILED	Gives the upload ID of an upload left open on the target.

These lines are described under Live View & History > Job Detail Flyout.

Large objects listed as UPLOAD KEPT continue from their first missing part only if the resumed job runs on the same copy agent. On another agent they are copied again from the start.

Prioritizing a critical bucket's pending jobs

1. Open the Copy Job Active Queue Management sub-tab.
2. Find the pending jobs for the bucket's vault config.
3. Select them using the row checkboxes.
4. Click Change Priority and select CRITICAL.

The jobs move to the front of the pending queue. They are dispatched on the next available agent slot.

Checking throughput after a concurrency change

1. Trigger a copy run and let it complete.
2. Find the completed job on the Live View & History sub-tab and open it.
3. Click View Long-term Trends.
4. Compare the throughput curve to previous runs for the same config.

A healthy increase in objects per second confirms that the concurrency change had the intended effect.

Tips

• A job stuck in running for much longer than similar jobs usually means the agent is struggling. Check Agent Fleet Management to confirm that the agent is still sending heartbeats and that its job count has not stalled.
• A long-running job can also be waiting for the network. A WAITING FOR NETWORK line in the job's bucket_walk step (full copy) or copy step (incremental or Batch Sync copy) means the copies are retrying an unreachable source or target, for up to 15 minutes per outage.
• The agent_pending_report countdown is based on the per-agent crash timeout override if set, otherwise on the global default. Agents with a long crash timeout hold jobs in this state longer before auto-failing them.
• To investigate partial syncs, use View Detailed Copy Log rather than the job detail flyout. The job detail shows only aggregate counts, while the copy log shows each object with its specific error.
• Job history is retained indefinitely. The per-object Copy Log rows are kept for the Copy Log Retention period (Setup & Settings > Database, default 90 days).
• Bulk priority changes affect only jobs currently in pending state. You cannot change the priority of jobs that are already running.