Copy Log

The Copy Log page shows the per-object detail log for a single copy job. Open it with View Detailed Copy Log in the job's Job #N Details flyout.

Use this page to see the outcome of every object the copy agent attempted to transfer, including failures and retries. It helps you understand why specific objects failed and whether retries resolved them.

Overview

Each row in the log represents one object transfer attempt. The copy agent writes the log during job execution, and it is preserved after the job completes.

You can filter, sort, and paginate the log to focus on specific objects or failure modes. The page uses server-side pagination, so it performs well even for jobs that copied millions of objects.

Summary Bar

The summary bar at the top of the page is a grid of stat cards that cover the whole job. Each card has a hover tooltip that explains its exact meaning.

Card	What it shows
Total Objects	Unique objects with a final outcome: copied, deleted, failed, skipped, and target-skipped. Intermediate retry rows are excluded.
Copied	Objects successfully transferred. Green when greater than 0.
Deleted	Objects deleted from the target by an incremental sync. Rose when present, dimmed otherwise.
Failed	Objects that still failed after all three retries. Red. If the source or target could not be reached at all, the object first waits for the network for up to 15 minutes. See Network waits.
Skipped	All objects that were not transferred and were not failures. It rolls up every skip reason. See the list below this table.
Target Skip	The subset of Skipped that Target Skip Mode matched against the live target bucket using its 4-tier match. Blue. This subset is also included in the Skipped card.
Deleted at Source	Shown only when it is above 0. The subset of Skipped that was listed but deleted at the source before it was read. See below this table.
Retried	Total intermediate retry attempts before eventual success or permanent failure. See below this table.
Total Size	Total bytes transferred for successfully copied objects.
Single-Part / Multipart	How many objects were copied as a single PUT and how many were split into multiple parts. Both are included in Copied.
Avg Duration	Average total duration in milliseconds per successfully copied object.
Max Duration	Duration in milliseconds of the slowest single object copy.

Skipped card

The Skipped card counts every skip reason:

• Smart Skip: the source ETag matched a previous copy-log record.
• Target Skip: the object matched the live target bucket.
• A source object that no longer existed when it was read, because it was deleted at the source after it was listed.

Because Skipped is the umbrella count, a job whose objects were all skipped against the target shows a value equal to the object count, not 0. The Target Skip card beside it breaks out the target-skip subset.

Deleted at Source card

This card counts objects that were listed but deleted at the source before they were read. An example is data that the source application expires on its own schedule.

Nothing was left to copy, so these objects are tried once, not retried, and are not failures. A job whose only issue is such objects completes.

Retried card

This card counts whole-object attempts only. A single part of a multipart copy that was re-read or re-uploaded in place is not counted, and neither is a wait for the network. See Part retries and Network waits.

The retry of a multipart object continues its upload instead of starting again from part 1, unless the source object changed.

Total Size card

An object that continued an earlier upload counts with its full size, although only its missing parts were sent in this job.

info

Full Copy: the cards count log rows. An object whose copy failed during this job's walk keeps its retry and failed rows. The end-of-walk retry then adds one more row with its final outcome.

That final row is one of the following:

• copied
• target_skipped ("end-of-walk retry: already on the target …")
• skipped ("source deleted: …")
• another failed

Such an object counts in Failed and again under its final outcome. An object that failed again counts twice in Failed.

A resume's retry of the failures it inherits adds only their outcome rows to the resume's log. Their failed rows stay with the job that failed them.

The job's own counts, shown in its Live Copy Summary and summary step, count each object once by its final outcome.

info

The Skipped card compared with the skipped row status. The Skipped card is a roll-up of every skip mode: Smart Skip, Target Skip, and stale-key skips.

The per-object status value is more specific. A Smart Skip or stale-key object is skipped, and a Target Skip object is target_skipped.

On a Target Skip job, the Skipped card might read 6 while every table row shows target_skipped. The card is the total, and the rows carry the exact mode. The status filter lets you isolate each mode.

Log Table

The main table has one row per object. The columns are described in the sections below.

Object key

The full S3 object key or file path of the object.

Status

The final outcome for this object.

Status	Badge	Meaning
copied		The object was transferred successfully through the full copy path. See below this table.
skipped	Gray	The object was not transferred, and that is not a failure. See Skipped status.
target_skipped	Blue	Target Skip matched the object on the live target bucket. The tier that made the call is in the Error message column.
deleted	Red	An incremental sync removed this object from the target because the source no longer holds it.
failed		The copy attempt failed permanently after all retries. See Failed status.
retry		The object required more than one attempt. Check the Error message column to see whether it ultimately succeeded.

In Standard mode, every object is copied. In Smart Skip and Target Skip modes, copied is the Tier 3 fallback. The object did not match any skip criterion (no copy-log baseline, not present on target, no marker match, no range-hash match), so a complete source-to-target byte transfer was performed.

Intermediate retry rows are shown separately from the final outcome row. The next attempt of a multipart object continues the same upload from the first part not yet on the target. See Part retries.

Skipped status

A skipped object has one of three causes: Smart Skip, the object was deleted at the source, or a stale or empty key.

Smart Skip matched the most recent earlier copy-log record of this object for the same config, source bucket, target bucket, and target prefix. The Error message column shows one of these:

• smart-skip: ETag matches baseline job #0 on a full copy. The #0 means all earlier copies, not one job.
• smart-skip: ETag matches baseline (all-jobs) on an incremental sync.

The source object no longer existed when it was read:

• On a full copy, the object was listed and then deleted at the source before its turn came. This is common for applications that expire their own data, such as audit journals. With large objects ahead of it, the gap between listing and copying can be hours.
• On an incremental sync, the object was created and then deleted between the change list being built and the sync running.

In either case there is nothing to copy and nothing wrong. The object is tried once, with no retries, and the Error message column says source deleted: the object no longer exists at the source ….

These objects do not count as failures. They do not fail the job, do not push a window to sync_partial or sync_failed, and need no attention. The Deleted at Source card counts them.

A missing source bucket (NoSuchBucket) is still a failure.

info

To tell a deleted object from a missing bucket, the copy agent checks that the bucket still exists. This check needs the source account's s3:ListBucket permission, the same permission a full copy needs to list the bucket.

If the account lacks it, a deleted object is retried and reported as failed, and the copy agent's log says the bucket check failed.

Failed status

On a Full Copy, the copy agent retries every failed object once all partitions are done. A resume retries the failures of the job it resumes.

As a result, the same object can have a later row in the same job:

• copied
• target_skipped with "end-of-walk retry: already on the target (same ETag and size)"
• skipped with "source deleted: …"

The job's own Failed count and summary count only the objects that are still failed.

A copy agent keeps the multipart upload of a failed object of 1 GiB or more on the target. A retry, or the next run on the same agent, then continues the upload instead of starting again. See Resumed copies.

Part retries

A multipart copy reads the source one part at a time and uploads each part before reading the next.

If reading or uploading a single part fails with a transient error, only that part is retried and the copy carries on. The parts already uploaded are kept. Transient errors include:

• A dropped or reset connection
• A read timeout
• A 5xx, 408, or 429 answer

A part is tried up to 4 times, with 3 pauses of 2 s, 4 s, and 8 s between the tries. Starting the multipart upload, completing it (Complete), and canceling it (abort) are retried the same way. A canceled job tries the abort only once, because it is stopping.

A part retry is not a new attempt of the object. It creates no retry row and is not counted in the Retried card.

If the copy then succeeds, its copied row says so in the Error message column. For example:

recovered after 2 part retries (last: part 1412 read — Premature end of Content-Length delimited message body…)

If a part still fails after its 4 tries, the object fails that attempt and gets a retry row. Its upload is kept. The next attempt checks that the source object is unchanged (same ETag and size) and continues the same upload from the first part not yet on the target. The copied row then says resumed from part K/N.

The upload is aborted only when it cannot be continued:

• The source object changed or was deleted.
• The source sends no ETag, so a same-size overwrite could not be detected.
• The target rejected the Complete.
• The error is one that no retry can fix.

After a source change or a rejected Complete, the next attempt starts a new upload from part 1.

When the object's last attempt fails, or the job is canceled, the upload is aborted. The exception is an object of 1 GiB or more, where a copy agent keeps the upload for the next run. See Resumed copies.

Where copy events are logged

The copy event lines named on this page (ABORT FAILED, WAITING FOR NETWORK, UPLOAD KEPT, and others) are in the job's own step log:

• The bucket_walk step of a full copy.
• The copy step of an incremental or Batch Sync copy.

See the Job Detail Flyout in Copy Jobs > Live View & History.

When the Complete call fails

If the Complete call fails, the copy agent checks the object on the target before anything else. The copy counts as done when the object there matches the source:

• It has the same size.
• It has the source-etag marker of this source version.
• Its multipart ETag has the same part count.

The object is not copied again, and the copied row says the Complete call failed, but the object on the target matches the source.

An abort that still fails leaves the upload and its parts on the target. An ABORT FAILED line in the job's step log names its upload ID, so you can abort it with an S3 tool.

Network waits

A request that never reached the server is retried with growing pauses instead of 4 tries in about 14 seconds. This applies to every object, including single-part copies. Causes include:

• The host name did not resolve.
• The connection was refused or timed out.
• There was no route to the host.

The pauses are 2, 4, 8, 16, and 32 s, then every 60 s, each plus or minus 20%. The copy agent keeps retrying for up to 15 minutes per outage.

A network wait is not a new attempt. It creates no retry row and is not counted in the Retried card.

A copy that had to wait says so on its copied row, for example waited 2m 05s for the network.

Step log lines

The job's step log gets:

• One WAITING FOR NETWORK line when an outage starts.
• One NETWORK RECOVERED line when the source or target answers again.

Any answer ends the outage, including an error answer such as 503, because the network works again.

If the same side fails again within a minute of a reported outage, the new outage gets its lines only once it has lasted a minute. This keeps a flapping connection from filling the log.

When the outage exceeds 15 minutes

If the outage lasts longer than 15 minutes, the objects it affects fail. The Error message column shows the source could not be reached for 15 min (network retry budget), or the target … for the target.

The 15 minutes are counted per outage, not per object. There is one clock for the target and one for each source endpoint. A new outage starts a new clock.

A geo-federated job reads each VDC separately. A VDC that is down uses up its own 15 minutes even while another VDC answers.

Once the time is used up, other objects that hit the same outage fail quickly instead of each waiting 15 minutes again.

A cancel ends the wait within about a second once it reaches the copy agent.

Resumed copies (copy agents)

When a copy agent stops copying an object of 1 GiB or more before it is finished, the multipart upload stays on the target. The agent keeps a record of it in its local state database.

The copy can stop for these reasons:

• The last attempt failed.
• The job was canceled.
• The agent restarted or crashed.

What the log shows

After a failed or canceled copy, the job's step log shows:

UPLOAD KEPT: <key> — K/N parts are on the target

The object's failed row ends with upload kept for the next run: K/N parts are on the target (uploadId …).

There are two exceptions:

• An upload with no part on the target yet is aborted, not kept.
• An upload that an earlier run left is aborted if it fails all its retries again for a reason other than the network. The next run then starts a fresh upload.

How a later run continues

A later run on the same agent that copies the object continues from the first missing part. Such a run can be a re-run, a Resume, the policy's next full copy, or an incremental copy.

The agent asks the target which parts it has. It continues only if the source object (ETag and size) is unchanged. The upload keeps the part size it started with, even if the policy's Min Part Size has changed since.

The copied row of a continued copy says resumed from part K/N (uploadId …, left by an earlier run; source MD5 not computed).

A run on another copy agent copies the object again from the start.

When an upload cannot be continued

The old upload is aborted and the object is copied from part 1. The job's step log shows:

UPLOAD DISCARDED: <key> — uploadId … cannot be continued: …; copying from part 1

This happens in cases such as:

• The source object changed or sends no ETag.
• The recorded part size does not fit the object.
• The target cannot list the upload's parts, or refuses to. An example is a 403 when the s3:ListMultipartUploadParts permission is missing.

Cleanup of old uploads

Until an upload is continued or aborted, its parts take up space on the target. The copy agent aborts old uploads at the end of a copy job, unless that job is canceled:

Upload not continued for	Aborted at the end of
7 days	The bucket's next full copy on that agent
35 days	The agent's next incremental or Batch Sync copy

Incremental and Batch Sync copies handle only the objects that changed, so they never continue an upload that a full copy kept. A RESUME JANITOR line appears in the job's step log.

If such an abort fails, the upload is tried again after another 7 or 35 days.

Other columns

Column	Description
Type	Whether the object was transferred as a single-part or multipart copy.
Retry #	The retry attempt number for this row: 0 for the original attempt, 1 or higher for retries. Sortable.
Parts	Number of multipart parts. Blank for single-part. Not sortable.
Size	The size of this specific object.
Duration	Total wall-clock duration in milliseconds for this copy attempt. Sortable.
Read latency	Time taken to read the object from the source. Sortable.
Write latency	Time taken to write the object to the destination. Sortable.
Thread	The worker thread ID that handled this copy. Use it to spot a single hot or slow thread across many rows. Not sortable.

Source MD5

The locally computed MD5 of the source bytes as they were streamed during the copy. It is set on copied objects only, and skip rows leave it blank.

For multipart objects, this is the MD5 of the whole assembled stream. It is not the multipart-composite hash that S3 returns as the ETag, so the value is comparable across multipart schemes.

It is also blank (—) on a copy that continued an upload left by an earlier run. The Error message column says left by an earlier run; source MD5 not computed. The parts already on the target were not read again, so the whole-object MD5 cannot be computed.

A copy that continued its own upload after a failed attempt still has the MD5.

Source ETag

The S3 ETag of the source object at the moment of the copy or skip decision.

• For single-part objects, this is the raw MD5 (32 hex characters).
• For multipart objects, this is the multipart-composite MD5(MD5(part1)…MD5(partN))-N value. The -N suffix is the part count.

Smart Skip and Target Skip Tier 1 and Tier 2 compare against the Source ETag. The column is shown so you can audit skip decisions row by row.

Error message

For a copied row, this column is normally empty. A note in amber, not red, means the copy succeeded but had to retry parts, wait for the network, continue an upload, or check the target. Several notes are joined with ;.

Note	Meaning
recovered after N part retries (last: …)	The copy needed part retries. See Part retries.
resumed from part K/N (uploadId …)	After a failed attempt, the next attempt continued the same multipart upload from part K instead of starting again.
resumed with all N parts already on the target (uploadId …)	Every part was already there, and only the final Complete call was left.
left by an earlier run; source MD5 not computed	The copy continued an upload that an earlier run on the same copy agent kept. See Resumed copies.
waited Ns for the network	The source or target could not be reached for a while. The copy waited and carried on. See Network waits.
the Complete call failed, but the object on the target matches the source	The final Complete call returned an error, for example because its response was lost. The object on the target was checked and matches this copy, so it was not copied again.

For a failed row, the error reason is shown. For a skipped or target_skipped row, the reason for the skip is shown.

Skip and failure messages

Message	Meaning
source deleted: the object no longer exists at the source …	The object was listed, or was in the change list, and then deleted at the source before it was read. It was tried once. This is a skip, not a failure.
Key cannot be empty	Incremental sync only. The change list held an empty key. This is a skip, not a failure.
NoSuchKey or Status Code: 404 on a failed row	The source bucket could not be found or read when the object was missing. The object is retried and reported as a failure, not skipped.
NetworkUnreachableException: the source could not be reached for 15 min (network retry budget) — …	On a retry or failed row. The copy agent could not reach that side for the whole 15-minute network wait. The message says the target … for the target. Check name resolution and routing from the copy agent.

On a Full Copy, an object that the end-of-walk retry found deleted after it failed in this job's walk also has that walk's retry and failed rows.

Smart Skip messages

Message	Meaning
smart-skip: ETag matches baseline job #0	Full copy. Smart Skip matched the most recent earlier copy-log record of this object for the same config, source bucket, target bucket, and target prefix.
smart-skip: ETag matches baseline (all-jobs)	Incremental sync. Smart Skip matched the same record as above.

Target Skip messages

Message	Meaning
target-skip: tier1 strict ETag+size	The source ETag and size matched the target listing exactly.
target-skip: tier1.5 date-trust (cutoff=YYYY-MM-DD, sourceLM=YYYY-MM-DD)	Date trust attested by the operator. The source size matched the target size, and the source LastModified was strictly before the operator's cutoff date. The message includes both the cutoff and the actual source LastModified for auditing.
target-skip: tier2 source-etag match	The x-amz-meta-source-etag metadata marker, written by a previous copy by this product, matched.
target-skip: tier2 x-emc-source-etag match	The x-emc-source-etag metadata marker matched.
target-skip: tier2.5 range-hash (begin+middle+end)	Three sampled byte ranges matched between source and target through MD5.

Metadata markers written on every successful Tier 3 copy

Every successful Tier 3 copy writes one user-metadata header on the target object. This applies in any mode: Standard, the Tier 3 fallback in Smart Skip, and the Tier 3 fallback in Target Skip.

Wire header	Value	Purpose
x-amz-meta-source-etag	The source object's ETag at the moment of the copy. It matches the Source ETag column for this row.	Lets a future Target Skip job recognize this copy through Tier 2 — source-etag match without re-reading bytes.

Tier 2 also accepts the x-amz-meta-x-emc-source-etag marker. The Error message column identifies which key matched.

Re-copies overwrite the metadata atomically. After any Tier 3 re-copy, the marker reflects the most recent source ETag.

Filters

Filter	Description
Status dropdown	Filters the table to objects with a specific status. Use Failed (permanent) to focus on objects that need attention. See the options below.
Copy type dropdown	Filters to single-part or multipart objects.
Object key prefix	Free-text filter that shows only rows whose key starts with the entered text. Useful for focusing on a specific folder or prefix within a large job.
Scan prefix dropdown	Filters to objects that belong to a specific changelist partition. Useful for incremental copy jobs that process multiple changesets.

Status dropdown options

• Copied
• Failed (permanent)
• Skipped (Smart Skip / deleted at source)
• Target Skip
• All Skips (Smart + Target)
• Deleted
• Retry

The two skip filters differ:

• Skipped (Smart Skip / deleted at source) matches only the skipped per-object status. That covers Smart Skip, objects deleted at the source after they were listed, and stale changelist keys.
• All Skips (Smart + Target) matches both skipped and target_skipped. It returns the same set of rows that the Skipped summary card counts.

Use All Skips (Smart + Target) when you reconcile the Skipped card against the table.

Pagination

Choose 50, 100, 250, or 500 rows per page. Pagination is server-side, so changing the page size reloads results from the backend.

Sorting

Click any column header to sort the table by that column in ascending order. Click again to sort in descending order. Sorting is server-side.

Workflows

Find all failed objects in a job

1. Open the Status dropdown filter and select Failed (permanent).
2. Review the Error message column for each row to understand the cause.
3. If the errors share a common pattern, such as a network timeout or access denied, that pattern points to the root cause.

Check whether retried objects ultimately succeeded

1. Filter by status Retry. Each row is one failed attempt, and its Error message says why it failed.
2. To see how an object ended, clear the status filter and enter its key in Object key prefix.
3. A copied row means a later attempt succeeded. A failed row means the object did not recover.
4. On a multipart object, resumed from part K/N on the copied row shows that the later attempt continued the same upload instead of copying the object again.

Spot a flaky source or network path

1. Look for large objects that keep needing part retries. They point to a connection problem between the agent and the source or target, even when every copy ends up copied. An example is a proxy or load balancer that drops long transfers.
2. Look for copied rows with an amber note in the Error message column. To find them across many jobs, use Job Log Search with Search in: Error Message and the text recovered after.
3. Read the part number and error in the note to see where and how the connection failed.

Notes that say waited Ns for the network mean something different. The same is true of WAITING FOR NETWORK lines in the job's step log (bucket_walk or copy step).

They mean the source or target could not be reached at all for a while, because of name resolution, refused or timed-out connections, or routing. This is a different problem from connections dropped mid-transfer.

To find these notes across jobs, search the Error Message for for the network.

Check that a large object continued instead of starting again

1. Enter the object's key in Object key prefix and find its copied row.
2. Read the Error message. resumed from part K/N (uploadId …) means parts 1 to K-1 were already on the target and were not sent again. With resumed with all N parts already on the target, no part was sent again.
3. left by an earlier run means an earlier job on the same copy agent kept the upload. If that job failed or was canceled, its step log (bucket_walk or copy step) has an UPLOAD KEPT line for the object.

If there is no such note although an earlier job kept an upload for the object, the object was copied from part 1. The likely reasons are:

• The new run was on another copy agent.
• The kept upload could not be continued. An UPLOAD DISCARDED line in the new job's step log says why, for example because the source object changed.
• The upload was aborted after 7 days without progress, or after 35 days if only incremental or Batch Sync copies ran since.

Investigate slow copy performance for large objects

1. Sort by Write latency in descending order.
2. The rows at the top are the objects that took the longest to write.
3. Cross-reference with the Type column. Very large objects should be multipart. If a large object was copied as single-part, that may explain the elevated latency.

Data Retention

Copy log records are stored in ClickHouse. They are kept for the Copy Log Retention period set in Setup & Settings > Database, counted from the time the object was copied. The default is 90 days, and the range is 7–1095 days.

ClickHouse deletes older records automatically through its TTL mechanism during background merge operations. This applies to all jobs, both full copy and incremental.

To audit a copy operation older than the retention period, use the job-level summary. Its status, object counts, bytes transferred, and duration remain permanently available in the job history. Only the per-object detail rows are subject to the copy log retention.

Tips

• The Retry status indicates the object required more than one attempt. Retries are normal for transient network errors. A high retry count is worth investigating if it persists across multiple jobs.
• Part retries inside a multipart copy and waits for the network are not Retry rows. Look for recovered after and waited … for the network notes on copied rows instead.
• A retry does not restart a multipart object from byte 0. Neither does a later run on the same copy agent for an object of 1 GiB or more. The upload continues from its first missing part (resumed from part K/N), as long as the source object is unchanged.
• With several copy agents, run the job again on the agent that kept the upload.
• The Object key prefix filter is case-sensitive and matches from the start of the key. To search within a key rather than from the start, leave the prefix filter empty and use the full key in a different tool.
• For very large jobs, use pagination rather than trying to view all rows at once. Server-side pagination keeps the page responsive regardless of the total row count.