Job Log Search

What this tab does

The Job Log Search tab lets you search across the copy-log records from all copy jobs at once — not just a single job. It has two modes, selectable via the pill toggle at the top of the panel:

• Search — find individual copy log rows by object path or error string
• Error Summary — aggregate all failures by error pattern and see a ranked report

---

Mode toggle

The pill buttons at the top of the panel switch between Search and Error Summary. The source bucket filter applies to both modes. The date controls differ: Search uses a free From/To range; Error Summary uses a single End date with a fixed 7-day window.

---

Search mode

What it does

Searches the cyber_vault_copy_log ClickHouse table using tokenbf_v1 bloom filter skip indexes on object_key and error_message. This avoids full table scans for most queries, even on tables with millions of rows.

This is useful when you need to:

• Find every attempt to copy (or fail to copy) a specific object path across multiple job runs.
• Search for all objects that failed with a particular error string.
• Audit whether a known object was successfully copied within a given date window.

Search fields

Field	Description
Search text	The string to search for. Matched with LIKE '%text%' — i.e. the text can appear anywhere in the field. Minimum 3 characters (unless a Job ID filter is active — see below).
Search in	Which column to search: Object Key (the S3 object path) or Error Message (the copy error text).
Status	Filter by outcome: copied, failed, skipped, or retry. Leave as "All statuses" to return all outcomes (including target_skipped and deleted rows, which have no filter option of their own).
Source bucket	Optional. Filters to a single source bucket (exact match). Providing this significantly improves query performance because it takes advantage of ClickHouse's sort order key which includes source_bucket.
From / To	Date range for the search. Required for all cross-job searches (when no job ID filter is active). Defaults to the last 7 days.

Job ID filter chip

When you drill down into a specific job from Error Summary mode (by clicking a job ID pill), the panel switches to Search mode with a Job: NNNNN × chip displayed above the search text box. This chip means:

• The search is scoped to that single job using an exact job_id = N filter in ClickHouse, not a LIKE text search.
• The search text field becomes optional — you can leave it empty to see all copy log rows for that job, or add a text filter to narrow within those rows.
• The date range pickers are ignored while a job ID filter is active (the backend uses job_id = directly, so no date range is needed).
• Click the × on the chip to clear the filter and return to date-range search mode.
• Typing new text in the search box while a job ID chip is active clears the chip — the search reverts to a text-based cross-job search.

The "complete path segment" requirement

The bloom filter index works by tokenizing object key strings at word boundaries (non-alphanumeric characters such as /, -, .). A query like reports will correctly match bucket/reports/2026/file.parquet, but a truncated fragment like repor will not benefit from the index — ClickHouse cannot filter granules using a partial token, so the query degrades to a linear scan.

Best practice: always search for at least one complete path segment or word. For example:

• Good: invoices, 2026/03, AccessDenied
• Avoid: inv, 2026/0, Access

Date range requirement

When searching across all jobs (no Job ID filter active), you must provide a from and to date. This is enforced to prevent accidental full-table scans on the copy log, which can grow very large in active deployments. When a Job ID filter is active, the date range is ignored — the backend queries by job_id = directly, which is both precise and fast.

Result table

Column	Description
Job ID	The ID of the copy job that produced this row. Shown in bold blue — select and copy it to look up the job with Find by ID in Copy Jobs - Live View & History.
Config	The vault configuration name (source bucket policy).
Bucket	The source bucket name.
Object Key	The S3 object path as stored in the copy log.
Status	The outcome for this specific copy attempt: copied (green), failed (red), skipped (gray), or retry (amber). retry rows are intermediate attempts; the final outcome row appears after all retries complete.
Size	Object size formatted in B / KB / MB / GB.
Error	First 60 characters of the error message (if any). Hover for the full text. On a copied row, a recovered after N part retries (last: …) note means the multipart copy succeeded but some of its parts had to be re-read or re-uploaded — search the Error Message for recovered after to find copies that hit a flaky source or network path. Other notes on copied rows are listed under Searching the copy notes.
Copied At	Timestamp when this copy attempt completed (UTC).

Results are ordered newest first.

Searching the copy notes

A copied row can carry notes in its error message; several are joined with ;. Set Search in to Error Message (and Status to copied to leave out other rows) and search for:

Search text	Finds
recovered after	Multipart copies that needed part retries (recovered after N part retries (last: …)).
resumed from part	Multipart copies that continued their upload from a later part instead of starting again (resumed from part K/N (uploadId …)), after a failed attempt or from an earlier run.
resumed with all	Multipart copies whose parts were all on the target already, so only the final Complete call was left (resumed with all N parts already on the target (uploadId …)). Search for resumed to find both kinds.
left by an earlier run	The resumed copies (both kinds) that continued an upload kept by an earlier job on the same copy agent (their Source MD5 in the Copy Log is blank).
for the network	Copies that waited for the source or target to become reachable again (waited Ns for the network).
Complete call failed	Multipart copies whose final Complete call returned an error, but whose object on the target matched the source, so it was not copied again.

To find objects that failed because the source or target could not be reached for the whole 15-minute network wait, search for network retry budget with Status failed or retry (the message reads … could not be reached for 15 min (network retry budget) …).

Truncation warning

When 500 or more rows match your query, a warning badge appears: "Results capped at 500 — narrow your filters." The table shows only the 500 most recent rows. To get a complete result set, add more specific filters (bucket, status, narrower date range) or use the Export CSV button to download the current 500-row set and continue analysis offline.

---

Error Summary mode

What it does

Error Summary aggregates all failed copy attempts within your selected date range into a ranked report grouped by error pattern. Instead of showing individual copy log rows, it answers the question: "What were the most impactful failure causes, and how many objects and jobs did they affect?"

The report queries the cyber_vault_copy_log table for rows with status = 'failed', then groups them by the first 200 characters of the error message. Results are ordered by object count descending — the most impactful error patterns appear first.

Date window — fixed at 7 days

Error Summary always searches exactly 7 days. Instead of a From/To pair, the panel shows a single End date picker. The start date is automatically set to 7 days before the end date and shown as a read-only label (e.g. 2026-04-08 > 2026-04-15).

This limit exists because Error Summary performs a full column scan of the copy log — there are no per-row skip indexes that help this aggregation query. A 7-day window scans roughly 4–5 million rows and returns results in 2–5 seconds. Wider windows would take significantly longer and are not exposed in the UI.

To investigate a different week, change the End date and click Generate Report again.

How to use it

1. Set the End date (defaults to today) and optionally a Source bucket filter.
2. Click Generate Report — the panel always searches the 7 days ending on the selected date.
3. The table shows one row per distinct error pattern.
4. Click the ▶ expand button on any row to see the full error message and a list of sample job IDs.
5. Click a job ID pill to switch to Search mode with a Job ID filter active — results show all copy log rows for that job (exact job_id = match, not a text search).

Result table columns

Column	Description
Error Pattern	The first 80 characters of the error prefix (hover for full 200-char prefix).
Objects	Total number of failed copy attempts matching this error pattern (red, bold).
Jobs	Number of distinct jobs that produced this error.
Buckets	Source buckets affected (up to 2 shown; hover for all).
Configs	Vault config names affected (up to 2 shown; hover for all).
Last Seen	Most recent timestamp when this error occurred (UTC).
▶ / ▼	Expand/collapse the row to see the full error message and sample job IDs.

Expanded row details

Clicking the expand button reveals:

• Full error message — the complete last error text for this pattern, shown in a scrollable <pre> block.

• Sample job IDs: up to 10 job IDs that produced this error. Each is a clickable button that switches the panel to Search mode and activates a Job ID filter, shown as a "Job: NNNNN ×" chip, for that exact job.

  The results show all copy log rows for that job, not just the rows matching the error pattern. You can review everything the job copied, skipped, and failed. See the Job ID filter chip section above.

CSV export

Click Export CSV to download the full report as a CSV file with the following columns:

Error Pattern, Objects Affected, Distinct Jobs, Sample Job IDs, Buckets, Configs, Last Seen, Full Error Message

Multi-value fields (sample job IDs, buckets, configs) are pipe-joined (|) and double-quoted.

Truncation

The report returns at most 100 error patterns per query (ordered by object count). If more than 100 distinct patterns exist in the selected range, a yellow banner appears:

info

Showing top 100 error patterns by object count — narrow your date range to see all patterns.

To see less-frequent patterns, narrow your date range or add a source bucket filter.

---

Exporting results

Both modes support Export CSV (shown in the results bar when rows are present). The export is client-side — it generates from the rows already in the browser without an additional server request.

• In Search mode: exports up to 500 rows with all copy log fields.
• In Error Summary mode: exports up to 100 error groups with all aggregated fields.