Jobs

The Jobs page shows the history and current state of the system jobs. To open it, click System Jobs in the top menu.

System jobs include audit ingest runs, ObjectScale inventory scans, incremental sync report analysis, and device purges. Use this page to monitor what the system does in the background, investigate failures, and manage job schedules.

The page has two tabs: Job Monitor and Schedules.

info

Copy jobs (full copies and incremental syncs) are not listed here. They appear under Cyber Vault > Copy Jobs - Live View & History.

The copy-job sections later on this page describe queue management, recovery, and Target Skip. They also describe the job details panel that copy jobs share with system jobs.

Overview

The Job Monitor tab combines a summary of job statistics with a filterable job table. Running jobs are also listed in an Active Jobs section at the top.

The tab refreshes automatically every 30 seconds. A countdown next to the page subtitle shows when. Click ↻ to refresh immediately.

Schedules tab

The Schedules tab shows one card for each schedulable job type:

• ECS Ingest (SSH)
• ECS Inventory
• Incremental Sync Report Analysis

The cards shown depend on the license. Each card has an on/off toggle, an interval selector, the last-run time, and, when the schedule is enabled, the next scheduled run. The tab refreshes every 30 seconds.

Control	Description
Enable (toggle on)	Turns on automatic runs at the configured interval.
Disable (toggle off)	Stops new automatic runs from starting. It takes effect immediately, and a run already in progress is allowed to finish.
Run Now	Starts a single on-demand run regardless of the toggle state. You can run an ingest or scan once without turning on the recurring schedule.

Run Now on Incremental Sync Report Analysis does nothing while its schedule is disabled or while a run is still in progress.

A run started with Run Now shows the Manual trigger badge. A scheduled run shows Auto.

Incremental Sync Report Analysis

This job computes the new, modified, and deleted counts of each sync window for every enabled vault policy. This lets the Incremental Sync dashboard load quickly. It also identifies windows without changes.

How a run works

The job runs in the background with a time budget of 10 minutes per run. Policies it does not reach are analyzed first in the next run.

The job ends Completed with a summary such as "120 of 500 analyzed, 40 waiting for ingest".

• A policy that is waiting for ingest is not a failure. It moves on as soon as the audit ingest has covered its windows.
• If ClickHouse is overloaded, the run stops early with "stopped: ClickHouse overloaded". This is not counted as a failure.
• Canceling the job stops it the same way.
• A failed run is tried again one interval later.

Step log

Step	What it shows
discover	Which policies are analyzed, stalest first.
compute	One line per policy with the range analyzed and the time taken, or the reason it failed.
store	The windows saved.

Syncs do not depend on this job. If it is disabled or behind, windows are still synced. See alarm CV-119.

Stat Cards

The stat cards at the top of the page give a quick summary across all jobs.

Card	Description
Total Jobs	Count of all jobs ever recorded.
Completed	Count of jobs that finished successfully.
Failed	Count of jobs that ended in an error state.
Running	Count of jobs currently executing.
Avg Duration	Mean wall-clock time across all completed jobs of any type.

Job Table

The job table lists all jobs, newest first, with 50 rows per page.

Each row shows the job ID, type, device or config, status badge, trigger, start time, duration, and step progress. The trigger is Auto for scheduled runs and Manual for runs someone started.

Click View on a row to open the detail flyout for that job. Click it again to close the flyout.

Filters

Filter	Description
Job Type	Narrows the table to ECS SSH Ingest, ECS Inventory Scan, Device Purge, or Incremental Sync Analysis. Copy jobs are not offered here. Use Cyber Vault > Copy Jobs - Live View & History instead.
Status	Narrows the table to jobs in one state: Completed, Failed, Running, Pending, or Canceled.
Device / Config	Narrows the table to jobs initiated by a specific device or configuration name.
Time range (copy jobs only)	Narrows the copy-job history to jobs created in the last N days.

The time range filter applies to the copy-job history under Cyber Vault > Copy Jobs - Live View & History. The System Jobs page does not include a time filter.

Choose All days (no filter, the default), Last 1 day, Last 2 days, and so on up to Last 30 days. Dates are filtered on job creation time, not job start time.

Device Purge jobs

When you permanently purge a deleted device, the cleanup runs as a reviewable Device Purge job. To start a purge, go to Setup & Settings > Storage > Deleted Devices > Purge. The job lets you see exactly what was removed.

Its steps mirror the cascade:

1. Resolve the device.
2. Pause and check ingests.
3. Revert source ACLs.
4. Delete the ObjectScale service accounts that the console created.
5. Purge the ClickHouse audit trail.
6. Delete the PostgreSQL configuration, inventory, and groups.
7. Strip the device from settings.
8. Remove the registry row.

Best-effort remote steps against an unreachable cluster complete with a warning and never fail the job. A failure in the PostgreSQL cascade or the settings strip fails the job.

Job history is never touched. You still see the device's past copy and ingest jobs after the purge.

Job Detail Flyout

Click View on a job row to open a flyout panel on the right side of the page. While the job is running, the flyout refreshes every 3 seconds.

Section	Description
Full configuration	The parameters the job was started with, such as cluster name, device target, or copy configuration name.
Log excerpt	The most recent log lines produced by the job. For long-running jobs, only the tail of the log is shown.
Error message	If the job failed, the error reason is shown at the top of the flyout.
Step-by-step progress	For multi-step jobs such as Cyber Vault copy or audit ingest, each step with its status and how long it took.

ObjectScale SSH ingest steps

ObjectScale SSH ingest jobs have five steps:

1. Connecting to ObjectScale nodes via SSH (setup)
2. Listing files and comparing against ingest state (scan)
3. Collecting and parsing log files (ingest)
4. Persisting to ClickHouse
5. Summarizing results

The scan step is where the listing and comparison happen. For each node it logs how many files were listed, how many were already complete and will be skipped, and which gz files still need processing, by name. You can see what the run decided to skip and process before any data is downloaded.

Step color

A step that recorded any failure ends failed, with a red cross and the error message under the step name. It keeps its counter chips. A green tick means no failures.

A job can show a failed step while the job itself is completed. This is the case for a degraded ingest run, described below.

Step logs

Each step that produced log output shows a N log lines toggle to expand the log, and a Copy button beside it.

Copy places the step's full log on your clipboard in chronological order, oldest line first. You can paste it into a support ticket or chat. You do not need to expand the log first.

The button briefly shows a green Copied checkmark to confirm. On consoles served over plain HTTP, where the browser clipboard API is unavailable, the button falls back to a legacy copy method automatically.

Ingest Summary cards (ObjectScale SSH ingest)

For ObjectScale SSH ingest jobs, the Ingest Summary section at the top of the flyout shows these stat cards:

• Records Decoded
• Records Inserted
• Inserted to CH (or "Dry Run (logged)")
• Elapsed time

These cards appear only when the value is greater than zero:

Card	Description
Skipped (gray)	Files already fully ingested on an earlier run. On steady-state runs this is most of the files.
Errors (red)	Log lines that could not be decoded as valid audit records.
IO Errors (amber)	Files that could not be read or written at all, such as an SSH connection drop or a corrupt archive.
CH Errors (red)	ClickHouse insert failures that remained after all retries. The ingest continues with the next batch.
CH Retries (amber)	Insert attempts that had to be repeated. Retries that eventually succeeded lose no data.

If CH Errors is above zero and Records Inserted is lower than Records Decoded, the difference is the number of rows that were not saved.

Degraded ingest runs

A run is degraded when it completes but part of the work failed. For example, some nodes refused the connection, or some files hit I/O errors.

For a degraded run:

• The Degraded badge names the cause, for example "1 of 3 node(s) failed to connect; 2 I/O error(s) — see Ingest step log — the run completed with the rest."
• The Ingest step is shown failed (red) with its counters and reason.
• Alarm AU-004 (ECS SSH audit ingest degraded) is raised.
• The job status stays completed, and the work that did succeed is kept.

Vault sync freshness line (ObjectScale SSH ingest)

Vault sync windows are synced only after the audit ingest has read all of their events. See When a Window Waits for Ingest on the Incremental Sync page.

Each ObjectScale SSH ingest run reports, for each node, whether it moved that point forward. The report is one line on the Summarizing results step, for example:

Vault sync freshness: 3 of 4 node(s) read to the end; 1 not counted this run: 10.1.1.4. ecs-vdc1 ingested through 2026-11-04 10:12:31 UTC (was 2026-11-04 09:42:19 UTC).

Term in the line	Meaning
read to the end	The node was read completely and counts from this run's start.
wait for the archive	The node's live log was rotated while it was being read. It counts once that archive has been read, normally on the next run.
not counted this run	Anything else, such as a connection or listing failure or a failed insert. The node keeps its earlier time.
ingested through	The device's time, which is the oldest of its nodes' times. Windows ending at least 10 minutes before it can be synced.

If the device's ingest state was reset after the run started, for example with Clear on the Ingest Offsets page, the line says the run is not counted. The next complete run counts.

Node / File Details table

The Node / File Details table lists every file the run touched, with one row per node and file. A State badge shows the result:

State	Meaning
New data (green)	The active log file had new records since the last run.
Up to date (blue)	The active log file had no new records since the last run.
Complete (gray)	A rotated archive was fully ingested.
partial_error	An I/O error cut the read short. The checkpoint reflects what was read, so the rest is picked up on the next run.
rotated_away / rotation_pending	The log was rotated while the run was reading it, or ObjectScale was still compressing the rotated copy. Nothing is lost, and the data is read on a later run.
Missing (red)	The expected file could not be found on the node.
CH Insert Failed (amber)	The file was read successfully but ClickHouse failed to store it.
error	The file could not be processed for another reason.

The header above the table shows how many rows are displayed versus the true total, for example showing 10 of 47. The stored list is capped for each job.

Active Jobs

The Active Jobs section sits above the stat cards. It lists running, pending, auto-paused, and awaiting-report jobs. Each entry shows its type, job ID, device or config, and N / M steps progress.

There is no hover popover. Open the job's flyout with View for live step-by-step progress.

Actions

Each job in the Active Jobs section has a Cancel button. Clicking it sends a cancellation signal.

The job finishes its current unit of work before it stops, so the status can take a short time to change to canceled.

Resume is not available on this page because copy jobs are not listed here. To resume a failed copy job, use Cyber Vault > Copy Jobs - Live View & History.

Bulk Actions (Multi-Select)

Use the checkbox column on the left of each row to select multiple jobs. The select-all checkbox in the header selects all jobs on the current page.

When any jobs are selected, a bulk action toolbar appears above the table:

• Cancel (N) sends a cancel signal to all selected jobs that are Running, Pending, or Awaiting Report. The count shows how many are eligible.
• Clear selection deselects all jobs.

Jobs that are not eligible for an action are silently skipped. If you select 50 jobs and only 30 are running or pending, only those 30 are canceled. The counts in the toolbar show exactly what will happen before you click.

The Cancel buttons in the Active Jobs section stay available and are unaffected by multi-select.

Manual ingest trigger

To start an on-demand ingest or inventory run without waiting for the next scheduled cycle, click Run Now on that job type's card in the Schedules tab.

Workflows

Monitor a running Cyber Vault copy job

1. Open Cyber Vault > Copy Jobs - Live View & History. Copy jobs are not listed on System Jobs.
2. Find the running job in the live view.
3. Open its detail flyout for step-by-step progress.

Investigate a failed ingest job

1. Set the Status filter to Failed.
2. Set the Job Type filter to ECS SSH Ingest.
3. Click View on the failed job row to open the detail flyout.
4. Read the error message and log excerpt to determine the cause.

Trigger a manual ingest run

1. Open the Job Monitor tab and confirm that no ingest job is running in the Active Jobs section.
2. Open the Schedules tab.
3. Click Run Now on the ECS Ingest (SSH) card.

Copy Job Active Queue Management

The Copy Job Active Queue Management sub-tab shows copy jobs as they move through dispatch, in real time. It is under Cyber Vault > Copy Jobs - Live View & History.

Running / Capacity bar

The bar at the top shows Running: N / M (pct%).

• N is the number of jobs currently executing.
• M is the total concurrent-job capacity across all online copy agents.
• Queued jobs waiting for a slot are shown separately as · X queued.

The bar turns amber when running is at 100% of capacity and jobs are still waiting.

Agent filter pills

A row of pills below the bar shows one pill for each online copy agent, with its active job count (running plus queued). Click a pill to filter the tabs to that agent's jobs. Click it again to clear the filter.

Live Jobs tab

This tab shows jobs that are executing: Running and Awaiting Report.

Each row has a Failure column. It is empty for running jobs and shows a badge for failed jobs.

Badge	Meaning
Canceled	An operator clicked Cancel.
Watchdog: never started	The job was assigned but the agent never acknowledged it.
Watchdog: timeout	The agent went offline mid-run and did not reconnect in time.
Watchdog: re-queued	The job was moved back to the queue because its assigned agent went offline. The job continues. This is not a failure.
Engine error	The copy agent reported an unrecoverable error.
Ransomware Event	The job was blocked or canceled by the ransomware-detection check.
No copy agent	The job failed before it started because no copy agent could take it.

For No copy agent, the fix depends on how the job was dispatched:

• Auto: no copy agent was online and undrained. Start or undrain a copy agent, then run the job again.
• Pinned to one agent: that agent was offline. Choose Auto or another agent, or start it.

Queue tab

This tab shows Queued jobs, which are assigned to an agent but waiting for an execution slot. Each copy agent runs up to its maximum number of concurrent jobs, which you set in Agent Fleet Management.

Queued jobs are never lost. They start as soon as a slot is free.

Copy Job Recovery

Awaiting Report status

When a copy agent goes offline while a job is running, or the console restarts, the job becomes Awaiting Report. The outcome is not known yet. The agent may still be running the job and reports back when it reconnects. The job is not failed.

• If the agent reconnects within the crash-timeout window, the job advances to completed or failed normally. The default is 30 minutes. You can set it globally or for each agent in Agent Fleet Management.
• If the agent does not reconnect in time, the job fails with the reason Agent did not reconnect within N minutes. Before it fails the job, the console asks the agent directly whether it is still running the job. If it is, the job continues.
• If a watchdog failed a job but the reconnecting agent later reports it as still running, the job returns to Running automatically. No operator action is needed.

Copy jobs are never moved to another agent

If the agent running a copy job goes offline, the console does not dispatch the job to a different agent. The original agent may still be running it, and both agents would copy the same data.

If the job is confirmed failed, run it again with Resume.

Target Skip Mode

Full-copy jobs run in Standard, Smart Skip, or Target Skip mode. You select the mode at dispatch.

In Target Skip mode, objects already present on the target are skipped. The copy log marks them target_skipped, with the reason that decided the skip.

Reason in the log	Meaning
tier1 strict ETag+size	The source ETag and size matched the target object exactly. This is the safest reason.
tier1.5 date-trust	You supplied a cutoff date, and the source object was last modified before it with a matching size. Correctness depends on the cutoff you attested.
tier2 source-etag match	The target carried a marker with the source ETag, and it matched the current source ETag.
tier2.5 range-hash	Sampled byte ranges of source and target matched. A change outside the sampled ranges is not detected, so use this only for read-only sources.

Objects that match no reason are copied. A skipped status is a Smart Skip match. A copied status means the object was transferred.

Counters in the job detail flyout

The top tiles are Objects Copied, Skipped, Failed, Single-Part, and Multipart. They update every few seconds and count all skips together, including Smart Skip.

For Target Skip jobs, a per-tier breakdown below the tiles shows how many objects each reason skipped and how many were copied. It is read when you open the flyout and does not refresh while the job runs. Close and reopen the flyout to update it.

Tips

• If a job seems stuck in the running state, check System Monitor for what it is waiting on.
• Canceled jobs keep their partial results. For ingest jobs, events ingested before cancellation are preserved.
• The average duration stat card covers all job types. For a type-specific baseline, filter the table to that type.
• A job showing Awaiting Report is not necessarily failed. Wait for the crash-timeout window before you conclude that the job is lost.