Incremental Sync Dashboard

Overview

The Incremental Sync tab is the operational control center for all incremental copy activity. From this tab you can:

• See live pipeline health across every protection group.
• Trigger syncs manually.
• Drill into per-bucket window history.
• Review the full audit trail of past jobs.

Incremental sync copies only the objects that changed since the last completed window. Each run is far faster than a full copy.

Global Enable / Disable

The Incremental Sync switch on the Settings tab turns manual incremental syncs on or off for groups in Preparation Mode. It appears after Sync Windows and Active Vault Syncs. Click Enable or Disable to toggle the setting.

State	Behavior
Disabled (default)	These controls are grayed out, with the hover text "Enable incremental sync first": Sync Group Now, a bucket's Sync All, a window's Run Sync, Resume, and Re-sync, and Resume and Re-submit of incremental jobs on the Copy Jobs pages. Full copies are not affected.
Enabled	You can use all of these controls to start or resume incremental syncs for Preparation Mode groups.

Turn the switch on only after all full vault copies are complete.

Groups in Vault Mode are not affected. Their vault agent runs the syncs on the group's schedule.

info

Vault agents own incremental sync scheduling for vault-mode protection groups. Preparation Mode has no automatic schedule. Use Sync Group Now or Sync All, or a window's buttons in the week view.

For a vault-mode group these buttons are disabled. Hover over them to see the reason.

Smart Skip

The second switch on the Settings tab controls Smart Skip. It applies to every incremental sync, however it is started.

State	Behavior
Disabled (default)	Every object marked as changed is copied again, even if its content is identical to what is already in the vault.
Enabled	Before a changed object is copied, its current ETag is compared with the most recent successful copy of that object. If they match, the object is skipped instead of transferred again.

The change takes effect on the next sync. Running jobs are not affected.

Smart Skip avoids re-transferring data that is already correct at the target, such as metadata-only updates or re-uploads of identical content.

Smart Skip requires ClickHouse. If the job log says Smart Skip is enabled but not active, check the ClickHouse connection.

Planned Sync Time

The third switch on the Settings tab, Planned Sync Time, controls when inside each protection group's schedule window the vault agent syncs. It applies to vault-mode groups only.

Option	Behavior
Random within window (default)	Once per window, the vault agent picks a random moment inside the window and syncs then. The moment changes every window, so someone watching the network cannot know in advance when the air gap will open.
Window start	The vault agent syncs at the start of the window, within 30 seconds. The time is predictable.

warning

Window start makes every group that shares a schedule policy sync at the same moment. It gives up the unpredictability that makes the air gap harder to target. Prefer Random within window unless you need a predictable start.

Where the planned time is shown

The planned time is shown in each group's header on the Sync Windows tab and on the Active Vault Syncs cards, for example Agent: Today 12:00.

After the sync starts, the badge also shows one of these:

• · syncing since 12:37
• · synced 12:37
• · failed 12:37 (the agent retries)
• · skipped 12:37 (an operator skipped the window)

A change to the setting reaches each vault agent on its next heartbeat and applies from the next window. A time already picked for the open window is kept.

How the vault agent runs the planned sync

• One sync per window. After the window's scheduled sync has started, the agent does not start another in the same window, even after a restart. Failed or partial windows are still retried.
• Windows that cross midnight. A window such as 22:00 for 8 hours stays open until it ends the next morning.
• Repeat every N days. This follows the policy's run calendar. A failed or missed run day does not shift the calendar and is not caught up the next day. A run day that did not sync at all raises CV-117.
• Retry when blocked. If the console cannot be reached, the sync is blocked (for example by an Eyeglass ransomware pre-flight check), or the sync is waiting for ingest, the agent tries again every 5 minutes while the window is open.
• Groups sync independently. Each group's sync keeps the vault open until its jobs finish. The vault closes when the last sync is done.
• Restarts. Restarting the vault agent does not stop running copy jobs. At its first heartbeat, the agent picks them up again and closes the vault when they finish.

Week Navigator

The week navigator at the top of the Sync Windows tab controls which 7-day range of sync history is displayed.

Mode	What it shows
Live (default)	Real-time pipeline health: pending and syncing windows and status badges for the trailing 30 days. It refreshes every 30 seconds while any bucket is syncing.
Week view	All sync windows in the selected 7-day range, grouped by calendar day.

Navigation controls:

• Use the ← and → arrows to move one week back or forward.
• Use the calendar icon to pick a start date.
• Click Live to return to the live view. If the range reaches today, the view also returns to live mode.
• At the bottom of a bucket's week view, click ← Load earlier week to move back one week.

tip

If a bucket shows "No window data" for a week, no changes were recorded for it in that period.

Protection Group Summary Table

Each row represents one protection group. In the live view, the table refreshes every 30 seconds while any bucket is syncing. Click Refresh at any time.

Column	What it shows
Group	The protection group name. The colored dot matches the group's color in Source Data Protection Group Config.
Status badge	Live pipeline health (live view only). See Group Status Badge.
N buckets	The number of vault configs (buckets) in the group.
N syncing / N failed	Buckets currently syncing and failed windows (live view only).
N pending / All synced	Windows still to sync, or all synced (live view only). In the week view, it shows "expand buckets to see windows".
RPO Nh	The group's recovery point objective.
Net Open / Net Closed	For vault-mode groups, the vault network state last reported by the vault agent.
Sync Group Now	A button. See Sync Now / Sync All.

Click any row to expand the Bucket Drill-Down for that group.

Group Status Badge

The colored badge in each group row header shows the live health of the group's sync pipeline. It updates with each live-view refresh.

Badge	Color	Condition
Syncing	Blue	At least one window for this group is currently at status = syncing, which means a job is actively running.
Partial	Amber	One or more windows are at sync_failed or sync_partial, and nothing is currently syncing.
Complete	Green	No window is syncing, failed, partial, or ready to sync. Windows that are still open, skipped, or abandoned do not count.
Idle	Gray	Windows are ready and waiting to sync, but none is syncing, failed, or partial.

Sync Now / Sync All

Sync Group Now

Sync Group Now is on a group row. It shows a distribution plan before it dispatches jobs. The plan spreads each vault config's job across the available copy agents (ONLINE and not draining), so heavy and large buckets do not saturate a single agent.

The popup shows:

• The total number of sync jobs
• The heavy versus standard count
• A per-agent bar chart
• A table of heavy bucket placements

After you confirm, each job is picked up by its copy agent.

• If only one copy agent is available, all jobs go to it. The popup still appears.
• If no copy agent is available, the group shows "No copy agent available — start or undrain a copy agent and try again". The per-bucket Sync All, a window's Run Sync or Resume, and Resume in Copy Jobs refuse in the same way.
• Jobs start only if closed change windows are ready to sync. Otherwise the plan shows 0 jobs.
• Incremental Sync must be Enabled on the Settings tab.

Sync All

Sync All is on each bucket row. It dispatches copy jobs for all ready, failed, and partial windows of that bucket.

The Trigger column in Sync History shows manual for syncs started here and automatic for syncs that the vault agent started on its schedule.

info

Excluded buckets are never synced. A bucket excluded from vault protection is skipped by Sync Group Now. You exclude a bucket with the shield toggle in the Bucket Browser of Source Data Protection Group Config.

The per-bucket Sync All and Resume actions reject an excluded bucket with "Source bucket excluded from vault protection". Clear the exclude toggle first if you want the bucket to sync again.

Bucket Drill-Down

Click a group row to expand the bucket drill-down. It contains a bucket search, the bucket table, and the group's Sync History.

Which change windows are shown

Every group lists and counts all of its pending, failed, and partial windows, whatever their age. The totals at the top of the page add up every bucket that belongs to a protection group. The totals are Buckets with Pending Data, Pending Windows, Pending Objects, and Pending Data.

Change windows that were never synced are kept as long as the console keeps the audit events. The limit is the console's audit retention (Setup & Settings > Database > Data Retention), which has no default limit. Windows older than that are no longer shown or synced.

A failed or partial window older than the audit retention period still shows here and is sent again by each scheduled sync. It does not count toward the group's sync result, the 3-attempt limit, or alarm CV-114. Use Skip if such a window can never succeed.

Bucket search filter

Type any part of a config name in the search bar above the bucket table to filter the rows. The search is case-insensitive. Click ✕ or clear the text to reset it.

Bucket table

The table has one row for each vault config (a bucket and source namespace pair) in the group. Groups with more than 500 configs load in batches. Click Load N more buckets at the bottom of the table to show the next 500.

Column	What it shows
Config	The vault job config (policy) name.
Bucket	The source bucket being protected, with its namespace below it.
Device	The source device.
Windows	Live counters for syncing, ready, failed, partial, waiting-for-ingest, and open windows, or All synced.
Objects	The total pending objects across all tracked windows for this config.
Data Size	The total pending data volume across all tracked windows. The size is an estimate taken from the source access log. The copy itself always reads each object's real size from the source.
Action	The Sync All button. It is disabled when no pending, failed, or partial windows exist, or when Incremental Sync is disabled.

Click a row to expand the Change Windows panel for that config.

Change Windows

The change windows panel lists the windows of the selected config in chronological order. The per-window Run Sync, Resume, Re-sync, and Skip buttons are available in the week view, for yesterday and earlier.

Column	What it shows
Window Start > End	When this change window opened and closed.
Changes	▲ N NEW, ✎ N MOD, and ✗ N DEL (or the object count), plus the data size.
Status	The current state of the window. Failed and partial windows also show copied/total.
Job #N	The job of the last sync attempt for this window. Click it to open the job detail.

Window status values

Status	Meaning
Ready	The window is closed and waiting for a sync. In Preparation Mode, use Sync Group Now, Sync All, or Run Sync. In Vault Mode, the vault agent's scheduled sync handles it. A window that is still open shows Open · closes …. A window without changes shows No activity.
Syncing / In Progress	A copy job is copying objects for this window now.
Done / Synced N	All objects were copied successfully.
Partial	Some objects were copied but some failed, or the job was canceled after it copied some objects. The vault agent retries the window on its next sync.
Failed	The job failed entirely, or was canceled before any objects were copied. The vault agent retries the window.
Skipped	An operator skipped this window with Skip. It is not synced again, so its changes are never copied to the vault.
Waiting for ingest	A closed window that is held back because the audit ingest has not read all of its events yet. Hover over the badge for the reason. See When a Window Waits for Ingest.
Abandoned	The vault agent failed this window 3 times in a run, and alarm CV-114 fires. The window is sent again at every scheduled run day with 3 new attempts. If it can never succeed, for example because its objects no longer exist at the source, click Skip.

Sync History (Audit Trail)

The Sync History section is below the bucket table, inside the expanded group accordion. Scroll past the bucket rows to find it.

Click the Sync History toggle (clock icon) to show or hide the audit table. The history loads when you expand the group. To load it again, collapse and expand the group, or refresh the page.

The table shows up to 30 of the most recent sync windows for this group across all its configs.

Column	What it shows
Job	The copy job (#N). Click it to open the job.
Window	The start of the changelist window that was synced.
Status	The color-coded status of this window: completed, partial, failed, skipped, abandoned, or syncing.
Objects	The total objects in the changelist (detected as changed).
Copied	The objects successfully copied to the vault target.
Failed	The objects that failed to copy.
Started	When the sync started.
Trigger	manual (you clicked Sync Now or Sync All), automatic (the vault agent started it on its schedule), or resume (a batch resume). A single-window resume is recorded as manual.

Retries and Abandoned Windows

When a group's sync finishes with failed or partial windows, the vault agent retries them automatically in the same sync. It keeps the vault open, waits 60 seconds, and re-sends only the failed and partial windows.

It repeats this until one of these happens:

• Everything succeeds.
• Nothing is left to retry.
• The retry is blocked, for example by an Eyeglass ransomware event.
• The windows are abandoned.

Attempt	Behavior
1 to 2	About 60 seconds later, the agent re-sends the failed and partial windows. Only the objects still missing are copied again, so a retry of a mostly successful window is much faster than the original attempt.
3	The window is marked Abandoned and alarm CV-114 is raised.
After abandon	No more retries in this run. The window is sent again at the next scheduled run day with 3 new attempts.

Failures that happen while no sync is watching, for example after a vault agent restart, are picked up at the next heartbeat. The heartbeat starts a retry for every group that is not already syncing.

Only enabled copy policies are retried. A failed window on a disabled policy is not retried, is never abandoned, and raises no CV-114. It keeps its status. If you enable the policy again, the window is retried normally.

Alarm CV-114

Alarm CV-114 reports an abandoned window. It lists the group, config, window start, and vault agent.

You clear it manually. Clearing it does not restart the sync. After the cause is fixed, nothing else is needed, because the window is sent again at the next scheduled run day. If the window can never succeed, Skip it in the week view to unblock newer windows.

Auto-Recovery

Every 60 seconds, the console checks for windows still shown as Syncing whose job is no longer running. It closes them with the outcome the job actually had.

Job outcome	Window becomes	Alarms
Completed, no object failures	Done	None
Completed, some objects failed	Partial (or Failed if nothing got through)	CV-101, IS-001
Canceled	Partial	CV-101, IS-001
Failed, or the job record is gone	Failed	CV-101, IS-001

IS-003 is raised for every window closed this way. Jobs that are waiting to report, for example after a console restart, are left alone. The vault agent retries a window that ends up failed, as described above.

Stuck windows

Newer windows of a config wait behind an unresolved older one. A window is resolved when it is Done, Skipped, or Abandoned.

To clear a Failed window, sync it again. The oldest unresolved window can always be synced. If the window can never be synced, Skip it.

Federation Device Scope Error

When a sync refuses to run, you may see "Federation device scope could not be resolved".

Incremental sync reads a bucket's changes only from the source devices that belong to the policy's federation. A different cluster that uses the same namespace and bucket name therefore cannot affect your sync. With Delete Sync enabled, it also cannot remove a good copy from the vault.

If the device list cannot be determined, the sync fails with this message instead of running:

Federation device scope could not be resolved for <namespace>/<bucket> — refusing to run an unscoped changelist query. Check that the policy's source device is enabled and named.

To fix it:

1. Check that the policy's source device is enabled in Setup & Settings > Storage.
2. Check that every registered source device has a name.
3. Check that the console and its database are healthy, then run the window again.

Nothing is lost. The window stays unresolved and syncs normally once the cause is fixed.

Which Windows a Sync Sends

Changes are grouped into windows of the policy's sync interval, which is 6 hours by default and aligned to the clock. A sync can be started by the vault agent's schedule, Sync Group Now, Sync All, or Resume. In every case, the console sends these windows:

1. Failed and partial windows, and abandoned ones except on a retry, oldest first.
2. Every closed window since the policy's sync start that is not proven empty.

Windows are always sent oldest first. A window that the audit ingest has not fully read yet waits. See When a Window Waits for Ingest.

Where a policy's sync starts

• A policy that has been synced continues from its newest synced window.
• A policy that has never been synced starts from its last completed full copy. If it has none, it starts from its creation time. For example, a policy added to a group that is already in Vault Mode starts from its creation time.

Right after a full copy

A policy that was never synced has nothing to send until the window the copy started in closes. That takes up to one sync interval, plus the ingest wait.

Until then, Sync All answers "No closed unsynced windows found". This is expected, because everything before the copy is already in the vault.

Limits and special cases

• Never further back than the audit retention. Audit events are kept according to Setup & Settings > Database > Data Retention, so older windows cannot be checked or copied. This can happen to a policy that is disabled for longer than the retention period, or to a vault agent that is offline that long. Run a full copy of the policy to bring it up to date. A full copy is not available while the group is in Vault Mode.
• Re-created policies. A re-created policy is a new policy and starts at its creation time. Run a full copy after you re-create a policy.
• Changing the sync interval. Every period is still sent exactly once, with no gap and no overlap.

When a Window Waits for Ingest

A change window is copied from the audit events that the console has ingested for it. To make sure no events are missed, a closed window is synced only after the ingest has read past the window's end plus a 10-minute margin.

This must be true for every source device that feeds the window:

• The policy's source device
• Its geo-federation peers
• Any enabled device with exactly the same name as one of them

A quiet cluster with a healthy ingest is current. The time follows ingest runs, not traffic.

What you see

A waiting window shows Waiting for ingest instead of Ready. A failed or partial window keeps its badge with a clock next to it. Hover over the badge for the reason, for example:

Waiting for ingest: Source-primary ingested through 08:41 UTC; the window ends 09:00 UTC (+10 min margin)

Other reasons say that a device or node has had no complete ingest run yet, or that the ObjectScale SSH ingest schedule is disabled. The bucket row counts waiting windows next to the ready and failed counters.

What waits

Action	Behavior
Scheduled vault syncs	The agent tries again every 5 minutes. Ready windows are still synced. Only the waiting ones are left for the next attempt.
Run Sync, Re-sync, or Resume on a window	Refused with the reason. You are asked whether to Sync anyway.
Sync All or batch Resume	Queues the windows that are ready and says how many later windows were left waiting.
Sync Group Now	Syncs the windows that are ready and shows the reason for the ones that wait.

After a console restart or upgrade, the ingest state is restored automatically, so windows do not wait for new ingest runs.

These actions reset the ingest state of a device. Its windows then wait for its next completed ingest runs.

• Click Clear on an ObjectScale SSH node's row on the Ingest Offsets page.
• Wipe audit data.
• Purge a device.

Alarm CV-118

CyberVault sync waiting for ingest (warning) is raised when a source device used by an enabled vault policy has not completed an ingest run for more than 2 hours.

One alarm lists all affected devices and says whether the ObjectScale SSH ingest schedule is disabled. It clears by itself once the devices catch up. Check the ECS SSH Ingest job on the Jobs page for nodes that are not counted or fail to connect.

Sync anyway

When you start a waiting window by hand, the console shows the reason and asks "Sync anyway?".

warning

Confirm only if you accept that changes the ingest has not read yet will not be copied, and that the window will be marked as synced. For example, confirm if a decommissioned node will never be read again. Scheduled and vault-agent syncs never sync anyway.

Window Analysis

The Incremental Sync Report Analysis job (System Jobs > Schedules) runs every 60 minutes by default. It computes the new, modified, and deleted counts shown on this dashboard. It also identifies windows without changes so they can be skipped.

Syncs do not depend on this job. If the analysis is disabled, failing, or behind, windows are still synced. Only the dashboard counts are incomplete, and empty windows cannot be skipped. As a result, Sync History may show more windows with 0 objects.

Alarm CV-119

CyberVault window analysis is behind (warning) is raised when an enabled policy has no analyzed window in its last 4 sync intervals. The alarm detail names the cause.

Detail	What to do
The Incremental Sync Report Analysis schedule is disabled	Enable it in System Jobs > Schedules.
The analysis is waiting for the audit ingest	Fix the audit ingest (see CV-118). The analysis then catches up by itself.
The analysis is not keeping up	See the latest Incremental Sync Report Analysis job for details.

The alarm clears by itself when every policy is current again.

Scheduling

Vault agents own incremental sync scheduling. For vault-mode groups, each vault agent manages its own schedule inside the secure vault network, so syncs do not depend on console availability. Preparation Mode groups have no automatic schedule and sync only when you start them by hand.

Sync Schedule Policies

Configure policies in the Sync Schedules tab. Each policy defines these fields:

Field	Description
Window start time	The hour and minute when the vault agent may begin syncing for this group. This is UTC. Every VM syncs to NTP and runs in UTC.
Window duration	How many hours the sync window stays open.
Repeat every N days	The run calendar. The policy syncs on its First Sync date and every N days from it (1 is every day, 7 is weekly), always at the window start.

Assign a policy to a protection group in Source Data Protection Group Config (group edit).

Window Ordering

Windows of a config are always synced oldest first, in one job. A newer window cannot be synced on its own while an older one is unresolved. The older window must first be Done, Skipped, or Abandoned. A Partial window is retried.

Pausing a Group

Protection groups have no enable or disable switch of their own. To pause sync for one group, for example during bucket maintenance, disable its copy policies:

1. Open Source Data Protection Group Config > Source Data Protection Policy Config (Step 5).
2. Click Edit.
3. Set Enabled to Disabled.
4. Click Save Policy.

Disabled policies get no new jobs and are left out of retries. Enable them again to resume.