Troubleshooting

This page collects the common problems in AirGap for ObjectScale, grouped by subject. Each entry gives the symptom, the cause, and the fix, and names the page that explains the feature in full.

tip

Before you dig in, check the Alarms page and System Monitor. Most problems on this page raise an alarm that names the cause.

Licensing

You install a license by uploading the license archive. For the steps, see the Licensing page in the Setup section.

Message	Cause	Fix
"not released for this appliance"	The archive was issued for a different Appliance ID.	Request an archive for this appliance's ID.
"This license key is for a ... product and not valid for a ... product"	The archive was issued for the other edition of the product. Each console accepts only licenses for its own edition.	Request an archive for this console's edition. If the console shows the wrong branding, ask support to correct it, then restart the console.
"Upload the license archive (.zip) issued for this appliance"	The file is not a .zip, is empty, or is larger than 1 MB.	Upload the original .zip archive you received.
"The license manager is not running"	The license manager failed to start. The console keeps running as R1, and no license can be installed.	Find the reason in the console log, fix it, and try again.

Database and ClickHouse

For the settings behind these entries, see the Database page in the Setup section and the ClickHouse Health page in the System Health section.

Connection and credentials

Symptom	Cause	Fix
Save rejects the ClickHouse password, or every ClickHouse request fails with the same message	The password contains non-ASCII characters, such as accented letters or emoji. HTTP headers cannot carry them.	Change the password to ASCII characters only. Generated passwords are always alphanumeric.
ClickHouse requests fail or time out after you changed the port	The port points to the native TCP interface (9000 or 9440). The console speaks only HTTP and HTTPS.	Use the HTTP port (8123) or the HTTPS port (8443).
The new PostgreSQL credentials have no effect	Credentials take effect only after a backend restart.	Restart the console backend service.
Apply on the PostgreSQL sizing card returns a permission denied error	ALTER SYSTEM SET needs a superuser, or GRANT ALTER SYSTEM ON PARAMETER on PostgreSQL 15 or later.	Use an application database user with that privilege.

Retention and mutations

Symptom	Cause	Fix
Disk space did not drop after Apply Retention	ClickHouse removes expired data in background merges over the following hours.	Wait a few hours, then check row counts and disk space.
A changed audit retention ends with an error, and a MATERIALIZE TTL mutation is still pending	materialize_ttl_recalculate_only is not set on the audit table, so ClickHouse rewrites the whole table.	Wait for the mutation to finish, then run ALTER TABLE pvc_audit_events MODIFY SETTING materialize_ttl_recalculate_only = 1 as a ClickHouse administrator. Check Mutations on the ClickHouse Health page before you apply again.
The console log warns that ClickHouse refused the MODIFY SETTING statement	The console's ClickHouse user cannot change table settings.	Run the statement above as a ClickHouse administrator. The console retries after a restart.
The cv_job_metrics retention change has no effect	Its TTL is set when the table is created. Apply Retention does not govern it.	Restart the ClickHouse service.

Memory, load, and slow inserts

Symptom	What to do
Alarm CH-003 or CH-004 (ClickHouse memory)	See the steps below.
Inserts are slow	Open Table Storage & Parts Health and check the table's parts count. If it is very high, check Merge Pressure. Inserts slow down while merges catch up and recover on their own.
High ClickHouse CPU	Open Currently Running Queries and look for queries with a high elapsed time. Expand Recent Queries to read the full text.
Mutations still listed after a data wipe	Open Mutations and look for any mutation that is not Done. If a completed mutation stays listed, contact support.

To handle CH-003 or CH-004:

1. Read the Memory panel. Note Resident, In use, Not in use, and the likely cause. If most memory is in use, the cause is load, not held memory. Check Currently Running Queries and Recent Queries for heavy queries.
2. Create a support bundle in Setup & Settings > Backup & Restore > Create Backup Now. It contains the memory history.
3. If memory not in use stays high, restart ClickHouse at a quiet moment. Audit ingest and agent writes retry by themselves. The alarms clear a few minutes later.
4. Send the bundle to support with the time of the alarm.

info

The console never purges or restarts ClickHouse by itself.

Audit ingest

For the dashboards, see the System Monitor page in the Monitoring section.

IS-002 stays active after the cluster caught up

IS-002 stays active while any cluster is more than 1 hour behind, including clusters its list does not show. It clears only when every cluster that went above 1 hour is back at or below 40 minutes.

1. Check System Monitor > Audit Ingest Lag. A red card means that cluster is still more than 1 hour behind.
2. In backend.log, find the lines that name every breaching cluster: IS-002 RPO breach — cluster='…'.
3. Find the line that starts with Is002RpoBreachClearHandler. It names the clusters that still keep the alarm open.

IS-002 clears itself within a minute or two once all clusters are back.

• A device you no longer ingest keeps breaching until its audit data is purged or ages out.
• While IS-004 is active, IS-002 is left as it is.
• If you clear IS-002 by hand while a cluster is still behind, a new one is raised within about a minute.

IS-004 Audit ingest RPO status unknown

The console could not read the audit ingest lag for 15 minutes. The Audit Ingest Lag card in System Monitor shows Stale or Unavailable, with the error.

1. Check the ClickHouse Health page for memory, circuit breakers, and running queries.
2. Check the Statistics index built detail. On a large audit table where it is below 100%, the first read after a console start, wipe, purge, retention change, or database switch is a full read. ClickHouse stops it after 360 seconds, and the console then reads only the last 2–6 hours.
3. The short read fails only when its window holds more than 100 million audit records (error 158 in the Error field), or when ClickHouse is in trouble.
4. Search the console log for ClickHouse audit statistics [lag] refresh failed.
5. In ClickHouse Health > Recent Queries, find the failed read. It shows the SQL text and error code.

IS-004 clears itself after the first successful read.

Copy jobs

For the feature, see the Copy Jobs page in the Operations section.

A job failed

1. On Live View & History, filter Status to failed.
2. Click the job row and read the error in the detail panel. Common causes are a lost agent connection, an unavailable vault device, and an expired source credential.
3. Read the copy event lines in the job's step log. Use the bucket_walk step of a full copy, or the copy step of an incremental or Batch Sync copy.
4. If the error was at the object level, click View Detailed Copy Log to see which objects failed.
5. Fix the cause, then click Resume on the job row, or Resume Failed Job in the flyout.

Log line	Meaning and fix
WAITING FOR NETWORK with no NETWORK RECOVERED after it	The source or target did not answer again. After 15 minutes of retries, the objects that needed it failed. Check name resolution and routing from the copy agent.
UPLOAD KEPT	Lists large objects whose upload the agent kept for the next run, with the parts already on the target.
UPLOAD DISCARDED	Names a kept upload that could not be continued, and why.
ABORT FAILED	Gives the ID of an upload left open on the target.

info

Objects listed as UPLOAD KEPT continue from their first missing part only if the resumed job runs on the same copy agent. On another agent, they are copied again from the start.

A re-run skips far fewer objects than expected

If a re-run of an unchanged bucket skips far fewer objects than it lists, the Smart Skip baseline lookup may have failed.

When ClickHouse rejects the lookup, times out, or cannot be reached, the objects on that page are copied instead of skipped. Nothing is lost, but the job copies more than it needs to.

Look for these signs:

• The bucket_walk step log shows the first failure with its reason, for example Smart Skip baseline lookup failed for bucket 'b' (1000 of 1000 keys unchecked): HTTP 500: ….
• The summary step log shows Smart Skip: baseline lookup failed for N page(s); M objects were copied without a skip check.
• The job summary details show smartSkipFailedLookups and smartSkipUncheckedObjects above zero.

Check ClickHouse load, timeouts, and connectivity. Changed data is not the cause.

Copies pause when DNS is down

Copy agents on a built-in profile keep no DNS answers, so copies pause while a lookup fails. A copy waits and retries for up to 15 minutes before the object fails, and its multipart upload is kept. The console and vault agents keep using the last good answer for up to 15 minutes.

Restore the DNS server. See the Advanced Settings page in the System Settings section.

Incremental sync

For the feature, see the Incremental Sync page in the Operations section.

Symptom	Cause	Fix
A window shows Waiting for ingest	A window syncs only after the audit ingest has read past its end plus a 10-minute margin, for every source device that feeds it.	Check the ECS SSH Ingest job on the Jobs page for nodes that are not counted or fail to connect. Scheduled syncs retry every 5 minutes.
The console asks Sync anyway?	You started a waiting window by hand.	Confirm only if you accept that changes the ingest has not read yet will not be copied. Scheduled syncs never sync anyway.
Alarm CV-114, window Abandoned	The window failed 3 attempts.	Fix the cause. The window is sent again at the next scheduled run day. If it can never succeed, Skip it in the week view. Clearing the alarm does not restart the sync.
Newer windows do not sync	An older window is unresolved. A window is resolved when it is Done, Skipped, or Abandoned.	Sync the oldest unresolved window again, or Skip it if it can never sync.
A failed window on a disabled policy is not retried	Only enabled policies are retried.	Enable the policy. The window is then retried normally.
The sync fails with "Federation device scope could not be resolved"	The console could not determine the policy's source devices.	Check that the policy's source device is enabled in Setup & Settings > Storage and that every source device has a name. Then check that the console and database are healthy and run the window again. Nothing is lost.
Run Sync is refused for a group in Vault Mode	The vault agent owns the sync schedule in Vault Mode.	Use the vault agent's schedule, or exit Vault Mode.
Dashboard counts are incomplete, or more windows show 0 objects	The Incremental Sync Report Analysis job is disabled, failing, or behind. Syncs still run.	See alarm CV-119 in the Alarms section of this page.

Vault agents and target authentication

For the feature, see the Target Authentication page in the Vault Operations section.

Symptom	Cause	Fix
No devices are listed	No vault-type ObjectScale devices are configured.	Add vault-type storage devices in Setup & Settings > Storage.
Ownership did not update after Exit Vault Mode	The tab loads ownership only when it is opened.	Refresh the tab.
A device shows Agent Owned, but the agent is offline	The ownership record persists while the agent is unreachable.	Use Exit Vault Mode, or contact the vault network administrator to release the lock.
The console blocks service account provisioning or key rotation	The vault agent owns target authentication for the device. This prevents credential conflicts.	On the Vault Data Sync Dashboard, use Exit Vault Mode in the Operational Mode panel.

Alarms

For the rules and codes, see the Alarms page. The IS-002, IS-004, CH-003, and CH-004 alarms are covered in the Audit ingest and Database and ClickHouse sections of this page.

Symptom	Cause	Fix
The Active tab is empty, but the badge shows a count	The tab and the badge poll independently.	Click Refresh, or wait up to 10 seconds.
An alarm you cleared came back	The condition is still true. The engine raised a new event on the next tick, subject to cooldown.	Fix the root cause, or suppress the alarm while you investigate.
AG-001 is still active after the agent restarted	The engine clears it on the next 60-second tick after the agent reconnects.	None. It clears automatically within 60 seconds.
A disabled rule still shows events in History	Disabling a rule clears active events and stops new ones. CLEARED and SUPPRESSED events are never deleted.	None. History is a permanent audit trail.
Load Default Rules shows "0 inserted, N skipped"	All default rules already exist, and none were overwritten.	None. This is normal. The console adds missing default rules at every startup.

CV-118 Cyber Vault sync waiting for ingest

Vault sync windows wait for the audit ingest of the listed sources. Check the Vault sync freshness line of the ECS SSH Ingest job on the System Jobs page. The alarm clears by itself when the devices catch up. See the Incremental sync section of this page.

CV-119 Cyber Vault window analysis is behind

Open System Jobs > Schedules. If Incremental Sync Report Analysis is disabled, enable it. The alarm then clears, and it is raised again with the current cause until every policy has a current analyzed window.

Otherwise, act on the cause named in the alarm:

Cause	Fix
"the analysis is waiting for the audit ingest"	Fix the audit ingest, not the analysis limits. The source device and its federation peers need a complete ingest run of every node. The analysis then catches up by itself.
"the analysis is not keeping up"	Open the analysis' latest job. The compute step shows the policies that failed and how many were left. The discover step shows how many wait for ingest.

The cause can switch between these two. Each switch clears the alarm and raises it again. Windows keep syncing meanwhile.

See Also

• Alarms
• ClickHouse Health
• System Monitor
• Licensing