Alarms

The Alarm System monitors your environment continuously using rule-driven logic. The rules are evaluated on a 60-second background tick.

When a condition is met, an alarm event is created. It appears in the count badge on the Alarms top-menu tab and on this page.

Most alarms clear themselves when the underlying condition goes away.

How raise and clear work

Raising

Alarms are raised in two ways:

• Service events. Specific failure points in the console and agent raise alarms in real time, the moment the condition occurs. Examples are an agent going offline, a ClickHouse write failure, an auth error, or a job duration that is exceeded. These fire immediately, not on the tick.
• Rule-based triggers. The 60-second evaluation tick checks metric thresholds, named database queries, job status, and job duration against the rules you have uploaded. When a rule fires, a new event is created.

Clearing

The alarm engine owns clearing, so you do not have to clear most alarms manually. On every 60-second tick, for each ACTIVE event, the engine:

1. Checks whether a custom clear handler is registered for that alarm code. For example, AG-001 is cleared when the offline agent reconnects.
2. If no custom handler fires, evaluates the rule's clear_condition block (timeout, metric drop, or job succeeded).
3. If either path returns true, moves the event to CLEARED.

Deduplication and cooldown

• Deduplication. Deduplication is per agent, per device, or per rule, depending on the alarm. While an alarm is ACTIVE, the engine does not create a second event for the same key, however often the trigger condition is true.

  • One event per agent: AG-001, VA-001, VA-003, VA-005 to VA-008, VA-010.
  • One event per device: VA-009.
  • One event per rule: everything else, including CV-114, CV-116 to CV-119, IS-002, IS-004, and every metric rule.

  A per-rule alarm hides later occurrences while one is active. For per-rule alarms, the cooldown counts from the latest event, even a cleared one.

• Cooldown. After an event is cleared, the rule does not fire again until the configured cooldown period has elapsed. The default is 60 minutes. This prevents rapid re-firing when a condition is borderline.

IS-002 and IS-004 are an exception to the cooldown. They clear themselves as soon as their condition is gone. Once their latest event is cleared, by the console or by hand, the cooldown no longer holds back the next one, so a breach that comes back is raised at once. Their own conditions limit how often they can repeat. See RPO alarms.

Nav badge

The Alarms tab in the top menu shows a red count badge when active alarms exist. The badge polls every 30 seconds. Clicking it opens this page on the Active tab.

On the Cyber Vault page, a compact inline badge in the page header shows the same count. Hover over it to see the newest active alarms, up to 8. Click it to go to this page.

Active tab

The Active tab shows all alarm events currently in the ACTIVE state, newest first. It shows them however long ago they were raised, so the list always matches the bell badge count.

Auto-refresh

The tab refreshes automatically every 10 seconds. A bar above the alarm list shows:

• Next assessment in Ns: the seconds until the next 60-second engine tick fires. It counts down in real time.
• Evaluating…: shown briefly while the tick is running.
• CSV button: downloads all current active alarms as a CSV file.
• Refresh button: fetches the latest data immediately and resets the 10-second auto-refresh timer.

Alarm cards

Each active alarm shows:

Field	Description
Severity badge	Critical (red), Warning (blue), or Info (gray). It has a pale fill with the severity name in normal text.
Rule name	The name from the rule document.
Code badge	The short alarm code, for example AG-001.
Triggered time	When the alarm was first raised.
Context fields	Key and value pairs from the alarm's context, such as agent name, config name, job ID, metric values, and error text. Each field is labeled and formatted: bytes as GB or MB, durations as h/m/s, percentages, and MB/s.
Auto-clears when chip	The clear logic from the rule, for example "Auto-clears after 24h", "Auto-clears when agent comes online", or "Manual clear only". It tells you what resolves the alarm without your action.

Actions

Action	Description
Clear	Moves the event to CLEARED immediately. Use it after you have investigated and resolved the condition, or when the alarm fired but the condition has already resolved itself.
Suppress	Silences the alarm for exactly 24 hours. The event moves to SUPPRESSED, the monitor skips re-evaluation, and the alarm leaves the Active list. After 24 hours, if the trigger condition is still true, a new ACTIVE event is created on the next tick. Use it when you know about the condition but cannot address it immediately and do not want repeated notifications.

CSV export

The CSV button downloads all current active alarms as a comma-separated file. The columns are:

• Alarm code
• Rule name
• Severity
• Triggered time
• One column per context field, as the union of all fields across all active alarms

Use the file to share a snapshot of the current alarm state or to feed it into an external system.

History tab

The History tab is a paginated table of all alarm events (ACTIVE, CLEARED, and SUPPRESSED), newest first.

Status badges

Status	Meaning
ACTIVE	The condition is currently true and unacknowledged.
CLEARED	The condition resolved. The engine cleared it automatically, or an operator cleared it manually.
SUPPRESSED	Manually snoozed for 24 hours. It returns to ACTIVE after expiry if the condition is still true.

Clear context

For automatically cleared events, the Cleared column shows the clear time.

Expand a row, or read the raw event, to see a clear_context field that records the reason. Examples:

• reason: handler:AG-001 means the AG-001 custom handler cleared it because the agent came back online.
• reason: auto_clear means the engine's rule-based clear_condition evaluation cleared it.
• reason: Agent came back online comes from a direct manual clear path.

CSV export

The CSV (this page) button downloads the current page of history as a CSV file. Each page holds up to 50 events. The page and date are in the filename. Use it to extract historical alarm data for reporting or auditing.

Rules tab

The Rules tab manages the alarm rule documents that drive monitoring.

Rules table

Column	Description
Rule name	The display name.
Code	The alarm code badge, for example AG-001.
Severity	Critical, Warning, or Info. In the rule JSON the value is CRITICAL, WARNING, or INFO.
Cooldown	The minimum time between re-fires.
Trigger	The trigger type: Service event, Metric threshold, DB query, Job failed, or Job duration.
Clear logic	A summary of the clear condition, for example "Auto-clears after 24h" or "Manual clear only".
Actions	The Enable/Disable toggle and the Delete button.

Expandable rule detail

Click a row to expand a three-panel detail view.

• Trigger shows the full trigger type and its parameters: metric name and threshold, query key and threshold, job type and multiplier, or "Service event — fired by internal logic".
• Clear Condition shows the clear type and parameters in plain language.
• Context Fields lists the expected context field names for that alarm code. These are the key and value pairs you see on Active alarm cards when the alarm fires, for example agentId, agentName, configName, jobId, error, and durationSeconds.

Examples of the Clear Condition text:

• timeout_minutes: 1440 is shown as "Auto-clears after 24h (1440m)".
• metric_below: heap_used_pct < 70 is shown as "Clears when heap_used_pct drops below 70".
• job_succeeded: CYBER_VAULT_COPY is shown as "Auto-clears when CYBER_VAULT_COPY succeeds".
• manual is shown as "Manual clear only".

Load Default Rules

Load Default Rules inserts the complete set of built-in alarm rules for all wired alarm codes. Rules whose codes are already present are silently skipped. It is safe to run at any time and never overwrites a customized rule. After loading, the toolbar shows how many rules were inserted and their codes.

You normally do not need it. At every console startup, any missing default rules, including codes added by a software upgrade, are added automatically in the same way. Existing rules, enabled or disabled, are never changed. After a software upgrade, the new alarm codes are therefore on the Rules tab without any action.

Use Load Default Rules to restore a default rule you deleted without restarting the console.

info

A default rule you delete comes back at the next console start. To silence an alarm, disable its rule instead.

Upload Rule

Paste or type a JSON alarm rule document and click Upload.

The system validates the document structure and alarm code, computes a SHA-256 checksum for tamper detection, and stores the canonical form. Invalid documents are rejected with a detailed error that shows which field is wrong.

Enable and Disable

The toggle button switches a rule between active evaluation (enabled) and dormant (disabled). When you disable a rule:

• The 60-second tick skips it entirely, so no new events fire.
• All currently ACTIVE events for that rule are cleared immediately and disappear from the Active tab at once.

Re-enabling the rule lets the monitor raise new events on the next tick if the condition is true.

Delete

Delete removes the rule document permanently. Historical alarm events are preserved and stay fully readable. The rule name and code are stored inside each event's trigger_context when it is raised, so deleting a rule never orphans history.

Rule document format

Rules are JSON documents.

Required fields

{
  "name": "Console heap high",
  "severity": "WARNING",
  "trigger": { "type": "metric_threshold", "metric": "heap_used_pct", "operator": "GT", "threshold": 85 },
  "clear_condition": { "type": "metric_below", "metric": "heap_used_pct", "threshold": 70 }
}

All fields

Field	Required	Description
name	Yes	The display name shown in the UI and stored in every event.
code	Recommended	A short identifier badge, for example SY-001. It must match a wired code to get context enrichment.
severity	Yes	INFO, WARNING, or CRITICAL.
cooldown_minutes	No	The minimum minutes between re-fires. The default is 60. Set it to 0 for edge-triggered alarms that should fire only once per incident.
trigger	Yes	What to evaluate. See the trigger types below.
clear_condition	No	When to clear automatically. See the clear condition types below. Omit it for manual-clear-only behavior.

Trigger types

metric_threshold

Fires when a JVM or system metric crosses a threshold.

"trigger": { "type": "metric_threshold", "metric": "heap_used_pct", "operator": "GT", "threshold": 85 }

• Available metrics: heap_used_mb, heap_used_pct, active_jobs_count, thread_count, gc_count.
• Operators: GT, GTE, LT, LTE, EQ.

db_query_threshold

Fires when a named database query result crosses a threshold.

"trigger": { "type": "db_query_threshold", "query": "vault_pending_windows_count", "operator": "GT", "threshold": 5 }

Available query keys: vault_pending_windows_count, failed_jobs_last_hour, active_alarm_count, offline_agent_count, avg_job_duration.

job_failed

Fires when the most recent completed job of a type has status failed. It fires only if the job failed within the last 2 × cooldown_minutes. This prevents stale re-fires from old failures.

"trigger": { "type": "job_failed", "job_type": "AUDIT_INGEST" }

job_duration_exceeded

Fires when a completed job took longer than threshold × rolling average of the last 20 completed runs for that type.

• It needs at least 3 completed jobs to establish a baseline.
• The threshold field is a multiplier. A value of 2.0 means 2× the average.

"trigger": { "type": "job_duration_exceeded", "job_type": "CYBER_VAULT_COPY", "threshold": 2.0 }

service_event

Raised directly by internal service logic and not evaluated by the tick. These fire in real time when specific failure conditions occur. No additional trigger fields are required. The tick always skips service_event rules.

"trigger": { "type": "service_event" }

Almost all built-in default rules use this type. AG-011, AG-012, AG-013, and SY-002 are metric thresholds. The Rules table shows the others as Service event in the Trigger column.

Clear condition types

Type	Behavior
manual	Never clears automatically. The operator must use the Clear button.
timeout_minutes	Clears when triggeredAt + timeout_minutes has passed.
metric_below	Clears when the named metric drops below threshold. It is re-checked every 60 seconds.
job_succeeded	Clears when the most recent job of job_type has status completed.

For metric_below, the metrics are the trigger metrics above plus one more. The default rule of IS-002 uses ingest_lag_seconds, the largest audit ingest lag of all clusters.

• It applies only while that lag is known. It never clears while the lag cannot be read.
• For IS-002, the console applies 2400 s (40 minutes) when the stored threshold is higher. The rule still shows 3600. This agrees with IS-002's own 40-minute clear level, so you never need to edit it.

Some alarm codes have custom clear handlers that take priority over the rule's clear_condition. For example, AG-001 (agent offline) always clears the moment the agent reconnects, whatever clear_condition the rule specifies. For these codes, the clear_condition is a fallback, for example a 24-hour timeout in case the agent never reconnects.

Built-in alarm codes

The following codes are wired to real conditions in the system. The pre-configured rule for each one is added automatically at console startup, or with Load Default Rules.

warning

Known issue: the default rules AG-011, AG-012, and AG-013 are not active in this release, so those codes never raise an alarm.

Agent (copy agent): AG-*

Code	Name	Severity	Clear
AG-001	Agent offline	WARNING	Custom handler when the agent reconnects; 24h timeout fallback
AG-002	Agent ClickHouse write failure (spool depth > 0)	WARNING	60m timeout
AG-004	Agent live stats push failed	WARNING	10m timeout
AG-005	Agent copy log spool accumulating	WARNING	60m timeout
AG-006	Agent spool approaching disk limit (>80% of 50k cap)	WARNING	60m timeout
AG-007	Agent copy throughput degraded	WARNING	When CYBER_VAULT_COPY succeeds
AG-008	Agent JVM heap pressure (>85%)	WARNING	30m timeout
AG-009	Agent delta queue stale (>7d undelivered, purged)	WARNING	4h timeout
AG-010	Agent delta queue write failed	CRITICAL	60m timeout
AG-011	Agent circuit breaker open >5 min	WARNING	Breaker closes (metric_below)
AG-012	Agent ephemeral port exhaustion (fewer than 1000 available)	CRITICAL	Metric recovers
AG-013	Agent connectivity degraded (≥2 breakers HALF_OPEN)	WARNING	Metric recovers

Audit ingest: AU-*

Code	Name	Severity	Clear
AU-001	ECS audit ingest failed (ClickHouse unreachable)	CRITICAL	2h timeout
AU-003	ClickHouse insert failed during audit ingest	WARNING	60m timeout
AU-004	ECS SSH audit ingest degraded	WARNING	2h timeout

AU-004 is raised when any of these happens:

• Some nodes failed to connect.
• Some nodes were skipped for lack of an SSH session permit.
• The run hit I/O errors, such as failed ClickHouse inserts.

ClickHouse: CH-*

Code	Name	Severity	Clear
CH-001	ClickHouse write failure	CRITICAL	60m timeout
CH-003	ClickHouse memory not released	WARNING	Custom handler, or manual
CH-004	ClickHouse at its memory limit	CRITICAL	Custom handler, or manual

CH-003 and CH-004 details

Alarm	Raised when	Clears when
CH-003	The memory ClickHouse holds but does not use (resident minus in use) stays at or above max(30% of its memory limit, 4 GiB) for 10 minutes.	It stays 5 minutes below max(20% of the limit, 2 GiB), for example after a ClickHouse restart. Or clear it manually.
CH-004	The larger of two readings stays at or above 95% of the limit for 3 minutes. The readings are the memory counted against ClickHouse's limit (its resident or cgroup reading) and the memory it tracks as in use by queries.	It stays 3 minutes below 85% of the same larger value. Or clear it manually.

For CH-004:

• When the tracked memory is the larger reading, the summary says "(memory tracked as in use by queries)".
• When ClickHouse refuses even the console's memory check with "(total) memory limit exceeded", the larger reading in that error ("current RSS" or "would use") counts.
• At the limit, ClickHouse refuses work server-wide, so audit ingest and vault syncs stop.

Both alarms come from the console's ClickHouse memory monitor:

• It samples ClickHouse memory every minute. The first sample is 2 minutes after the console starts.
• A sample that ClickHouse does not answer completely within 13 seconds is canceled and logged at WARN. Sampling continues.
• Between the alarm and clear levels, neither alarm is raised or cleared, so a value hovering near a level does not flap.
• With no recent sample, for example when ClickHouse is unreachable, an active alarm stays active.
• Both alarms clear if the monitor is switched off or ClickHouse is no longer configured.

The alarm detail shows a summary, the memory figures, whether ClickHouse is refusing queries, the likely cause (an estimate), and What to do. What to do points to Help > ClickHouse Health > Memory not released. Set the levels with APP_CH_MEMORY_MONITOR_* environment variables. See the ClickHouse Health help.

Cyber Vault: CV-*

Code	Name	Severity	Clear
CV-001	Source auth failed (full copy)	CRITICAL	Manual
CV-002	Target auth failed (full copy)	CRITICAL	Manual
CV-003	Prefix scan fallback	WARNING	Job succeeds
CV-004	Object copy failed (full copy)	WARNING	Job succeeds
CV-006	Copy throughput degraded	WARNING	Job succeeds
CV-007	Page checkpoint timeout	WARNING	Job succeeds
CV-009	Full copy failed (objects failed)	WARNING	Manual
CV-100	Sync window not picked up by agent	WARNING	Multi-incremental job succeeds
CV-101	Sync window failed (zombie healer)	WARNING	4h timeout
CV-102	Sync window skipped (max retries)	WARNING	4h timeout
CV-103	Target bucket not configured (incremental)	WARNING	Manual
CV-104	Source auth failed (incremental)	CRITICAL	Manual
CV-105	Target auth failed (incremental)	CRITICAL	Manual
CV-106	Cyber Vault ClickHouse changelist query failed	CRITICAL	Manual
CV-108	Object copy failed (incremental)	WARNING	Incremental job succeeds
CV-109	Object delete failed (incremental)	WARNING	Incremental job succeeds
CV-110	MD5/ETag mismatch on single-part copy	WARNING	Manual
CV-111	Window sync partial (some failures)	WARNING	Manual
CV-112	Window sync failed completely	WARNING	Manual
CV-113	Batch incremental completed with window failures	WARNING	Manual
CV-114	Window abandoned (vault agent exhausted 3 retries)	WARNING	Manual
CV-115	Ransomware event detected	CRITICAL	Auto-clears on a clean Eyeglass poll; or manual
CV-116	Cyber Vault sync exceeded time limit	WARNING	Auto-clears when all listed jobs have finished; or manual
CV-117	Cyber Vault scheduled sync did not run	WARNING	Auto-clears at the group's next scheduled start; or manual
CV-118	Cyber Vault sync waiting for ingest	WARNING	Custom handler; or manual
CV-119	Cyber Vault window analysis is behind	WARNING	Custom handler; or manual
CV-300	Heap pressure during copy (>80%)	WARNING	Job succeeds
CV-TPS-001	TPS limit utilization high	WARNING	10m timeout, plus custom handler when below 85%

CV-106: changelist query failed

An incremental sync could not read a window's change list from ClickHouse. The window is not sealed and is synced again later.

CV-115: ransomware event detected

Setup & Settings > Storage > Eyeglass Device reported an active ransomware event. It also raises when Eyeglass could not be reached for more polls than the Missed polls before treated as detection setting allows.

It is raised in two cases:

• An incremental sync is blocked before it starts. The sync never runs, and Copy Jobs history shows it as Not Started with Failure reason Ransomware Event.
• The background check finds a detection while syncs are running. Queued incremental jobs are then canceled, and running ones too if Kill in-flight jobs on detection is on.

Full copies are not affected.

One alarm stands for one detection event, not one per job. To see every affected job, filter Copy Jobs history by Failure reason Ransomware Event.

CV-115 clears automatically when Eyeglass's next poll is clean, meaning it is reachable and has no active event. You can also clear it manually. Deleting the device or turning monitoring off does not clear it.

CV-116: sync exceeded time limit

Jobs are still running after the Sync Time Limit (Vault Data Sync Dashboard > Settings). The vault agent released the vault and re-adopts the jobs at its next heartbeat.

The alarm clears when all listed jobs have finished (completed, failed, or canceled), or manually.

CV-117: scheduled sync did not run

A run day's schedule window ended without its scheduled sync having started. The run calendar is the First Sync date plus k × Repeat Every.

Usual causes:

• The vault agent, the vault, or the console was down.
• Another sync of the same group was still running for the whole window. This can be a retry, a sync re-adopted after an agent restart, or the previous run day's long sync. A group runs one sync at a time. In this case the agent was up, and that run day's changes go with the next run.

A group that was just moved to another schedule policy whose window has already passed today can also raise it once. This is a known issue.

How it behaves:

• The vault agent reports it when it can reach the console again.
• There is no catch-up. The next run is the next calendar day.
• Each missed window raises CV-117 once. A report of a window that already had one, even a cleared one, is not raised again.
• No alarm is raised when a scheduled start of the group did reach the console in that window, for example when Eyeglass pre-flight blocked it (CV-115).
• No alarm is raised when the policy's schedule was changed after the run day began, or before the run calendar was introduced.
• Only one CV-117 is active at a time. While one group's CV-117 is active, a missed run of another group raises no second alarm. The vault agent reports it again every 30 minutes, and it gets its own CV-117 once the first one clears. The SYNC_MISSED message in the QTP audit log is recorded every time.

The alarm clears automatically at the group's next scheduled start, or manually.

CV-118: sync waiting for ingest

One or more sources that an enabled vault policy reads from have an ingested-through time more than 2 hours old. Their change windows wait for the audit ingest instead of being synced with objects missing.

The sources can be:

• The policy's source device.
• A federation peer.
• Another source device with exactly one of their names, whose events a sync also copies.

It is not raised while the ingest gate switch (waitForIngest) is off. See the Incremental Sync help, When a window waits for ingest.

The alarm clears when the listed sources have caught up. When what it reports changes, it is cleared and raised again with the current content within 5 minutes. You can also clear it manually.

CV-119: window analysis is behind

An enabled vault policy has no analyzed window in its last 4 sync intervals. A policy that was never analyzed counts once it is that old.

The console checks 20 minutes after it starts, then every 15 minutes.

Windows are still synced. Only the dashboard counts are incomplete, and windows that the analysis has not proven empty are synced instead of skipped.

The cause says which of these applies:

• The Incremental Sync Report Analysis schedule is disabled.
• The analysis is waiting for the audit ingest. Every policy behind is waiting for its source device's ingest to cover its next windows.
• The analysis is not keeping up.

The alarm clears when every policy is current again. When the cause changes, it is cleared and raised again with the current content at the next check. You can also clear it manually.

Incremental sync system: IS-*

Code	Name	Severity	Clear
IS-001	Window stuck syncing	CRITICAL	30m timeout
IS-002	Audit ingest RPO breach	WARNING	Custom handler; or manual. See RPO alarms.
IS-003	Zombie window healed (informational)	INFO	5m timeout
IS-004	Audit ingest RPO status unknown	WARNING	Custom handler; or manual. See RPO alarms.

RPO alarms (IS-002 and IS-004)

• IS-002 is raised when the newest audit event of one or more clusters is more than 1 hour old. One alarm lists the clusters that were breaching when it was raised, worst first.
• IS-004 is raised when the audit ingest lag could not be read for 15 minutes, with at least 2 failed reads in a row. RPO breaches (IS-002) cannot be detected in that state.

The console checks the audit ingest RPO every minute, whether or not Incremental Sync is enabled. The lag of a cluster is the time since its newest audit event, per ObjectScale cluster.

The check reads the console's cached audit statistics. These are the same values as the Audit Ingest Lag card in System Monitor, refreshed about once a minute. The check never runs a ClickHouse query of its own.

The result is one of these states:

State	Meaning
Known	The lag was read in the last 15 minutes. When reads are slow, it was read within one refresh cycle of the statistics: the read, the pause before the next one, and that next read, at most about 18 minutes. A slow but working read therefore never makes the state drop to "not known yet". If any cluster is more than 1 hour behind, IS-002 is raised or kept.
Not known yet	The first read after a console start is still running, or only one read in a row failed. The same applies after an audit data wipe, a device purge, a retention change, or a switch to another ClickHouse database. Nothing is raised or cleared. On a large audit table this usually takes under a minute.
Unknown	No successful read for 15 minutes and at least 2 failed reads in a row. IS-004 is raised. While the lag is unknown, IS-002 is neither raised nor cleared, and an existing IS-002 stays as it is.
Not configured	Setup & Settings > Database has no ClickHouse host. There is no lag to check. Nothing is raised, and an active IS-002 or IS-004 is cleared.

Large audit tables without a usable statistics index

If the statistics index is being built, covers too little of the table, or is not usable, the lag is not read from the whole table every minute. See System Monitor help, Where the numbers come from.

• Every cluster's last event is remembered. The console keeps each cluster's newest audit event from every read. It saves it in ClickHouse in the table audit_lag_baseline, at most every 10 minutes and at once when a cluster appears or goes. After a console start, a retention change, or an audit data wipe, the first read is already the short read described below. An audit data wipe removes only the wiped source's clusters. The state is Known within about a minute. A cluster that has sent nothing for days is in the check from the start with its full lag.
• A full read happens first only when nothing is remembered. That is the first start on a ClickHouse database, the first start after an upgrade, a device purge, or when the saved list cannot be read. After a device purge, the clusters cannot be told apart, so what was remembered is dropped. The full read counts every cluster, which can take minutes on a large table. ClickHouse stops it after 360 seconds, and a read that ran into that limit is tried again 10 minutes later. The state stays Not known yet until it succeeds. If it fails twice and 15 minutes pass, IS-004 is raised.
• Otherwise, the short read runs. The lag is read every minute or two for the last 6 hours only. On a busy installation the window is shorter, but at least 2 hours. It reads at most 100 million audit records and merges the result with the clusters read before. A cluster without events in that window keeps its last event, and its lag keeps growing, so it keeps breaching. A short read that fails counts toward IS-004 like any other. This includes one that exceeds the 100 million row cap (ClickHouse error 158, which means more than about 50 million audit records an hour).
• The full read repeats in the background at most every 6 hours. Its failures never make the state not known yet or unknown and never raise IS-004. They only leave System Monitor's totals stale. A cluster whose audit data has all expired under the retention leaves the check at the next full read, at most 6 hours later, or later while that read fails.

IS-002 detail fields

Field	Description
Cluster, Ingest lag, Last event	The values for the worst cluster.
RPO threshold	1 hour.
Breaching clusters	Every cluster over the threshold when the alarm was raised, with its lag at that time, worst first. For example: ecs-vdc2 (174.0 h), ecs-vdc1 (98.0 h). At most 20 are listed, then "… and N more".
As of	When that lag was read.

The details of an alarm cannot change once it is raised, so the list is not updated while the alarm is active. To see current values:

• The Audit Ingest Lag card in System Monitor shows the current lag of every cluster.
• The console log has a WARN for each cluster when it goes above 1 hour. It repeats hourly while the cluster stays above 40 minutes. It has an INFO line when the cluster is back at or below 40 minutes.

Log lines:

SyncScheduler: IS-002 RPO breach — cluster='…' lagSeconds=… (…) threshold=3600 …
SyncScheduler: IS-002 RPO breach continues — cluster='…' … threshold=3600 (over at or below 2400) …
SyncScheduler: RPO breach over — cluster='…' lagSeconds=… (at or below 2400; threshold 3600)

How IS-002 raises and clears

Raise. IS-002 is raised when any cluster is more than 1 hour (3600 s) behind.

Stays active. The alarm stays active in these cases:

• While any cluster is more than 1 hour behind, also a cluster it does not list. When another cluster falls behind, or a listed one recovers while another still breaches, the alarm stays as it is. It is neither cleared nor replaced, so one alarm covers the whole breach. A cluster that starts breaching later is not named in this alarm. See the detail fields above for where to find it.
• While a cluster that went above 1 hour is not yet back at or below 40 minutes (2400 s). This gives a 20-minute margin per cluster. It applies to every cluster that went above 1 hour, also one that did so after the alarm was raised.
• A cluster that never went above 1 hour never keeps the alarm open, whatever its lag. For example, a cluster steady at 55 minutes does not keep alive an alarm raised for another cluster that has recovered. If that cluster later goes above 1 hour itself, it counts like any breach: it keeps an active IS-002 open or raises a new one.

When the console keeps IS-002 active only because of the margin, the log has one INFO line that names the clusters still above 40 minutes:

Is002RpoBreachClearHandler: no cluster exceeds the 3600 s RPO any more, but IS-002 event=… is kept until every cluster that breached it is back at or below 2400 s (1200 s clear margin) — still above: vdc-a (3300 s)

Why 20 minutes. The lag shown is the lag at the last read plus the age of that read. With slow reads on a large installation, it climbs by up to about 18 minutes between reads even when the real lag is steady. A narrower margin let the normal refresh clear and re-raise the alarm, with an email each time when the rule's Email is on. With 20 minutes, a lag hovering around 1 hour keeps one alarm open, and clearing at 40 minutes is still well within the 1-hour RPO.

Clear. IS-002 clears by itself (handler:IS-002 in the clear history) when both of these are true:

• No cluster is more than 1 hour behind.
• Every cluster that went above 1 hour is back at or below 40 minutes, or none has audit data any more.

Log line:

Is002RpoBreachClearHandler: no cluster exceeds the 3600 s RPO and every cluster that breached it is back at or below 2400 s (or none has audit data) — clearing IS-002 event=…

If a cluster falls behind again afterward, the next check, within a minute, raises a new IS-002 with the clusters breaching then. It is raised at once. The rule's 30-minute cooldown does not delay a new alarm once the previous one is cleared, whether it was cleared automatically or by hand.

Repeats are limited by the condition itself. Going from 40 minutes back above 1 hour takes at least 20 minutes without ingest progress. A still-active alarm is never duplicated, and a suppressed one stays suppressed for its 24 hours.

IS-002 also clears by itself when ClickHouse is no longer configured (Setup & Settings > Database without a ClickHouse host), like IS-004:

Is002RpoBreachClearHandler: ClickHouse is not configured any more — clearing IS-002 event=…

If ClickHouse is configured again and a cluster is still behind, a new IS-002 is raised.

Other behavior:

• After a console restart, the record of which clusters went above 1 hour starts empty. An active IS-002 whose clusters are all at or below 1 hour then clears once the lag has been read. It is raised again only if a cluster goes above 1 hour.
• An old IS-002 whose rule was deleted or reloaded, and which shows no rule, is cleared as soon as the lag is known. If a cluster still breaches, a current one is raised at the next check, within a minute.
• Every cluster that has audit data is checked. A decommissioned device keeps IS-002 active until its audit data is purged or ages out.
• The IS-002 rule shows the clear condition metric_below ingest_lag_seconds 3600 and cooldown 30. The console applies 2400 s when it evaluates that condition, so there is nothing to edit.

IS-004

IS-004 clears as soon as the lag is read again, or when ClickHouse is no longer configured. You can also clear it manually.

It is raised again at once when the lag becomes unknown again after a clear, which takes another 15 minutes without a successful read. The 30-minute cooldown does not delay it. If you clear it by hand while the lag is still unknown, it is raised again within a minute.

Every other alarm code keeps its cooldown.

IS-004 detail fields:

Field	Description
Reason	Why the status is unknown.
Last successful check	The time of the last successful read, or "none since console start …".
Failed checks	The number of failed reads.
Error	The last read error.
Statistics index built	How much of the audit statistics index (the proj_audit_stats_v1 projection) is built, in %.

The Audit Ingest Lag card in System Monitor shows the same state.

Job duration: JD-*

Code	Name	Severity	Clear
JD-001	Job duration exceeded threshold	WARNING	Job succeeds

Licensing: LI-*

Code	Name	Severity	Clear
LI-001	License expiring	WARNING	Auto, when no license is about to expire
LI-002	License expired	CRITICAL	Auto, when no license is expired
LI-003	Licensed capacity exceeded	WARNING	Auto, when the use is within the license
LI-004	No support license installed	WARNING	Auto, when a support license is installed

• LI-001 is raised by the license manager 2 months before a license expires.
• LI-003 is raised when the license manager finds more usable TB or nodes than the installed license covers. Nothing is disabled.

PostgreSQL: PG-*

Code	Name	Severity	Clear
PG-001	PostgreSQL connection failure	CRITICAL	15m timeout
PG-002	PostgreSQL analytics query failed	WARNING	15m timeout

Console and system: SY-*

Code	Name	Severity	Clear
SY-001	Console JVM near-OOM (98% via MemoryPoolMXBean)	CRITICAL	Manual
SY-002	Console JVM heap pressure (>85%)	WARNING	heap_used_pct drops below 75
SY-003	Agent version drift	WARNING	Manual
SY-004	Console identity key unavailable	WARNING	Auto-clears when the key is available
SY-005	Software delivery key unavailable	WARNING	Auto-clears when the key is available
SY-006	Console identity key regenerated (informational)	INFO	24h timeout

• SY-004: vault agents cannot send encrypted log uploads or device health reports.
• SY-005: the System Public Key cannot be shown, and QCE packages cannot be installed.
• SY-006: the old key file is kept as .bak-<timestamp>.

The Vault Operations banner and the Setup & Settings > Security panel show the same key problems as SY-004 and SY-005, independently of the alarms.

Vault agent: VA-*

Code	Name	Severity	Clear
VA-001	Vault agent offline (hours threshold)	CRITICAL	service_event when the agent reconnects
VA-002	Vault agent upgrade failed	WARNING	Manual
VA-003	Vault agent upgrade stalled (vault network may be open)	CRITICAL	Manual
VA-004	Vault agent version mismatch	WARNING	service_event when the version aligns
VA-005	Airgap failed to open (no interface came up)	WARNING	The next open succeeds
VA-006	Airgap opened degraded (some interfaces did not come up)	WARNING	An open brings every interface up
VA-007	Airgap close not confirmed (vault link may be open)	CRITICAL	The close is confirmed
VA-008	Interface up while the vault should be closed	CRITICAL	Every interface reads shut again
VA-009	Air-gap switch health problem	WARNING	A later health report for that switch is healthy
VA-010	Vault device credentials must be re-entered	MAJOR	Manual

VA-010 is raised when a vault agent registers a new identity key (reinstall, host restore, or VM clone) and the switch or ObjectScale management API device credentials had to be cleared. Re-enter the credentials on the vault device, then clear the alarm. One alarm is raised per agent, and email is on by default.

VAULT_NETWORK_OPEN and VAULT_NETWORK_CLOSE have no default rule, so they never fire. If you upload a rule for either code, set the name, severity, and clear condition yourself. Without them, the name is the code, the severity is WARNING, and the alarm clears only manually.

Email notifications

The Rules table also has Email and Real-time columns.

Toggle	Description
Email	When enabled, email notifications fire whenever the rule raises an event.
Real-time	When enabled, the email bypasses the batch window and is sent immediately. All other email-enabled codes batch into the configured digest window.

Defaults:

• Email is off for most codes. It is on for AG-001, LI-001, LI-002, CV-001, CV-002, CV-104, CV-105, CV-106, PG-001, SY-001, AU-001, IS-002, CH-003, and CH-004.
• Real-time is on for AG-001, LI-002, PG-001, SY-001, and CH-004. These issues need immediate attention. For example, at its memory limit ClickHouse refuses audit ingest and vault sync work.

Defaults are set when the default rule is first added, at console startup or by Load Default Rules. A rule that already exists keeps its own setting.

Both toggles update immediately in the UI, and the row stays in place while the request is in flight. A filter pill bar above the table (All, Email Off, Email On) lets you focus on rules where email is not yet configured.

Troubleshooting

For common alarm questions and the IS-002, IS-004, CV-118, CV-119, CH-003, and CH-004 alarms, see the Alarms section of the Troubleshooting page.