Diagnostics

This tab provides Logging settings, on-demand state snapshots, agent fleet snapshots, and JVM profile management for remote copy agents.

Overview

The Diagnostics tab is the first place to look when you troubleshoot ingest failures, performance issues, or unexpected agent behavior.

• JVM Profiles let you tune the heap, garbage collector, and thread settings for copy agents without redeploying.
• The snapshot logs (System and Agent Dashboard) are JSONL files written to disk on a schedule. They are bundled into backups for offline review.

Logging

The Logging card controls how the console writes its own log files. The settings apply to the running console JVM. Where applicable, they are pushed to copy agents on their next heartbeat.

Setting	Description	Range and default	Applies
Log Level	DEBUG, INFO, or WARN. It is propagated to all agents on their next heartbeat. INFO is the production default. DEBUG can produce several GB per day and noticeably slows the JVM.	Default INFO	Immediately on save, no restart
Max File Size (MB)	The size at which the active log file is rolled to a .gz archive.	10–500, default 50	On restart
Retention (days)	Rolled .gz archives older than this are deleted at startup. Active log files are never deleted by this setting.	1–365, default 30	On restart
Total Size Cap (MB)	A hard ceiling on the combined size of all rolled archives.	100–10,000, default 500	On restart

System Snapshot Log

This log writes a JSONL entry to data/logs/application.log at a configurable interval. Each entry captures the full system state.

Controls

Control	Description
Trigger Snapshot	Writes one entry to application.log immediately, regardless of the configured interval. Use it before you report a support issue to capture a consistent point-in-time picture.
Snapshot interval	How often the scheduler writes a snapshot. The range is 10–3600 seconds and the default is 30 seconds. Changes take effect on the next scheduler tick, with no restart.

What each entry captures

Each snapshot entry is a JSON object with these fields:

Field	Content
ts / iso	Epoch milliseconds and ISO-8601 timestamp of the snapshot.
jvm	Heap used and max (MB and %), JVM CPU %, host CPU %, thread count, GC counts, and metaspace.
apiCalls	Per-endpoint call counts and average latency since the last restart.
clickhouseHealth	Connectivity check result and round-trip latency (ms).
clickhouseBreakers	The console's ClickHouse circuit breakers. See Circuit breaker field.
dbStats	ClickHouse compressed and uncompressed size, and row count.
ingestLag	Per-cluster lag between the latest ingested event and now. See Ingest lag field.
eventsByStorageNode	Total events per storage cluster, from the last full read of the audit statistics. See Ingest lag field.
ingestStats	State of the cached audit statistics. See Ingest statistics field.
endpointHealth	Cached endpoint reachability results (System Monitor > Endpoint Reachability).
ingestThroughput	The last completed ingest jobs with their rates and counts.
diskStats / diskStatsOs	Disk read and write latency from the OS-level probe, and the platform it ran on.
clickhouseAnalytics	ClickHouse version, uptime, connections, queries per second, inserts, mutations, merge pressure, and memory. See ClickHouse memory fields.
dbHealth	PostgreSQL ping latency, active connections, pool utilization %, and database size.

A section that could not be read is written as {"error": "<message>"}. The exception's name is used when it has no message. The rest of the line is still written.

Circuit breaker field

clickhouseBreakers has one circuit breaker per subsystem. See the ClickHouse Health help. It lists only lanes that opened, have failures, or are not closed, in this form:

{"<lane>": {state, retryInSec, failures, opens, fastFails, last, lastClass}}

The value is {} when every lane is quiet. Counters run since the console started or since the ClickHouse settings changed.

Ingest lag field

ingestLag comes from the console's cached audit statistics. The console reads them from ClickHouse at most once a minute for all users. The snapshot never waits for ClickHouse. The field is empty until the first read after start-up.

Each cluster entry has these values:

• as_of: when the statistics were read.
• stale: whether the value is stale.
• beyond_window: true when the lag was read for the last hours only and that cluster had no event in them. Its time then comes from an earlier read and is an upper bound. See the System Monitor help.

eventsByStorageNode comes from the last full read of the same statistics. It is the same read as ingestLag while the statistics index is usable. Otherwise it can be up to 6 hours old. Its time is ingestStats.totals.as_of.

Ingest statistics field

ingestStats describes the state of the cached audit statistics.

Key	Meaning
status	ok, stale, loading, unavailable, or not_configured.
as_of, age_seconds	When the statistics were read, and their age.
refresh_seconds	The current spacing between reads.
duration_ms	How long the last read took.
consecutive_failures, error, last_success	Failure count, last error, and time of the last success.
rpo_state	What the IS-002 and IS-004 check sees: KNOWN, PENDING, UNKNOWN, or NOT_CONFIGURED.
clusters, max_lag_seconds	Cluster count and the largest lag.
detail, buckets	Statistics behind the filter lists and the Cyber Vault lists: status, as_of, failures, and error.
raw_distinct_cached	Cached distinct-value data.
projections	The statistics index build. See the next table.
lag_window_hours	The hours of the last lag read when only the last hours were read. null for a full read.
totals	status, as_of, consecutive_failures, and error of the full read the totals come from.

A value counts as stale when any of these is true:

• It is older than 15 minutes.
• When reads are slow, it is older than one refresh cycle (at most about 18 minutes).
• The last read failed.

The projections object holds:

• mode, taken from CV_AUDIT_STATS_PROJECTIONS.
• status.
• Coverage %, partitions missing, pending builds, and the last error.
• stats_unusable_reason and buckets_unusable_reason. These say why ClickHouse cannot use an index although it may be built. They are null when it can.

projections.status has these values:

Status	Meaning
materializing	The console is building the index.
complete	The index is built.
add-only / off	The console does not build the index.
paused	The build was stopped with KILL MUTATION, or new parts without the index kept appearing. It stays paused until the console restarts.
retrying	ClickHouse is unreachable or overloaded, or the audit table is not created yet. It is retried about every 2 minutes. Any other unexpected ClickHouse error, for example a column that an upgrade has not added yet, is retried after 2, 4, 8, 16, and 32 minutes, then every hour.
failed	ClickHouse rejected the build statement itself, for example because of a syntax error, an unsupported feature, a read-only user, or denied access. It stays failed until the console restarts.
not-started	The build has not started.
not-configured	The build is not configured.

ClickHouse memory fields

clickhouseAnalytics carries the memory history that a support bundle includes. See the ClickHouse Health help.

Field	Meaning
memResidentMb, memTrackingMb, memVirtualMb	Resident, tracked, and virtual memory.
memLimitMb	The server memory limit.
memGapMb, memGapPctOfLimit	Resident minus tracked: memory that is held but not in use.
memCountedMb	The value that ClickHouse compares with its limit.
jemAllocatedMb, jemActiveMb, jemResidentMb, jemRetainedMb, jemMetadataMb, jemDirtyEstMb	Allocator breakdown.
allocatorPurges, memoryWorkerRuns	Allocator and memory worker counters.
memLikelyCause	The likely cause of memory pressure.
memError	Present when memory could not be read.

Agent Dashboard Snapshot Log

This log writes a JSONL entry to data/logs/agentdashboard.jsonl at a configurable interval. Each entry captures the full state of the registered agent fleet. It is the agent equivalent of the System Snapshot Log.

Controls

Control	Description
Trigger Snapshot	Writes one entry to agentdashboard.jsonl immediately, regardless of the configured interval.
Snapshot interval	How often the scheduler writes a snapshot. The range is 10–3600 seconds and the default is 60 seconds.

What each entry captures

Each entry is a JSON object with a fleet summary header and a per-agent array.

Fleet summary fields

Field	Content
ts / iso	Epoch milliseconds and ISO-8601 timestamp.
agentCount	Total registered agents.
onlineCount	Agents with status ONLINE.
offlineCount	Agents with status OFFLINE or DEGRADED.
totalActiveCopyJobs	Sum of active copy jobs across all online agents.

Per-agent fields

These fields are in the agents[] array.

Field	Content
agentId	Agent UUID.
name	Display name.
type	COPY_ENGINE or VAULT_AGENT.
status	ONLINE, OFFLINE, or DEGRADED.
version	Agent software version string.
hostname / ip	Host reported by the agent.
lastSeenMs	Epoch ms of the last heartbeat received.
chConnected	Whether the agent can reach ClickHouse directly.
draining	Whether the agent is in maintenance drain mode (no new jobs assigned).
siteId	Optional site or location label.
upgradeStatus	IDLE, PENDING, UPGRADING, DONE, or FAILED.
activeCopyJobs	Number of copy jobs currently running on this agent.
maxConcurrentJobs	Maximum concurrent jobs the agent advertises.
bufferDepth	Peak in-memory copy-log buffer rows since the last heartbeat, before the ClickHouse flush.
spoolDepth	Number of copy-log rows waiting in the agent's SQLite spool, the ClickHouse outage recovery queue.
spoolOldestSec	Age in seconds of the oldest spool entry. 0 if the spool is empty.
chFlushLastSuccessMs	Epoch ms of the last successful ClickHouse flush.
activeJobIds	Comma-separated list of active job IDs on this agent.
hostCpuPct	Host system CPU utilization %.
jvmCpuPct	JVM process CPU utilization %.
heapUsedPct	JVM heap used %.
heapUsedMb / heapMaxMb	JVM heap used and maximum, in MB.
threadCount	JVM thread count.

info

All per-agent metrics come from the most recent heartbeat. If an agent is OFFLINE, its last-known values are included.

Use cases

• Capacity trending. Track how activeCopyJobs and bufferDepth change across the fleet as job load increases.
• ClickHouse connectivity issues. chConnected and chFlushLastSuccessMs show which agents lost their ClickHouse path and when.
• Spool monitoring. A rising spoolDepth means ClickHouse is unreachable from an agent. spoolOldestSec shows how stale the backlog is.
• Upgrade auditing. upgradeStatus and version track the rollout of new agent builds across the fleet.

Job Monitoring on the Database tab

The Sample Interval and Retention Period controls for the cv_job_metrics time series are on Setup & Settings > Database > Database Settings > Job Metrics Sampling. This data powers the View Long-term Trends link on the Jobs flyout.

All ClickHouse retention controls are on that one tab. The JSON paths are cvMonitoring.sampleIntervalSeconds and cvMonitoring.retentionMonths. The defaults are 60 seconds and 6 months.

See the Database help page for full documentation.

JVM Profiles

JVM Profiles are named sets of JVM arguments for remote copy agents. The console sends each agent its assigned profile on every heartbeat.

When the assigned profile differs from the one the agent runs, the agent writes the new arguments to its jvm.args file and restarts itself with them. It restarts only when no copy job is running. See Assign to Agent below.

Built-in profiles

Six profiles are provided. All of them include:

• ZGC and AlwaysPreTouch.
• An OOM heap dump.
• --add-opens java.base/java.net=ALL-UNNAMED, which lets the agent read its JVM DNS cache for the Fleet DNS Cache Monitor.
• -Dsun.net.inetaddr.ttl=0.

Profile	Initial heap	Max heap	Intended host RAM
default	6 GB	6 GB	8 GB host
host-16g	12 GB	12 GB	16 GB host
host-24g	18 GB	18 GB	24 GB host
host-32g	24 GB	24 GB	32 GB host
host-48g	36 GB	36 GB	48 GB host
host-64g	48 GB	48 GB	64 GB host

With -Dsun.net.inetaddr.ttl=0, the agent keeps no DNS answers. Every new connection resolves the name again, and connections spread across the nodes behind a round-robin DNS name. Because nothing is cached, the Fleet DNS Cache Monitor has no entries for these agents.

When a lookup fails, for example because the DNS server is down, the copy waits and retries for up to 15 minutes instead of failing the object.

The default profile is assigned automatically to every newly deployed agent. When an agent has no explicit profile, the heartbeat delivers profileId = "default". Once you know the agent's host RAM, assign a larger host profile from Cyber Vault > Agent Fleet Management > Live Agents.

Custom profiles

Control	Description
Profile Name	A unique label for the profile.
JVM Arguments	The raw JVM argument string applied to the agent process, for example heap size flags, GC algorithm selection, or thread stack size overrides. Arguments are passed to the JVM launcher unchanged.
Add Profile	Adds a named profile. Enter a name and the argument string, then save.
Edit Profile	Changes the argument string of an existing custom profile.
Delete Profile	Removes a custom profile. An agent that is assigned the deleted profile falls back to the built-in default at its next restart.
Assign to Agent	Done from Cyber Vault > Agent Fleet Management > Live Agents, not from this tab. See below.

Assign a profile to an agent

1. Open Cyber Vault > Agent Fleet Management > Live Agents and find the agent. It must be online.
2. Select the profile from its Profile dropdown.

The assignment is saved as soon as you select it. The agent shows "pending restart" until it applies the profile.

The agent applies a new profile by restarting itself. It does this only when no copy job is running, because a restart would fail running jobs.

While jobs run, the change waits. The agent logs it once as "deferred" and retries on every heartbeat.

To apply the profile on a busy agent, Drain the agent on Cyber Vault > Agent Fleet Management > Live Agents. It then gets no new jobs, except jobs pinned to it. It restarts once its running jobs finish. No new job starts while the agent restarts.

Workflows

View recent backend errors

1. In the Application Logs card on this tab, click Backend Log. The last 200 lines of backend.log open inline.
2. Frontend Log, Application Log, and Agent Dashboard Log work the same way.
3. Click Clear to close the viewer.

For older entries, use one of these options:

• Run a backup with Setup & Settings > Backup & Restore > Create Backup Now. The archive includes backend.log plus 7 days of rolled archives. Download it from Stored Backups and open the file in a text editor.

Capture a system state snapshot

1. Click Trigger Snapshot in the System Snapshot Log card.
2. Wait for the confirmation message.
3. Read the entry, which is appended to data/logs/application.log on the console. You can read the last 200 lines in Application Logs > Application Log on this tab, or use a backup or the shell.

Capture an agent fleet snapshot

1. Click Trigger Snapshot in the Agent Dashboard Snapshot Log card.
2. Wait for the confirmation message.

The new entry is a single JSONL line appended to data/logs/agentdashboard.jsonl. It shows the state of all registered agents at that moment.

Assign the right profile for a host

1. Open Cyber Vault > Agent Fleet Management > Live Agents and find the agent.
2. In its Profile dropdown, select the profile that matches the host RAM, for example host-32g for a 32 GB machine.

The assignment is saved immediately. The agent restarts itself with the new profile once no copy job is running. To apply it on a busy agent, drain the agent and let its jobs finish.

Create a custom JVM profile for non-standard tuning

1. Click Add Profile.
2. Enter a descriptive Profile Name.
3. Enter the JVM arguments. Copy the arguments from the nearest host-RAM profile and adjust them as needed.
4. Click Save.
5. Open Cyber Vault > Agent Fleet Management > Live Agents and select the new profile in the target agent's Profile dropdown.

The agent restarts itself with the new profile once no copy job is running. No manual restart is needed.

Revert an agent to the default profile

1. Open Cyber Vault > Agent Fleet Management > Live Agents and find the agent.
2. Select the default profile in its Profile dropdown.

The assignment is saved immediately. The agent restarts itself with default once no copy job is running.

Tips

• The backend log is the most useful starting point for any ingest or connectivity problem. Look for lines tagged with the job ID or device name.
• Each system snapshot is timestamped inside the log entry. Compare consecutive snapshots to see how job state or memory changed between two points in time.
• Agent dashboard snapshots help with post-incident analysis. Compare bufferDepth and spoolDepth around a known outage window to see when ClickHouse connectivity was lost and recovered.
• A profile takes effect only when the agent restarts to apply it. The agent does this by itself once no copy job is running.
• An agent applies a profile only when its assigned profile changes. Editing the arguments of the profile it already runs does not restart it.
• For a custom profile on a larger host, set -Xmx to no more than about 75% of total host RAM, as the built-in profiles do. For example, use -Xmx24g for a 32 GB machine.

warning

Do not set -Xmx higher than the available memory. The OS out-of-memory killer then ends the JVM instead of producing a clean error.

• All built-in profiles include --add-opens java.base/java.net=ALL-UNNAMED. The agent needs it to report its DNS cache to the Fleet DNS Cache Monitor.
• The DNS Cache TTL setting on the Advanced tab does not apply to copy agents. Their DNS cache comes from the profile's -Dsun.net.inetaddr.* flags.