System Monitor
The System Monitor page gives you real-time visibility into JVM health, external endpoint reachability, audit ingest performance, and API performance. To open it, click System Monitor in the navigation bar.
Use it first when you investigate slowness, high memory usage, or a suspected stalled ingest.
Overview
The page is divided into panels, each covering a different aspect of system health. The panels show the current snapshot only. The one exception is the Outbound API Calls table, which accumulates counts since the last backend restart.
Most panels refresh automatically, each on its own schedule, and only while the browser tab is visible. The throughput figures in Audit Ingest Performance load when the page opens and have their own refresh link.
| Panel | Refresh interval |
|---|---|
| JVM metrics, operations, and API calls | 15 seconds. A countdown timer shows the time until the next refresh. |
| ClickHouse and PostgreSQL health | 10 seconds |
| Endpoint Reachability | 60 seconds |
| Audit Ingest Lag and Total Events by Storage Node | 60 seconds |
Polling pauses in hidden tabs
Each panel loads once when the page opens. After that, it refreshes only while the browser tab is visible.
A tab in the background, such as another tab selected or a minimized window, sends no requests. When you return to the tab, every panel whose refresh is due reloads at once.
A refresh never starts while the previous one for the same panel is still waiting for an answer. Several System Monitor tabs left open add no load to the console or to ClickHouse.
JVM Memory and CPU
This panel shows detailed memory and CPU metrics for the running JVM process.
| Metric | Description |
|---|---|
| Heap used / committed / max | Displayed as a bar. Use it to gauge how close the JVM is to its heap limit. If heap used stays near max, the process is at risk of an OutOfMemoryError. |
| Non-heap memory | Memory used for JVM internals outside the heap, including code cache and compiler buffers. |
| Direct buffers | Off-heap buffers allocated by NIO channels and similar APIs. High direct buffer usage alongside high heap can indicate I/O-heavy operations. |
| Mapped buffers | Memory used by memory-mapped files, with the buffer count and capacity. It shows "no mapped files" when none are in use. |
| Metaspace | Memory used for class metadata. It grows with the number of loaded classes and does not shrink unless classes are unloaded. |
| Thread count (total / daemon / peak) | The current number of JVM threads, how many are daemon threads, and the peak since startup. |
| GC run count and time | How many garbage collection cycles have run and the cumulative time spent in GC. High GC time relative to uptime can indicate heap pressure. |
| System memory | Total and free memory on the host OS, as reported by the JVM. |
Memory Advisor
The Memory Advisor sits under the JVM metrics. It compares the running process with the memory and thread limits it was started with, and gives a short verdict for each area. The header shows the peak thread count, the configured heap, and the host RAM it is based on.
| Area | What it shows |
|---|---|
| Heap | Heap used and maximum, with a verdict such as "Heap sizing OK for current load". It also shows the peak thread count and the estimated in-flight data. |
| Direct Memory | Direct memory used and its limit, which defaults to the heap maximum. It estimates the direct memory the peak thread count needs, at about 1 MB of TLS buffer per thread, and says whether the limit is enough. |
| GC & Thread Stacks | Garbage collection time and count, the native memory used by thread stacks, and the active collector. It adds a tip when a smaller thread stack size would save memory. |
The advisor reports on the running process only. It does not change any setting.
Active JVM parameters
This panel lists the JVM flags the process is running with and shows how many there are. Use it to confirm which memory, garbage collection, and diagnostic options are active.
Below the list, Recommended JVM flags for high-parallelism copy workload shows a suggested command line with a copy button. It also shows the host RAM, how much is free, and the largest safe heap size, which is 75% of the host RAM.
Active Operations
This panel lists the operations that are running now. For each one it shows:
- Wall-clock elapsed time.
- CPU time consumed.
- Thread efficiency (CPU time ÷ wall time).
- Memory allocated.
Each entry represents a tracked code path, such as an ingest run, a ClickHouse query, or a copy job step.
An operation that stays here for an unusually long time can indicate a stuck job. It can also indicate a slow dependency, such as a network timeout while waiting on an external device.
Recent Completed Operations
This panel lists the 50 most recently completed operations, sorted by completion time. It shows their wall-clock and CPU duration.
Use it to review what the system did in the last few minutes and to spot operations that took unexpectedly long.
Endpoint Reachability
This table lists all configured external endpoints and shows whether each is currently reachable. Only endpoints relevant to this product are shown. The panel refreshes every 60 seconds, and a refresh button in the panel header reloads it on demand.
| Column | Description |
|---|---|
| Endpoint name | The name of the endpoint. |
| Type badge | Storage, ClickHouse, or App Database. |
| Status | UP, DOWN, or Not configured. Rows with DOWN status are highlighted in red. |
| Latency | Response time in milliseconds. |
| URL | The endpoint URL. |
| Last checked | When the endpoint was last checked. |
| Error | For ClickHouse, ClickHouse's own error, for example Cannot connect: HTTP 500: Code: 202 … TOO_MANY_SIMULTANEOUS_QUERIES. |
After 5 health checks in a row fail, the console pauses its own ClickHouse health checks for 30 seconds. The error column then shows ClickHouse circuit open — skipping for Ns [health lane …].
Other console subsystems have their own circuit breakers, which are shown on the ClickHouse Health page.
ClickHouse and PostgreSQL Health
ClickHouse Summary
The ClickHouse Audit Database panel shows these values for the ClickHouse analytics database:
- Connection status
- Latency
- Version
- Compressed and uncompressed table size
Click Full DB Health Analytics to open the ClickHouse Health page for deeper diagnostics.
PostgreSQL Summary
This panel shows these values for the application PostgreSQL database:
- Connection status
- Latency
- Version
- Active connection count and idle connections
- Pool utilization percentage, color-coded green, amber, or red
- Total connections and max connections
- Database size
A ping slower than 100 ms is shown in bold with a yellow warning icon. Hover over the icon to see "Slow".
The panel is hidden when PostgreSQL is not configured. It links to the Postgres Health page for detailed analytics.
Audit Ingest Lag
This panel shows one lag card for each configured source device or cluster. Each card shows the time elapsed since the most recent ingested audit event for that source.
Cards are color-coded:
- Green: the last event was less than 5 minutes ago.
- Amber: the last event was 5 to 60 minutes ago.
- Red: the last event was more than 60 minutes ago.
Use this panel to quickly identify devices that have stopped delivering events. A cluster that stays red for more than an hour also raises alarm IS-002 (Audit ingest RPO breach).
Where the numbers come from
The lag cards and Total Events by Storage Node come from cached audit statistics. The console reads them from ClickHouse at most once a minute, so they are never fresher than about a minute.
On a large audit table, a statistics index is built in the background after an upgrade. Meanwhile the header shows statistics index build in progress (NN%).
The lag is read from the most recent events only, and the totals refresh every 6 hours, in these cases:
- While the index builds.
- After a console start.
- After an audit data wipe.
- After a device purge.
- After a retention change.
The time in the chart header shows when the totals were last read. A cluster that is catching up on events older than the last few hours shows no progress on its card until its events reach that window.
Header and states
The header shows as of HH:MM:SS, the browser time when the statistics were read. Next to it, a badge shows the state of the statistics.
| State | What it means |
|---|---|
| OK ("as of" time only) | The last read succeeded and its values are recent. |
| Stale (amber) | The latest read failed, or the values are more than 15 minutes old. The last values received are shown with a banner naming the error, and the console keeps retrying. |
| Unavailable (red) | No value could be read. The console keeps retrying. Alarm IS-004 is raised if this lasts 15 minutes. |
| Loading… | The first read after a console start, wipe, purge, or retention change is still running. |
| Not configured | No ClickHouse database is configured. |
If the panel says No ingest data available — run an audit ingest job first., the statistics were read and the audit table has no events yet. If it says the console could not be reached, the cards keep the last values received.
The refresh link in the Audit Ingest Performance panel reloads the statistics and the throughput chart at once.
Audit Ingest Performance
This panel shows historical throughput data for completed ingest jobs. The throughput chart loads when the page opens. The refresh link reloads it together with the audit ingest statistics.
Per-type summary pills
The pills show the average events per second, total events inserted, and job count for the last 10 completed jobs of each ingest type. ObjectScale audit logs are collected only over SSH.
Total Events by Storage Node
This horizontal bar chart shows accumulated event counts per source device. It uses the same cached statistics as the lag cards. Summarized rows count as the number of operations they stand for.
Click a bar to filter the throughput scatter chart to that device.
The header reads "N total · as of HH:MM:SS". This is the time of the full read the totals come from:
- While the index is usable, it matches the lag cards.
- While the lag is read for the last hours only, it can be up to 6 hours earlier. See Where the numbers come from.
If the last background full read failed, the header adds "(stale: the last full read failed)" and keeps the totals read before. When the statistics are loading or unavailable, the chart shows the same message as the lag panel instead of bars.
Ingest Throughput per Job
This scatter chart plots events per second for each completed ingest job. Dots are color-coded by job type, matching the colored type pills above the chart.
Hover over a dot to see the job ID, device name, event rate, and insert count.
Outbound API Calls
This table lists the calls the console makes to external systems, such as the ObjectScale management API or the vault target. Calls are grouped by endpoint, with metrics accumulated since the last backend restart.
| Column | Description |
|---|---|
| Calls | Total number of calls made to this endpoint group. |
| Errors | Number of calls that returned an error. Shown in red when above zero. |
| Avg / Min / Max / P95 | Response time distribution in milliseconds. P95 is the 95th percentile of the last 20 calls, so 95% of those calls were faster than this value. |
| Trend (last 20) | A small inline chart of the latency of the last 20 calls. |
| Last URL | The most recent URL called. |
Slow values stand out:
- Over 300 ms is shown in bold with a yellow warning icon. Hover over it to see "Slow" or, above 1 second, "Very slow".
- Over 5 seconds is shown in bold red.
Use this table to identify slow endpoints or endpoint groups with elevated error rates. ObjectScale management API entries, named "ECS: …", are highlighted for easy identification.
Disk I/O
This panel appears on Linux hosts only. It shows read and write IOPS, I/O latency, and utilization percentage for each block device visible to the OS.
High utilization on the console's data disk can explain slow ingest or slow ClickHouse writes.
Workflows
Diagnose high memory usage
- Check the Heap used bar in the JVM Memory panel. If it is near max, memory pressure is the likely cause.
- Check the GC run count and time. Frequent GC with a long cumulative time confirms heap pressure.
- Check Active Operations for any operation that has run for an unusually long time. It may be holding references to large objects.
Investigate a stalled ingest
- Open the Audit Ingest Lag panel.
- Check the header first. If it shows Stale or Unavailable, the lag itself could not be read, and the banner names the error. Investigate ClickHouse on the ClickHouse Health page before the device.
- Find the card for the affected device and check the lag value.
- If the lag is red and growing, check the Endpoint Reachability table for that device. It may have become unreachable.
- Check Active Operations for any ingest operation that appears stuck.
- If the ingest is running and inserting events but the lag still grows, the cluster may be catching up on a backlog older than the last few hours. Its card updates once its events reach that window, or at the next full read, at most 6 hours later.
Identify the slowest API endpoint
- Open the Outbound API Calls table.
- Look for the highest P95 values, shown in bold or red when slow.
- Investigate the top entries. Cross-reference with Active Operations to see if a long-running operation corresponds to that endpoint.
Tips
- If the JVM heap is consistently above 80% of max, consider increasing the heap allocation in the deployment configuration before you hit an OutOfMemoryError.
- Outbound API Calls statistics reset on each backend restart. After a recent restart, the counts and averages reflect only the current session.
- The JVM countdown timer shows the seconds until the next automatic refresh of the memory and GC panels. The 15-second interval keeps backend overhead low.
- A System Monitor tab left open in the background sends no requests. When you switch back to it, the panels that are due refresh immediately.
- The audit ingest panels never show fresher values than about a minute. Total Events by Storage Node can be up to 6 hours old on a large table, so check the time in its header.