System Monitor
The System Monitor page gives real-time visibility into application memory and CPU health, external endpoint reachability, audit ingest performance, and outbound API calls. Use it first when you investigate slowness, high memory usage, or a suspected stalled ingest.
Where: System Operations > System Monitor
Overview
The page is divided into panels, each covering one aspect of system health. Panels refresh on their own schedules:
| Panel | Refresh |
|---|---|
| JVM memory and CPU, Active Operations, Recent Completed Operations, Outbound API Calls, Disk I/O | Every 15 seconds. A countdown next to the page title shows the time until the next refresh. |
| ClickHouse Audit Database, PostgreSQL Application Database | Every 10 seconds |
| Audit Ingest Lag | Every 30 seconds |
| Endpoint Reachability | Every 60 seconds |
| Audit Ingest Performance | When the page loads, or when you click refresh in the panel header |
The panels show the current snapshot only. The exception is the Outbound API Calls table, which accumulates counts over time and keeps them across console restarts.
JVM Memory and CPU
These cards and panels show memory and CPU metrics for the running application process.
| Metric | Meaning |
|---|---|
| Heap Memory | Heap used and maximum, shown as a bar with the percentage used. Heap used consistently near max puts the process at risk of running out of memory. |
| Host CPU | CPU use of the whole host, across all processes, with the number of cores. |
| JVM CPU | CPU use of the application process, with the current thread count and how many are daemon threads. |
| Off-Heap Memory | Memory the JVM uses outside the heap: Direct Buffers (NIO buffers, with count and capacity), Mapped Buffers (memory-mapped files), and Metaspace (class metadata, with the non-heap total). |
| GC runs, GC time and Peak threads | Number of garbage collection cycles, the cumulative time spent in them, and the peak thread count since startup. High GC time relative to uptime can indicate heap pressure. |
Active Operations
This panel lists the operations running now, with the context, wall-clock time elapsed, CPU time consumed, thread efficiency (CPU time divided by wall time) and memory allocated. Each operation is a tracked unit of work such as an ingest run or a ClickHouse query.
An operation that stays here for an unusually long time can indicate a stuck job or a slow dependency, such as a network timeout waiting on an external device.
Recent Completed Operations
This panel lists the last 50 completed operations with the same columns as Active Operations, plus the completion time. Use it to review what the system did in the last few minutes and to spot operations that took unexpectedly long.
Endpoint Reachability
This table shows whether each configured external endpoint is reachable. The panel refreshes every 60 seconds, and a refresh button in the panel header refreshes it immediately.
| Column | Meaning |
|---|---|
| Endpoint | Endpoint name |
| Type | Badge: Kubernetes, CVE Scanner, RunAI, Storage (PowerScale clusters), ClickHouse, or App Database. CVE Scanner shows Not configured, because CVE scanning is not available in this release. |
| Status | UP, DOWN, or Not configured |
| Latency | Response time in milliseconds |
| URL | Endpoint address |
| Last checked | Time of the last check |
Rows with DOWN status are highlighted in red.
ClickHouse and PostgreSQL health
These summary panels link to the detailed health pages through Full DB Health Analytics, and each has a refresh button.
- ClickHouse Audit Database — connection status, version, latency, the number of audit rows, the size on disk (with the uncompressed size), and the compression ratio. It links to the ClickHouse Health page.
- PostgreSQL Application Database — connection status, version, ping time, active and idle connections, pool utilization percentage (green, amber, or red) with total and maximum connections, and database size. It links to the PostgreSQL Database Health page. The panel is hidden when PostgreSQL is not configured.
Audit Ingest Lag
One lag card per device or cluster shows the time since the most recent ingested audit event from that source. Use these cards to find devices that have stopped delivering events.
| Card color | Time since last event |
|---|---|
| Green | Less than 5 minutes |
| Amber | 5 to 60 minutes |
| Red | More than 60 minutes |
Audit Ingest Performance
This panel shows throughput for completed ingest jobs.
- Per-type summary pills — average events per second, total events inserted, and job count for the last 10 completed jobs of each ingest type (PowerScale, ECS SSH).
- Total Events by Storage Node — a horizontal bar chart of accumulated event counts per source device. Click a bar to filter the throughput chart to that device.
- Ingest Throughput per Job — a scatter chart of events per second for each completed ingest job. Colors identify the job type: PowerScale is purple and ECS SSH is amber. Hover over a dot to see the job ID, device name, event rate, and insert count.
Outbound API Calls
This table lists the console's calls to external systems, such as the Kubernetes API, RunAI and ObjectScale, grouped by endpoint. The counts are kept across console restarts.
| Column | Meaning |
|---|---|
| Endpoint | The external system called |
| Calls | Total calls made |
| Errors | Calls that returned an error |
| Avg / Min / Max | Average, fastest and slowest call latency in milliseconds |
| P95 | The 95th percentile latency of the last 20 calls |
| Trend (last 20) | A small inline chart of the latency of the last 20 calls |
| Last URL | The address of the most recent call |
Use the table to find slow external systems and those with errors. ObjectScale entries are highlighted in sky blue.
Disk I/O
This panel appears on Linux hosts only. For each block device visible to the operating system, it shows average read and write latency, read and write IOPS, in-flight I/O, and utilization percentage. High utilization on the device that holds the application data can explain slow ingest or slow ClickHouse writes.
Workflows
Diagnose high memory usage
- In the JVM Memory and CPU panel, check the Heap used bar. If it is near max, memory pressure is the likely cause.
- Check the GC run count and time. Frequent GC with a long cumulative time confirms heap pressure.
- In Active Operations, look for an operation that has run unusually long. It may hold references to large objects.
Investigate a stalled ingest
- Open the Audit Ingest Lag panel.
- Find the card for the affected device and check the lag value.
- If the lag is red and growing, check the Endpoint Reachability table for that device. It may be unreachable.
- Check Active Operations for an ingest operation that appears stuck.
Identify the slowest external system
- Open the Outbound API Calls table.
- Compare the P95 and Max values across the rows.
- Investigate the slowest entries. Compare them with Active Operations to see whether a long-running operation corresponds to that system.
Tips
- If the heap stays above 80% of max, consider increasing the heap allocation in the deployment configuration before the process runs out of memory.
- Outbound API call counts are kept across console restarts, so the counts and averages can include calls made before the last restart.