ClickHouse Health

The ClickHouse Health page (ClickHouse Database Health) provides diagnostics for the analytics database that stores audit events, copy job metrics and ingest telemetry. Use it to investigate database performance, memory problems, stuck mutations and storage growth. The page refreshes every 15 seconds; Pause / Resume in the header freezes the view while you investigate.

If ClickHouse can't answer part of the page, that section shows Could not load: <error> and its card shows —; the rest of the page keeps working.

Summary Cards

Card	What it shows
Memory (Resident)	The memory the ClickHouse process holds, with In use and the server memory Limit below. Green below 75% of the limit, amber from 75%, red from 90%, gray when unknown.
Active Queries	Queries running now, with active merges and delayed inserts below. Red when ClickHouse is delaying inserts (too many parts).
Connections	Open connections to ClickHouse.
Mutation Health	Pending mutations, with the total and failed counts. Red when any mutation is pending.
Merge Pressure	The largest number of parts in any partition, with the number of active merges. Amber above 50, red above 150. A high value can temporarily slow inserts while merges catch up.

Below the cards, Cumulative rates since server start shows queries, inserts and selects per second, average query and insert time, and the error rate (amber above 1%, red above 5%).

Memory

The Memory panel shows how much memory ClickHouse holds compared with what it actually uses. ClickHouse refuses work when its resident memory reaches its limit, however little its queries are using, so memory that is held but not in use matters as much as query load.

Figure	What it is
Resident	The memory ClickHouse holds, with its peak since start.
In use	What ClickHouse counts as used by queries, merges and caches.
Not in use	Resident minus in use. On a healthy server this is small, under 10% of the limit. Green, amber or red as it approaches the CH-003 levels.
Counted against limit	What ClickHouse compares with its limit. Amber from 85%, red from 95%.
Limit	ClickHouse's server memory limit.
Allocator purges since start	How often ClickHouse has released freed memory back since it started.

The bar is scaled to the limit: blue is memory in use, the remainder of resident memory is memory not in use, and Free is what is left below the limit.

Likely cause (estimate). When memory not in use is high, the panel suggests where it probably sits:

Shown	What to do
Normal	Nothing.
Cannot tell	This ClickHouse version doesn't report the needed statistics. Use the figures above.
Freed memory the allocator keeps	ClickHouse releases it by itself once memory is above 90% of its limit.
Allocator fragmentation, Allocations ClickHouse does not track, Memory outside the allocator	Restart ClickHouse at a quiet moment (see the steps below).

Memory alarms

The console samples ClickHouse memory every 60 seconds and raises two alarms:

• CH-003 — ClickHouse memory not released (warning): memory not in use stays at or above max(30% of the limit, 4 GiB) for 10 minutes. It clears after 5 minutes below max(20% of the limit, 2 GiB).
• CH-004 — ClickHouse at its memory limit (critical, emailed at once by default): the larger of the counted and in-use memory stays at or above 95% of the limit for 3 minutes. It clears after 3 minutes below 85%.

Between the alarm and clear levels neither alarm is raised nor cleared. The console never purges or restarts ClickHouse by itself. To silence an alarm for a while, disable its rule on the Alarms page.

When CH-003 or CH-004 is raised

1. Read the Memory panel. Note Resident, In use, Not in use and the likely cause. If most of the memory is in use, the cause is load, not held memory: check Currently Running Queries and Recent Queries for heavy queries.
2. Take a support bundle from Setup & Settings > Backup & Restore > Create Backup Now. It contains the memory history.
3. If memory not in use stays high, restart ClickHouse at a quiet moment. Audit ingest and agent writes retry by themselves. CH-003 / CH-004 clear a few minutes later, once fresh samples are below their clear levels.
4. Send the bundle to support with the time of the alarm.

Table Storage & Parts Health

Lists each table with its row count (Rows), compressed size (Compressed), compression Ratio, number of parts (Parts, amber above 50, red above 100) and when it was last modified. A very high part count means many small inserts were written faster than ClickHouse could merge them. This is a warning sign but usually resolves once the insert rate drops.

Currently Running Queries

The queries running now, with elapsed time (amber above 5 s, red above 30 s), memory used, rows read and the user. Click a query to see its full text. Use this to identify runaway queries that consume disproportionate resources.

Mutations

The 20 most recent table mutations: table, command, creation time, status (Pending, Done or Failed with a reason) and the parts still to do. Pending ones are listed first.

A pending mutation that is not making progress needs attention. Completed mutations that are still registered re-apply to all new data inserted into that table, so after any wipe or delete operation, check this panel and contact support if a completed mutation is not being cleaned up.

Maintenance mutations started by the console itself (the audit statistics index build, or a retention change) also appear here while they run. It is safe to let them finish.

Merge Pressure

The largest number of parts in any partition, and the merges running now with their progress.

Console Circuit Breakers

The console protects itself from a failing ClickHouse with a circuit breaker per subsystem (lane), so one subsystem's failures don't block the others. A lane opens after 5 failures in a row; while it is open, its calls fail at once with "ClickHouse circuit open — skipping for Ns", and it retries by itself every 30 seconds, closing as soon as ClickHouse answers.

Lane	What uses it
core	Start-up setup, settings, admin operations, retention, wipes and purges
ingest	Audit ingest
sync	Vault sync and copy
analysis	Window previews and change estimates
status	Dashboards, Cyber Vault lists, MCP reads
diagnostics	This page, snapshot and storage statistics
alarm checks	Alarm inputs such as the audit ingest lag and throughput degradation (AG-007)
telemetry	Heartbeat and job metrics
health	The ClickHouse health check (System Monitor > Endpoint Reachability)

The table shows, per lane, its State (Closed, Open or Probing), Retry in, Consecutive failures, Opened since start, Fast-failed calls and Last failure. If a lane is open, read its last failure to find the cause (memory, overload or connectivity); the other lanes keep working meanwhile.

Recent Queries — Last 10 Minutes (slowest first)

The 50 slowest queries in the last 10 minutes: start time, duration (amber above 100 ms, red above 1 s), rows and bytes read, memory, and whether the query finished or failed. Click a query to see its full text. The panel opens by itself when a query took more than a second. Use it to spot inefficient query patterns.

Cumulative Event Counters Since Server Start

Expand this section for ClickHouse's own event counters since it started (queries, inserts, failures, merged rows, network traffic), with descriptions and a filter box.

Workflows

Check for stuck mutations after a data wipe

1. Open the Mutations panel.
2. Look for any mutation that is not Done.
3. If a completed mutation is still present, contact support.

Investigate a table with slow insert performance

1. Open the Table Storage & Parts Health panel and check the table's Parts count.
2. If it is very high, check Merge Pressure and Active Queries (delayed inserts).
3. If many merges are active, the table is catching up; inserts slow temporarily until merges complete.

Find what is causing high ClickHouse CPU usage

1. Open the Currently Running Queries panel and look for queries with a high elapsed time.
2. Expand Recent Queries for the full text of suspicious queries.

Tips

• After any wipe or delete operation, return to this page and check the Mutations panel.
• High merge pressure usually resolves itself within minutes once insert pressure eases. It needs attention only if it persists for hours.
• Use Pause to read a slow-query entry or a mutation without the panel refreshing. Remember to Resume.
• A Not in use value that keeps growing after each heavy job, while Resident does not fall, is the pattern CH-003 reports: plan a ClickHouse restart at a quiet moment.