Skip to main content
Migration Notice
We're migrating documentation from the old portal into this one. Some things may look a little different or out of place in the meantime — we know, and we're working to get it right. If something's unclear or doesn't look right, let us know.

Database

The Database tab provides operational tuning and sizing for the two databases the system uses: ClickHouse (analytics and audit store) and PostgreSQL (operational state). It tunes an already-connected system.

Overview​

  • ClickHouse — a column-oriented analytics database. It holds every audit event ingested from PowerScale and ObjectScale/ECS devices, plus the high-volume security data: pipeline logs, Kubernetes audit events, and analyzer results. It is the authoritative record for file activity and posture history.
  • PostgreSQL — the application database. It holds operational state: jobs, settings, ingest offsets, agent registrations, and the Kubernetes topology.

The two stores are independent. ClickHouse audit data is not affected by the application database.

note

Both databases are provisioned and connected by the deployment, and their schemas are created on first boot. You do not need to enter hosts, ports, or credentials. The tab shows the PostgreSQL and ClickHouse connection fields, and they are editable. Do not change them unless Superna support asks you to.

Ingest and Retention​

  • Ingest Batch Size — the number of records sent in each ClickHouse insert during audit-log ingest for PowerScale and ECS SSH audit events. Default: 100,000, which covers one complete ECS audit file in a single insert. For PowerScale, any value above 1,000 works correctly. The minimum recommended value is 1,000.
  • Apply Retention — applies the Storage audit events (months) value from the Advanced tab to the audit events in ClickHouse, and the same retention to K8 inventory snapshots. Takes effect only with Apply Retention; enter the months first — 0 removes the expiry. See Retention Settings.
tip

ClickHouse processes the deletion asynchronously. Allow a few minutes after you apply retention before you check row counts.

K8 Inventory Snapshots​

Snapshot Frequency sets how often the system captures K8 inventory snapshots (pods, PVCs, and mounts) after each K8 scan.

OptionEffect
Every ScanMaximum visibility
HourlyReduces data volume in environments with frequent scans
DailyMinimal storage overhead

Snapshots power the Lifecycle Watch page, and the time-travel and deleted-pod detection on K8 Data Security and Data Auditing.

Retention Settings​

The retention period for the highest-volume ClickHouse data is set on the Advanced tab under ClickHouse retention. A saved value applies immediately without a restart, except Storage audit events. ClickHouse drops rows older than the window on its next merge pass.

SettingData coveredDefaultRange
Storage audit events (months)File create, write, and delete events from PowerScale and ObjectScale. Takes effect only with Apply Retention; enter the months first — 0 removes the expiry.0 in the field; the audit data starts with a 12-month expiry0–60
Pipeline logs (days)Diagnostics logs shown in the Pipeline Diagnostics flyout71–30
CVE findings (months)CVE trend history. Not used in this release.123–24
K8s audit events (days)Kubernetes API audit events36530–1095
Analyzer results (months)Output of the linguistic coherence (LC), classification, and object PII analyzers123–36

Lower Analyzer results if you are short on disk space. Raise it to meet legal or eDiscovery retention requirements.

Console Resource and Database Sizing​

The Console Resource & Database Sizing sub-tab provides host-resource estimates and tuning recommendations for ClickHouse and PostgreSQL. It has no Save button. Each database card has an Apply button, but neither works in this release (see below). Use the tiers, estimates and parameter tables as a reference.

Console Resource and Host Sizing​

This card estimates the minimum host resources when the console, ClickHouse, and PostgreSQL share one host. Enter the Number of Agents and Number of Buckets Managed. The card calculates CPU, RAM, and disk for each component and for the combined total, plus a recommended JVM heap size.

A table lists recommended values for ten ClickHouse server and MergeTree parameters, each with a short rationale.

TierTarget scale
SmallAbout 1 billion rows, 16 GB RAM host
MediumAbout 5 billion rows, 32 GB RAM host
LargeAbout 10 billion rows, 64 GB RAM host

The parameters are max_server_memory_usage, max_memory_usage, mark_cache_size, uncompressed_cache_size, background_pool_size, background_schedule_pool_size, max_bytes_to_merge_at_max_space_in_pool, parts_to_delay_insert, parts_to_throw_insert, and merge_tree_max_rows_per_part.

A storage estimate card shows the disk usage per tier for audit events, using a fixed sample compression ratio of 13.2x. A copyable ClickHouse query lets you read your current compression ratios and row counts at any time.

Apply ClickHouse Table Configuration​

warning

Apply to ClickHouse Tables does not work in this release: it ends with an error and applies nothing. ClickHouse keeps its default values for parts_to_delay_insert, parts_to_throw_insert and max_bytes_to_merge_at_max_space_in_pool.

The other seven parameters require a change to the ClickHouse config.xml file and a ClickHouse restart. A copyable config.xml snippet for the selected tier is shown on screen. A badge in the card header shows the tier that was last applied.

A table lists recommended PostgreSQL parameters for the write-heavy, small-connection-pool workload of this application, including shared_buffers and effective_cache_size.

TierTarget host
Small4–8 GB RAM
Large16–32 GB RAM

Apply PostgreSQL Configuration​

warning

Do not click Apply in the Apply PostgreSQL Configuration card. It does not work in this release and ends with an error. On the VMware appliance (OVA) it first changes PostgreSQL server settings, including shared_buffers and max_connections, which take effect at the next PostgreSQL restart and can break the appliance's PostgreSQL. If it was clicked on an appliance, contact Superna support before you restart PostgreSQL or the appliance. On Kubernetes, the database refuses the change.

Workflows​

Apply an audit retention policy​

  1. Set the window in Settings > Advanced Settings > ClickHouse retention > Storage audit events (months).
  2. Click Apply Retention on this tab. Saving the Advanced tab alone does not apply this value. Takes effect only with Apply Retention; enter the months first — 0 removes the expiry.
  3. Wait for the confirmation message.
  4. Verify that the ClickHouse row count dropped as expected.

Change database sizing​

The Apply buttons on this tab do not work in this release. Change the sizing where your deployment manages it:

  • VMware appliance (OVA): change the memory limits of the console and databases with sudo aisec-apply-sizing. See Install on VMware (OVA).
  • Kubernetes (Helm): set postgres.resources and clickhouse.resources in your values file and run helm upgrade. See Install on Kubernetes (Helm).