Skip to main content
Migration Notice
We're migrating documentation from the old portal into this one. Some things may look a little different or out of place in the meantime — we know, and we're working to get it right. If something's unclear or doesn't look right, let us know.

Ingest Offsets

The Ingest Offsets page is a debug view of the byte-level read position for every audit log file that the ingest pipeline tracks. It shows how far into each log file the system has read and processed. You can also reset the positions of all files on one node or device to force a re-read.

Where: the page is not in the menu. Open an audit ingest job on System Operations > System Jobs, then click View ingest offsets (or View all N files) in its Node / File Details section. You can also browse to https://<console>/ingest-offsets.

Overview​

The Jobs page shows a 24-hour summary of the same data. The ingest pipeline stores its position in each log file as a byte offset. After a restart, or when it continues an interrupted ingest, the pipeline reads from the stored offset instead of from the start of the file. This page shows the stored offsets and their metadata. Use it to diagnose ingest gaps, verify progress, or re-read the files of a node or device.

Offset table​

ColumnDescription
DeviceThe ECS or PowerScale device that the log file belongs to.
TypeThe device type, ECS or PowerScale, and whether the file is a rotated, compressed log (gz) or the current log (active). For both device types, the pipeline connects over SSH and reads the log files over SFTP.
FileThe name of the log file on the remote device.
OffsetThe number of bytes read and processed from the file. A value of zero means that the file has not been read yet or that the offset was cleared.
RecordsThe number of log events parsed from the file so far.
StatusThe state of the file in the pipeline. See Status values.
Last UpdatedWhen the offset record was last written. Use it to compare progress between checks.

Status values​

StatusMeaning
IN_PROGRESSThe file is being read, or was partially read and is not marked complete.
COMPLETEThe file is fully processed.
CLEAREDThe offset was reset manually. The file is re-read on the next ingest run.

Filters​

Each column has its own filter control at the top of the table.

  • Device and File accept free-text partial matches.
  • Type and Status are dropdowns.
  • Filters apply instantly and can be combined.
  • The row count badge in the header shows how many records match the current filters.

Sorting​

Click a column header to sort the table by that column. Click it again to reverse the order.

Infinite scroll​

Rows load in batches of 50 as you scroll down. In environments with many tracked files, scrolling to the bottom of the list loads more rows, so there are no pagination controls.

Clear offset​

Each row has a Clear action. It resets the byte position of every file on that row's node or device to zero, not only the file on that row, so the ingest pipeline re-reads all of those files from the beginning on its next run. Click Clear, then confirm with Yes. Read the warning under Tips first.

Workflows​

Verify that a log file is being processed​

  1. Type the file name or device name in the filter bar to narrow the list.
  2. Check the Status column. IN_PROGRESS with a non-zero byte offset confirms that the file is being read.
  3. Check Records to confirm that events are being parsed.

Force a re-read of log files after a parse error​

  1. Filter the table to find the file.
  2. Click Clear on that row.
  3. Click Yes to confirm.

The next ingest run for that node or device re-reads all of its files from byte zero.

Check whether ingest has stalled on a device​

  1. Filter by device name to show all files for the device.
  2. Look for files in IN_PROGRESS with a non-zero offset that has not changed since your last check.
  3. Compare with System Monitor > Audit Ingest Lag to see the lag for that device.

Tips​

warning

Clearing offsets reprocesses every file of that node or device from the beginning. If events from those files are already in ClickHouse, duplicate audit events are created. Use Clear only when duplicates are acceptable, or when you plan to wipe and re-ingest the ClickHouse data.

  • If a file is stuck in IN_PROGRESS with no progress, ingest can be stalled at the network level. Check System Monitor > Endpoint Reachability to verify that the device is reachable.
  • After an unexpected backend restart, the stored offset can be slightly behind the actual read position. The pipeline can re-read a small number of events after a restart, and those events are stored twice.