Skip to main content
Migration Notice
We're migrating documentation from the old portal into this one. Some things may look a little different or out of place in the meantime — we know, and we're working to get it right. If something's unclear or doesn't look right, let us know.

Ingest Offsets

The Ingest Offsets page is a debug view of the byte-level read position for every audit log file that the ingest pipeline tracks. It shows how far into each log file the system has read and processed. You can also reset the position of individual files to force a re-read.

Overview​

The Jobs page shows a 24-hour summary of the same data. The ingest pipeline stores its position in each log file as a byte offset. After a restart, or when it continues an interrupted ingest, the pipeline reads from the stored offset instead of from the start of the file. This page shows the stored offsets and their metadata. Use it to diagnose ingest gaps, verify progress, or re-read a specific file.

Offset table​

ColumnDescription
Device nameThe ECS or PowerScale device that the log file belongs to.
Device typeThe ingest method for the device: SSH (the pipeline connects over SSH to retrieve log files) or syslog (the device pushes events over UDP syslog).
File nameThe name of the log file on the remote device.
Byte offsetThe number of bytes read and processed from the file. A value of zero means that the file has not been read yet or that the offset was cleared.
Records readThe number of log events parsed from the file so far.
StatusThe state of the file in the pipeline. See Status values.
Last UpdatedWhen the offset record was last written. Use it to compare progress between checks.

Status values​

StatusMeaning
IN_PROGRESSThe file is being read, or was partially read and is not marked complete.
COMPLETEThe file is fully processed.
CLEAREDThe offset was reset manually. The file is re-read on the next ingest run.

Filters​

Each column has its own filter control at the top of the table.

  • Device and file name accept free-text partial matches.
  • Device type and status are dropdowns.
  • Filters apply instantly and can be combined.
  • The row count badge in the header shows how many records match the current filters.

Sorting​

Click a column header to sort the table by that column. Click it again to reverse the order.

Infinite scroll​

Rows load in batches of 50 as you scroll down. In environments with many tracked files, scrolling to the bottom of the list loads more rows, so there are no pagination controls.

Clear offset​

Each row has a Clear Offset action. It resets the byte position of the file to zero, so the ingest pipeline re-reads the whole file from the beginning on its next run. The action needs a two-step confirmation click to prevent accidental resets. Read the warning under Tips first.

Workflows​

Verify that a log file is being processed​

  1. Type the file name or device name in the filter bar to narrow the list.
  2. Check the Status column. IN_PROGRESS with a non-zero byte offset confirms that the file is being read.
  3. Check Records read to confirm that events are being parsed.

Force a re-read of a log file after a parse error​

  1. Filter the table to find the file.
  2. Click Clear Offset on that row.
  3. Confirm the two-step prompt.

The next ingest run for that device re-reads the file from byte zero.

Check whether ingest has stalled on a device​

  1. Filter by device name to show all files for the device.
  2. Look for files in IN_PROGRESS with a non-zero offset that has not changed since your last check.
  3. Compare with System Monitor > Audit Ingest Lag to see the lag for that device.

Tips​

warning

Clearing an offset reprocesses the whole file from the beginning. If events from that file are already in ClickHouse, duplicate audit events are created. Use Clear Offset only when duplicates are acceptable, or when you plan to wipe and re-ingest the ClickHouse data.

  • If a file is stuck in IN_PROGRESS with no progress, ingest can be stalled at the network level. Check System Monitor > Endpoint Reachability to verify that the device is reachable.
  • After an unexpected backend restart, the stored offset can be slightly behind the actual read position. This is by design. The pipeline can re-read a small number of events after a restart, and ClickHouse deduplication handles them.