Skip to main content
Migration Notice
We're migrating documentation from the old portal into this one. Some things may look a little different or out of place in the meantime — we know, and we're working to get it right. If something's unclear or doesn't look right, let us know.

Ingest Offsets

The Ingest Offsets page is a diagnostic view of the byte-level read position for every audit log file that the ingest pipeline tracks.

It shows how far into each log file the system has read and processed. You can also reset a node's file positions to force a re-read.

Open it from Setup & Settings > Database > Ingest Offset Management > View Full List, or from the View all files link in an ingest job's details.

Overview​

The ingest pipeline stores a byte offset for each log file in the application database. After a restart, or when it continues an interrupted ingest, the pipeline reads from the stored offset instead of from the beginning of the file.

This page shows those offsets and their metadata. Use it to diagnose ingest gaps, verify progress, or trigger a re-read of a specific file.

First sync and historical archives​

The first time a node is ingested over SSH, the pipeline does not pull the weeks or months of rotated .gz archives on the node. Pulling them would put heavy load on the storage nodes and the console.

Instead, the pipeline:

  • Records every existing archive as SKIPPED without downloading it.
  • Ingests only the active dataheadsvc-access.log, the log that is currently being written.

Logs that rotate after the first sync are new files and are ingested normally. This includes logs that rolled over while the product was down.

The first sync is detected per node, so a node added later to an existing device gets the same treatment. Clearing offsets or wiping audit data returns a node to this state.

Offset table columns​

ColumnDescription
Device nameThe ObjectScale device that the log file belongs to.
Device typeThe source platform. The filter dropdown has the options All and one for the ObjectScale source type, which is the Dell ObjectScale audit log pulled over SSH.
File nameThe name of the log file on the remote device.
Byte offsetThe number of bytes read and processed from this file. Zero means the file has not been read yet or the offset was cleared.
Records readThe number of log events parsed from this file so far.
StatusThe current state of the file in the pipeline. See the next table.
Last UpdatedWhen the offset record was last written. Use it to compare progress between checks.

Status values​

StatusMeaning
IN_PROGRESSThe file is being read, or was partially read. The active log normally sits here between runs.
COMPLETEThe file was fully downloaded, decoded, and inserted. It has a non-zero Records read count.
SKIPPEDA pre-existing .gz archive that the device's first sync recorded as done without downloading it. Records read is 0. It is never read again.
CLEAREDThe offset was manually reset. The file is re-read on the next ingest run.

COMPLETE and SKIPPED are terminal. The file is skipped on future runs. Only IN_PROGRESS is re-checked.

A .gz archive at IN_PROGRESS means an earlier attempt did not finish, for example because of a ClickHouse or transfer error. It is retried on the next run and resumes from its stored offset, so nothing is skipped.

An archive that stays at IN_PROGRESS across many runs keeps failing. Check the job log for its error.

Filters​

Each column has its own filter control at the top of the table.

  • Device and file name accept free-text partial matches.
  • Device type and status are dropdowns.
  • Filters apply instantly and can be combined.

The row count badge in the header shows how many records match the current filters.

Time window selector​

A dropdown in the header limits which COMPLETE rows are loaded. IN_PROGRESS rows are always shown, regardless of the window.

  • Last 24 hours (default, fastest)
  • Last 3 days
  • Last 7 days
  • Last 30 days
  • Last 90 days
  • All records (shown in amber as a warning, because it may be slow on busy clusters)

The row count badge shows the active window as a short label, such as 24h or 7d, next to the count.

Sorting and scrolling​

Click a column header to sort by that column. Click again to reverse the direction.

Rows load in batches of 50 as you scroll down. There are no pagination controls.

Clear Offset​

Each row has a Clear button. Clearing applies to every file of that row's node. All of the node's stored offsets are removed and the node returns to the first-sync state.

On the next run, the existing archives are recorded as SKIPPED again and the active log is read from the beginning.

Clearing a node's offsets also resets the ingest freshness of that node's whole device. Its sync windows then show Waiting for ingest until its next completed ingest run. See When a Window Waits for Ingest on the Incremental Sync page.

Workflows​

Verify that a specific log file is being processed​

  1. Type the file name or device name in the filter bar.
  2. Check the Status column. IN_PROGRESS with a non-zero byte offset confirms that the file is being read.
  3. Check the Records read column to confirm that events are being parsed.

Force a re-read of a node's active log after a parse error​

  1. Filter the table to find the file.
  2. Click Clear on that row.
  3. Click Yes to confirm.

The next ingest run for that node re-reads the active log from byte zero. Existing archives on the node are recorded as SKIPPED and are not re-read.

Check whether ingest has stalled on a device​

  1. Filter by device name to show all files for the device.
  2. Look for files stuck at IN_PROGRESS with a non-zero offset that has not changed since your last check.
  3. Cross-reference with System Monitor > Audit Ingest Lag to check the lag for that device.

A .gz archive at IN_PROGRESS is waiting for a retry. It is not a stalled live read.

Tips​

warning

Clearing a node's offsets re-reads its active log from the beginning. If those events were already ingested, duplicate audit events are created. Use Clear only when duplicates are acceptable or you intend to wipe and re-ingest the audit data.

  • If a file is stuck at IN_PROGRESS with no progress, ingest may have stalled at the network level. Check System Monitor > Endpoint Reachability to verify that the device is reachable.
  • After an unexpected restart, the stored offset may be slightly behind the actual read position. The pipeline may re-read a small number of events, and duplicates are handled automatically.
  • If the connection to a node drops while the active log is being read, events already inserted are kept. Only the remainder is read on the next run.