Skip to main content
Migration Notice
We're migrating documentation from the old portal into this one. Some things may look a little different or out of place in the meantime — we know, and we're working to get it right. If something's unclear or doesn't look right, let us know.
Version: 2.15.0

Archive Engine Guide for Data Orchestration for Dell

Introduction​

Archive Engine is the licensed Data Orchestration capability that provides end-user self-service archiving and recall, administrator-defined age-based archive policies, and AWS S3 Deep Archive tier transparency. This guide covers configuring Archive Engine folder definitions, the end-user archive/recall workflow, data lifecycle tracking, Deep Archive tier transparency, policy-based archiving, and dry-run testing. For licensing and conceptual background, see Concept and Pre-Requisites.

Prerequisites​

  • Archive Engine subscription license.
  • Two S3 buckets: one to hold archived data, and one to stage recalled data (a third, optional trash bucket can hold deleted staging data with a lifecycle policy).
  • The Pipeline NFS mount for data recall configured on Golden Copy (see Installation Guide), with the mount path matching the --recall-sourcepath value used below.

Configuring Self-Service Archiving​

An Archive Engine configuration is built from two folder definitions — one for on-premises-to-cloud archiving, and one for cloud-to-on-premises recall — linked together.

Prepare the Recall Data Location​

Set the cluster's recall data location, used to uniquely store all data arriving from the cloud for pipeline-based workflows:

searchctl isilons modify --name <cluster name> --recall-sourcepath /ifs/fromcloud

Add the Upload (On-Premises to Cloud) Folder Definition​

The staging path for data pending archive is typically /ifs/tocloud. Key flags for this folder:

  • --archive-engine-sourcepaths — authorizes one or more paths (comma-separated) that end users can select for self-service archiving, for example /ifs/projects,/ifs/home.
  • --archive-engine-type upload — marks this definition as the on-premises-to-cloud direction.
  • --full-archive-schedule — a cron schedule for when staged data is moved to the cloud (a daily off-peak schedule is recommended).
  • --delete-from-source — deletes data from the staging path once it has been copied (required for a move, rather than backup, workflow).
  • --storeFileHistory — encodes each file's archive/recall history as a custom S3 object property.
  • --type AE — marks the folder definition as an Archive Engine configuration.
  • --tier (optional) — for AWS or Azure targets; see AWS Deep Archive Tier Transparency below for the deep archive option.
searchctl archivedfolders add --isilon <cluster name> --folder /ifs/tocloud --accesskey <access key> --secretkey <secret key> --endpoint <endpoint> --region <region> --bucket <archive bucket> --cloudtype aws --archive-engine-sourcepaths /ifs/projects --full-archive-schedule "0 0 * * *" --delete-from-source --archive-engine-type upload --storeFileHistory --type AE

Add the Recall (Cloud to On-Premises) Folder Definition​

Create the recall staging path on the cluster (as root), for example mkdir -p /ifs/fromcloud/<staging-bucket-name>, then add the recall folder definition:

  • --source-path <cluster name>/<archived path> — authorizes which archived data is presented to end users for recall; all data under this path becomes visible for recall.
  • --bucket — must match the archive bucket used in the upload folder definition above, tying the two definitions together.
  • --recall-schedule — cron schedule for how often the staging bucket is scanned before recalling data on-premises.
  • --recall-from-sourcepath, --archive-engine-type recall, --trash-after-recall, --recyclebucket, --storeFileHistory, --type AE — as with the upload definition, plus --trash-after-recall to delete staging data once recalled, and --recyclebucket to move it to a lifecycle-managed trash bucket instead.
searchctl archivedfolders add --isilon <cluster name> --folder /ifs/fromcloud/<staging-bucket-name> --accesskey <access key> --secretkey <secret key> --endpoint <endpoint> --region <region> --bucket <staging bucket> --cloudtype aws --source-path <cluster name>/ifs/projects --recall-schedule "*/5 * * * *" --recall-from-sourcepath --archive-engine-type recall --trash-after-recall --recyclebucket <trash bucket> --storeFileHistory --type AE

Retrieve the recall folder definition's ID with searchctl archivedfolders list, then reference it on the upload folder definition with --archive-engine-recall-folder <recall folder ID>. The two folder definitions must be created in this order (recall first, then upload), since the upload definition needs the recall folder's ID to complete the link.

End-User Archive and Recall Workflow​

  1. Archiving: the end user logs in to the Archive Engine GUI as an Active Directory user, selects the Archive Engine icon, and browses to a path authorized above (eligible folders show an "up to cloud" icon). Clicking the archive button submits the job; the folder shows as staged until the archive job completes, after which the archived data appears on the right-hand panel when browsing that folder.
  2. Recall: the end user browses the file system on the left pane and sees archived data for the selected folder on the right pane. Clicking the recall (cloud-down) icon stages the data for recall; a scheduled recall job then restores it to its original path.
note

If a recalled file's name already exists in the file system, the incoming file is renamed to avoid a collision, using the pattern <original file name>-smart-archiver-recall-collision-<year>-<month>-<day>-<time>.

Data Lifecycle Tracking​

The --storeFileHistory flag tags each object with the date and time it moved between the file system and object storage, providing traceability of a file's full archive/recall history directly within the object's metadata, without needing a separate tracking system.

AWS Deep Archive Tier Transparency​

Deep Archive is AWS S3's lowest-cost, longest-retrieval-time storage tier, normally requiring specialized APIs and a multi-hour wait to make an object accessible again. Archive Engine's transparency feature manages this entire recall process automatically, so end users can request their data back through the normal Archive Engine recall workflow without being aware of the underlying Deep Archive mechanics.

Prerequisites:

  • Golden Copy release 1.1.13 or later.
  • Golden Copy Archive Engine Archived TB subscription license.
  • An AWS S3 endpoint (this feature is AWS-specific).

How it works:

  • Configure the upload folder definition with --tier DEEP_ARCHIVE; no other Archive Engine configuration changes are required.
  • Archive Engine automatically issues the AWS Deep Archive recall request and runs a background job to monitor recall status, until the requested data becomes available.
  • Because AWS's own Deep Archive recall makes data temporarily available for a limited window, Archive Engine performs a server-side copy of the recalled data into a separate recall bucket, where it can be retained indefinitely or purged via a lifecycle policy.

Configuring the temporary retention window:

By default, restored objects remain accessible for 7 days (AWS allows a range of 4–14 days). To change this, on Golden Copy node 1:

export RESTORE_EXPIRATION_DAYS=<number of days>

Set this in /opt/superna/eca/eca-env-common.conf, restart the cluster, and adjust the corresponding container override (docker-compose.overrides.yml) as directed by Superna Support if further tuning is needed.

While data is in flight from Deep Archive, end users see a visual indicator (a star icon) on the requested file/folder in the Archive Engine portal until it becomes available.

Policy-Based Data Archive​

Policy-based archiving lets administrators target one or more file system paths with an age-based scan (last-modified date older than a specified number of months/years) and automatically move matching data to S3, without requiring end users to manually select data for archive.

Prerequisites:

  • Golden Copy with the Advanced license key.
  • Archive Engine add-on license.
  • Administrative access to the target S3 storage.

Workflow:

  1. Create an upload folder definition (staging path, typically /ifs/tocloud) and a recall folder definition, linked together as in the self-service configuration above. AWS Deep Archive tiering can be combined with policy-based archiving for maximum storage cost savings.
  2. Create one or more archive policies that target specific paths with an age-based rule, attached to the upload folder definition.

Configure a Policy​

searchctl archivedfolders policies add --folder <upload folder ID> --path /ifs/olddata --older-than <months[-years]>

The --older-than value subtracts the given number of months and years from the job's start date to compute the cutoff date. For example, --older-than 3-1 targets files older than 3 months and 1 year at job start; values of 12 or more months automatically carry over into years.

Additional policy options:

  • --remove-empty-folders true — enables a background task that deletes empty folders left behind once all files in them are archived. Contact Support before enabling; the default is false, since many customers prefer to retain the visible folder structure.
  • --pause true|false — pauses archiving for a specific policy without deleting it. Default is false.

Multiple policies with different paths and age criteria can be attached to the same folder definition.

Test Before Production Use​

warning

Never run a policy-based archive job in production without testing it first.

  1. Create a test policy against a disposable test path with a short --older-than window (for example, 1 month).
  2. Copy a small tree of test data into that path, and (as root on the cluster) use touch -m -d to backdate the files' last-modified timestamps so they fall within the policy's matching window.
  3. Run the archive job against the test policy only (searchctl archivedfolders archive --id <folder ID> --follow) and confirm the reported policy file/folder counts and error rates look correct.
  4. Log in as an AD user with SMB read permission at or below the test path and confirm the recall option is available and functions correctly.

Switch to Production​

  1. Remove the test policy: searchctl archivedfolders policies remove --id <policy ID>.
  2. Add the intended production path policies to the upload folder definition.
  3. Run the archive job against the folder definition to begin production archiving.

Dry-Run Mode​

Dry-run jobs walk through all the normal steps of a policy-based archive job and collect the same statistics — total bytes scanned, files/bytes matching the policy, and percentage matched — without copying, moving, or deleting any data. This lets administrators or prospective users estimate the impact of a policy before committing to it.

  • Easy-start dry run (no existing folder definition required): searchctl archivedfolders archive --path <policy path> --isilon <cluster name> --older-than <rule> --dry-run creates a temporary folder definition for the duration of the job (using a placeholder blackhole cloud type) and removes it afterward. This mode does not require Archive Engine, Pipeline, or Advanced licensing, since no data actually moves.
  • Dry run on an existing folder definition: requires policies configured with --dry-run true. Running searchctl archivedfolders archive --id <folder ID> --dry-run processes only the dry-run-flagged policies and leaves the folder definition in place afterward.
  • Dry-run job statistics are tagged with job type DRY and can be viewed with searchctl stats view, or exported like any other job with searchctl jobs export (exported events are synthesized for the dry run, since no real archive events occur).

Error Handling​

When archiving very large file counts (for example, over a million files), a small percentage of transient errors is expected — commonly a PowerScale API timeout (too many requests) or an expired token session. These conditions surface as INFO-level messages distinguishing a timeout from a connection/session error, and are typically resolved by a recurring incremental or policy scan picking up any files missed on a prior run.

See Also​

  • Configuration Guide — General folder definition, scheduling, and monitoring reference.
  • Use Cases — Where self-service archiving fits among Data Orchestration's use cases.
  • Compliance Guide — Immutability and auditability posture.