Archive Engine Guide for Data Orchestration for Dell
Introduction
Archive Engine is the licensed Data Orchestration capability that provides end-user self-service archiving and recall, administrator-defined age-based archive policies, and AWS S3 Deep Archive tier transparency. This guide covers configuring Archive Engine folder definitions, the end-user archive/recall workflow, data lifecycle tracking, Deep Archive tier transparency, policy-based archiving, and dry-run testing. For licensing and conceptual background, see Concept and Pre-Requisites.
Prerequisites
- Archive Engine subscription license.
- Two S3 buckets: one to hold archived data, and one to stage recalled data (a third, optional trash bucket can hold deleted staging data with a lifecycle policy).
- The Pipeline NFS mount for data recall configured on Golden Copy (see Installation Guide), with the mount path matching the
--recall-sourcepathvalue used below.
Configuring Self-Service Archiving
An Archive Engine configuration is built from two folder definitions — one for on-premises-to-cloud archiving, and one for cloud-to-on-premises recall — linked together.
Prepare the Recall Data Location
Set the cluster's recall data location, used to uniquely store all data arriving from the cloud for pipeline-based workflows:
searchctl isilons modify --name <cluster name> --recall-sourcepath /ifs/fromcloud
Add the Upload (On-Premises to Cloud) Folder Definition
The staging path for data pending archive is typically /ifs/tocloud. Key flags for this folder:
--archive-engine-sourcepaths— authorizes one or more paths (comma-separated) that end users can select for self-service archiving, for example/ifs/projects,/ifs/home.--archive-engine-type upload— marks this definition as the on-premises-to-cloud direction.--full-archive-schedule— a cron schedule for when staged data is moved to the cloud (a daily off-peak schedule is recommended).--delete-from-source— deletes data from the staging path once it has been copied (required for a move, rather than backup, workflow).--storeFileHistory— encodes each file's archive/recall history as a custom S3 object property.--type AE— marks the folder definition as an Archive Engine configuration.--tier(optional) — for AWS or Azure targets; see AWS Deep Archive Tier Transparency below for the deep archive option.
searchctl archivedfolders add --isilon <cluster name> --folder /ifs/tocloud --accesskey <access key> --secretkey <secret key> --endpoint <endpoint> --region <region> --bucket <archive bucket> --cloudtype aws --archive-engine-sourcepaths /ifs/projects --full-archive-schedule "0 0 * * *" --delete-from-source --archive-engine-type upload --storeFileHistory --type AE
Add the Recall (Cloud to On-Premises) Folder Definition
Create the recall staging path on the cluster (as root), for example mkdir -p /ifs/fromcloud/<staging-bucket-name>, then add the recall folder definition:
--source-path <cluster name>/<archived path>— authorizes which archived data is presented to end users for recall; all data under this path becomes visible for recall.--bucket— must match the archive bucket used in the upload folder definition above, tying the two definitions together.--recall-schedule— cron schedule for how often the staging bucket is scanned before recalling data on-premises.--recall-from-sourcepath,--archive-engine-type recall,--trash-after-recall,--recyclebucket,--storeFileHistory,--type AE— as with the upload definition, plus--trash-after-recallto delete staging data once recalled, and--recyclebucketto move it to a lifecycle-managed trash bucket instead.
searchctl archivedfolders add --isilon <cluster name> --folder /ifs/fromcloud/<staging-bucket-name> --accesskey <access key> --secretkey <secret key> --endpoint <endpoint> --region <region> --bucket <staging bucket> --cloudtype aws --source-path <cluster name>/ifs/projects --recall-schedule "*/5 * * * *" --recall-from-sourcepath --archive-engine-type recall --trash-after-recall --recyclebucket <trash bucket> --storeFileHistory --type AE
Link the Two Folder Definitions
Retrieve the recall folder definition's ID with searchctl archivedfolders list, then reference it on the upload folder definition with --archive-engine-recall-folder <recall folder ID>. The two folder definitions must be created in this order (recall first, then upload), since the upload definition needs the recall folder's ID to complete the link.