Configuration Replication

Introduction

Configuration replication reads the data access configuration of a source cluster and uses pre-established replication relationships to mirror the data access configuration to a target. This includes metadata that controls access to data, as well as failback and replication mappings, schedules, and properties.

For detailed information on what is specifically replicated, please refer to the sections for different storage vendors below.

Prerequisites

Before starting configuration replication, ensure you have:

1. A Directory Structure on Target Cluster:

   • On PowerScale OneFS, one or more syncIQ jobs must exist and have been successfully run to ensure that the data on the target cluster is a mirror of the source.

2. Access Zones on Target Cluster:

   • Access zones corresponding to the shares, exports, and quotas must already exist on the target PowerScale OneFS cluster. If needed, zone replication can be run to create these access zones on the target cluster.

Replicating Shares for Security on Target Clusters

Configuration replication on the DR cluster ensures that configurations such as shares, aliases, exports, and quotas are created on the target cluster and ready to be used in case of failover.

Use Cases

• DFS mode for SMB failover: Configurations are renamed and hidden on failover on the DR cluster.
• Access Zone failover: Configurations are synced and hidden on the DR cluster.

Hidden

To clarify "hidden": If you suffix a share name with the dollar sign character ($), Windows will not display the share, effectively hiding it from users.

Configuration

1. Enable a policy for DFS mode:

   

   • Navigate to the Jobs icon and enable the policy for DFS mode.
   • This configuration will also work for Access Zone failover.
   warning

   Selecting DFS mode will rename SMB shares in target clusters configured on any selected SyncIQ policy, Click OK to continue

   Click OK to proceed or cancel to return.

2. Change the DFS configuration suffix:

   • Edit the /opt/superna/sca/data/system.xml file.

   • Change the <dfssharesuffix> tag to include $ as shown below:

     <dfssharesuffix>$</dfssharesuffix>
   note

   On an existing installation, the old DFS renamed share will need to be manually deleted.

Enable/Disable Jobs

This section will walk you through the process of enabling or disabling jobs, including bulk actions for managing multiple jobs at once.

1. Open the Eyeglass Main Menu

   From the main dashboard, locate and click the Jobs icon, which is located in the top-middle section of the screen and represented by a checklist icon.

2. View the Jobs List

   Once the Jobs window is open, you'll see a list of available jobs. You can view details such as the job name, policy, type, last run date, and the current state.

3. Select Jobs to Enable or Disable

   Check the box next to the jobs you want to enable or disable.

   info

   You can select more than one job at the same time by checking multiple boxes.

4. Use the Bulk Action to Enable or Disable Jobs
   After selecting the jobs, look at the bottom right of the window for the dropdown button labeled Select a bulk action. Click on this button and choose either Enable or Disable based on the action you want to perform.

5. Effect of Enabling the Job

   info

   On the next Configuration Replication cycle, the enabled Job will be run.

   This process allows you to manage multiple jobs efficiently by enabling or disabling them in bulk.

What Properties are Replicated by Eyeglass?

PowerScale OneFS

PowerScale OneFS

Configuration replication for PowerScale OneFS’s cluster covers the following elements:

• SMB Shares
• NFS Exports
• Failover Support Policies
• Access Zones
• Quotas
• Snapshot Schedules
• User Access Controls

VAST

VAST

Configuration replication for VAST covers the following elements:

• Views
• ViewPolicies
• S3 Lifecycle Rules
• QoS Policies

Qumulo

Qumulo

Configuration replication for Qumulo covers the following elements:

• SMB Shares
• NFS Exports
• Snapshot Policies

How does Eyeglass Determine Uniqueness?

Uniqueness and Equality between Shares

The combination of share name and zone name is the unique value used to determine if one share equals another. This applies to records on the same cluster as well as when comparing shares between two PowerScale OneFS clusters.

Uniqueness and Equality between Exports

Uniqueness is determined by the client list and the replication path between Access Zones on replicating clusters. The same path, but with a different client list (all client lists), will be treated as different exports to sync within a given Access Zone

Uniqueness and Equality between Zones

Zone name is used to determine if one zone equals another. This applies to records on the same cluster as well as when comparing zones between two PowerScale OneFS clusters.

Further Details: Configuration

Shares

All configuration properties for shares are replicated with the following exceptions:

• Local users & their permissions: These are not replicated.
• Filesystem users & their permissions: Filesystem users are replicated.
  • To ensure that Filesystem users are replicated, make sure File:System is added to the "Authentication Providers" list on the target cluster’s Access Zone.
• SID replication: SIDs for local and filesystem users are replicated but not mapped to users on the target. AD at the target cluster must be set up to resolve SIDs using the same AD forest.

note

Shares with variable expansion, such as %u in share names, sync correctly to the target cluster.

note

Locally created users or groups on PowerScale OneFS generate unique UIDs or GIDs per cluster. As a result, the same username on two clusters will have different UIDs and GIDs, which is standard Linux behavior. Syncing local users or groups has no value because userX on cluster A will not be the same as userX on cluster B and will not have access to data secured to userX on cluster A. Best practice: Use LDAP or Kerberos for user security for exports or Active Directory for a unified security database.

Exports

All configuration information for exports is replicated to the target with the following exceptions:

• Export ID: Source and target exports will not have the same export ID.
• FQDN clients: Clients listed by FQDN must be resolvable by the target cluster. If this is not possible, Eyeglass full sync mode must be used to override API validation and force the creation of exports. See ilgs CLI command documentation for more details.

Restriction

• Dual path exports: Both paths must fall under the same SyncIQ policy.

Quotas

All configuration information for quotas is replicated to the target with the following exceptions:

• Usage Accounting - include Snapshot Data: Best practice is not to replicate this setting with SyncIQ.

• Usage Accounting - include Data-Protection Overhead: Best practice is not to replicate this setting with SyncIQ.

• Quota definitions: Quotas that use local or filesystem users or groups are unique across clusters, even if they share the same name. Therefore, well-known accounts and local/filesystem users should not be used with the Quota Replication feature.

  Best practice: Use Active Directory for user and group quotas and custom or automatic quota replication jobs.

  note

  Quota replication is not run on a schedule but only during failover

Zones

The following configuration properties are replicated for Zones (excluding the System zone):

• Access Zone Name
• Zone base directory
• Authentication Providers: Name only.
• User Mapping Rules: Applicable to PowerScale OneFS 7.1.x.x only.

note

Authentication providers or local providers must exist on the target to resolve SIDs used in share permissions and filesystem ACLs.

Snapshots

Snapshot schedule properties are replicated only for paths that match SyncIQ policies cluster-wide, with the following exceptions:

• next_run: Auto-populated by PowerScale OneFS.
• next_snapshot: Auto-populated by PowerScale OneFS.

Purpose:

• Sync Snapshot schedules found on SyncIQ paths.
• Syncs schedules as per SyncIQ policy paths.
• Reads dedupe path settings for SyncIQ Policies and applies them to the target cluster.
• Can be disabled independently of snapshots using igls command.

Job Creation:

• Automatically built, but user disabled by default and must be enabled.

Schedule:

• FILESYSTEM Jobs run automatically on a 5-minute schedule.

Initial State:

• Disabled and must be enabled to execute at the defined configuration replication schedule.

info

Note: Snapshot schedule config sync does not overwrite existing Snapshot schedules on the target cluster with different Snapshot schedule names.

Dedupe

You can configure deduplication settings on both the source and DR (Disaster Recovery) clusters. This allows the dedupe operation to process data on the DR cluster, achieving the same disk space savings. By doing this, you ensure that after a failover, the disk usage on the DR cluster matches that of the source cluster. This setup avoids any delays in starting dedupe operations post-failover, allowing the target cluster to maintain normal operational levels without waiting for deduplication jobs to run.

Dedupe properties are replicated only for paths that match SyncIQ policies cluster-wide, including:

• Dedupe path for job processing.
• Dedupe assessment paths.

How it Works

1. You must add file system paths and assessment paths to the source cluster, and they must match a SyncIQ policy path that will sync to the target cluster.
2. Eyeglass manages the SyncIQ policies.
3. The dedupe scheduled job is not synced and must be set up or changed manually on the target cluster.
4. Ensure that licenses for dedupe exist on the target cluster.

How to Enable

1. Dedupe paths are automatically synced with normal configuration replication, so no setup is required.
2. Dedupe paths added to the target cluster are only those that match a SyncIQ policy path or below in the file system.
3. Dedupe settings for paths are synced once for all policies, not per policy.
4. You must verify that the paths are added to the target cluster.

SyncIQ Policies

• On Failover: Only the SyncIQ schedule is re-applied to newly created mirror policies.
• File pattern filters: These are not synced on failover. File pattern filters can result in unprotected data during failover and failback. Failover and failback workflows require customer testing. All file filter use cases are untested and are not supported or documented.

Troubleshooting

AEC_NOT_FOUND: "Path 'X/Y/Z' Not Found: No Such File Or Directory" for Eyeglass Configuration Replication Job

The Eyeglass Configuration Replication Job fails with error AEC_NOT_FOUND: "Path 'x/y/z' not found: No such file or directory" (Alarm Code: SCA0004).

This error occurs when the Eyeglass Configuration Replication Job runs and attempts to replicate a share, export, or quota, but the associated directory does not exist on the target.

Resolution

The following are common causes of the AEC_NOT_FOUND error and their resolutions:

• SyncIQ Policy has not been run:
  The SyncIQ policy associated with the path has not been run, and therefore, the path does not exist on the target cluster.

  Resolution: Ensure the SyncIQ policy has recently run on the cluster.

• SMB Share or NFS Export path mismatch:
  The SMB Share or NFS Export path points to a path that does not exist on the source cluster filesystem, or the share path does not match the path on the filesystem exactly (case-sensitive).

  Resolution: Review the SMB Share or NFS Export path on the source cluster, copy it to the clipboard, and SSH to the source cluster. Run the command:
  cd <pasted path>
  If the cd command fails, either the path does not exist or there is a case-sensitivity mismatch.

• SyncIQ Policy path exclusion or inclusion:
  The SyncIQ policy has paths in the included or excluded list, and the missing path is protected by the policy but is not listed.

  Resolution: Review the policy configuration and verify if the excludes or includes are configured correctly. Note that using these options is not supported by Dell EMC for failover/failback.

• SMB Share path with trailing slash:
  The SMB Share path has a trailing "/" at the end, for example, /ifs/home/.

  Resolution: Remove the trailing "/" from the share path to resolve the error.

Once resolved the next Configuration Replication job will succeed and the alarm will be cleared.

Other Possible Causes for Missing Path on Target Cluster

• Path is on the SyncIQ Policy Excluded list:
  The path is explicitly excluded by the SyncIQ policy.

AEC_FORBIDDEN Error for Eyeglass Configuration Replication Job

The Eyeglass Configuration Replication Job fails with the error: AEC_FORBIDDEN....

The Isilon system is provisioned in Eyeglass with a user who does not have the minimum required privileges.

Troubleshooting Steps

1. Verify User Permissions:
   Cross-reference the permissions of the Isilon PowerScale OneFS user used in Eyeglass provisioning with the Minimum Required Privileges documentation.

2. Update User Privileges:
   If the PowerScale OneFS user does not have the required privileges, update the user privileges in PowerScale OneFS. The next Eyeglass Configuration Replication Job will use these updated privileges.

3. Verify Privileges via Command Line:
   If the PowerScale OneFS user already has the required privileges, verify the user privileges directly from the Isilon command line to ensure they are set as required.

AEC_NOT_FOUND: Zone <Zone Name> Not Found for Eyeglass Configuration Replication Job

The Eyeglass Configuration Replication Job fails with the error: AEC_NOT_FOUND Zone <Zone Name> not found. (Alarm Code: SCA0004)

This error occurs when the Eyeglass Configuration Replication Job attempts to replicate a share or export, but the associated Zone does not exist on the target cluster.

Resolution

To resolve the AEC_NOT_FOUND: Zone <Zone Name> error, follow these steps:

1. Review SyncIQ Policy Information:

   • Check the SyncIQ policy associated with the Configuration Replication Job that triggered the error.
   • Determine the SyncIQ policy source path on the source cluster.
   • Identify the Access Zone on the source cluster where the SyncIQ policy source path is located.
   • Review the SyncIQ policy target path on the target cluster.
   • Determine the Access Zone on the target cluster where the SyncIQ policy target path is replicating.

2. Verify Access Zone Consistency:
   Ensure that the Access Zone names on both the source and target clusters are the same. If they differ, you will encounter this error.

3. Confirm Zone Existence:
   Ensure that all Zones associated with the shares and exports exist on the target cluster.

4. Rerun the Configuration Replication Job:
   Once the Zones exist on the target cluster, the next configuration replication job will succeed, and the alarm will be cleared.

AEC_EXCEPTION: Bad Hostname for Eyeglass Configuration Replication Job

The Eyeglass Configuration Replication Job fails with the error: AEC_EXCEPTION message bad hostname 'hostname'. (Alarm Code: SCA0004)

This error occurs because Eyeglass cannot replicate the NFS Export to the target cluster due to a hostname listed in the NFS Export "Clients" list not resolving on the target cluster.

Best Practice

To avoid this issue, ensure that the DR cluster can resolve hostnames. If hostnames cannot be resolved, the data will not be mountable after a failover.

One possible cause is that the NFS Exports "Clients" field has a host name entry that cannot be resolved on replication of the Export.

Resolution

To resolve the AEC_EXCEPTION: Bad Hostname error, follow these steps:

1. Verify Hostname Entry:
   Ensure the "Clients" field on the NFS Export on the source cluster contains valid hostname entries.

2. Check Hostname Resolution:
   Run the command nslookup <hostname> to determine if the hostname resolves correctly.

3. Remove Unnecessary Hostnames:
   If the hostname is no longer required, remove it from the "Clients" list.

4. Use Eyeglass CLI:
   If the hostname cannot be resolved or removed, you can use the following Eyeglass CLI command to ignore unresolved hosts:
   igls admin ignoreunresolvablehosts

5. Remove Export on Target Cluster:
   If the issue persists after ignoring unresolved hosts, remove the export from the target cluster and allow Eyeglass to recreate it during the next Configuration Replication Job.

AEC_NOT_FOUND: Zone 'x' Not Found for Eyeglass Configuration Replication Job

This error occurs when Eyeglass attempts to replicate a share or export, and the associated Zone does not exist on the target cluster.

Possible Cause

The Zone associated with the share or export on the source cluster does not exist on the target cluster with the exact same name.

Solution

To resolve the AEC_NOT_FOUND: Zone 'x' Not Found error, follow these steps:

1. Ensure Directory Exists on Target:
   The directory associated with the share, export, or quota being replicated must exist on the target cluster.

   • Run the SyncIQ Policy to create the necessary paths on the target.

2. Verify SyncIQ Policy Includes/Excludes:
   For SyncIQ policies with included or excluded paths, manually verify that the error relates to excluded paths and confirm that the job has succeeded for included paths.

3. Ensure Matching Zone Exists:
   The Zone associated with the share or export being replicated must exist on the target cluster with the exact same name.

See Also

• Advanced DR Robot Configuration Guide

• Eyeglass Jobs