Storage Failover Steps

Introduction

The storage layer failover process is a critical part of disaster recovery (DR). A complete DR plan should also address application shutdown and restart procedures to ensure seamless failover. The storage layer provides the foundation for all higher-level failover processes, and Eyeglass streamlines this step while enabling error detection during failover.

For complex or application-layer failover scenarios, Superna Professional Services offers support through Proof of Concept (POC) engagements, recommendations, and assessments. Examples include:

• VMware SRM with externally mounted storage for virtual machines.
• Oracle RAC Data Guard with file system dependencies for applications.

After identifying the Failover Type suitable for your environment, the next section describes the high-level steps for each type.

Failover Types and High-Level Steps

Ordered Steps for Non-DFS and DFS Type

This step-by-step process details failover actions for SyncIQ Policy or Microsoft DFS Type. Each item specifies its purpose, describes the action performed, and explains how the action is initiated based on the selected failover type.

1. Ensure There Is No Live Access to Data (Source)

   Verify that no active operations are accessing the data to prevent inconsistencies during failover.
   Perform a manual check for open files. If open files are found, determine whether to wait for their closure or proceed with the failover.

   • How Action Is Initiated:
     • SyncIQ Type: Manual
     • DFS Type: Manual
   note

   The Failover Wizard's Data Integrity failover option (version 2.0 or later) blocks IO to SMB shares before failover starts.

   tip

   Disable SMB and NFS protocols on the source cluster. This is a cluster-wide operation designed to eliminate data loss.

   1. Enable Data Integrity Failover (SMB Only)
      Applies the "Deny Everyone" setting to SMB shares before the failover process begins.

      • How Action Is Initiated:
      • SyncIQ Type: Automated by Eyeglass
      • DFS Type: Automated by Eyeglass

   2. Cache Schedule for SyncIQ Policies Being Failed Over
      Retrieves the schedule for SyncIQ policies being failed over on PowerScale OneFS and sets policies to manual to ensure they do not run again during failover.

      • How Action Is Initiated:
      • SyncIQ Type: Automated by Eyeglass
      • DFS Type: Automated by Eyeglass

2. Begin Failover With the Failover Wizard (Eyeglass)
   Initiates the failover process from Eyeglass.

   • How Action Is Initiated:
     • SyncIQ Type: Manual or Eyeglass REST API
     • DFS Type: Manual or Eyeglass REST API

3. Validation of Failover Job (Eyeglass)
   Verifies all warnings before submitting the failover job to ensure successful execution.

   • How Action Is Initiated:
     • SyncIQ Type: Automated by Eyeglass
     • DFS Type: Automated by Eyeglass

   1. Validation - Block on Warning Enabled
      Prevents the failover process from continuing if warnings, such as quota scan requirements, are detected.

      • How Action Is Initiated:
        • SyncIQ Type: Automated by Eyeglass
        • DFS Type: Automated by Eyeglass

   2. Set Eyeglass Config Jobs to User-Disabled
      Sets Eyeglass configuration jobs to a user-disabled state to prevent failed steps from running. Jobs can only run if the user re-enables them post-failover.

      • How Action Is Initiated:

        • SyncIQ Type: Automated by Eyeglass
        • DFS Type: Automated by Eyeglass

4. Synchronize Data (Run SyncIQ Policies)
   Runs all PowerScale OneFS SyncIQ policy jobs related to the Access Zone being failed over.

   • How Action Is Initiated:
     • SyncIQ Type: Automated by Eyeglass
     • DFS Type: Automated by Eyeglass

5. Synchronize Configuration
   Executes Eyeglass configuration replication, including shares, export aliases, snapshot schedules, and dedupe paths.

   • How Action Is Initiated:
     • SyncIQ Type: Automated by Eyeglass (configuration exists on source and target)
     • DFS Type: Automated by Eyeglass

6. Renaming Shares for DFS Type to Redirect DFS Clients For DFS failover, shares are renamed on the source and target clusters to redirect clients with dual DFS target paths to the target cluster.

   • How Action Is Initiated:

     • SyncIQ Type: Not Applicable
     • DFS Type: Automated by Eyeglass
     info

     DFS Type: Special handling renames shares on the source and target so only one DFS target UNC is reachable and active for DFS clients to switch over.

7. Provide Write Access to Data on Target Allow writes to SyncIQ policies related to failover.

   • How Action Is Initiated:
     • SyncIQ Type: Automated by Eyeglass
     • DFS Type: Automated by Eyeglass

8. Disable SyncIQ on Source and Activate on Target

   Prepare for failover by creating a mirror policy on the target. Disable the source cluster policy and enable the target cluster policy on PowerScale OneFS.

   • How Action Is Initiated:
     • SyncIQ Type: Automated by Eyeglass
     • DFS Type: Automated by Eyeglass

9. Reset SyncIQ Schedule on Target Mirror Policy

   Apply the schedule to the target mirror policy using the schedule retrieved from Step 1.

   • How Action Is Initiated:
     • SyncIQ Type: Automated by Eyeglass
     • DFS Type: Automated by Eyeglass

10. Failover Quotas (Optional) The Eyeglass Failover Wizard automatically fails over quotas by running quota jobs related to the SyncIQ policies being failed over.

    • How Action Is Initiated:

      • SyncIQ Type: Automated by Eyeglass

        info

        Quotas are deleted on the source cluster and created on the target cluster.

      • DFS Type: Automated by Eyeglass

        info

        Quotas are deleted on the source cluster and created on the target cluster so that post-failover quotas are applied.

11. Remove Quotas on Directories That Are Targets of SyncIQ Eyeglass deletes all quotas on the source cluster for the specified policies.

    • How Action Is Initiated:
      • SyncIQ Type: Automated by Eyeglass
      • DFS Type: Automated by Eyeglass

    1. Run Mirror Policy Executes the mirror policy to resync data in the reference direction.

       • How Action Is Initiated:
         • SyncIQ Type: Automated by Eyeglass
         • DFS Type: Automated by Eyeglass

    2. Set Eyeglass Config Jobs to Enabled
       Enables configuration synchronization only if the Resync Prep step completes successfully for the policy.

       • How Action Is Initiated:
         • SyncIQ Type: Automated by Eyeglass
         • DFS Type: Automated by Eyeglass

12. Change SmartConnect Zone on Source
    Rename SmartConnect zones and aliases on the source cluster to prevent names from being resolved by clients.

    • How Action Is Initiated:
      • SyncIQ Type: Manual
      • DFS Type: Not Required (source and destination clusters can use existing SmartConnect zones)

13. Avoid SPN Collision
    Sync SPNs in all Active Directory (AD) providers to match the current SmartConnect zone names and aliases on the source cluster.

    • How Action Is Initiated:
      • SyncIQ Type: Manual (deletes SmartConnect SPN from the source cluster machine account)
      • DFS Type: Not Applicable (DFS SPNs are not changed during failover)

14. Move SmartConnect Zone to Target
    Add source SmartConnect zones as aliases on the target cluster.

    • How Action Is Initiated:
      • SyncIQ Type: Manual
      • DFS Type: Not Required (source and destination clusters can use existing SmartConnect zones)

15. Create SPNs for Kerberos Authentication on Target for SMB Shares
    Sync SPNs in all Active Directory (AD) providers to match current SmartConnect zone names and aliases on the target cluster.

    • How Action Is Initiated:
      • SyncIQ Type: Manual (adds new SmartConnect alias SPNs to the target cluster machine account)
      • DFS Type: Not Applicable (DFS SPNs are not changed or registered to cluster machine accounts)

16. Repoint DNS to the Target Cluster IP Address
    Update DNS delegations for all SmartConnect zones that are members of the access zone.

    • How Action Is Initiated:
      • SyncIQ Type: Manual
      • DFS Type: Not Applicable (no updates are needed as DFS resolution has not changed in DNS, only the target UNC with an active share)

17. Refresh Session to Pick Up DNS Change
    Remount the SMB/NFS shares.

    • How Action Is Initiated:
      • SyncIQ Type: Manual on clients
      • DFS Type: Automatic (Windows 7 or later with DFS support)

SyncIQ Policy Failover Procedure with DR Assistant

This procedure is for application failover when host-side automation scripts are required. It is recommended for NFS failover when no SPN or SmartConnect zone automation is required.

It offers flexibility to design a failover solution customized for an application use case, while using DFS Type or Access Zone failover for standardized failover workflows.

This failover type can be used with SMB. However, manual steps are required for SPN and SmartConnect zones that are not required with DFS Type or Access Zone failover.

Recommended use cases:

1. NFS failover with post-script host automation.
2. Application failover with a script engine to assist with application automation steps.
3. Failover without DNS updates, using SmartConnect zone names and unmount and remount automation.

Failover Steps

1. Initiate Failover of the SyncIQ policy with Eyeglass.

2. Secondary cluster: create the SmartConnect zone alias name (manual; automated with Access Zone failover).

   Command:

   isi networks modify pool <subnet>:<zone-name> --add-zone-aliases <zone-alias-name>

   Example:

   isi network modify pool subnet0:zone01-pool --add-zone-aliases primary.ad1.test

3. Secondary cluster: ignore the missing SPN (manual).

   Command:

   isi auth ads spn check --domain <domain-name>

   When this command is run on the secondary cluster, a missing SPN is reported. If the scenario uses the local provider as the authentication provider, this missing SPN message can be ignored.

4. Update name resolution to the new IP address (automated by Eyeglass with the script engine).

   The Post Failover Eyeglass script can be used to update name resolution.

5. Re-mount the NFS mount (automated by Eyeglass with the script engine).

   Unmount and remount the NFS mounting folder on the client to refresh the connection to the NFS export (accessed through the secondary cluster).

   Command:

   umount -fl /mnt/z01-nfs01

   Then, for NFSv3:

   mount -t nfs -o vers=3 primary.ad1.test:/ifs/data/zone01/z01-nfs01 /mnt/z01-nfs01

   For NFSv4:

   mount -t nfs -o vers=4 primary.ad1.test:/ifs/data/zone01/z01-nfs01 /mnt/z01-nfs01

   You might need to unmount the export with the force and lazy options to resolve an NFS stale issue. For NFSv4, you might also need to kill the process that was accessing the export mount before the failover.

Failback Steps

1. Initiate Failback of the SyncIQ policy with Eyeglass.

   • Automated by Eyeglass: on the secondary cluster, replicate data to the primary cluster with the mirror policies.
   • Automated by Eyeglass: on the primary cluster, allow writes to the target directories of the mirror policies.
   • Automated by Eyeglass: on the secondary cluster, complete the failback process.

2. Secondary cluster: delete the SmartConnect zone alias name (manual; automated with Access Zone failover).

   Command:

   isi networks modify pool <subnet>:<zone-name> --remove-zone-aliases <zone-alias-name>

   Example:

   isi network modify pool subnet0:zone01-pool --remove-zone-aliases primary.ad1.test

3. Update name resolution to the IP address (automated by Eyeglass with the script engine).

   The Post Failover Eyeglass script can be used to update name resolution.

4. Re-mount the NFS mount (automated by Eyeglass with the script engine).

   Unmount and remount the NFS mounting folder on the client to refresh the connection back to the primary cluster's NFS export.

   Command:

   umount /mnt/z01-nfs01

   Then, for NFSv3:

   mount -t nfs -o vers=3 primary.ad1.test:/ifs/data/zone01/z01-nfs01 /mnt/z01-nfs01

   For NFSv4:

   mount -t nfs -o vers=4 primary.ad1.test:/ifs/data/zone01/z01-nfs01 /mnt/z01-nfs01

Ordered Steps for Access Zone or IP Pool Failover

This section outlines the step-by-step process for failover involving Access Zones or IP Pools. Each step describes the action performed, its purpose (if necessary), and how the action is initiated.

1. Ensure That There Is No Live Access to Data (Source)
   Perform a manual check for open files. If open files are found, decide whether to wait for them to close or proceed with the failover.

   tip

   Disable SMB and NFS protocols on the source cluster before failover. This is a cluster-wide operation required to prevent data loss.

   • How Action Is Initiated:
     • Access Zone: Manual

   1. Enable Data Integrity Failover (SMB Only)

      Applies the "Deny Everyone" setting to SMB shares before the failover process begins.

      • How Action Is Initiated:
        • Access Zone: Automated by Eyeglass

   2. Cache Schedule for SyncIQ Policies

      Retrieves the schedule associated with SyncIQ policies being failed over on PowerScale OneFS. Sets policies to manual to ensure they do not run again during failover.

      • How Action Is Initiated:
        • Access Zone: Automated by Eyeglass

2. Begin Failover With the Failover Wizard (Eyeglass)
   Initiate the failover process from Eyeglass.

   • How Action Is Initiated:
     • Access Zone: Manual or Eyeglass REST API

3. Validation of Failover Job (Eyeglass)
   Verify all warnings before submitting the failover job.

   • How Action Is Initiated:
     • Access Zone: Automated by Eyeglass

   1. Validation - Block on Warning Enabled
      Prevents the failover process from continuing if warnings are detected, such as quota scan requirements.

      • How Action Is Initiated:
        • Access Zone: Automated by Eyeglass

   2. Set Eyeglass Config Jobs to User-Disabled
      Configures Eyeglass jobs to a user-disabled state to prevent failed steps from running unless re-enabled manually after the failover.

      • How Action Is Initiated:
        • Access Zone: Automated by Eyeglass

4. Synchronize Data (Run SyncIQ Policies)
   Run all PowerScale OneFS SyncIQ policy jobs related to the Access Zone being failed over.

   • How Action Is Initiated:

     • Access Zone: Automated by Eyeglass

       info

       All policies in the Access Zone.

5. Synchronize Configuration
   Run Eyeglass configuration replication, including shares, export aliases, snapshot schedules, and dedupe paths.

   • How Action Is Initiated:

     • Access Zone: Automated by Eyeglass

       info

       Based on matching Access Zone base path.

6. Redirect DFS Data (If Applicable)
   If DFS is configured, redirect rename steps would be executed at this point.

   note

   It is possible to integrate DFS-protected data inside an Access Zone failover to protect shares, exports, and DFS data as part of the Access Zone failover process.

7. Change SmartConnect Zone on Source
   Rename SmartConnect zones and aliases on the source cluster to prevent client resolution. Dual delegation eliminates the need for DNS updates.

   • How Action Is Initiated:

     • Access Zone: Automated by Eyeglass

       info

       Based on matching Access Zone base path.

8. Avoid SPN Collision
   Sync SPNs in all Active Directory (AD) providers to match current SmartConnect zone names and aliases. The proxy is configured through the target cluster.

   • How Action Is Initiated:

     • Access Zone: Automated by Eyeglass

       info

       AD delegation must be completed as per install documentation.

9. Move SmartConnect Zone to Target Add source SmartConnect zones and aliases on the target cluster.

   • How Action Is Initiated:
     • Access Zone: Automated by Eyeglass

10. Update SPN to Allow Authentication Against Target Sync SPNs in all Active Directory (AD) providers to match current SmartConnect zone names and aliases. Proxy configuration is handled through the target cluster.

    • How Action Is Initiated:
      • Access Zone: Automated by Eyeglass

11. Repoint DNS to the Target Cluster IP Address
    Perform DNS dual delegation for all SmartConnect zones that are members of the Access Zone.

    • How Action Is Initiated:

      • Access Zone: Automated by Eyeglass

        tip

        See Configure Access Zone DNS Dual Delegation.

12. Remove Quotas on Directories That Are Targets of SyncIQ Eyeglass deletes all quotas on the source cluster for the specified policies.

    • How Action Is Initiated:
      • Access Zone: Automated by Eyeglass

    1. Run Mirror Policy
       Execute the mirror policy to resync data in the reference direction.

       • How Action Is Initiated:
       • Access Zone: Automated by Eyeglass

    2. Set Eyeglass Config Jobs to Enabled
       Enables configuration synchronization only if the Resync Prep step completes successfully for the policy.

       • How Action Is Initiated:
       • Access Zone: Automated by Eyeglass

13. Disable SyncIQ on Source and Make Active on Target

    Perform the resync prep SyncIQ policy step to failover. This creates a mirror policy on the target, disables the source cluster policy, and enables the target cluster policy in PowerScale OneFS.

    • How Action Is Initiated:
      • Access Zone: Automated by Eyeglass

14. Set Proper SyncIQ Schedule on Target

    Set the schedule on the mirror policy (target) using the schedule retrieved from step 6 in PowerScale OneFS for the policies related to the failover.

    • How Action Is Initiated:
      • Access Zone: Automated by Eyeglass

15. Synchronize Quotas

    Run Eyeglass quota jobs related to the SyncIQ policy or Access Zone being failed over.

    • How Action Is Initiated:
      • Access Zone: Automated by Eyeglass

16. Repoint DNS to the Target Cluster IP Address

    Dual delegation with Eyeglass avoids DNS steps during failover for all SmartConnect zones that are failed over.

    • How Action Is Initiated:

      • Access Zone: Automated by Eyeglass

        tip

        See Configure Access Zone DNS Dual Delegation.

17. Refresh Session to Pick Up DNS Change

    Remount the SMB/NFS shares or remount exports if data integrity remount is not required for SMB.

    • How Action Is Initiated:

      • Access Zone: Automated by Eyeglass

        tip

        See Configure Access Zone DNS Dual Delegation.

Controlled Failover Option Results Summary

The following tables indicate which failover steps are executed based on whether the Controlled failover option was selected when the failover was initiated. Step numbers in these tables differ from the ordered lists above.

SyncIQ Policy Failover

Step	Executed on	Initiated	Controlled selected	Controlled NOT selected
1 - Ensure that there is no live access to data (check for open files)	Source	Manual	Not applicable - manual step	Not applicable - manual step
2 - Begin failover (initiate failover from Eyeglass)	Eyeglass	Manual	Not applicable - manual step	Not applicable - manual step
3 - Validation (wait for other Eyeglass failover jobs to complete)	Eyeglass	Automated by Eyeglass	Step executed	Step executed
4 - Synchronize data (run all OneFS SyncIQ policy jobs)	Source	Automated by Eyeglass	Step executed	Step NOT executed
5 - Synchronize configuration (shares, exports, aliases)	Eyeglass	Automated by Eyeglass (configuration exists on source and target)	Step executed	Step executed based on last known data in Eyeglass. To skip this step, uncheck the Config Sync option.
6 - Synchronize quotas	Eyeglass	Automated by Eyeglass (deleted on source cluster and created on target cluster)	Step executed	Step executed based on last known data in Eyeglass
7 - Record schedule for SyncIQ policies being failed over	Source	Automated by Eyeglass	Step executed	Step NOT executed
8 - Prevent SyncIQ policies being failed over from running (set schedule to manual on source)	Source	Automated by Eyeglass	Step executed	Step NOT executed
9 - Provide write access to data on target	Target	Automated by Eyeglass	Step executed	Step executed
10 - Disable SyncIQ on source and make active on target (resync prep, creates mirror policy on target)	Source	Automated by Eyeglass	Step executed	Step NOT executed
11 - Set proper SyncIQ schedule on target mirror policy	Target	Automated by Eyeglass	Step executed	Step NOT executed
12 - Remove quotas on directories that are targets of SyncIQ (PowerScale best practice)	Source	Automated by Eyeglass	Step executed	Step NOT executed
13 - Change SmartConnect zone on source so it is not resolved by clients	Source	Manual	Not applicable - manual step	Not applicable - manual step
14 - Avoid SPN collision	Source	Manual (deletes SmartConnect SPN from source cluster machine account)	Not applicable - manual step	Not applicable - manual step
15 - Move SmartConnect zone to target (add source zones as aliases on target)	Target	Manual	Not applicable - manual step	Not applicable - manual step
16 - Update SPN to allow authentication against target	Target	Manual (adds new SmartConnect alias SPNs to target cluster machine account)	Not applicable - manual step	Not applicable - manual step
17 - Repoint DNS to the target cluster IP address	DNS	Manual	Not applicable - manual step	Not applicable - manual step
18 - Refresh session to pick up DNS change (remount SMB shares)	SMB client machines	Manual on clients	Not applicable - manual step	Not applicable - manual step

Access Zone Failover

Step	Executed on	Initiated	Controlled selected	Controlled NOT selected
1 - Ensure that there is no live access to data (check for open files)	Source	Manual	Not applicable - manual step	Not applicable - manual step
2 - Begin failover (initiate failover from Eyeglass)	Eyeglass	Manual	Not applicable - manual step	Not applicable - manual step
3 - Validation (wait for other Eyeglass failover jobs to complete)	Eyeglass	Automated by Eyeglass	Step executed	Step executed
4 - Synchronize data (run all OneFS SyncIQ policy jobs related to the Access Zone)	Source	Automated by Eyeglass (all policies in the Access Zone)	Step executed	Step NOT executed
5 - Synchronize configuration (shares, exports, aliases)	Eyeglass	Automated by Eyeglass (based on matching Access Zone base path)	Step executed	Step executed based on last known data in Eyeglass. To skip this step, uncheck the Config Sync option.
6 - Synchronize quotas	Eyeglass	Automated by Eyeglass (based on matching Access Zone base path)	Step executed	Step executed based on last known data in Eyeglass
7 - Record schedule for SyncIQ policies being failed over	Source	Automated by Eyeglass	Step executed	Step NOT executed
8 - Prevent SyncIQ policies being failed over from running (set schedule to manual on source)	Source	Automated by Eyeglass	Step executed	Step NOT executed
9 - Provide write access to data on target	Target	Automated by Eyeglass (only for policies that match the Access Zone base path)	Step executed	Step executed
10 - Disable SyncIQ on source and make active on target (resync prep, creates mirror policy on target)	Source	Automated by Eyeglass	Step executed	Step NOT executed
11 - Set proper SyncIQ schedule on target mirror policy	Target	Automated by Eyeglass	Step executed	Step NOT executed
12 - Remove quotas on directories that are targets of SyncIQ (PowerScale best practice)	Source	Automated by Eyeglass	Step executed	Step NOT executed
13 - Change SmartConnect zone on source so it is not resolved by clients	Source	Automated by Eyeglass (requires IP pool hints to be configured)	Step executed	Step NOT executed
14 - Avoid SPN collision	Source	Automated by Eyeglass (AD delegation must be completed as per the installation documentation)	Step executed	Step executed
15 - Move SmartConnect zone to target (add source zones as aliases on target)	Target	Automated by Eyeglass (requires IP pool hints to be configured)	Step executed	Step executed
16 - Update SPN to allow authentication against target	Target	Automated by Eyeglass (requires IP pool hints to be configured)	Step executed	Step executed
17 - Repoint DNS to the target cluster IP address	DNS	Potentially automated by Eyeglass with the script engine	Not applicable - manual or scripted step	Not applicable - manual or scripted step
18 - Refresh session to pick up DNS change (remount SMB shares)	SMB client machines	Manual on clients	Not applicable - manual step	Not applicable - manual step

note

DNS servers and clients cache DNS entries, which requires touching the client or intermediate DNS servers to clear DNS caches. On Windows, use ipconfig /flushdns.

Additional Information

1. Initiates Eyeglass Configuration Replication task for all Eyeglass jobs.

   
   

2. SyncIQ does NOT modify the ACL (Access control settings on the file system); it locks the file system.

   
   

3. A system.xml change is required to enable parallel step mode.

   note

   All policy steps are attempted. On failure, the failover job will continue to attempt all steps and skip downstream steps per policy if the previous SyncIQ step failed.

See Also

• Failover Design – What each failover type is and the results it produces.
• Failover Planning – Pre-failover checklists and readiness preparation.
• DR Rehearsal Mode – Test write access at the DR site without completing a full failover.