Post Failover Steps

Introduction

After executing any type of failover, there are certain steps which must be taken to ensure proper functioning of your Eyeglass VM and all associated Clusters.

Post Failover Steps

There are several procedures that may be followed after Failover to verify that everything is in its expected State and Location. To make sure you only encounter relevant information, we must separate these steps into sub-articles as follows:

Access Zone

Post Failover Steps for Access Zone Failover

Verify that DNS Updates Were Completed Correctly

1. SSH into your Eyeglass IP address using a Terminal.

2. Run the following command:

   nslookup

3. Run the Server Command, replacing the X.X.X.X with the IP of the subnet service on the target Cluster.

   server X.X.X.X

4. Enter a SmartConnect Zone name (format shown below)and press enter.

   Format: 
   <name>.<domain>
   
   Example:
   isi01-s0.example.com

5. The expected response should be an IP address from the Target Cluster's IP Pool.

   tip

   If the received output is indeed from the Target Cluster IP Pool, then failover (as far as SmartConnect delegation to the target Cluster is concerned), was correct.

6. Run the Server Command again, using the Production DNS server that has a modified CNAME.

7. Repeat the tests above using a Production DNS Client IP address.

8. Verify that the output returns an IP address from the Target Cluster IP Pool consistently.

Check for SPN Errors

1. Review the Failover Log and verify that all the SPN steps were successful.

   info

   Failover logs are located under:

   /opt/data/failover_logs/

2. If any SPN steps show as failed, manual recovery of the SPN will be required using the ADSIedit AD tool to perform delete and add of SmartConnect names or Alias names. This tool requires permissions to the computer account for the Cluster being edited.

   info

   The Failover Log contains all SPN SmartConnect names that were included in the failover.

Automated SMB Connection Switch to Target Cluster

warning

This procedure can't be used if you disabled the SMB Protocol on the Cluster.

To start, complete failover as normal with Eyeglass DR Assistant.

Complete the following steps for each Access Zone IP Pool that was part of the Failover, and has SMB connections that should switch to the Target Cluster.

1. View/edit the IP Pool. (To do this, on your PowerScale OneFS UI: click on Cluster Management, then External Network, and finally View/Edit the specific Pool.)
2. Record the interfaces that are members of the pool. (this will be required to reconfigure the pool).
3. Select all interfaces and click on Remove.
4. Save the pool with no member interfaces.
5. Once you've completed steps 1-4, all connected SMB clients will query DNS for the SmartConnect name and will re-mount and re-authenticate to the Target Cluster.
6. View/edit the same IP Pool.
7. Re-add the interfaces that were recorded in step 2.
8. Save the Pool again.
9. Repeat steps 1-7 for each IP Pool included in the Failover.
10. Test data access to the Target Cluster to verify SMB clients have switched and can write data.

Manually Switch SMB Connection to Target Cluster

To see this procedure, consult the How to Manually Switch SMB Connection to Target Cluster After Failover guide.

Refresh NFS Mounts

NFS Mounts require an unmount and remount on the host (whichever it is that you're using).

1. To unmount an export with open files, use the following option (force and lazy flag).

   umount -fl

2. Remount the export, or, if configured in fstab, use the following command to remount any unmounted entries in the file.

   mount -a

Quota Updates

After the update, there should be no quotas on the Source Cluster for the SyncIQ Policies in the Access Zone. On the Target Cluster, you should find all quotas for the SyncIQ Policies in the Access Zone.

SPN Updates

After the update, there should be SPNs for all SmartConnect Zones, and SmartConnect Zone Aliases related to the subnet pools associated with the access zone that was failed over.

SmartConnect Zone Updates

Post Failover, the following should be true regarding the SmartConnect Zones and Aliases related to the subnet pools associated with the Access Zone that was failed over.

1. Eyeglass created SmartConnect one alias on Target Cluster, with the same name as the SmartConnect Zone on the Source Cluster partner IP Pool.

2. Eyeglass updated the Source Cluster SmartConnect Zone name with the prefix "igls-original-".

3. The Alias for the Target SmartConnect Zone is removed from the Source Cluster.

4. After Failover is completed, DNS Admin or Post Failover scripting updates DNS entry for the SmartConnect Zone name must use the SmartConnect Service IP address from the Target Cluster.

   
   

SyncIQ Policy Updates

To see this procedure, please consult the How to Verify SyncIQ Policy Updates guide.

SyncIQ Policy

Post Failover Steps for Policy Failover

Networking Changes

Make the necessary networking changes required to redirect your clients to the Target Cluster. This may involve:

• PowerScale OneFS SmartConnect Zone updates
• Zone Alias Updates
• DNS Updates

Update SPNs

It may be necessary to update SPNs upon completion of Failover, if you add or delete any SmartConnect Zone or Zone Alias.

Manually Refresh SMB Connection

Please consult the guide on How to Manually Switch SMB Connection to Target Cluster After Failover at the link provided.

Refresh NFS Connection

After Failover is completed and DNS is updated, make sure to refresh NFS Connection.

Checklist

1. Check File System Updates:

   On the Source Cluster, the directories and subdirectories corresponding to the SyncIQ Policies that were failed over should be read-only. On the Target Cluster, the corresponding directories and subdirectories should be writeable.

2. Check Quota Updates:

   After the update, there should be no quotas on the Source Clusters, and they should all be in the Target Cluster.

SyncIQ Policy Updates

To see this procedure, consult the SyncIQ Policy Updates guide.

DFS

Post Failover Steps for DFS Failover

Verify Job Status

1. From the Eyeglass Web UI, open the DR Dashboard and select the DFS Readiness Tab.
2. Make sure two (2) jobs are shown: one for the mirror policy (which should be active), and one for the original policy (which should be disabled now).
3. Login to the source cluster (the one that you failed over from) and verify that all shares that are part of the failover are renamed with DFS Prefixes on the cluster.

Consult Which Shares were Part of the Failover

1. From the Eyeglass Web UI, open the Jobs window.
2. Select the DFS Mode SyncIQ Policy Job using its checkbox.
3. Click on Bulk Actions
4. Click on Edit Configuration
5. Under the chosen Cluster, expand the Configuration folder. Then, under that, expand the SMB Folder. All the shares that were part of the failover will have a blue checkmark next to them.

Check Your SMB Clients

This must be done from a Windows machine in the domain which has access to the DFS mount.

1. Open with explorer:

   \\<domain name>\<dfs root name>\

2. Right click any DFS folder involved in the Failover, and click on Properties.

3. Open the DFS Tab.

4. Check the status and verify that the active path is "Okay" and active.

5. Check the alternate path and verify that it's unreachable.

IMPORTANT

If the failover was done with the Controlled Failover option unchecked (a real DR event Failover), this means the share rename step on the Source cluster was not executed.

The Source Cluster should not be allowed to be reachable on the network until the shares are renamed using PowerScale OneFS UI or DFS referrals are edited to disable the target folder pointing to the source cluster.