Post Failover Steps

Introduction

After executing any type of failover, there are certain steps which must be taken to ensure proper functioning of your Eyeglass VM and all associated Clusters.

Post Failover Steps

There are several procedures that may be followed after Failover to verify that everything is in its expected State and Location. To make sure you only encounter relevant information, we must separate these steps into sub-articles as follows:

Access Zone

Post Failover Steps for Access Zone Failover

Verify that DNS Updates Were Completed Correctly

1. SSH into your Eyeglass IP address using a Terminal.

2. Run the following command:

   nslookup

3. Run the Server Command, replacing the X.X.X.X with the IP of the subnet service on the target Cluster.

   server X.X.X.X

4. Enter a SmartConnect Zone name (format shown below)and press enter.

   Format: 
   <name>.<domain>
   
   Example:
   isi01-s0.example.com

5. The expected response should be an IP address from the Target Cluster's IP Pool.

   tip

   If the received output is indeed from the Target Cluster IP Pool, then failover (as far as SmartConnect delegation to the target Cluster is concerned), was correct.

6. Run the Server Command again, using the Production DNS server that has a modified CNAME.

7. Repeat the tests above using a Production DNS Client IP address.

8. Verify that the output returns an IP address from the Target Cluster IP Pool consistently.

Check for SPN Errors

1. Review the Failover Log and verify that all the SPN steps were successful.

   info

   Failover logs are located under:

   /opt/data/failover_logs/

2. If any SPN steps show as failed, manual recovery of the SPN will be required using the ADSIedit AD tool to perform delete and add of SmartConnect names or Alias names. This tool requires permissions to the computer account for the Cluster being edited.

   info

   The Failover Log contains all SPN SmartConnect names that were included in the failover.

Automated SMB Connection Switch to Target Cluster

warning

This procedure can't be used if you disabled the SMB Protocol on the Cluster.

To start, complete failover as normal with Eyeglass Failover Wizard.

Complete the following steps for each Access Zone IP Pool that was part of the Failover, and has SMB connections that should switch to the Target Cluster.

1. View/edit the IP Pool. (To do this, on your PowerScale OneFS UI: click on Cluster Management, then External Network, and finally View/Edit the specific Pool.)
2. Record the interfaces that are members of the pool. (this will be required to reconfigure the pool).
3. Select all interfaces and click on Remove.
4. Save the pool with no member interfaces.
5. Once you've completed steps 1-4, all connected SMB clients will query DNS for the SmartConnect name and will re-mount and re-authenticate to the Target Cluster.
6. View/edit the same IP Pool.
7. Re-add the interfaces that were recorded in step 2.
8. Save the Pool again.
9. Repeat steps 1-7 for each IP Pool included in the Failover.
10. Test data access to the Target Cluster to verify SMB clients have switched and can write data.

Manually Switch SMB Connection to Target Cluster

To see this procedure, consult the How to Manually Switch SMB Connection to Target Cluster After Failover guide.

Refresh NFS Mounts

NFS Mounts require an unmount and remount on the host (whichever it is that you're using).

1. To unmount an export with open files, use the following option (force and lazy flag).

   umount -fl

2. Remount the export, or, if configured in fstab, use the following command to remount any unmounted entries in the file.

   mount -a

Quota Updates

After the update, there should be no quotas on the Source Cluster for the SyncIQ Policies in the Access Zone. On the Target Cluster, you should find all quotas for the SyncIQ Policies in the Access Zone.

SPN Updates

After the update, there should be SPNs for all SmartConnect Zones, and SmartConnect Zone Aliases related to the subnet pools associated with the access zone that was failed over.

With SPN Delegation configured as required in preparation for Eyeglass Assisted Access Zone Failover, Eyeglass creates SPNs related to the SmartConnect Zones and Aliases it detects. No manual SPN management is required.

note

• SPNs are not created for SmartConnect Zones or SmartConnect Zone Aliases that are prefixed with igls.
• SPNs are not created for HDFS or NFS. These will need to be repaired manually.

IMPORTANT

Eyeglass does not create SPNs for any SmartConnect Zone or SmartConnect Zone Alias that is prefixed with igls. An SPN check from clusters configured with Eyeglass mapping hints indicates that there are missing SPNs for these SmartConnect Zones and Aliases. This is expected, because these SmartConnect Zones and Aliases are not used for cluster access.

Do not run SPN repair. It fails if executed on both clusters because of the conflict created by having identical mapping hints on the clusters.

IMPORTANT

Eyeglass does not remove "extra" SPNs that do not correspond to detected SmartConnect Zones and Aliases. If you need these SPNs removed, remove them manually.

IMPORTANT

Due to a PowerScale issue, executing the SPN repair step may result in an error, both when executed by Eyeglass and when executed directly from the PowerScale command line. In this case, the SPNs must be repaired manually, after which the SPN repair command resumes as expected.

Use the ADSIedit tool to verify the machine account SPNs.

SmartConnect Zone Updates

Post Failover, the following should be true regarding the SmartConnect Zones and Aliases related to the subnet pools associated with the Access Zone that was failed over.

1. Eyeglass created SmartConnect one alias on Target Cluster, with the same name as the SmartConnect Zone on the Source Cluster partner IP Pool.

2. Eyeglass updated the Source Cluster SmartConnect Zone name with the prefix "igls-original-".

3. The Alias for the Target SmartConnect Zone is removed from the Source Cluster.

4. After Failover is completed, DNS Admin or Post Failover scripting updates DNS entry for the SmartConnect Zone name must use the SmartConnect Service IP address from the Target Cluster.

   
   

Example: SmartConnect Zone Update

Failover from Cluster 1 to Cluster 2

Subnet and Pool	Cluster 1	Cluster 2
subnet1:Prod	Eyeglass renames SmartConnect Zone prod.example.com to igls-original-prod.example.com
subnet1:DR		Eyeglass creates SmartConnect Zone alias prod.example.com
subnet0:synciq-prod	No changes

Failover again from Cluster 2 to Cluster 1

Subnet and Pool	Cluster 1	Cluster 2
subnet1:Prod	Eyeglass renames SmartConnect Zone igls-original-prod.example.com to prod.example.com. Eyeglass creates SmartConnect Zone alias dr.example.com.
subnet1:DR		Eyeglass renames SmartConnect Zone dr.example.com to igls-original-dr.example.com. Eyeglass removes the SmartConnect Zone alias prod.example.com created by the previous failover.
subnet0:synciq-prod	No changes

IMPORTANT

If the failover was done with the Controlled failover option unchecked, some steps on the failover source cluster will not have been executed.

SyncIQ Policy Updates

To see this procedure, please consult the How to Verify SyncIQ Policy Updates guide.

SyncIQ Policy

Post Failover Steps for Policy Failover

Networking Changes

Make the necessary networking changes required to redirect your clients to the Target Cluster. This may involve:

• PowerScale OneFS SmartConnect Zone updates
• Zone Alias Updates
• DNS Updates

Update SPNs

It may be necessary to update SPNs upon completion of Failover, if you add or delete any SmartConnect Zone or Zone Alias.

Manually Refresh SMB Connection

Please consult the guide on How to Manually Switch SMB Connection to Target Cluster After Failover at the link provided.

To refresh an SMB connection after failover is completed and DNS is updated:

1. If the client was connected to the share during the failover, unmount (disconnect) the share.

2. Remount (connect) the share. Check Kerberos and the SPN on the target cluster machine account.

3. Test read/write against the newly mounted shares.

4. If step 3 fails, the original connection information is likely cached on the client machine. The data in this case would continue to be available, however it would be read-only and writes would fail. To remove the cached connection information, open a Windows command prompt and run:

   ipconfig /flushdns

5. Test read/write against the newly mounted share again.

6. If step 5 fails, it is likely a DNS cache issue that needs to be looked at on the upstream DNS servers.

Refresh NFS Connection

After Failover is completed and DNS is updated, make sure to refresh NFS Connection.

Checklist

IMPORTANT

If the failover was done with the Controlled failover option unchecked, some steps on the failover source cluster will not have been executed.

1. Check File System Updates:

   On the Source Cluster, the directories and subdirectories corresponding to the SyncIQ Policies that were failed over should be read-only. On the Target Cluster, the corresponding directories and subdirectories should be writeable.

2. Check Quota Updates:

   After the update, there should be no quotas on the Source Clusters, and they should all be in the Target Cluster.

SyncIQ Policy Updates

To see this procedure, consult the SyncIQ Policy Updates guide.

DFS

Post Failover Steps for DFS Failover

Verify Job Status

1. From the Eyeglass Web UI, open the DR Dashboard and select the DFS Readiness tab.
2. Make sure two (2) jobs are shown: one for the mirror policy (which should be active), and one for the original policy (which should be disabled now).
3. Login to the source cluster (the one that you failed over from) and verify that all shares that are part of the failover are renamed with DFS Prefixes on the cluster.

Consult Which Shares were Part of the Failover

1. From the Eyeglass Web UI, open the Jobs window.
2. Select the DFS Mode SyncIQ Policy Job using its checkbox.
3. Click on Bulk Actions
4. Click on Edit Configuration
5. Under the chosen Cluster, expand the Configuration folder. Then, under that, expand the SMB Folder. All the shares that were part of the failover will have a blue checkmark next to them.

Check Your SMB Clients

This must be done from a Windows machine in the domain which has access to the DFS mount.

1. Open with explorer:

   \\<domain name>\<dfs root name>\

2. Right click any DFS folder involved in the Failover, and click on Properties.

3. Open the DFS tab.

4. Check the status and verify that the active path is "Okay" and active.

5. Check the alternate path and verify that it's unreachable.

IMPORTANT

If the failover was done with the Controlled Failover option unchecked (a real DR event Failover), this means the share rename step on the Source cluster was not executed.

The Source Cluster should not be allowed to be reachable on the network until the shares are renamed using PowerScale OneFS UI or DFS referrals are edited to disable the target folder pointing to the source cluster.

See Also

• Readiness Guide – Validate that your environment is prepared before a planned or unplanned failover event.
• Failover Guide – Reference for the failover modes (Access Zone, IP Pool, SyncIQ Policy, DFS) covered in this guide.
• Failover Recovery – If a failover did not complete all of its steps successfully.