Configure VAST Mesh Failover

Introduction

VAST Mesh Failover is an advanced disaster recovery capability that enables seamless failover between three or more VAST clusters without requiring manual intervention after the initial setup.

Traditional failover approaches involve failing over from a primary cluster (A) to a secondary cluster (B), then failing back from B to A, or alternatively failing over from A to C and failing back from C to A. This creates a limitation if there is an issue with cluster A, preventing a return after failing over to B.

The mesh failover concept addresses this limitation by allowing users to fail over from A to B, then from B to C, and finally from C back to A—all without needing to log in to VAST after the initial setup. This circular failover pattern provides greater flexibility and resilience in disaster recovery scenarios.

Requirements

• Three VAST clusters running version 5.4.0 or later
• Eyeglass version 4.4.0 or later
• All three clusters must be connected via Replication Peers
• Active Directory configured
• DNS configured
• Mesh failover feature enabled in system.xml

Environment Configuration

Enable Mesh Failover

To enable the mesh failover feature, you must configure the following environment variable in Eyeglass's system.xml file:

<enableVastDRMeshMode>true</enableVastDRMeshMode>

This setting enables mesh failovers for VAST clusters within Eyeglass.

Cluster Topology

For this guide, we will use a three-cluster configuration:

• Cluster A (PROD): Production cluster
• Cluster B (DR1): First disaster recovery cluster
• Cluster C (DR2): Second disaster recovery cluster

Configuration Steps

VAST Operations

The following steps must be performed on the VAST clusters to prepare for mesh failover.

Create Tenants

On all three clusters (PROD, DR1, and DR2):

1. Go to Element Store > Tenants. Click Create Tenant.

2. Complete the information in the Add Tenant window.

   Enter a Name for the Tenant. Ensure that the tenant names match across all three clusters.

   Go to the Providers tab. Select the correct Active Directory and LDAP values.

3. Click Create to create the Tenant and close this window.

   If presented with a dialog asking to Create an inaccessible tenant?, select Yes, Create Tenant Anyway.

info

The tenant name must be identical on all three clusters. For example: Demo

Create Virtual IP Pools

On all three clusters (PROD, DR1, and DR2):

1. Go to Network Access > Virtual IP Pools. Click Create Virtual IP Pool.

2. Complete the information in the Add Virtual IP Pool window.

   Select the previously created Tenant. Enter a Name for the VIP Pool.

   Enter the appropriate Subnet CIDR IPv4 value that matches your network requirements.

   Go to the IP Range List tab. Enter the Start IP and End IP, then click + Add.

   Go to the DNS Configurations tab. Enter the Virtual IP Pool Domain Name using the following convention:

   • On PROD cluster: Use the standard domain name (e.g., demo-vip1)
   • On DR1 cluster: The domain name must include the igls-original- prefix (e.g., igls-original-demo-vip1)
   • On DR2 cluster: The domain name must include the igls-original- prefix (e.g., igls-original-demo-vip1)

3. Click Create to add the virtual IP pool and close this window.

warning

All DR cluster VIP pool domain names must have the igls-original- prefix appended. During failover, Eyeglass will automatically rename domain names by adding or removing this prefix to redirect client traffic.

Create View Policy (PROD Cluster Only)

On the PROD cluster only:

1. Go to Element Store > View Policies. Click Create Policy.

2. Complete the information in the Add Policy window.

   Select the Tenant and Virtual IP Pool created earlier.

   Enter a Name for the view policy (e.g., Demo-ViewPolicy1) and select Security flavor.

   Go to the NFS tab. Select Group membership source.

3. Click Create to add the view policy and close this window.

Create View (PROD Cluster Only)

On the PROD cluster only:

1. Go to Element Store > Views. Click Create View.

2. Complete the information in the Add View window.

   Select the Tenant and Policy name you created earlier.

   Enter names for the Path and SMB share name.

   Select SMB from Protocols. Select the option Create new directory for the view.

   Go to the Share-level ACL tab. Turn on the option Enable Share-level ACL.

   Select the Domain and Grantee type. Search and select the Name, then select + Add ACE. Add any other user(s) to be included.

3. Click Create to add the view and close this window.

Create Replication Peers

All three clusters must be connected via Replication Peers. Follow the instructions in the Tenant/SMB Guide to create replication peers between:

• PROD and DR1
• PROD and DR2
• DR1 and DR2

Create Protection Policies

On PROD Cluster:

1. Go to Data Protection > Protection Policies. Click Create Policy.

2. Create two protection policies, one for each target:

   • Protection Policy 1: Target = DR1 (e.g., Demo-ProtectionPolicy-DR1)
   • Protection Policy 2: Target = DR2 (e.g., Demo-ProtectionPolicy-DR2)

For detailed guidance on creating protection policies, refer to the Tenant/SMB Guide.

On DR1 Cluster:

1. Create a protection policy targeting the DR2 cluster (e.g., Demo-ProtectionPolicy-DR2).

This completes the circular replication setup required for mesh failover.

Create Protected Paths

On PROD Cluster:

1. Go to Data Protection > Protected Paths. Click Create Protected Path.

2. Create a protected path for the view you created earlier.

3. Add the first destination (DR1) to the protected path.

4. After creating the protected path with the first destination, edit the protected path to add the second destination (DR2).

note

You can only set one target at a time when creating a protected path. To add another target, edit the protected path after setting the first destination.

On DR1 Cluster:

1. Edit the protected path to add DR2 as a destination.

note

On VAST 5.4.0, adding a second stream to the protected path on DR1 must be performed via CLI.

Step 1: Find the protected path on DR1

protectedpath list

Step 2: Find the protection policy for DR1 → DR2

protectionpolicy list

Step 3: Find the replication peer (DR2 cluster)

replicationpeer list

Step 4: List remote tenants on DR2

tenant list-remote --peer-id <id>

Step 5: Add the second stream

protectedpath add-stream --id <protected-path-id> --name "<stream-name>" --protection-policy-id <policy-id> --remote-tenant-name "<remote-tenant-name>"

note

• Specify --protection-policy-id only — the remote target ID is automatically derived from the policy.
• Use --remote-tenant-name (the tenant name on the remote cluster), not the GUID.
• The command tenant list-remote requires --peer-id to query the remote tenant.
• To list existing streams, run replicationstream list.

This completes the setup of the mesh replication topology.

Eyeglass Operations

Add Managed Devices

Log in to Eyeglass and perform the following steps:

1. Load the VAST DR licenses.

2. From the Add Managed Device window, add all three clusters (PROD, DR1, and DR2) and assign licenses to them.

For detailed instructions, see Add Clusters.

After adding new clusters to the Eyeglass appliance, the inventory collection job starts automatically. Monitor this job by opening the Jobs window and going to the Running Jobs tab. Wait for the job to complete before continuing.

Configuration Replication and Tenant Readiness

1. Open the Jobs window and go to the Job Definitions tab.

2. In the Jobs window, two jobs will appear. Eyeglass creates separate jobs for the same protected path with two destinations.

3. Enable and run the Configuration Replication jobs. This step ensures that the necessary View Policies and Views are created on both DR clusters.

4. Confirm that the job statuses are OK.

5. Run the Tenant Readiness job.

6. Ensure that the Tenant Readiness job has successfully completed.

Verify Readiness with DR Dashboard

1. Open DR Dashboard and confirm both jobs' protected path readiness is in OK or INFO state.

2. Navigate to Tenant Readiness and ensure both tenant jobs are in OK or INFO state.

Failover Procedures

Protected Path Failover

Failover from A to B

1. Open the DR Assistant window.

2. Select the Source Cluster (A), set Failover Type to Protected Path, and select Failover Mode. Click Next.

3. Acknowledge the support process and click Next.

4. From cluster A, select cluster B as the target. Click Next.

note

You can select only one target for a protected path such as Demo-ProtectedPath. Selecting both targets will trigger an alert.

5. Confirm the configuration is valid and click Next.

6. Review everything and click Run Failover.

7. Review the Protected Path Failover job and Logs.

After successful failover, cluster B becomes the source, while clusters A and C serve as destinations. Eyeglass will detect new jobs with updated flows, ready for configuration replication.

After successful Configuration Replication and Tenant Readiness, you are ready to fail over from B to either A or C.

Failover from B to C

1. Repeat the steps above, selecting B as the source and C as the target.

2. Verify everything and click Run Failover.

3. Review the Protected Path Failover job and Logs.

After completion, verify that new jobs use C as the source and A and B as destinations.

After successful runs of Configuration Replication and Tenant Readiness, you are ready to fail over from C to either A or B.

Failover from C to A

1. Repeat the steps above, selecting C as the source and A as the target.

2. Verify everything and click Run Failover.

3. Review the Protected Path Failover job and Logs.

You are now at the original state with A as the source and B and C as the targets.

Confirm Configuration Replication and Tenant Readiness jobs are OK.

Tenant Failover

Before beginning tenant failover, verify tenant readiness in the DR Dashboard.

Understanding Network Mapping

Open DR Dashboard and verify Tenant Readiness. In Network Mapping, you will see that all tenants contain the VIP Pool with the following domain name convention:

• Cluster A: demo-vip1 (no prefix)
• Cluster B: igls-original-demo-vip1 (with prefix)
• Cluster C: igls-original-demo-vip1 (with prefix)

info

All DR cluster VIP pool domain names will have the igls-original- prefix appended. During failover, Eyeglass automatically renames domain names to redirect client traffic.

Failover from A to B

1. Open the DR Assistant window.

2. Select the Source Cluster (A), set Failover Type to Tenant, and select Failover Mode. Click Next.

3. Acknowledge the support process and click Next.

4. Select the tenant with A as the source and B as the target, then click Next.

5. Verify the protected paths scheduled for failover under the selected tenant. Acknowledge the failover release notes and click Next.

6. Check everything one last time, acknowledge the settings, and then click Run Failover.

7. Review the Tenant Failover job and verify the Logs. The logs will show the protected path failover and the domain name renaming for VIP pools for the Demo tenant.

Domain Name Changes:

• Before: A = demo-vip1, B = igls-original-demo-vip1
• After: A = igls-original-demo-vip1, B = demo-vip1

New Configuration Replication and Tenant Readiness jobs with updated flows are now visible in the Jobs window.

The Demo tenant from B can now be failed over to either A or C.

Failover from B to C

1. Open DR Dashboard to verify Tenant Readiness.

2. Repeat DR Assistant steps with B as the source and C as the target, then initiate failover.

3. Review the Tenant Failover job and verify the Logs. Protected path failover and renaming of domain names can be seen below:

Domain Name Changes:

• Before: B = demo-vip1, C = igls-original-demo-vip1
• After: B = igls-original-demo-vip1, C = demo-vip1

New Configuration Replication and Tenant Readiness jobs with updated flows are now visible in the Jobs window.

The Demo tenant from C can now be failed over to either A or B.

Failover from C to A

1. Open DR Dashboard to verify Tenant Readiness.

2. Repeat DR Assistant steps with C as the source and A as the target, then initiate failover.

3. Review the Tenant Failover job and verify the Logs. Protected path failover and renaming of domain names can be seen below:

Domain Name Changes:

• Before: C = demo-vip1, A = igls-original-demo-vip1
• After: C = igls-original-demo-vip1, A = demo-vip1

New Configuration Replication and Tenant Readiness jobs with original flows are now visible in the Jobs window.

You have successfully returned to the original state.

Summary

VAST Mesh Failover provides a robust circular failover pattern across three or more clusters, eliminating single points of failure in disaster recovery scenarios. By configuring mesh replication and leveraging Eyeglass's automated failover orchestration, organizations can achieve greater resilience and flexibility in their data protection strategies.

Key benefits of mesh failover include:

• Circular failover capability: Fail over from A → B → C → A without manual VAST intervention
• Automatic domain name management: Eyeglass handles VIP pool domain name renaming during failover
• Flexible granularity: Support for both protected path-level and tenant-level failover
• Simplified recovery: No need to log in to VAST after initial configuration

See Also

• Configure Tenant/SMB Failover
• Configure VIP Pool Failover