Parallel Failover Job and Thread Configuration

Introduction

These configurations are aimed at environments with more than 50 SyncIQ policies that require a faster failover option to maintain SLA on data recovery. Three related capabilities are available:

• Parallel Threads — Allows Make Writable and Resync Prep operations to run in parallel, up to a thread limit. Eyeglass keeps at least the configured number of policies executing at a time throughout the failover process across all failover jobs.
• Parallel Jobs — Allows more active jobs to share threads for failover. This value and the thread value should be set to the same number. The default is 10 jobs.
• Concurrent Failovers — The default is 5, and can be increased using the parameters described below. This may be required in a multi-tenant scenario where multiple users can submit failover jobs.

Parallel Failover Job and Concurrent Failover Configuration

Parallel Failover Jobs allow multiple failover jobs of any type to be run at the same time. This feature shares a single thread limit across all failover jobs, and can be combined with Parallel Threads to increase the parallelization of each individual failover job.

warning

Test this configuration in advance of a real failover. Three separate values must be changed together to increase parallel jobs, threads, and the concurrent failover limit.

A failover job can contain any type of failover (Access Zone, IP Pool, DFS, or SyncIQ policy) with any number of policies.

How to Increase Parallel Threads, Failover Jobs, and Concurrent Failover Limit

These options default to 10 failover jobs and 10 threads in the pool. This can be increased; testing has validated up to 50 failover jobs and 50 threads. It is not recommended to increase beyond these tested limits. The default concurrent failover limit is 5, and can also be increased.

1. Log in via SSH as admin to the Eyeglass appliance.

2. Run sudo -s and enter the admin password to become the root user.

3. Edit the parallel task limits file:

   nano /opt/superna/sca/data/parallelTaskLimits.json
   • To increase to 20 failover jobs and 20 threads, set both values to 20:

   {"parallelJobs":20,"totalParallelTasks":20}
   note

   The minimum value should be 2. Increasing the job count requires increasing the thread count to the same number.

   • Save and exit with CTRL+X, then answer Y to save the changes.

4. To change the default concurrent failover limit — required if the parallel jobs count is increased, or if 5 concurrent failovers is not enough — edit the system configuration file:

   nano /opt/superna/sca/data/system.xml
   • Add a new tag inside the <process> tag, setting the desired value. Testing has validated up to 50 concurrent failovers; it is not recommended to increase beyond this tested limit.

   <max_concurrent_failovers>5</max_concurrent_failovers>
   warning

   Do not set this value higher than the parallel jobs value.

   • Save and exit with CTRL+X, then answer Y to save the changes.

5. Restart the SCA process to apply the changes:

   systemctl restart sca

6. To submit parallel Access Zone, policy, DFS, or IP Pool jobs, use the Failover Wizard to start a failover job, close it, reopen it, and start another failover. Repeat this to submit additional parallel failover jobs.

7. Monitor all running failovers from the Failover Wizard's Running Failovers tab.

warning

Cluster resources may be exhausted when running a very large number of concurrent failovers. Testing is mandatory prior to attempting this in production.

Parallel Threads for Failover Mode (Legacy Mode)

This mode runs Make Writable steps for multiple policies in parallel, using up to 10 threads. It is enabled by default in all current releases and does not need to be changed unless directed by Support.

Key differences between sequential and parallel mode:

• On 8.x clusters, up to 50 policies can run at a time, using a maximum of 10 threads to send 10 policy Make Writable or Resync Prep commands at once. On 7.2 clusters, 5 policies execute at a time with 5 queued; as each policy completes, another is started to keep the maximum number queued.
• Testing has shown 3x to 4x improvements in overall time to complete Make Writable, though results in production may vary.

How to Enable Parallel Threads

note

This is enabled by default in current releases.

Run the following command to enable multi-threaded, parallel step execution across all failover jobs:

igls adv failovermode set --parallel=true

To disable:

igls adv failovermode set --parallel=false