Skip to main content
Migration Notice
We're migrating documentation from the old portal into this one. Some things may look a little different or out of place in the meantime — we know, and we're working to get it right. If something's unclear or doesn't look right, let us know.
Version: 2.15.0

Design Guide for Data Orchestration for Dell

Introduction​

This guide covers the architecture and performance considerations involved in planning a Data Orchestration deployment. It is intended for architects and administrators deciding how many Golden Copy VMs and Virtual Accelerator Nodes to deploy, and how to configure them for the required copy throughput. For step-by-step configuration procedures, see Managing the Solution.

Deployment Architecture​

A Data Orchestration deployment consists of a Golden Copy control VM and, optionally, one or more Virtual Accelerator Nodes (VANs):

  • The control VM manages configuration, job scheduling, and reporting.
  • VAN nodes exist solely to perform copy work, increasing overall throughput.
  • All nodes (control and VAN) require an NFS mount to the source PowerScale cluster.
  • A standard scale-out configuration is 6 VMs (1 control node plus 5 VAN nodes); deployments can scale out further, up to 99 nodes, for higher-throughput requirements.

Performance Planning​

Scale-out vs. scale-up​

Golden Copy defaults to 100 copy threads per VM. Vertical scaling (adding more threads to a single VM) does not scale copy throughput as effectively as horizontal scaling (adding more VMs) — this is why a multi-VM scale-out configuration is the recommended starting point for higher-throughput requirements. Each thread processes files in batches, and depending on file size, throughput can reach 1,200+ files per second on properly sized, scaled-out deployments.

warning

Thread-count and memory tuning changes should only be made with Superna Support guidance. Increasing thread counts without a corresponding understanding of memory, CPU, and threading behavior can create a cross-over point where throughput decreases rather than increases. In most cases, the actual bottleneck lies elsewhere in the end-to-end path from source storage to the target object storage bucket.

Small files vs. large files​

  • Most file systems are dominated by small files, where throughput (bytes/second) is not a meaningful metric — files-per-second is tracked instead (last 15 minutes, last hour, last 24 hours).
  • Large files (64 MB or larger) benefit from multipart upload/download, where portions of a single file are copied in parallel (10 threads per file by default) and reassembled at the target. This is the primary way to achieve high aggregate throughput, but requires a file population dominated by large files (for example, media workloads) to see the benefit.
  • File systems with a significant proportion of large files require additional VM memory to support multipart transfers without impacting stability.

Compression and encryption​

  • Data Orchestration does not apply compression during copy. The CPU time required to compress data generally outweighs any bandwidth savings, and many common file types (for example, Office documents, PDFs) are already compressed by the originating application.
  • In-flight encryption uses end-to-end TLS 1.3. At-rest encryption is provided by the target cloud provider (typically via provider-managed keys). No additional encryption is applied during the copy process itself, since doing so would add CPU overhead without a corresponding security benefit given the in-flight and at-rest protections already in place.

Incremental job performance​

For clusters with a high change rate (for example, over 3 million files per daily incremental), a fast incremental mode is available that skips a secondary metadata-lookup API call and relies only on the metadata already included in the PowerScale change list. This trades some metadata completeness (owner, group, and mode bits are not collected in this mode; folder ACLs and file timestamps are still collected) for significantly faster incremental processing.

Recall (restore) performance​

Recall/restore throughput can be tuned similarly to copy throughput, using additional parallel download threads for multipart recall operations. This requires a sufficiently current Golden Copy release and applies primarily to AWS S3-compatible targets.

Deployment Patterns​

Deployment sizing should follow the intended use case (see Use Cases for the full list):

  • Backup use cases (single or versioned synced backup, one-time archive copy) are typically less throughput-sensitive per job, since they run on a scheduled cadence, but benefit from scale-out sizing when the data set is large or the incremental change rate is high.
  • Data shuffle and hybrid cloud/Pipeline workflows that move newly created or changed data continuously benefit most from scale-out VAN sizing, since they are triggered more frequently by ongoing data creation.
  • Media and large-file workflows benefit most from multipart upload/download tuning and additional VM memory, rather than simply adding more VAN nodes.

See Also​

  • Concept – What Data Orchestration is and the terminology used throughout this documentation.
  • Pre-Requisites – Platform, licensing, and infrastructure requirements.
  • Use Cases – Common scenarios for Data Orchestration.