Troubleshooting for Data Orchestration for Dell

Introduction

This page covers common issues encountered when running Golden Copy archive, incremental, recall, and data audit jobs: copy job failures, performance problems, Virtual Accelerator Node (VAN) and cluster operational issues, and connectivity or cloud storage access problems. For CLI command syntax referenced below, see References; for performance tuning parameters, also see References.

Common Copy Job Failures

1. Get the reason for failed files in a job.
   • Run searchctl archivedfolders errors --jobId <job-id> --count 20 --tail to retrieve the most recent errors for a job. The job ID can be found with searchctl jobs running or searchctl jobs history.
   • The full error log is also written to /opt/superna/var/logs/archiveworker/error.log on the Golden Copy node and can be attached to a support case if needed.
2. Re-run only the failed files instead of the entire job.
   • Find the failed job in searchctl jobs history, then run searchctl job rerun --id <job-id> to retry only the files that failed, rather than re-running the whole copy job.
3. Re-run a failed tree walk (folder listing) failure.
   • If the folder stats show API failures retrieving files/folders from the cluster (visible with searchctl stats view --folder <folder-id>), run searchctl archivedfolders rerun --id <job-id> --only-folders after the job has completed (requires Golden Copy 1.1.9 or later).
   • The treewalk failure log on each node can be reviewed with ecactl cluster exec cat /opt/superna/var/logs/indexworker/folders_processing.log.
4. Time sync (clock skew) causes all uploads to be rejected.
   • The source PowerScale cluster and the Golden Copy appliance must be time-synced within 15 minutes of an NTP source. If they are not, S3 targets will reject all uploads due to excessive time skew. Configure NTP on both the source cluster and Golden Copy.
5. Test target permissions before troubleshooting further.
   • Before assuming a job configuration problem, run searchctl archivedfolders test --id <folder-id> to validate S3 connectivity, file upload, upload verification, and delete permissions against the target. All four checks must pass.

Performance Issues

If copy or recall throughput is lower than expected, review the following before changing any tuning parameters — most bottlenecks are not resolved simply by increasing thread counts, and thread/memory changes should only be made with Superna Support guidance:

• Small-file throughput is bound by files-per-second rather than raw bandwidth. Golden Copy defaults to 100 threads per VM; scaling out to additional Virtual Accelerator Nodes (VANs) generally improves small-file throughput more effectively than increasing threads on a single VM.
• Large-file throughput (files 64 MB or larger) benefits from multipart upload parallelism; a file system dominated by small files will not see the same throughput gains from multipart tuning.
• High incremental change rates (over roughly 3 million files per daily incremental) can be sped up by enabling ARCHIVE_SMART_INCREMENTAL=true, which skips an extra per-file metadata API call (owner/group/mode-bit metadata is not collected in this mode).
• Recall (restore) performance can be tuned with dedicated parallel-thread and skip-existing-file settings, but this configuration requires Golden Copy 1.1.17 or later and is documented specifically for AWS S3 targets.
• On a folder that has never been copied before, disabling the --skip-s3-file-exists check (searchctl archivedfolders modify --id <id> --skipS3FileExists=true) can speed up the initial full copy; re-enable it once the full copy completes and incremental mode is active.
• Full details on scale-out sizing, VAN deployment, and memory/CPU sizing for large-file environments are summarized in References.

VM and Cluster Operational Issues

• Starting and stopping the cluster: use ecactl cluster down to shut down all services and ecactl cluster up to bring them back up. A cluster restart is required after most configuration file changes.
• Removing a node from active copy operations: if a Golden Copy node needs maintenance or is showing performance issues, it can be removed from the active IP list for a cluster with searchctl isilons modify --name <cluster> --isilon-ips <remaining IPs>, which stops sending files to the removed node without needing to take the whole job down.
• Wiping configuration while retaining clusters and folder definitions: ecactl cluster rediscover (requires 1.1.9 or later).
• Backing up configuration before a factory reset: searchctl settings config export and searchctl settings config importconfig can protect cluster and folder-definition configuration, including encrypted credentials, before a factory reset (requires 1.1.9 or later).
• Isolating whether the bottleneck is the source storage or the network path: Simulated Data Upload Mode can mock archive operations with randomly generated data (in ARRAY or FILE mode) to test upload performance independent of PowerScale NFS reads. This is primarily intended for use by Superna Support, and every archive operation on the affected Golden Copy installation will use simulated data while it is enabled, so it should only be run with no scheduled archive jobs active.

Connectivity and Cloud Storage Access Issues

• Firewall/port requirements: Golden Copy requires SSH and HTTPS (port 8080 REST API) access to the PowerScale IP pool in the System zone, NFS access from Golden Copy and VAN nodes to the System zone IP pool, and outbound HTTPS (443) to the target S3 storage.

• Internet-based targets (AWS, Azure, or any other Internet-reachable S3 target) require Golden Copy's IP addresses to be NAT'd to the Internet, or a proxy must be configured.

• Proxy configuration: if transparent source-IP NAT is not available, Golden Copy supports proxy access to reach Internet cloud providers (use the AWS proxy variables below for AWS or any other S3-compatible target). On node 1 as ecaadmin, edit /opt/superna/eca/eca-env-common.conf and add the variables needed for your proxy:

  Variable	Purpose
  AWS_PROXY_HOST	Proxy host the client connects through.
  AWS_PROXY_PORT	Proxy port (for example, 8080 or 3128).
  AWS_PROXY_PROTOCOL	HTTP or HTTPS.
  AWS_PROXY_USER	Optional; proxy username, if required by the proxy.
  AWS_PROXY_PASS	Optional; proxy password, if required by the proxy.

  Save the file, then restart the cluster for the change to take effect (ecactl cluster down / ecactl cluster up).

• Restricting outbound access by IP range: AWS and Azure both publish IP range lists that can be used to scope outbound firewall rules to just the storage service ranges, rather than opening broad Internet access — see Installation Guide.

• Confirming the issue is connectivity and not credentials/permissions: run searchctl archivedfolders test --id <folder-id> to isolate a connectivity failure from a permissions or configuration issue on the target bucket.

Disk Space Management

The appliance ships with 400 GB of disk per node to hold queued copy data and per-job reporting records. Additional disk space may be needed if:

• The number of configured folder definitions grows beyond roughly 25 (reporting records are tracked per folder) — plan for an additional 100 GB per 25 folders.
• A rate limit is applied to WAN copy traffic, since queued file records can accumulate for longer while throttled.
• Job summary exports are run and stored on the appliance itself.
• The error rate on copy jobs is elevated, since error records also consume disk space until the underlying issue is resolved.
• Multiple full archive jobs are run concurrently.

See Also

• References – Full CLI command reference, folder definition parameters, and performance tuning variables.
• Compliance Guide – Auditability and job history reporting.