Skip to main content
Migration Notice
We're migrating documentation from the old portal into this one. Some things may look a little different or out of place in the meantime — we know, and we're working to get it right. If something's unclear or doesn't look right, let us know.
Version: 2.15.0

Bulk Data Loading with AWS Snowball and Azure Data Box for Data Orchestration for Dell

Introduction​

For very large initial data sets where available Internet bandwidth would make an over-the-network first copy impractically slow, Golden Copy can load data onto a physical bulk-transfer appliance — AWS Snowball or Azure Data Box — that is shipped to the cloud provider's data center and imported into the target bucket or container. File system metadata is preserved on the objects copied to the device, and once the data lands in cloud storage, Golden Copy can switch the folder to normal incremental sync over the Internet. The same physical-device approach can also be used in reverse — exporting a large volume of data back out of cloud storage — see Bulk Recall of Data with AWS Snowball and Azure Data Box below.

note

This option should only be used for very large data sets on very low-bandwidth Internet connections. It is not faster for small files, and is generally not faster than a direct Internet copy once roughly 100 Mbps or more of bandwidth is available. Consult Superna Support before choosing this option — in most cases, copying directly over the Internet is faster.

Bulk Loading Data with AWS Snowball​

AWS Snowball is an on-site device used to bulk load data into Amazon S3 faster than an Internet connection would allow.

Order, Load, and Return the Device​

  1. Order an AWS Snowball device and complete the AWS-side steps to create the export/import job, receive the device, and connect it to your local network, following AWS's own Snowball documentation.
  2. Once the device is connected and reachable from the Golden Copy appliance, continue with the Golden Copy steps below to load data.
  3. After the copy completes, return the device to AWS following AWS's return process.

Prepare Snowball Edge Credentials​

Using the AWS Snowball client tools on a workstation with network access to the device:

  1. Retrieve the manifest file and unlock code for the job from the AWS Snowball console.

  2. Configure a named profile referencing the manifest file, unlock code, and the device's on-network endpoint:

    snowballEdge.bat configure --profile gcsnowball

  3. Unlock the device:

    snowballEdge.bat unlock-device --endpoint https://<snowball-ip> --manifestfile <path-to-manifest> --unlock-code <unlock-code>

  4. Retrieve a local access key and secret access key for the device:

    snowballEdge.bat list-access-keys --profile gcsnowball
    snowballEdge.bat get-secret-access-key --access-key-id <access-key-id> --profile gcsnowball

Load Data with Golden Copy​

Add the Snowball device as a folder definition target, using the --force flag to bypass connectivity validations that don't apply to a Snowball device, and disabling incremental mode since the device will be offline in transit:

searchctl archivedfolders add --folder /ifs/data/archivedata --isilon <cluster name> --accesskey <access key> --secretkey <secret key> --endpoint http://<snowball-ip>:8080 --bucket <bucket name> --region snow --disable-incremental true --cloudtype other --force

Start the archive job (also with --force), and monitor it like any other job:

searchctl archivedfolders archive --id <folder ID> --force
searchctl jobs running
searchctl jobs view --id <job name>

Once the job completes, return the device to AWS.

Bulk Loading Data with Azure Data Box​

Azure Data Box is the equivalent on-site device for bulk loading data into Azure Blob Storage.

Order and Prepare​

  1. Order a Data Box, create the target Storage Account and Blob container in Azure, and follow Microsoft's Data Box quick-start steps to unpack, connect, and unlock the device.
  2. Create a container on the Data Box device itself with the same (case-sensitive) name as the container created in the Azure Storage Account — this is required before uploading data.
  3. DNS resolution is mandatory before loading data. The Data Box's blob service endpoint (in the form <storage-account-name>.blob.<serial-number>.microsoftdatabox.com) must resolve, from the PowerScale cluster's configured DNS servers, to the on-site IP address of the Data Box. Add a DNS zone and A record matching the endpoint shown in the Data Box console, then verify resolution from the cluster (as root) before proceeding — do not continue until the endpoint resolves correctly.

Load Data with Golden Copy​

Add the Data Box as a folder definition target, disabling incremental mode since the device will be shipped and imported before it is reachable again:

searchctl archivedfolders add --isilon <cluster name> --folder /ifs/archive --secretkey <secret key> --endpoint <databox endpoint DNS name> --container <container name> --bucket <storage account name> --disable-incremental true --cloudtype azure

Start and monitor the archive job:

searchctl archivedfolders archive --id <folder ID>
searchctl jobs running
searchctl jobs view --id <job ID>

Once the job completes, return the Data Box to Azure following Microsoft's return process, and remove the temporary DNS entry created for the device. Verify the data has landed in the Blob container once Azure notifies you that the import is complete.

Switching to Incremental Sync After Bulk Load​

Once data has been verified in the cloud target, the same folder definition used for the bulk load can be switched to normal incremental sync over the Internet, since the metadata already encoded on the objects supports incremental operation:

searchctl archivedfolders modify --id <folder ID> --endpoint <production regional endpoint> --disable-incremental false

Confirm an incremental schedule is configured on the folder and verify the next scheduled incremental run completes successfully before relying on it as the ongoing sync path.

Bulk Recall of Data with AWS Snowball and Azure Data Box​

The reverse operation — recalling a very large volume of data back out of cloud storage — has the same bandwidth and egress-cost trade-offs as the initial bulk load, and can also use a physical bulk-transfer device rather than an Internet-based recall.

note

As with bulk loading, this option should only be used for very large data sets where Internet bandwidth is limited, or where avoiding egress charges on a very large recall is a priority. It is not faster for small files, and is generally not faster than a direct Internet-based recall once roughly 100 Mbps or more of bandwidth is available. Consult Superna Support before choosing this option.

AWS Snowball Export (Bulk Recall)​

  1. Order an AWS Snowball export job to have data copied from your S3 bucket onto the device and shipped to your location, following AWS's Snowball export job documentation.
  2. Once the device arrives and is connected to your network, use the AWS Snowball client tools to copy data from the device to a mounted export on the PowerScale cluster.
info

Recalling data directly from a Snowball export device this way does not reapply Golden Copy's file metadata (owner, group, ACLs, timestamps) during the copy, since the copy is performed with the Snowball client tools rather than a Golden Copy recall job.

Azure Data Box Export (Bulk Recall)​

Azure Data Box also supports an export job that copies data from an Azure Blob container or Storage Account onto the device for shipment to your location, following Microsoft's Data Box export documentation. As with the Snowball export path above, data copied directly from the device does not have Golden Copy metadata reapplied automatically.

See Also​