Source Data Protection Policy Config
Overview
The Source Data Protection Policy Config step (Step 5) creates and manages per-bucket vault protection policies. Each policy links a source bucket to a target vault account, a copy mode, and copy performance settings — these drive the actual copy jobs that transfer data to the vault.
Filter Bar
The filter bar at the top of the page narrows the bucket list:
| Filter | Description |
|---|---|
| Search | Free-text search on bucket name or namespace. Queries the server on each change. |
| Device | Filter to a specific ObjectScale device, or leave at "All ECS devices" to see all. |
| Group | Filter to a specific protection group. Leave at "All groups" to see all. |
| Protection Status | Filter by policy state — All, Unprotected (no policy), Protected (has policy), or Vaulted. |
The Namespace filter is not shown in this mode — use the Group filter to scope by group instead.
Bucket Table
Each row represents one bucket. Columns shown in this mode:
| Column | Meaning |
|---|---|
| Device | Source ObjectScale device the bucket belongs to |
| Namespace | ObjectScale namespace |
| S3 Bucket | Bucket name |
| Group | Protection group the bucket is assigned to |
| Protection Status | Badge showing Unprotected / Protected / Vaulted |
| Policy Name | Name of the existing policy, or — if none |
| Actions | Edit button for protected/vaulted buckets |
Creating Policies — Bulk (ObjectScale Wizard)
This is the standard flow for bulk-provisioning ObjectScale vault policies. Source and target accounts are auto-assigned per namespace from the registered service accounts — no manual account selection is required.
- Use the filters to scope to the device and group you want to work with.
- Select one or more unprotected buckets using the row checkboxes, or click Select All (N) to select all unprotected buckets across every page.
- The action bar appears showing the auto-assigned accounts for the selected buckets.
- Choose a Copy Mode and configure the Delete Sync toggle as needed.
- Optionally expand Advanced Settings to tune copy performance parameters.
- Click Create Policies. One policy record is created per selected bucket.
Action Bar Controls
| Control | Default | Description |
|---|---|---|
| Source Account Profile | Auto-assigned | Shows which source S3 service account was matched to each selected bucket's namespace. Read-only — set up source accounts in Source Authentication (Step 2). |
| Vault Target | Auto-assigned | Shows which target vault account was matched per namespace. Read-only — set up target accounts in Target Authentication. |
| Mode | Auto | Read-only badge showing the group's operational mode — ◎ Prep Mode (Online) or ⬡ Vault Mode (Airgap). Automatically set from the protection group; cannot be changed per-policy. To change mode, use Vault Operations > Vault Data Sync Dashboard > Switch to Vault Mode. |
| Delete Sync | Off | When on, DELETE events in the changelist are replicated to the vault target. Disable to keep deleted objects in the vault. With Delete Sync on, a mass delete in production is copied into the vault. |
If an account cannot be matched for a namespace, an amber warning shows which namespaces are missing — provision the account in the Source or Target Authentication step before creating policies.
Advanced Settings
Click the Advanced Settings chevron to expand performance controls that apply to all policies created in the current bulk operation:
| Setting | Default | Description |
|---|---|---|
| Threads per Job | 32 | Concurrent S3 copy threads per job. With prefix partitions detected at runtime, the engine scales up to 4× (max 128). |
| Multipart Threshold (MB) | 128 | Objects at or above this size use S3 multipart upload; smaller objects are copied as a single PUT. Default 128 MB means objects up to 128 MB copy in one PUT (fewer S3 API calls, but each buffers its full size in memory during the copy). S3 minimum is 5 MB. |
| Min Part Size (MB) | 8 | Minimum size of each multipart part. S3 enforces a 5 MB hard minimum (lower values are raised to 5 MB). See the object-size ceiling note below. |
Object-size ceiling and automatic part-sizing
S3 allows at most 10,000 parts per multipart upload. A fixed part size therefore caps the largest object at part size × 10,000:
| Part size | Largest object |
|---|---|
| 8 MB | About 80 GB |
| 64 MB | About 640 GB |
| 128 MB | About 1.28 TB |
You do not need to raise Min Part Size by hand for large objects. The copy agent computes a safe part size for each object and raises it to keep the part count at or below 10,000, up to a 1 GiB part-size cap. That cap covers the S3 maximum object size of 5 TB. Automatic part-sizing is logged for each object in the copy agent log.
An object that is too large for the 10,000-part limit even at the cap fails with the message "object too large for multipart upload".
Larger parts need more memory, because each copy thread holds one part buffer. When the copy agent raises a policy's Min Part Size for a large object, it also adjusts two other settings, once, and saves them. See the list below.
- Threads per Job is lowered so that threads × part size stays within a per-job memory budget of about 1 GiB. For example, a 128 MB part size trims the policy to 8 threads. It never goes below 4. Both the full-copy and the incremental-copy thread counts are trimmed.
- Multipart Threshold is raised toward the part size, up to 128 MB. Objects smaller than one part are then not uploaded as pointless one-part multipart uploads.
The adjustments only move each value in its safe direction: Min Part Size up, Threads per Job down, and Multipart Threshold up. They never overwrite unrelated settings. You see them the next time you open the policy's Advanced Copy Settings, and you can still edit all three by hand.
Min Part Size and kept uploads
When the copy of an object of 1 GiB or more fails or is canceled, the copy agent keeps its multipart upload. The next run on that agent continues it with the part size the upload started with.
A Min Part Size change made in between, by hand or by the automatic adjustment, does not discard the kept upload. The new part size applies only to uploads that start after the change.
Read-only information cards below the inputs show derived values:
| Card | Value | Meaning |
|---|---|---|
| App Thread Pool | Threads you set | OS/virtual threads running concurrent copy tasks |
| Netty Connection Pool | Threads × 4 | S3AsyncClient HTTP connection pool — sized for ~4 parallel part connections per in-flight upload |
| Netty I/O Threads | ~2 × CPU cores | Netty event-loop threads for async S3 I/O — not user-configurable |
| Connections / Upload | ~4 parts parallel | Effective parallel multipart parts per concurrent upload |
Creating Policies — Manual (Full Configuration)
Click Manual Policy (always visible in the action bar, no bucket selection required) to open a full configuration panel for creating a single vault policy with complete control over all settings. Use this when you need a policy that the bulk wizard cannot build.
Policy Details
| Field | Default | Description |
|---|---|---|
| Policy Name | — | Unique display name. Required. |
| Mode | Auto | Read-only badge showing ◎ Prep Mode or ⬡ Vault Mode, derived from the protection group. Cannot be changed per-policy — set at the group level in Vault Operations. |
| Enabled | Yes | When off, the policy is skipped by the scheduler. |
Source
Select a registered source-side S3 service account.
| Field | Description |
|---|---|
| Namespace | ObjectScale namespace (e.g. s3). |
| Source Bucket | Name of the bucket to copy from. Required. |
Target
Select a registered target-side vault account.
| Field | Description |
|---|---|
| Region | Target bucket region. Default: us-east-1. |
| Target Bucket | Vault bucket to copy into. Required. |
| Key Prefix | Optional prefix for all object keys written to the target (e.g. vault/prod/). |
Sync & Copy Settings
| Field | Default | Description |
|---|---|---|
| Sync Interval | 6 hours | The change-window length (1 / 6 / 12 / 24 hours). The schedule policy decides when syncs run. |
| Delete Sync | Off | Replicate DELETE events to the vault target when on. |
| Schedule Policy | Default | Which schedule policy controls when copy jobs run. |
| Threads / Job | 32 | Concurrent copy threads (1–128). Scales up to 4× with prefix partitions. Netty pool = threads × 4. |
| MP Threshold (MB) | 128 | Objects above this size use multipart upload (8–512 MB). |
| Min Part (MB) | 8 | Minimum multipart part size (S3 enforces 5 MB minimum). |
| Prefix Scan Depth | 4 | How many prefix directory levels to scan when discovering parallel copy partitions (1–10). |
Copy Modes
Copy mode is determined by the protection group's operational mode — it is not a per-policy choice. All policies in a group share the same mode.
◎ Prep Mode (Online) — the group is in Preparation Mode. The copy engine writes directly to the vault target S3 endpoint. Used while the vault network is reachable and you are building up the initial vault dataset.
⬡ Vault Mode (Airgap) — the group has been switched to Vault Mode via Vault Operations > Vault Data Sync Dashboard. The vault is isolated by the air-gap switch. The vault agent opens the switch port only for each sync window. The copy agent writes to the vault's S3 endpoint (port 9021) only while the window is open. The mode badge in the policy creation action bar and the edit panel reflects this automatically — no manual selection is needed or allowed.
Why you cannot select mode per-policy: Mixed modes within a group are not technically supported. When the air-gap port is down, S3 access to the vault target fails for all policies in that group. Running some policies in Prep Mode and others in Vault Mode would result in copy failures. Mode is therefore a group-level setting only.
To switch a group from Prep Mode to Vault Mode: Go to Vault Operations > Vault Data Sync Dashboard and click Switch to Vault Mode (in the Operational Mode panel). This bulk-converts all prep-mode groups to Vault Mode, assigns them to the selected vault agent, and updates all their existing policies automatically. New policies created after the switch automatically receive Vault Mode.
Selecting Buckets
Row checkboxes — select individual buckets for bulk operations.
Select All (N) — when clicked, selects all unprotected buckets across every page matching the current filters, not just the rows visible on screen. The count N reflects the server-side filtered total.
When all rows on the current page are checked and more exist, an offer banner appears: "Select all N buckets". Clicking that link activates the full virtual select-all.
When virtual select-all is active, a confirmation banner shows the total and a Clear link to deselect.
Editing an Existing Policy
Protected and vaulted rows have an Edit button. Clicking it opens an inline edit panel below that row with all policy fields:
| Field | Description |
|---|---|
| Config Name | Display name for the policy |
| Copy Mode | Read-only badge — shows ◎ Prep Mode or ⬡ Vault Mode derived from the group. Cannot be edited here; change via Vault Operations. |
| Source Account | Source-side S3 service account dropdown |
| Target Account | Target vault account dropdown |
| Schedule Policy | Schedule controlling when copy jobs run |
| Target Bucket | Vault bucket name |
| Sync deletes to vault target | Replicate deletes when checked |
| Threads per Job | Concurrent copy threads (1–128) |
| Multipart Threshold (MB) | Multipart upload threshold (8–512 MB, default 128) |
| Min Part Size (MB, min 5) | Minimum multipart part size (at least 5 MB, default 8) |
| Prefix Scan Depth | Prefix partition scan depth (1–10) |
| Enabled | Enable or disable this policy |
Click Save Policy to apply. The copy engine picks up the new settings on the next scheduled run.
Deleting Policies
- Select one or more protected bucket rows.
- A red action bar appears showing the count of selected protected buckets.
- Click Delete N Protection Policies.
Deletion is immediate — there is no confirmation dialog. Policy records are removed; no data is deleted from the source or vault. Cancel any in-progress copy jobs for those buckets before deleting if you want to avoid orphaned job state.
Tips
- Source and target accounts must be configured before policies can be created. If accounts cannot be auto-assigned, add them in Source Authentication (Step 2) or Target Authentication.
- The bulk wizard auto-matches accounts by device ID and namespace. If a bucket's namespace has no matching account, the action bar shows an amber warning naming the missing namespace.
- Policies created here are independent of provisioning status. Ensure the target bucket has been provisioned (Step 4) before copy jobs run, or the first write will fail.
- Schedule policies are shared across groups. Changing a schedule policy affects every policy that references it.