TPS Monitor

The TPS Monitor gives real-time and historical visibility into how the global TPS rate limiter distributes S3 API call quota across all active copy agents.

This page covers what the monitor shows, how the rate-limiting system works, and how to interpret unusual readings.

Why TPS rate limiting exists

Copy operations fan out into large virtual thread pools. A fleet of agents can together generate tens of thousands of S3 API calls per second against an ObjectScale cluster.

ObjectScale clusters have documented performance cliffs at high aggregate TPS:

• Metadata service latency degrades.
• The cluster starts returning 503 SlowDown responses.
• Those responses cascade into per-agent auto-pauses and lower overall throughput.

The TPS rate limiter is a distributed global governor. It caps the total S3 API call rate across the entire fleet and protects the source and target storage infrastructure.

Set the global limit from the budget that production can give replication. Agree that budget with the storage team, and start 30–40% below that headroom.

Enable TPS limiting

Go to Setup & Settings > TPS Rate Limiting and turn on Enable TPS Rate Limiting. Then configure these settings:

Setting	Description
Global Limit	The total budget shared across all agents, entered as an absolute TPS value (for example, 10000 means 10,000 TPS). Any whole number from 100 upwards is accepted. The minimum is 100 TPS. A lower value is raised to 100 when you leave the field, and the coordinator never enforces a global budget below 100 TPS.
Transaction Types	The S3 operation types that count toward the limit. LIST, GET, and PUT are counted by default. DELETE is excluded by default because deletes are rare in normal copy workflows.
Alarm Threshold	The utilization percentage at which alarm CV-TPS-001 fires. The default is 95%.

When TPS limiting is disabled, the TPS Monitor tab shows an informational card with a link to the settings page. The tab is hidden from the left navigation when the feature is off.

How the credit window token bucket works

The rate limiter uses a credit-window token bucket, timed to the 30-second heartbeat cycle between agents and the console.

1. Credit budget delivery. At every 30-second heartbeat, the console (the coordinator) tells each agent how many S3 API calls it may make in the next window. This is the agent's credit budget: creditBudget = allocatedTps × 30. For example, an agent allocated 1,000 TPS receives 30,000 credits per window.
2. Credit spending. Every S3 call deducts from the agent's credit counter before the call is dispatched. If an operation would exhaust the remaining credits, the copy thread parks in a 50-millisecond sleep loop until the next heartbeat restores the window.
3. Metric reporting. At each heartbeat, the agent resets its period counter and reports the actual TPS it used. The coordinator uses this to adjust allocations for the next window.
4. Bursts. Because the window is 30 seconds, bursts are set by thread concurrency, not by the limit. An agent can use its entire budget at the start of a window before the coordinator can react. See Efficiency above 100%.

Parking costs no platform thread. The copy paths run on virtual threads, so thousands of threads can park at once without affecting other work.

Coordinator allocation algorithm

The console runs the TPS coordinator. It is the single source of truth for allocations across the fleet.

All state is in memory. After a console restart, allocations converge again within one 30-second heartbeat cycle.

The coordinator recomputes allocations for all active agents on every heartbeat from any agent, and every 30 seconds on the console's own timer. It follows these steps:

1. Active agent list. Active agents have a running copy job and a heartbeat within the last 90 seconds. Stale agents are excluded from the budget split but keep their last allocation.
2. Equal-split baseline. baseAlloc = totalBudget / activeAgents.
3. Identify hungry agents. Agents whose utilization (actualTps / allocatedTps) is at or above 90% are hungry. They need more quota.
4. Reclaim unused quota. This happens only when hungry agents exist. Each active agent below 80% utilization returns 50% of its unused quota to a shared pool. If no agent is hungry, all agents keep the full base split. This preserves burst headroom and stops allocations from slowly converging down to actual TPS.
5. Redistribute to hungry agents. The pool is divided equally among hungry agents as a bonus allocation.
6. Exponential smoothing (40/60). Allocation changes are damped to prevent oscillation: newAllocation = 0.4 × computedAllocation + 0.6 × previousAllocation.
7. Safety cap. If the sum of all allocations exceeds the global budget, all allocations are scaled proportionally to bring the total back to the limit.
8. Per-agent minimum. No active agent gets less than 1 TPS. This matters only with a small global limit split across many agents, where it stops a share from rounding down to 0.

Smoothing means allocation changes take about 5 heartbeat cycles (about 2.5 minutes) to fully converge. A newly active agent, or one coming off a throttle period, takes about 2.5 minutes before its allocation normalizes to a fair share.

When an agent accepts a new job, the coordinator makes a simple equal split across all online agents (at least 1 TPS each). This seeds the credit budget before the first heartbeat arrives and prevents an unconstrained burst in the first 30-second window.

What each S3 operation costs

Operation	Default counted	TPS cost
LIST	Yes	1 per ListObjectsV2 call. Each call returns up to 1,000 objects.
GET	No effect	GET is never charged.
PUT (single-part)	Yes	1
PUT (multipart)	Yes	11, a fixed charge once per object.
DELETE	No effect	DELETE is never charged.

The limiter charges a fixed amount once per object: 11 for a multipart object and 1 for a single-part object. It does not count CreateMultipartUpload, HEAD, ListParts, ranged GETs, retries, or Target Skip checks. An N-part copy really makes 2N+3 requests (2N+4 for an incremental copy): about 36 for a 128 MB object and 260 for a 1 GiB object at 8 MB parts. The limit caps the copy rate, not the source's real request rate. Objects at or above the multipart threshold, default 128 MB, are counted at the higher rate automatically.

Monitor display

Summary banner

Three stat cards appear at the top of the page:

• Total System TPS is the sum of actual TPS across all agents, shown against the global limit (for example, 7,832 / 10,000).
• Utilization is the percentage of the global budget currently in use.
• Active Agents is the number of agents with a running copy job, compared with the total registered agents.

Total System TPS and Utilization use the same colors: green below 80%, amber at 80–95%, and red above 95%.

Per-agent table

Column	What it means
Agent	The display name of the registered copy agent. A row named "console" is always listed. The console reports itself to the TPS coordinator, but copy jobs never run on it. That row stays Idle with 0 actual TPS and takes no share of the budget.
Status	A green Running badge when the agent has an active copy job. A gray Idle badge otherwise.
Allocated TPS	The ops-per-second budget that the coordinator assigned to the agent for the current 30-second window.
Actual TPS	The TPS observed in the last 30-second period, as reported in the agent's last heartbeat. It uses the same green, amber, and red bands as the summary.
Efficiency %	actualTps / allocatedTps × 100, with a mini progress bar. It shows how well the agent uses its quota. Agents below 80% may be releasing unused quota to other agents.
Actual Mbps	Always shows 0.0. Bandwidth-based limiting is not enforced.

Time-series chart

The chart shows per-agent TPS and total TPS over the last 30 minutes, as 60 data points at 30-second intervals.

• Each registered agent has its own colored line.
• A bold Total line shows the aggregate across all agents.
• A dashed red reference line marks the configured global limit.
• The X-axis shows wall-clock time (HH:MM). The tooltip shows per-agent TPS at each sampled point.

The chart updates every 30 seconds, matching the heartbeat cadence. A countdown next to the Refresh button shows the seconds until the next refresh. A query-latency badge in ms appears alongside.

Expected behavior and unusual readings

Normal patterns

• Running agents at 70–95% efficiency are well throttled and use their quota effectively.
• Idle agents show 0 actual TPS. They may still show a non-zero allocated TPS, because the coordinator keeps the last computed allocation until the agent's next heartbeat delivers new state.
• Allocations are roughly equal across running agents at steady state.

Efficiency above 100%

An agent with efficiency above 100% is working correctly. It has more copy work than its current allocation covers, and the coordinator gives it a larger quota in the next 30-second window.

Read it as a signal of which agents are demand-constrained:

• High efficiency means more quota is coming.
• Low efficiency means quota will be redistributed away.

Why the rate appears over quota

The credit budget caps total ops per 30-second window, not the instantaneous rate per second.

For example, an agent allocated 164 TPS receives 164 × 30 = 4,920 credits. With many small, fast objects and high thread concurrency, it can use all 4,920 credits in about 10 seconds. It then parks for the remaining 20 seconds.

The reported actualTps is 4,920 ops / 10 s = 492 TPS, which appears as 300% efficiency. Total ops are still within the window budget. The measurement looks inflated because the burst happened in a fraction of the window.

The storage cluster sees about 492 ops/s during that 10-second burst instead of a smooth 164 ops/s. On clusters with hard per-second limits, this can still trigger 503 responses even though the TPS accounting is correct. See Tuning guidance.

What the coordinator does next

An agent that reports high utilization is marked hungry on the next heartbeat. The coordinator then raises its allocation gradually, using the 40/60 smoothing. See Coordinator allocation algorithm and the worked example on the TPS Rate Limiting settings page.

The smoothing is intentional. Without it, a 300% agent would jump to maximum allocation immediately and starve other agents.

New agent with very low allocation while running

This is caused by coordinator convergence lag. A previously idle agent starts with near-zero actualTps history. The 60% historical weight in the smoothing pulls its allocation down for about 5 heartbeat cycles (about 2.5 minutes) before it normalizes to its fair share.

During this time the agent shows Running status with a very low Allocated TPS. This is expected and corrects itself.

Idle agents with non-zero allocated TPS

The coordinator keeps the last computed allocation until it receives a new heartbeat from the agent. An agent that just stopped its copy job shows its previous running allocation for up to 30 seconds, until the next heartbeat clears it to idle.

All agents suddenly at maximum allocation

This typically happens after a console restart. The coordinator starts empty and makes a simple equal split based on the count of online agents. Within one heartbeat cycle (30 seconds), actual demand is reported and allocations converge to the correct distribution.

Color coding reference

Utilization	Color	Meaning
Below 80%	Green	Normal. Plenty of headroom.
80–95%	Amber	Approaching the limit. Monitor closely.
Above 95%	Red	At or exceeding the limit. Alarm CV-TPS-001 may be active.

The colors apply to the summary banner, the Total System TPS card, and the Actual TPS column in the per-agent table.

Alarm CV-TPS-001

CV-TPS-001 (TPS Limit Utilization High) is a WARNING-severity alarm.

Behavior	Description
Fires when	Global TPS utilization exceeds the configured alarm threshold (default 95%). A 5-minute cooldown stops it from firing again every heartbeat cycle during sustained high utilization.
Auto-clears when	Utilization drops below 85%. The 10-point gap between the fire threshold (95%) and the clear threshold (85%) is intentional. It prevents rapid fire and clear cycles when utilization bounces around the boundary. The alarm engine checks this condition every 60 seconds.
Fallback	If utilization stays above 85% but no clear event arrives, for example during a console restart, the alarm clears automatically after 10 minutes.

If CV-TPS-001 fires persistently, raise the global limit if the source has headroom, or raise the alarm threshold percentage (default 95%). Both are valid. CV-TPS-001 fires above the threshold percentage of the limit, so lowering the limit makes it fire more.

TPS limiting versus 503 auto-pause

These are separate, complementary mechanisms. Both can be active at the same time.

TPS limiting is proactive. The credit governor keeps aggregate S3 call rates from reaching the storage cluster's performance cliff. A correctly tuned global limit should eliminate 503 responses.

503 auto-pause is reactive. The storage cluster may still return a 503 SlowDown response despite the governor. This can happen during the first 30-second window after a new agent starts, or when the cluster has per-client limits separate from aggregate limits. The agent then backs off for a random holdoff of 30–300 seconds. All running jobs on that agent pause until the holdoff expires.

tip

If you see frequent 503 auto-pause events in the Copy Jobs queue, the global TPS limit is set too high. The auto-pause works as intended, but a well-tuned limit should rarely or never trigger it.

Tuning guidance

1. Start from the agreed replication budget. The budget is what production can give replication, agreed with the storage team. Start 30–40% below that headroom. If you do not know the budget, start conservatively, for example 5,000–8,000 TPS for a medium cluster, and raise the limit in steps.
2. Monitor over several hours of active copy work. Short-window readings can mislead because of the 30-second credit burst behavior. Look at the 30-minute chart trend instead of individual data points.
3. Raise the limit in steps if agents are consistently throttled at low efficiency. If total actual TPS is well below the global limit but agents park often, the per-agent allocation may be uneven. Check whether some agents run at high efficiency while others are idle.
4. If 503 auto-pause events appear, the storage cluster is under stress. Lower the global limit and let the system stabilize over about 5 heartbeat cycles before you assess further. If CV-TPS-001 fires without 503 events, raise the limit if the source has headroom, or raise the alarm threshold percentage.
5. Account for bursts. The 30-second credit model means instantaneous rates can exceed the nominal limit, because bursts are set by thread concurrency, not by the limit. If the cluster has hard per-second limits, keep the global TPS limit well below them.
6. Plan upgrades during low-activity periods. During a rolling upgrade, old agents that do not support TPS throttling run unthrottled and exceed their allocated quota. Disable TPS limiting during the rollout, or upgrade all agents before you re-enable it.