ArticlesAI platform

A 429 is not a bug: governing inference capacity at AWS, Microsoft and Google

The “too many requests” code most often hides an allocation nobody decided. How to govern scarce, shared and expensive compute capacity: who decides, by which rules, at what price, with which risks and which indicators.

Published 3 November 2026

Sources checked between 23 October and 5 November 2026. Model names, tiers, quotas, prices and regulations change quickly: check the cited documentation before any decision.

When a generative AI application calls a model at AWS, Microsoft or Google, the answer can be a refusal: HTTP code 429, “too many requests”. Most of the time nothing is broken: the refusal comes from a rate limit, a spending cap or available capacity, at the provider or in your own platform, and the code alone does not say which. Behind the code lies a management question: compute capacity is scarce, shared and expensive, and a company that does not formalise how it is shared lets its uses fight it out at the moment of saturation.

This article is written for CIOs, platform and architecture leads, FinOpsFinOpsThe practice of managing and optimising cloud spending, bringing finance, procurement and engineering together.See the glossary and risk functions. It explains where 429s come from, then offers what is needed to govern inference capacity: who decides, by which rules, at what price, with which risks and which indicators. The technical detail and the diagnosis are in the appendices.

An incident, to begin with

A fictional scenario, built to illustrate the mechanisms described in this article.

One Monday at 9:05, a company’s internal assistant answers “service temporarily unavailable” to some of its employees. Yet the dashboard shows the model using only 40% of the text volume allowed per minute. The team’s reflex: ask the provider for a higher limit. It does not need one.

The technical logs tell another story. Since 9:00, an automated document job has been sending its requests in batches of several dozen within the same second. Each request is short, so the text volume stays low. But the provider also limits the number of requests per second, and each batch exceeds that limit. And the assistant and the document job go through the same access to the model: employees’ questions that arrive during those seconds are refused. The application retries at once, the software library retries in turn, and the attempts pile up. Finally, the assistant cleanly displays an error message instead of crashing: seen from technical monitoring, everything looks normal.

The fix comes down to four moves, without asking the provider for anything: make the document job wait in a queueInterconnection queueThe list of projects (power plants, storage, large consumers) waiting to be connected to the transmission grid.See the glossary , reserve part of the capacity for the assistant, cap the total number of retries while respecting the delay indicated by the provider (Retry-After), and track refusals by origin. But these moves assume a decision nobody had made: which of the two uses should go first when capacity runs short. That decision, and all those around it, is what this article calls governing inference capacity.

The essentials in seven ideas

1. LLM rate limiting is no longer a technical safeguard, it is a market.
The scarcity of compute is physical, its allocation is commercial: guaranteed throughput is bought, priority is paid for, patience is rewarded.
2. A 429 is often the symptom of an allocation nobody decided.
Before asking for more quota, you need to know who consumes what, by which path, and who should have gone first.
3. The quota does not see what you see.
At AWS on bedrock-runtimeIsolated runtimeThe environment an agent runs in, separated from others and limited to its scope, to contain errors and misuse.See the glossary , and at Microsoft at least for PTUs, it judges an estimate that includes the requested output cap; with pay-per-tokenTokenTwo meanings. For a model: a piece of a word, the unit that measures processed text and therefore cost. In security: a temporary key proving an access right.See the glossary , the bill covers the tokens actually processed, with PTUs the deployed capacity. Consumption metrics are therefore not enough to explain refusals.
4. Governing starts with naming the decision-makers.
Who classifies a use, who sets its floors and ceilings, who opens a subscription, who reserves capacity, who authorises a flow outside the EU.
5. Reserving capacity is a financial commitment, not just a search for a discount.
It is paid for whether it is used or not; the decision belongs to finance, on a business case.
6. Dependence on a provider, a model or a region is an enterprise risk.
For financial entities, DORA makes it an obligation: a register of arrangements, a concentration assessment, a tested exit strategy.
7. The provider manages physical scarcity, you have to manage yours.
With agentsAgentA program that uses an AI model to carry out a task: it reads information, calls tools and suggests or performs actions.See the glossary , one request becomes dozens of calls: per-run budgets, criticality classes and indicators followed by management protect critical uses when capacity drops.

Why this has become a management issue

In 2026, providers no longer hide that capacity is rationed. In late June, the Financial Times reported that Google had capped Meta’s use of Gemini because it could not deliver the capacity requested; other customers were reportedly affected to a lesser degree (Rappler/Reuters). On the AWS side, Matt Garman said in May that supply would remain constrained (Handelsblatt). When one of Google’s largest customers gets capped, no quota can be taken for granted.

Physical scarcity, commercial allocation. The constraint is first material: GPUsGPUGraphics processing unit: the massively parallel chip used to train and run AI models. It is what draws and heats the most.See the glossary , memory, network, cooling and above all electricity. An AI data centerData centerA building that hosts servers, storage and networking, powered and cooled continuously, day and night.See the glossary can be built in a few months; a grid connectionGrid connectionHooking a site up to the public power grid, with the guaranteed capacity that comes with it. It takes years to obtain and is often the scarcest asset.See the glossary takes five years on average in the United States. Europe accounts for only about 5% of the world’s AI compute capacity, against nearly 80% for the United States, and that European capacity mostly serves inference (companion article). But how this capacity is shared between customers, and how its priorities and prices are set, remains a commercial decision. Quotas, tiers and capacity 429s express both at once.

For a company, rate limiting touches four stakes at once:

Availability.
An assistant, an agent or a processing chain that receives 429s can fail for the user even though nothing is “broken”, unless a queue, a retry or a degraded mode absorbs the refusal. On a well-sized platform these refusals remain rare, but they come in bursts, over a few minutes and often on the smallest deployments.
Cost.
Every provider now sells several prices for the same model: priority, standard, deferred, reserved. Choosing badly means paying for capacity you do not use, or suffering refusals on what you paid for.
Compliance.
The most effective lever for gaining throughput is to let the provider process your data elsewhere. Throughput and data residency are traded against each other.
Internal fairness.
A provider quota is shared by every application that uses it. Without an allocation, the noisiest application decides for the others.

Rate limiting is therefore an architecture and governance decision, not an SDK setting. The provider manages a physical scarcity; the company manages an organisational one: deciding which applications and which agents must keep running when capacity runs short.

What this article adds. Taken separately, the mechanisms of rate limiting are already documented, often by gatewayGatewayA single checkpoint placed in front of services, controlling everything that goes in and out: identity, rights, quotas, logging.See the glossary vendors (Requesty, TrueFoundry, Palo Alto Networks) or platforms (Palantir), and the FinOps Foundation covers the choice between shared and reserved capacity. What is missing is the link between the mechanics and the decision: that link is what this article builds.

Read alsoThe gigawatt and the dollarThe speed gap between AI and the grid also exists in dollars, and it is bridged the same way: with other people’s money.

Understanding the mechanism

Admission versus billing. This is the most important and least understood point. At AWS (bedrock-runtime), the provider decides whether to accept a request on arrival, based on an estimate: promptPromptThe written instructions given to the AI model to steer its answer.See the glossary size plus the requested generation cap (max_tokens). It bills afterwards, on actual consumption. A request that asks for 4,000 output tokens and produces 200 has first locked quota for 4,000 (AWS, token counting); Microsoft’s Dev ProxyReverse proxyA server placed in front of others that receives every request and forwards it while applying rules.See the glossary documentation gives the same example for Azure (Microsoft, LLM rate limits in Dev Proxy). Consequence: billed-token metrics are not enough to explain a 429.

At admission: an estimate

prompt
the size of the request
+ 4,000
the requested output cap (max_tokens), locked in the quota

At billing: the actual

prompt
the same request
+ 200
the tokens actually produced, the only ones billed
The case where admission relies on an estimate, as at AWS on bedrock-runtime. Appendix A details the variants.

The rule is not universal. The Anthropic APIAPIThe interface that lets two pieces of software exchange data automatically.See the glossary used directly counts only the output tokens actually generated: a high max_tokens does not reduce the limit there (Claude Platform, rate limits). At Microsoft, the official pages contradict each other for Standard deployments: the Dev Proxy documentation includes max_tokens in the estimate (Microsoft, LLM rate limits in Dev Proxy), the quota management page counts only actual tokens (Microsoft, quota management); for PTUs, the estimate includes it before being reconciled. Google does not document its role. Counting depends on the provider, the endpoint and the deployment type: lowering max_tokens is a remedy only where it enters the estimate, and a simple test will tell you.

The units.

TPM (tokens per minute):
the limit that bites first for LLMs. It counts input and output, sometimes separately, sometimes with a multiplier on output.
RPM (requests per minute):
in decline. AWS has removed it for some of its models, Microsoft derives it mechanically from TPM. It remains decisive for ancillary models (images, OCR), often capped at a few dozen requests per minute.
Capacity units:
at Microsoft, you do not allocate TPM but units, converted into TPM and RPM according to a ratio specific to each model.

The windows. A “per minute” limit is often enforced over 1 to 10 seconds. Microsoft documents it: a deployment at 600 RPM can refuse requests as soon as it receives more than 10 in the same second. Likewise, a limit of 500 requests per 10 seconds does not allow a burst of 3,000 in one second, even though it is equivalent to 3,000 per minute.

Two families of 429. A quota 429 says you have exceeded your allocation. A capacity 429 says the shared pool is saturated, whatever your quota. Confusing them leads to the wrong remedy: asking for a quota increase when the problem is contention.

The 429 is not always the dominant error. In the April 2026 traffic of the Requesty gateway, 65.8% of errors returned by providers were 429s, against 19.4% of 400s and less than 5% of server errors (Requesty). But that is the measurement of a gateway pooling many customers, often close to their limits. On an enterprise platform with generously sized deployments, the field often shows the opposite: rare 429s, concentrated over a few minutes and a few small deployments, buried among far more numerous request errors. Before building an anti-429 strategy, measure your own error breakdown.

The seven levels: factors to examine, not steps

Seven factors can limit an LLM request. They are not seven successive steps each issuing its own 429: they include components, rules, allocation parameters and a physical constraint, each with its own unit. They are not all of the same nature and fall into three families: traffic control mechanisms (levels 1 to 4: your code, your limiter, your gateway, the provider quota), capacity allocation policies (levels 5 and 6: the service tier and the geographic scope, which you choose and the provider applies) and the provider’s physical constraint (level 7). This is the reading grid for the restRESTThe most common style of web API: data is exchanged through addresses and simple verbs (read, create, update, delete).See the glossary of the article.

Seven factors to examine, three at home and four at the provider

Components, rules, allocation parameters and a physical constraint, each in its own unit

At home: you set it

  1. 1. Application instancetraffic controlLimiter in the code: counts requests, multiplies with instances
  2. 2. Distributed limitertraffic controlCounter shared between instances: must count in weighted tokens
  3. 3. Gatewaytraffic controlBudgets per consumer: only sees the traffic that goes through it

At the provider: you choose, it applies

  1. 4. Provider quotatraffic controlTPM and RPM per model: counting specific to the service and deployment
  2. 5. Service tierallocation policyStandard, Priority, Flex, reserved: sets your place in the queue
  3. 6. Geographic scopeallocation policyRegion, zone or global: sets available capacity and services
  4. 7. Physical capacityphysical constraintThe provider’s GPUs: contention in the shared pool, beyond your control
LevelWho controls itUnitWhat breaksLever
1. Application instanceYou (code)HTTP requestsMultiplies with the number of instances; ignores prompt sizeMove to level 2 as soon as there is more than one instance
2. Distributed limiterYou (Redis or equivalent)Requests or tokensCounting in requests when the provider counts in tokensCount in weighted tokens, output included
3. GatewayYou or your platformTokens per key, budgetsSeparate counters per gateway; paths that bypass itBudgets per consumer, routing between backends
4. Provider quotaProvider, adjustableTPM / RPM per model, account or subscriptionCounting specific to the service and deployment, sometimes on an estimate (max_tokens included)Size max_tokens, request a justified increase
5. Service tierYou choose, the provider appliesStandard, Priority, Flex, reservedLower priority under heavy demandOne tier per type of flow
6. Geographic scopeYou chooseRegion, zone, globalAvailable capacity and servicesThe regional lever
7. Physical capacityProvider onlyGPUs, compute timeContention in the shared poolReserved capacity, multi-provider

Levels 1 and 3 give a false sense of control. LangChain’s built-in limiter only applies to one process and only counts requests (LangChain, rate limiters); the llm-token-limit policy of Azure API ManagementAPI managementThe platform that publishes, secures, documents and monitors a company’s APIs (MuleSoft, Kong, Azure API Management…).See the glossary counts separately on each gateway (Azure, llm-token-limit policy). In both cases, several instances multiply the budget, and a gateway only protects the traffic that goes through it.

What changes from one provider to another

The three providers share these mechanisms but apply them differently. The table keeps only the differences that change a decision; the per-provider detail, endpoints, coefficients and paid options, is in appendix A.

CriterionAWS BedrockMicrosoft FoundryGoogle Agent Platform
Quota scopeAccount, model, source regionSubscription, model, deployment typeOrganisation, model
What you setNothing per deployment: account quotasCapacity units per deploymentNothing in Standard: your tier depends on your spend
Main unitTPM input + output (runtime); input and output separate (mantle)TPM, derived RPMPriority threshold in TPM
Role of max_tokensReserved at admission, refunded at the end of the requestContradictory documentation in Standard; counted then reconciled in PTUNot detailed in the docs
Counting trapBurndown of ×5 to ×15 on output, depending on the modelRPM windows of 1 to 10 sPer-second peaks limited
Scaling upIncrease request (prioritised if usage is real)Automatic quota tiersUsage tiers on 30 days of spend
Reserved capacityReserved tier, Provisioned ThroughputPTUProvisioned Throughput (GSU)
Paid priorityPriority (+75%), depending on the modelPriority processingPriority PayGo
Economy optionFlex (-50%), Batch, depending on the modelFlex processing, BatchFlex PayGo (~-50%), Batch
Multi-region routingGeographic and global profilesGlobal and Data Zone deploymentsGlobal endpoint

The regional lever, and the price of residency

Geographic scope is the cheapest throughput lever: it adds no routing fees, and at AWS the global profile even costs about 10% less than geographic profiles. It is paid for in control over where your prompts are processed. All three providers offer the same ladder: a region, a zone (EU, US), the world. The higher you go, the more capacity you generally get, depending on the model and the offer; the lower you go, the simpler compliance becomes.

Microsoft puts it bluntly: Global offers the highest default quotas and receives new models first; regional types come last, with no guaranteed date, and depend on capacity freed up by older models (Microsoft, deployment types). The reason is material: pooling several regions makes it possible to use free GPUs wherever they are.

RungAWS BedrockMicrosoft FoundryGoogle Agent Platform
RegionSingle-region callStandard, Regional ProvisionedRegional endpoint
EU zoneeu. profile: European destinations, which depend on the source regionData Zone EU: processed within the EU Data BoundaryEU multi-region endpoint
Worldglobal. profile: all commercial regionsGlobal Standard, Global ProvisionedGlobal endpoint
List of regionsFrozen: any addition gives a new profile identifierMicrosoft can add regions to a zone without noticeGuarantees published model by model
Specific trapAn SCP blocking one destination makes the request failThe EU Data Boundary may include Norway and SwitzerlandOutside the listed regions, no guarantee of processing location

Sources: AWS, geographic profiles; Microsoft, deployment types; Google, data residency.

The EU zone is often a good compromise for a European company, depending on its residency obligations and model availability: more capacity without taking data out of the zone, provided you check what it covers: Norway and Switzerland in Microsoft’s EU Data Boundary, destinations that vary with the source region at AWS. At Microsoft, the quota is pooled for the whole zone; at AWS, the list of regions in an eu. profile never changes without a new identifier.

The price of residency is paid mostly in services. Priority, Flex or Batch often disappear in the EU zone or in a region: anti-429 levers live mostly in global mode. The per-provider detail, and the cases of Sweden Central and of models without a European profile, are in appendix C.

Two warnings. Multi-region routing increases throughput, not resilience: Microsoft states that an outage of the primary region affects all traffic routed there. And residency is not sovereigntySovereigntyKeeping control of your data and tools: hosting in Europe, applicable law, no dependency on a foreign provider.See the glossary : the AWS European Sovereign Cloud, opened in January 2026 with Bedrock, offers one more step, at the cost of an isolated capacity pool and a narrower catalogue (AWS, European Sovereign Cloud).

Who decides what

Without explicit governance, inference capacity has no owner. The platform team creates deployments, each project requests quota increases as it goes, finance discovers the bill afterwards, and nobody arbitrates when two uses want the same capacity at the same time. The first act of governance is to name who decides what.

DecisionWho proposesWho decidesOn what basis
Classify a use: critical, standard or deferredThe business owner of the useThe AI platform lead, with the businessThe impact of an outage, the service objective
Set an application’s floor and ceilingThe platform teamThe platform lead; management in case of conflict between business unitsMeasured consumption, criticality
Request a quota increase, open a subscription or an accountThe platform teamThe platform lead, with finance informedThe checks in appendix F, the cost of tracking one more envelope
Reserve capacity (PTU, Reserved tier, GSU)The platform team and FinOpsFinance, on a business caseExpected utilisation, commitment length, alternatives
Authorise a flow outside the EU zoneThe businessAuthorised management, after advice from the DPO and complianceThe nature of the data, residency obligations
Add a provider or a modelArchitectureThe AI committee, with procurement and riskSubstitutability, concentration, exit cost
Set agent budgetsThe business and the platform teamThe platform leadCost per run, criticality

Titles vary from one organisation to another; this table is the author’s proposal, not a standard. What matters is that each decision has a single decision-maker and a written rule, and that the platform can enforce it. A rule nobody wrote applies anyway: the noisiest application sets it, at the moment of saturation.

The example of extra subscriptions. Opening one more subscription or account is often the fastest way to gain throughput, since each brings its own quota envelope (see “The means”). Decided by the platform, it is a lever. Left to each project, it multiplies the envelopes to track, the keys to protect and the bills to reconcile, without anyone knowing how much capacity the company actually holds.

Allocation rules: who goes first

Once the decision-makers are named, they need rules. The provider allocates scarce capacity by its own rules; the company must decide which uses go first. No organisation can let each application decide alone what it consumes.

In the shared pool, Standard traffic comes after contractually reserved capacity and after priority traffic; beyond that, providers’ allocation rules are not public. What remains predictable is what you buy and what you allocate yourself. Governing, in practice, means setting four rules:

Criticality classes,
each tied to a provider service tier: critical (reserved capacity, Priority), standard (Standard), deferred (Flex, Batch).
Floors and ceilings per application.
A floor reserves part of the envelope for a critical application, within the capacity the provider actually delivers; a ceiling stops a secondary application from consuming everything.
Residency as an attribute of the flow, not of the whole platform.
Only the flows that require it stay in the EU zone; the others can benefit from global mode and its levers.
Internal commitments.
Each class has its availability and latency objective, and the platform measures whether it is met.

Palantir documents a similar model, with three levels: the overall envelope per model, limits per project and limits per user. Its example makes the floor concrete: on an envelope of one million TPM, a production application can reserve 250,000 TPM, which it consumes first, before drawing on the 750,000 shared; 20% of the envelope also stays reserved for interactive requests, batch jobs being unable to exceed 80% (Palantir, LLM capacity management).

The goal is not to push as many requests through as possible, but to protect critical uses within the capacity actually available, with service objectives and fallback mechanisms, even when everyone shares the same quota. This is a governed allocation of inference capacity, and it does not depend on any provider.

Money: reserve or pay as you go

Inference capacity can be bought in three ways: per token, with variable spend and no guarantee; as reserved capacity, with fixed spend; with priority or discount options (Priority, Flex, Batch), which change the price of a token according to urgency. Choosing between them is not a technical setting: it is an investment decision.

Reserving means committing. The FinOps Foundation speaks of “a strategic commitment rather than a simple discount mechanism”: reserved capacity is paid for continuously, whether it is used or not (FinOps Foundation, GenAI capacity options). In Microsoft’s worked example, with the price assumptions used, PTU and pay-per-token cost roughly the same at full utilisation, and PTU is more expensive per token below that (appendix B). The break-even point must be recalculated for the actual model, contract and load. The FinOps Foundation even cites a reserved offer whose price per token is 67% above standard at full utilisation: you take it for a latency or availability commitment, not to save money.

Four questions before reserving:

1. Is the traffic steady?
A stable, high load makes a reservation pay off; bursts leave it unused most of the time.
2. Does the use require a service commitment?
If a critical use can tolerate neither refusals nor variable latency, a reservation is justified even at a higher price.
3. How long will the model remain the right one?
At AWS, a reservation is tied to a model; at Google, to a publisher, and it cannot be reduced; at Microsoft, PTUs can remain without a deployment if capacity is lacking for the new model (same source).
4. Who carries unused capacity?
The FinOps Foundation calls allocated but unused capacity “the most common form of waste”. Without an owner, it appears nowhere.

Make consumption visible, then charge it back. A shared quota is a common good: without visibility per business unit, everyone consumes at no apparent cost. The first step is to show each business unit what it consumes, in tokens, in euros and in reserved capacity held for it. ChargebackRebillingCharging each team the real cost of its AI usage, to manage the budget.See the glossary comes next, once the allocation rules are stable. It requires every call to carry the identity of its consumer, which only a gateway covering all routes can guarantee.

A throughput cap is not a spending cap. In April, a developer reported on the Google forum a bill of €54,000 in 13 hours, caused by an unrestricted key used to call Gemini (Google AI forum). Some providers offer a monthly cap, like the Anthropic API used directly, which then returns a 429 with no retry delay (appendix F). Elsewhere, the company must set its own budgets and spending alerts, per business unit and per key.

Risk: concentration, reversibility, residency

When an assistant or an agent becomes indispensable to a business unit, its dependence on a provider, a model and a region becomes an enterprise risk. The 429 is its most visible sign: it shows what happens when the only available capacity gives way.

RiskThe signalThe countermeasure
Concentration on a providerAll critical uses at the same oneA second provider tested for these uses, and a router able to switch to it
Concentration on a modelNo replacement model evaluatedA fallback model validated per critical use, with its own quality tests
Concentration on a regionAll traffic in one region, sometimes the only one offering the model (appendix C)An EU zone or a second region, after checking the catalogue
Contractual lock-inA long reservation, tied to a model or a publisherShort commitments and partial coverage
SprawlSubscriptions, accounts and keys opened project by projectA central inventory; opening decided by the platform

Reversibility has a cost that must be paid before the incident. Changing model or provider does not happen in a day: prompts, evaluations, call formats and reservations are often specific to one model. A fallback that has never been tested is not one. Remember too that multi-region routing increases throughput, not resilience: an outage of the primary region hits all the traffic routed there.

For finance, it is an obligation. The DORA regulation has applied to EU financial entities since 17 January 2025 (CSSF). It requires them to keep a register of all arrangements covering ICT services and, for services supporting critical or important functions, documented and tested exit strategies (DORA, article 28). Before contracting, they must also assess concentration risk, notably when services are not easily substitutable or when several such arrangements are made with the same provider (DORA, article 29). An assistant that supports a critical function and relies on a single model at a single provider falls within this frameworkFrameworkA development toolkit that provides an application’s structure; for agents: LangGraph, Strands, CrewAI…See the glossary ; outside the financial sector, the same grid remains good discipline.

Residency is a trade-off, not an absolute. Requiring EU processing for every flow loses anti-429 levers (Priority, Flex, sometimes entire models; appendix C). Requiring global mode for every flow can breach the obligations of some. The decision is taken flow by flow by the authorised manager, after advice from compliance and from the DPO where personal data is involved; the platform enforces it as a routing constraint.

Agents: the run budget as an enterprise rule

With an agent, the unit that counts is no longer the request but the run. Take a request that triggers a main agent, which delegates to three specialised agents each making five calls to the model: that is 16 calls for a single request. If each refused call is retried twice, the worst case rises to 48. Ten simultaneous users are then enough to send several hundred calls in a burst on the same quota, while “user” traffic looks modest.

This is not theoretical. HiveMind describes coding agents launched in parallel on the same API that fail even though capacity was sufficient sequentially (arXiv 2604.17111). Uncoordinated retries worsen the contention they try to get around.

Control must therefore cover the whole run. The values below are orders of magnitude to adjust, not measured recommendations.

ControlExampleWhat it prevents
Token budget per run100,000 tokens per workflowWorkflowA sequence of steps defined in advance. Some steps can be handed to an AI model, but the order stays fixed.See the glossary A loop that exhausts everyone’s quota
Delegation depth5 levels at mostAgents delegating to each other endlessly
Parallelism4 simultaneous sub-agentsA burst that saturates the pool
Retry budget3 cumulative retries per runRetries multiplying at each level
Business priorityA customer-service agent goes before a document jobA background job blocking a critical use
Monetary budget€2 per runA cost drift invisible in the quotas
TimeoutStop after 60 secondsA run that monopolises capacity

Where to enforce these bounds. The gateway sees each call, but not the execution tree that connects them. Per-run bounds therefore live in the agent runtime. For the gateway to check or complement them, each call must carry a run identifier propagated end to end: it is what makes it possible to add up the tokens of one request, cut a run that exceeds its budget, and link a 429 to the agent that caused it.

Degrade rather than fail. When the budget is reached or the provider refuses, an agent can switch to a lighter model, reduce its depth, put a sub-task in a deferred queue or return a partial answer flagged as such. These choices are decided in advance, by criticality class, not at the moment of the incident. And monitoring must follow the business run, not only the transport (appendix E, error no. 12).

Steering: what management needs to see

An allocation policy that is not measured exists only on paper. Six indicators are enough for a committee to know whether it holds.

IndicatorWhat it tells youWhat it does not tell you
Refusal rate per criticality class, and by origin (application, gateway, provider)Whether critical uses are protectedThe cause of a refusal: that is the job of the diagnosis (appendix F)
Service objectives met per class: availability, latencyWhether internal commitments are keptWhether the objective was the right one
Capacity held versus capacity consumed: allocated, reserved, consumedWaste and headroomSaturation: a quota allocated at 100% can be lightly used
Spend per business unit and per use, cost per agent runWho consumes what, and at what priceThe value created by that spend
Traffic split by provider, model and regionThe level of concentrationThe real ability to switch: only a test proves it
Share of traffic bypassing the gatewayThe real coverage of the policyWhat that traffic does

A maturity grid. It helps place an organisation and choose the next step; it is the author’s proposal, with no survey measuring it.

LevelWhat you observe
1. EnduredEach project creates its own deployments; 429s are discovered in production, the bill afterwards
2. VisibleConsumption and refusals measured per application and by origin; inventory of subscriptions, accounts and keys
3. AllocatedCriticality classes, floors and ceilings, budgets per application and per agent, enforced by the gateway
4. ArbitratedNamed decision-makers, reservations decided on a business case, residency chosen flow by flow, regular review
5. ResilientA tested fallback per critical use, an exit strategy, service objectives followed by management

The opening incident illustrates the improvements an organisation can make internally, without asking its provider for anything: making consumption visible, setting priorities and enforcing budgets, in other words moving from level 1 to level 3.

The means: a platform that enforces the policy

The rules above are only worth something if the platform can enforce them, and a counter in front of an endpoint is not enough. A platform that withstands 429s chains five functions: identify and budget at the gateway, admit according to an estimate faithful to the provider’s, order flows by criticality, route to the right model, region or provider, and observe each refusal at its origin. For agents, a sixth function lives in the runtime: bounding the whole run.

A 429 is handled before the provider, not after

Applicationsassistants, APIs, batch jobs
Agent runtimebudget per run, depth, parallelism, retries, timeout
AI gatewayauthenticated identity, budget per consumer, all routes
Admission controlweighted tokens, max_tokens, bounded concurrency, budget in euros
Scheduler
Criticalreserved, Priority
Standardshared quota
DeferredFlex, Batch
Routermodel, region, provider, overflow, degradation
AWSBedrock
MicrosoftFoundry
GoogleAgent Platform
The gateway sees every call; only the runtime sees an agent’s whole run.
A provider 429 goes back up to the router, which decides: retry, overflow or degrade.

Who controls what.

ComponentWhat it controlsWhat it does not see
AI gatewayIdentity, budget per consumer, all routes, refusal logAn agent’s complete run; the state of the provider’s pool
Admission controlTokens weighted as the provider weights them (burndown, max_tokens), concurrency, monetary budgetBusiness criticality, if nobody declares it
SchedulerThe order between critical, standard and deferred flowsWhat happens once the request is sent
RouterModel, region, provider, overflow, degradationResidency rules, unless they are coded as a constraint
Agent runtimeBudget per run, depth, parallelism, retries, timeoutOther applications’ traffic
ObservabilityObservabilitySeeing what happens inside agents: every run, its sources, tool calls, errors and costs.See the glossary 429s by origin, estimated versus billed, retries, queueNothing usable without a propagated request and run identifier

The provider quota sets a usage envelope and admission rules; it says nothing about the allocation between your applications. That is yours to build. A complete policy, per consumer and per model, answers ten questions:

ElementQuestion
IdentityWhich authenticated identity carries the quota?
EnvironmentDo dev, staging and production share the same provider quota?
TargetWhich deployment or profile, in which region?
RequestsHow many calls per window?
TokensWhich budget, weighted as the provider weights it?
ConcurrencyHow many simultaneous calls at most?
QueueHow many calls waiting, and for how long at most?
RetriesHow many attempts, over what total duration, for which errors?
DegradationWhat happens if the counting service goes down: fail open or fail closed?
ConsistencyDoes the sum of budgets stay under the provider ceiling, with a margin?

Implementation principles.

1. A counter shared across replicas and gateways.
A local counter multiplies the budget by the number of replicas (appendix E, error no. 4); you need an external counting service, often backed by Redis.
2. Cover every path.
Either everything goes through the gateway, or the counting service is also consulted by direct calls. The allocation key must include the account, the source region, the model and the inference mode.
3. Bound concurrency, not just throughput.
In streaming, tokens are only known at the end: a budget counted afterwards does not guarantee it will not be exceeded. Bounded concurrency and, if needed, prior reservation with reconciliation complete the setup.
4. Retry properly.
Respect Retry-After and retry-after-ms with the right unit; otherwise exponential backoff with jitter. Bound the number and total duration of attempts, taking the SDK’s retries into account. Never replay a deterministicDeterministicAlways producing the same result for the same input. A classic flow is deterministic; an agent deciding on its own is not.See the glossary error.
5. Use the prompt cache.
At AWS, cache reads do not count in the bedrock-runtime quota; at Microsoft, they reduce PTU sizing. Elsewhere, the documentation talks mostly about billing: check per provider, model and deployment mode.
6. Separate environments, and subscriptions if needed.
Separate deployments already separate counters: at Microsoft, a quota of 240,000 TPM split into two deployments of 120,000 TPM gives two independent limits (Microsoft Learn, managing quota in Foundry). Separate subscriptions or accounts also separate quota envelopes and governance, and each brings its own: a possible lever for gaining throughput, at Microsoft as at AWS, provided increases obtained in staging are carried over to production (AWS Builder Center, managing Bedrock quotas at scale). At Google, multiplying projects within one organisation does not multiply its Standard PayGo threshold. None of these partitions isolates the provider’s physical capacity, and each extra subscription is one more quota to track, which the gateway must be able to locate.

Limits of this analysis

  • Capacity and burndown figures change with every model, as does the availability of Priority and Flex at AWS. Those cited here hold at the date given at the top: reread each model card before building a strategy on them.
  • 429 rates. Forum figures are testimonies; the only public measurement found comes from a gateway (Requesty), on its own traffic. No provider publishes its refusal rate.
  • The twelve field errors describe recurring patterns. We found no survey measuring their frequency; the only related figure comes from HiveMind: on a test bench and under heavy contention, uncoordinated agents fail in 72 to 100% of cases, far more than in the real case that opens the study (3 agents out of 11).
  • The three research papers cited are arXiv preprints; their records do not mention acceptance by a conference or a journal.
  • HyperscalersHyperscalerThe cloud giants that build and run their own data centers at very large scale: AWS, Microsoft, Google, Meta, Oracle.See the glossary ’ real AI capacity in Europe. None of the three publishes the chips installed per region; the 5% share cited above is a floor, not an exact figure, and does not measure the capacity accessible through an API. The companion article details what is known about the European fleet.
  • Providers’ allocation rules. How a provider arbitrates between reserved capacity, priority traffic and Standard is not public, nor is the share of compute rented or owned by each lab.
  • The bounds proposed for agents are illustrative orders of magnitude, not measured values.
  • The proposed governance framework (decision rights, rules, indicators, maturity grid) is the author’s. It draws on practices documented by Palantir and the FinOps Foundation and on the text of DORA, but no survey measures its effect. It is not legal advice on how DORA applies to a given organisation.

And this reading can be proven wrong. If providers return to fixed, guaranteed quotas, if capacity 429s become rare without reserved capacity, or if the gap between Global and the EU zone disappears, then the rationing was cyclical, and part of this article with it.

Conclusion: you do not manage a quota, you manage an allocation

In one year, AWS, Microsoft and Google have turned rate limiting into a market for GPU capacity. The scarcity is physical, its allocation is commercial: guaranteed throughput is bought, priority is paid for, patience is rewarded, spending history influences the priority threshold and geography the ceiling.

But the market is only half the problem. The other half is at home: capacity nobody owns, subscriptions opened as things go, reservations nobody tracks, agents that turn one request into dozens of calls. The question is no longer “how many TPM?” but “who decides the allocation, by which rules, at what price, with which risks, and how do we know?”.

Answering it takes named decision-makers, a written allocation policy, a tracked budget, assessed risks, and a platform that admits, orders, routes and observes to enforce all of it. As long as capacity remains scarce, the 429 will be a design and governance element, not an anomaly. The organisations that understand this will have more reliable applications, more predictable spend and choices that are easier to defend before a finance department, a DPO or a regulator.

Appendix A: per-provider detail

AWS Bedrock: two endpoints, two ledgers

bedrock-runtime (InvokeModel, Converse) applies quotas per model and per region: a single-region TPM, a cross-region TPM and a daily cap. The TPM adds input and output in a single counter, shared by all the APIs calling the same model. Two routes, two applications or two IAM identities in the same account therefore consume the same quota. RPM now exists only for some models (AWS, bedrock-runtime quotas).

bedrock-mantle, which serves the OpenAI-compatible and Anthropic Messages APIs, separates an input quota and an output quota, with no RPM. On arrival, Bedrock adds the input tokens and the requested max_tokens; if that exceeds the input quota, the refusal is immediate. If the output quota runs out during generation, the response is cut off (AWS, bedrock-mantle quotas).

Burndown is the main trap. On bedrock-runtime, one output token consumes several quota tokens depending on the model: 5 for Claude 4.7 and earlier, 10 for Claude Opus 5.5, Sonnet 5, Opus 5 and Fable 5.1, 15 for Claude 4.8, while the bill counts 1 for 1. Official formula: input + cache writes + (output × burndown rate) (AWS, token counting). A raw input + output sum therefore does not reproduce quota consumption.

The rate varies from one model to another: on bedrock-runtime, the GPT-5.6 Sol model card gives a burndown of 10, one output token consuming 10 quota tokens (GPT-5.6 Sol model card).

Cross-region is the native throughput lever: a geographic (eu.) or global (global.) profile routes the request to an available destination region, with a separate quota. Beware: the entry endpoint says nothing about where processing happens. A request entering through a European region with a global. profile can be processed outside Europe. And a single destination region blocked by an SCP makes the request fail (AWS, cross-region inference).

Service tiers are chosen per request via service_tier (AWS, service tiers):

Reserved:
input and output TPM guaranteed over 1 or 3 months, automatic overflow to Standard, 99.5% availability target.
Priority:
priority processing, billed 75% more than Standard.
Standard:
the default behaviour.
Flex:
50% cheaper, deprioritised during periods of heavy demand.

Priority, Standard and Flex share the same on-demand quota. Quota increases go first to customers who already consume their allocation, and none is granted on an end-of-life model.

Beware: these tiers do not exist for every model. Support depends on the model, not the region, and is read card by card. As of 23 October 2026, Claude Sonnet 5.5 offers only the Standard tier (Claude Sonnet 5.5 model card) and GPT-5.6 Sol excludes Priority and Flex (GPT-5.6 Sol model card), while Nova Pro and Grok 4.6 offer both (Nova Pro, Grok 4.6). On these four cards, Bedrock’s priority levers concern secondary models rather than flagship models.

Microsoft Foundry: quota as a budget to allocate

At Microsoft, the quota is a stock of capacity units that you distribute yourself between your deployments. Each unit gives a number of TPM and RPM that depends on the model; RPM cannot be set independently (Microsoft, quota management). Two practical consequences:

Allocated is not consumed.
A quota allocated at 100% blocks the creation of capacity, not traffic (see error no. 1 in appendix E).
Pools are not interchangeable.
The quota for a model in Global Standard does not add to that of the same model in Data Zone. Headroom in one deployment type is not a reserve for the other.

An Azure specificity: refusals count too. Rejected requests are added to the minute’s counter, which makes retries without delay dangerous.

The “invisible” 429 is documented. Standard deployments share a common pool; when demand approaches capacity, the service can temporarily lower a deployment’s effective limit. The signal is the x-ratelimit-limit-tokens header: if it is lower than the configured TPM, an adjustment is under way, which usually clears within a few hours.

Quota tiers automate scaling up. Seven tiers exist, from Free Tier to Tier 6. Moving up a tier is automatic when usage hits the limit, taking into account the commercial relationship and payment history. In practice, some companies do not wait for the next tier: they open extra subscriptions, each with its own quota (see “Who decides what” and “The means”). Since 7 May 2026, for the models concerned, the quota has moved to the subscription level: a single pool for all regions in Global Standard, one pool per zone in Data Zone (Microsoft, quotas and limits).

The paid options:

PTU:
dedicated capacity billed by the hour. Immediate 429 at 100% utilisation, with retry-after-ms; Microsoft presents it as a throttling signal, not an outage (Microsoft, getting started with PTUs).
Spillover:
the overflow of a saturated PTU deployment goes to a Standard deployment of the same resource.
Priority processing:
per deployment or per request, in Global Standard and Data Zone US only (Microsoft, Priority processing).
Flex processing:
50% cheaper, in Global Standard only, with no automatic fallback to Standard (Microsoft, Flex processing).

Google Agent Platform (formerly Vertex AI): a priority threshold, a variable capacity

Since Google Cloud Next on 22 April 2026, Vertex AI has been called Gemini Enterprise Agent Platform (Google, Agent Platform release notes). The API remains aiplatform.googleapis.com. Google is the provider that has most clearly abandoned the fixed per-project quota.

Dynamic Shared Quota and usage tiers. In Standard PayGo, a model’s capacity is pooled between all customers. Each organisation receives a priority throughput threshold set by its tier, itself calculated on its spend over the last 30 days. This threshold is not reserved capacity: under heavy contention, 429s remain possible below it. Above it, traffic can overflow on a best-effort basis. Example from the docs: an organisation in Tier 3 gets a threshold of 10 million TPM on Gemini 2.5 Flash (Google, Dynamic Shared Quota).

Consequence: in Standard PayGo, a 429 most often signals temporary contention rather than a quota reached; other models and Provisioned Throughput have their own limits. In an older Google forum thread, paying customers reported 10 to 15% of 429s over 24 hours, up to 45% at peak, on Gemini 2.0 Flash, a model now superseded (Google Cloud forum).

The consumption options form a clear ladder (Google, consumption options):

Provisioned Throughput:
capacity reserved in GSUs, the only option isolated from the shared pool.
Priority PayGo:
a header is enough, billed at the premium rate. Baseline at organisation level: 10M TPM per Pro model, 50M TPM per Flash model (Google, Priority PayGo).
Standard PayGo:
DSQ and usage tiers.
Flex PayGo:
about 50% cheaper, for jobs that tolerate latency and refusals.
Batch:
asynchronous, with a delay on the order of 24 hours.

Google recommends a hybrid approach: reserved capacity for critical real time, Priority for fluctuating traffic, Standard for the rest, Flex and Batch for offline work. Its first anti-429 lever is the global endpoint, which spreads requests over several regions (Google Cloud, reducing 429 errors).

Appendix B: guaranteed throughput, what the provider must absorb and how to audit it

With reserved capacity, the provider commits to absorbing a throughput. But each measures it in its own unit and converts your tokens according to its own scale. Auditing means recalculating your consumption with that scale and comparing it with the provider’s metrics.

CriterionAWS Bedrock (Reserved tier)Microsoft Foundry (PTU)Google Agent Platform (GSU)
Unit purchasedInput TPM and output TPM, reserved separatelyPTU, usable on any supported modelGSU, a throughput in tokens per second specific to each model
Minimum100,000 input TPM and 10,000 output TPM15 PTU (Global, Data Zone), 25 to 50 (Regional)Depending on the model
Output weightingOutput reserved separately1 output token = 8 input (gpt-5, gpt-5.2), 6 (gpt-5.5), 4 (gpt-4.1)Burndown by modality: 1 text output token = 4 on gemini-2.0-flash
Prompt cacheCache writes count in the reserved TPMCache reads deducted (weighted for GPT-6)Cache reads reduced
Enforcement windowPer minuteLeaky bucket, 429 at 100% utilisation120 s for 1 GSU, 30 s for 25 GSU, 5 s for 250 GSU (gemini-2.5-flash)
Commitment1 or 3 months, fixed price per 1,000 TPMHourly, or a 1-month or 1-year reservationFirm order
Service promise99.5% availability targetLatency: 99% of requests above 50 tokens/s (gpt-5), 80 (gpt-5-mini); a published target, which Microsoft calculates as a p50 over 5 minutesIsolation from the shared pool
Beyond the reserved amountOverflow to StandardSpillover to Standard (optional)Overflow to Standard or Priority

Sources: AWS, service tiers; Microsoft, PTU sizing; Google, PT calculation; Google, PT usage.

Two examples from the docs. At Microsoft, 1,000 requests per minute of 200 input tokens and 20 output tokens on gpt-5.2 give 360,000 normalised TPM (200,000 + 8 × 20,000), or 110 PTU; with 50% caching, 80 are enough. At Google, 10 requests per second on gemini-2.0-flash represent 57,000 adjusted tokens per second, or 17 GSU.

The contract’s traps.

A reservation is not capacity.
At Microsoft, a reservation is a discount on the meter, not a GPU guarantee: you create the deployment first, then buy the reservation (Microsoft, provisioned throughput).
Unconsumed capacity is lost.
In Microsoft’s worked example, which relies on an illustrative price, PTU and pay-per-token cost roughly the same at full utilisation; below that, PTU is more expensive per token. Microsoft states that this calculation does not guarantee the real cost: the break-even point must be recalculated for the model, the contract and the load.
The latency promise is a median.
Microsoft’s target is calculated as a p50 per 5-minute window. It covers neither the tail of the distribution nor the time to first token.

An audit grid in five checks.

1.
Recalculate normalised consumption from your logs and the published scale, then compare with the provider’s counter: Provisioned-managed Utilization V2 (Azure), EstimatedTPMQuotaUsage (AWS), consumed_token_throughput against dedicated_gsu_limit (Google).
2.
Check where the traffic was actually served: trafficType at Google, service_tier in the response at Azure, the ResolvedServiceTier dimension at AWS.
3.
Measure the promised latency: the Tokens per Second metric at Azure, TimeToFirstToken at AWS (AWS, March 2026).
4.
Track the paid utilisation rate, hour by hour, and reduce the reservation below the break-even point.
5.
Distinguish availability and performance in the contract. An availability SLA does not cover degraded throughput.

A point of attention: EstimatedTPMQuotaUsage includes burndown and cache writes, but not the initial reservation linked to max_tokens (AWS, TTFT and quota metrics). A refusal can therefore occur while the metric stays under the quota.

Appendix C: residency, the levers lost provider by provider

The real price of residency: fewer services, not just fewer models. Anti-429 levers live mostly in Global mode.

LeverMicrosoft FoundryGoogle Agent PlatformAWS Bedrock
PriorityGlobal Standard or Data Zone US only; neither Data Zone EU nor regionalGlobal, us and eu endpoints in general; some models in global onlyDepends on the model, not the region: absent on Claude Sonnet 5.5 and GPT-5.6 Sol, present on Nova Pro and Grok 4.6
FlexGlobal Standard onlyPreview models: global endpoint onlySame rule: absent on Claude Sonnet 5.5 and GPT-5.6 Sol
BatchGlobal Batch (world) or Data Zone Batch (EU)Unavailable through the global endpointBatch inference, with its own quotas
Reserved capacityGlobal, Data Zone or Regional PTUFor Claude: regional endpoints onlyReserved tier or Provisioned Throughput
Cost of residencyNo stated premium, but fewer optionsClaude on a regional endpoint: +10%Global profile about 10% cheaper than geographic profiles

Sources: Microsoft, Priority; Microsoft, Flex; Google, Flex PayGo; Anthropic, Claude on Vertex.

The Sweden Central case. Azure has about ten European regions, but Sweden Central concentrates the catalogue and is sometimes the only option, notably for fine-tuned models (Microsoft, models sold by Azure). Microsoft does not explain why; it only highlights Sweden’s abundant, low-carbon electricity. Concentrating everything on this region also creates a single point of failure.

The opposite trap exists at AWS. Claude Sonnet 5.5 has an eu. geographic profile on Bedrock; GPT-5.6 Sol does not: its card offers only a us. profile and a global. profile. To call it from Frankfurt or Paris on bedrock-runtime, you have to go through the global profile, with no guarantee of processing in Europe (GPT-5.6 Sol model card). Before choosing a model, you therefore need to check not only where it is available, but through which type of profile.

Appendix D: what the research says

Three recent papers shed direct light on the proposed architecture. They are arXiv preprints; their records do not yet mention acceptance by a conference or a journal.

Agents, on the client side.
HiveMind starts from a concrete case: 3 coding agents out of 11, launched in parallel on the same API, failed even though capacity was sufficient sequentially. The proposed proxy applies admission, backpressure, a circuit breaker, a token budget and priority queues, which is the core of the architecture described above (arXiv 2604.17111).
Fairness in latency, not just throughput.
A team including Matei Zaharia and Scott Shenker shows that current policies equalise customers’ throughput without guaranteeing per-token latency; their FairInference system bounds that gap (arXiv 2609.18112). It is an argument for criticality classes rather than a simple throughput split.
Isolation in gateways.
KeyPooling shows that gateways pooling provider keys can make customers share the prompt cache (arXiv 2608.17485): a gateway must allocate capacity without breaking isolation.

Appendix E: twelve errors to look for before asking for more quota

Provider docs describe levels 4 to 7. But when you open the hood of an enterprise AI platform, several weaknesses sit at levels 1 to 3, at home. Here are twelve, grouped by family.

Misreading the numbers

1. Confusing allocation and consumption.
A quota allocated at 100% says you can no longer create capacity, not that traffic is saturated. A platform can have several models almost fully allocated while showing a near-zero refusal rate. Conversely, a backup resource can show no visible traffic for days: that is neither proof that it is useless nor a green light to delete it.
2. Looking for the bottleneck in the wrong place.
You watch the large text model, and the refusals come from elsewhere: an OCR, image or embeddingsEmbeddingsA numeric representation of text or images that lets you find content close in meaning.See the glossary model capped at a few dozen requests per minute. A batch of documents sent all at once is enough to saturate it. The right reflex is a queue and bounded concurrency for these ancillary models, not a general quota increase.
3. Classifying every error as rate limiting.
400, 408, 499, 500 and 503 are not 429s. A parameter or format-translation error is deterministic: replaying it only adds traffic.

Counters that do not count what you think

4. Per-process counters.
A middlewareMiddlewareA component that slots into an agent’s loop (before or after the model, around a tool) to add a control without changing the agent itself.See the glossary that stores its counters in memory gives each pod its own budget. Two replicas set to N requests per minute accept 2N; a restart resets everything to zero.
5. Fixed windows that double bursts.
A counter reset every minute lets a full quota through just before the boundary, then another just after. You can thus accept nearly double the limit within a few milliseconds. Sliding-window or token-bucket algorithms avoid this effect.
6. A badly chosen identity.
A limit keyed on the IP address often sees the proxy’s: all users then share the same counter. A limit keyed on a client-supplied header is bypassed by changing the header. A quota’s key must be an authenticated identity.
7. A quota checked after the fact.
Many per-user quotas read past consumption before authorising a call, without reserving the tokens of the call in progress. Ten simultaneous calls all pass under the threshold before the first is counted. And if writing the consumption log fails silently, consumption is underestimated afterwards.

Protections that turn against you

8. Health probes inside the quota.
If the limiter applies to every path, the health route can receive a 429 when the counter is saturated. Depending on the configuration, the orchestratorOrchestrationCoordinating several agents and tools to run an end-to-end process: who does what, in which order, with which checks.See the glossary may then consider the service unhealthy. Technical probes must be taken out of the business quota.
9. A silent local 429.
A refusal with no Retry-After and no distinct error code says neither when to retry nor whether the refusal comes from the application, the gateway or the provider.
10. Implicit retries.
When the code sets neither a number of attempts, nor a total duration, nor an output cap, the SDK’s defaults decide, and they change between versions. A retry on a stream already partly sent, or on an operation with side effects, can duplicate work.

Paths nobody sees

11. Calls that bypass the gateway.
All it takes is a client configured to call the provider’s private endpoint directly, or a “native” route that passes the body through without decoding it, for a policy set only on the gateway to miss part of the traffic. Yet all these paths consume the same provider quota.
12. Monitoring that mixes everything.
An alert on “more than N 4xx errors in five minutes” at cluster level says neither which model, nor which cause, nor which level refused. Worse, an application that cleanly turns a 429 into an unavailability message inside a stream makes the incident disappear from HTTP monitoring: the initial status remains 200.

Appendix F: diagnosing a 429, identify the level before acting

A 429 is not a cause, it is a symptom. Before any remedy, you need to know which level issued it.

What you observeProbable levelWhat to look atRemedy
429 with no provider header, a message from your application1 or 2Application logs, the counter’s keyFix the key, share the counter
429 with a gateway header or message3Gateway policy, the consumer’s budgetReview the budget, check counter sharing
429 with x-ratelimit-remaining-tokens at zero4Admission estimate, max_tokens, burstsLower max_tokens where it counts, smooth traffic, and only then request an increase
429 while token metrics are low4Prompt size, max_tokens, short windowsMeasure the estimate, not the billed amount
x-ratelimit-limit-tokens lower than the configured quota7Temporary adjustment by the providerRetry with delay, overflow, reserved capacity
429 in bursts at a fixed time, across all customers7Contention in the shared poolFirst rule out scheduled jobs and internal bursts; then global endpoint or zone, Priority, multi-provider
429 on a reserved deployment5Utilisation at 100%Spillover, resizing
429 on a single ancillary model4That model’s request capQueue and bounded concurrency
429 with no retry-after, code enforced_spend_limit_reached (Anthropic API used directly)4The tier’s monthly spending capDo not retry: move to the next tier, or wait until the 1st of the following month

What to always collect: the exact error message, the headers (Retry-After, retry-after-ms, x-ratelimit-*), the provider’s request identifier, the service tier actually applied and the number of attempts already made. Metrics aggregated per minute tell you when; only these elements tell you why.

Three measurement traps. Do not add a global series to its per-model sub-series. Do not read the absence of a series as the absence of traffic. And remember that retries inflate the number of refusals: at AWS, the Invocations metric excludes refused requests, which are counted separately.

Sources

  • AWS: bedrock-runtime quotas
  • AWS: bedrock-mantle quotas
  • AWS: token counting and burndown
  • AWS: service tiers
  • AWS: cross-region inference
  • AWS: geographic profiles
  • AWS: TTFT and quota metrics
  • AWS Builder Center: managing Bedrock quotas at scale
  • AWS: European Sovereign Cloud
  • Microsoft: quotas and limits
  • Microsoft: quota management
  • Microsoft: LLM rate limits in Dev Proxy
  • Microsoft: deployment types
  • Microsoft: provisioned throughput
  • Microsoft: PTU sizing
  • Microsoft: Priority processing
  • Microsoft: Flex processing
  • Microsoft: APIM llm-token-limit policy
  • Google: Dynamic Shared Quota
  • Google: consumption options
  • Google: Priority PayGo
  • Google: Flex PayGo
  • Google: Provisioned Throughput calculation
  • Google: data residency
  • Google Cloud: reducing 429 errors
  • Google: Agent Platform release notes
  • LangChain: rate limiters
  • Anthropic: Claude on Vertex
  • Claude Platform: rate limits
  • Rappler/Reuters, citing the Financial Times: Google limits Meta’s use of Gemini
  • Handelsblatt: Matt Garman
  • Requesty: error code breakdown (April 2026)
  • Google AI forum: Gemini billing spike
  • Google Cloud forum: 429 rates on Gemini 2.0 Flash
  • HiveMind (arXiv 2604.17111)
  • FairInference (arXiv 2609.18112)
  • KeyPooling (arXiv 2608.17485)
  • Bruegel: Policy Brief 18/2026 on Europe’s AI compute gap
  • Europe2031.ai: Compute Forecast
  • IEA: Energy and AI (summary)
  • Palantir: LLM capacity management
  • Requesty: 429 Too Many Requests, causes and fixes (updated 29 September 2026)
  • TrueFoundry: rate limiting in an AI Gateway (updated 12 October 2026)
  • Palo Alto Networks: Rate Limiting for LLM Applications and Agents
  • FinOps Foundation: Navigating GenAI Capacity Options (updated 4 February 2026)
  • FinOps Foundation: FinOps for AI
  • DORA: article 28
  • DORA: article 29
  • CSSF: ICT and cyber risk for DORA entities