Rate Limiter Pattern
Overview
Suppose one client calls a public search API hundreds of times per second in a loop. Every request queries the database, so that client consumes most of the processing capacity and slows requests from others. Discarding the response afterward does not reduce the load: the database query has already run. Admission must be decided before work begins, based on how much this client has sent recently.
A rate limiter bounds requests admitted during a period and rejects excess requests before processing. The unit for a separate allowance is the partition key, such as an API key, user ID or IP address. These examples give each API key its own allowance, so one client's burst does not consume another's share.
A limiter holds the state for one key and decides whether to admit each request. The example uses a token bucket. It stores tokens up to a capacity, spends one per request, and replenishes one at regular intervals. No token means rejection. The limiter reads the current time from a clock supplied from outside, letting examples and tests control time without waiting.
Rejection is returned explicitly rather than silently dropping work. HTTP uses 429 Too Many Requests and a Retry-After header indicating when to retry. A client given no reason may immediately retry and increase the load.
That protects a server. In the other direction, clients can regulate outbound traffic to stay below an external service's limit. Microsoft's Azure Architecture Center calls the latter Rate Limiting,1 and describes rejecting requests exceeding per-user allowances as one strategy in the Throttling pattern.2 Both directions use the same limiter, so this article covers both, with server-side examples.
Structure
RateLimitGate has one TokenBucketLimiter per API key. A bucket reads IClock to replenish tokens and returns a Decision containing admission and the wait until another attempt.
At t=300 ms, alice's request has no token and is rejected before reaching actual processing. The gate rounds the bucket's calculated 700 ms up to whole seconds and sends Retry-After: 1.
History
Token buckets appear in network traffic-control specifications. RFC 1363 (1992) defines Token Bucket Rate and Token Bucket Size as flow-specification fields. Tokens accumulate in an imaginary bucket at a fixed rate. A host must withdraw tokens equal to the packet size before sending; otherwise it waits for tokens. New tokens are discarded when the bucket is full. The same specification bounds data sent in any interval T to B + (R * T) bytes, where B is bucket size and R is rate.3 RFC 2215 (1997), defining common characterization parameters for Integrated Services network elements, specifies TOKEN_BUCKET_TSPEC as a token-bucket specification augmented with values such as peak rate. The token-bucket part consists of rate r and depth b.4 The single-rate three-color marker in RFC 2697 (1999) starts with full C and E buckets and refills both at the common CIR rate, filling C first and then E. It marks packets green, yellow or red according to whether enough tokens remain for their size.5
RFC 6585 added 429 Too Many Requests to HTTP in 2012. It means the user sent too many requests in a period. A response MAY include a Retry-After header indicating how long to wait. The RFC leaves user identification and request counting to the server.6 Under HTTP Semantics, RFC 9110, Retry-After is either an HTTP date or a non-negative integer number of seconds.7
In a 2017 engineering post introducing four limiters used by its API, Stripe said that its request-rate limiter used the token-bucket algorithm implemented with Redis.8 A 2022 .NET Blog post announced built-in rate limiting for .NET 7: ConcurrencyLimiter, TokenBucketRateLimiter, FixedWindowRateLimiter and SlidingWindowRateLimiter in System.Threading.RateLimiting, together with ASP.NET Core middleware in Microsoft.AspNetCore.RateLimiting.9
Why use it?
Separate allowances prevent one user from monopolizing capacity. Excess requests are rejected before actual work such as database queries, making rejection cheaper than processing. Azure's Throttling document likewise recommends rejecting as early as possible in the request pipeline.10 Receiving 429 and Retry-After removes the need for a client to guess when to resend.
On the client side, limiting outbound traffic below a service's allowance reduces rejections and retransmissions. Azure's Rate Limiting document gives an example where sending 10,000 records and retrying only rejected records ends up sending 30,000 records before completion.11
If concurrency or CPU is the bottleneck, a request-count limit does not directly protect that resource. Consider a bulkhead or concurrency limiter that bounds calls currently running.12 A rate limiter is also not a complete defense against distributed denial-of-service attacks.13
With several servers, decide where counters live. Per-server counters are fast but undercount a caller distributed across replicas. Shared storage such as Redis sees all requests but adds latency to every decision.14 Shared storage can still race if one decision uses separate commands. Redis documents the case where INCR increments a counter but EXPIRE is missed, leaving a key that never expires, and closes the gap by combining both in one Lua script.15 The key also changes the outcome. Trusting an unvalidated client forwarding header as an IP lets callers rotate keys to evade limits. Users sharing one public IP can consume one another's allowance. Creating a limiter for every unvalidated value also grows memory indefinitely.16
Choosing an algorithm
Even the same N requests per second can admit different bursts depending on the counting algorithm. Azure's Throttling document compares four:17
Fixed window: reset the counter at fixed boundaries such as 0 and 1 second. Simple, but it admits consecutive bursts immediately before and after a boundary.
Sliding window: move the window with time to reduce boundary effects, at the cost of more state.
Token bucket: admit a burst up to capacity, then enforce the replenishment rate.
Leaky bucket: release work at a constant rate, in front of a backend requiring steady arrivals.
In .NET, FixedWindowRateLimiter, SlidingWindowRateLimiter, TokenBucketRateLimiter implement the first three, while ConcurrencyLimiter limits concurrent execution. The ASP.NET Core middleware (AddRateLimiter, UseRateLimiter) can use PartitionedRateLimiter to separate users or IP addresses.18 This middleware's default rejection status is 503.19 Set RejectionStatusCode to 429 to change it, but this alone does not add Retry-After. In OnRejected, read the lease's MetadataName.RetryAfter and round it up to whole seconds for the header.20
Example
Send eight requests to a bucket with capacity three and one token replenished every 1000 ms. An injected FakeClock controls time, so the program never waits.
using System;
var clock = new FakeClock();
// Capacity three; replenish one token every 1000 ms.
var bucket = new TokenBucketLimiter(capacity: 3, refillPeriodMs: 1000, clock);
long[] requestTimesMs = [0, 100, 200, 300, 500, 1300, 1400, 3500];
foreach (var t in requestTimesMs)
{
clock.NowMs = t;
var d = bucket.TryAcquire();
Console.WriteLine(d.Allowed
? $"t={t,4} ms allowed tokens left {d.TokensLeft}"
: $"t={t,4} ms rejected Retry-After: {RetryAfterSeconds(d.RetryAfterMs)} (next token in {d.RetryAfterMs} ms)");
}
// Round Retry-After up to whole seconds; rounding down makes clients return too early.
static long RetryAfterSeconds(long ms) => (ms + 999) / 1000;
interface IClock { long NowMs { get; } }
// A fake clock controlled by the example; no real waiting.
sealed class FakeClock : IClock { public long NowMs { get; set; } }
readonly record struct Decision(bool Allowed, int TokensLeft, long RetryAfterMs)
{
public static Decision Allow(int tokensLeft) => new(true, tokensLeft, 0);
public static Decision Reject(long retryAfterMs) => new(false, 0, retryAfterMs);
}
sealed class TokenBucketLimiter(int capacity, long refillPeriodMs, IClock clock)
{
private readonly int capacity = capacity;
private readonly IClock clock = clock;
private int tokens = capacity; // Start with a full bucket.
private long lastRefillMs = clock.NowMs;
public Decision TryAcquire()
{
var now = clock.NowMs;
Refill(now);
if (tokens == 0)
{
// Also return the wait until the next token.
return Decision.Reject(lastRefillMs + refillPeriodMs - now);
}
tokens--;
return Decision.Allow(tokens);
}
// On a request, refill from elapsed time since the last replenishment; no timer is needed.
private void Refill(long now)
{
var periods = (now - lastRefillMs) / refillPeriodMs;
tokens = (int)Math.Min(capacity, tokens + periods);
// Discard fractional accumulated time when full; a full bucket accepts no further tokens.
lastRefillMs = tokens == capacity ? now : lastRefillMs + periods * refillPeriodMs;
}
}t= 0 ms allowed tokens left 2
t= 100 ms allowed tokens left 1
t= 200 ms allowed tokens left 0
t= 300 ms rejected Retry-After: 1 (next token in 700 ms)
t= 500 ms rejected Retry-After: 1 (next token in 500 ms)
t=1300 ms allowed tokens left 0
t=1400 ms rejected Retry-After: 1 (next token in 600 ms)
t=3500 ms allowed tokens left 1Once the first three requests empty the bucket, only one request per 1000 ms passes. Rejections include the time until the next token, rounded up to whole seconds.
Extended example
Mix alice's requests, bob's requests and requests without a key through a per-key bucket gate. Then compare a fixed window and a token bucket with the same nominal two-per-second limit. The token bucket has capacity two and refills one token every 500 ms. Both receive the same flow crossing the 1000 ms boundary.
using System;
using System.Collections.Generic;
var clock = new FakeClock();
var gate = new RateLimitGate(clock, capacity: 3, refillPeriodMs: 1000);
(long At, string? ApiKey)[] trace =
[
(0, "alice"), (100, "alice"), (200, "alice"), (250, "bob"),
(300, "alice"), (350, null), (400, "bob"), (1300, "alice"),
];
Console.WriteLine("== per-client token buckets (capacity 3, 1 token per 1000 ms)");
foreach (var (at, key) in trace)
{
clock.NowMs = at;
Console.WriteLine($"t={at,4} {key ?? "(none)",-6} -> {gate.Handle(new Request(key))}");
}
// Send the same request flow to two algorithms, both limited to two per second.
var clock2 = new FakeClock();
var fixedWindow = new FixedWindowLimiter(limit: 2, windowMs: 1000, clock2);
var bucket = new TokenBucketLimiter(capacity: 2, refillPeriodMs: 500, clock2);
long[] burst = [900, 950, 1000, 1050, 1400];
int fixedInSpan = 0, bucketInSpan = 0;
Console.WriteLine();
Console.WriteLine("== limit 2 per second: fixed window vs token bucket");
foreach (var at in burst)
{
clock2.NowMs = at;
var f = fixedWindow.TryAcquire();
var b = bucket.TryAcquire();
if (at <= 1050 && f.Allowed) fixedInSpan++;
if (at <= 1050 && b.Allowed) bucketInSpan++;
Console.WriteLine($"t={at,4} fixed window: {Show(f),-16} token bucket: {Show(b)}");
}
Console.WriteLine($"admitted from 900 to 1050 ms: fixed window {fixedInSpan}, token bucket {bucketInSpan}");
static string Show(Decision d) => d.Allowed ? "allowed" : $"rejected {d.RetryAfterMs} ms";
interface IClock { long NowMs { get; } }
sealed class FakeClock : IClock { public long NowMs { get; set; } }
readonly record struct Decision(bool Allowed, int TokensLeft, long RetryAfterMs)
{
public static Decision Allow(int tokensLeft) => new(true, tokensLeft, 0);
public static Decision Reject(long retryAfterMs) => new(false, 0, retryAfterMs);
}
sealed record Request(string? ApiKey);
sealed record Response(int Status, string Text, long? RetryAfterSeconds)
{
public override string ToString() =>
RetryAfterSeconds is { } s ? $"{Status} {Text}, Retry-After: {s}" : $"{Status} {Text}";
}
sealed class RateLimitGate(IClock clock, int capacity, long refillPeriodMs)
{
// One bucket per key keeps a client's burst from consuming another's tokens.
private readonly Dictionary<string, TokenBucketLimiter> buckets = new();
public Response Handle(Request request)
{
// Key authentication is omitted. A bucket for every unvalidated value can grow memory without bound.
if (request.ApiKey is null) return new Response(401, "Unauthorized", null);
if (!buckets.TryGetValue(request.ApiKey, out var bucket))
{
bucket = new TokenBucketLimiter(capacity, refillPeriodMs, clock);
buckets[request.ApiKey] = bucket;
}
var d = bucket.TryAcquire();
if (!d.Allowed)
{
// Return 429 and the retry delay, rounded up to whole seconds, rather than silently dropping work.
return new Response(429, "Too Many Requests", (d.RetryAfterMs + 999) / 1000);
}
return new Response(200, $"OK (tokens left {d.TokensLeft})", null);
}
}
sealed class TokenBucketLimiter(int capacity, long refillPeriodMs, IClock clock)
{
private readonly int capacity = capacity;
private readonly IClock clock = clock;
private int tokens = capacity;
private long lastRefillMs = clock.NowMs;
public Decision TryAcquire()
{
var now = clock.NowMs;
Refill(now);
if (tokens == 0) return Decision.Reject(lastRefillMs + refillPeriodMs - now);
tokens--;
return Decision.Allow(tokens);
}
private void Refill(long now)
{
var periods = (now - lastRefillMs) / refillPeriodMs;
tokens = (int)Math.Min(capacity, tokens + periods);
// Discard fractional accumulated time when full; a full bucket accepts no further tokens.
lastRefillMs = tokens == capacity ? now : lastRefillMs + periods * refillPeriodMs;
}
}
sealed class FixedWindowLimiter(int limit, long windowMs, IClock clock)
{
private long windowStartMs;
private int count;
public Decision TryAcquire()
{
var now = clock.NowMs;
// Windows start at fixed boundaries, 0, 1000, 2000 ms; crossing one resets the count.
var start = now - now % windowMs;
if (start != windowStartMs) { windowStartMs = start; count = 0; }
if (count == limit) return Decision.Reject(windowStartMs + windowMs - now);
count++;
return Decision.Allow(limit - count);
}
}== per-client token buckets (capacity 3, 1 token per 1000 ms)
t= 0 alice -> 200 OK (tokens left 2)
t= 100 alice -> 200 OK (tokens left 1)
t= 200 alice -> 200 OK (tokens left 0)
t= 250 bob -> 200 OK (tokens left 2)
t= 300 alice -> 429 Too Many Requests, Retry-After: 1
t= 350 (none) -> 401 Unauthorized
t= 400 bob -> 200 OK (tokens left 1)
t=1300 alice -> 200 OK (tokens left 0)
== limit 2 per second: fixed window vs token bucket
t= 900 fixed window: allowed token bucket: allowed
t= 950 fixed window: allowed token bucket: allowed
t=1000 fixed window: allowed token bucket: rejected 400 ms
t=1050 fixed window: allowed token bucket: rejected 350 ms
t=1400 fixed window: rejected 600 ms token bucket: allowed
admitted from 900 to 1050 ms: fixed window 4, token bucket 2While alice receives 429, bob passes through his own bucket. The fixed window admits four requests, twice the limit, in the 150 ms from 900 to 1050 ms; the token bucket stops at its capacity of two. That comparison covers only the 150 ms interval. A token bucket can admit its capacity plus replenished tokens over one second. With these settings, requests at 0, 0 and 500 ms all pass within one second; including 1000 ms brings the count to four. The token bucket also admits 1400 ms while the fixed window rejects it, so which admits more depends on the request flow. This gate stores state in a process-local Dictionary, so several servers count separately. It never removes unused key buckets and has no locks, allowing concurrent threads to corrupt the token count. Retry-After is guidance; a client that ignores it and returns too early can receive another 429.
TypeScript example
Translate the keyed gate into TypeScript. Decision is a union discriminated by allowed, so only a rejected decision exposes retryAfterMs.
interface IClock { readonly nowMs: number; }
// A fake clock controlled by the example; no real waiting.
class FakeClock implements IClock { nowMs = 0; }
type Decision =
| { allowed: true; tokensLeft: number }
| { allowed: false; retryAfterMs: number };
class TokenBucketLimiter {
#tokens: number;
#lastRefillMs: number;
constructor(
private readonly capacity: number,
private readonly refillPeriodMs: number,
private readonly clock: IClock,
) {
this.#tokens = capacity; // Start with a full bucket.
this.#lastRefillMs = clock.nowMs;
}
tryAcquire(): Decision {
const now = this.clock.nowMs;
this.#refill(now);
if (this.#tokens === 0) {
// Also return the wait until the next token.
return { allowed: false, retryAfterMs: this.#lastRefillMs + this.refillPeriodMs - now };
}
this.#tokens--;
return { allowed: true, tokensLeft: this.#tokens };
}
// On a request, refill from elapsed time since the last replenishment; no timer is needed.
#refill(now: number): void {
const periods = Math.floor((now - this.#lastRefillMs) / this.refillPeriodMs);
this.#tokens = Math.min(this.capacity, this.#tokens + periods);
// Discard fractional accumulated time when full; a full bucket accepts no further tokens.
this.#lastRefillMs = this.#tokens === this.capacity ? now : this.#lastRefillMs + periods * this.refillPeriodMs;
}
}
type RateLimitResponse = { status: number; text: string; retryAfterSeconds?: number };
class RateLimitGate {
// One bucket per key keeps a client's burst from consuming another's tokens.
readonly #buckets = new Map<string, TokenBucketLimiter>();
constructor(
private readonly clock: IClock,
private readonly capacity: number,
private readonly refillPeriodMs: number,
) {}
handle(apiKey: string | null): RateLimitResponse {
// Key authentication is omitted. A bucket for every unvalidated value can grow memory without bound.
if (apiKey === null) return { status: 401, text: "Unauthorized" };
let bucket = this.#buckets.get(apiKey);
if (bucket === undefined) {
bucket = new TokenBucketLimiter(this.capacity, this.refillPeriodMs, this.clock);
this.#buckets.set(apiKey, bucket);
}
const d = bucket.tryAcquire();
if (!d.allowed) {
// Return 429 and the retry delay, rounded up to whole seconds, rather than silently dropping work.
return { status: 429, text: "Too Many Requests", retryAfterSeconds: Math.ceil(d.retryAfterMs / 1000) };
}
return { status: 200, text: `OK (tokens left ${d.tokensLeft})` };
}
}
function show(r: RateLimitResponse): string {
return r.retryAfterSeconds === undefined
? `${r.status} ${r.text}`
: `${r.status} ${r.text}, Retry-After: ${r.retryAfterSeconds}`;
}
const clock = new FakeClock();
const gate = new RateLimitGate(clock, 3, 1000);
const trace: Array<[number, string | null]> = [
[0, "alice"], [100, "alice"], [200, "alice"], [250, "bob"],
[300, "alice"], [350, null], [400, "bob"], [1300, "alice"],
];
for (const [at, key] of trace) {
clock.nowMs = at;
console.log(`t=${String(at).padStart(4)} ${(key ?? "(none)").padEnd(6)} -> ${show(gate.handle(key))}`);
}t= 0 alice -> 200 OK (tokens left 2)
t= 100 alice -> 200 OK (tokens left 1)
t= 200 alice -> 200 OK (tokens left 0)
t= 250 bob -> 200 OK (tokens left 2)
t= 300 alice -> 429 Too Many Requests, Retry-After: 1
t= 350 (none) -> 401 Unauthorized
t= 400 bob -> 200 OK (tokens left 1)
t=1300 alice -> 200 OK (tokens left 0)The output matches the first part of the extended C# example. State resides in one Node.js process's Map, with the same multi-server limitation.
See also
Each C# example is a standalone .NET 10 console program.
Compile the TypeScript examples to JavaScript targeting ES2022 or later, then run them in Node.js.
Footnotes
- Microsoft Azure Architecture Center. Rate Limiting pattern. Introduction. ↩
- Microsoft Azure Architecture Center. Throttling pattern. Per-principal rate limits under Solution. ↩
- C. Partridge. "A Proposed Flow Specification". RFC 1363 (1992). RFC Editor. Token Bucket Rate and Token Bucket Size under Definition of Individual Fields. ↩
- S. Shenker, J. Wroclawski. "General Characterization Parameters for Integrated Service Network Elements". RFC 2215 (1997). RFC Editor. TOKENBUCKETTSPEC. ↩
- J. Heinanen, R. Guerin. "A Single Rate Three Color Marker". RFC 2697 (1999). RFC Editor. Section 3, Metering. ↩
- M. Nottingham, R. Fielding. "Additional HTTP Status Codes". RFC 6585 (2012). RFC Editor. Section 4, 429 Too Many Requests. ↩
- R. Fielding, M. Nottingham, J. Reschke (eds.). "HTTP Semantics". RFC 9110 (2022). 10.2.3 Retry-After. ↩
- Paul Tarjan. Scaling your API with rate limiters. Stripe blog (2017). ↩
- Brennan Conroy. Announcing Rate Limiting for .NET. .NET Blog (2022). ↩
- Microsoft Azure Architecture Center. Throttling pattern. Problems and considerations: Make rejection cheaper than the work that it prevents. ↩
- Microsoft Azure Architecture Center. Rate Limiting pattern. Context and problem. ↩
- Microsoft Azure Architecture Center. Throttling pattern. Problems and considerations: Align throttling limits with the component that saturates first. ↩
- Microsoft Learn. Rate limiting middleware in ASP.NET Core. Prevent DDoS attacks. ↩
- Microsoft Azure Architecture Center. Throttling pattern. Problems and considerations: Decide where the counter resides when one limit spans multiple nodes. ↩
- Redis. INCR. Pattern: rate limiter. ↩
- Microsoft Learn. Rate limiting middleware in ASP.NET Core. Testing endpoints with rate limiting; comments in the Chain limiters in a named policy example. ↩
- Microsoft Azure Architecture Center. Throttling pattern. Pick a limiting algorithm intentionally table under Problems and considerations. ↩
- Microsoft Learn. Rate limiting middleware in ASP.NET Core. Rate limiter algorithms; Rate limiting partitions. ↩
- Microsoft Learn. RateLimiterOptions.RejectionStatusCode. Default: Status503ServiceUnavailable. ↩
- Microsoft Learn. Rate limiting middleware in ASP.NET Core. Choosing what happens when a request is rate limited. ↩