There is a class of bugs that only happen at midnight.

Your API has a rate limit: 1000 requests per minute. A client hits 999 at 11:59:59 PM, then fires another 999 at 12:00:00 AM. Two windows, two clean counters, all 1998 requests allowed. Your database gets a spike it was never designed to handle, and you spend the next hour wondering how your rate limiter let this through.

It did let it through. You just did not know it.

This is the boundary spike problem, and it is one of about ten things that will go wrong when you design a rate limiter from scratch. The algorithm is the easy part. The hard parts are what happens when two servers see the same request at the same time, what happens when Redis goes down, and what happens when a client figures out they can route around your limits by spreading requests across IP addresses.

This post designs Throttle, a rate limiting service you can embed as middleware or run as a standalone sidecar, the same way I built Relay in the previous post. We will go through every major algorithm, how each one breaks in a distributed system, how to fix those breaks, and how to deploy it.

What Rate Limiting Is Actually For

Most people think of rate limiting as one thing. It is not. It is four different problems wearing the same name.

The first is infrastructure protection. A single client with a buggy retry loop can take down your API. The rate limiter is the wall between that client and your database.

The second is fair use. On a multi-tenant system like Relay, one tenant having a bad day should not become everyone else’s problem. This is resource isolation at the API layer.

The third is monetization. Free tier gets 100 requests per minute. Pro gets 1000. Rate limits are what make pricing tiers real rather than theoretical.

The fourth is security. Slowing down brute-force login attempts, stopping credential stuffing, limiting the blast radius of a stolen API key.

These four problems lead to different design decisions. A security rate limiter needs to stay effective even when Redis is down. A billing rate limiter is fine with a few seconds of lag. If you mix them up, you either overbuild the wrong thing or build the right thing in a way that fails in the wrong direction.

Throttle handles all four. That means multiple limit scopes (per IP, per key, per tenant), multiple time windows (per second, per minute, per day, per month), and a design that fails gracefully instead of all at once.

The Four Algorithms

Before picking an algorithm, understand that every rate limiting algorithm is making a bet about what “fair” means. Some bet on time windows. Some bet on token accumulation. The bet determines what kind of abuse gets through and what kind of legitimate traffic gets punished.

There are four algorithms that actually matter in production. They are not interchangeable.

Fixed Window Counter

The simplest one. Divide time into buckets of fixed length W (say, 60 seconds). Keep a counter per bucket. Add one on every request. Reject when the counter goes over the limit.

1window:  [0s ----------- 60s][60s ----------- 120s]
2counter:  897                  0
3limit:    1000

Implementation in Redis:

 1func (r *RedisLimiter) AllowFixed(ctx context.Context, key string, limit int, window time.Duration) (bool, error) {
 2    now := time.Now().Unix()
 3    windowStart := now - (now % int64(window.Seconds()))
 4    bucketKey := fmt.Sprintf("rl:fixed:%s:%d", key, windowStart)
 5
 6    pipe := r.client.Pipeline()
 7    incr := pipe.Incr(ctx, bucketKey)
 8    pipe.Expire(ctx, bucketKey, window*2) // 2x window so key expires naturally
 9    _, err := pipe.Exec(ctx)
10    if err != nil {
11        return false, err
12    }
13
14    return incr.Val() <= int64(limit), nil
15}

The window start is calculated by rounding now down to the nearest window boundary. Every request in that 60-second bucket hits the same counter.

The boundary spike problem. Fixed windows have one well-known flaw. A client can send limit requests at 23:59:59.999 and limit more at 00:00:00.001, and you just let 2 * limit requests through in 2 milliseconds.

1limit = 1000
2
3[      window 1      ][      window 2      ]
4...            999 req|1000 req ...
5                      ^
6                2 ms, 1999 requests allowed

This is not a hypothetical. It is how you get database crashes even when you think you have rate limiting in place.

When to use it anyway. Fixed window is fine when the boundary burst does not matter much. Monthly quota enforcement is the classic example. If a user burns 1000 API calls in the last second of their billing month and 1000 in the first second of the next, that is a real problem for strict quota. But for most monthly caps, simple is better.

Sliding Window Log

Instead of counting requests per window, you store the timestamp of every request in the last W seconds and count those.

 1func (r *RedisLimiter) AllowSlidingLog(ctx context.Context, key string, limit int, window time.Duration) (bool, error) {
 2    now := time.Now()
 3    windowStart := now.Add(-window).UnixMilli()
 4    logKey := fmt.Sprintf("rl:log:%s", key)
 5
 6    pipe := r.client.Pipeline()
 7    // Remove timestamps older than the window
 8    pipe.ZRemRangeByScore(ctx, logKey, "0", strconv.FormatInt(windowStart, 10))
 9    // Count remaining timestamps (requests in the current window)
10    count := pipe.ZCard(ctx, logKey)
11    // Add this request's timestamp
12    pipe.ZAdd(ctx, logKey, redis.Z{Score: float64(now.UnixMilli()), Member: now.UnixNano()})
13    pipe.Expire(ctx, logKey, window)
14    _, err := pipe.Exec(ctx)
15    if err != nil {
16        return false, err
17    }
18
19    return count.Val() < int64(limit), nil
20}

We use a Redis sorted set. The score is the timestamp in milliseconds. On every request: remove old entries, count what is left, add the current timestamp. The window always looks backward exactly W seconds from right now.

This is the most accurate algorithm. No boundary spike. Every point in time sees the correct count. A client who sends 1000 requests spread across the last 60 seconds will be rejected on request 1001 no matter when it arrives.

The problem is memory. You store one entry per request per user. If a client is allowed 10,000 requests per minute, you store up to 10,000 sorted set members. At 1 million users each with a 1000 req/min limit, that is up to 1 billion sorted set entries. This is why production systems at high volume almost never use sliding log.

Use it for low-volume, high-precision cases: login attempts, password resets, admin APIs where correctness matters more than memory.

Sliding Window Counter

A hybrid. It approximates the sliding window using two fixed-window counters: the current window and the previous one, weighted by how far into the current window you are.

1window = 60s
2current window started 15s ago (we are 25% into it)
3
4estimated count = prev_count * (1 - 0.25) + curr_count * 1.0
5                = prev_count * 0.75 + curr_count

 1func (r *RedisLimiter) AllowSlidingCounter(ctx context.Context, key string, limit int, window time.Duration) (bool, error) {
 2    now := time.Now()
 3    windowSecs := int64(window.Seconds())
 4    currentWindowStart := now.Unix() - (now.Unix() % windowSecs)
 5    prevWindowStart := currentWindowStart - windowSecs
 6
 7    currentKey := fmt.Sprintf("rl:sw:%s:%d", key, currentWindowStart)
 8    prevKey := fmt.Sprintf("rl:sw:%s:%d", key, prevWindowStart)
 9
10    pipe := r.client.Pipeline()
11    curr := pipe.Get(ctx, currentKey)
12    prev := pipe.Get(ctx, prevKey)
13    _, _ = pipe.Exec(ctx)
14
15    currentCount, _ := strconv.ParseFloat(curr.Val(), 64)
16    prevCount, _ := strconv.ParseFloat(prev.Val(), 64)
17
18    // How far are we into the current window? (0.0 = start, 1.0 = end)
19    elapsed := float64(now.Unix()-currentWindowStart) / float64(windowSecs)
20    estimated := prevCount*(1-elapsed) + currentCount
21
22    if estimated >= float64(limit) {
23        return false, nil
24    }
25
26    // Increment current window counter
27    r.client.Incr(ctx, currentKey)
28    r.client.Expire(ctx, currentKey, window*2)
29    return true, nil
30}

This is what Cloudflare uses. Their blog post on rate limiting describes exactly this approach. You get O(1) memory per user (just two counters), near-accurate sliding window behavior, and no meaningful boundary spikes. The approximation error is a few percent at most.

The formula assumes requests were spread evenly through the previous window. If they were all at the very end (a burst), the estimate will be a bit too low. In practice this error is small enough that this is the right default for most production systems.

Token Bucket

Token bucket works differently from the counter-based algorithms. Instead of counting past requests, you maintain a bucket that fills with tokens at a fixed rate. Each request uses one token. If the bucket is empty, the request is rejected.

1bucket capacity = 100 tokens
2refill rate     = 10 tokens/second
3
4t=0:   bucket = 100
5t=5:   100 requests arrive, bucket = 0, all allowed
6t=5.1: 1 request arrives, bucket = 1 (0.1s * 10), allowed
7t=5.1: 1 more request, bucket = 0, rejected
8t=6:   bucket = 9, next request allowed

Bucket capacity controls the maximum burst. Refill rate controls sustained throughput. If someone has not touched the API in 100 seconds and the refill rate is 10 tokens/second, they can burst 1000 requests when they come back, then trickle at 10 per second.

Redis implementation using timestamps (no background thread needed):

 1type TokenBucketState struct {
 2    Tokens     float64
 3    LastRefill int64 // Unix timestamp in milliseconds
 4}
 5
 6func (r *RedisLimiter) AllowTokenBucket(ctx context.Context, key string, capacity float64, refillRate float64) (bool, error) {
 7    // refillRate = tokens per millisecond
 8    bucketKey := fmt.Sprintf("rl:tb:%s", key)
 9
10    // Lua script for atomicity - read-modify-write in one round trip
11    script := redis.NewScript(`
12        local key = KEYS[1]
13        local capacity = tonumber(ARGV[1])
14        local refill_rate = tonumber(ARGV[2])  -- tokens per ms
15        local now = tonumber(ARGV[3])
16
17        local state = redis.call("HMGET", key, "tokens", "last_refill")
18        local tokens = tonumber(state[1]) or capacity
19        local last_refill = tonumber(state[2]) or now
20
21        -- Compute how many tokens have been added since last request
22        local elapsed = now - last_refill
23        tokens = math.min(capacity, tokens + elapsed * refill_rate)
24
25        if tokens < 1 then
26            -- Update last_refill even on rejection so we track time correctly
27            redis.call("HMSET", key, "tokens", tokens, "last_refill", now)
28            redis.call("PEXPIRE", key, math.ceil(capacity / refill_rate))
29            return 0
30        end
31
32        tokens = tokens - 1
33        redis.call("HMSET", key, "tokens", tokens, "last_refill", now)
34        redis.call("PEXPIRE", key, math.ceil(capacity / refill_rate))
35        return 1
36    `)
37
38    now := time.Now().UnixMilli()
39    result, err := script.Run(ctx, r.client, []string{bucketKey}, capacity, refillRate/1000.0, now).Int()
40    if err != nil {
41        return false, err
42    }
43    return result == 1, nil
44}

The Lua script is the key part. Redis runs Lua scripts atomically, so the read, compute, and write happen as a single operation. Nothing else can interleave. This kills the race condition you would get with a WATCH/MULTI/EXEC approach.

Token bucket is what most production systems use for per-key limiting. It is why API clients can send a short burst if they have been idle. The burst is intentional. It handles legitimate traffic spikes without punishing users.

The tradeoff is state. Each key stores two fields: tokens and last_refill. Cheap in Redis, but more than a simple counter. And you have to decide: what does a brand-new user start with? Full bucket. Otherwise new users get rejected before they even get started.

Leaky Bucket

Leaky bucket is the reverse of token bucket. Requests flow in and the bucket drains at a fixed rate.

1requests go in -> [queue] -> processed at 1 req/10ms -> response
2                     |
3                  if full: reject

The bucket is a FIFO queue. Requests enter, a worker drains it at a constant rate. If the queue is full, new requests are dropped. Output is perfectly smooth. You never send more than 1 request per 10ms no matter how bursty the input is.

You use this for shaping traffic toward a dependency, not for user-facing limits. If you are calling an external API that handles exactly 100 req/s and has no burst tolerance, a leaky bucket in front of it smooths your traffic and stops you from getting throttled. Users do not see it because it is between your code and theirs.

For user-facing rate limiting, leaky bucket is usually wrong. Users experience queuing delay instead of a clean rejection, which is confusing. And you have to cap the queue size, which means deciding whether to reject immediately or wait when it fills up. Waiting adds unpredictable latency. This is more of a traffic shaper than a rate limiter.

Choosing an Algorithm

For Throttle:

• Token bucket for per-key and per-IP short-window limits. Configurable burst, O(1) memory, accurate enough.
• Sliding counter for per-tenant limits over longer windows (hourly, daily). Low memory, near-exact, no thundering herd at window boundaries.
• Fixed window for monthly quota only. When the window is 30 days, simplicity wins.

The Distributed Problem

So far the algorithms look reasonable on paper. Pick token bucket, write the Lua script, done. But all of that assumes one Redis instance and one application server. Production is neither, and that changes everything.

When you run multiple API servers behind a load balancer, rate limit state needs to be consistent across all of them. If it is not, clients can bypass the limit just by having their requests spread across servers.

Both servers see 500 requests. Both think the user is under the limit. The user sent 1000 and all went through. If each server has its own local counter with no sync, your “1000 req/min” limit is really 1000 * num_servers in practice.

The fix is shared state. This is why Redis is the standard for distributed rate limiting. Every server reads and writes to the same Redis instance.

Now the counter is shared. But you just added a network hop on every request for the rate limit check. That is on the critical path. Redis is fast (under a millisecond), but it is not free.

The Race Condition

Even with shared Redis, there is a race condition in the naive implementation.

1Server 1: GET counter -> 999
2Server 2: GET counter -> 999    (reads before Server 1 writes)
3Server 1: SET counter = 1000 -> allow
4Server 2: SET counter = 1000 -> allow  (wrong, should be 1001)

GET followed by SET is not atomic. Between the read and the write, another server can jump in. The fix is atomic operations.

For simple counters, INCR and INCRBY are atomic in Redis. A single INCR atomically reads, increments, and returns the new value. No race.

For complex state like token bucket, use a Lua script. Redis runs Lua atomically with no interleaving. This is exactly why the token bucket implementation above uses a script.

There is also WATCH/MULTI/EXEC, Redis’s optimistic transaction mode. You watch a key, start a transaction, read, then exec. If someone else modified the key between WATCH and EXEC, the transaction aborts and you retry. It works, but you need retry logic and it fails badly under high contention.

Lua scripts are simpler and more reliable. Use them.

The Redis Failure Problem

Redis is on the critical path now. If it goes down, every request fails the rate limit check. You have to decide: fail open or fail closed.

Fail open means if Redis is down, let all requests through. The system runs unprotected. You take the traffic risk to stay available. Right choice for non-security limits like billing tiers and usage dashboards.

Fail closed means if Redis is down, reject everything. You stay protected at the cost of total unavailability. Right choice for security limits like login brute force and account lockout.

The hybrid is local fallback. Each server keeps a local in-process rate limiter, just a simple atomic counter or token bucket in memory. When Redis is unreachable, fall back to local limits. Local limits are per-server, so the effective limit becomes local_limit * num_servers, but that beats either zero protection or a total outage.

 1type TieredLimiter struct {
 2    redis  *RedisLimiter
 3    local  *LocalLimiter // in-process token bucket per key
 4    policy FailurePolicy // FailOpen or FailClosed or LocalFallback
 5}
 6
 7func (t *TieredLimiter) Allow(ctx context.Context, key string, limit Limit) (Decision, error) {
 8    decision, err := t.redis.Allow(ctx, key, limit)
 9    if err == nil {
10        return decision, nil
11    }
12
13    // Redis is down
14    switch t.policy {
15    case FailOpen:
16        return Allow, nil
17    case FailClosed:
18        return Deny, nil
19    case LocalFallback:
20        return t.local.Allow(key, limit.ScaledBy(1.0/float64(t.serverCount)))
21    }
22    return Allow, nil
23}

The ScaledBy call matters. If you run 10 servers and the real limit is 1000 req/min, the local fallback limit should be 100 req/min. Across all 10 servers, the aggregate stays at 1000.

The Sliding Window Race

The sliding counter has its own distributed race. The read-modify-write across the current and previous window counters is not atomic.

1// This is NOT safe without additional atomicity
2curr := pipe.Get(ctx, currentKey)  // read
3prev := pipe.Get(ctx, prevKey)     // read
4// ... compute estimated ...
5pipe.Incr(ctx, currentKey)         // write

Another server can increment between your read and your write. Wrap it in Lua.

 1var slidingScript = redis.NewScript(`
 2    local curr_key = KEYS[1]
 3    local prev_key = KEYS[2]
 4    local limit = tonumber(ARGV[1])
 5    local window_secs = tonumber(ARGV[2])
 6    local elapsed_ratio = tonumber(ARGV[3])  -- 0.0 to 1.0
 7
 8    local curr = tonumber(redis.call("GET", curr_key)) or 0
 9    local prev = tonumber(redis.call("GET", prev_key)) or 0
10
11    local estimated = prev * (1 - elapsed_ratio) + curr
12    if estimated >= limit then
13        return 0
14    end
15
16    redis.call("INCR", curr_key)
17    redis.call("EXPIRE", curr_key, window_secs * 2)
18    return 1
19`)

One round trip, atomic. This is the rule: any time you have a read-compute-write on rate limit state, put it in Lua.

Who to Limit

A rate limiter is only as good as the identity it limits on. The key you choose determines how the limit behaves.

By IP address. Simple. Works against naive scrapers. Breaks behind NAT (a company office might have 500 employees sharing one IP), breaks against distributed attackers who rotate IPs, and it punishes shared infrastructure like CI runners.

By API key. The right choice for authenticated APIs. Each key gets its own limit. One bad key does not affect anyone else. This is the main identity for Throttle.

By tenant. Useful for aggregate limits across an organization. An enterprise tenant with 50 API keys should not get 50 times the limit just by spreading requests across keys.

By user. For consumer products with session-based auth.

In production you apply all of these at once, in layers:

Each layer catches a different kind of abuse. IP limits stop anonymous scrapers. Key limits handle a compromised key. Tenant limits stop one big customer from saturating the system. Global limits are the last line of defense.

The layered approach has a nice property: you only check deeper layers if the outer ones pass. If the IP is already over the limit, you never touch the API key counter. This saves Redis reads on already-rejected traffic.

The Full Data Model

Throttle stores limit configurations in Postgres and live state in Redis.

 1-- Limit policies, referenced by API keys and tenants
 2CREATE TABLE rate_limit_policies (
 3    id              TEXT PRIMARY KEY,
 4    name            TEXT NOT NULL,          -- "free_tier", "pro_tier", "enterprise"
 5    
 6    -- Per-key limits
 7    req_per_second  INT  NOT NULL DEFAULT 10,
 8    req_per_minute  INT  NOT NULL DEFAULT 500,
 9    req_per_hour    INT  NOT NULL DEFAULT 10000,
10    req_per_day     INT  NOT NULL DEFAULT 100000,
11    req_per_month   INT  NOT NULL DEFAULT 1000000,
12    
13    -- Burst headroom (token bucket capacity multiplier)
14    burst_multiplier FLOAT NOT NULL DEFAULT 1.5,
15    
16    -- Algorithm override (null = system default)
17    algorithm       TEXT,
18    
19    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
20    updated_at      TIMESTAMPTZ NOT NULL DEFAULT now()
21);
22
23-- Per-key overrides (when a key needs a non-default policy)
24CREATE TABLE api_key_rate_limits (
25    api_key_id      TEXT PRIMARY KEY REFERENCES api_keys(id),
26    policy_id       TEXT NOT NULL REFERENCES rate_limit_policies(id),
27    custom_override JSONB           -- ad-hoc overrides without a new policy row
28);
29
30-- Blocking rules (IPs, tenants, keys that are permanently blocked)
31CREATE TABLE rate_limit_blocks (
32    id          TEXT PRIMARY KEY,
33    scope       TEXT NOT NULL,      -- 'ip', 'key', 'tenant'
34    value       TEXT NOT NULL,      -- the IP or key ID or tenant ID
35    reason      TEXT,
36    blocked_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
37    expires_at  TIMESTAMPTZ,        -- null = permanent
38    UNIQUE (scope, value)
39);
40
41CREATE INDEX idx_blocks_scope_value ON rate_limit_blocks(scope, value);
42CREATE INDEX idx_blocks_expires_at  ON rate_limit_blocks(expires_at) WHERE expires_at IS NOT NULL;
43
44-- Persistent rate limit violations log (for billing, audit, anomaly detection)
45CREATE TABLE rate_limit_events (
46    id          TEXT PRIMARY KEY,
47    tenant_id   TEXT,
48    api_key_id  TEXT,
49    ip          INET,
50    scope       TEXT NOT NULL,      -- 'ip', 'key', 'tenant', 'global'
51    limit_key   TEXT NOT NULL,      -- the Redis key that was hit
52    limit_value INT NOT NULL,       -- the limit that was exceeded
53    count       INT NOT NULL,       -- the count at time of rejection
54    window      INTERVAL NOT NULL,
55    created_at  TIMESTAMPTZ NOT NULL DEFAULT now()
56) PARTITION BY RANGE (created_at);

The rate_limit_events table is append-only and partitioned by month. It feeds dashboards (“you are at 90% of your limit”), anomaly detection (“this IP hit rate limits 500 times in the last hour”), and the upsell pipeline (“this tenant exceeded their plan limit 3 times this week”).

The Middleware

Here is how Throttle wires into an HTTP server. It resolves context from the request, makes a decision, and either passes through or rejects.

 1type Limiter interface {
 2    Allow(ctx context.Context, req *LimitRequest) (*Decision, error)
 3}
 4
 5type LimitRequest struct {
 6    IP       string
 7    APIKeyID string
 8    TenantID string
 9    Policy   *Policy
10}
11
12type Decision struct {
13    Allowed    bool
14    Limit      int
15    Remaining  int
16    ResetAt    time.Time
17    RetryAfter time.Duration // set when Allowed=false
18    Scope      string        // which scope denied: "ip", "key", "tenant"
19}
20
21func RateLimitMiddleware(limiter Limiter) func(http.Handler) http.Handler {
22    return func(next http.Handler) http.Handler {
23        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
24            ctx := r.Context()
25            tc := TenantFromContext(ctx) // set by auth middleware upstream
26
27            req := &LimitRequest{
28                IP:       realIP(r),
29                APIKeyID: tc.APIKeyID,
30                TenantID: tc.TenantID,
31                Policy:   tc.RateLimitPolicy,
32            }
33
34            decision, err := limiter.Allow(ctx, req)
35            if err != nil {
36                // Redis is unavailable - policy decision (see failure handling)
37                if isRedisDown(err) {
38                    slog.Warn("rate limiter degraded", "err", err)
39                    // For this example: fail open, continue to handler
40                    next.ServeHTTP(w, r)
41                    return
42                }
43                http.Error(w, "internal error", http.StatusInternalServerError)
44                return
45            }
46
47            // Always set rate limit headers, even on success
48            w.Header().Set("X-RateLimit-Limit",     strconv.Itoa(decision.Limit))
49            w.Header().Set("X-RateLimit-Remaining", strconv.Itoa(decision.Remaining))
50            w.Header().Set("X-RateLimit-Reset",     strconv.FormatInt(decision.ResetAt.Unix(), 10))
51
52            if !decision.Allowed {
53                w.Header().Set("Retry-After", strconv.Itoa(int(decision.RetryAfter.Seconds())))
54                w.WriteHeader(http.StatusTooManyRequests)
55                json.NewEncoder(w).Encode(map[string]any{
56                    "error":       "rate_limit_exceeded",
57                    "scope":       decision.Scope,
58                    "retry_after": decision.RetryAfter.Seconds(),
59                    "message":     fmt.Sprintf("Rate limit exceeded. Retry in %.1fs.", decision.RetryAfter.Seconds()),
60                })
61                return
62            }
63
64            next.ServeHTTP(w, r)
65        })
66    }
67}
68
69// realIP handles X-Forwarded-For and X-Real-IP headers from load balancers.
70// Never trust the first IP in XFF blindly. Take the last IP added
71// by a trusted proxy if you control the proxy chain.
72func realIP(r *http.Request) string {
73    if xff := r.Header.Get("X-Forwarded-For"); xff != "" {
74        parts := strings.Split(xff, ",")
75        return strings.TrimSpace(parts[len(parts)-1])
76    }
77    if xri := r.Header.Get("X-Real-IP"); xri != "" {
78        return xri
79    }
80    host, _, _ := net.SplitHostPort(r.RemoteAddr)
81    return host
82}

The response headers matter. Clients that read X-RateLimit-* headers can back off on their own. Retry-After tells them exactly when to retry. Stripe’s client libraries implement exponential backoff using these headers. A rate limiter that only returns 429 with no headers forces clients to guess, and guessing usually means retrying immediately, which makes the problem worse.

The Full Request Flow

Each check is one Redis read. A clean request with three limit scopes costs three Redis round trips. In practice you run them concurrently:

 1func (l *TieredRedisLimiter) Allow(ctx context.Context, req *LimitRequest) (*Decision, error) {
 2    blocked, err := l.checkBlockList(ctx, req)
 3    if err != nil || blocked {
 4        return &Decision{Allowed: false, Scope: "block"}, err
 5    }
 6
 7    // Run limit checks concurrently using errgroup
 8    var (
 9        ipDec     *Decision
10        keyDec    *Decision
11        tenantDec *Decision
12    )
13    g, ctx := errgroup.WithContext(ctx)
14    g.Go(func() error {
15        var err error
16        ipDec, err = l.ipLimiter.Allow(ctx, req.IP, req.Policy.IPLimit)
17        return err
18    })
19    g.Go(func() error {
20        var err error
21        keyDec, err = l.keyLimiter.Allow(ctx, req.APIKeyID, req.Policy.KeyLimit)
22        return err
23    })
24    g.Go(func() error {
25        var err error
26        tenantDec, err = l.tenantLimiter.Allow(ctx, req.TenantID, req.Policy.TenantLimit)
27        return err
28    })
29    if err := g.Wait(); err != nil {
30        return nil, err
31    }
32
33    // Return the most restrictive decision
34    for _, d := range []*Decision{ipDec, keyDec, tenantDec} {
35        if !d.Allowed {
36            return d, nil
37        }
38    }
39
40    // All passed - return the tightest remaining quota (for response headers)
41    return mostRestrictive(ipDec, keyDec, tenantDec), nil
42}

Three Lua scripts run in parallel. In practice the three checks take about the same time as one.

Returning Useful Rate Limit Information

The 429 response is a user-facing feature. It is what developers see when their integration breaks. Getting it right cuts support tickets.

1X-RateLimit-Limit: 1000          # The limit that applies
2X-RateLimit-Remaining: 847       # Requests left in the current window
3X-RateLimit-Reset: 1720425600    # Unix timestamp when the window resets
4Retry-After: 14                  # Seconds until they can retry (on 429 only)

GitHub and Stripe both use this format. It is close enough to a standard that most HTTP clients handle it.

The response body on a 429 should give the developer exactly what they need to understand and fix the problem:

1{
2  "error": "rate_limit_exceeded",
3  "scope": "tenant",
4  "limit": 50000,
5  "current": 50001,
6  "window": "1h",
7  "retry_after": 847.3,
8  "message": "Tenant aggregate limit exceeded. Your plan allows 50,000 requests/hour. Retry in 847 seconds or upgrade to Pro."
9}

The scope field tells the developer which limit they hit. Without it, they cannot tell if one bad client is the problem or the entire organization is over quota. The window tells them whether waiting 1 second or 15 minutes is appropriate. The upsell nudge in message also happens to reduce support tickets.

Scalability

A single Redis instance handles roughly 100,000 commands per second. Each rate limit check runs 1 to 3 Redis commands depending on the Lua script. At 100,000 req/s of API traffic with three limit scopes per request, you need around 300,000 Redis commands per second. That means Redis Cluster.

Redis Cluster shards keys across multiple primary nodes. Rate limit keys shard naturally by hash. Each user’s counter lives on one shard and stays consistent within that shard. Cross-shard operations are rare in rate limiting because each scope’s key is self-contained.

Memory. With token bucket (two fields per key per scope) and sliding counter (two counters per key per window), memory per active user is roughly:

• Token bucket state: about 200 bytes per scope
• Sliding counter: about 200 bytes per active window
• 3 scopes times 2 windows each = about 1.2 KB per active user

At 1 million concurrent active users, that is about 1.2 GB of Redis memory. Fine for a dedicated cluster.

Local admission control. At very high traffic, even one Redis round trip per request starts to hurt. Twitter’s rate limiter uses a two-layer approach: a local approximate counter per server tracks recent requests. If clearly under the limit, skip Redis. If near the limit, do the authoritative Redis check. This cuts Redis load by around 80% under normal traffic.

 1type LocalApproximateLimiter struct {
 2    local     *LocalTokenBucket
 3    remote    *RedisLimiter
 4    threshold float64 // 0.8 = check Redis when local is at 80% of limit
 5}
 6
 7func (l *LocalApproximateLimiter) Allow(ctx context.Context, key string, limit int) (bool, error) {
 8    localDecision := l.local.Estimate(key, limit)
 9    if localDecision.RatioUsed < l.threshold {
10        // Clearly under limit, skip Redis
11        l.local.Increment(key)
12        return true, nil
13    }
14    // Near or over limit - do authoritative check
15    return l.remote.Allow(ctx, key, limit)
16}

The local estimate is wrong in a multi-server world because it only sees its own server’s traffic. But “clearly under limit” is almost always correct. If this server has seen 10% of the limit, the other servers would have to account for 90% for the real limit to be close. In practice traffic distributes roughly evenly.

Availability

Redis Sentinel provides automatic failover for a single Redis primary. Sentinel watches the primary and promotes a replica if it goes down. Failover takes 10 to 30 seconds. During that window your rate limiter is unavailable, which is why the local fallback from earlier matters.

Redis Cluster gives you both sharding and HA. Each shard has a primary and replicas. If a shard primary fails, a replica is promoted. A cluster with 3 primaries and 3 replicas survives any single-node failure without meaningful disruption.

Rate limit state must be written to the primary. But block list checks and policy reads can go to replicas. Routing read-only operations to replicas reduces primary load.

What happens during a network partition? Some servers can reach Redis, some cannot. The ones that cannot fall back to local limiting. The ones that can behave normally. The two groups now have inconsistent views of the rate limit. At most limit * (num_isolated_servers / total_servers) extra requests leak through during the partition. This is the CAP tradeoff. To stay available, you accept some inconsistency. For rate limiting, that is the right call. A small violation of rate limits during a partition beats 100% request failure.

Deployment Patterns

Embedded Middleware

Throttle runs as a Go package imported into your service. No extra network hop for the rate limit check beyond Redis. The simplest deployment.

1Your service
2  HTTP server
3    Auth middleware
4    RateLimitMiddleware (Throttle, embedded)
5    Route handlers

The downside is that every service needs to import and configure Throttle separately. Configuration drifts across services. Centralized changes require redeployments.

Sidecar

Throttle runs as a sidecar process alongside each service instance. The service calls Throttle over a local unix socket or loopback. The network hop is under a millisecond.

1Pod (Kubernetes)
2  Service container  ->  POST localhost:8081/check
3  Throttle sidecar   ->  Redis Cluster

This is how Envoy’s rate limiting works. The sidecar model gives you centralized configuration while keeping latency low.

API Gateway / Reverse Proxy

Rate limiting happens at the reverse proxy before traffic reaches your service. Nginx, HAProxy, Kong, Envoy, and cloud load balancers all support this. The service never sees over-limit requests.

1Internet -> Cloudflare (IP and DDoS limits) -> ALB -> API Gateway (key and tenant limits) -> Your service

This is the model for public APIs at scale. Cloudflare handles volumetric attacks. The API gateway handles per-key business logic limits. Your service handles nothing.

The limitation is context. Rate limiting at the proxy means less information about the request. You cannot limit on application-level concepts like “number of tokens consumed” for an LLM API without custom logic. Proxies are good at “N requests per minute”. They are not good at “N output tokens per minute”.

Centralized Rate Limit Service

Run Throttle as a standalone service. Every other service calls it before processing a request.

1Service A  ->  Throttle service  ->  Redis
2Service B  ->  Throttle service
3Service C  ->  Throttle service

Centralized configuration, no drift, easy to update policies. The cost: every request has a synchronous network call to Throttle on the critical path. Throttle becomes a hard dependency. If it goes down, all services fail unless each one has its own fallback logic. At high scale, Throttle itself needs to be distributed.

Lyft’s Ratelimit service (open source) uses this model. It is the right choice for large organizations with many independent services that need consistent rate limiting policy.

Trade-offs and Costs

Accuracy vs. memory. Sliding window log is perfectly accurate. Sliding counter approximates it. Token bucket is accurate for burst but cannot give you an exact count of total requests in a window. The more accurate the algorithm, the more memory or complexity it needs.

Consistency vs. latency. Authoritative Redis checks give you global consistency. Local approximate checks give you low latency. Under high load, the local check should be the fast path, and you accept that a few extra requests might slip through.

Fairness across users. A per-key limit of 1000 req/min is fair in isolation. But if 10% of your users are on enterprise plans with 100,000 req/min each, one enterprise customer can still saturate your infrastructure while every free-tier user is comfortably under their limit. You need per-scope limits at every layer, not just per-key.

The header trust problem. X-Forwarded-For is set by load balancers and proxies, but it can also be set by attackers. A client can send X-Forwarded-For: 1.2.3.4 and trick a naive rate limiter into thinking they are a different IP on every request. Always use the IP added by your own infrastructure, not the one claimed by the client.

Clock skew. Sliding window counters depend on the current time. Across servers with clock skew, window boundaries can differ slightly. NTP usually keeps skew under 1ms, which does not matter for second-level windows. For millisecond-precision limits, be careful comparing timestamps across machines.

The thundering herd at window reset. At midnight, when daily limits reset, every user who was at the limit can send requests at the same moment. This creates a traffic spike. Sliding windows help compared to fixed windows. You can also add random jitter to reset times per user so not everyone resets at once.

What Production Systems Actually Do

Stripe uses token bucket per API key with per-endpoint weights. Some endpoints cost more tokens than others. They have written publicly about decrementing tokens based on actual compute cost, not request count. A heavy request costs more tokens than a lightweight one. This stops “request inflation,” where clients make many cheap requests to stay under count-based limits while consuming far more resources than the limit intended.

GitHub uses sliding window with separate limits per authenticated user and per IP for unauthenticated users. Their secondary rate limit targets repeated failed requests, which signals brute-forcing or scraping rather than legitimate volume.

Cloudflare uses the sliding window counter from this post. Their rules run at edge nodes in every data center globally, evaluated in-process with no Redis round trip. They use a custom in-memory structure per edge PoP that gossips state with nearby PoPs. That is how they rate-limit at millions of requests per second.

Twilio applies limits at the product level. Some Twilio APIs are limited by concurrent connections, not requests per second. That requires a semaphore (a Redis counter with a max value) rather than a windowed counter.

The common thread: production rate limiters are layered, they make explicit choices about consistency vs. availability, and they put real work into the developer experience of the 429 response.

One Improvement Worth Building

Most teams treat rate limiting as a static policy: the limit is 1000 req/min, and that is it forever. But the thing rate limiting is protecting (your database, your inference cluster, your third-party budget) does not have a fixed capacity. It changes with load, with time of day, with deploys.

Adaptive rate limiting is the idea that the limit itself should respond to system health. If your database is at 90% CPU, tighten rate limits by 20% across all keys. When load drops back to normal, loosen them. The rate limiter becomes a feedback loop rather than a fixed gate.

 1type AdaptiveLimiter struct {
 2    base    Limiter
 3    metrics SystemMetrics // CPU, latency p99, error rate
 4}
 5
 6func (a *AdaptiveLimiter) scaleFactor() float64 {
 7    cpu := a.metrics.DatabaseCPU()
 8    switch {
 9    case cpu > 0.90:
10        return 0.5  // cut limits in half
11    case cpu > 0.75:
12        return 0.75
13    default:
14        return 1.0
15    }
16}
17
18func (a *AdaptiveLimiter) Allow(ctx context.Context, req *LimitRequest) (*Decision, error) {
19    scaled := req.Policy.ScaledBy(a.scaleFactor())
20    return a.base.Allow(ctx, &LimitRequest{
21        IP:       req.IP,
22        APIKeyID: req.APIKeyID,
23        TenantID: req.TenantID,
24        Policy:   scaled,
25    })
26}

This is simple to build on top of the existing structure. The base limiter does not change. You just scale the policy before passing it in.

The tricky part is choosing the right signal. CPU is a lagging indicator. A better signal is p99 latency of your slowest dependency, because it responds faster to the start of a problem. If your database p99 goes from 20ms to 200ms, something is wrong and you want the rate limiter to react before CPU follows.

The other tricky part is communicating the change to clients. If you tighten limits dynamically, X-RateLimit-Limit in the response headers now lies. The number the client stored from the last response is no longer accurate. You can mitigate this by returning the effective limit (already scaled) in the header, not the policy limit, so clients always see the limit that actually applies to their next request.

Further Reading

• Cloudflare blog: How we built rate limiting - the source for the sliding window counter approach.
• Stripe rate limiting - the canonical write-up on layered rate limiting in production.
• Lyft’s Ratelimit service - open source centralized rate limiting service used with Envoy.
• Redis documentation: INCR, Lua scripting - the primitives everything here is built on.
• GitHub REST API rate limiting - a real-world example of multi-tier rate limiting with decent documentation.

The thing I keep coming back to is that rate limiting is not a feature you add. It is a set of decisions you make about who you trust, how much, and what you do when that trust runs out. Every choice in this post, the algorithm, the identity key, the failure policy, flows from that.

Most teams add a Redis counter, call it done, and discover the boundary spike on a bad Tuesday. The engineers who designed Cloudflare’s edge rate limiting and Stripe’s API limits thought about these tradeoffs deliberately. They chose approximation over exactness, local over global, graceful degradation over hard failure. Those choices let their systems handle traffic that would have broken a naive implementation.

The counter in Redis was never the hard part.