A few weeks ago I designed Relay, a multi-tenant AI API gateway, as a system design exercise. That post covered the database, the auth, the billing. It did not cover the thing every one of those systems is sitting behind: the API itself.

Here is the question that post left open. When someone builds GET /v1/requests to list a tenant’s API call history, what should that endpoint actually look like? Offset or cursor pagination? What does the response envelope contain? What happens when Relay needs to add a filter that does not fit in a query string? What does a webhook payload look like when Relay tells a tenant “your batch job finished”? None of that is specific to AI gateways. It is the same decisions every backend API makes, made badly often enough that “REST API” has become a phrase that means “JSON over HTTP, structure unspecified.”

So this post is about those decisions. We will use Relay as the running example, specifically its dashboard and management API (tenants managing API keys, viewing usage, receiving webhooks, exporting data), not the OpenAI-compatible hot path, which already has its shape fixed by the providers it mirrors. Everything here is REST with JSON. At the end there is a pointer to what changes if you reach for GraphQL, RPC, or SOAP instead, but that is its own post.

Examples are in Go. The principles are not Go-specific, or even REST-specific. If you write APIs in any language, the same decisions will apply.

What Makes an API Good

Before naming conventions and pagination schemes, it helps to state the actual goal. An API is good when a developer who has never seen your system can read one endpoint, guess the shape of the next ten, and be right most of the time.

That is a narrower goal than “clean” or “RESTful.” It comes down to four properties.

Predictability. If GET /v1/api-keys/{id} returns an object with created_at as an RFC 3339 string, every other endpoint in the API returns timestamps the same way. If list endpoints wrap results in {"data": [...], "pagination": {...}}, every list endpoint does, not just the ones someone remembered to be consistent on.

One way to do a thing. If there are two ways to filter a list (a query parameter and a request body), developers will use both inconsistently across your own codebase, and support will have to debug both. Pick one path per capability.

Evolvability without breakage. You will need to add fields, deprecate endpoints, and fix mistakes. The API needs a built-in way to do all three without an existing client waking up broken. This is what versioning and additive schema changes are for, covered later.

Explicit contracts, not implied ones. “The list is usually sorted by creation date” is not a contract. "sort": "created_at desc" in the response, or documented and fixed behavior, is. Anything a client might reasonably depend on should either be guaranteed or explicitly called out as unstable.

Everything below is really just these four properties applied to a specific decision: URLs, schemas, pagination, caching, and so on.

Naming and Resource Structure

Relay’s dashboard API exposes tenants, API keys, usage, and webhooks. All of them are resources, and resources get nouns, not verbs.

1GET    /v1/api-keys              list API keys for the authenticated tenant
2POST   /v1/api-keys              create an API key
3GET    /v1/api-keys/{id}         fetch one API key
4DELETE /v1/api-keys/{id}         revoke an API key

Not POST /v1/createApiKey or GET /v1/getApiKeys. The HTTP method already carries the verb. Repeating it in the path is redundant, and once you have fifty endpoints named as functions instead of nouns, you no longer have a resource model. You have RPC that happens to run over HTTP.

A few rules that keep this consistent as the API grows:

• Plural nouns for collections. /api-keys, not /api-key. The singular form is only used when the ID follows it: /api-keys/{id}.
• Nest only one level deep, and only for true ownership. /v1/api-keys/{id}/rotate is fine because rotation belongs to a specific key. /v1/tenants/{id}/api-keys/{id}/rotate is not, because the tenant is already implied by the auth token. If you find yourself nesting three levels, it usually means the child resource should be its own top-level collection with a filter, like /v1/requests?api_key_id={id}.
• Actions that are not CRUD get a verb, deliberately. rotate, revoke, cancel, retry are real operations that do not map to PUT or DELETE cleanly. POST /v1/api-keys/{id}/rotate is a controlled escape hatch, not a violation of the noun rule. The rule is “resources are nouns,” not “there shall be no verbs anywhere.”
• snake_case in JSON bodies, kebab-case in URLs. key_hint and created_at in the payload, /api-keys in the path. This is a convention fight with no universally right answer (Stripe uses snake_case bodies and hyphenated paths; some Microsoft APIs use camelCase throughout). Pick one and never mix both cases inside the same field set. Mixed casing inside one JSON object is the single most common thing that makes an API feel unpolished.

Request and Response Schema

Envelope or bare object

A single resource response should be the object itself, not wrapped:

1{
2  "id": "key_9fL2xQ",
3  "name": "Production",
4  "key_prefix": "rly_live_",
5  "key_hint": "...1eA",
6  "status": "active",
7  "created_at": "2026-07-20T09:14:00Z"
8}

A list response needs an envelope, because a list carries more than just the items: pagination cursors, total counts, sort order.

 1{
 2  "data": [
 3    { "id": "key_9fL2xQ", "name": "Production", "status": "active" },
 4    { "id": "key_2vM8pR", "name": "CI", "status": "active" }
 5  ],
 6  "pagination": {
 7    "next_cursor": "eyJpZCI6ImtleV8ydk04cFIifQ",
 8    "has_more": true
 9  }
10}

Do not wrap the single-resource response in a {"data": {...}} envelope too, just to be consistent with the list response. It buys nothing and forces every client to unwrap one extra layer on every single call. Consistency should apply to shapes that need the same thing, not blindly to everything.

Field conventions

A few small decisions, made once, save a hundred future arguments in code review:

• IDs are prefixed strings, not raw UUIDs or integers. key_9fL2xQ, req_7yN3kM, ten_xyz789. The prefix tells you what kind of object you are looking at from the ID alone, in a log line, in a support ticket, in a stack trace. This is the same pattern the multi-tenant post used for tenants, api_keys, and requests, and it should be consistent across every resource in the API, not just some of them.
• Timestamps are RFC 3339 strings in UTC, always. "2026-07-20T09:14:00Z". Never Unix epoch integers in a public API. Epoch integers are ambiguous about seconds versus milliseconds, and unreadable in a raw log dump. If you need epoch millis for a specific high-frequency internal API, that is a different contract than your public REST API.
• Money is a string or an integer in minor units, never a float. "amount": "9.00" or "amount_cents": 900. A float64 for currency will eventually produce 8.999999999 somewhere and someone will spend an afternoon on it.
• Null means “explicitly no value.” Omitted means “not applicable to this resource state.” A revoked API key might have revoked_at: "2026-07-21T10:00:00Z" and an active one has revoked_at: null, present but null, because the field is always meaningful for that resource type. A next_cursor field is omitted entirely (not null) once there is nothing left to fetch, because at that point the pagination concept does not apply anymore. Be deliberate about which of these two you mean, and do not let the serializer decide for you.
• Enums are lowercase strings, not integers. "status": "active", not "status": 1. An integer enum forces every client to keep a lookup table in sync with your source code. A string enum is self-documenting in every request log you will ever grep.

Partial updates

Relay’s dashboard lets a tenant rename an API key or change its expires_at without resending the whole object. Two real options exist for this.

JSON Merge Patch (RFC 7396): send only the fields you want to change, and a JSON body with null means “set this field to null.”

1PATCH /v1/api-keys/key_9fL2xQ
2{ "name": "Production (rotated July)" }

JSON Patch (RFC 6902): send an array of explicit operations (add, remove, replace).

1PATCH /v1/api-keys/key_9fL2xQ
2[{ "op": "replace", "path": "/name", "value": "Production (rotated July)" }]

Merge Patch is simpler and covers almost every real case. Reach for JSON Patch only when you genuinely need array element operations (insert at index 2, remove a specific item from a list) that Merge Patch cannot express. For Relay’s dashboard API, Merge Patch is the right call everywhere. Most APIs never need JSON Patch’s complexity, and adding it “for completeness” is exactly the kind of premature flexibility that makes a schema harder to reason about later.

Pagination: Offset vs Keyset (Cursor)

Relay’s requests table is the one from the multi-tenant post: every API call the tenant makes gets a row, and at 1,000 tenants making 100 calls a day, that is 3 million rows a month. GET /v1/requests has to paginate through that, and the pagination strategy you pick here is the single decision most likely to bite you in production six months from now.

Offset pagination

1GET /v1/requests?page=3&limit=50

1SELECT * FROM requests
2WHERE tenant_id = $1
3ORDER BY created_at DESC
4LIMIT 50 OFFSET 100;

Trivial to implement, and it gives you random access: jump straight to page 40. This is the right choice for small, mostly static collections where users expect page numbers, like a list of API keys (a tenant has a handful, not millions).

It falls apart on large, frequently changing tables for two separate reasons:

1. It gets slow at depth. OFFSET 100000 still has to scan and discard 100,000 rows before returning anything. The query gets linearly slower the deeper a client pages, even though the page size never changes.
2. It is unstable under writes. If a new row is inserted while a tenant is paginating page 2 of requests, every row after it shifts by one. The client can see the same row twice, or skip one entirely, depending on sort direction. For a table that gets a new row every time the tenant makes an API call, this is not a rare edge case, it happens on every active tenant’s very first page-through.

Keyset (cursor) pagination

1GET /v1/requests?cursor=eyJjcmVhdGVkX2F0IjoiMjAyNi0wNy0yMFQwOTowMDowMFoifQ&limit=50

Instead of an offset, the cursor encodes the last row’s sort key. The next page is “give me rows after this specific point,” not “skip this many rows.”

1SELECT * FROM requests
2WHERE tenant_id = $1
3  AND (created_at, id) < ($2, $3)   -- values decoded from the cursor
4ORDER BY created_at DESC, id DESC
5LIMIT 50;

Note the compound key: (created_at, id), not created_at alone. Two requests can share the same millisecond timestamp under load, and a cursor on created_at alone would either skip or repeat rows in that case. id breaks the tie and makes the ordering total and deterministic. This is the detail that makes keyset pagination correct instead of “usually correct.”

 1type Cursor struct {
 2    CreatedAt time.Time `json:"created_at"`
 3    ID        string    `json:"id"`
 4}
 5
 6func EncodeCursor(c Cursor) string {
 7    b, _ := json.Marshal(c)
 8    return base64.URLEncoding.EncodeToString(b)
 9}
10
11func DecodeCursor(s string) (Cursor, error) {
12    var c Cursor
13    b, err := base64.URLEncoding.DecodeString(s)
14    if err != nil {
15        return c, ErrInvalidCursor
16    }
17    if err := json.Unmarshal(b, &c); err != nil {
18        return c, ErrInvalidCursor
19    }
20    return c, nil
21}
22
23func (h *Handler) ListRequests(w http.ResponseWriter, r *http.Request) {
24    tenantID := TenantFromContext(r.Context())
25    limit := clampLimit(r.URL.Query().Get("limit"), 50, 200)
26
27    var after *Cursor
28    if raw := r.URL.Query().Get("cursor"); raw != "" {
29        c, err := DecodeCursor(raw)
30        if err != nil {
31            writeError(w, 400, "invalid_cursor")
32            return
33        }
34        after = &c
35    }
36
37    rows, err := h.store.ListRequests(r.Context(), tenantID, after, limit+1)
38    if err != nil {
39        writeError(w, 500, "internal_error")
40        return
41    }
42
43    hasMore := len(rows) > limit
44    if hasMore {
45        rows = rows[:limit]
46    }
47
48    resp := ListResponse{Data: rows, Pagination: Pagination{HasMore: hasMore}}
49    if hasMore {
50        last := rows[len(rows)-1]
51        resp.Pagination.NextCursor = EncodeCursor(Cursor{CreatedAt: last.CreatedAt, ID: last.ID})
52    }
53    writeJSON(w, 200, resp)
54}

The limit+1 trick (fetch one extra row) is how you compute has_more without a separate COUNT(*) query. If you get back 51 rows for a limit of 50, there is more data, and you trim the 51st before returning.

The cursor is opaque to the client on purpose. It is base64-encoded JSON here for simplicity, but treat it as an implementation detail you are free to change, not a contract. If you switch the underlying sort key next year, old cursors from bookmarked URLs should fail with a clear invalid_cursor error, not silently return wrong data.

Which one, when

Relay’s dashboard uses offset for /v1/api-keys (a tenant has maybe a dozen) and keyset for /v1/requests and /v1/usage (thousands to millions of rows, growing every second). Do not pick one pagination style for the whole API. Pick it per resource, based on how that resource actually grows and gets queried.

The New Verb: QUERY

Every API eventually needs to filter on more than a URL can comfortably hold. Relay’s usage dashboard wants to filter requests by a combination of model, status, date range, cost threshold, and cache hit state, at once. That does not fit cleanly in query parameters:

1GET /v1/requests?model=gpt-5.5&status=success&cache_hit=true&min_cost=0.01&created_after=2026-07-01&created_before=2026-07-20

It technically works, until the filter needs an array of models, a nested boolean condition, or hits a URL length limit some proxy enforces. The traditional workaround is POST /v1/requests/search with the filter in the body. It works, but it is semantically wrong: a search is a read, and POST tells every cache, proxy, and retry mechanism between the client and your server “this is a write, do not cache it, do not safely retry it, it might not be idempotent.” That mismatch is a real cost, and it is why Stripe, GitHub, and most search-heavy APIs use POST /search anyway: it is the wrong method, and it is still the pragmatic choice, because nothing better existed until recently.

As of June 2026, HTTP has an actual answer: QUERY, standardized as RFC 10008. It is the first new HTTP method since PATCH in 2010. QUERY is what the name suggests: a read, like GET, that carries a request body, like POST.

 1QUERY /v1/requests
 2Content-Type: application/json
 3
 4{
 5  "filter": {
 6    "model": ["gpt-5.5", "claude-3-5-sonnet"],
 7    "status": "success",
 8    "cache_hit": true,
 9    "cost": { "gte": 0.01 },
10    "created_at": { "after": "2026-07-01", "before": "2026-07-20" }
11  },
12  "sort": "created_at desc",
13  "limit": 50
14}

The important part is not the syntax, it is the contract. QUERY is defined as safe and idempotent, the same guarantees GET carries. That means a proxy or gateway is allowed to cache a QUERY response and safely retry it on a network failure, the same way it would a GET, something it can never safely assume about a POST body. For an endpoint like Relay’s usage search, that is a real operational win: it means Relay’s own gateway can add caching in front of QUERY /v1/requests without special-casing it, because HTTP already tells it that this method is safe to cache.

The honest caveat: QUERY took eleven years to standardize (it spent six of those years named SEARCH before the rename), and as of mid-2026 support is still filling in. Node.js parses it natively, ASP.NET Core 10 recognizes it, Jetty and Tomcat support is in flight. Older reverse proxies, some corporate firewalls, and plenty of client libraries do not know what to do with a request body on a method they have never heard of. The pragmatic default for a production API in 2026 is still POST /v1/requests/query (name the endpoint query, not search, to signal the intent even while using POST as the transport), with a plan to add QUERY as the primary method once your infrastructure and clients catch up, keeping POST around as a fallback. Relay’s API does exactly this: POST /v1/requests/query today, documented as “QUERY-semantics over POST,” with QUERY itself supported as soon as the load balancer in front of it does.

Verbs, Actions, and Idempotency

Standard CRUD covers create, read, update, delete. Real APIs need a fifth category: actions that change state without being any of the four. Revoking a key. Retrying a failed batch job. Canceling a subscription.

1POST /v1/api-keys/{id}/rotate
2POST /v1/batches/{id}/cancel
3POST /v1/webhooks/{id}/test

These are POST because they are not safe (they change state) and not idempotent by default (calling rotate twice generates two different new keys). That second property is the actual problem, because networks fail mid-request. A client calls POST /v1/api-keys/{id}/rotate, the connection drops before the response arrives, and the client cannot tell whether the rotation happened or not. Retrying blindly might rotate the key twice, invalidating a key the client thinks is still valid.

The fix is the Idempotency-Key header, the same pattern Stripe popularized:

1POST /v1/api-keys/key_9fL2xQ/rotate
2Idempotency-Key: 7c9e6679-7425-40de-944b-e07fc1f90ae7

 1func IdempotencyMiddleware(store IdempotencyStore) func(http.Handler) http.Handler {
 2    return func(next http.Handler) http.Handler {
 3        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 4            key := r.Header.Get("Idempotency-Key")
 5            if key == "" || r.Method == http.MethodGet {
 6                next.ServeHTTP(w, r)
 7                return
 8            }
 9
10            cached, found := store.Get(r.Context(), key)
11            if found {
12                w.Header().Set("Content-Type", "application/json")
13                w.WriteHeader(cached.StatusCode)
14                w.Write(cached.Body)
15                return
16            }
17
18            rec := httptest.NewRecorder()
19            next.ServeHTTP(rec, r)
20
21            // Only successful responses are safe to replay verbatim.
22            // A 5xx might mean the write never committed, so let a retry try again for real.
23            if rec.Code < 500 {
24                store.Set(r.Context(), key, rec.Code, rec.Body.Bytes(), 24*time.Hour)
25            }
26
27            w.WriteHeader(rec.Code)
28            w.Write(rec.Body.Bytes())
29        })
30    }
31}

The store is keyed on the idempotency key, scoped to the tenant (two different tenants can reuse the same UUID by coincidence), and holds the response for 24 hours. The first request with a given key executes and its response is cached. Every retry with the same key, whether it is the client retrying after a timeout or an accidental double-click, gets the original response replayed, not a second execution.

1CREATE TABLE idempotency_keys (
2    tenant_id     TEXT        NOT NULL REFERENCES tenants(id),
3    key           TEXT        NOT NULL,
4    status_code   INT         NOT NULL,
5    response_body JSONB       NOT NULL,
6    created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
7    expires_at    TIMESTAMPTZ NOT NULL,
8    PRIMARY KEY (tenant_id, key)
9);

Apply this to every non-idempotent POST, not just the “dangerous” ones. POST /v1/api-keys (create) needs it too. Without it, a client retrying a timed-out create request ends up with two API keys instead of one, and neither the client nor the tenant has any way to know that happened until they see two keys in the dashboard.

Versioning

Relay’s hot path is already /v1/chat/completions. The version lives in the URL, and that choice was made for a reason worth stating explicitly: it is the simplest option to get right, and simple is what you want for a contract other people’s production systems depend on.

The alternative, header-based versioning (Accept: application/vnd.relay.v2+json or a custom X-API-Version header), keeps URLs clean and lets you version at a finer grain than “the whole API.” It also means every debugging session starts with “what version is this request even using,” because it is invisible in the URL, invisible in a browser address bar, invisible in a curl command someone pastes into a support ticket. For an API whose primary consumers are backend developers pasting example requests into their own code (which is every API in this post), that invisibility is a real cost that outweighs the cleanliness argument. Use URL versioning unless you have a specific reason not to.

Whichever you pick, the harder discipline is not breaking v1 while v2 doesn’t exist yet. Two rules cover almost every case:

Additive changes do not need a new version. Adding a new optional field to a response, adding a new endpoint, adding a new optional request parameter: none of these break an existing client, because existing clients simply ignore fields they do not recognize. This is why a JSON API should never validate against a schema that rejects unknown fields on the way in from the server to the client.

Breaking changes need a new version, full stop. Removing a field, renaming a field, changing a field’s type (a string ID becoming an integer), changing the meaning of an existing status code: all of these break something, somewhere, for someone. There is no clever way around this. You cut v2.

When v2 ships, v1 does not disappear immediately. It gets a deprecation window, communicated in the response headers of the old version so that anyone still calling it finds out from their own logs, not from a changelog they never read:

1Deprecation: true
2Sunset: Sat, 1 May 2027 00:00:00 GMT
3Link: <https://relay.lorbic.com/docs/v2/migration>; rel="successor-version"

Deprecation and Sunset are real, registered HTTP headers (RFC 8594 and the corresponding Deprecation header draft), not something Relay invented. A well-behaved client library can check for the Deprecation header and log a warning automatically, which is a much more reliable distribution channel for “please migrate” than an email that goes to whoever signed up eighteen months ago and has since left the company.

Caching

The multi-tenant post covered caching the actual LLM responses, exact-match and semantic. This is a different layer: standard HTTP caching for the dashboard’s REST resources, GET /v1/api-keys, GET /v1/requests/{id}, the things a tenant’s own dashboard fetches on every page load.

The two headers that matter are ETag and Cache-Control.

 1func (h *Handler) GetAPIKey(w http.ResponseWriter, r *http.Request) {
 2    key, err := h.store.GetAPIKey(r.Context(), chi.URLParam(r, "id"))
 3    if err != nil {
 4        writeError(w, 404, "not_found")
 5        return
 6    }
 7
 8    etag := fmt.Sprintf(`"%x"`, sha256.Sum256(mustJSON(key)))
 9    if inm := r.Header.Get("If-None-Match"); inm == etag {
10        w.WriteHeader(http.StatusNotModified)
11        return
12    }
13
14    w.Header().Set("ETag", etag)
15    w.Header().Set("Cache-Control", "private, max-age=30")
16    writeJSON(w, 200, key)
17}

ETag is a hash of the current representation. A client that already has a copy sends it back as If-None-Match on the next request, and if nothing changed, the server returns 304 Not Modified with an empty body instead of resending the whole object. For a dashboard polling GET /v1/api-keys every few seconds to keep its UI fresh, this turns most of those requests into a few hundred bytes of headers instead of a full JSON payload.

Cache-Control: private, max-age=30 tells any cache (browser, CDN, intermediate proxy) that this response belongs to one tenant (private, never shared across tenants) and can be reused for 30 seconds without even asking the server. no-store on anything containing a secret, like the one-time display of a freshly created API key, which must never be cached anywhere, by any layer, for any amount of time.

Cache invalidation on writes is the part people forget. When a tenant renames an API key, the write handler needs to either bump a version number embedded in the ETag computation (so the next read naturally produces a different hash) or explicitly purge the cached entry if you are running a CDN or reverse proxy cache in front of the API. An ETag that is a hash of the current row’s content, computed fresh on every read as in the example above, sidesteps this entirely: it is always correct because it is never stored, only recomputed. The tradeoff is a hash computation on every request instead of a cheap lookup, which is a fine trade for a dashboard’s read volume and a bad one for a hot path serving millions of requests a second, where you would maintain a version column instead and hash nothing.

BFF: Backend for Frontend

Relay’s dashboard home screen needs, in one page load: the tenant’s plan and status, their last 5 API keys, this month’s usage rollup, and any active billing alerts. That is four different internal queries (tenants, api_keys, usage_daily, billing_periods), and if the dashboard frontend calls Relay’s general-purpose API directly, it makes four round trips, each carrying its own auth check, its own network latency, and its own over-fetched fields the dashboard does not render.

A Backend for Frontend is a thin API layer that exists for exactly one client (the dashboard, in this case) and shapes its response around what that one screen needs, not around the general resource model underneath.

 1type DashboardHomeResponse struct {
 2    Tenant       TenantSummary   `json:"tenant"`
 3    RecentKeys   []APIKeySummary `json:"recent_keys"`
 4    UsageThisMonth UsageSummary  `json:"usage_this_month"`
 5    BillingAlert  *BillingAlert  `json:"billing_alert,omitempty"`
 6}
 7
 8func (bff *DashboardBFF) Home(w http.ResponseWriter, r *http.Request) {
 9    tenantID := TenantFromContext(r.Context())
10
11    var tenant Tenant
12    var keys []APIKey
13    var usage UsageSummary
14    var billing *BillingAlert
15
16    g, ctx := errgroup.WithContext(r.Context())
17    g.Go(func() (err error) { tenant, err = bff.tenants.Get(ctx, tenantID); return })
18    g.Go(func() (err error) { keys, err = bff.keys.ListRecent(ctx, tenantID, 5); return })
19    g.Go(func() (err error) { usage, err = bff.usage.ThisMonth(ctx, tenantID); return })
20    g.Go(func() (err error) { billing, err = bff.billing.ActiveAlert(ctx, tenantID); return })
21
22    if err := g.Wait(); err != nil {
23        writeError(w, 500, "internal_error")
24        return
25    }
26
27    writeJSON(w, 200, DashboardHomeResponse{
28        Tenant:         toTenantSummary(tenant),
29        RecentKeys:     toAPIKeySummaries(keys),
30        UsageThisMonth: usage,
31        BillingAlert:   billing,
32    })
33}

The four internal calls still happen, but they happen concurrently, server-side, over a fast internal network, and the dashboard gets one response shaped exactly like the screen it renders. No over-fetching (the dashboard never sees the full api_keys row, key_hash, created_by, none of that, just the five fields the summary card shows), and no round trips from the browser, which is the network path that actually has latency worth worrying about.

The tradeoff is real, not free: a BFF is another service to deploy, monitor, and keep in sync with the resources it aggregates. It is worth it when a specific client (a mobile app, a dashboard, a public-facing website) has a shape of need that is meaningfully different from your general resource API, and painful when teams reach for it reflexively for every client, producing a pile of near-identical BFFs that all drift independently. Relay has exactly one BFF, for the dashboard. The mobile SDK and CLI tooling call the general API directly, because their needs match the resource model closely enough that a BFF would just be a pass-through layer adding latency for no reshaping benefit.

SDUI: Server-Driven UI

The BFF above still assumes the dashboard’s frontend code knows how to render a “billing alert” card, because that layout is baked into the frontend’s own code. Server-driven UI goes one step further: the server sends not just the data, but a description of what to render and how to lay it out, and the client is a generic renderer that interprets that description.

Relay’s onboarding wizard is the natural candidate. It has steps that change: today it is “add a payment method, create your first API key, pick a default model.” Next quarter product wants to add a step, reorder them, or A/B test two different copies of step 2, all without shipping a new frontend build or waiting on a mobile app store review for the same change on iOS.

 1GET /v1/dashboard/onboarding-ui
 2
 3{
 4  "version": "2026-07-20",
 5  "screen": {
 6    "type": "wizard",
 7    "steps": [
 8      {
 9        "id": "payment_method",
10        "title": "Add a payment method",
11        "components": [
12          { "type": "text", "content": "You get $5 in free credits during trial." },
13          { "type": "button", "action": "open_stripe_portal", "label": "Add card" }
14        ]
15      },
16      {
17        "id": "first_key",
18        "title": "Create your first API key",
19        "components": [
20          { "type": "button", "action": "create_api_key", "label": "Generate key" },
21          { "type": "code_block", "bind": "generated_key" }
22        ]
23      }
24    ]
25  }
26}

The client renderer knows how to draw a text, a button, and a code_block, and how to execute a fixed, small set of named actions (open_stripe_portal, create_api_key). It does not know anything about onboarding specifically. Add a third step, reorder the two existing ones, or change the copy, and every client picks it up on next load, no release required, on web, iOS, and Android at once.

The real system has a few more pieces than the JSON above suggests: a small, versioned component vocabulary the client renderer supports (this is the part that has to ship in a client release, so keep it deliberately small and stable), a schema-validated payload so a malformed server response cannot crash the renderer, and a fallback screen for when the client’s renderer version is older than the payload it received (an unrecognized component type should render as nothing, or a generic placeholder, never a crash).

SDUI is worth the investment for exactly the kind of surface that changes often and is annoying to gate behind app store review: onboarding flows, promotional banners, plan comparison screens, empty states. It is a bad fit for anything with complex client-side interaction or heavy platform-specific behavior (a rich text editor, a data table with column resizing), where forcing the interaction model through a generic JSON schema costs more in renderer complexity than it saves in release cycles. Relay uses it for onboarding and dashboard banners. The actual usage charts and the API key management table are hand-built in the dashboard frontend, because a chart’s interaction model does not compress into a generic component vocabulary without losing the thing that makes it a good chart.

WebSocket and Server-Sent Events

The multi-tenant post already covers SSE for streaming chat completions: the provider sends chunks, Relay pipes them straight through as data: events, one direction, client to open connection and read. That pattern generalizes past LLM streaming to anything that is a server pushing a sequence of updates: a live token counter on the dashboard, a tail of the tenant’s most recent requests as they land.

 1func (h *Handler) StreamUsage(w http.ResponseWriter, r *http.Request) {
 2    tenantID := TenantFromContext(r.Context())
 3    flusher, ok := w.(http.Flusher)
 4    if !ok {
 5        writeError(w, 500, "streaming_unsupported")
 6        return
 7    }
 8
 9    w.Header().Set("Content-Type", "text/event-stream")
10    w.Header().Set("Cache-Control", "no-cache")
11    w.Header().Set("Connection", "keep-alive")
12
13    updates := h.usage.Subscribe(r.Context(), tenantID)
14    defer h.usage.Unsubscribe(tenantID, updates)
15
16    for {
17        select {
18        case <-r.Context().Done():
19            return
20        case u := <-updates:
21            fmt.Fprintf(w, "data: %s\n\n", mustJSON(u))
22            flusher.Flush()
23        }
24    }
25}

SSE runs over plain HTTP, reconnects automatically in every browser’s built-in EventSource client, and passes through ordinary HTTP infrastructure (load balancers, proxies, corporate firewalls) without special handling. Its one real limitation is that it is one-way. The server pushes, the client only ever reads.

WebSocket is the right tool when the client needs to talk back on the same connection, not just make a new HTTP request. Relay’s dashboard has exactly one feature that needs this: a live request log viewer where a tenant can pause the stream, adjust a filter (show only errors, show only one model), and have that filter applied server-side without reopening the connection.

Default to SSE. It solves the actual problem (server pushes updates) with less protocol surface, less client code, and fewer things that can go wrong at the network layer. Reach for WebSocket only when the client genuinely needs to send messages back on the same open connection, not just make occasional separate HTTP calls alongside a stream it is reading.

API Security

Auth headers

Relay’s hot path already established the pattern: Authorization: Bearer rly_live_sk_.... The dashboard API’s JWT-based session auth uses the same header, same scheme, different token. This consistency matters more than it looks: one auth header format across the entire API means one code path for every client library to implement, instead of “use this header for these endpoints, that header for those.”

Never accept credentials in the query string (?api_key=...). Query strings end up in access logs, browser history, and Referer headers sent to third-party resources on the same page. A credential that only ever appears in the Authorization header stays out of all three by default.

Server-to-server communication

Bearer tokens work when a human or a client SDK is calling Relay. A different problem shows up when two backend services need to trust each other directly, with no human in the loop: Relay’s billing worker calling Stripe, or an enterprise tenant’s own internal service calling Relay’s BYOK credential endpoint.

Two patterns cover almost every case:

mTLS (mutual TLS) where both sides present a certificate, is the right choice for a small, fixed set of trusted internal services that rarely change, like Relay’s own microservices talking to each other inside a VPC. It is strong (the private key never leaves the holding service) but operationally heavy: certificate issuance, rotation, and revocation all need their own infrastructure.

HMAC request signing is the more common choice for API-to-API integrations across organizational boundaries, because it needs no certificate infrastructure on the caller’s side, just a shared secret.

 1func SignRequest(secret []byte, method, path string, body []byte, timestamp string) string {
 2    mac := hmac.New(sha256.New, secret)
 3    mac.Write([]byte(method + "\n" + path + "\n" + timestamp + "\n"))
 4    mac.Write(body)
 5    return hex.EncodeToString(mac.Sum(nil))
 6}
 7
 8func VerifySignedRequest(secret []byte, r *http.Request, body []byte) error {
 9    ts := r.Header.Get("X-Relay-Timestamp")
10    sig := r.Header.Get("X-Relay-Signature")
11
12    t, err := strconv.ParseInt(ts, 10, 64)
13    if err != nil || time.Since(time.Unix(t, 0)) > 5*time.Minute {
14        return ErrStaleRequest
15    }
16
17    expected := SignRequest(secret, r.Method, r.URL.Path, body, ts)
18    if !hmac.Equal([]byte(expected), []byte(sig)) {
19        return ErrInvalidSignature
20    }
21    return nil
22}

The timestamp check matters as much as the signature itself. Without it, a captured request (from a compromised log, a network dump, a misconfigured proxy that echoes requests) can be replayed indefinitely with a valid signature, because the signature alone only proves the request was not tampered with, not that it is fresh. Rejecting anything older than a few minutes closes that window without needing a stateful nonce store for most use cases. If you need to guard against replay within that window too (the same signed request sent twice, both within 5 minutes), track seen (timestamp, signature) pairs in Redis with a TTL matching the window, and reject duplicates.

Webhook receiver APIs

A webhook receiver is unusual among “APIs you build” because you are not the one designing the request format, the sender is (Stripe, a provider, another tenant’s system), and you have to accept it anyway. Relay has two separate webhook relationships to get right, in opposite directions.

Receiving: Stripe calls Relay’s POST /v1/webhooks/stripe when a payment succeeds or fails. Every webhook receiver needs the same three defenses, regardless of who is sending it:

1. Verify the signature before doing anything else. Stripe signs every webhook with a secret only Relay and Stripe know. Skipping this check means anyone who finds the URL can POST a fake “payment succeeded” event and reactivate a suspended tenant’s account without paying anything.

 1func StripeWebhookHandler(secret string) http.HandlerFunc {
 2    return func(w http.ResponseWriter, r *http.Request) {
 3        body, _ := io.ReadAll(r.Body)
 4        sig := r.Header.Get("Stripe-Signature")
 5
 6        event, err := webhook.ConstructEvent(body, sig, secret)
 7        if err != nil {
 8            writeError(w, 400, "invalid_signature")
 9            return
10        }
11
12        // Ack immediately, process async. Stripe retries on anything but 2xx,
13        // and a slow handler here is Relay's problem, not Stripe's to wait on.
14        if err := enqueueWebhookEvent(r.Context(), event); err != nil {
15            writeError(w, 500, "enqueue_failed")
16            return
17        }
18        w.WriteHeader(http.StatusOK)
19    }
20}

2. Acknowledge fast, process async. The handler’s only job is to verify the signature and enqueue the event. Anything slower (updating the tenant’s billing status, sending a follow-up email) happens in a background worker reading off that queue. Stripe retries a webhook it does not get a 2xx for within its timeout, and a synchronous handler that occasionally takes 8 seconds under load will get duplicate deliveries for the exact same event, on top of whatever else went wrong.

3. Deduplicate by event ID, because “at least once” delivery is the norm, not the exception. Every webhook sender you integrate with, Stripe included, documents that a given event might be delivered more than once. Store the event ID with a unique constraint and skip processing on a duplicate.

1CREATE TABLE processed_webhook_events (
2    source     TEXT NOT NULL,        -- stripe | provider_x
3    event_id   TEXT NOT NULL,
4    processed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
5    PRIMARY KEY (source, event_id)
6);

Sending: Relay also sends webhooks, to tenants, when their batch job finishes or a provider outage affects their traffic. This is the same problem from the other side, and the same three defenses apply in reverse: Relay signs every outbound webhook so the tenant’s receiver can verify it actually came from Relay, retries with backoff on anything but a 2xx from the tenant’s endpoint, and includes an event_id so the tenant’s own receiver can deduplicate.

 1CREATE TABLE webhook_endpoints (
 2    id            TEXT        PRIMARY KEY,
 3    tenant_id     TEXT        NOT NULL REFERENCES tenants(id),
 4    url           TEXT        NOT NULL,
 5    secret        TEXT        NOT NULL,        -- shared secret, shown once at creation
 6    status        TEXT        NOT NULL DEFAULT 'active',
 7    created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
 8);
 9
10CREATE TABLE webhook_deliveries (
11    id            TEXT        PRIMARY KEY,
12    endpoint_id   TEXT        NOT NULL REFERENCES webhook_endpoints(id),
13    event_id      TEXT        NOT NULL,
14    event_type    TEXT        NOT NULL,        -- batch.completed | provider.degraded
15    payload       JSONB       NOT NULL,
16    status        TEXT        NOT NULL DEFAULT 'pending', -- pending | delivered | failed
17    attempt_count INT         NOT NULL DEFAULT 0,
18    last_attempt_at TIMESTAMPTZ,
19    next_attempt_at TIMESTAMPTZ,
20    created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
21    UNIQUE(endpoint_id, event_id)
22);

webhook_deliveries is the audit trail tenants need when they ask “why didn’t I get notified my batch finished.” It records every attempt, every failure, and lets the dashboard show a delivery log, plus a manual “resend” button that replays the same event_id and payload rather than generating a new event, keeping the tenant’s own deduplication intact.

Bulk Data Load APIs

Everything so far has been synchronous: request in, response out, within one HTTP round trip. Some operations do not fit that shape. A tenant wants to submit 50,000 chat completion requests at once for an overnight batch job, or export a year of requests history for their own analytics warehouse. Neither belongs on the request-response path.

Bulk submission: the async job pattern

1POST /v1/batches
2Content-Type: application/jsonl
3
4{"custom_id": "req-1", "model": "gpt-5.5", "messages": [...]}
5{"custom_id": "req-2", "model": "gpt-5.5", "messages": [...]}
6... (50,000 lines)

1201 Created
2{
3  "id": "batch_4mK9pL",
4  "status": "validating",
5  "request_count": 50000,
6  "created_at": "2026-07-26T10:00:00Z"
7}

The client gets back a job ID immediately, not 50,000 responses. Relay validates and processes the batch asynchronously (typically at a lower priority and lower cost than the synchronous API, since there is no client waiting on the other end of an open connection), and the tenant finds out it is done one of two ways: polling GET /v1/batches/{id} for a status field, or, better, registering a webhook for batch.completed and finding out the moment it happens instead of however often they happen to poll.

 1CREATE TABLE batches (
 2    id              TEXT        PRIMARY KEY,
 3    tenant_id       TEXT        NOT NULL REFERENCES tenants(id),
 4    status          TEXT        NOT NULL DEFAULT 'validating', -- validating | in_progress | completed | failed | expired
 5    request_count   INT         NOT NULL,
 6    completed_count INT         NOT NULL DEFAULT 0,
 7    failed_count    INT         NOT NULL DEFAULT 0,
 8    input_file_id   TEXT        NOT NULL,
 9    output_file_id  TEXT,
10    error_file_id   TEXT,
11    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
12    completed_at    TIMESTAMPTZ,
13    expires_at      TIMESTAMPTZ NOT NULL
14);

Each line in the input file is independent and carries its own custom_id, so a failure on line 30,000 does not invalidate the other 49,999. The output file matches each result back to its custom_id, and a separate error file lists exactly which lines failed and why, so the tenant does not have to diff the whole input against the whole output to find the two that broke.

Bulk export: the other direction

Exporting requests history works the same way in reverse: POST /v1/exports with a date range and format (CSV or JSONL), returns a job ID, and a webhook or poll tells the tenant when a download URL is ready. Never generate a large export synchronously inside a request handler. A “small” export today is a multi-gigabyte one in a year, and an HTTP request handler holding a connection open for minutes while it streams a database cursor to a response is exactly the kind of resource that starves every other tenant sharing that gateway pod, which is the noisy neighbor problem the multi-tenant post covered from the database side. The async job pattern sidesteps it entirely: the actual export work runs in a background worker with its own resource budget, not inside the request-handling pool.

The one rule that makes both directions safe: never make a bulk operation block a request handler. If it takes longer than a user is willing to stare at a spinner (a rough rule of thumb: a few hundred milliseconds), it is a job, not a request.

The Full Surface, at a Glance

Putting the whole dashboard and management API together:

Every row on this table made an explicit choice: which pagination style, whether it needs an idempotency key, whether it is sync or async. None of those are free defaults. Each one is a deliberate answer to a question raised earlier in this post, applied to one specific resource.

FAQ

Why not use JSON Patch everywhere instead of JSON Merge Patch, since it is more powerful?

Power you do not need is complexity you pay for anyway. Merge Patch covers “change these fields” with a plain JSON object anyone can write by hand while testing with curl. JSON Patch requires an array of op/path/value triples for even a single field change, and its real advantage (precise array operations) rarely comes up outside of collaborative editing tools. Default to Merge Patch and only add JSON Patch support if a specific endpoint genuinely needs array-level operations Merge Patch cannot express.

Should the dashboard’s list endpoints (like /v1/api-keys) use keyset pagination too, just to be consistent with /v1/requests?

No. Consistency should apply within a category of decision (all list responses share an envelope shape), not force the same pagination strategy onto every resource regardless of how it actually grows. A tenant has a handful of API keys, ever. Offset pagination there is simpler to implement, gives useful random page access, and never encounters the depth or write-instability problems keyset solves, because the table it is querying never gets big enough for those problems to exist.

Does GraphQL solve the over-fetching problem better than a BFF?

It solves a similar problem differently: instead of a purpose-built endpoint per client, GraphQL lets each client specify exactly the fields and nested resources it wants in one query, against one general schema. That is a genuine strength for clients with very different, evolving data needs (a mobile app and a web dashboard wanting different subsets of the same resources). It comes with its own costs: caching gets harder because every query is potentially unique, and query complexity needs its own guardrails so a client cannot accidentally request a deeply nested, expensive-to-resolve tree. Whether that tradeoff beats a small number of purpose-built BFFs depends on how many genuinely different clients you have and how often their needs diverge. This post’s promised GraphQL follow-up will design Relay’s dashboard API both ways and compare them directly.

Is QUERY actually going to replace POST /search any time soon?

Not in the next year or two, no. Standardization is necessary but not sufficient. Every intermediary between a client and your server (corporate proxies, older load balancers, some client HTTP libraries) needs to know what to do with a request carrying a body on an unfamiliar method, and that kind of infrastructure upgrades slowly. The realistic path is: design your search endpoints today as if they were QUERY (safe, idempotent, filter in the body, name the endpoint query not search), implement them as POST for now, and switch the method once your gateway and client libraries support it. The design decision and the transport are separable, and getting the design right now costs nothing extra.

Why sign outbound webhooks if the connection is already over HTTPS?

HTTPS proves the payload was not tampered with in transit and that the tenant is talking to the right server. It says nothing about whether the payload the tenant’s server receives on /hooks/relay actually came from Relay and not from anyone else who discovered that URL and decided to POST a fake batch.completed event to it. TLS secures the pipe. A signature over the payload proves who put something into the pipe. Both are needed, and neither substitutes for the other.

What should a webhook receiver return if it cannot process the event right away?

A 2xx anyway, as soon as the signature is verified and the event is safely enqueued, not after it is fully processed. The sender’s retry logic exists to handle real failures (the receiver was down, the network dropped the request), not to serve as your queue. If your handler holds the connection open until processing finishes, you have turned every sender’s retry timeout into your own processing deadline, and a legitimately slow-but-successful process now looks like a failure and triggers a duplicate delivery on top of the one already in flight.

Things Worth Their Own Post

This post stayed inside REST with JSON on purpose, because that combination alone has enough unresolved decisions to fill it. A few things named along the way that deserve their own full treatment:

• GraphQL version of this API: the same Relay dashboard, redesigned around a single schema and client-specified queries instead of purpose-built REST endpoints and a BFF. Where it genuinely wins, where it just relocates the same complexity, and what N+1 resolver problems look like once requests and usage_daily are both graph edges.
• RPC and gRPC for internal services: Relay’s own microservices (the router talking to the health monitor, the billing worker talking to the usage aggregator) do not need REST’s discoverability or HTTP semantics. Protobuf contracts, streaming RPCs, and what changes when the client is always your own code, not a third-party developer.
• SOAP and XML for legacy enterprise integration: some of Relay’s enterprise prospects still run integration layers built for a SOAP world: WSDL contracts, XML schemas, and the reasons this stack still shows up in real RFPs from large, regulated companies, whether anyone likes it or not.