In How Multi-Tenant SaaS Actually Works, I left one major promise unfulfilled at the end: “Billing is an entire system design on its own. The billing pipeline gets its own post.”

This is that post.

Building a billing pipeline is different from building almost any other backend component. If your caching layer has a bug, latencies increase by 50ms. If your log aggregator drops a packet, a dashboard chart dips for a minute. But if your billing pipeline has a bug, it operates silently in the dark. It either undercharges your customers (and your business bleeds cash without throwing a single 500 status code) or overcharges them (and you wake up to customer support tickets, disputes, and ruined trust).

Relay pays upstream LLM providers per token, marks up the cost by 20%, and bills tenants monthly based on their total usage. In this post, we will design the complete billing system that makes this work reliably.

Whether you use Stripe, Razorpay, Paddle, PayU, or PayPal, the fundamental engineering problems are identical. They all emit webhooks, they all encounter payment failures, and they all model subscriptions using the same underlying mechanics. We are going to design this system using clean, provider-agnostic architecture.

How Relay’s Billing Model Works

Before diving into webhooks and database schemas, let us lay out how data moves through Relay’s billing pipeline across three distinct layers:

1. Real-Time Cost Tracking: Every API request passing through Relay logs token usage and cost calculation into the requests table.
2. Asynchronous Aggregation: A background worker continuously aggregates raw request data into the usage_daily rollup table.
3. Monthly Metered Invoicing: At the end of a billing cycle, Relay’s billing worker queries the daily usage, compiles invoice line items, and submits them to the payment gateway.

Local Database State vs Gateway State

The payment gateway is always the source of truth for payment status, customer card details, and invoice settlements. Relay maintains a local replica in Postgres to serve API authorization checks and dashboard views without making remote HTTP calls on every incoming API request.

 1CREATE TABLE tenant_billing (
 2    tenant_id               TEXT PRIMARY KEY REFERENCES tenants(id),
 3    gateway                 TEXT NOT NULL,              -- stripe | razorpay | paddle | payu
 4    gateway_customer_id     TEXT NOT NULL UNIQUE,       -- cus_... | cust_... | ctm_...
 5    gateway_subscription_id TEXT,                       -- sub_... | null until subscribed
 6    subscription_status     TEXT,                       -- active | past_due | canceled | ...
 7    current_period_start    TIMESTAMPTZ,
 8    current_period_end      TIMESTAMPTZ,
 9    created_at              TIMESTAMPTZ NOT NULL DEFAULT now(),
10    updated_at              TIMESTAMPTZ NOT NULL DEFAULT now()
11);
12
13CREATE TABLE tenant_subscription_items (
14    tenant_id               TEXT NOT NULL REFERENCES tenants(id),
15    model_family            TEXT NOT NULL,              -- openai_gpt4 | anthropic_claude | ...
16    gateway_item_id         TEXT NOT NULL,              -- subscription item ID from gateway
17    gateway_price_id        TEXT NOT NULL,
18    updated_at              TIMESTAMPTZ NOT NULL DEFAULT now(),
19    PRIMARY KEY (tenant_id, model_family)
20);

Always create the gateway customer object eagerly during tenant provisioning rather than lazily when they add a card. Every downstream flow (adding payment methods, applying credits, emitting metered usage) assumes a valid gateway_customer_id exists. Handling lazy customer creation forces conditional logic across dozens of handlers, creating unnecessary surface area for bugs.

Subscription Lifecycle Management

Payment gateways model recurring billing using finite state machines. While vendor documentation uses different terms for these states, their operational behavior is identical.

Here is how common gateway states map to Relay’s internal subscription model:

Subscription Creation and Initial Payment

When a customer signs up for a paid plan, Relay creates a subscription object at the gateway.

1. Synchronous Card Verification: For traditional card payments without extra verification, the initial charge processes immediately. The subscription transitions directly from pending to active.
2. Asynchronous 3D Secure / SCA: Under regulations like PSD2 in Europe or RBI guidelines in India, payments frequently require multi-factor authentication (3D Secure). The gateway returns a pending status alongside a redirect URL. Relay keeps the subscription state as pending until the customer completes authentication on their bank’s portal.

Renewal Cycles and Invoice Draft Windows

At the end of a billing cycle, the gateway automatically generates a renewal invoice.

Most payment gateways provide a temporary draft window (typically 1 hour) between invoice creation and charge execution. This draft window acts as an operational buffer. If Relay’s hourly usage worker experienced a transient delay near midnight, the system uses the invoice.created webhook event to perform a final reconciliation pass, injecting any missing line items before the gateway finalizes the bill.

Idempotent Webhook Processing

Webhooks are delivered over public networks using at-least-once delivery guarantees. This means your system will receive identical events multiple times, and events will occasionally arrive out of sequence.

A webhook processing pipeline must obey three strict engineering rules:

1. Verify signature and enqueue instantly: Validate signatures against raw payload bytes, insert the event ID into a storage table, enqueue a worker task, and return HTTP 200 immediately.
2. Deduplicate atomically: Use database constraints to prevent double-processing.
3. Re-fetch live state: Do not rely on event payload snapshots. Fetch the latest state directly from the gateway API before applying changes.

Raw Signature Verification

Always verify HMAC signatures on the raw byte stream from the HTTP request body. Never parse JSON first and verify second; JSON parsers do not preserve key ordering or whitespace formatting, which invalidates the computed signature digest.

 1package billing
 2
 3import (
 4	"crypto/hmac"
 5	"crypto/sha256"
 6	"encoding/hex"
 7	"errors"
 8	"net/http"
 9)
10
11type GatewayEvent struct {
12	ID         string
13	Type       string
14	ObjectType string
15	ObjectID   string
16	RawPayload []byte
17}
18
19type PaymentGateway interface {
20	Name() string
21	VerifyWebhook(body []byte, headers http.Header) (GatewayEvent, error)
22	FetchEvent(ctx context.Context, eventID string) (GatewayEvent, error)
23	GetSubscription(ctx context.Context, subID string) (Subscription, error)
24	GetInvoice(ctx context.Context, invoiceID string) (Invoice, error)
25	ReportUsage(ctx context.Context, record UsageRecord) error
26	CreateCreditNote(ctx context.Context, req CreditNoteRequest) (string, error)
27}
28
29func VerifyHMACSignature(rawBody []byte, signatureHeader, secret string) bool {
30	mac := hmac.New(sha256.New, []byte(secret))
31	mac.Write(rawBody)
32	expectedMAC := hex.EncodeToString(mac.Sum(nil))
33	return hmac.Equal([]byte(signatureHeader), []byte(expectedMAC))
34}

Atomic Database Deduplication

We track processed events using a dedicated database table. Using PostgreSQL’s ON CONFLICT (event_id) DO NOTHING, we guard against concurrent or repeated event processing inside a single database transaction.

1CREATE TABLE webhook_events (
2    event_id     TEXT PRIMARY KEY,
3    gateway      TEXT NOT NULL,
4    event_type   TEXT NOT NULL,
5    status       TEXT NOT NULL DEFAULT 'pending', -- pending | done | failed
6    payload      JSONB,
7    processed_at TIMESTAMPTZ
8);

 1func (w *BillingWorker) ProcessEvent(ctx context.Context, eventID, eventType string, payload []byte) error {
 2	return w.db.WithTx(ctx, func(tx *sql.Tx) error {
 3		res, err := tx.ExecContext(ctx, `
 4			INSERT INTO webhook_events (event_id, gateway, event_type, status, payload)
 5			VALUES ($1, $2, $3, 'pending', $4)
 6			ON CONFLICT (event_id) DO NOTHING`,
 7			eventID, w.gateway.Name(), eventType, payload)
 8		if err != nil {
 9			return err
10		}
11
12		rows, err := res.RowsAffected()
13		if err != nil {
14			return err
15		}
16		if rows == 0 {
17			// Event already processed by another worker; exit safely.
18			return nil
19		}
20
21		// Re-fetch the live object from the gateway API to handle out-of-order events.
22		event, err := w.gateway.FetchEvent(ctx, eventID)
23		if err != nil {
24			return err
25		}
26
27		if err := w.applyEvent(ctx, tx, event); err != nil {
28			return err // Rollback transaction so retry can attempt later
29		}
30
31		_, err = tx.ExecContext(ctx,
32			`UPDATE webhook_events SET status = 'done', processed_at = now() WHERE event_id = $1`,
33			eventID)
34		return err
35	})
36}

If your business logic fails, the database transaction rolls back, reverting both the tenant state update and the webhook_events insert. When the gateway retries the webhook delivery later, the worker can safely try processing it again.

Dunning Logic and Failed Payment Handling

Dunning is the automated process of managing failed payments. Involuntary churn (failures caused by expired cards, temporary bank holds, or insufficient funds) accounts for a large percentage of revenue loss in SaaS.

Dunning consists of two synchronized layers:

1. Technical Dunning: Automated retry schedules managed by the payment gateway.
2. Communication & Access Dunning: In-app banners, email notifications, feature restrictions, and eventual account suspension managed by Relay.

Soft vs Hard Declines

Payment declines fall into two broad categories:

• Soft Declines (insufficient_funds, network_timeout, processing_error): These indicate temporary issues. The gateway retries the charge automatically over a multi-day schedule. Relay grants a grace period during soft declines.
• Hard Declines (stolen_card, lost_card, invalid_account): These failures are permanent. Retrying will not succeed. Relay immediately prompts the customer for a new payment method and restricts administrative operations.

Graduated Grace Period Implementation

Relay handles dunning progressively without immediately shutting down a customer’s production traffic on the first payment failure:

 1func (w *BillingWorker) OnPaymentFailed(ctx context.Context, tx *sql.Tx, payment FailedPayment) error {
 2	tenant, err := w.tenants.GetByGatewayCustomer(ctx, payment.CustomerID)
 3	if err != nil {
 4		return err
 5	}
 6
 7	mrr, err := w.billing.GetMRR(ctx, tenant.ID)
 8	if err != nil {
 9		return err
10	}
11
12	if isHardDecline(payment.DeclineCode) {
13		w.notifyOps(ctx, tenant.ID, payment.DeclineCode, "Hard decline encountered")
14	}
15
16	switch payment.AttemptCount {
17	case 1:
18		portalURL := w.gateway.BillingPortalURL(ctx, tenant.GatewayCustomerID)
19		w.mailer.Send(ctx, tenant.BillingEmail, "payment_failed_initial", map[string]string{
20			"portal_url": portalURL,
21		})
22	case 2:
23		w.mailer.Send(ctx, tenant.BillingEmail, "payment_failed_reminder", nil)
24		if mrr > 50000 { // High value customer (> $500 MRR)
25			w.notifySales(ctx, tenant.ID, "medium_priority")
26		}
27	default:
28		w.mailer.Send(ctx, tenant.BillingEmail, "payment_failed_final", nil)
29		if mrr > 50000 {
30			w.notifySales(ctx, tenant.ID, "high_priority")
31		}
32		// Restrict administrative setup while keeping runtime proxy endpoints working
33		w.features.Restrict(ctx, tenant.ID, "create_api_keys", "upgrade_models")
34	}
35
36	return w.tenants.UpdateBillingStatus(ctx, tx, tenant.ID, "PAYMENT_FAILING", map[string]any{
37		"attempt_count": payment.AttemptCount,
38	})
39}

If payment retries fail completely and the gateway marks the subscription as uncollectible, Relay transitions the tenant status to SUSPENDED. The API gateway middleware immediately begins rejecting runtime API requests with HTTP status 402 Payment Required.

Credit Notes and Accounting Adjustments

A credit note is a formal accounting document that offsets a previously issued invoice. You should never edit or delete an existing finalized invoice in your database. Instead, issue a credit note to maintain a clean financial audit trail.

Common scenarios requiring credit notes include:

1. Outage Credits: SLA compensation issued after an upstream provider outage.
2. Partial Overcharge Refunds: Correcting token calculation overcharges.
3. Bad Debt Write-Offs: Clearing unpaid invoices for canceled accounts.

 1CREATE TABLE billing_adjustments (
 2    id                      TEXT PRIMARY KEY,
 3    tenant_id               TEXT NOT NULL REFERENCES tenants(id),
 4    gateway_credit_note_id  TEXT NOT NULL,
 5    type                    TEXT NOT NULL, -- service_credit | refund | write_off
 6    amount_cents            BIGINT NOT NULL,
 7    currency                TEXT NOT NULL DEFAULT 'USD',
 8    memo                    TEXT,
 9    issued_at               TIMESTAMPTZ NOT NULL DEFAULT now(),
10    issued_by               TEXT NOT NULL
11);

The Paid Invoice Allocation Rule

When issuing a credit note against a paid invoice, the sum of all credit allocations must equal the total invoice value being adjusted:

1Refund Amount + Customer Balance Credit + Write-off Amount = Invoice Amount Adjusted

 1type CreditNoteRequest struct {
 2	InvoiceID      string
 3	AmountCents    int64
 4	Type           string // "balance_credit" | "refund" | "write_off"
 5	Memo           string
 6	IdempotencyKey string
 7}
 8
 9func (s *BillingService) IssueServiceCredit(ctx context.Context, tenantID string, amountCents int64, memo string) error {
10	billing, err := s.db.GetTenantBilling(ctx, tenantID)
11	if err != nil {
12		return err
13	}
14
15	invoice, err := s.gateway.GetLastPaidInvoice(ctx, billing.GatewayCustomerID)
16	if err != nil {
17		return err
18	}
19
20	idempotencyKey := fmt.Sprintf("credit-%s-%d", tenantID, time.Now().UnixMilli())
21	creditNoteID, err := s.gateway.CreateCreditNote(ctx, CreditNoteRequest{
22		InvoiceID:      invoice.ID,
23		AmountCents:    amountCents,
24		Type:           "balance_credit",
25		Memo:           memo,
26		IdempotencyKey: idempotencyKey,
27	})
28	if err != nil {
29		return err
30	}
31
32	return s.db.InsertBillingAdjustment(ctx, BillingAdjustment{
33		TenantID:            tenantID,
34		GatewayCreditNoteID: creditNoteID,
35		Type:                "service_credit",
36		AmountCents:         amountCents,
37		Memo:                memo,
38		IssuedBy:            "system",
39	})
40}

Usage-Based Invoice Generation

Unlike fixed monthly software subscriptions, metered billing requires reporting cumulative consumption metrics to the gateway throughout the billing cycle.

Reporting Hourly Usage Safely

To report consumption without double-counting usage during retries, construct a deterministic idempotency key for every usage payload.

Format: usage-{tenant_id}-{model_family}-{hour_timestamp}

 1type UsageRecord struct {
 2	ItemID         string
 3	Quantity       int64
 4	At             time.Time
 5	IdempotencyKey string
 6}
 7
 8func (w *BillingWorker) ReportHourlyUsage(ctx context.Context) error {
 9	hourBucket := time.Now().UTC().Truncate(time.Hour).Add(-time.Hour)
10
11	rows, err := w.db.QueryHourlyUsage(ctx, hourBucket)
12	if err != nil {
13		return err
14	}
15
16	for _, row := range rows {
17		itemID, err := w.db.GetGatewayItemID(ctx, row.TenantID, row.ModelFamily)
18		if err != nil {
19			if errors.Is(err, ErrItemNotFound) {
20				if refreshErr := w.refreshSubscriptionItems(ctx, row.TenantID); refreshErr != nil {
21					return refreshErr
22				}
23				itemID, err = w.db.GetGatewayItemID(ctx, row.TenantID, row.ModelFamily)
24			}
25			if err != nil {
26				return err
27			}
28		}
29
30		idempotencyKey := fmt.Sprintf("usage-%s-%s-%d",
31			row.TenantID, row.ModelFamily, hourBucket.Unix())
32
33		err = w.gateway.ReportUsage(ctx, UsageRecord{
34			ItemID:         itemID,
35			Quantity:       row.TotalTokens,
36			At:             hourBucket,
37			IdempotencyKey: idempotencyKey,
38		})
39		if err != nil {
40			w.log.Error("Failed to report usage", "tenant", row.TenantID, "error", err)
41			// Continue processing other tenants
42		}
43	}
44	return nil
45}

If the hourly job fails midway through processing 1,000 tenants, re-running the job is completely safe. The gateway uses the deterministic idempotency key to ignore duplicate reports.

End-to-End Billing Architecture

Here is the complete blueprint showing how real-time request tracking, hourly aggregation, webhook processing, and dunning workflows fit together:

FAQ

Why create gateway customer objects eagerly at signup instead of lazily when adding a payment method?
Creating customer objects at signup guarantees that every tenant record in your database has a valid gateway_customer_id. This eliminates conditional checks across your codebase when attaching payment methods, processing trials, issuing promotional credits, or fetching invoices.

What happens if our webhook processor drops offline for 12 hours?
Payment gateways store undelivered webhooks and retry delivery for 3 to 7 days using exponential backoff schedules. When your processor comes back online, it processes accumulated events. Because your worker re-fetches live state from the gateway API before making database updates, processing stale events remains safe and correct.

How do we handle mid-cycle plan upgrades?
When a tenant upgrades their plan, the payment gateway replaces existing subscription items with new ones. Your system must listen for subscription.updated webhooks and update local tenant_subscription_items mappings immediately. Otherwise, hourly usage reports will target deleted item IDs and fail.

Can daily usage rollups fall behind during traffic spikes?
Yes. Aggregation jobs can lag during heavy traffic surges. For this reason, period-end invoice reconciliation queries the primary requests log table directly rather than relying solely on rollups.