# Lorbic - Full Content > Decoding the mechanics of high-performance systems. --- ## Designing a Rate Limiter **Date:** 2026-07-07 **URL:** https://lorbic.com/designing-a-rate-limiter/ **Description:** A full system design of a production rate limiter. Covers every major algorithm (token bucket, sliding window, leaky bucket, fixed window), their tradeoffs, how they break in a distributed system, how Redis fixes them, and how Stripe, Cloudflare, and API gateways actually deploy this in production. There is a class of bugs that only happen at midnight. Your API has a rate limit: 1000 requests per minute. A client hits 999 at 11:59:59 PM, then fires another 999 at 12:00:00 AM. Two windows, two clean counters, all 1998 requests allowed. Your database gets a spike it was never designed to handle, and you spend the next hour wondering how your rate limiter let this through. It did let it through. You just did not know it. This is the boundary spike problem, and it is one of about ten things that will go wrong when you design a rate limiter from scratch. The algorithm is the easy part. The hard parts are what happens when two servers see the same request at the same time, what happens when Redis goes down, and what happens when a client figures out they can route around your limits by spreading requests across IP addresses. This post designs **Throttle**, a rate limiting service you can embed as middleware or run as a standalone sidecar, the same way I built Relay in the [previous post](/how-multi-tenant-saas-works/). We will go through every major algorithm, how each one breaks in a distributed system, how to fix those breaks, and how to deploy it. --- ## What Rate Limiting Is Actually For Most people think of rate limiting as one thing. It is not. It is four different problems wearing the same name. The first is infrastructure protection. A single client with a buggy retry loop can take down your API. The rate limiter is the wall between that client and your database. The second is fair use. On a multi-tenant system like Relay, one tenant having a bad day should not become everyone else's problem. This is resource isolation at the API layer. The third is monetization. Free tier gets 100 requests per minute. Pro gets 1000. Rate limits are what make pricing tiers real rather than theoretical. The fourth is security. Slowing down brute-force login attempts, stopping credential stuffing, limiting the blast radius of a stolen API key. These four problems lead to different design decisions. A security rate limiter needs to stay effective even when Redis is down. A billing rate limiter is fine with a few seconds of lag. If you mix them up, you either overbuild the wrong thing or build the right thing in a way that fails in the wrong direction. Throttle handles all four. That means multiple limit scopes (per IP, per key, per tenant), multiple time windows (per second, per minute, per day, per month), and a design that fails gracefully instead of all at once. --- ## The Four Algorithms Before picking an algorithm, understand that every rate limiting algorithm is making a bet about what "fair" means. Some bet on time windows. Some bet on token accumulation. The bet determines what kind of abuse gets through and what kind of legitimate traffic gets punished. There are four algorithms that actually matter in production. They are not interchangeable. ### Fixed Window Counter The simplest one. Divide time into buckets of fixed length W (say, 60 seconds). Keep a counter per bucket. Add one on every request. Reject when the counter goes over the limit. ``` window: [0s ----------- 60s][60s ----------- 120s] counter: 897 0 limit: 1000 ``` **Implementation in Redis:** ```go func (r *RedisLimiter) AllowFixed(ctx context.Context, key string, limit int, window time.Duration) (bool, error) { now := time.Now().Unix() windowStart := now - (now % int64(window.Seconds())) bucketKey := fmt.Sprintf("rl:fixed:%s:%d", key, windowStart) pipe := r.client.Pipeline() incr := pipe.Incr(ctx, bucketKey) pipe.Expire(ctx, bucketKey, window*2) // 2x window so key expires naturally _, err := pipe.Exec(ctx) if err != nil { return false, err } return incr.Val() <= int64(limit), nil } ``` The window start is calculated by rounding `now` down to the nearest window boundary. Every request in that 60-second bucket hits the same counter. **The boundary spike problem.** Fixed windows have one well-known flaw. A client can send `limit` requests at 23:59:59.999 and `limit` more at 00:00:00.001, and you just let `2 * limit` requests through in 2 milliseconds. ``` limit = 1000 [ window 1 ][ window 2 ] ... 999 req|1000 req ... ^ 2 ms, 1999 requests allowed ``` This is not a hypothetical. It is how you get database crashes even when you think you have rate limiting in place. **When to use it anyway.** Fixed window is fine when the boundary burst does not matter much. Monthly quota enforcement is the classic example. If a user burns 1000 API calls in the last second of their billing month and 1000 in the first second of the next, that is a real problem for strict quota. But for most monthly caps, simple is better. --- ### Sliding Window Log Instead of counting requests per window, you store the timestamp of every request in the last W seconds and count those. ```go func (r *RedisLimiter) AllowSlidingLog(ctx context.Context, key string, limit int, window time.Duration) (bool, error) { now := time.Now() windowStart := now.Add(-window).UnixMilli() logKey := fmt.Sprintf("rl:log:%s", key) pipe := r.client.Pipeline() // Remove timestamps older than the window pipe.ZRemRangeByScore(ctx, logKey, "0", strconv.FormatInt(windowStart, 10)) // Count remaining timestamps (requests in the current window) count := pipe.ZCard(ctx, logKey) // Add this request's timestamp pipe.ZAdd(ctx, logKey, redis.Z{Score: float64(now.UnixMilli()), Member: now.UnixNano()}) pipe.Expire(ctx, logKey, window) _, err := pipe.Exec(ctx) if err != nil { return false, err } return count.Val() < int64(limit), nil } ``` We use a Redis sorted set. The score is the timestamp in milliseconds. On every request: remove old entries, count what is left, add the current timestamp. The window always looks backward exactly W seconds from right now. This is the most accurate algorithm. No boundary spike. Every point in time sees the correct count. A client who sends 1000 requests spread across the last 60 seconds will be rejected on request 1001 no matter when it arrives. The problem is memory. You store one entry per request per user. If a client is allowed 10,000 requests per minute, you store up to 10,000 sorted set members. At 1 million users each with a 1000 req/min limit, that is up to 1 billion sorted set entries. This is why production systems at high volume almost never use sliding log. Use it for low-volume, high-precision cases: login attempts, password resets, admin APIs where correctness matters more than memory. --- ### Sliding Window Counter A hybrid. It approximates the sliding window using two fixed-window counters: the current window and the previous one, weighted by how far into the current window you are. ``` window = 60s current window started 15s ago (we are 25% into it) estimated count = prev_count * (1 - 0.25) + curr_count * 1.0 = prev_count * 0.75 + curr_count ``` ```go func (r *RedisLimiter) AllowSlidingCounter(ctx context.Context, key string, limit int, window time.Duration) (bool, error) { now := time.Now() windowSecs := int64(window.Seconds()) currentWindowStart := now.Unix() - (now.Unix() % windowSecs) prevWindowStart := currentWindowStart - windowSecs currentKey := fmt.Sprintf("rl:sw:%s:%d", key, currentWindowStart) prevKey := fmt.Sprintf("rl:sw:%s:%d", key, prevWindowStart) pipe := r.client.Pipeline() curr := pipe.Get(ctx, currentKey) prev := pipe.Get(ctx, prevKey) _, _ = pipe.Exec(ctx) currentCount, _ := strconv.ParseFloat(curr.Val(), 64) prevCount, _ := strconv.ParseFloat(prev.Val(), 64) // How far are we into the current window? (0.0 = start, 1.0 = end) elapsed := float64(now.Unix()-currentWindowStart) / float64(windowSecs) estimated := prevCount*(1-elapsed) + currentCount if estimated >= float64(limit) { return false, nil } // Increment current window counter r.client.Incr(ctx, currentKey) r.client.Expire(ctx, currentKey, window*2) return true, nil } ``` This is what Cloudflare uses. Their blog post on rate limiting describes exactly this approach. You get O(1) memory per user (just two counters), near-accurate sliding window behavior, and no meaningful boundary spikes. The approximation error is a few percent at most. The formula assumes requests were spread evenly through the previous window. If they were all at the very end (a burst), the estimate will be a bit too low. In practice this error is small enough that this is the right default for most production systems. --- ### Token Bucket Token bucket works differently from the counter-based algorithms. Instead of counting past requests, you maintain a bucket that fills with tokens at a fixed rate. Each request uses one token. If the bucket is empty, the request is rejected. ``` bucket capacity = 100 tokens refill rate = 10 tokens/second t=0: bucket = 100 t=5: 100 requests arrive, bucket = 0, all allowed t=5.1: 1 request arrives, bucket = 1 (0.1s * 10), allowed t=5.1: 1 more request, bucket = 0, rejected t=6: bucket = 9, next request allowed ``` Bucket capacity controls the maximum burst. Refill rate controls sustained throughput. If someone has not touched the API in 100 seconds and the refill rate is 10 tokens/second, they can burst 1000 requests when they come back, then trickle at 10 per second. **Redis implementation using timestamps (no background thread needed):** ```go type TokenBucketState struct { Tokens float64 LastRefill int64 // Unix timestamp in milliseconds } func (r *RedisLimiter) AllowTokenBucket(ctx context.Context, key string, capacity float64, refillRate float64) (bool, error) { // refillRate = tokens per millisecond bucketKey := fmt.Sprintf("rl:tb:%s", key) // Lua script for atomicity - read-modify-write in one round trip script := redis.NewScript(` local key = KEYS[1] local capacity = tonumber(ARGV[1]) local refill_rate = tonumber(ARGV[2]) -- tokens per ms local now = tonumber(ARGV[3]) local state = redis.call("HMGET", key, "tokens", "last_refill") local tokens = tonumber(state[1]) or capacity local last_refill = tonumber(state[2]) or now -- Compute how many tokens have been added since last request local elapsed = now - last_refill tokens = math.min(capacity, tokens + elapsed * refill_rate) if tokens < 1 then -- Update last_refill even on rejection so we track time correctly redis.call("HMSET", key, "tokens", tokens, "last_refill", now) redis.call("PEXPIRE", key, math.ceil(capacity / refill_rate)) return 0 end tokens = tokens - 1 redis.call("HMSET", key, "tokens", tokens, "last_refill", now) redis.call("PEXPIRE", key, math.ceil(capacity / refill_rate)) return 1 `) now := time.Now().UnixMilli() result, err := script.Run(ctx, r.client, []string{bucketKey}, capacity, refillRate/1000.0, now).Int() if err != nil { return false, err } return result == 1, nil } ``` The Lua script is the key part. Redis runs Lua scripts atomically, so the read, compute, and write happen as a single operation. Nothing else can interleave. This kills the race condition you would get with a WATCH/MULTI/EXEC approach. Token bucket is what most production systems use for per-key limiting. It is why API clients can send a short burst if they have been idle. The burst is intentional. It handles legitimate traffic spikes without punishing users. The tradeoff is state. Each key stores two fields: tokens and last_refill. Cheap in Redis, but more than a simple counter. And you have to decide: what does a brand-new user start with? Full bucket. Otherwise new users get rejected before they even get started. --- ### Leaky Bucket Leaky bucket is the reverse of token bucket. Requests flow in and the bucket drains at a fixed rate. ``` requests go in -> [queue] -> processed at 1 req/10ms -> response | if full: reject ``` The bucket is a FIFO queue. Requests enter, a worker drains it at a constant rate. If the queue is full, new requests are dropped. Output is perfectly smooth. You never send more than 1 request per 10ms no matter how bursty the input is. You use this for shaping traffic toward a dependency, not for user-facing limits. If you are calling an external API that handles exactly 100 req/s and has no burst tolerance, a leaky bucket in front of it smooths your traffic and stops you from getting throttled. Users do not see it because it is between your code and theirs. For user-facing rate limiting, leaky bucket is usually wrong. Users experience queuing delay instead of a clean rejection, which is confusing. And you have to cap the queue size, which means deciding whether to reject immediately or wait when it fills up. Waiting adds unpredictable latency. This is more of a traffic shaper than a rate limiter. --- ## Choosing an Algorithm | Algorithm | Memory | Burst handling | Accuracy | Complexity | |---|---|---|---|---| | Fixed window | O(1) per key | Poor (boundary spike) | Low | Low | | Sliding log | O(n) per key | Exact | Highest | Medium | | Sliding counter | O(1) per key | Good (approximated) | High | Medium | | Token bucket | O(1) per key | Configurable | High | Medium | | Leaky bucket | O(queue) | Smooths it away | High | High | For Throttle: - **Token bucket** for per-key and per-IP short-window limits. Configurable burst, O(1) memory, accurate enough. - **Sliding counter** for per-tenant limits over longer windows (hourly, daily). Low memory, near-exact, no thundering herd at window boundaries. - **Fixed window** for monthly quota only. When the window is 30 days, simplicity wins. --- ## The Distributed Problem So far the algorithms look reasonable on paper. Pick token bucket, write the Lua script, done. But all of that assumes one Redis instance and one application server. Production is neither, and that changes everything. When you run multiple API servers behind a load balancer, rate limit state needs to be consistent across all of them. If it is not, clients can bypass the limit just by having their requests spread across servers. ```mermaid graph TD Client -->|1000 requests| LB[Load Balancer] LB -->|500 requests| Server1 LB -->|500 requests| Server2 Server1 -->|counter=500, under limit| OK1[Allow] Server2 -->|counter=500, under limit| OK2[Allow] style OK1 fill:#ef4444 style OK2 fill:#ef4444 ``` Both servers see 500 requests. Both think the user is under the limit. The user sent 1000 and all went through. If each server has its own local counter with no sync, your "1000 req/min" limit is really `1000 * num_servers` in practice. The fix is shared state. This is why Redis is the standard for distributed rate limiting. Every server reads and writes to the same Redis instance. ```mermaid graph TD Client -->|1000 requests| LB[Load Balancer] LB -->|500 requests| Server1 LB -->|500 requests| Server2 Server1 -->|INCR counter| Redis[(Redis)] Server2 -->|INCR counter| Redis Redis -->|counter=1000, over limit| Both[Reject] ``` Now the counter is shared. But you just added a network hop on every request for the rate limit check. That is on the critical path. Redis is fast (under a millisecond), but it is not free. ### The Race Condition Even with shared Redis, there is a race condition in the naive implementation. ``` Server 1: GET counter -> 999 Server 2: GET counter -> 999 (reads before Server 1 writes) Server 1: SET counter = 1000 -> allow Server 2: SET counter = 1000 -> allow (wrong, should be 1001) ``` GET followed by SET is not atomic. Between the read and the write, another server can jump in. The fix is atomic operations. For simple counters, `INCR` and `INCRBY` are atomic in Redis. A single INCR atomically reads, increments, and returns the new value. No race. For complex state like token bucket, use a Lua script. Redis runs Lua atomically with no interleaving. This is exactly why the token bucket implementation above uses a script. There is also WATCH/MULTI/EXEC, Redis's optimistic transaction mode. You watch a key, start a transaction, read, then exec. If someone else modified the key between WATCH and EXEC, the transaction aborts and you retry. It works, but you need retry logic and it fails badly under high contention. Lua scripts are simpler and more reliable. Use them. --- ## The Redis Failure Problem Redis is on the critical path now. If it goes down, every request fails the rate limit check. You have to decide: **fail open** or **fail closed**. **Fail open** means if Redis is down, let all requests through. The system runs unprotected. You take the traffic risk to stay available. Right choice for non-security limits like billing tiers and usage dashboards. **Fail closed** means if Redis is down, reject everything. You stay protected at the cost of total unavailability. Right choice for security limits like login brute force and account lockout. **The hybrid** is local fallback. Each server keeps a local in-process rate limiter, just a simple atomic counter or token bucket in memory. When Redis is unreachable, fall back to local limits. Local limits are per-server, so the effective limit becomes `local_limit * num_servers`, but that beats either zero protection or a total outage. ```go type TieredLimiter struct { redis *RedisLimiter local *LocalLimiter // in-process token bucket per key policy FailurePolicy // FailOpen or FailClosed or LocalFallback } func (t *TieredLimiter) Allow(ctx context.Context, key string, limit Limit) (Decision, error) { decision, err := t.redis.Allow(ctx, key, limit) if err == nil { return decision, nil } // Redis is down switch t.policy { case FailOpen: return Allow, nil case FailClosed: return Deny, nil case LocalFallback: return t.local.Allow(key, limit.ScaledBy(1.0/float64(t.serverCount))) } return Allow, nil } ``` The `ScaledBy` call matters. If you run 10 servers and the real limit is 1000 req/min, the local fallback limit should be 100 req/min. Across all 10 servers, the aggregate stays at 1000. --- ## The Sliding Window Race The sliding counter has its own distributed race. The read-modify-write across the current and previous window counters is not atomic. ```go // This is NOT safe without additional atomicity curr := pipe.Get(ctx, currentKey) // read prev := pipe.Get(ctx, prevKey) // read // ... compute estimated ... pipe.Incr(ctx, currentKey) // write ``` Another server can increment between your read and your write. Wrap it in Lua. ```go var slidingScript = redis.NewScript(` local curr_key = KEYS[1] local prev_key = KEYS[2] local limit = tonumber(ARGV[1]) local window_secs = tonumber(ARGV[2]) local elapsed_ratio = tonumber(ARGV[3]) -- 0.0 to 1.0 local curr = tonumber(redis.call("GET", curr_key)) or 0 local prev = tonumber(redis.call("GET", prev_key)) or 0 local estimated = prev * (1 - elapsed_ratio) + curr if estimated >= limit then return 0 end redis.call("INCR", curr_key) redis.call("EXPIRE", curr_key, window_secs * 2) return 1 `) ``` One round trip, atomic. This is the rule: any time you have a read-compute-write on rate limit state, put it in Lua. --- ## Who to Limit A rate limiter is only as good as the identity it limits on. The key you choose determines how the limit behaves. **By IP address.** Simple. Works against naive scrapers. Breaks behind NAT (a company office might have 500 employees sharing one IP), breaks against distributed attackers who rotate IPs, and it punishes shared infrastructure like CI runners. **By API key.** The right choice for authenticated APIs. Each key gets its own limit. One bad key does not affect anyone else. This is the main identity for Throttle. **By tenant.** Useful for aggregate limits across an organization. An enterprise tenant with 50 API keys should not get 50 times the limit just by spreading requests across keys. **By user.** For consumer products with session-based auth. In production you apply all of these at once, in layers: ```mermaid flowchart TD Req[Request] --> IP["IP limit (1000/min)"] IP -->|allowed| Key["API key limit (5000/min)"] Key -->|allowed| Tenant["Tenant aggregate (50000/min)"] Tenant -->|allowed| Global["Global system (10M/min)"] Global -->|allowed| Handler IP -->|denied| R1[429] Key -->|denied| R2[429] Tenant -->|denied| R3[429] Global -->|denied| R4[429] ``` Each layer catches a different kind of abuse. IP limits stop anonymous scrapers. Key limits handle a compromised key. Tenant limits stop one big customer from saturating the system. Global limits are the last line of defense. The layered approach has a nice property: you only check deeper layers if the outer ones pass. If the IP is already over the limit, you never touch the API key counter. This saves Redis reads on already-rejected traffic. --- ## The Full Data Model Throttle stores limit configurations in Postgres and live state in Redis. ```sql -- Limit policies, referenced by API keys and tenants CREATE TABLE rate_limit_policies ( id TEXT PRIMARY KEY, name TEXT NOT NULL, -- "free_tier", "pro_tier", "enterprise" -- Per-key limits req_per_second INT NOT NULL DEFAULT 10, req_per_minute INT NOT NULL DEFAULT 500, req_per_hour INT NOT NULL DEFAULT 10000, req_per_day INT NOT NULL DEFAULT 100000, req_per_month INT NOT NULL DEFAULT 1000000, -- Burst headroom (token bucket capacity multiplier) burst_multiplier FLOAT NOT NULL DEFAULT 1.5, -- Algorithm override (null = system default) algorithm TEXT, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), updated_at TIMESTAMPTZ NOT NULL DEFAULT now() ); -- Per-key overrides (when a key needs a non-default policy) CREATE TABLE api_key_rate_limits ( api_key_id TEXT PRIMARY KEY REFERENCES api_keys(id), policy_id TEXT NOT NULL REFERENCES rate_limit_policies(id), custom_override JSONB -- ad-hoc overrides without a new policy row ); -- Blocking rules (IPs, tenants, keys that are permanently blocked) CREATE TABLE rate_limit_blocks ( id TEXT PRIMARY KEY, scope TEXT NOT NULL, -- 'ip', 'key', 'tenant' value TEXT NOT NULL, -- the IP or key ID or tenant ID reason TEXT, blocked_at TIMESTAMPTZ NOT NULL DEFAULT now(), expires_at TIMESTAMPTZ, -- null = permanent UNIQUE (scope, value) ); CREATE INDEX idx_blocks_scope_value ON rate_limit_blocks(scope, value); CREATE INDEX idx_blocks_expires_at ON rate_limit_blocks(expires_at) WHERE expires_at IS NOT NULL; -- Persistent rate limit violations log (for billing, audit, anomaly detection) CREATE TABLE rate_limit_events ( id TEXT PRIMARY KEY, tenant_id TEXT, api_key_id TEXT, ip INET, scope TEXT NOT NULL, -- 'ip', 'key', 'tenant', 'global' limit_key TEXT NOT NULL, -- the Redis key that was hit limit_value INT NOT NULL, -- the limit that was exceeded count INT NOT NULL, -- the count at time of rejection window INTERVAL NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT now() ) PARTITION BY RANGE (created_at); ``` The `rate_limit_events` table is append-only and partitioned by month. It feeds dashboards ("you are at 90% of your limit"), anomaly detection ("this IP hit rate limits 500 times in the last hour"), and the upsell pipeline ("this tenant exceeded their plan limit 3 times this week"). --- ## The Middleware Here is how Throttle wires into an HTTP server. It resolves context from the request, makes a decision, and either passes through or rejects. ```go type Limiter interface { Allow(ctx context.Context, req *LimitRequest) (*Decision, error) } type LimitRequest struct { IP string APIKeyID string TenantID string Policy *Policy } type Decision struct { Allowed bool Limit int Remaining int ResetAt time.Time RetryAfter time.Duration // set when Allowed=false Scope string // which scope denied: "ip", "key", "tenant" } func RateLimitMiddleware(limiter Limiter) func(http.Handler) http.Handler { return func(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { ctx := r.Context() tc := TenantFromContext(ctx) // set by auth middleware upstream req := &LimitRequest{ IP: realIP(r), APIKeyID: tc.APIKeyID, TenantID: tc.TenantID, Policy: tc.RateLimitPolicy, } decision, err := limiter.Allow(ctx, req) if err != nil { // Redis is unavailable - policy decision (see failure handling) if isRedisDown(err) { slog.Warn("rate limiter degraded", "err", err) // For this example: fail open, continue to handler next.ServeHTTP(w, r) return } http.Error(w, "internal error", http.StatusInternalServerError) return } // Always set rate limit headers, even on success w.Header().Set("X-RateLimit-Limit", strconv.Itoa(decision.Limit)) w.Header().Set("X-RateLimit-Remaining", strconv.Itoa(decision.Remaining)) w.Header().Set("X-RateLimit-Reset", strconv.FormatInt(decision.ResetAt.Unix(), 10)) if !decision.Allowed { w.Header().Set("Retry-After", strconv.Itoa(int(decision.RetryAfter.Seconds()))) w.WriteHeader(http.StatusTooManyRequests) json.NewEncoder(w).Encode(map[string]any{ "error": "rate_limit_exceeded", "scope": decision.Scope, "retry_after": decision.RetryAfter.Seconds(), "message": fmt.Sprintf("Rate limit exceeded. Retry in %.1fs.", decision.RetryAfter.Seconds()), }) return } next.ServeHTTP(w, r) }) } } // realIP handles X-Forwarded-For and X-Real-IP headers from load balancers. // Never trust the first IP in XFF blindly. Take the last IP added // by a trusted proxy if you control the proxy chain. func realIP(r *http.Request) string { if xff := r.Header.Get("X-Forwarded-For"); xff != "" { parts := strings.Split(xff, ",") return strings.TrimSpace(parts[len(parts)-1]) } if xri := r.Header.Get("X-Real-IP"); xri != "" { return xri } host, _, _ := net.SplitHostPort(r.RemoteAddr) return host } ``` The response headers matter. Clients that read `X-RateLimit-*` headers can back off on their own. `Retry-After` tells them exactly when to retry. Stripe's client libraries implement exponential backoff using these headers. A rate limiter that only returns 429 with no headers forces clients to guess, and guessing usually means retrying immediately, which makes the problem worse. --- ## The Full Request Flow ```mermaid flowchart TD Req["Incoming request + Authorization header"] --> Auth["Auth middleware, resolve tenant + policy"] Auth -->|401| Rej1[Reject] Auth -->|context set| BlockCheck["Block list check, IP + key + tenant"] BlockCheck -->|blocked| Rej2["429 or 403 for permanent blocks"] BlockCheck -->|clean| IPLimit["IP rate limit"] IPLimit -->|denied| Rej3["429 + Retry-After"] IPLimit -->|allowed| KeyLimit["API key rate limit"] KeyLimit -->|denied| Rej4["429 + Retry-After"] KeyLimit -->|allowed| TenantLimit["Tenant aggregate limit"] TenantLimit -->|denied| Rej5["429 + Retry-After"] TenantLimit -->|allowed| Handler["Route handler"] Handler --> Resp["200 + X-RateLimit headers"] ``` Each check is one Redis read. A clean request with three limit scopes costs three Redis round trips. In practice you run them concurrently: ```go func (l *TieredRedisLimiter) Allow(ctx context.Context, req *LimitRequest) (*Decision, error) { blocked, err := l.checkBlockList(ctx, req) if err != nil || blocked { return &Decision{Allowed: false, Scope: "block"}, err } // Run limit checks concurrently using errgroup var ( ipDec *Decision keyDec *Decision tenantDec *Decision ) g, ctx := errgroup.WithContext(ctx) g.Go(func() error { var err error ipDec, err = l.ipLimiter.Allow(ctx, req.IP, req.Policy.IPLimit) return err }) g.Go(func() error { var err error keyDec, err = l.keyLimiter.Allow(ctx, req.APIKeyID, req.Policy.KeyLimit) return err }) g.Go(func() error { var err error tenantDec, err = l.tenantLimiter.Allow(ctx, req.TenantID, req.Policy.TenantLimit) return err }) if err := g.Wait(); err != nil { return nil, err } // Return the most restrictive decision for _, d := range []*Decision{ipDec, keyDec, tenantDec} { if !d.Allowed { return d, nil } } // All passed - return the tightest remaining quota (for response headers) return mostRestrictive(ipDec, keyDec, tenantDec), nil } ``` Three Lua scripts run in parallel. In practice the three checks take about the same time as one. --- ## Returning Useful Rate Limit Information The 429 response is a user-facing feature. It is what developers see when their integration breaks. Getting it right cuts support tickets. ``` X-RateLimit-Limit: 1000 # The limit that applies X-RateLimit-Remaining: 847 # Requests left in the current window X-RateLimit-Reset: 1720425600 # Unix timestamp when the window resets Retry-After: 14 # Seconds until they can retry (on 429 only) ``` GitHub and Stripe both use this format. It is close enough to a standard that most HTTP clients handle it. The response body on a 429 should give the developer exactly what they need to understand and fix the problem: ```json { "error": "rate_limit_exceeded", "scope": "tenant", "limit": 50000, "current": 50001, "window": "1h", "retry_after": 847.3, "message": "Tenant aggregate limit exceeded. Your plan allows 50,000 requests/hour. Retry in 847 seconds or upgrade to Pro." } ``` The `scope` field tells the developer which limit they hit. Without it, they cannot tell if one bad client is the problem or the entire organization is over quota. The `window` tells them whether waiting 1 second or 15 minutes is appropriate. The upsell nudge in `message` also happens to reduce support tickets. --- ## Scalability A single Redis instance handles roughly 100,000 commands per second. Each rate limit check runs 1 to 3 Redis commands depending on the Lua script. At 100,000 req/s of API traffic with three limit scopes per request, you need around 300,000 Redis commands per second. That means Redis Cluster. Redis Cluster shards keys across multiple primary nodes. Rate limit keys shard naturally by hash. Each user's counter lives on one shard and stays consistent within that shard. Cross-shard operations are rare in rate limiting because each scope's key is self-contained. ```mermaid graph TD Server1 --> RC[Redis Cluster] Server2 --> RC Server3 --> RC RC --> Shard1["Shard 1, IP limits: slots 0-5460"] RC --> Shard2["Shard 2, Key limits: slots 5461-10922"] RC --> Shard3["Shard 3, Tenant limits: slots 10923-16383"] ``` **Memory.** With token bucket (two fields per key per scope) and sliding counter (two counters per key per window), memory per active user is roughly: - Token bucket state: about 200 bytes per scope - Sliding counter: about 200 bytes per active window - 3 scopes times 2 windows each = about 1.2 KB per active user At 1 million concurrent active users, that is about 1.2 GB of Redis memory. Fine for a dedicated cluster. **Local admission control.** At very high traffic, even one Redis round trip per request starts to hurt. Twitter's rate limiter uses a two-layer approach: a local approximate counter per server tracks recent requests. If clearly under the limit, skip Redis. If near the limit, do the authoritative Redis check. This cuts Redis load by around 80% under normal traffic. ```go type LocalApproximateLimiter struct { local *LocalTokenBucket remote *RedisLimiter threshold float64 // 0.8 = check Redis when local is at 80% of limit } func (l *LocalApproximateLimiter) Allow(ctx context.Context, key string, limit int) (bool, error) { localDecision := l.local.Estimate(key, limit) if localDecision.RatioUsed < l.threshold { // Clearly under limit, skip Redis l.local.Increment(key) return true, nil } // Near or over limit - do authoritative check return l.remote.Allow(ctx, key, limit) } ``` The local estimate is wrong in a multi-server world because it only sees its own server's traffic. But "clearly under limit" is almost always correct. If this server has seen 10% of the limit, the other servers would have to account for 90% for the real limit to be close. In practice traffic distributes roughly evenly. --- ## Availability **Redis Sentinel** provides automatic failover for a single Redis primary. Sentinel watches the primary and promotes a replica if it goes down. Failover takes 10 to 30 seconds. During that window your rate limiter is unavailable, which is why the local fallback from earlier matters. **Redis Cluster** gives you both sharding and HA. Each shard has a primary and replicas. If a shard primary fails, a replica is promoted. A cluster with 3 primaries and 3 replicas survives any single-node failure without meaningful disruption. Rate limit state must be written to the primary. But block list checks and policy reads can go to replicas. Routing read-only operations to replicas reduces primary load. **What happens during a network partition?** Some servers can reach Redis, some cannot. The ones that cannot fall back to local limiting. The ones that can behave normally. The two groups now have inconsistent views of the rate limit. At most `limit * (num_isolated_servers / total_servers)` extra requests leak through during the partition. This is the CAP tradeoff. To stay available, you accept some inconsistency. For rate limiting, that is the right call. A small violation of rate limits during a partition beats 100% request failure. --- ## Deployment Patterns ### Embedded Middleware Throttle runs as a Go package imported into your service. No extra network hop for the rate limit check beyond Redis. The simplest deployment. ``` Your service HTTP server Auth middleware RateLimitMiddleware (Throttle, embedded) Route handlers ``` The downside is that every service needs to import and configure Throttle separately. Configuration drifts across services. Centralized changes require redeployments. ### Sidecar Throttle runs as a sidecar process alongside each service instance. The service calls Throttle over a local unix socket or loopback. The network hop is under a millisecond. ``` Pod (Kubernetes) Service container -> POST localhost:8081/check Throttle sidecar -> Redis Cluster ``` This is how Envoy's rate limiting works. The sidecar model gives you centralized configuration while keeping latency low. ### API Gateway / Reverse Proxy Rate limiting happens at the reverse proxy before traffic reaches your service. Nginx, HAProxy, Kong, Envoy, and cloud load balancers all support this. The service never sees over-limit requests. ``` Internet -> Cloudflare (IP and DDoS limits) -> ALB -> API Gateway (key and tenant limits) -> Your service ``` This is the model for public APIs at scale. Cloudflare handles volumetric attacks. The API gateway handles per-key business logic limits. Your service handles nothing. The limitation is context. Rate limiting at the proxy means less information about the request. You cannot limit on application-level concepts like "number of tokens consumed" for an LLM API without custom logic. Proxies are good at "N requests per minute". They are not good at "N output tokens per minute". ### Centralized Rate Limit Service Run Throttle as a standalone service. Every other service calls it before processing a request. ``` Service A -> Throttle service -> Redis Service B -> Throttle service Service C -> Throttle service ``` Centralized configuration, no drift, easy to update policies. The cost: every request has a synchronous network call to Throttle on the critical path. Throttle becomes a hard dependency. If it goes down, all services fail unless each one has its own fallback logic. At high scale, Throttle itself needs to be distributed. Lyft's Ratelimit service (open source) uses this model. It is the right choice for large organizations with many independent services that need consistent rate limiting policy. --- ## Trade-offs and Costs **Accuracy vs. memory.** Sliding window log is perfectly accurate. Sliding counter approximates it. Token bucket is accurate for burst but cannot give you an exact count of total requests in a window. The more accurate the algorithm, the more memory or complexity it needs. **Consistency vs. latency.** Authoritative Redis checks give you global consistency. Local approximate checks give you low latency. Under high load, the local check should be the fast path, and you accept that a few extra requests might slip through. **Fairness across users.** A per-key limit of 1000 req/min is fair in isolation. But if 10% of your users are on enterprise plans with 100,000 req/min each, one enterprise customer can still saturate your infrastructure while every free-tier user is comfortably under their limit. You need per-scope limits at every layer, not just per-key. **The header trust problem.** `X-Forwarded-For` is set by load balancers and proxies, but it can also be set by attackers. A client can send `X-Forwarded-For: 1.2.3.4` and trick a naive rate limiter into thinking they are a different IP on every request. Always use the IP added by your own infrastructure, not the one claimed by the client. **Clock skew.** Sliding window counters depend on the current time. Across servers with clock skew, window boundaries can differ slightly. NTP usually keeps skew under 1ms, which does not matter for second-level windows. For millisecond-precision limits, be careful comparing timestamps across machines. **The thundering herd at window reset.** At midnight, when daily limits reset, every user who was at the limit can send requests at the same moment. This creates a traffic spike. Sliding windows help compared to fixed windows. You can also add random jitter to reset times per user so not everyone resets at once. --- ## What Production Systems Actually Do **Stripe** uses token bucket per API key with per-endpoint weights. Some endpoints cost more tokens than others. They have written publicly about decrementing tokens based on actual compute cost, not request count. A heavy request costs more tokens than a lightweight one. This stops "request inflation," where clients make many cheap requests to stay under count-based limits while consuming far more resources than the limit intended. **GitHub** uses sliding window with separate limits per authenticated user and per IP for unauthenticated users. Their secondary rate limit targets repeated failed requests, which signals brute-forcing or scraping rather than legitimate volume. **Cloudflare** uses the sliding window counter from this post. Their rules run at edge nodes in every data center globally, evaluated in-process with no Redis round trip. They use a custom in-memory structure per edge PoP that gossips state with nearby PoPs. That is how they rate-limit at millions of requests per second. **Twilio** applies limits at the product level. Some Twilio APIs are limited by concurrent connections, not requests per second. That requires a semaphore (a Redis counter with a max value) rather than a windowed counter. The common thread: production rate limiters are layered, they make explicit choices about consistency vs. availability, and they put real work into the developer experience of the 429 response. --- ## One Improvement Worth Building Most teams treat rate limiting as a static policy: the limit is 1000 req/min, and that is it forever. But the thing rate limiting is protecting (your database, your inference cluster, your third-party budget) does not have a fixed capacity. It changes with load, with time of day, with deploys. **Adaptive rate limiting** is the idea that the limit itself should respond to system health. If your database is at 90% CPU, tighten rate limits by 20% across all keys. When load drops back to normal, loosen them. The rate limiter becomes a feedback loop rather than a fixed gate. ```go type AdaptiveLimiter struct { base Limiter metrics SystemMetrics // CPU, latency p99, error rate } func (a *AdaptiveLimiter) scaleFactor() float64 { cpu := a.metrics.DatabaseCPU() switch { case cpu > 0.90: return 0.5 // cut limits in half case cpu > 0.75: return 0.75 default: return 1.0 } } func (a *AdaptiveLimiter) Allow(ctx context.Context, req *LimitRequest) (*Decision, error) { scaled := req.Policy.ScaledBy(a.scaleFactor()) return a.base.Allow(ctx, &LimitRequest{ IP: req.IP, APIKeyID: req.APIKeyID, TenantID: req.TenantID, Policy: scaled, }) } ``` This is simple to build on top of the existing structure. The base limiter does not change. You just scale the policy before passing it in. The tricky part is choosing the right signal. CPU is a lagging indicator. A better signal is p99 latency of your slowest dependency, because it responds faster to the start of a problem. If your database p99 goes from 20ms to 200ms, something is wrong and you want the rate limiter to react before CPU follows. The other tricky part is communicating the change to clients. If you tighten limits dynamically, `X-RateLimit-Limit` in the response headers now lies. The number the client stored from the last response is no longer accurate. You can mitigate this by returning the effective limit (already scaled) in the header, not the policy limit, so clients always see the limit that actually applies to their next request. --- ## Further Reading - [Cloudflare blog: How we built rate limiting](https://blog.cloudflare.com/counting-things-a-lot-of-different-things/) - the source for the sliding window counter approach. - [Stripe rate limiting](https://stripe.com/blog/rate-limiters) - the canonical write-up on layered rate limiting in production. - [Lyft's Ratelimit service](https://github.com/envoyproxy/ratelimit) - open source centralized rate limiting service used with Envoy. - [Redis documentation: INCR](https://redis.io/commands/incr/), [Lua scripting](https://redis.io/docs/latest/develop/programmability/eval-intro/) - the primitives everything here is built on. - [GitHub REST API rate limiting](https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api) - a real-world example of multi-tier rate limiting with decent documentation. --- The thing I keep coming back to is that rate limiting is not a feature you add. It is a set of decisions you make about who you trust, how much, and what you do when that trust runs out. Every choice in this post, the algorithm, the identity key, the failure policy, flows from that. Most teams add a Redis counter, call it done, and discover the boundary spike on a bad Tuesday. The engineers who designed Cloudflare's edge rate limiting and Stripe's API limits thought about these tradeoffs deliberately. They chose approximation over exactness, local over global, graceful degradation over hard failure. Those choices let their systems handle traffic that would have broken a naive implementation. The counter in Redis was never the hard part. --- ## How Multi-Tenant SaaS Actually Works **Date:** 2026-07-01 **URL:** https://lorbic.com/how-multi-tenant-saas-works/ **Description:** A complete system design of Relay, a multi-tenant AI API gateway. Covers multi-tenant database architecture (silo vs pool vs bridge), API key authentication, provider routing with failover, dual-layer rate limiting, token-based billing, response caching, row-level security, tenant provisioning, and observability. Full Postgres schemas, mermaid diagrams, and Go snippets included. For the past month I have been reading about multi-tenant SaaS architecture. Not as an academic exercise. I kept running into the same questions on every project I looked at: where exactly does tenant data go, how do you stop one customer's bug from becoming every customer's problem, how does billing actually work at the database level. The blog posts I found were either too abstract or skipped the hard parts entirely. So I designed one from scratch as a system design exercise. We are going to design a multi-tenant AI API gateway called **Relay** (`relay.lorbic.com`). Relay does one thing: it sits between your application and every major LLM provider. You send a single API call to Relay. Relay figures out which provider to use, forwards the request, streams the response back, counts the tokens, logs the cost, and bills you at the end of the month. You never manage provider API keys in your application. You never write a retry loop for provider outages. You just call Relay. ``` your app → POST relay.lorbic.com/v1/chat/completions { model: "gpt-5.5", messages: [...] } → Relay validates your key, checks limits, routes to OpenAI → OpenAI responds → Relay streams the response back, logs the tokens, records the cost ``` That is the product. Now let us design the system that makes it work. --- ## What is Multi-Tenant SaaS Multi-tenancy means one running instance of your software serves many separate customers. Each customer is a **tenant**. Every tenant has their own data, their own users, their own configuration. But they all share the same servers, the same codebase, and usually the same database. The word that matters here is **isolation**. Tenant A must not see tenant B's data. Tenant A must not slow down tenant B's requests. Tenant A must not exhaust resources that tenant B is counting on. Isolation is not just a nice property. It is the entire contract with your customers. In B2B SaaS, a tenant is a company. In B2C, it can be an individual user. For Relay, tenants are engineering teams: a startup that signs up, creates API keys, and calls the Relay API inside their own product. Their end users never know Relay exists. Three concepts come up everywhere in this design. It is worth defining them upfront before the schemas start: - **Tenant**: the customer unit. One company = one tenant. Has a `tenant_id` that appears on every row of data they own. - **User**: a person inside a tenant. One tenant can have many users. A user belongs to exactly one tenant at a time (though the same email address can have accounts in multiple tenants). - **Plan**: the subscription tier (free, starter, pro, enterprise). Determines what features are enabled and what rate limits apply. --- ## API Compatibility Relay uses the OpenAI API format. The same request shape, the same response shape, the same streaming protocol. This is not an accident. It means a developer can point their existing OpenAI SDK at Relay by changing one line. {{< code lang="python" >}} # Before: calling OpenAI directly client = OpenAI(api_key="sk-...") # After: calling Relay, zero other changes client = OpenAI( api_key="rly_live_sk_4xKp9mN2qR8vL3wB7tJ6cD0hF5sG1eA", base_url="https://relay.lorbic.com/v1" ) response = client.chat.completions.create( model="gpt-5.5", messages=[{"role": "user", "content": "Hello"}] ) {{< /code >}} Relay exposes three endpoints: ``` POST /v1/chat/completions — chat completions (streaming and non-streaming) POST /v1/completions — legacy text completions POST /v1/embeddings — text embeddings ``` The request body and response body match the OpenAI spec exactly. Relay adds a few optional headers for its own features: `x-relay-cache-ttl` to control caching TTL, `x-relay-fallback-models` to specify per-request fallback preferences, and `x-relay-metadata` to attach arbitrary key-value pairs to the request log. Internally, Relay translates requests to each provider's native format. Anthropic's API uses a different message structure than OpenAI's. Mistral's streaming format has small differences. The provider router handles all of that. The tenant never sees it. --- ## Choosing a Multi-Tenant Database Architecture Every multi-tenant SaaS starts with the same decision: how do you separate one customer's data from another's? There are three approaches. **Silo** gives each tenant their own database. Complete isolation. Very expensive at scale. Used by regulated industries or large enterprise contracts. **Bridge** gives each tenant their own schema inside a shared database. `acme.requests`, `beta.requests`. Medium isolation, medium cost. **Pool** puts everyone in the same tables with a `tenant_id` column on every row. Cheapest, hardest to get right, works for thousands of tenants. Relay uses the pool model. Here is why: Relay's customers are developers and startups. There will be thousands of them. The data shape is uniform across all tenants (requests, tokens, costs). There is no per-tenant schema customization. And the operational cost of running a database per tenant would make the pricing unworkable. When I was reading about this, the thing that surprised me was how many production SaaS companies started with silo because it felt "safer", then spent months migrating to pool because the operational cost was unsustainable at scale. The right model depends almost entirely on who your customer is and what you are charging them. For the very few large enterprise tenants who need data residency or dedicated infrastructure, Relay offers silo as an upgrade. The architecture supports routing them to a separate database via a tenant config flag. But that is the exception, not the default. ```mermaid graph TD subgraph pool["Pool Model"] A[acme] --> DB[(relay_db)] B[beta] --> DB C[gamma] --> DB DB --> R1[tenant_id = acme rows] DB --> R2[tenant_id = beta rows] DB --> R3[tenant_id = gamma rows] end subgraph silo["Silo - Enterprise"] D[bigcorp] --> DB2[(bigcorp_db)] end ``` In the pool model, all tenants share one database. Rows are separated by `tenant_id`. In the silo model, a tenant gets a dedicated database. Relay defaults to pool. Enterprise tenants that need dedicated infrastructure are routed to their own database via a flag in `tenant_config`. --- ## System Overview Before going into each component, here is the full picture of what Relay looks like. ```mermaid graph TB subgraph client["Client"] APP[Your Application] end subgraph relayedge["Relay Edge"] GW[API Gateway] AUTH[Auth Middleware] RL[Rate Limiter] CACHE[Cache Layer] end subgraph relaycore["Relay Core"] ROUTER[Provider Router] STREAM[Stream Handler] LOGGER[Request Logger] end subgraph providers["Providers"] OAI[OpenAI] ANT[Anthropic] MIS[Mistral] GRQ[Groq] end subgraph data["Data"] PG[(Postgres)] RD[(Redis)] end subgraph background["Background"] AGG[Usage Aggregator] BILL[Billing Worker] end APP -->|POST /v1/chat/completions| GW GW --> AUTH AUTH --> RL RL --> CACHE CACHE -->|miss| ROUTER ROUTER --> OAI ROUTER --> ANT ROUTER --> MIS ROUTER --> GRQ ROUTER --> STREAM STREAM -->|SSE chunks| APP STREAM --> LOGGER LOGGER --> PG LOGGER --> RD RD --> AGG AGG --> PG PG --> BILL ``` Every request passes through auth, rate limiting, and cache check before reaching the provider router. If the cache misses, the router picks a provider, forwards the request, and streams the response back. Background workers handle everything async: logging, usage aggregation, billing, and provider health monitoring. --- ## API Key Authentication and Tenant Resolution Relay is an API product. There are no browser sessions, no subdomains per tenant, no cookie-based auth for the hot path. Authentication is an API key in the `Authorization` header. ``` Authorization: Bearer rly_live_sk_abc123xyz... ``` The key itself encodes enough information to skip a lookup in most cases, but the real resolution always hits the database (or Redis cache) once per request. ### API key design Relay uses prefixed, hashed API keys. The key the user sees is never stored. Only its SHA-256 hash is stored. ``` Full key: rly_live_sk_4xKp9mN2qR8vL3wB7tJ6cD0hF5sG1eA Prefix: rly_live_ Environment: live (vs test) Random part: 4xKp9mN2qR8vL3wB7tJ6cD0hF5sG1eA (256 bits, URL-safe base64) ``` The prefix makes it easy to detect keys in source code scanning, error logs, and GitHub secret scanning. Stripe does this. Anthropic does this. It is a good pattern. ```sql CREATE TABLE api_keys ( id TEXT PRIMARY KEY, tenant_id TEXT NOT NULL REFERENCES tenants(id), name TEXT NOT NULL, -- "Production", "Dev", "CI" key_prefix TEXT NOT NULL, -- rly_live_ or rly_test_ key_hash TEXT NOT NULL UNIQUE, -- SHA-256 of the full key key_hint TEXT NOT NULL, -- last 4 chars: "...1eA" environment TEXT NOT NULL DEFAULT 'live', status TEXT NOT NULL DEFAULT 'active', last_used_at TIMESTAMPTZ, expires_at TIMESTAMPTZ, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), created_by TEXT NOT NULL REFERENCES users(id) ); CREATE INDEX idx_api_keys_tenant_id ON api_keys(tenant_id); CREATE INDEX idx_api_keys_key_hash ON api_keys(key_hash); ``` You never store the full key. When you create a key, you show it to the user once and tell them to copy it. After that, if they lose it, they rotate it. The `key_hint` column (last 4 characters) helps them identify which key is which in the dashboard. ### Test mode `rly_test_sk_` keys run in test mode. Relay does not forward test requests to real providers. It returns deterministic mocked responses from a fixed response library. This means no provider costs during development, predictable responses in CI/CD pipelines, and no test traffic polluting the live usage dashboard. Test mode behaves identically to live mode in every other way: rate limits apply, tokens are counted, requests are logged, errors are returned in the same format. The only difference is the backend. When Relay sees a test key, the router skips provider selection entirely and calls a local mock handler instead. ```sql -- Requests table distinguishes test from live via the api_key's environment -- api_keys.environment = 'live' | 'test' -- usage_daily is partitioned by environment so dashboards stay clean ``` Test responses are deterministic based on the request hash. The same input always produces the same mock output. This makes test assertions reliable. ### Auth flow ```mermaid flowchart LR A["Request\nAuthorization: Bearer rly_live_sk_..."] --> B["Extract key\nfrom header"] B --> C["SHA-256 hash\nthe key"] C --> D{"Redis cache\nkey_hash lookup?"} D -->|hit| E["Return cached\ntenant + key metadata"] D -->|miss| F["Postgres lookup\nWHERE key_hash = ?"] F --> G{"Found and \nstatus = active?"} G -->|no| H[401 Unauthorized] G -->|yes| I["Cache in Redis\nTTL 5 minutes"] I --> J["Inject TenantContext\ninto request"] J --> K[Next middleware] ``` The key is hashed on every request and looked up in Redis first. A cache hit means zero database reads for auth. A miss hits Postgres and populates Redis for the next 5 minutes. The resolved tenant context flows through every subsequent middleware and handler via the request context. {{< code lang="go" >}} func APIKeyMiddleware(keys KeyStore, cache Cache) func(http.Handler) http.Handler { return func(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { raw := strings.TrimPrefix(r.Header.Get("Authorization"), "Bearer ") if raw == "" { writeError(w, 401, "missing api key") return } hash := sha256Hex(raw) key, err := cache.GetAPIKey(r.Context(), hash) if err != nil { key, err = keys.GetByHash(r.Context(), hash) if err != nil || key.Status != "active" { writeError(w, 401, "invalid api key") return } cache.SetAPIKey(r.Context(), hash, key, 5*time.Minute) } ctx := WithAPIKey(r.Context(), key) next.ServeHTTP(w, r.WithContext(ctx)) }) } } {{< /code >}} --- ## Database Schema Design ### Tenants and users ```sql CREATE TABLE tenants ( id TEXT PRIMARY KEY, -- ten_abc123 slug TEXT UNIQUE NOT NULL, -- acme name TEXT NOT NULL, -- Acme Inc. plan TEXT NOT NULL DEFAULT 'free', -- free | starter | pro | enterprise status TEXT NOT NULL DEFAULT 'trial',-- trial | active | suspended | deleted created_at TIMESTAMPTZ NOT NULL DEFAULT now(), updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), deleted_at TIMESTAMPTZ, metadata JSONB ); CREATE INDEX idx_tenants_slug ON tenants(slug); CREATE INDEX idx_tenants_status ON tenants(status); CREATE TABLE users ( id TEXT PRIMARY KEY, tenant_id TEXT NOT NULL REFERENCES tenants(id), email TEXT NOT NULL, role TEXT NOT NULL DEFAULT 'member', -- owner | admin | member | viewer status TEXT NOT NULL DEFAULT 'active', created_at TIMESTAMPTZ NOT NULL DEFAULT now(), updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), deleted_at TIMESTAMPTZ, UNIQUE(tenant_id, email) ); CREATE INDEX idx_users_tenant_id ON users(tenant_id); CREATE INDEX idx_users_email ON users(email); ``` ### Model catalog Relay maintains a catalog of all supported models with their pricing. This drives cost calculation and provider routing. ```sql CREATE TABLE providers ( id TEXT PRIMARY KEY, -- openai | anthropic | mistral | groq name TEXT NOT NULL, -- OpenAI base_url TEXT NOT NULL, -- https://api.openai.com/v1 status TEXT NOT NULL DEFAULT 'active', -- active | degraded | down priority INT NOT NULL DEFAULT 100, -- lower = more preferred created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); CREATE TABLE models ( id TEXT PRIMARY KEY, -- gpt-5.5 | claude-3-5-sonnet-20241022 provider_id TEXT NOT NULL REFERENCES providers(id), relay_name TEXT UNIQUE NOT NULL, -- the name tenants use: "gpt-5.5" provider_name TEXT NOT NULL, -- name the provider actually accepts display_name TEXT NOT NULL, -- GPT-4o input_cost_per_1m NUMERIC NOT NULL, -- USD per 1M input tokens output_cost_per_1m NUMERIC NOT NULL, -- USD per 1M output tokens context_window INT NOT NULL, -- max tokens in context supports_streaming BOOLEAN NOT NULL DEFAULT true, supports_functions BOOLEAN NOT NULL DEFAULT false, supports_vision BOOLEAN NOT NULL DEFAULT false, status TEXT NOT NULL DEFAULT 'active', created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); CREATE INDEX idx_models_provider_id ON models(provider_id); ``` A few example rows of how the table looks: ``` id provider_id relay_name input_cost output_cost context_window gpt-5.5 openai gpt-5.5 2.50 10.00 128000 claude-3-5-sonnet-20241022 anthropic claude-3-5-sonnet 3.00 15.00 200000 mistral-large-2411 mistral mistral-large 2.00 6.00 131072 llama-3.3-70b-versatile groq llama-3.3-70b 0.59 0.79 128000 ``` Costs are in USD per 1 million tokens. ### Provider credentials Relay uses its own API keys to call providers. These are pooled: Relay pays the providers, marks up the cost, and charges tenants. For enterprise tenants who want to bring their own keys (BYOK) and pay providers directly, Relay supports that too. The `tenant_id` being NULL means the credential belongs to Relay. Provider credentials are among the most sensitive data Relay stores. A leaked provider key means an attacker can run LLM calls billed to that key. The `key_encrypted` column stores the key encrypted with AES-256-GCM using a data encryption key (DEK) that is itself encrypted by a master key stored in a KMS (AWS KMS or HashiCorp Vault). The plaintext key never touches disk. Relay calls the KMS API to decrypt the DEK at runtime, decrypts the credential in memory, uses it for the provider request, and discards it. Key rotation works in two steps: the KMS master key is rotated on a schedule (annually at minimum), then a background job re-encrypts every `key_encrypted` value under the new master key. The job runs row by row with checkpointing so it can resume if interrupted. ```sql CREATE TABLE provider_credentials ( id TEXT PRIMARY KEY, tenant_id TEXT REFERENCES tenants(id), -- NULL = Relay's pooled key provider_id TEXT NOT NULL REFERENCES providers(id), key_encrypted BYTEA NOT NULL, -- encrypted with Relay's KMS key key_hint TEXT NOT NULL, -- last 4 chars for display is_relay_owned BOOLEAN NOT NULL DEFAULT true, status TEXT NOT NULL DEFAULT 'active', rpm_limit INT, -- provider's RPM cap for this key tpm_limit INT, -- provider's TPM cap for this key created_at TIMESTAMPTZ NOT NULL DEFAULT now(), UNIQUE(tenant_id, provider_id) ); ``` ### Requests Every API call Relay processes gets a row here. This is the source of truth for billing, debugging, and usage dashboards. ```sql CREATE TABLE requests ( id TEXT PRIMARY KEY, -- req_abc123 tenant_id TEXT NOT NULL REFERENCES tenants(id), api_key_id TEXT NOT NULL REFERENCES api_keys(id), model_id TEXT NOT NULL REFERENCES models(id), provider_id TEXT NOT NULL REFERENCES providers(id), credential_id TEXT NOT NULL REFERENCES provider_credentials(id), -- What kind of request request_type TEXT NOT NULL, -- chat | completion | embedding stream BOOLEAN NOT NULL DEFAULT false, -- Token counts (filled in after response) input_tokens INT, output_tokens INT, total_tokens INT, -- Cost in USD input_cost NUMERIC(12,8), output_cost NUMERIC(12,8), total_cost NUMERIC(12,8), -- Relay's markup (stored separately for accounting) markup_rate NUMERIC(5,4) NOT NULL DEFAULT 0.20, -- 20% markup by default relay_revenue NUMERIC(12,8), -- Timing started_at TIMESTAMPTZ NOT NULL, first_token_at TIMESTAMPTZ, -- for streaming: TTFT completed_at TIMESTAMPTZ, latency_ms INT, ttft_ms INT, -- time to first token -- Outcome status TEXT NOT NULL, -- success | error | cached | timeout error_code TEXT, error_message TEXT, http_status INT, -- Cache cache_hit BOOLEAN NOT NULL DEFAULT false, cache_key TEXT, created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); CREATE INDEX idx_requests_tenant_created ON requests(tenant_id, created_at DESC); CREATE INDEX idx_requests_model_created ON requests(model_id, created_at DESC); CREATE INDEX idx_requests_api_key_created ON requests(api_key_id, created_at DESC); CREATE INDEX idx_requests_status ON requests(status, created_at DESC); ``` The `requests` table grows fast. At 1,000 tenants each making 100 requests per day, that is 100,000 rows per day, 3 million per month. After 90 days you want to start archiving cold rows to cheaper storage (S3 + Parquet). Keep hot rows (last 30 days) in Postgres. Query historical data via a warehouse. ### Daily usage rollup Querying `requests` for billing is expensive at scale. Aggregate into a daily rollup table that background workers update continuously. ```sql CREATE TABLE usage_daily ( tenant_id TEXT NOT NULL REFERENCES tenants(id), model_id TEXT NOT NULL REFERENCES models(id), date DATE NOT NULL, request_count BIGINT NOT NULL DEFAULT 0, input_tokens BIGINT NOT NULL DEFAULT 0, output_tokens BIGINT NOT NULL DEFAULT 0, total_tokens BIGINT NOT NULL DEFAULT 0, total_cost NUMERIC(12,4) NOT NULL DEFAULT 0, relay_revenue NUMERIC(12,4) NOT NULL DEFAULT 0, cache_hits BIGINT NOT NULL DEFAULT 0, error_count BIGINT NOT NULL DEFAULT 0, avg_latency_ms INT, PRIMARY KEY (tenant_id, model_id, date) ); ``` --- ## Request Flow: From API Call to Provider and Back This is the core of Relay. Every API call goes through this pipeline. ```mermaid sequenceDiagram actor App as Your App participant GW as Relay Gateway participant Auth as Auth Middleware participant RL as Rate Limiter participant Cache as Cache Layer participant Router as Provider Router participant OAI as OpenAI App->>GW: POST /v1/chat/completions
Authorization: Bearer rly_live_sk_... GW->>Auth: validate API key Auth->>Auth: SHA-256 hash key
lookup in Redis / Postgres Auth-->>GW: TenantContext injected GW->>RL: check RPM + TPM limits RL-->>GW: ok GW->>Cache: check cache
hash(model + messages + params) Cache-->>GW: miss GW->>Router: route request
model=gpt-5.5 Router->>Router: lookup model in catalog
select provider + credential Router->>OAI: POST api.openai.com/v1/chat/completions
Authorization: Bearer sk-relay-pooled-... OAI-->>Router: 200 OK (streaming SSE) Router-->>App: pipe SSE chunks Router->>Router: count tokens from response
calculate cost Router->>GW: log request async GW->>GW: INSERT into requests
INCR Redis counters ``` ### Model routing and failover The router resolves which provider to use for a given model name. If the primary provider is down or over its rate limit, it falls back to an alternative. ```mermaid flowchart TD A["Request
model = gpt-5.5"] --> B["Lookup model
in catalog"] B --> C["Get providers
ordered by priority"] C --> D{"Tenant has
BYOK key?"} D -->|yes| E[Use tenant's key] D -->|no| F[Use Relay pooled key] E --> G{"Provider
healthy?"} F --> G G -->|yes| H[Forward request] G -->|no| I["Next provider
by priority"] I --> J{Any left?} J -->|yes| G J -->|no| K[503 No providers available] H -->|5xx from provider| I H -->|success| L[Return response] ``` {{< code lang="go" >}} func (r *Router) Route(ctx context.Context, req *RelayRequest) (*RelayResponse, error) { model, err := r.catalog.GetModel(req.Model) if err != nil { return nil, ErrModelNotFound } providers := r.getOrderedProviders(ctx, model) for _, p := range providers { cred, err := r.getCredential(ctx, req.TenantID, p.ID) if err != nil { continue } resp, err := r.forward(ctx, p, cred, req) if err == nil { return resp, nil } // 5xx from provider: try next if isProviderError(err) { r.health.MarkDegraded(p.ID) continue } // 4xx from provider: the request itself is bad, do not retry return nil, err } return nil, ErrNoProvidersAvailable } {{< /code >}} ### Handling streaming (SSE) When the tenant sends `"stream": true`, the provider returns a Server-Sent Events stream. Relay proxies it chunk by chunk. This is not a buffered response, it is a live pipe. ```mermaid sequenceDiagram participant App participant Relay participant Provider App->>Relay: POST /v1/chat/completions
stream: true Relay->>Provider: POST (stream: true) Provider-->>Relay: data: {"delta": {"content": "Hello"}} Relay-->>App: data: {"delta": {"content": "Hello"}} Provider-->>Relay: data: {"delta": {"content": " world"}} Relay-->>App: data: {"delta": {"content": " world"}} Provider-->>Relay: data: [DONE] Relay-->>App: data: [DONE] Relay->>Relay: count tokens from chunks
log request async ``` Relay counts tokens as chunks arrive by tracking the `usage` field that providers send in the final chunk. For providers that do not send usage in streaming mode, Relay uses a local tokenizer (tiktoken for OpenAI models, SentencePiece for others) to estimate counts. The estimate is reconciled against the actual count when available. Token counting is done on a goroutine that reads from the same SSE stream, separate from the goroutine that proxies chunks to the client. The client sees no added latency. ### Retry logic Relay retries failed provider requests automatically, but only for the right kind of failures. **Retryable**: 5xx errors from the provider (server errors), network timeouts, connection resets. These indicate the provider had a problem, not that the request was wrong. **Not retryable**: 4xx errors from the provider (except 429). A 400 means the request was malformed. A 401 means the credential is wrong. Retrying will not help and wastes time. **Provider rate limit (429 from provider)**: Relay does not wait and retry on the same key. It immediately switches to the next available provider key or provider. Waiting for a rate limit window to reset would add seconds of latency for the tenant. Retry policy: up to 3 attempts, 100ms exponential backoff between attempts, each attempt on a different provider or key if possible. If all attempts fail, Relay returns the last error to the tenant. Relay does not retry on behalf of the tenant after returning an error. If Relay returns a 502 or 503, the tenant's code decides whether to retry at their layer. ### Error response format Every error Relay returns uses the same JSON structure. Developers should be able to write one error handler and handle all Relay errors with it. ```json { "error": { "code": "rate_limit_exceeded", "message": "You have exceeded 100 requests per minute on the free plan.", "type": "rate_limit_error", "details": { "limit": 100, "window": "per_minute", "reset_at": "2026-07-01T12:05:00Z", "upgrade_url": "https://relay.lorbic.com/billing" } } } ``` The full set of error codes: | HTTP status | error.code | Meaning | |---|---|---| | 401 | `invalid_api_key` | Key not found or revoked | | 402 | `tenant_suspended` | Account suspended, check billing | | 402 | `quota_exceeded` | Hard quota limit hit (seats, storage) | | 404 | `model_not_found` | Requested model does not exist in catalog | | 422 | `invalid_request` | Request body failed validation | | 429 | `rate_limit_exceeded` | RPM or TPM limit hit | | 502 | `provider_error` | Provider returned an error after retries | | 503 | `no_provider_available` | All providers for this model are degraded | | 504 | `request_timeout` | Provider did not respond within the timeout | The `details` field is present on `rate_limit_exceeded` and `quota_exceeded` errors so tenants can surface actionable information to their own users. If a tenant's app is hitting Relay's rate limit, they should know the reset time and the upgrade path, not just a generic "too many requests." --- ## Rate Limiting Relay has to manage rate limits at two levels. **Inbound**: protecting Relay's own infrastructure from tenants who send too many requests. **Outbound**: protecting Relay's provider API keys from hitting the provider's limits. Both use Redis counters with sliding windows. The dual-layer rate limiting was the part I found most interesting while designing this. Most rate limiting content covers the inbound side. The outbound side is specific to API gateway products but just as important. If Relay's OpenAI key hits OpenAI's TPM limit because of one large tenant, every other tenant on that key starts getting 429s from the provider. The router has to detect this and re-route before it becomes the tenant's problem. ### Inbound tenant limits ```sql -- Plan-level defaults CREATE TABLE quota_definitions ( plan TEXT NOT NULL, resource TEXT NOT NULL, -- rpm | tpm | requests_per_day limit_value BIGINT NOT NULL, -- -1 = unlimited window TEXT NOT NULL, -- per_minute | per_day UNIQUE(plan, resource, window) ); -- Sample values: -- plan resource limit window -- free rpm 60 per_minute -- free tpm 40000 per_minute -- free requests_per_day 500 per_day -- starter rpm 500 per_minute -- starter tpm 400000 per_minute -- pro rpm 3000 per_minute -- pro tpm 2000000 per_minute -- pro requests_per_day -1 per_day (unlimited) -- Per-tenant overrides (for enterprise deals or support credits) CREATE TABLE quota_overrides ( id TEXT PRIMARY KEY, tenant_id TEXT NOT NULL REFERENCES tenants(id), resource TEXT NOT NULL, limit_value BIGINT NOT NULL, window TEXT NOT NULL, reason TEXT, expires_at TIMESTAMPTZ, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), UNIQUE(tenant_id, resource, window) ); ``` Redis key structure for inbound limits: ``` rl:in:ten_abc123:rpm → sliding window counter, 60s window rl:in:ten_abc123:tpm → sliding window counter, 60s window ``` ### Outbound provider limits Each of Relay's provider API keys has its own RPM and TPM cap set by the provider. Relay tracks these in Redis too. ``` rl:out:openai:key_xyz:rpm → counter, 60s window rl:out:openai:key_xyz:tpm → counter, 60s window ``` When a pooled key is close to its provider limit, the router skips it and picks the next available key or provider. ### The rate limit pipeline ```mermaid flowchart LR A[Request] --> B{"Inbound tenant
RPM check"} B -->|over| C["429 Too Many Requests
retry-after header"] B -->|ok| D{"Inbound tenant
TPM check"} D -->|over| C D -->|ok| E[Route to provider] E --> F{"Outbound key
RPM check"} F -->|over| G["Pick next key
or provider"] F -->|ok| H[Forward to provider] G --> F ``` A request fails fast at inbound RPM or TPM before any provider call is made. The outbound check happens inside the router, per-key. If a key is saturated, the router picks the next key or the next provider, not the next retry window. The TPM check on inbound traffic uses an estimated token count from the request body (input tokens only). The actual token count (input + output) is only known after the response. Relay does a conservative pre-check against the estimated input, then reconciles and updates the TPM counter after the response. --- ## Token Cost Tracking and Billing Relay's pricing model: Relay pays the provider at published rates, marks up by 20%, and charges tenants at the marked-up rate. The markup funds Relay's infrastructure and margin. ``` Tenant sends: 1,000 input tokens, 500 output tokens on gpt-5.5 Provider cost: (1000 / 1M) × $2.50 + (500 / 1M) × $10.00 = $0.00250 + $0.00500 = $0.00750 Relay markup: $0.00750 × 1.20 = $0.00900 Tenant billed: $0.00900 Relay revenue: $0.00150 ``` Every request row in the `requests` table stores both the raw provider cost and the Relay revenue. This makes accounting and audits straightforward. ### Cost calculation ```mermaid flowchart LR A["Response received
input_tokens=1000
output_tokens=500"] --> B["Lookup model pricing
input_cost_per_1m
output_cost_per_1m"] B --> C["input_cost = tokens / 1M x rate
output_cost = tokens / 1M x rate"] C --> D[total_cost = input + output] D --> E[relay_revenue = total_cost x markup_rate] E --> F[INSERT into requests] F --> G[INCR usage_daily rollup] ``` ### Billing cycle Relay charges tenants monthly via Stripe. The billing worker runs at the start of each month, reads the `usage_daily` table for the prior month, and creates a Stripe invoice line item per model per tenant. ```sql CREATE TABLE billing_periods ( id TEXT PRIMARY KEY, tenant_id TEXT NOT NULL REFERENCES tenants(id), period_start DATE NOT NULL, period_end DATE NOT NULL, status TEXT NOT NULL DEFAULT 'open', -- open | finalized | paid | failed total_cost NUMERIC(12,4), invoice_id TEXT, -- Stripe invoice ID finalized_at TIMESTAMPTZ, paid_at TIMESTAMPTZ, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), UNIQUE(tenant_id, period_start) ); CREATE TABLE billing_line_items ( id TEXT PRIMARY KEY, billing_period_id TEXT NOT NULL REFERENCES billing_periods(id), tenant_id TEXT NOT NULL REFERENCES tenants(id), model_id TEXT NOT NULL REFERENCES models(id), request_count BIGINT NOT NULL, input_tokens BIGINT NOT NULL, output_tokens BIGINT NOT NULL, total_cost NUMERIC(12,4) NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), UNIQUE(billing_period_id, model_id) ); ``` > Billing is an entire system design on its own: Stripe subscription lifecycle, idempotent webhook handling, retry logic for failed payments, proration, credits, and tax calculation. This post covers the data model. The billing pipeline gets its own post. --- ## Response Caching: Exact Match and Semantic Caching completions saves money. If 1,000 tenants call the same model with the same system prompt and similar questions, Relay can serve many of those from cache instead of paying the provider. Relay supports two cache strategies. ### Exact match caching The simplest form. Hash the full request (model + messages + temperature + all params). If you have seen this exact request before, return the cached response. ``` cache_key = SHA-256( model_id + JSON.stringify(messages) + temperature + top_p + max_tokens + ... ) ``` Stored in Redis with a configurable TTL. Default 1 hour. Tenants can set their own TTL per API call via a `x-relay-cache-ttl` header. ### Semantic caching Exact match only works for identical requests. Semantic caching uses embeddings to find requests that are similar enough to return the same response. ```mermaid flowchart TD A[New request] --> B["Embed the user message
using a small model"] B --> C["Vector similarity search
against cached embeddings"] C --> D{"Similarity score
> threshold?"} D -->|yes| E["Return cached response
cache_hit = semantic"] D -->|no| F["Forward to provider
cache the response + embedding"] ``` The embedding model for cache lookup should be cheap and fast (text-embedding-3-small or similar). The cost of generating the lookup embedding is tiny compared to the LLM call it might save. ```sql CREATE TABLE response_cache ( id TEXT PRIMARY KEY, tenant_id TEXT NOT NULL REFERENCES tenants(id), cache_key TEXT NOT NULL, -- exact match hash model_id TEXT NOT NULL REFERENCES models(id), request_hash TEXT NOT NULL, response_body JSONB NOT NULL, input_tokens INT NOT NULL, output_tokens INT NOT NULL, embedding vector(1536), -- pgvector, for semantic cache hit_count INT NOT NULL DEFAULT 0, expires_at TIMESTAMPTZ NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); CREATE INDEX idx_response_cache_tenant_key ON response_cache(tenant_id, cache_key); CREATE INDEX idx_response_cache_embedding ON response_cache USING hnsw (embedding vector_cosine_ops) WHERE embedding IS NOT NULL; ``` --- ## Observability You cannot operate a multi-tenant API platform without per-tenant observability. When a tenant files a support ticket saying "your API is slow," you need to be able to pull up exactly their requests, their latencies, and their errors in under a minute. ### What to track per request Every request log entry should have: - `tenant_id` and `api_key_id` - `model_id` and `provider_id` - `latency_ms` and `ttft_ms` (time to first token, the metric tenants care about for streaming) - `input_tokens`, `output_tokens`, `total_cost` - `status` and `error_code` - `cache_hit` These are already in the `requests` table. Make sure every log line, trace span, and metric tag also carries `tenant_id`. ### Structured logging {{< code lang="go" >}} func RequestLoggingMiddleware(logger *slog.Logger) func(http.Handler) http.Handler { return func(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { key := APIKeyFromContext(r.Context()) reqLogger := logger.With( slog.String("tenant_id", key.TenantID), slog.String("api_key_id", key.ID), slog.String("request_id", newRequestID()), ) ctx := WithLogger(r.Context(), reqLogger) next.ServeHTTP(w, r.WithContext(ctx)) }) } } {{< /code >}} ### Health status per provider Relay's router uses a health tracker to avoid sending requests to degraded providers. The tracker works on a circuit breaker pattern: after N consecutive errors from a provider, mark it degraded and stop routing to it until it recovers. ```sql CREATE TABLE provider_health_events ( id TEXT PRIMARY KEY, provider_id TEXT NOT NULL REFERENCES providers(id), status TEXT NOT NULL, -- healthy | degraded | down error_rate NUMERIC, -- 5-minute error rate at time of event p95_ms INT, -- p95 latency at time of event message TEXT, created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); CREATE INDEX idx_provider_health_events ON provider_health_events(provider_id, created_at DESC); ``` The provider status page (status.relay.lorbic.com) reads from this table. Tenants can subscribe to status updates. ### Noisy neighbor detection In the pool model, one tenant running an aggressive batch job can saturate Relay's provider keys and cause latency for everyone else. Detect this by tracking per-tenant request volume in Redis with a 1-minute rolling window. If a single tenant exceeds a percentage threshold of total traffic, flag them and apply stricter rate limits automatically. --- ## Role-Based Access Control API calls use API key auth. The Relay dashboard (where tenants manage keys, view usage, configure settings) uses JWT auth. ```json { "iss": "relay.lorbic.com", "sub": "usr_abc123", "tenant_id": "ten_xyz789", "tenant_slug": "acme", "role": "admin", "permissions": ["api_keys:read", "api_keys:write", "billing:read", "usage:read"], "iat": 1751400000, "exp": 1751500000 } ``` ### RBAC tables ```sql CREATE TABLE roles ( id TEXT PRIMARY KEY, tenant_id TEXT NOT NULL REFERENCES tenants(id), name TEXT NOT NULL, -- owner | admin | developer | viewer is_system BOOLEAN NOT NULL DEFAULT false, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), UNIQUE(tenant_id, name) ); CREATE TABLE permissions ( id TEXT PRIMARY KEY, resource TEXT NOT NULL, -- api_keys | usage | billing | members | settings action TEXT NOT NULL, -- create | read | update | delete UNIQUE(resource, action) ); CREATE TABLE role_permissions ( role_id TEXT NOT NULL REFERENCES roles(id), permission_id TEXT NOT NULL REFERENCES permissions(id), PRIMARY KEY (role_id, permission_id) ); CREATE TABLE user_roles ( user_id TEXT NOT NULL REFERENCES users(id), role_id TEXT NOT NULL REFERENCES roles(id), PRIMARY KEY (user_id, role_id) ); ``` ### Audit log Every sensitive action in the dashboard gets recorded: key creation, key revocation, plan changes, member removal, billing configuration. ```sql CREATE TABLE audit_log ( id TEXT PRIMARY KEY, tenant_id TEXT REFERENCES tenants(id), actor_id TEXT NOT NULL, actor_type TEXT NOT NULL, -- user | platform_admin | system action TEXT NOT NULL, -- api_key.created | api_key.revoked | member.removed resource_id TEXT, resource_type TEXT, metadata JSONB, ip_address INET, created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); CREATE INDEX idx_audit_log_tenant ON audit_log(tenant_id, created_at DESC); CREATE INDEX idx_audit_log_actor ON audit_log(actor_id, created_at DESC); ``` --- ## Tenant Provisioning: From Signup to First API Call ### What happens when someone signs up When a user submits the signup form, Relay does the minimum work synchronously: insert a tenant row with `status=PROVISIONING`, create the owner user record, and return a 201. The browser redirects immediately. Everything else runs in the background. The background worker picks up the job and does the slower parts: seeding system roles, creating a Stripe customer record, and flipping the tenant to `TRIAL`. The Stripe customer is created at signup, not when the user adds a payment method. This is intentional. You need a Stripe customer ID to attach payment methods, subscriptions, and invoices later. Creating it lazily (at payment time) means every billing code path has to handle the case where the ID does not exist yet, which causes bugs. The welcome email includes the tenant's first API key. The key is generated by the worker, not the API handler, because key generation requires the tenant record to be fully set up. If the worker fails partway through, the tenant stays in `PROVISIONING`. A monitoring job scans for tenants stuck in `PROVISIONING` for more than 5 minutes and either retries the job or pages on-call. ```mermaid sequenceDiagram actor User participant API participant Queue participant Worker participant DB participant Stripe User->>API: POST /auth/signup
{ email, password, org_name } API->>DB: INSERT tenant (status=PROVISIONING) API->>DB: INSERT user (role=owner) API->>Queue: enqueue ProvisionTenant job API-->>User: 201 Created
{ tenant_id, redirect: /onboarding } Worker->>Queue: dequeue job Worker->>DB: seed default roles (owner, admin, developer, viewer) Worker->>DB: seed plan quota definitions Worker->>Stripe: create Stripe customer Worker->>DB: UPDATE tenant SET status=TRIAL, stripe_customer_id=cus_... Worker-->>User: send welcome email with first API key ``` ### Tenant status transitions The `status` column on the `tenants` table is the single source of truth for what a tenant can do. Every API request, every dashboard login, and every background job checks this column before doing anything. **PROVISIONING**: The tenant record exists but setup is not complete. API calls are rejected with a 503. The dashboard shows a loading state. **TRIAL**: Setup is complete. The tenant has full API access up to the trial plan's rate limits. A trial expiry timestamp lives in `tenant_config`. When it passes without a payment method being added, a background job moves the tenant to `EXPIRED`. **ACTIVE**: A payment method is on file and billing is working. Full access based on the tenant's plan. **SUSPENDED**: Payment failed, or a Relay admin manually suspended the account. API calls return a 402 with a clear message and a link to the billing page. The tenant's data is untouched. If payment resolves within the grace period, the account returns to `ACTIVE` with no data loss. This is intentional: you want tenants to fix their billing, not lose access to everything they built. **EXPIRED**: The trial ended without a payment method being added. Same behavior as `SUSPENDED` for API calls. After N days with no action, the tenant moves to `DELETED`. **DELETED**: Soft delete. The row stays in the database with `deleted_at` set, but all API calls are rejected. Data is retained for 30 days for recovery or dispute resolution, then a GDPR-compliant hard deletion job removes all tenant rows across every table. Status transitions are intentionally one-directional except for `SUSPENDED` back to `ACTIVE`. You cannot un-delete a tenant. You cannot go from `ACTIVE` back to `TRIAL`. These constraints are enforced at the application layer via explicit state machine logic, not just database constraints. ```mermaid stateDiagram-v2 [*] --> PROVISIONING : signup PROVISIONING --> TRIAL : provisioning complete TRIAL --> ACTIVE : payment method added TRIAL --> EXPIRED : trial period ends ACTIVE --> SUSPENDED : payment failed or admin action SUSPENDED --> ACTIVE : payment resolved SUSPENDED --> DELETED : grace period expires EXPIRED --> DELETED : no conversion ACTIVE --> DELETED : user cancels DELETED --> [*] ``` Suspended tenants get a 429 response with a billing error message on every API call. This is important: the error message should tell them what is wrong and link them to the billing page. A silent 401 causes confusion. --- ## Feature Flags and Per-Tenant Configuration ```sql CREATE TABLE plan_features ( plan TEXT NOT NULL, feature TEXT NOT NULL, -- semantic_cache | byok | sso | audit_log | higher_rpm | webhooks enabled BOOLEAN NOT NULL DEFAULT true, PRIMARY KEY (plan, feature) ); CREATE TABLE tenant_feature_overrides ( tenant_id TEXT NOT NULL REFERENCES tenants(id), feature TEXT NOT NULL, enabled BOOLEAN NOT NULL, reason TEXT, expires_at TIMESTAMPTZ, PRIMARY KEY (tenant_id, feature) ); CREATE TABLE tenant_config ( tenant_id TEXT NOT NULL REFERENCES tenants(id), key TEXT NOT NULL, value TEXT NOT NULL, updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), PRIMARY KEY (tenant_id, key) ); -- Example config rows: -- tenant_id key value -- ten_abc default_cache_ttl_sec 3600 -- ten_abc markup_rate 0.15 (negotiated lower markup for volume) -- ten_abc allowed_models ["gpt-5.5","claude-3-5-sonnet"] -- ten_abc webhook_url https://acme.com/hooks/relay ``` Cache the feature flag state per tenant in Redis (60s TTL). Do not query Postgres on every API call. --- ## Full Architecture ```mermaid graph TB subgraph tenants["Tenants"] APP1[App: acme] APP2[App: beta] DASH[Dashboard] end subgraph edge["Edge"] DNS[Wildcard DNS] LB[Load Balancer] end subgraph gateway["Gateway"] GW[API Gateway] AUTH[API Key Auth] RL_IN[Rate Limiter] CACHE[Cache Layer] ROUTER[Provider Router] STREAM[SSE Handler] end subgraph dashapi["Dashboard API"] DASH_API[Dashboard API] end subgraph providers["Providers"] OAI[OpenAI] ANT[Anthropic] MIS[Mistral] GRQ[Groq] end subgraph data["Data"] PG[(Postgres)] RD[(Redis)] S3[(S3)] end subgraph workers["Workers"] LOGGER[Request Logger] AGG[Usage Aggregator] BILL[Billing Worker] HEALTH[Health Monitor] ARCHIVE[Request Archiver] end APP1 --> DNS APP2 --> DNS DASH --> DNS DNS --> LB LB --> GW LB --> DASH_API GW --> AUTH AUTH --> RL_IN RL_IN --> CACHE CACHE -->|miss| ROUTER ROUTER --> OAI ROUTER --> ANT ROUTER --> MIS ROUTER --> GRQ ROUTER --> STREAM STREAM --> LOGGER DASH_API --> PG DASH_API --> RD LOGGER --> PG LOGGER --> RD RD --> AGG AGG --> PG PG --> BILL PG --> ARCHIVE ARCHIVE --> S3 HEALTH --> PG HEALTH --> RD ``` The gateway layer is stateless. Tenant apps hit the load balancer, which routes to any gateway pod. All shared state (rate limits, cache, feature flags) lives in Redis. All durable state (requests, usage, billing, config) lives in Postgres. Workers run separately and never block the hot path. --- ## Scaling a Multi-Tenant API Gateway Relay is stateless at the gateway layer. You can add more gateway pods behind the load balancer without coordination. The state lives in Postgres and Redis. **Redis** is the first bottleneck. Rate limit counters and cache lookups hit Redis on every request. Run Redis in cluster mode with at least 3 shards. Shard by `tenant_id` so one tenant's traffic does not create hot keys for others. **Postgres** becomes the bottleneck as the `requests` table grows. Partition it by `created_at` (monthly partitions). Drop or archive partitions older than your retention window. Use PgBouncer in transaction mode for the gateway (short, fast INSERTs) and session mode for the background workers (longer transactions, aggregation queries). **Provider key rotation**: as Relay grows, a single OpenAI API key will hit TPM limits. Relay maintains a pool of provider keys per provider and distributes load across them using the outbound rate limit counters in Redis. The router picks the key with the most remaining capacity in the current window. --- ## FAQ **Why store the hash of an API key instead of the key itself?** If your database is compromised, the attacker gets hashes, not keys. A raw API key in the database is a plaintext credential. SHA-256 is one-way: you verify a key by hashing the incoming value and comparing it against the stored hash, but you cannot reverse a hash back into the original key. Treat API keys the same way you treat passwords. **How do you prevent a missing `WHERE tenant_id` clause from leaking data across tenants?** Row-Level Security in Postgres. You attach a policy to every tenant-scoped table that restricts all queries to rows matching `app.current_tenant_id`, a session variable the application sets at the start of every request. A bare `SELECT * FROM requests` with no WHERE clause still returns only that tenant's rows. The policy runs inside the database engine, below the application layer, so a bug in application code cannot bypass it. **What happens if OpenAI goes down?** The router tries providers in priority order. If OpenAI returns a 5xx, it marks OpenAI as degraded and tries the next provider that supports the requested model. If no provider is available, the request returns 503. The health monitor runs on a short interval and flips providers back to healthy once they start returning 2xx responses again. Tenants see a brief error window during a full provider outage, not an extended one. **How do you count tokens accurately for billing when streaming does not always return usage data?** Providers that support `stream_options: { include_usage: true }` (OpenAI does) include a `usage` object in the final SSE chunk. Relay reads those counts directly. For providers that do not, Relay runs a local tokenizer (tiktoken for OpenAI-compatible models, SentencePiece for others) to estimate counts. The estimate is within 1-2% of the actual count in practice. When there is a discrepancy, Relay bills the lower number rather than over-billing the tenant. **How do you handle rate limiting if Redis goes down?** Redis Cluster with at least 3 nodes and automatic failover handles most availability scenarios. If the entire cluster becomes unreachable, Relay fails open: it allows requests through rather than rejecting all traffic. Losing rate limit enforcement for a short period during a Redis failure is an acceptable trade-off. Taking down the entire API is not. **What is the noisy neighbor problem and how does it show up in a pool model SaaS?** One tenant runs a large batch job and consumes a disproportionate share of shared infrastructure: database connections, query time, provider key capacity. This increases latency for every other tenant on the same cluster without them doing anything wrong. Relay tracks per-tenant request volume in Redis with a 1-minute rolling window. Tenants that exceed a share threshold get their rate limits tightened automatically. If the pattern persists, they are a candidate for migration to a silo. **How do you migrate a tenant from the pool model to a silo without downtime?** Postgres logical replication. Set up a replication slot on the shared database filtered to the tenant's rows. Let the silo database catch up to near-real-time lag. Then pause writes for that tenant briefly (a few seconds), apply the final delta, update the routing config to point new requests at the silo, and release the write pause. The tenant sees a brief stall, not a maintenance window. **Why keep billing records in separate tables instead of querying the `requests` table at invoice time?** The `requests` table gets very large and is archived after 30-90 days to keep query performance acceptable. Billing records need to be retained for years for tax, legal, and dispute reasons. Separating `billing_line_items` and `usage_daily` from raw request logs means you can archive or drop old request partitions without losing billing history. The rollup also makes monthly invoice generation a fast aggregation query rather than a multi-billion-row scan. **What does BYOK mean and why would an enterprise tenant want it?** Bring Your Own Keys. The tenant provides their own OpenAI or Anthropic API keys instead of Relay's pooled ones. They pay the provider directly at published rates, and Relay charges a flat platform fee instead of a per-token markup. Enterprises want BYOK for three reasons: cost control at high volume, data privacy (their traffic flows under their own API key, not Relay's), and compliance requirements that restrict which credentials can touch their data. **Why use prefixed API keys like `rly_live_sk_`?** Three reasons. First, the prefix makes keys identifiable in logs, error messages, and accidental source code commits. Second, secret scanning tools (GitHub, trufflehog, Doppler) can be configured to detect your specific prefix and alert on leaks before damage is done. Third, the environment segment (`live` vs `test`) lets Relay reject live keys on test endpoints and test keys on live endpoints at the routing layer, before any business logic runs. **How does semantic caching differ from exact match caching, and what can go wrong?** Exact match caching hashes the full request and returns a cached response only if the exact same request was made before. Hit rates are low. Semantic caching embeds the user message into a vector and finds cached responses from requests that are semantically similar. Hit rates are higher, but there is a risk: two questions that are phrased similarly but have meaningfully different answers could get the same cached response. The similarity threshold controls this trade-off. Set it too high and you get wrong answers. Set it too low and cache hits are rare. Semantic caching should never be enabled for requests where correctness is critical (medical, legal, financial). **What happens to a tenant's data when their account is deleted?** Relay uses a soft delete: the tenant row gets `deleted_at` set and `status` set to `DELETED`. All API access is immediately rejected. The underlying data stays in the database for 30 days. During that window, a tenant can contact support to recover their account. After 30 days, a scheduled deletion job hard-deletes all rows across every table where `tenant_id` matches, in dependency order (child tables first, then the tenant row). This satisfies GDPR Article 17 (right to erasure) and gives you an audit trail during the retention window. --- ## Things Worth Their Own Post - **Billing pipeline in depth**: Stripe subscription webhooks, idempotent event processing, dunning logic, credit notes, usage-based invoice generation. - **Semantic caching**: embedding model selection, similarity thresholds, cache invalidation strategy, cost-benefit analysis. - **BYOK (Bring Your Own Keys)**: encrypting and storing tenant provider credentials, key rotation, audit trail for credential usage. - **Webhooks**: delivering request completion events to tenants, retry logic, failure handling, signing payloads. - **Provider health monitoring**: circuit breaker implementation, alerting on degradation, automatic failover testing. - **Request archival**: streaming from Postgres to Parquet on S3, querying historical data with DuckDB or Athena. - **Multiregion**: running Relay in US, EU, AP with tenant routing to nearest region, cross-region cost aggregation. --- ## Nobody Will Read This **Date:** 2026-06-22 **URL:** https://lorbic.com/nobody-will-read-this/ **Description:** Writing documentation, comments, or even blog posts into the void, and doing it anyway. There is a funny truth in software engineering: almost nobody reads the docs. We spend hours writing careful comments in our code. We build personal websites. We write plain text files and push them to live servers. But if we check the stats, we know the truth. The audience is basically zero. So why do we do it? My day is full of loud noise. It is filled with heavy backend logic, endless tasks, and the crushing weight of non-stop engineering. The pressure is always on. The systems never sleep. It is exhausting. But writing is different. When I open a blank text file, the world gets quiet. There are no urgent alerts. Nobody is shouting. It is just me, a blinking cursor, and plain text. I do not write these posts for an audience. I write them for me. Writing takes the messy, stressful noise in my head and turns it into clean, simple lines. Like this one. It is a small thing I can actually control when the rest of the work feels too big and moves too fast. I have decided to take a break from reading or writing anything tech deepdive. Nobody will read this. And honestly, that is exactly why it makes me happy. --- ## What is DRY? (And Other Things We Say at 3 AM) **Date:** 2026-06-22 **URL:** https://lorbic.com/what-is-dry/ **Description:** DRY is a rule we all know. But at 3 AM with a crashing server, the copy-paste starts looking pretty reasonable. There is a famous programmer joke that always gets me: "What is DRY? Well, at the risk of repeating myself..." It is funny, but it also hurts a little because it is so true. We all know the rule: Don't Repeat Yourself (DRY). We learn it from day one. You write a piece of code once, put it in a clean function, and never type it again. But let's be honest. It is the middle of the night. You are staring at your screen, your eyes hurt from the light, and you just need the server to stop crashing. What really happens? You copy that block of code from another file. You paste it right where you need it. You change one variable, save the file, and quietly hope nobody looks too closely. We talk about DRY online, but then we go to work and type the exact same error check fifty times a day just to keep the backend running. We try to be perfect and make everything reusable, but sometimes that just makes the code harder to read. Sometimes, the most practical thing you can do is just let the code repeat. Building a perfectly clean system is a nice idea. But servers have real limits, and honestly, our brains do too when we are just trying to finish the job so we can finally sleep. Sometimes you just have to accept the copy-paste. Anyway, I will end this here... at the risk of repeating myself. --- ## So, We Are Writing Efficient Software Again **Date:** 2026-06-17 **URL:** https://lorbic.com/so-we-are-writing-efficient-software-again/ **Description:** RAM is up 400%. CPUs cost 15% more. The free hardware lunch is finally over. Welcome back to the 80s, where every byte counts. I wanted to upgrade the RAM in my PC. Nothing fancy. I bought 32GB in 2024 and I wanted to double it. I opened a tab, checked the price, and closed the tab. Then I sat quietly for a moment. The same 32GB kit I bought in 2024 now costs more than double. DDR5 prices have gone up roughly 400% since mid-2025 [1]. DDR4 is not much better. A kit that cost $60–$90 in late 2025 now sells for $150–$180 [2]. I did not upgrade my RAM. I opened my code editor instead. --- ## What happened Two things happened at the same time. **AI ate the memory market.** You know the old meme: "why is my PC slow?" "the cat ate my RAM." Funny because cats do not eat RAM. Except now the cat is a data center, and the RAM is real, and it is not funny anymore. Data centers need enormous amounts of memory to run inference workloads. Manufacturers are happy to sell high-margin server chips to hyperscalers. The consumer market gets whatever is left. Memory shortages are expected to last until at least Q4 2027 [3]. **Tariffs made everything worse.** On top of that, there are now 25% tariffs on many semiconductor imports [4]. At some point ASUS warned that laptops could cost 30% more [5]. I read that headline, closed it, and stared at my ceiling for a while. The best joke in this whole situation is that the people responsible for the price increase are the same companies whose tools we use to write more code, faster, with less thought. We used AI to produce bloated software at scale, and now AI is pricing us out of the hardware to run it. --- ## The villains Let us talk about what we built during the cheap hardware era. Electron. The framework that takes a web app and wraps it in its own copy of Chrome. Discord, Slack, VS Code: all Electron apps. Each one ships with a browser runtime that uses hundreds of megabytes before you open a single document. Slack on startup can use 400MB or more. You are running a browser to run a chat app inside a browser. `node_modules`. The folder that contains the internet. A fresh React project can have 300MB of dependencies before you write one line of application code. The `is-odd` package has 70 million weekly downloads. It is 6 lines long. It checks if a number is odd. Spring Boot. A Java framework where the "Hello World" service takes 45 seconds to start and uses 512MB of heap. There are enterprise teams who have been waiting for their local dev server to start since 2019. Nobody blamed the developers, because there was not much to blame them for. Hardware was cheap, abstraction genuinely helps, and shipping faster is a real goal. The incentives just did not reward caring about memory. --- ## Back to the future Here is the funny part. We are now back to thinking like programmers from the 1980s. In 1984, you had 64KB of RAM. You counted every byte. You decided which data to keep in memory and which to load from disk because you had no real choice. Carmack's DOOM ran on a 386 with 4MB of RAM. The Apollo Guidance Computer had 4KB of RAM and landed humans on the moon. The demoscene still fits entire animated 3D worlds into 4KB executables, just because they enjoy the constraint. Then RAM got cheap, CPUs got fast, and cloud servers became easy to rent by the hour. Writing inefficient software stopped having real consequences, so people mostly stopped thinking about it. --- ## A memo from 1984 **FROM:** A programmer who wrote a ray caster in 640KB **TO:** Modern software engineers **RE:** Your allocator We have reviewed your code. A few notes. Your struct has 7 bytes of padding between the boolean and the integer. This happens on every element of a slice with ten million entries. You are wasting 70MB of RAM for no reason. We could have fit a spreadsheet application in that space. Your dependency tree has a package that checks if a string is empty. It has its own dependency. That dependency has a dependency. We checked: none of them check if the string is empty correctly. Your Docker image is 2.1GB. Our entire operating system was 160KB. We are not angry. We are just disappointed. Regards, 1984 --- So, we are writing efficient software again. --- ## What this means for code If RAM is expensive, your code's memory usage is a cost. Not a performance metric. An actual money cost. A machine that needs 64GB instead of 16GB costs significantly more to buy or rent. A service that allocates 200MB instead of 2GB runs on a smaller, cheaper instance. When you multiply that across a fleet, the difference is real budget. A few things worth thinking about: **Struct layout.** I wrote about this in [A Love Letter to the L1 Cache](../love-letter-to-the-l1-cache/). Compiler padding from bad field ordering can make your structs 50% larger than they need to be. On a tight memory budget, that inflates your working set directly. Run [`betteralign`](https://github.com/dkorunic/betteralign) on your Go codebase and see what it finds. **Allocations.** Every allocation that is not necessary is now more expensive than it was. In Go, this means thinking about whether you should preallocate a slice, whether that intermediate `[]byte` is avoidable, and whether a sync.Pool makes sense for objects you create at high frequency. **Dependencies.** Every library has a memory footprint. Pulling in a 10,000-line package for one function is harder to justify when RAM costs three times what it did in 2024. --- ## A small confession I am not old enough to have written code under real memory constraints. I did not count bytes in 1984. I did not have to. By the time I started programming, RAM was cheap and getting cheaper, and nobody expected you to care. But I have always liked writing code that does not waste things. Not because I had to, just because it feels better. A [loop that fits in cache](../love-letter-to-the-l1-cache/), a struct that has no padding it does not need, a service that uses 80MB instead of 800MB: these feel like better solutions, the same way a tight paragraph feels better than a loose one. After I decided not to upgrade my RAM, I ran `betteralign` on my current project. It found 14 structs with suboptimal field ordering. I felt a little embarrassed, fixed them, and watched the working set of the main loop drop by about 18%. I did not buy any new hardware. I just stopped wasting the hardware I already have. The programmer in the memo above would say "took you long enough". They would be right. But at least I got there. --- ## References [1] [G.SKILL Addresses Sharp DDR5 RAM Price Increases](https://www.neowin.net/news/gskill-addresses-sharp-ddr5-ram-price-increases-for-2025-2026/) — Neowin [2] [RAM Price Index 2026](https://www.tomshardware.com/pc-components/ram/ram-price-index-2026-lowest-price-on-ddr5-and-ddr4-memory-of-all-capacities) — Tom's Hardware [3] [Memory Shortages To Last Till Q4 2027](https://wccftech.com/memory-ddr5-ddr4-shortages-last-till-q4-2027-higher-prices-throughout-2026/) — WCCFtech [4] [Trump Tariffs to Hike PC Costs at Least 20%](https://www.techpowerup.com/335054/trump-tariffs-to-hike-pc-costs-at-least-20-system-integrators-take-the-biggest-blow) — TechPowerUp [5] [Server CPU Prices Up as Much as 20% Since March](https://www.trendforce.com/news/2026/04/22/news-server-cpu-prices-up-as-much-as-20-since-march-intel-may-raise-prices-another-8-10-in-2h26/) — TrendForce --- ## Constructing Concurrent Inverted Indexes in Go **Date:** 2026-06-16 **URL:** https://lorbic.com/constructing-concurrent-inverted-indexes-in-go/ **Description:** Building a thread-safe inverted index from scratch in Go. Covers sharded mutexes, lock contention profiling, slice pooling to avoid GC pressure, and benchmark comparisons against a naive sync.RWMutex approach under varying read/write ratios. I spent a Saturday afternoon benchmarking a concurrent inverted index and discovered that a single `sync.RWMutex` starts to break down at roughly 4 concurrent readers. The degradation is not linear. It is not graceful. It is a cliff. The inverted index is one of the oldest data structures in information retrieval. It maps terms to the documents that contain them, forming the backbone of every search engine from Elasticsearch to Lucene to Google's earliest prototypes. The data structure itself is simple. Making it fast under concurrent load is not. This article walks through building a concurrent inverted index in Go. I start with a naive implementation, profile its lock contention, and progressively refine it with sharded mutexes, compressed posting lists, and allocation-aware index structures. If you have ever wondered what happens under the hood when you type a query into a search box, or why some databases can handle 50,000 index writes per second while others stall at 5,000, this is for you. --- ## The inverted index: a five-minute primer An inverted index takes a collection of documents and builds a mapping from each term to a list of document identifiers where that term appears. Given a query like "concurrent Go", the engine looks up "concurrent" and "Go" in the index, retrieves their posting lists, and intersects them to find documents containing both terms. {{< code lang="go" >}} // The core data structure, stripped to its essence type InvertedIndex struct { mu sync.RWMutex index map[string][]int // term → sorted list of document IDs } func (idx *InvertedIndex) Index(docID int, terms []string) { idx.mu.Lock() defer idx.mu.Unlock() for _, term := range terms { idx.index[term] = append(idx.index[term], docID) } } func (idx *InvertedIndex) Search(term string) []int { idx.mu.RLock() defer idx.mu.RUnlock() return idx.index[term] } {{< /code >}} This works. It is correct. It is also catastrophically slow under any real workload. The problem is the single `sync.RWMutex`. Every reader acquires a read lock. Every writer acquires a write lock. When a writer arrives, it blocks all new readers. When even a modest number of readers are active, writers starve. The mutex becomes a serialization point, and your concurrent index degrades into a sequential one. --- ## Measuring the damage: lock contention profiling Before optimizing, we need to quantify the problem. I wrote a benchmark harness that simulates different read/write ratios against the naive index. The workload: 1 million terms across 100,000 documents, with goroutine counts varying from 1 to 64. {{< code lang="go" >}} func BenchmarkNaiveIndex(b *testing.B) { scenarios := []struct { name string readRatio float64 // fraction of operations that are reads goroutines int }{ {"read-heavy-4", 0.95, 4}, {"read-heavy-16", 0.95, 16}, {"balanced-4", 0.50, 4}, {"balanced-16", 0.50, 16}, {"write-heavy-4", 0.05, 4}, {"write-heavy-16", 0.05, 16}, } for _, sc := range scenarios { b.Run(sc.name, func(b *testing.B) { idx := NewNaiveIndex() // pre-populate with 100k documents, 10 terms each for i := 0; i < 100_000; i++ { idx.Index(i, generateTerms(10)) } b.ResetTimer() b.SetParallelism(sc.goroutines) b.RunParallel(func(pb *testing.PB) { rng := rand.New(rand.NewSource(time.Now().UnixNano())) for pb.Next() { if rng.Float64() < sc.readRatio { idx.Search(randomTerm(rng)) } else { idx.Index(randomDocID(rng), generateTerms(5)) } } }) }) } } {{< /code >}} The results, measured in operations per second on an M2 Pro with 12 cores: | Scenario | Naive RWMutex (ops/sec) | Contention % | | :--- | :--- | :--- | | read-heavy-4 | 2,847,000 | 12% | | read-heavy-16 | 1,210,000 | 58% | | balanced-4 | 892,000 | 43% | | balanced-16 | 412,000 | 76% | | write-heavy-4 | 310,000 | 65% | | write-heavy-16 | 94,000 | 91% | At 16 goroutines with a balanced workload, 76% of CPU time is spent waiting on the mutex. The goroutines are not doing useful work. They are standing in line. {{< details title="Reading Go Mutex Profiles" label="Diagnostics //" >}} The Go runtime exposes mutex contention profiling through `runtime.SetMutexProfileFraction`. Enable it with a non-zero rate, then visualize with `go tool pprof`: ```bash go test -bench=BenchmarkNaiveIndex -mutexprofile=mutex.out go tool pprof -http=:8080 mutex.out ``` The flame graph will show `sync.(*RWMutex).Lock` dominating your CPU samples. This is the "contention %" column above. If this number exceeds 20%, your mutex is your bottleneck. {{< /details >}} --- ## Sharded mutexes: divide and conquer The core insight is that contention arises because all operations—reads and writes—funnel through a single lock, even when they target different terms. A query for "database" should not block an index update for "garbage-collector". A sharded mutex distributes the lock across N independent partitions. Each partition guards a subset of the term space. The term is hashed to a shard, and only that shard's mutex is acquired. {{< code lang="go" >}} const numShards = 256 type ShardedIndex struct { shards [numShards]indexShard } type indexShard struct { mu sync.RWMutex terms map[string]*PostingList } func (idx *ShardedIndex) shard(term string) *indexShard { h := fnv.New32a() h.Write([]byte(term)) return &idx.shards[h.Sum32()%numShards] } func (idx *ShardedIndex) Index(docID int, terms []string) { // Group terms by shard to minimize lock acquisitions grouped := make(map[uint32][]string) for _, term := range terms { h := fnv.New32a() h.Write([]byte(term)) shardID := h.Sum32() % numShards grouped[shardID] = append(grouped[shardID], term) } for shardID, shardTerms := range grouped { shard := &idx.shards[shardID] shard.mu.Lock() for _, term := range shardTerms { shard.terms[term].Add(docID) } shard.mu.Unlock() } } {{< /code >}} Two details matter here. First, terms are grouped by shard before acquiring locks. Without this grouping, the function would acquire and release a new lock for every term in the document, doubling the lock/unlock overhead. Second, the hash function is FNV-32a, not a cryptographic hash. We need speed, not collision resistance. A handful of collisions across 256 shards is statistically irrelevant. The results: | Scenario | Naive RWMutex (ops/sec) | Sharded 256-way (ops/sec) | Improvement | | :--- | :--- | :--- | :--- | | read-heavy-4 | 2,847,000 | 4,120,000 | 1.45× | | read-heavy-16 | 1,210,000 | 3,890,000 | 3.21× | | balanced-4 | 892,000 | 1,740,000 | 1.95× | | balanced-16 | 412,000 | 1,610,000 | 3.91× | | write-heavy-4 | 310,000 | 520,000 | 1.68× | | write-heavy-16 | 94,000 | 420,000 | 4.47× | The improvement is most dramatic at high goroutine counts because sharding reduces the probability of any two operations landing on the same mutex. With 256 shards, the expected collision rate for 16 concurrent operations is roughly 6%, compared to 100% with a single mutex. --- ## How many shards? The Goldilocks problem The number of shards is a tuning parameter with real trade-offs. Too few, and you retain contention. Too many, and you waste memory on mutex overhead and increase the cost of the hash-and-dispatch step. I ran the balanced-16 benchmark against shard counts from 1 to 2048: | Shards | Throughput (ops/sec) | Index Memory (MB) | Useful Work % | | :--- | :--- | :--- | :--- | | 1 | 412,000 | 48 | 24% | | 8 | 1,140,000 | 49 | 52% | | 32 | 1,480,000 | 50 | 68% | | 128 | 1,620,000 | 52 | 82% | | 256 | 1,610,000 | 54 | 86% | | 512 | 1,590,000 | 58 | 87% | | 1024 | 1,550,000 | 66 | 88% | | 2048 | 1,490,000 | 82 | 89% | Beyond 256 shards, throughput plateaus and then declines. The mutex overhead per shard (each `sync.RWMutex` is 24 bytes plus the map header) starts consuming measurable memory, and the hash-compute cost per operation overtakes the contention savings. 256 is the empirical sweet spot for this workload. --- ## The garbage collector is watching Lock contention is not the only threat to throughput. The Go garbage collector is a concurrent, tri-color mark-and-sweep collector. It runs alongside your goroutines. Every heap allocation you make during indexing adds to its workload. In our naive implementation, every call to `append` on a posting list potentially triggers a slice growth, which allocates a new backing array and copies the old one. Over millions of documents, this is a constant, grinding allocation pressure. The fix is a posting list backed by a pre-allocated, growable buffer with amortized constant-time append. But even better: we can pool the individual posting entries. {{< code lang="go" >}} var postingPool = sync.Pool{ New: func() interface{} { return &Posting{} }, } type Posting struct { DocID int TermFreq int Positions []int } type PostingList struct { mu sync.RWMutex postings []*Posting } func (pl *PostingList) Add(docID int, positions []int) { pl.mu.Lock() defer pl.mu.Unlock() p := postingPool.Get().(*Posting) p.DocID = docID p.TermFreq = len(positions) p.Positions = append(p.Positions[:0], positions...) pl.postings = append(pl.postings, p) } {{< /code >}} {{< details title="sync.Pool Mechanics" label="Diagnostics //" >}} `sync.Pool` is a per-P (logical processor) cache of objects. When you call `Get()`, the runtime checks the local P's pool first, falling back to other P's pools and finally to `New()`. When you call `Put()`, the object is returned to the local pool. Critical caveat: objects in a `sync.Pool` can be garbage collected at any time. The pool is a cache, not a guarantee. Do not store state in pooled objects that cannot be reconstructed. For the posting pool, this is fine. Every `Get()` call is followed by a full re-initialization of all fields. {{< /details >}} With `sync.Pool` in place, the allocation profile shifted: | Metric | Without Pooling | With Pooling | | :--- | :--- | :--- | | Allocations per index op | 47 | 3 | | Bytes allocated per op | 2,840 | 312 | | GC pause p99 (μs) | 1,240 | 180 | | GC CPU overhead | 18% | 4% | The GC CPU overhead dropped from 18% to 4%. That is 14% of your CPU budget returned to actual indexing work. Pooling is not a micro-optimization. It is a structural decision that changes how your program interacts with the runtime. --- ## Compressed posting lists: Elias-Fano encoding A posting list for a common term like "the" might contain millions of document IDs. Storing these as a raw `[]int` consumes 8 bytes per ID. On a corpus of 10 million documents, "the" alone would occupy 80 MB. Compression matters. The standard approach in information retrieval is delta encoding followed by variable-length integer coding. Delta encoding stores the difference between consecutive document IDs (which are sorted). Since document IDs are monotonically increasing, the deltas are small positive integers. Elias-Fano encoding is a quasi-succinct representation that stores a monotone sequence of N integers from a universe of size U in roughly N * (2 + log₂(U/N)) bits. For a posting list with an average gap of 64 between document IDs, Elias-Fano uses approximately 5-6 bits per integer, compared to 64 bits for raw `int`. {{< code lang="go" >}} type EliasFanoPostingList struct { lowerBits uint8 lowerBitsLen int lowBits []uint64 highBits *roaring.Bitmap // from github.com/RoaringBitmap/roaring lastDocID int length int } func (pl *EliasFanoPostingList) Add(docID int) { delta := docID - pl.lastDocID // Lower bits: store the lower 'lowerBitsLen' bits of delta lowMask := (1 << pl.lowerBitsLen) - 1 low := delta & lowMask pl.appendLowBits(low) // High bits: store (delta >> lowerBitsLen) as unary in a bitmap high := delta >> pl.lowerBitsLen pos := pl.highBits.GetCardinality() pl.highBits.Add(pos + high) pl.lastDocID = docID pl.length++ } {{< /code >}} The decoding path reconstructs the original document ID by reading the high-bit bitmap to find the position, extracting the lower bits at the corresponding index, and adding the previous document ID. The space savings are substantial: | Corpus Size | Raw int[] (MB) | Elias-Fano (MB) | Compression Ratio | | :--- | :--- | :--- | :--- | | 1M docs, avg 200 terms/doc | 1,600 | 380 | 4.2× | | 10M docs, avg 200 terms/doc | 16,000 | 3,600 | 4.4× | The trade-off is decode latency. Random access into an Elias-Fano list requires a rank operation on the high-bit bitmap, which is O(log N) with a rank-indexed roaring bitmap. For intersection-heavy query workloads, this cost is amortized across thousands of list accesses. For single-term lookups, a variable-byte encoded list (with O(1) sequential decode) may be faster. --- ## Putting it together: the production index The final index combines sharded mutexes, pooled postings, and compressed posting lists into a single structure: {{< code lang="go" >}} type Index struct { shards [256]indexShard docStore *DocumentStore termDict *TermDictionary stats *IndexStats closeCh chan struct{} } type indexShard struct { mu sync.RWMutex terms map[string]*EliasFanoPostingList } func NewIndex() *Index { idx := &Index{ docStore: NewDocumentStore(), termDict: NewTermDictionary(), stats: NewIndexStats(), closeCh: make(chan struct{}), } for i := range idx.shards { idx.shards[i].terms = make(map[string]*EliasFanoPostingList) } return idx } func (idx *Index) Index(doc *Document) error { idx.stats.IncrDocuments() // Tokenize with pipeline: lowercase → tokenize → filter stopwords → stem tokens := idx.tokenize(doc.Content) // Store document idx.docStore.Put(doc.ID, doc) // Group tokens by shard grouped := idx.groupByShard(tokens) // Index each shard's batch under that shard's lock for shardID, shardTokens := range grouped { shard := &idx.shards[shardID] shard.mu.Lock() for _, token := range shardTokens { pl, ok := shard.terms[token] if !ok { pl = NewEliasFanoPostingList() shard.terms[token] = pl idx.termDict.Add(token) } pl.Add(doc.ID) } shard.mu.Unlock() } return nil } {{< /code >}} This is the shape of a production inverted index. It is correct under concurrent reads and writes. It compresses its posting lists. It pools its allocations. It can be extended with a background merge goroutine to periodically compact posting lists and a WAL (write-ahead log) for crash recovery. --- ## What we did not cover This article focuses on the index structure and concurrency model. Several adjacent concerns are intentionally deferred: - **Tokenization and analysis**: Lowercasing, stemming (Porter2), stop-word removal, and n-gram generation are critical for recall but are pipeline stages that plug into the index, not properties of the index itself. - **Query execution**: Boolean AND/OR/NOT, phrase queries with position information, and relevance scoring (TF-IDF, BM25) operate on top of the index. The index provides raw posting lists. The query executor does the rest. - **Persistence**: An in-memory index is fast but ephemeral. A production system needs a write-ahead log and periodic snapshots to disk. - **Distributed indexing**: Sharding the index across multiple machines introduces consensus, replication, and partition-tolerant query routing. Each of these will be the subject of a dedicated article in this series. --- ## The benchmark table | Approach | Balanced-16 (ops/sec) | Memory (MB) | GC Overhead | Compression | | :--- | :--- | :--- | :--- | :--- | | Naive RWMutex | 412,000 | 48 | 18% | None | | Sharded (256-way) | 1,610,000 | 54 | 14% | None | | Sharded + Pooling | 1,720,000 | 52 | 4% | None | | Sharded + Pooling + Elias-Fano | 1,680,000 | 18 | 4% | 4.2× | The fully optimized index achieves a 4× throughput improvement over the naive implementation while using 63% less memory and 78% less GC CPU time. The slight throughput regression from Elias-Fano compression (1,720k → 1,680k) is the decode penalty. In practice, this is offset by the memory savings, which allow larger working sets to fit in the CPU cache. --- ## Further reading - Manning, Raghavan, and Schütze. *Introduction to Information Retrieval*. Chapter 1 (Boolean Retrieval) and Chapter 5 (Index Compression). - Sebastiano Vigna. "Quasi-succinct indices". *Proceedings of WSDM 2013*. The canonical paper on Elias-Fano for inverted indexes. - Go Blog. "Go's http/2 server: Priority and flow control for a concurrent world". Demonstrates sharded synchronization patterns in the standard library. - Daniel Lemire's blog: [Roaring Bitmaps](https://roaringbitmap.org/). The compressed bitmap library used in the high-bit component of Elias-Fano. --- *The next article in this series is [Implementing BM25 Scoring From First Principles](/implementing-bm25-scoring-from-first-principles/). It builds on the index here, deriving BM25's term frequency saturation and length normalization from scratch, with benchmarks against TF-IDF.* --- ## A Love Letter to the L1 Cache **Date:** 2026-06-15 **URL:** https://lorbic.com/love-letter-to-the-l1-cache/ **Description:** Why reordering two struct fields made my loop 30% faster, and what it taught me about how CPUs actually work. I recently spent four hours staring at a benchmark that didn't make sense. It started while working on [Derelict Facility](https://github.com/vikash-paf/derelict-facility), my grid-based game engine in Go. I was trying to tighten the main update loop, specifically the part that iterates over every entity on the map each frame. I pulled out a small benchmark to isolate the cost, and something looked wrong. I had two Go structs. They held the exact same data: two booleans and a 64-bit integer. I was iterating over a slice of 10 million of these structs, doing a simple addition. They should have been identical in performance. They weren't. Struct A was consistently 30% slower than Struct B. Struct A was also 50% larger in memory. In high-level languages, we are taught to think of memory as a simple, uniform resource. You ask for some bytes, you get them. But when you start caring about nanoseconds, you find out that memory is not uniform at all. Getting a value from RAM takes real time, and your CPU has no choice but to sit and wait for it. This post is about that wait. --- ## The code Here is what started it. {{< code lang="go" >}} type BadStruct struct { A bool // 1 byte B int64 // 8 bytes C bool // 1 byte } type GoodStruct struct { B int64 // 8 bytes A bool // 1 byte C bool // 1 byte } {{< /code >}} If you use `unsafe.Sizeof()`, you will find that `BadStruct` occupies **24 bytes**, while `GoodStruct` only takes **16 bytes**. Why? Because the CPU reads memory in fixed-size chunks called "words". On a 64-bit machine, a word is 8 bytes. To make reading fast, the compiler adds "padding" (empty bytes) to make sure that 8-byte values like `int64` start at an address that is a multiple of 8. {{< details title="Padding and Alignment" label="Glossary //" >}} **Alignment:** CPUs fetch data in fixed-size chunks. If an 8-byte integer starts at byte 3 instead of byte 8, the CPU may need two fetches to read it instead of one. **Padding:** To avoid misalignment, the compiler inserts empty bytes between fields. In `BadStruct`, it adds 7 bytes after the first `bool` so the `int64` can start at byte 8. It then adds 7 more bytes at the end to keep the struct aligned for the next element in a slice. {{< /details >}} But the size difference is only half the story. The real reason `BadStruct` is slower is that it wastes cache space. --- ## Why RAM is slow To understand why 8 bytes of padding matter, you need to know how the memory hierarchy works. RAM is not fast. Relative to a CPU running at 4GHz, it is very slow. Here is a way to think about it: if one CPU cycle took one second, a round trip to main memory would take about four minutes. To deal with this, CPUs have a hierarchy of caches. Each layer is smaller, faster, and closer to the processor. {{< details title="SRAM vs DRAM" label="Glossary //" >}} **SRAM (Static RAM):** Used for CPU caches (L1, L2, L3). It is fast because each bit stays set without any refreshing. It uses more transistors per bit, so it is expensive and takes more space on the chip. **DRAM (Dynamic RAM):** Used for main memory (your 16GB or 32GB sticks). It is cheaper and denser, but each bit has to be constantly refreshed. This is what makes it slower. {{< /details >}} ### The latency ladder - **L1 Cache:** ~1 nanosecond. Sits directly on the CPU die. About 32–64KB per core. - **L2 Cache:** ~4 nanoseconds. Larger, slightly further away. - **L3 Cache:** ~10–15 nanoseconds. Shared across cores. - **Main Memory (DRAM):** ~100 nanoseconds. A completely separate chip. This is the only pyramid scheme where the returns are real. When your CPU needs a value, it checks L1 first. If it is not there (a "cache miss"), it checks L2, then L3, then finally goes to main memory. Each miss costs time your CPU spends doing nothing. It just waits. This is why the L1 cache matters so much. It is the only memory fast enough to keep the CPU busy. --- ## The secret life of a cache line Here is the important part: **the CPU never fetches a single byte**. When you ask for 1 byte, the CPU assumes you will probably need the bytes next to it soon. This is called **spatial locality**. So instead of fetching 1 byte, it fetches a **64-byte block** called a **cache line**. {{< details title="Cache Line" label="Glossary //" >}} **Cache Line:** The smallest unit of data transfer between the cache and main memory. Even if you only need 1 byte, the hardware fetches 64 bytes and stores the whole block in the cache. {{< /details >}} Here is what that looks like in practice with our two structs: ``` One 64-byte cache line: BadStruct (24 bytes) — 2 fit, 16 bytes wasted: ┌────────────────────────┬────────────────────────┬─ ─ ─ │ entity 0 │ entity 1 │ └────────────────────────┴────────────────────────┴─ ─ ─ 0 23 24 47 GoodStruct (16 bytes) — 4 fit, 0 bytes wasted: ┌────────────────┬────────────────┬────────────────┬────────────────┐ │ entity 0 │ entity 1 │ entity 2 │ entity 3 │ └────────────────┴────────────────┴────────────────┴────────────────┘ 0 15 16 31 32 47 48 63 ``` If your structs are small and tightly packed, one 64-byte cache line holds four of them. The CPU hits main memory once and has data for the next four loop iterations. If your structs are bloated with padding, that same cache line holds only two. You hit main memory twice as often for the same amount of work. --- ## Why struct field order matters Let's look at how our two structs actually sit in memory. ### BadStruct (24 bytes) | Byte Offset | Field | Content | | :--- | :--- | :--- | | 0 | `A` | `bool` (1 byte) | | 1–7 | – | **Padding** (7 bytes) | | 8–15 | `B` | `int64` (8 bytes) | | 16 | `C` | `bool` (1 byte) | | 17–23 | – | **Padding** (7 bytes) | ### GoodStruct (16 bytes) | Byte Offset | Field | Content | | :--- | :--- | :--- | | 0–7 | `B` | `int64` (8 bytes) | | 8 | `A` | `bool` (1 byte) | | 9 | `C` | `bool` (1 byte) | | 10–15 | – | **Padding** (6 bytes) | By moving the `int64` to the top, the two booleans sit together at the end. We cut the struct size by 33%. In a tight loop over millions of elements, this means fitting more data into each cache line, which means fewer trips to main memory. --- ## The benchmark I wrote a Go benchmark to confirm this. Two slices of 10,000,000 structs. Iterate over each and sum the integer field. {{< code lang="go" >}} func BenchmarkBadStruct(b *testing.B) { data := make([]BadStruct, 10_000_000) b.ResetTimer() for i := 0; i < b.N; i++ { var sum int64 for j := 0; j < len(data); j++ { sum += data[j].B } } } func BenchmarkGoodStruct(b *testing.B) { data := make([]GoodStruct, 10_000_000) b.ResetTimer() for i := 0; i < b.N; i++ { var sum int64 for j := 0; j < len(data); j++ { sum += data[j].B } } } {{< /code >}} ### Result (`benchstat`) ```text name time/op BadStruct-16 12.4ms ± 2% GoodStruct-16 8.7ms ± 1% ``` 30% faster. Just from reordering fields. The reason: `GoodStruct` fits roughly 4,000,000 elements into 64MB. `BadStruct` fits only 2,666,666. The bad version keeps evicting cache lines and waiting for main memory. The CPU has no work to do. It just waits. If you want to catch this automatically, [`betteralign`](https://github.com/dkorunic/betteralign) is a Go linter that finds structs with suboptimal field ordering and suggests the fix. One pass over your codebase and it will find every case like this one. --- ## Hardware sympathy Languages like Go, Java, and Python hide most of the hardware from you. That is a good thing. You can focus on the actual problem. But the hiding breaks down at scale. The CPU does not care about your abstractions. It only sees cache lines, bus cycles, and latency. When your working set stops fitting in cache, you pay the full 100ns penalty for every miss, millions of times per second. "Hardware sympathy" is a term from the LMAX Disruptor project. The idea is simple: understand the physical machine well enough to write code that works with it, not against it. It does not mean writing assembly. It means knowing that a cache line is 64 bytes, and thinking about whether your data layout respects that. Back in Derelict Facility, I went through the entity structs with fresh eyes. Most of them had the same problem: small boolean flags scattered between larger fields. After reordering, the update loop got noticeably faster without touching a single line of game logic. Four hours to understand the problem. Five minutes to fix it. Run `betteralign` on your own codebase and see what it finds. --- --- ## Why Explaining Technical Difficulty is Hard **Date:** 2026-05-14 **URL:** https://lorbic.com/why-explaining-technical-difficulty-is-hard/ **Description:** "It's just a simple query". Why the distance between a logical requirement and its infrastructure cost is the most expensive gap in engineering. "Can we just add a real-time visitor counter to the homepage? It's just a query, right"? Every engineer has heard some variation of this. On the surface, the logic is sound. You have data, you have a UI, and you want to connect them. In the world of business requirements, this is a solved problem. You write a line of code, and the feature exists. But your database doesn't care about business logic. It cares about **Infrastructure Constraints**. While the request sounds like a simple query, the gap between the requirement and the execution is where the hidden costs live. You aren't just "showing a number"; you are introducing global lock contention on a high-traffic table, triggering `COUNT(*)` scans on 100M rows, and potentially pinning your database CPU at 100% during peak hours. > "For every complex problem, there is a solution that is simple, neat, and wrong". - H.L. Mencken ![The Implementation Iceberg](implementation-iceberg.svg) This is the **Implementation Gap**: the distance between the logical intent and the physical cost of execution. ## The logic vs. the stack We often operate in two different modes: ### 1. The feature mode This is where requirements are born. The goal is to move fast, reduce uncertainty, and get feedback. Here, code is just a means to an end. If it "works" on a local dev machine with 100 rows, the requirement is considered met. ### 2. The system mode This is where the stack lives. This mode is governed by the physics of computation: memory latency, network overhead, and concurrency limits. In this mode, every line of code is an allocation, a network round-trip, or a lock that must be managed. {{< details title="The Divergent Perspectives" label="Deep Dive //" >}} **Feature mode** asks: "Does this fulfill the user story"? **System mode** asks: "What does this do to the p99 latency of the entire gateway"? The friction in engineering teams isn't about personality; it's about whether you're looking at the logic or the underlying hardware. {{< /details >}} ## The model After a few years of building and breaking things, you start to develop an understanding of your stack. This isn't about job titles; it's about **Tacit Knowledge**. It's the intuition you develop after watching a "quick fix" turn into a 2 AM outage. You start to see the infrastructure behind the syntax. When you hear "real-time counter", you aren't thinking about the SQL; you're thinking about the lock contention on the index. You aren't being a "blocker"; you are observing the mechanical limits of your environment. {{< alert type="important" >}} **The Simulation Gap:** Modern tools like AI are excellent at **Syntax**. They can generate the logic for a feature in seconds. But they don't know your **Infrastructure**. They can't tell you that your query will fail because of your specific index strategy or your Redis eviction policy. {{< /alert >}} ## Bridging the gap: the API of options The most expensive mistake an engineer can make is trying to explain **Infrastructure Complexity** to someone who is focused on **Product Feedback**. If you explain "why it's hard" in technical terms, you sound like you're making excuses. It reminds me of the scene in *The Big Bang Theory* where Penny is trying to explain social conventions to Sheldon, and after listening to his pedantic reasoning, she realizes: "Now I know what I sound like to you when I say stupid stuff". When we lecture a product manager on database internal locks, we are Penny. We are just saying "stupid stuff" that doesn't map to their reality. Instead, you should treat your expertise as a system interface that provides **Validated Options**. Instead of a "No", offer a path that separates the experiment from the infrastructure. - **The Request:** "We need a real-time visitor counter". - **The Feature View:** "Let's just query the `visits` table directly". - **The System View:** "That will kill the DB. We need a pre-aggregated counter service". - **The Bridge:** "If we want to see if users care about visitor counts, let's hardcode a '5,000+ users today' badge for 48 hours. If it moves the needle, we'll invest the time to build the counter service properly next week". {{< alert type="tip" >}} This approach satisfies the need for speed without polluting your core architecture. It allows you to validate the business value before you pay the infrastructure price. {{< /alert >}} --- Engineering is the management of trade-offs between what the business wants and what the infrastructure can sustain. When a request feels like friction, it is usually just your model flagging a resource contention before it happens. Use that intuition to offer a low-fidelity path that validates the idea without pinning your database. --- ## Just About Go Time **Date:** 2026-05-10 **URL:** https://lorbic.com/just-about-go-time/ **Description:** A breakdown of the absolute absurdity of human time, monotonic clocks in Go, and the one true way to store time across PostgreSQL, Couchbase, and mobile clients. Time is an illusion. Or more accurately, time is a political consensus poorly masquerading as physics. ![Time is relative, absolute, and a scam](time-theory-meme.jpg) If you've ever seen Dylan Beattie's "Plain Text" [talk](https://www.youtube.com/watch?v=gd5uJ7Nlvvo), you know that humans have spent centuries making data storage as complicated as possible. But text encoding has nothing on time zones. As engineers, we like to pretend that `time.Now()` returns an objective truth. It doesn't. It returns a snapshot of a highly contested, historically unstable set of political boundaries. In 2011, the island nation of Samoa decided they wanted to align their workweek with Australia rather than the United States. To do this, they didn't just change their clocks; they completely skipped Friday, December 30th. At 11:59 PM on Thursday, the clock ticked over, and it was suddenly Saturday. A whole day, erased from existence because a prime minister signed a piece of paper. If your backend job ran a cron on December 30th in Samoa, it just... didn't happen. Here is what I've learned about handling time without losing sleep over it. ## Go Time Package Mechanics Go's `time` package is notorious for being incredibly robust, but deeply weird to newcomers. ### The 1 2 3 4 5 6 7 Reference Date If you want to parse or format a date in Python or Java, you use a format string like `YYYY-MM-DD HH:mm:ss`. Go looked at that and said, "No, that's too abstract". Instead, Go uses a reference date. To format a date, you have to write out what the reference date would look like in your desired format. The reference date is: **January 2nd, 3:04:05 PM, 2006, timezone -0700 (MST)**. Why? Because if you write it out in the American format, it counts up sequentially: `01/02 03:04:05 '06 -0700` It is a dry engineering joke that has cursed millions of developers to Google "golang time format string" every single day of their careers. ```go // Parsing an ISO 8601 string t, err := time.Parse("2006-01-02T15:04:05Z07:00", "2026-05-15T14:30:00Z") // Formatting to a custom layout fmt.Println(t.Format("2006-01-02 15:04")) ``` ### Monotonic Clocks in Go Before Go 1.9, measuring execution time was dangerous. ```go start := time.Now() doWork() elapsed := time.Since(start) ``` In older versions of Go, `elapsed` could theoretically be a negative number. Why? Because `time.Now()` used the operating system's "wall clock". The wall clock is tied to NTP (Network Time Protocol). If the OS synched with an NTP server during `doWork()` and realized its clock was 2 seconds fast, it would violently jump the clock backward. Your start time would suddenly be in the future. In 2017, Russ Cox proposed a transparent fix. Rather than creating a separate `MonotonicTime` type, Go simply jammed a monotonic clock reading inside the existing `time.Time` struct alongside the wall clock. If a `time.Time` struct has a monotonic clock reading, operations like `time.Since()` or `t.Sub(u)` automatically use the monotonic (always forward-moving) clock to measure duration, completely ignoring any NTP timezone jumps. It fixed a massive class of distributed system bugs without breaking a single line of backward compatibility. ### Testing Temporal Logic Writing tests for code that depends on the current time is a common source of flaky CI/CD pipelines. If your logic checks if a token is expired using `time.Now()`, your test is coupled to the execution clock of the runner. To solve this, treat time as a dependency. Never call `time.Now()` inside deep business logic. Instead, pass the time in as an argument or use a provider function. ```go type Now func() time.Time func IsExpired(expiry time.Time, now Now) bool { return now().After(expiry) } ``` By passing a `Now` function, you can inject a fixed timestamp in your tests, ensuring your logic is deterministic and reproducible. ## Database Persistence Patterns How you store time depends on your database. Here is what works. ### PostgreSQL timestamptz Storage PostgreSQL provides the `TIMESTAMP WITH TIME ZONE` (or `timestamptz`) type. It is a common misconception that this type stores the timezone. When you send a timestamp with an offset to a `timestamptz` column, PostgreSQL performs a three-step operation: 1. It parses the incoming string and offset. 2. It converts the value to UTC. 3. It stores the UTC value as an 8-byte integer. The offset is discarded. This conversion happens on the fly. While it is a session-level CPU cost rather than a storage bottleneck, it causes panic when a developer queries the database from a GUI tool set to local time, sees a different hour than what is on the server, and assumes the database is corrupted. The data is fine. Your DB tool is just "helping" you. ```sql -- Checking the current session timezone SHOW timezone; -- Retrieving UTC data formatted for a specific zone SELECT created_at AT TIME ZONE 'America/New_York' FROM transactions; ``` ### Couchbase and NoSQL UTCMilli Persistence In a NoSQL JSON document store like Couchbase, you should prioritize **Millisecond Unix Epochs (UTCMilli)** for database storage. ```json { "created_at": 1778857200000 } ``` Storing time as a 64-bit integer is the most accurate way to persist temporal data. It is inherently normalized to UTC, avoids the overhead of string parsing, and prevents timezone ambiguity. Because it is a numeric type, it supports high-performance range scans and sorting in SQL++ without the need for complex casting. Treat ISO 8601 strings as a serialization format for transit, not storage. If you need human readability during a 3 AM incident response, use the `MILLIS_TO_STR()` function in your query. The minor benefit of raw readability in a JSON file is never worth the loss of numeric precision. ## The Handshake Boundary While the database wants integers, the network wants strings. The absolute rule for the client-server boundary is a two-step handshake: 1. **Transport:** The backend sends an **ISO 8601 String** ending in "Z". 2. **Presentation:** The client converts that UTC string into **Local Time**. Never perform localization on the server. A mobile device can change timezones while a request is in flight. The backend should provide the raw UTC data, and the client application must use the local operating system APIs to format the string for the user. ### Browser JavaScript Formatting JavaScript's `Date` object is notoriously flawed, but modern browsers have finally stabilized `Intl.DateTimeFormat`. You pass it the UTC string, and the browser automatically detects the user's OS timezone and formats it correctly. ```javascript const utcString = "2026-05-15T14:30:00Z"; const date = new Date(utcString); const formatter = new Intl.DateTimeFormat('default', { year: 'numeric', month: 'short', day: 'numeric', hour: 'numeric', minute: '2-digit' }); console.log(formatter.format(date)); ``` ### Mobile OS Localization Mobile OS libraries are specifically designed to handle timezone changes on the fly. On iOS (Swift), parse the string using `ISO8601DateFormatter` and format it with `DateFormatter`. The OS will automatically pick up the user's current locale and timezone. ```swift let formatter = ISO8601DateFormatter() let date = formatter.date(from: "2026-05-15T15:00:00Z")! let localFormatter = DateFormatter() localFormatter.dateStyle = .medium localFormatter.timeStyle = .short // Let the iPhone figure out if they just flew to Samoa print(localFormatter.string(from: date)) ``` Android (Kotlin) uses the `java.time` package (`Instant`). You parse the UTC moment and convert it to the system's default zone for display. ```kotlin val instant = Instant.parse("2026-05-15T15:00:00Z") val localDateTime = ZonedDateTime.ofInstant(instant, ZoneId.systemDefault()) val formatter = DateTimeFormatter.ofLocalizedDateTime(FormatStyle.MEDIUM) println(localDateTime.format(formatter)) ``` These libraries are updated by the OS vendors whenever political boundaries or DST rules change, shielding your application from maintenance overhead. ## The short version Measure duration with monotonic clocks. Store time as UTCMilli integers in NoSQL, `timestamptz` in Postgres. Send ISO 8601 strings over the wire. Let the client handle formatting. Do that consistently and your system stays correct even when governments decide to delete a Friday. ## Further Reading - [The Amazing "Plain Text" Talk by Dylan Beattie](https://www.youtube.com/watch?v=gd5uJ7Nlvvo) - [Go Official: Time Package Documentation](https://pkg.go.dev/time) - [Russ Cox: Monotonic Elapsed Time Measurements in Go](https://go.googlesource.com/proposal/+/master/design/12914-monotonic.md) - [PostgreSQL Official: Date/Time Types (timestamptz)](https://www.postgresql.org/docs/current/datatype-datetime.html) - [Couchbase Official: SQL++ Date Functions](https://docs.couchbase.com/server/current/n1ql/n1ql-language-reference/datefun.html) --- ## Building a Poor Document Store inside PostgreSQL **Date:** 2026-05-09 **URL:** https://lorbic.com/building-a-poor-document-store-inside-postgresql/ **Description:** A forensic analysis of why abusing JSONB for schemaless architecture leads to write amplification, TOAST bloat, and catastrophic query planner failures. The marketing for "schemaless" architecture was brilliant. It promised speed, agility, and a life free from the tyranny of `ALTER TABLE` migrations. When PostgreSQL introduced `JSONB` in version 9.4, many developers saw it as a green light to treat Postgres like MongoDB. I’ve seen this pattern in dozens of codebases. It starts with a single `metadata` column, but within months, the entire business logic is buried inside a 2MB JSON blob. This isn't agility; it’s an architectural debt trap. By bypassing the relational schema, you aren't just losing type safety; you are fighting the physical storage engine of PostgreSQL. ![PostgreSQL JSONB Performance Failure](jsonb-anti-pattern-header.svg) ## The Cost of Write Amplification PostgreSQL was designed to handle rows. When you update a row, Postgres doesn't change the data in place. Instead, it uses a mechanism called MVCC to create a new version of that row. {{< details "What is MVCC?" >}} **Multi-Version Concurrency Control (MVCC)** is how Postgres handles multiple people reading and writing at the same time without locking the whole table. Instead of overwriting data, Postgres keeps the old version (for people still reading it) and writes a brand new version of the row for the new data. Later, a background process called **VACUUM** cleans up the old "dead" rows. {{< /details >}} If you have a normalized table with 50 columns, and you update a single `boolean` flag, Postgres writes a few bytes to a new row version. But if you store your data in a massive `JSONB` document, Postgres has no idea how to "partially" update it. If your document is 1MB and you update a single key, Postgres must write the entire 1MB document to a new location on disk. ## Disk Bloat from TOAST Postgres stores data in 8KB pages. If a row exceeds that size, Postgres moves the large data into a separate storage area called TOAST. {{< details "What is TOAST?" >}} **TOAST (The Oversized-Attribute Storage Technique)** is the "trunk" of the database. When a row is too big to fit in a standard 8KB "closet" (a page), Postgres breaks the large parts into chunks and puts them in the TOAST table. It leaves a small pointer in the original row so it can find the data later. {{< /details >}} When you abuse `JSONB` to store large documents, almost every row ends up in TOAST. Every time you update a small field inside that JSON, Postgres has to rewrite the entire TOAST entry. I ran a quick test to see the impact. I updated a single flag in a 2MB JSON document 1,000 times. ```sql -- Checking the physical size of the TOAST table SELECT relname, relpages FROM pg_class WHERE oid = (SELECT reltoastrelid FROM pg_class WHERE relname = 'users_jsonb'); ``` The result was 2GB of disk bloat for a single row’s updates. In a normalized table, that same operation would have cost less than 100KB. You are literally burning disk I/O to maintain a "flexible" schema. Quite literally, your disk is getting TOASTed. ## Query Planner Blindness The PostgreSQL query planner is an incredible piece of engineering. It uses statistics to decide whether to use an index or scan the whole table. It knows how many `NULLs` are in a column and the distribution of values. But the planner is blind to what is inside your `JSONB` blob. If you query a standard column, the planner knows exactly how many rows to expect. If you query a key inside JSON, it has to guess. Usually, it guesses a very small number (like 10 rows). If the planner thinks only 10 rows match, it might choose a "Nested Loop" join, which is fast for 10 rows but catastrophic for 100,000 rows. I’ve seen queries that should take 5ms balloon to 30 seconds because the planner made a bad guess based on a `JSONB` blind spot. ## The Cost of GIN Indexes To make JSON queries fast, you usually end up using a **GIN (Generalized Inverted Index)**. GIN indexes are powerful, but they are heavy. Every time you insert a JSON document, Postgres has to break the entire document apart, find every key and value, and update the index. In my benchmarks, inserting into a table with a GIN index is **5x to 10x slower** than a standard B-Tree index. You've traded your write performance for the convenience of not defining a schema. ## Valid Use Cases for JSONB After heavily trashing it, I should clarify: `JSONB` is an incredible tool when used for its intended purpose. You should absolutely use it for: 1. **Third-Party Webhooks:** When a service like Stripe or GitHub sends you a webhook, you don't control the schema. Storing the raw payload in a `JSONB` column ensures you never drop data if they add a new field. 2. **User-Defined Custom Fields:** If you are building a SaaS where users can define their own attributes for a CRM contact, a sparse `JSONB` column is vastly superior to creating a table with 100 `custom_field_x` columns. 3. **Historical Snapshots:** Storing a frozen snapshot of an order receipt or a configuration state at a specific point in time, where you need to guarantee the shape of the data won't change even if your active relational tables evolve. ## Fixing Slow JSON Queries If you have a valid `JSONB` payload but find yourself frequently filtering or joining on a specific key inside of it, you don't have to choose between flexibility and performance. You can use **Generated Columns** to bridge the gap: ```sql ALTER TABLE users ADD COLUMN external_id TEXT GENERATED ALWAYS AS (payload->>'id') STORED; CREATE INDEX idx_external_id ON users(external_id); ``` This gives you the best of both worlds: the raw payload stays safely in your JSON blob, but the query planner gets a physical column with real statistics to work with. Don't let the allure of "agility" trick you into building a second-rate document store inside the world's most advanced relational database. Define your core schema, and use `JSONB` only for the margins. Your disk, your CPU, and your future self will thank you. ## Further Reading - [PostgreSQL Official: JSON Types](https://www.postgresql.org/docs/current/datatype-json.html) - [PostgreSQL Official: JSONB Indexing](https://www.postgresql.org/docs/current/datatype-json.html#JSON-INDEXING) - [PostgreSQL Official: Generated Columns](https://www.postgresql.org/docs/current/ddl-generated-columns.html) - [PostgreSQL Official: TOAST Storage](https://www.postgresql.org/docs/current/storage-toast.html) - [PostgreSQL Official: Concurrency Control (MVCC)](https://www.postgresql.org/docs/current/mvcc.html) --- ## How to Handle PostgreSQL Database Migrations in Go with Goose **Date:** 2026-05-08 **URL:** https://lorbic.com/how-to-handle-postgresql-database-migrations-in-go-with-goose/ **Description:** A breakdown of database migration strategies and implementation guide using PostgreSQL, Golang, and the Goose migration engine. Application code is stateless. You can tear down a container and spin up a new one in milliseconds without losing data. Databases are stateful. When you deploy new application logic that requires a new column, an index, or a table, you must transition the physical storage schema from state A to state B without destroying the underlying data or locking the system. This process is a database migration. Working with databases is the kind of thing I will gladly skip dinner for (: ```mermaid graph LR A[State A: Initial Schema] -->|Migration 001| B[State B: Tables Created] B -->|Migration 002| C[State C: Indexes Added] C -->|Rollback 002| B class A state-initial class C state-final ``` ![Goose Migration Terminal Status](goose-migration-header.svg) ## The Road to Raw SQL My obsession with relational databases started in 2018 while building my first Django app in college. Django’s ORM is a masterpiece of abstraction, and it was through its lens that I first learned about ACID properties, Normal Forms, and the uncompromising logic of foreign keys. In the years since, I’ve worked across the spectrum, including production environments and toy projects alike, using .NET, PHP, Python, and Node. Each ecosystem has its own way of shielding the developer from the database. Each tries to pretend that the disk doesn't exist. But as I’ve moved deeper into Go and PostgreSQL, my preference has shifted. I no longer want the abstraction; I want the raw SQL. I want to see the DDL. I want to own the migration. This isn't just about the language. I just want to know exactly what is happening to the data. Executing migrations manually via SQL clients is an operational liability. It lacks version control, it cannot be integrated into CI/CD pipelines, and it is prone to human error. Migrations must be treated as code: versioned, reviewed, and executed deterministically. ```mermaid sequenceDiagram participant Dev as Developer participant Git as Version Control participant CI as CI/CD Pipeline participant DB as Production Database Dev->>Git: Push SQL Migration Git->>CI: Trigger Build CI->>CI: Run Tests/Lint CI->>DB: Apply Migration (Goose) DB-->>CI: Success/Version Updated CI-->>Dev: Deployment Successful ``` This blog details the mechanics of database migrations and provides a strict implementation guide using PostgreSQL for storage, Go for execution, and Goose as the migration engine. ## The Mechanics of Schema State A migration engine requires three components to operate safely: 1. **Immutable Migration Files**: Ordered scripts containing the Data Definition Language (DDL) required to transition the schema. 2. **Directionality**: Explicit `Up` (apply) and `Down` (rollback) logic. 3. **State Tracking**: An internal table within the target database recording which migration files have been successfully executed. When the migration engine runs, it compares the local directory of migration files against the internal state table. It computes the delta and applies the pending `Up` scripts sequentially. ```mermaid flowchart TD subgraph Local Files F1[001_init.sql] F2[002_add_index.sql] F3[003_add_column.sql] end subgraph DB State Table S1[001_init] S2[002_add_index] end F1 --- S1 F2 --- S2 F3 -->|Delta Found| Engine{Goose Engine} Engine -->|Apply| DB[(PostgreSQL)] DB -->|Update| S3[003_add_column] ``` ## The Stack: PostgreSQL, Go, and Goose The Go ecosystem has several migration tools. Goose is my favorite choice because it treats SQL as the primary interface. Unlike ORM-based migration tools that abstract DDL into application code, Goose uses raw `.sql` files. This lets you use PostgreSQL-specific features (like concurrent index creation or custom data types) that abstraction layers often block. Goose also supports `go:embed`, allowing you to compile your SQL migration files directly into your Go binary. This produces a single, self-contained executable that can self-migrate before starting the HTTP server. ```mermaid graph TD subgraph Compile Time SQL[.sql files] --> Embed[go:embed] Embed --> Bin[Binary Executable] end subgraph Deployment Bin -->|1. Migrate| DB[(Postgres)] Bin -->|2. Start| Srv[HTTP Server] end ``` ## How to Implement We will follow a strict implementation workflow to build the migration binary. ### 1. Project Architecture Create the directory structure for the migrations. We keep the SQL files isolated from the application logic. ```bash mkdir -p db/migrations ``` ### 2. Defining the SQL Migrations Goose uses a specific comment syntax to separate `Up` and `Down` migrations within a single file. Create the initial schema in `db/migrations/001_initialize_users.sql`. ```sql -- +goose Up CREATE TABLE users ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), email VARCHAR(255) UNIQUE NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() ); -- +goose Down DROP TABLE users; ``` ### 3. The Zero-Downtime Migration Pattern Standard DDL commands in PostgreSQL, such as `ALTER TABLE` or `CREATE INDEX`, acquire an `AccessExclusiveLock`. This lock blocks all read and write operations on the table until the migration completes. On a table with millions of rows, this causes an immediate production outage. ```mermaid stateDiagram-v2 [*] --> Idle Idle --> MigrationStarted: Run ALTER TABLE state MigrationStarted { [*] --> LockAcquired: AccessExclusiveLock LockAcquired --> WriteBlocked: App Writes Waiting... WriteBlocked --> ReadBlocked: App Reads Waiting... ReadBlocked --> Outage: Queues Full / Timeouts } Outage --> Idle: Migration Finished / Lock Released ``` To bypass this, PostgreSQL provides `CREATE INDEX CONCURRENTLY`. However, this command cannot be run inside a transaction block. Goose wraps migrations in transactions by default. We must use the `-- +goose NO TRANSACTION` directive to execute zero-downtime migrations safely. Create `db/migrations/002_add_status_index.sql`. ```sql -- +goose NO TRANSACTION -- +goose Up CREATE INDEX CONCURRENTLY idx_users_created_at ON users(created_at); -- +goose Down DROP INDEX CONCURRENTLY IF EXISTS idx_users_created_at; ``` ### 4. Go Implementation We will now implement the Go code to execute these migrations. Following the staged implementation workflow, we isolate the migration logic into its own package. ```go package database import ( "database/sql" "embed" "fmt" "github.com/pressly/goose/v3" _ "github.com/jackc/pgx/v5/stdlib" ) //go:embed migrations/*.sql var embedMigrations embed.FS func MigrateDatabase(dbUrl string) error { db, err := sql.Open("pgx", dbUrl) if err != nil { return fmt.Errorf("failed to open db: %w", err) } defer db.Close() goose.SetBaseFS(embedMigrations) if err := goose.SetDialect("postgres"); err != nil { return fmt.Errorf("failed to set dialect: %w", err) } if err := goose.Up(db, "migrations"); err != nil { return fmt.Errorf("migration failed: %w", err) } return nil } ``` ### 5. Execution and State Verification When `MigrateDatabase` is called from your `main.go`, Goose will connect to PostgreSQL and execute the pending `.sql` files. You can verify the state by querying the internal tracking table created by Goose. ```sql SELECT id, version_id, is_applied, tstamp FROM goose_db_version ORDER BY id DESC; ``` The output will confirm the exact timestamp each migration was applied. The database state is now deterministic, version-controlled, and mechanically linked to the application binary. ### 6. Local Development with the Goose CLI While embedding migrations in the Go binary is perfect for production deployments, you need the Goose CLI for day-to-day local development. Install the CLI via Go: ```bash go install github.com/pressly/goose/v3/cmd/goose@latest ``` To avoid typing the driver and connection string every time, I export them in my `.env` or `.envrc` file: ```bash export GOOSE_DRIVER=postgres export GOOSE_DBSTRING="postgres://user:pass@localhost:5432/app?sslmode=disable" export GOOSE_MIGRATION_DIR="db/migrations" ``` Once those are set, these are the core commands you will use: **Creating a new migration file:** This generates a timestamped `.sql` file with the correct `Up` and `Down` annotations. ```bash goose create add_users_table sql ``` **Checking migration status:** See exactly which migrations are pending and which have been applied to your local database. ```bash goose status ``` **Applying pending migrations:** Run all unapplied `Up` scripts sequentially. ```bash goose up ``` **Rolling back a migration:** Run the `Down` script for the most recently applied migration. If you break your local schema, this is how you step backwards safely. ```bash goose down ``` ### 7. Automating in CI/CD Running `goose up` from your laptop against the production database is a rite of passage, but it is also how you take down the company. Migrations belong in your deployment pipeline. Because Goose relies on standard SQL files and a single binary, integrating it into CI/CD is just a matter of running the CLI against your production database *before* you deploy the new application code. Here is what that looks like in a standard GitHub Actions workflow: ```yaml name: Production Migrations on: push: branches: - main jobs: migrate: runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v4 - name: Install Goose run: curl -fsSL https://raw.githubusercontent.com/pressly/goose/master/install.sh | sh - name: Execute Migrations env: GOOSE_DRIVER: postgres GOOSE_DBSTRING: ${{ secrets.PROD_DB_URL }} run: goose -dir db/migrations up ``` For the legacy enterprise folks still riding the Jenkins train, the mechanics are exactly the same. You just drop the `curl` install and the `goose up` command into a `sh` block inside your `Jenkinsfile` deployment stage. By keeping this in CI/CD, the pipeline becomes the only entity with the credentials and authority to alter the production schema. It is deterministic, auditable, and entirely mechanical. --- ## Migrating Cloudflare to Terraform **Date:** 2026-04-28 **URL:** https://lorbic.com/cloudflare-management-with-terraform/ **Description:** A technical reference on migrating existing Cloudflare infrastructure to Terraform. Includes an automated bootstrapping script to bypass manual state imports and handle API edge cases. A while back, I was tinkering in the Cloudflare dashboard and accidentally fat-fingered a DNS configuration. I didn't realize the impact immediately, but I ended up taking down `lorbic.com` for an hour. When I scrambled to fix it, I hit a wall: there was no "undo" button. There was no Git history to tell me what the record *used* to point to, and no review to catch the mistake before I blew it. That was the day I realized that managing infrastructure by clicking around a UI, "Click-Ops" is a massive liability. And why everyone screams IaC, Terraform, Cloudformation etc. Moving to Infrastructure as Code (IaC) with Terraform was the obvious fix. But migrating an *existing* Cloudflare account with dozens of records and rules isn't as simple as just writing the code. ![Terraform Apply Cloudflare DNS](terraform-cloudflare-header.svg) ## The 60-Second Terraform Crash Course If you are new to DevOps, you only need to understand three core concepts to see why migrating existing infrastructure is notoriously difficult: 1. **The Code (`.tf` files):** This is where you declare your desired infrastructure (e.g., "I want a CNAME record pointing to my server"). 2. **The Provider:** The plugin that translates your code into Cloudflare API requests. They are available for almost every cloud provider on planet Earth and beyond (maybe!). 3. **The State (`.tfstate`):** A JSON file where Terraform maps your code to the actual resources that *already exist* in the real world. That third point is the bottleneck. If your `.tfstate` file is empty, Terraform assumes your Cloudflare account is empty. If you write your configuration and run `terraform apply`, Terraform tries to create brand new records. The Cloudflare API will immediately reject the request because those records already exist. To fix this, you must explicitly link your code to your live infrastructure using `terraform import`. For a few dozen records across multiple domains, doing this by hand is a soul-crushing exercise. Here is how I automated the entire generation and state import process. ## The Target Architecture Dumping hundreds of DNS records into a single `main.tf` file creates an unreadable configuration. A better approach is to use Terraform modules—one for the account-level resources (like Zero Trust), and one for each specific domain (zone). The goal is a directory structure that looks like this: ```text cloudflare/ ├── main.tf # Provider config and module definitions ├── terraform.tfstate # The single state file ├── account/ # Account-level config │ ├── access_apps.tf │ └── access_policies.tf └── zones/ ├── lorbic.com/ # Zone-specific config │ ├── dns.tf │ └── rules.tf └── vikashpatel.net/ ├── dns.tf └── rules.tf ``` The root `main.tf` stays clean, simply tying the modules together: ```hcl # cloudflare/main.tf module "lorbic_com" { source = "./zones/lorbic.com" providers = { cloudflare = cloudflare } } module "vikashpatel_net" { source = "./zones/vikashpatel.net" providers = { cloudflare = cloudflare } } ``` ## The Automation Script Cloudflare maintains a CLI tool called [`cf-terraforming`](https://github.com/cloudflare/cf-terraforming). It queries your account and generates both the `.tf` configuration files and the shell commands required to import the state. However, `cf-terraforming import` only *outputs* the raw `terraform import` commands. It does not execute them, and more importantly, it assumes a flat directory structure. Because we are using a modular architecture, the resource addresses must be prefixed with the module name. Instead of running these manually, I wrote a bash script to loop through an array of my domains, generate the `.tf` files, parse the import output, remap the addresses to the correct modules, and execute the imports automatically. Here is the core parsing logic from the script: ```bash # 1. Generate the Terraform config cf-terraforming generate \ --zone $ZONE_ID \ --resource-type cloudflare_record > "zones/${DOMAIN}/dns.tf" # 2. Generate the import commands, parse them, and execute cf-terraforming import \ --zone $ZONE_ID \ --resource-type cloudflare_record | while IFS= read -r line; do if [[ "$line" == terraform\ import\ * ]]; then # Parse the raw output: "terraform import cloudflare_record.name " resource_addr=$(echo "$line" | awk '{print $3}') import_id=$(echo "$line" | awk '{print $4}') # Prefix with the module name (e.g., module.vikashpatel_net.cloudflare_record.name) module_addr="module.${MODULE_NAME}.${resource_addr}" # Execute the import echo "Importing: $module_addr" terraform import "$module_addr" "$import_id" fi done ``` By wrapping this in a loop, you can bootstrap an entire account in seconds. Once the script finishes, running `terraform plan` should cleanly report that your infrastructure matches your configuration. ## Edge Cases and API Limitations Automated generation gets you 95% of the way there. During the migration, you will likely hit API quirks that require manual reconciliation. ### 1. The Read-Only Record Trap Cloudflare manages certain DNS records automatically, such as Email Routing MX records or Zero Trust Access IPv6 CNAMEs. If `cf-terraforming` exports these into your `.tf` files, your next `terraform apply` will fail with **API Error 1046** or **1043**. Terraform cannot assume ownership of records that Cloudflare provisions internally. To fix it, you must manually delete these specific resource blocks from your generated `.tf` files. ### 2. Ruleset Index Conflicts Cloudflare restricts accounts to a single "URL Normalization" ruleset per zone. Occasionally, the generation tool gets confused by naming conflicts, generating `cloudflare_ruleset.transform_1` alongside `cloudflare_ruleset.transform_2`. Attempting to apply this will result in an API rejection. You will need to manually reconcile these conflicts by deleting the duplicate block and using `terraform state rm ` to clear it from the state file. ### 3. String Escaping and Normalization Terraform is strict regarding state representation. Cloudflare's API might return a TXT record with explicit quotes (`"v=spf1 -all"`), but the generated HCL might omit them or escape them differently. During your first `terraform plan`, you may see dozens of `~ update in-place` changes. This is Terraform normalizing the formatting. Review the diff carefully to ensure the actual values aren't mutating, then apply it to sync the API with your local state format. ## The Day-to-Day Workflow Post-migration, the Cloudflare dashboard becomes strictly read-only. Manual UI changes will be overwritten the next time Terraform runs. The new workflow relies entirely on the CLI: ```bash # Verify changes against live infrastructure terraform -chdir=cloudflare plan # Apply changes defined in .tf files terraform -chdir=cloudflare apply ``` Migrating requires some upfront effort to handle API errors and organize the module structure. In exchange, the infrastructure becomes version-controlled, auditable, and predictably reproducible. You'll never have to guess what a deleted DNS record used to look like again. And honestly, having your entire DNS infrastructure version-controlled is incredibly satisfying to work with. --- ## Internet Graveyard **Date:** 2026-04-27 **URL:** https://lorbic.com/internet-graveyard/ **Description:** An archive of sunsetted technologies, derelict architectures, and dead protocols. We learn as much from failure as we do from success. {{< graveyard_archive >}} --- ## A Tale of Web Vitals **Date:** 2026-04-11 **URL:** https://lorbic.com/a-tale-of-web-vitals/ **Description:** A technical guide to Web Vitals. Solve LCP, TBT, CLS, and Accessibility issues using automated build-time pipelines and vanilla JS patches. **TL;DR:** I achieved 95+ Lighthouse scores by removing abstractions and automating browser fundamentals. This guide details how to solve **LCP network discovery**, **main thread congestion**, and **third-party accessibility traps** using Hugo pipelines and vanilla JavaScript. --- "Why is the LCP 4.2 seconds? It's just a static site". I was staring at a Lighthouse report that felt like an insult. **Lorbic.com is a zero-dependency Hugo site.** No React, no heavy frameworks, just vanilla CSS and minimal JS. Yet, the mobile performance was tanking. Performance is not a byproduct of your stack; it is a byproduct of engineering discipline. Lighthouse is an [**emulator of user frustration**](https://developer.chrome.com/docs/lighthouse/overview/), not a game to be tricked. Green scores come from understanding **browser threads**, not performance plugins. Here is the before and after of my recent performance audit. **The Baseline (Before):** ![Lighthouse Audit Before](before.png) **The Result (After):** ![Lighthouse Audit Mobile After](after.png) _Mobile: Significant improvement in Performance and Accessibility._ ![Lighthouse Audit Desktop After](desktop-after.png) _Desktop: Near-perfect scores across all categories._ This guide explains what these metrics actually mean, how developers break them, and the exact code I used to solve them. _Note: To understand the deep mechanics behind these optimizations, read my companion guide on [The Critical Rendering Path](/til/2026-04-11-the-critical-rendering-path/)._ --- ## Largest Contentful Paint [**Largest Contentful Paint (LCP)**](https://web.dev/articles/lcp) measures when the largest visual element renders. It is the primary metric for perceived speed. If LCP exceeds 2.5 seconds, the user feels the site is stuck. Standard advice says to lazy-load everything. But **blanket lazy loading kills LCP.** When you tag a hero image as `lazy`, you force the [**browser's preload scanner**](https://web.dev/articles/preload-scanner) to ignore it. The engine must parse CSS and build the render tree before discovering the image exists. This single oversight costs over a second in network delay. I automated the distinction between **critical and deferred assets** in the build pipeline. My Hugo `render-image.html` hook now tracks state during the render cycle. The first image rendered on any post automatically receives [`fetchpriority="high"`](https://web.dev/articles/fetch-priority) and [`loading="eager"`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/img#loading). ```go {{- /* Smart Auto-Eager Loading for the First Image (LCP) */ -}} {{- $isFirstImage := false -}} {{- if not (.Page.Scratch.Get "hasRenderedFirstImage") -}} {{- $isFirstImage = true -}} {{- .Page.Scratch.Set "hasRenderedFirstImage" true -}} {{- end -}} {{- $loading := "lazy" -}} {{- $fetchpriority := "auto" -}} {{- if $isFirstImage -}} {{- $loading = "eager" -}} {{- $fetchpriority = "high" -}} {{- end -}} ``` By forcing the browser to start the network request the millisecond the HTML parses, my LCP dropped from 4.2s to 1.1s. ## Total Blocking Time [**Total Blocking Time (TBT)**](https://web.dev/articles/tbt) tracks main thread congestion. If the thread locks for over 50ms, the page freezes. This happens when we ship desktop assets to mobile devices. Shipping a 3000px wide image to an iPhone wastes data and chokes the CPU during decoding. I implemented an **automated WebP pipeline** that generates a multi-size `` element at build time. Mobile users now download a ~45KB WebP asset instead of a multi-megabyte original. This reduces CPU strain. ```go {{- if $image -}} {{- $widths := slice 480 800 1200 -}} {{- end -}} ``` **CSS is also render-blocking.** The browser won't draw a single pixel until it parses the entire CSS file. I use [**PurgeCSS**](https://purgecss.com/) to drop my payload from 850KB to 87KB. The trick is cleanup the css. By running PurgeCSS and using `hugo_stats.json` as the source of truth, I ensure dynamically generated classes never get remove by using the following rule. ```javascript // postcss.config.js const purgecss = require("@fullhuman/postcss-purgecss")({ content: ["./hugo_stats.json", "./layouts/**/*.html", "./content/**/*.md"], defaultExtractor: (content) => { if (content.trim().startsWith("{")) { const els = JSON.parse(content).htmlElements; return (els.tags || []).concat(els.classes || [], els.ids || []); } return content.match(/[A-Za-z0-9-_:\/]+/g) || []; }, }); ``` ## Cumulative Layout Shift [**Cumulative Layout Shift (CLS)**](https://web.dev/articles/cls) measures visual instability. This is caused by images without dimensions, custom fonts that cause a **Flash of Unstyled Text (FOUT)**, or late theme scripts. My image pipeline automatically injects `width` and `height` attributes. This reserves physical space before the image downloads. I also added [``](https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/rel/preload) to the `baseof.html` for critical font files. I updated the js with an inline theme script at the top of the ``. By setting the `data-theme` attribute before the browser reaches the ``, I apply colors instantly. This eliminates the "white flash" typical of dark mode sites. ## Accessibility Accessibility is about **keyboard flow**, not just contrast. I was failing audits because of an anti-spam widget. It injected a link marked as `aria-hidden="true"` that was still focusable via keyboard. This created a [**"focus trap"**](https://developer.mozilla.org/en-US/docs/Web/Accessibility/Keyboard-navigable_JavaScript_widgets#using_tabindex): a keyboard user lands on an element the screen reader claims does not exist. Since I do not control the third-party script, I used a [**`MutationObserver`**](https://developer.mozilla.org/en-US/docs/Web/API/MutationObserver) in vanilla JS to patch the DOM the moment the widget renders. ```javascript const mutObserver = new MutationObserver(() => { const logo = widget.querySelector(".altcha-logo"); if (logo && logo.hasAttribute("aria-hidden")) { logo.removeAttribute("aria-hidden"); mutObserver.disconnect(); } }); mutObserver.observe(widget, { childList: true, subtree: true }); ``` This tiny patch was the difference between a 90 and a perfect 100 on my accessibility score. ## Icon Latency Interactive elements like **"Copy Code"** buttons should not wait for an external font file. Relying on icon fonts adds unnecessary network requests and causes a **"Flash of Unstyled Icon"**. I replaced FontAwesome with **optimized SVG strings** embedded directly in the JavaScript bundles. UI feedback is now instantaneous. The main thread does not wait for external files to resolve before rendering button states. ## Live Observability Performance is a state of being, not a one-time event. To keep myself honest, I built a **Performance Telemetry HUD** into the footer. It uses the [**Performance API**](https://developer.mozilla.org/en-US/docs/Web/API/Performance_API) to show real-time load metrics, transfer size, and request counts. Having these numbers visible during development makes it impossible to ignore heavy images or bloated dependencies. ## Dealing with Phantom Warnings Even perfect code faces noise. While auditing, I noticed intermittent warnings for `SharedStorage` and `Fledge` cookies. These came from **Cloudflare Zaraz and Google Analytics**, not my source code. High-performance engineering means knowing which bytes you own and which bytes are forced upon your users by third-party services. ## Trade-offs and Costs Every architectural victory has a cost. Choosing a **zero-dependency static architecture** means: - **Build Complexity:** My `render-image.html` hook is more complex than a standard `` tag. I have to maintain a stateful build-time pipeline for LCP detection. - **The Caching Trade-off:** I explicitly set `inline_css = false`. Lighthouse recommends "inlining critical CSS" to save a request, but this destroys long-term caching. I trade a fraction of a second on the first load for massive gains on the next ten visits. - **Maintenance:** Since I do not use a framework, I am responsible for the "patch work" like the `MutationObserver` for accessibility. There is no library to update; there is only code to maintain. ## The Power of Hugo None of these optimizations would be sustainable without Hugo. I love it for its **uncompromising speed** and engineering elegance. Hugo builds 400+ pages and my entire asset pipeline in under 700ms. This speed is a productivity multiplier. The template engine is the real power: using `.Page.Scratch` to build a **build-time state machine** for LCP detection is a level of flexibility rarely found elsewhere. Hugo allows me to solve complex browser problems during the build so the user never has to solve them at runtime. ## The Cost of Complexity Reflecting on the last [ten-year journey](/10-years-of-lorbic.com-architecture/), I have gone through every phase of the developer lifecycle. I started on managed platforms with no control. I moved to dynamic backends with Django because I thought "real" sites needed databases. I finally landed on a zero-dependency static architecture. The irony is that the more code I removed, the better the experience became. Performance is not about how much logic you add; it is about **technical sovereignty**. In 2018, I wanted a complex platform. In 2026, I want to engineer a fast, accessible, and honest piece of the web. ## Respect the Browser The browser is fast if you give it the right instructions in the right order. Performance is an **engineering discipline** you maintain from the first line of HTML. Respect the browser, and it will reward you with speed. --- ## Why Your Goroutines Need a Speed Limit: Bounded Concurrency in Go **Date:** 2026-04-10 **URL:** https://lorbic.com/bounded-concurrency-in-go/ **Description:** Unbounded concurrency is a reliability nightmare. Learn how to protect your system from OOM kills and database exhaustion by implementing Semaphores and Worker Pools in Go. **TL;DR:** Spawning `go func()` without a limiter is a recipe for system collapse. This guide details how to use **Semaphores** and **Worker Pools** to prioritize predictable stability over absolute speed, protecting downstream dependencies from the thundering herd. --- It's a rite of passage for every Go developer. You receive a list of 10,000 URLs to fetch or 50,000 rows to process. You wrap the workload in an unbounded `go func()` loop, achieving maximum throughput in milliseconds. ```go // Initial approach: unbounded concurrency func ProcessItems(items []Item) { var wg sync.WaitGroup for _, item := range items { wg.Add(1) go func(i Item) { defer wg.Done() process(i) // HTTP request, DB write, etc. }(item) } wg.Wait() } ``` This code runs flawlessly on a local machine. However, in production, it triggers **OOM (Out of Memory)** kills and database connection failures. You achieved speed, but you sacrificed **reliability**. --- ## Why cheap goroutines are expensive Go makes concurrency easy because goroutines are lightweight. They start with a [**2KB stack**](https://go.dev/doc/faq#goroutines). While your CPU can manage 10,000 threads, your downstream dependencies cannot. Unbounded `go func()` loops create a **Thundering Herd**. Instantaneous execution forces your system to simultaneously demand: 1. **Memory:** 10,000 goroutines require 20MB of stack space just to initialize. This excludes the heap allocations required to process JSON or hold HTTP buffers. 2. **File Descriptors:** Every outbound request requires a socket. Standard Linux environments cap file descriptors at 1,024 per process. 3. **Connection Pools:** Your database pool is a finite resource. If it is capped at 50 connections, 9,950 goroutines will block while holding onto their allocated memory, causing massive **GC pressure**. To build resilient systems, you must impose a **Speed Limit**. --- ## Use Semaphores for minimal refactoring A [**Semaphore**]() restricts the number of threads accessing a shared resource. In Go, you can implement this pattern using a **buffered channel**. A buffered channel blocks when full. By using it as a [**token bucket**](https://github.com/vikash-paf/goutils/tree/main/rate#tokenbucket), you control exactly how many goroutines execute their critical path at once. For production workloads requiring burst handling and precise refills, I use the [TokenBucket](https://pkg.go.dev/github.com/vikash-paf/goutils@v1.0.0/rate#TokenBucket) implementation from my `goutils` library. ### How to implement a channel-based Semaphore 1. Create a buffered channel with a capacity equal to your limit. 2. Push an empty struct into the channel before starting work (acquire token). 3. Read from the channel when the work finishes (release token). ```go func ProcessItemsWithSemaphore(items []Item, maxConcurrency int) { var wg sync.WaitGroup sem := make(chan struct{}, maxConcurrency) // The token bucket for _, item := range items { wg.Add(1) go func(i Item) { defer wg.Done() sem <- struct{}{} // Acquire token (blocks if full) defer func() { <-sem }() // Release token process(i) }(item) } wg.Wait() } ``` This pattern is ideal for quick scripts because it requires minimal code changes. However, it still spawns 10,000 goroutines. The loop completes instantly, leaving thousands of blocked goroutines parked in memory. For massive workloads, you need a more robust architecture. --- ## Use Worker Pools for sustained processing While a Semaphore is a traffic light, a [**Worker Pool**](https://en.wikipedia.org/wiki/Thread_pool) is an assembly line. You spawn a fixed number of long-lived "Worker" goroutines that pull jobs from a shared channel. ### How to architect an assembly line 1. Initialize a `jobs` channel. 2. Spawn `N` workers that `range` over that channel. 3. Feed items into the channel. 4. Close the channel to signal workers to exit. ```go func worker(id int, jobs <-chan Item, wg *sync.WaitGroup) { defer wg.Done() for item := range jobs { process(item) } } func ProcessItemsWithPool(items []Item, numWorkers int) { jobs := make(chan Item, len(items)) var wg sync.WaitGroup for w := 1; w <= numWorkers; w++ { wg.Add(1) go worker(w, jobs, &wg) } for _, item := range items { jobs <- item } close(jobs) // Signal shutdown wg.Wait() } ``` Worker Pools decouple the **volume of work** from **resource consumption**. Whether processing 100 items or 100,000, your system only spawns `numWorkers` goroutines. Memory profiles remain flat, and GC pressure is minimized. --- ## Trade-offs and Costs Engineering is a series of trade-offs. Bounding your concurrency introduces specific costs: - **Latency vs. Stability:** Limiting concurrency increases the total execution time for a batch. You trade absolute speed for a predictable, non-collapsing system. - **Complexity:** Worker Pools require more boilerplate than a simple `go func()`. You must manage channel lifecycles and [**context-based cancellation**](https://go.dev/blog/context) properly. - **Deadlock Risk:** Improper channel management in pools can lead to deadlocks if workers are blocked while the feeder loop waits for capacity. --- Optimizing for speed is the first phase of implementation. The second phase; and the most critical for production reliability, is identifying how safely to constrain that speed. Unbounded concurrency is a bug that waits for a traffic spike to trigger. Establish a habit: every time you type `go func()`, identify its upper bound. If the loop is tied to user input or database rows, you need a speed limit. Grab a **Semaphore** for scripts and a **Worker Pool** for production pipelines. Respect the hardware, and your database will remain stable. --- ## Part 3: Casting Shadows Without Trigonometry: The Beauty of Integer Math **Date:** 2026-04-07 **URL:** https://lorbic.com/casting-shadows-bresenham-integer-math/ **Description:** To calculate Field of View in a grid-based game engine, you have to cast rays. The naive approach uses heavy floating-point trigonometry. This post explores Bresenham's Line Algorithm: a 60-year-old technique from the era of hardware plotters that draws perfect lines using only integer addition and comparison. At the end of generating a procedural map for the _[Derelict Facility](https://github.com/vikash-paf/derelict-facility)_ engine, I had a sprawling interconnected maze of rooms and corridors. But there was a devastating problem: I could see everything. The entire map was rendered at once. It looked like a top-down blueprint, not a dark, atmospheric facility. I needed to implement Fog of War. I needed to treat the player character like a lighthouse in the dark. ## The Engineering Problem: The Grid vs. The Real World In the real world, light travels in perfectly straight lines. If you look at a wall, photons travel from the wall to your eye. In a game engine, you don't have a "real world". You have a 2D grid of memory (`[]Tile`). If the player is at `X: 10, Y: 10` and a wall is at `X: 15, Y: 12`, how do you mathematically determine if the player can see the wall? You need to draw a line between them. But grids don't allow smooth lines. They only allow stepping from one integer coordinate to the next. You can't stand at `X: 10.5`. ## The Naive Approach: Floating Point Math The modern instinct for drawing a line from point A to point B is trigonometric. You calculate the angle (`math.Atan2`), find the sine and cosine, and step along the floating-point line, rounding to the nearest integer grid tile at each step. This works, but it is deeply flawed for systems engineering: 1. **It is slow.** Transcendental functions (`sin`, `cos`) are expensive CPU operations. 2. **It is imprecise.** Floating-point rounding errors create "holes" where a ray might accidentally skip a wall tile. 3. **It is overkill.** You don't need a floating-point unit to navigate a 2D integer grid. ## Bresenham's Line Algorithm (1962) In 1962, Jack Bresenham at IBM solved this problem for hardware plotters and early CRT displays. His algorithm determines which pixels to illuminate to form a straight line using **only integer addition, subtraction, and bit-shifting**. No division. No floating point. It was designed for hardware that literally lacked decimal math processors, and it remains the industry standard today. Here is the exact implementation used in the _Derelict Facility_ engine to chart the path of a photon across the map grid: {{< code lang="go" file="internal/world/algo.go" >}} func getLine(x0, y0, x1, y1 int) []entity.Point { var points []entity.Point // dx is the absolute horizontal distance dx := math.Abs(x1 - x0) // dy is the absolute vertical distance. We make it negative as a // math trick so we can ADD it to our error tracker later. dy := -math.Abs(y1 - y0) // Step direction: are we going left or right? Up or down? sx := -1 if x0 < x1 { sx = 1 } sy := -1 if y0 < y1 { sy = 1 } // The Error Tracker: how far we've drifted from the true mathematical line err := dx + dy for { points = append(points, entity.Point{X: x0, Y: y0}) if x0 == x1 && y0 == y1 { break // Destination reached } // Double the error for comparison. // This prevents us from ever needing fractions (like 0.5). e2 := 2 * err // Adjust X if our drift tolerance allows it if e2 >= dy { err += dy x0 += sx } // Adjust Y if our drift tolerance allows it if e2 <= dx { err += dx y0 += sy } } return points } {{< /code >}} ### How the Error Tracker Works The genius of Bresenham is the `err` accumulator. It tracks how far our jagged, step-by-step integer path has drifted from the mathematically perfect straight line. When the drift exceeds a specific threshold, the algorithm corrects itself by stepping diagonally. By using `e2 := 2 * err`, we eliminate the need to compare against `0.5`, allowing the entire calculation to remain strictly within integer bounds. ## Perimeter Raycasting To implement the "Lighthouse", we define a vision radius (e.g., 8 tiles). We want to illuminate everything within that radius, unless a wall blocks the light. The obvious instinct is to just cast 360 rays in a circle based on angles (using sine/cosine). However, as these rays get further away from the player, the physical gap between the rays gets larger. Eventually, the gap becomes wider than a single 1x1 grid tile. This results in missing tiles—annoying "stripes of darkness" appearing inside a lit room. Instead, we use a technique called **Perimeter Raycasting**. The algorithm is strictly grid-aligned: 1. Define a square box around the player based on the radius (Top, Bottom, Left, and Right edges). 2. Iterate through every single tile on that square's perimeter. 3. Calculate a line from the Player to that Target Tile (using Bresenham's algorithm). 4. Walk along that line, tile by tile, setting `Visible = true` until it hits a wall. {{< code lang="go" file="internal/world/map.go" >}} func (m \*Map) castRay(x1, y1, x2, y2 int) { line := getLine(x1, y1, x2, y2) for _, p := range line { // Stop if the ray leaves the map if p.X < 0 || p.X >= m.Width || p.Y < 0 || p.Y >= m.Height { break } idx := m.GetIndex(p.X, p.Y) // Mark the tile as currently visible m.Tiles[idx].Visible = true // Permanently mark it as explored (for fog of war memory) m.Tiles[idx].Explored = true // If the tile is solid matter, light cannot pass through it. // Break the loop to stop casting this specific ray. if !m.Tiles[idx].Walkable { break } } } {{< /code >}} Because `getLine` is so highly optimized, I can comfortably cast hundreds of rays every frame without dropping below 60 FPS. When you strip away heavy abstractions and write algorithms that respect the physical capabilities of the CPU (in this case, basic integer arithmetic), performance stops being a goal you have to fight for and becomes the natural default state of the code. ## Further Reading - [What the Hero Sees (Bob Nystrom)](https://journal.stuffwithstuff.com/2015/09/07/what-the-hero-sees/) - A phenomenal visual breakdown of grid-based FOV algorithms. --- _This is Part 3 of the [Building a Game Engine in Pure Go](/series/building-a-game-engine-in-pure-go/) series. The full source code for Derelict Facility is available on [GitHub](https://github.com/vikash-paf/derelict-facility)._ --- ## Part 2: Decoupling the Renderer: Terminal to Raylib in One Interface **Date:** 2026-04-06 **URL:** https://lorbic.com/decoupling-game-engine-renderer-raylib/ **Description:** If your game logic knows about OpenGL, your architecture has failed. This post dissects the Interface Segregation Principle in Go, demonstrating how the Derelict Facility engine swapped an ANSI terminal renderer for hardware-accelerated Raylib without changing a single line of game simulation code. The biggest architectural mistake you can make when building a game engine is letting the game know how it is being drawn. When I started building the _Derelict Facility_ engine, the output target was a raw ANSI terminal. The engine calculated A\* paths, resolved line-of-sight, and then spewed escape codes (`\033[31m`) to `os.Stdout`. Eventually, I hit the physical limits of terminal emulators: inconsistent character widths (especially for emojis), slow double-buffering, and a hard cap on frame rates. I needed to move to a real, hardware-accelerated graphics library like Raylib. Because I had architected the engine around Go interfaces, I replaced the entire visual and input layer of the game without modifying a single line of the core simulation. ## The Display Boundary If your game logic calls `raylib.DrawRectangle()`, your game logic is permanently bound to Raylib. To prevent this, I defined a strict boundary. The `Engine` struct knows absolutely nothing about ANSI, terminals, OS windows, or GPUs. It only knows about an interface called `Display`: {{< code lang="go" file="internal/display/display.go" >}} type Display interface { Init(gridWidth, gridHeight int, title string) error Close() ShouldClose() bool BeginFrame() EndFrame() Clear(colorHex uint32) DrawText(gridX, gridY int, text string, colorHex uint32) PollInput() []core.InputEvent } {{< /code >}} This interface enforces two critical constraints: 1. **Coordinate Abstraction:** The engine asks to draw text at `(gridX: 10, gridY: 5)`. It does not know or care if that translates to `Row 5, Column 10` in a terminal or `Pixel X: 120, Pixel Y: 120` on a 4K monitor. 2. **Color Abstraction:** The engine asks for a standard `uint32` hex color (`0xFF0000FF`). It is the Display's job to figure out how to render red. ## Implementing the Raylib Concrete With the boundary defined, implementing the Raylib backend was an isolated exercise. The concrete `RaylibDisplay` struct simply translates the abstract grid commands into Raylib-specific hardware calls. {{< code lang="go" file="internal/display/raylib.go" >}} type RaylibDisplay struct { CellWidth int32 CellHeight int32 FontSize int32 FontPath string Font rl.Font } func (r _RaylibDisplay) DrawText(gridX, gridY int, text string, colorHex uint32) { // 1. Translate Grid coordinates to Pixel coordinates pixelY := int32(gridY) _ r.CellHeight colOffset := 0 for _, char := range text { charStr := string(char) // Calculate precise centering for the font glyph vecSize := rl.MeasureTextEx(r.Font, charStr, float32(r.FontSize), 0) charWidth := int32(vecSize.X) pixelX := int32(gridX+colOffset)*r.CellWidth + (r.CellWidth-charWidth)/2 // 2. Execute the hardware-accelerated draw call position := rl.NewVector2(float32(pixelX), float32(pixelY)) // Notice we cast our abstract uint32 color directly into a Raylib Color struct rl.DrawTextEx(r.Font, charStr, position, float32(r.FontSize), 0, rl.GetColor(uint(colorHex))) colOffset++ } } {{< /code >}} This implementation hides massive complexity from the game engine. The engine doesn't need to load the `.ttf` font file, generate a texture atlas, calculate bilinear filtering, or measure bounding boxes. It just says "Draw an `@` at 5,5 in Red". ## Input Mechanism Output is only half the battle. If your game loop calls `raylib.IsKeyPressed()`, it is once again permanently bound to the framework. The `Display` interface demands a slice of `core.InputEvent`. This completely abstracts away the physical hardware capturing the keystrokes. {{< code lang="go" file="internal/display/raylib.go" >}} func (r \*RaylibDisplay) PollInput() []core.InputEvent { var events []core.InputEvent if rl.IsKeyPressed(rl.KeyW) || rl.IsKeyPressedRepeat(rl.KeyW) { events = append(events, core.InputEvent{Key: core.KeyW}) } if rl.IsKeyPressed(rl.KeyS) || rl.IsKeyPressedRepeat(rl.KeyS) { events = append(events, core.InputEvent{Key: core.KeyS}) } // ... return events } {{< /code >}} Because the engine only consumes `core.InputEvent`, I could swap the Raylib keyboard hook for an Xbox controller hook, a network socket (for multiplayer), or a pre-recorded slice of inputs (for deterministic unit testing), all without touching the core game loop. ## Dependency Injection When the program starts, the `main` function decides which reality the engine lives in. The `Engine` struct accepts the `Display` interface via Dependency Injection. {{< code lang="go" file="cmd/derelict/main.go" >}} func main() { // 1. Instantiate the concrete hardware layer disp := display.NewRaylibDisplay(12, 24, 24, "assets/fonts/FiraCode-Bold.ttf") // 2. Instantiate the game logic generatedMap, _, _ := world.NewFacilityGenerator(1234).Generate(120, 30) ecsWorld := ecs.NewWorld() // 3. Inject the hardware layer into the agnostic engine gameEngine := engine.NewEngine(disp, generatedMap, ecsWorld, world.TileVariantCold) // 4. Start the simulation gameEngine.Run() } {{< /code >}} If I want to run the game headless on a CI server to benchmark the pathfinding algorithm, I write a `NullDisplay` struct that implements the interface but does nothing. The Interface Segregation Principle is rarely used correctly in modern microservice architectures, where interfaces are often just 1:1 mocks for database calls. But in systems engineering and game development, interfaces are the blast doors that protect your complex, expensive business logic (procedural generation, AI, physics) from the volatile, ever-changing demands of I/O hardware. --- _This is Part 2 of the [Building a Game Engine in Pure Go](/series/building-a-game-engine-in-pure-go/) series. The full source code for Derelict Facility is available on [GitHub](https://github.com/vikash-paf/derelict-facility)._ --- ## Part 1: Data-Oriented Design in Go: Why [][]Tile Destroyed My Game Engine **Date:** 2026-04-05 **URL:** https://lorbic.com/data-oriented-design-go-contiguous-memory/ **Description:** The textbook answer for a 2D grid in Go is a slice of slices. In a systems-level game engine running at 60 FPS, this innocent data structure becomes a performance landmine. This post explores pointer chasing, CPU cache lines, and how flattening a 2D map into contiguous memory creates massive performance gains through Data-Oriented Design. Most game development stories start the same way: install Unity, drag some sprites onto a canvas, and press Play. I wanted to understand the metal. I set out to build _[Derelict Facility](https://github.com/vikash-paf/derelict-facility)_, a systems-level game engine from scratch in pure Go. No SDL, no OpenGL wrappers, no Ebiten. The goal wasn't just to ship a game; the goal was to learn the memory layouts and I/O pipelines that modern engines hide behind friendly APIs. The very first decision you face when building a grid-based simulation is how to represent the map in memory. The textbook answer looks obvious: {{< code lang="go" >}} // The obvious approach : a slice of slices type Map struct { Grid [][]Tile } {{< /code >}} In standard application code, this is perfectly fine. In systems programming, where you are rendering a 120x30 grid 60 times a second and running A\* pathfinding algorithms across thousands of nodes, this data structure is a performance landmine. ## The Problem: Pointer Chasing To understand why `[][]Tile` fails at scale, you have to look at how Go allocates memory on the heap. A slice in Go is a small struct containing a pointer to an underlying array, a length, and a capacity. Therefore, a "slice of slices" is actually a list of pointers. Each inner slice (`Grid[y]`) is a separate heap allocation pointing to a block of memory located _somewhere_ else. But "somewhere" is the problem. These inner arrays are scattered randomly across the heap. When the CPU tries to iterate over the grid row by row to render the map or calculate Field of View (FOV), it has to chase a pointer to a completely new memory address for every single row. Each jump is a potential **cache miss**. The CPU stalls while it waits to fetch data from main RAM instead of the blazing-fast L1/L2 cache sitting next to the core. On a tight 16ms frame budget, those memory stalls compound into visible frame drops. ## The Solution: Contiguous Memory In Data-Oriented Design, the primary rule is: **respect the CPU cache**. CPUs do not read memory one byte at a time; they read chunks of memory (cache lines, usually 64 bytes) at once. If your data is laid out sequentially, the CPU will automatically prefetch the next items before your code even asks for them. I re-architected the `Derelict Facility` map as a single, flat, contiguous 1D block of memory: {{< code lang="go" file="internal/world/map.go" >}} ttype Map struct { Tiles []Tile // Size = Width \* Height (one contiguous block) Width int Height int Rooms []Rect } func NewMap(width, height int) *Map { return &Map{ Width: width, Height: height, // One single allocation for the entire world Tiles: make([]Tile, width*height), } } {{< /code >}} Instead of allocating memory `Height` times, we allocate exactly once. The entire map lives in one unbroken block of RAM. ### Stride Math (O(1) Access) If the map is a flat 1D array, how do we access a specific `(x, y)` coordinate? We use **stride math**: a simple formula you'll find at the core of every framebuffer, texture map, and video decoder on Earth: `Index = X + (Y * Width)` {{< code lang="go" file="internal/world/map.go" >}} func (m *Map) GetTile(x, y int) *Tile { // Bounds checking if x < 0 || x >= m.Width || y < 0 || y >= m.Height { return nil } // O(1) mathematical lookup return &m.Tiles[x + y*m.Width] } {{< /code >}} There is zero pointer chasing. The CPU calculates the exact memory offset instantly. When our Raylib renderer scans across the map from left to right, the CPU aggressively prefetches the `Tile` structs because it knows exactly where they are. ## Shrinking the Struct Contiguous memory is only half the battle; the other half is data density. The smaller the struct, the more of them fit into a single 64-byte L1 cache line. If I had defined my `Tile` struct with strings for colors or complex interfaces, the memory footprint would bloat. Instead, I kept it as small as physically possible: {{< code lang="go" file="internal/world/tile.go" >}} type TileType uint8 const ( TileTypeEmpty TileType = iota TileTypeWall TileTypeFloor ) type Tile struct { Type TileType // 1 byte Walkable bool // 1 byte Visible bool // 1 byte Explored bool // 1 byte } {{< /code >}} This struct packs perfectly into **4 bytes**. Do the math: A 120x30 map contains 3,600 tiles. At 4 bytes per tile, the entire map of the _Derelict Facility_ fits into exactly **14.4 KB** of RAM. A modern CPU L1 data cache is typically 32 KB or 64 KB. This means the engine can load the _entire physical game world_ into the absolute fastest layer of CPU memory simultaneously. ## The Trade-offs Is a contiguous 1D array always the right answer? No. 1. **Immutability of Size:** A flat slice is brilliant for a fixed-size grid (like a game map or an image matrix). If your map needs to dynamically grow or shrink in unpredictable directions during runtime, managing a massive contiguous reallocation is expensive. You would need to implement a chunking system (like Minecraft). 2. **Cognitive Overhead:** `Grid[y][x]` is undeniably easier to read and write than `m.Tiles[x + y*m.Width]`. You must abstract the math behind reliable helper functions (like `GetTile`) to prevent developers from messing up the index calculations. Before you write a single line of business logic or rendering code, your data layout is already dictating your system's performance ceiling. By flattening our 2D structures, leaning on simple mathematical strides, and ruthlessly stripping our structs of heap-allocated pointers, we stop fighting the Go garbage collector and start working in harmony with the CPU cache. In systems engineering, mechanical sympathy is everything. ## Further Reading If you want to dive deeper into Data-Oriented Design and Engine Architecture, these are the foundational resources that shaped my approach to _Derelict Facility_: **Books:** - _"Game Engine Architecture"_ by Jason Gregory - _"Game Programming Patterns"_ by Robert Nystrom (specifically the [Data Locality](https://gameprogrammingpatterns.com/data-locality.html) chapter) - _"Data-Oriented Design"_ by Richard Fabian **Lectures & Talks:** - _"Data-Oriented Design and C++"_ by Mike Acton (GDC 2014) - _"A Practical Guide to Applying Data-Oriented Design"_ by Andrew Kelley **Online Resources:** - [The `dbartolini/data-oriented-design` GitHub Repository](https://github.com/dbartolini/data-oriented-design) - Unity DOTS (Data-Oriented Technology Stack) Documentation --- _This is Part 1 of the [Building a Game Engine in Pure Go](/series/building-a-game-engine-in-pure-go/) series. The full source code for Derelict Facility is available on [GitHub](https://github.com/vikash-paf/derelict-facility)._ --- ## 10 years of lorbic.com architecture **Date:** 2026-04-02 **URL:** https://lorbic.com/10-years-of-lorbic.com-architecture/ **Description:** Reviewing the technical evolution of this blog over a decade: from managed platforms and dynamic backends to a custom, zero-dependency Hugo architecture. ![Lorbic Evolution: 2017 vs 2026](evolution.webp) **TL;DR:** This post tracks four architectural rewrites of my personal site over ten years. I cover the move from managed platforms (Blogger) to dynamic backends (Django), and finally to static generators (Jekyll/Hugo). Beginning in 2025, I stripped away all external themes and heavy frameworks to build a zero-dependency, vanilla-web architecture from the ground up for absolute control and performance. --- ## Technical Context For an engineer, a personal site is the only project where you have absolute authority over the stack. There are no product managers, no legacy constraints, and no enterprise technical debt: you own the metal. Over the last decade, my approach to this site has evolved from simple content publishing to a pursuit of complete technical control. This is the timeline of that evolution. ## 2015–2018: Managed Platforms (WordPress & Blogger) **Stack:** WordPress.com -> Blogger.com **Links:** [villageprogrammer.blogspot.com](https://villageprogrammer.blogspot.com) The initial goal was low-friction publishing. I started on WordPress in 2015 because it was the default, then moved to Blogger in late 2017 to create "Village Programmer" and get slightly better access to the underlying HTML/XML templates. ![The 2017 villageprogrammer.blogspot.com design](2017.webp) **Technical Trade-off:** Control was completely sacrificed for convenience. The rendering pipeline was a black box. The UI was composed of unmanaged CSS and proprietary widgets injected by the platform. Portability was non-existent; the content was locked in a managed ecosystem. I could write, but I couldn't engineer. ## 2018–2020: Dynamic Architecture (Django & PostgreSQL) **Stack:** Django + PostgreSQL + Nginx **Links:** Self-hosted Linux instance By college, I had learned MVC patterns and relational databases. Naturally, I assumed my blog needed a backend. I migrated everything to a custom Django application. **The Trade-off:** Operational overhead was the primary failure point. Managing a relational database and a Python runtime for a read-heavy, low-write blog is an architectural mismatch. The maintenance burden, including security patches, database backups, and server monitoring, became a distraction from actually writing or building new features. I learned that "Full-Stack" often just means "more things to break". ## 2020–2024: Static Site Generators (Jekyll & Hugo) **Stack:** Jekyll -> Hugo (Even Theme) **Links:** [vk4s.github.io](https://vk4s.github.io) I eventually recognized that a blog is an immutable dataset that should be pre-compiled. I migrated to the Jamstack: first using Jekyll on GitHub Pages, then porting to Hugo for sub-second build times. My content was finally decoupled from the infrastructure, living purely as Markdown in a Git repository. ![The 2021 vk4s.github.io static site using the Even theme](2021.webp) **The Trade-off:** Performance and portability were solved, but I was still using an off-the-shelf theme ("Even"). Modifying the site's behavior meant fighting against foreign CSS conventions and non-essential JavaScript dependencies. It was fast, but it wasn't mine. ## 2025–2026: Custom Architecture (Zero Dependencies) **Stack:** Hugo + Vanilla Web Technologies **Links:** [lorbic.com](https://lorbic.com) (Engineering Blog) / [vikashpatel.net](https://vikashpatel.net) (Portfolio) The current iteration of `lorbic.com` is a complete rewrite. Beginning in early 2025, I started stripping away the layers of accumulated technical debt: removing heavy CSS frameworks like Bootstrap, and deleting bloated JavaScript dependencies like jQuery and Zoom JS. The mandate was strict: zero external dependencies, zero CSS frameworks, and maximum performance through native primitives. I separated my portfolio (`vikashpatel.net`) from my engineering writing (`lorbic.com`). ![The current 2026 Lorbic dark-mode architecture](2026.webp) ### Engineered Features Instead of importing "black box" plugins, I implemented focused, vanilla solutions for specific needs. **1. Command Palette (Terminal-inspired UI)** Navigation is handled via a native terminal overlay (`Cmd+K`). It utilizes a pre-compiled JSON search index and a custom routing engine. This allows for instant theme toggling and site-wide search without the overhead of heavy third-party search libraries. ![The Cmd+K Command Palette](command-pallet.webp) **2. Contextual Selection Tooltip** Standard anchor links are static. I engineered a tooltip that hooks into the browser's Selection API. Highlighting text creates a [Text Fragment](https://developer.mozilla.org/en-US/docs/Web/Text_fragments) URL (`#:~:text=...`), allowing readers to share exact paragraph references. ![Deep Linking Context Menu](select-share-context-menu.webp) **3. Interactive Data Tables** All Markdown tables are wrapped in a custom JS observer. This adds an interactive action bar to every dataset, enabling one-click "Copy to Markdown" and "Download as CSV" functionality. It relies purely on the DOM; zero external CSV-parsing libraries are required. **4. Performance Telemetry HUD** To maintain performance hygiene, a real-time HUD tracks metrics via the `Navigation Timing API`. It displays DOM load times (typically < 40ms) and network payload sizes directly in the viewport. ![Live Performance Telemetry HUD](live-perf.webp) ## The Backstage Workflow While the site is static at the edge, the management layer is automated. I manage the publication pipeline using custom Python tooling via Poetry. Scripts residing in the `scripts/` directory handle automated backups of my self-hosted Listmonk newsletter instance and schedule content distributions. The deployment pipeline is a standard Git-based CI/CD flow via GitHub Actions, which builds the static site via Hugo and pushes the optimized assets to the CDN. --- Building your own blog engine is not about necessity. It's about forcing yourself to understand the boundaries of the systems you use. By stripping away managed platforms and heavy themes, I had to solve core problems in CSS containment, the critical rendering path, and DOM performance manually. The result is an infrastructure that behaves exactly as designed, served as fast as physics allows. ![Terminal 404 Page Design](not-found.webp) --- ## My Reading List **Date:** 2026-04-02 **URL:** https://lorbic.com/reading-list/ **Description:** A living collection of the best engineering blogs, essays, and deep dives I've read and want to remember. {{< resource_archive >}} --- ## Kubernetes on WSL2 and the macOS tunnel **Date:** 2026-04-01 **URL:** https://lorbic.com/kubernetes-on-wsl2-macos-access/ **Description:** A pragmatic guide to running k3s inside WSL2 on a Windows gaming PC and accessing it securely from a MacBook. No LAN exposure, no fragile port proxies: just SSH and control-plane tunneling. **TL;DR:** I run **k3s** on a Windows gaming PC via **WSL2** to master Kubernetes networking without cloud costs. This guide details a secure access model using **SSH** and **kubectl port-forward** to bypass NAT boundaries and maintain technical sovereignty. --- This guide omits Kubernetes feature tutorials. Instead, it details how to architect an environment where Kubernetes can exist without auxiliary hardware, idle cloud bills, or hidden networking realities. I wanted to learn Kubernetes properly, which meant understanding how **scheduling**, **access paths**, and **network boundaries** behave under real constraints. The problem was simple: I lacked a reasonable execution environment. ## Why my cloud server is full I maintain a crowded cloud server running a **Grafana observability stack**, **ClickHouse**, and **Couchbase**. While this setup is robust, the machine is effectively full. Mixing orchestration models on a single host would create unnecessary **debugging overhead** and risk the stability of existing services. Consequently, adding Kubernetes there would violate my goal of learning the system in isolation. ## Why a dedicated server was overkill I evaluated several alternatives, but most provided poor value. A **Mac Mini** or **Intel NUC** is expensive for the performance offered, while **Raspberry Pi clusters** are underpowered and diverge too far from production hardware. Furthermore, used datacenter servers are impractical due to noise and power consumption. My gaming PC, however, has a modern CPU, high-density RAM, and fast NVMe storage. Although it runs Windows, its raw capacity makes it an ideal Kubernetes host. ## Why WSL2 was the least-bad option On Windows, the choice is between a full **Linux VM** or **WSL2**. I chose WSL2 because a full VM adds a redundant virtualization layer and higher overhead, whereas WSL2 provides a **real Linux kernel** with near‑native performance. Crucially, WSL2 is a Linux VM with a specific networking model. This detail is critical for cross-device access. {{< details title="The 9P Boundary" label="Deep Dive //" >}} As I wrote in [The WSL2 performance tax](/posts/wsl2-performance-tax-go-windows/), the **9P protocol** bridges the Linux guest and Windows host. While it is efficient for file I/O, it does not resolve the **NAT boundary**. Because WSL2 sits in its own virtual network, its IP is not reachable from the LAN. {{< /details >}} ## Why k3s was the correct engine I avoided **Docker Desktop**, **Minikube**, and **MicroK8s** because these tools often hide technical complexity or add unnecessary layers. Instead, I chose **k3s** because it is **CNCF‑certified**, uses upstream components, and runs efficiently in constrained environments. It behaves like a production cluster without the resource tax. ## Installing k3s in a single command Installing k3s inside WSL2 is trivial: ```bash curl -sfL https://get.k3s.io | sh - ``` This command starts the **control plane** and **kubelet**, then generates a valid kubeconfig at `/etc/rancher/k3s/k3s.yaml`. ## WSL2 is trapped behind Windows NAT My primary machine is a **MacBook Air**. It is where I write manifests and run `kubectl`, yet the cluster lives inside WSL2, which sits behind Windows NAT. This architecture creates three friction points: 1. **WSL2 IPs** are not reachable from the LAN. 2. **NodePorts** fail to resolve because the node is not LAN-addressable. 3. **Firewall rules** on Windows manage permission but do not establish routing into the WSL2 subsystem. ## Tunneling through the control plane Rather than fighting WSL2 networking, I settled on a model that mirrors private production clusters. Nothing is exposed to the LAN; instead, we secure the **Kubernetes API** and tunnel application traffic through the control plane using **SSH** and `kubectl port-forward`. ## Setting up the Windows SSH bridge The first step is enabling **OpenSSH Server** on Windows. Run these commands in PowerShell as Administrator: ```powershell Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0 Start-Service sshd Set-Service sshd -StartupType Automatic ``` Then, add a firewall rule to restrict SSH to **Private** networks: ```powershell New-NetFirewallRule \ -Name "OpenSSH-Server-In-TCP" \ -DisplayName "OpenSSH Server (Private)" \ -Enabled True \ -Direction Inbound \ -Protocol TCP \ -Action Allow \ -LocalPort 22 \ -Profile Private ``` At this stage, the only exposed port is **22**. Kubernetes remains private. ## Mapping the cluster to macOS From WSL2, retrieve the kubeconfig: `sudo cat /etc/rancher/k3s/k3s.yaml`. Copy this file to the Mac at `~/.kube/config`, then update the **server address** to use localhost: ```yaml server: https://127.0.0.1:6443 ``` Keep all other certificates and keys unchanged. ## Securing the API endpoint From macOS, create an **SSH tunnel**: ```bash ssh -N -L 6443:127.0.0.1:6443 @ ``` This forwards the **Kubernetes API** securely from WSL2 to the Mac's localhost. With the tunnel active, `kubectl get nodes` and `kubectl get pods -A` function normally. ## Why NodePorts fail on WSL2 **NodePort** services assume nodes have routable IPs reachable by clients, but WSL2 violates this assumption. The node lives behind NAT, and Windows does not forward arbitrary ports into the Linux guest. Opening firewall ports only grants **permission**; it does not establish **routing**. This behavior is not a Kubernetes bug, but rather a mismatch between environment reality and service expectations. ## Forwarding application traffic Instead of NodePorts, use `kubectl port-forward` for service access: ```bash kubectl port-forward svc/app1 8081:8081 ``` The service is now available at `http://localhost:8081` on the Mac. **Important:** Even if the Service defines `nodePort: 30081`, `port-forward` ignores it because it tunnels directly to the **Service port** and the **Pod targetPort**. This is why attempting to forward 30081 fails. ## How traffic flows through the tunnel When `port-forward` is active, traffic flows from the Mac to `kubectl`, through the **Kubernetes API** over **mTLS**, to the `kubelet`, and finally to the Pod port. This is an **application-layer tunnel**, meaning no node-level networking is involved. ## Why this setup is actually secure The Kubernetes API is not exposed to the LAN. **SSH** is the only entry point, and it can be restricted to a single client IP. Because Kubernetes authentication remains the primary gatekeeper and application access is bound strictly to localhost, this is more secure than exposing NodePorts or using fragile Windows port‑proxy rules. ## The maintenance tax Every architectural decision has a cost: * **Persistent Tunneling:** You must maintain an active SSH session to interact with the cluster. * **DNS Resolution:** Localhost access bypasses standard Kubernetes DNS, meaning you cannot use internal cluster URLs from the Mac browser. * **Gaming PC Availability:** The cluster is only reachable when the host PC is awake and the WSL2 subsystem is running. ## Kubernetes is a system of constraints This setup allowed me to learn Kubernetes without auxiliary hardware or cloud overhead. It forced me to understand **networking boundaries** instead of bypassing them with convenience tools. Ultimately, it made Kubernetes less like "Docker with YAML" and more like the **distributed system** it actually is. --- {{< newsletter >}} --- ## ClickHouse data masking with regex **Date:** 2026-03-31 **URL:** https://lorbic.com/clickhouse-regex-data-masking/ **Description:** Ingesting PII is easy; cleaning it up is hard. Here is how to use ClickHouse's data masking policies and regex to redact sensitive logs in flight without killing your query performance. If you're running a production observability stack, you've already leaked PII. An engineer forgot to redact an email in a log line, or a JWT token ended up in a stack trace. In most databases, the only fix is to `DELETE` the data—killing your metrics along with the sensitive info. But ClickHouse has a more elegant approach: **Data Masking Policies**. By defining a policy at the role level, you can use regex to swap sensitive patterns with `[REDACTED]` in real-time, before the query results even leave the server. --- {{< details title="How Regex Masking works" label="Glossary //" >}} ClickHouse uses the **re2** engine for regex. When a masking policy is applied, the query planner injects a `regexpReplace` function into the `SELECT` path. This means your raw data remains on disk untouched (preserving its checksums and history), but the "View" layer ensures unauthorized users only see the masked version. {{< /details >}} --- ## Defining the policy The core of the "Clean Room" strategy is identifying the patterns that shouldn't be there. Let's look at a policy for redacting emails and IPv4 addresses using `multiRegexpReplace`. {{< code lang="sql" title="Masking Policy with Regex" >}} -- 1. Create the policy CREATE MASKING POLICY email_masking ON logs.payload USING ( multiRegexpReplace( payload, ['[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}', '\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}'], ['[EMAIL_REDACTED]', '[IP_REDACTED]'] ) ); -- 2. Apply it to an analyst role GRANT SELECT ON logs.payload MASKING POLICY email_masking TO analyst_role; {{< /code >}} {{< alert type="important" >}} Notice the use of `multiRegexpReplace`. It is significantly faster than chaining multiple `regexpReplace` calls because it traverses the string once for all patterns, rather than rescanning the entire string for every individual regex. {{< /alert >}} --- ## The Throughput Penalty: A Billion Row Tax Regex is computationally expensive. When you apply a masking policy, you are effectively adding a "CPU tax" to every `SELECT` query. If your query is scanning a billion rows, the regex engine has to run a billion times. In a high-throughput OLAP system like ClickHouse, where we expect sub-second aggregations, a "busy" regex can turn a 100ms query into a 10-second wait. Here is how to optimize your patterns to keep your "beast" status. ### 1. Anchors are your friend If you know the sensitive data always appears after a specific key in a JSON log, anchor your regex. Without anchors, the engine has to attempt a match at every single character position in the string. {{< code lang="text" title="Regex Anchoring" >}} # BAD: Scans every character for a potential email [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,} # GOOD: Only looks for patterns following the "user_email" key (?i)"user_email"\s*:\s*"([^"]+)" {{< /code >}} By using the key as an anchor, the RE2 engine can quickly skip over irrelevant parts of the log line using fast substring matching (like Boyer-Moore) before even engaging the complex regex state machine. ### 2. Avoid back-tracking and "The Greediness Trap" ClickHouse uses **RE2**, which is designed to prevent "catastrophic backtracking" (a common vulnerability in other engines like PCRE). However, RE2 still pays a price for ambiguity. The most common performance killer is the `.*` operator when used inside a capture group. {{< code lang="text" title="Avoiding Ambiguity" >}} # DANGEROUS: Highly ambiguous, forces the engine to explore many paths ID:.*Email:.* # FAST: Uses negated character classes to limit the search space ID:[^,]+,Email:[^,\s]+ {{< /code >}} By being explicit about what *cannot* be in the match (e.g., `[^,]+` means "anything that isn't a comma"), you provide the engine with a clear "stop" signal, reducing the number of states the machine has to track. --- ## Benchmarking the Tax If you are deciding whether to mask at the database level or the application level, consider these numbers (improvised based on typical RE2 behavior): | Strategy | Rows/sec | Latency (p99) | | :--- | :--- | :--- | | Raw SELECT (No Masking) | 500M | 80ms | | Anchored Regex Masking | 120M | 450ms | | Unanchored/Naive Regex | 15M | 4200ms | **The takeaway:** A poorly written regex is 30x slower than an optimized one. If your masking policy is poorly designed, it doesn't just protect data; it effectively DOSes your own database. --- ## Closing thoughts Data masking isn't a replacement for proper sanitization at the edge. But in the real world, things leak. ClickHouse's regex masking gives you a safety net that protects your data without requiring a full database wipe. The goal of a senior engineer isn't just to "make it work", but to "make it work within the constraints of the hardware". Stay slow and steady. Secure your logs, but don't kill your throughput. --- {{< newsletter >}} --- ## Understanding CPU Caches in Go **Date:** 2026-03-31 **URL:** https://lorbic.com/cpu-caches-in-go/ **Description:** A practical guide to understanding how CPU caches (L1/L2/L3) impact Go service performance, with benchmarks on modern hardware. When you're building Go services that handle millions of operations per second, the hardware beneath your abstractions starts to matter. Specifically, the CPU cache hierarchy, and whether your data fits in it. --- #### The Hardware Context: It's Not Just "RAM" Your server has 32GB or 64GB of RAM, but the CPU avoids touching it whenever possible. Instead, it works through a chain of caches: - **L1 Cache (Instruction & Data)**: Tiny (~64KB data per core), near-instant access (1–4 cycles). - **L2 Cache**: Larger (1–4MB per core), still extremely fast, your true private workspace. - **L3 / Last Level Cache (LLC)**: Shared across cores, much larger (32MB+), but noticeably slower. On modern ARM server chips (Ampere Altra, AWS Graviton) and Apple M-series silicon, **L2 is your most valuable per-core resource**. If your Go service's active working set fits in L2, it runs fast. Spill into L3 or RAM, and you hit a latency cliff. The practical question isn't "do you know what a cache is?" It's: **what does modern hardware actually penalize you for?** --- #### What Modern Hardware Gets Right (And What It Doesn't) I ran benchmarks on an Apple M4 to find out. The results changed some assumptions. ##### Sequential vs. Stride Access: The Prefetcher Wins The classic cache benchmark is matrix traversal: iterating a 1000×1000 integer matrix by row (sequential, cache-friendly) vs. by column (jumping 8KB every step, cache-hostile). On older CPUs, column traversal is reliably ~10× slower. On the M4: ```text BenchmarkMatrixRowTraversal 236,982 ns/op BenchmarkMatrixColTraversal 239,203 ns/op ``` Identical. The hardware prefetcher detected the stride pattern and pre-fetched the next chunk before the CPU needed it, hiding the latency entirely. This holds for stride-walk benchmarks too, even 64KB jumps don't fool it: ```text BenchmarkStrideWalk/Stride_64 0.30 ns/op BenchmarkStrideWalk/Stride_65536 0.28 ns/op ``` **Takeaway**: Modern CPUs are remarkably good at hiding predictable access patterns. Simple, linear data structures (slices, arrays) are still best, but you probably don't need to hand-tune memory layout unless profiling shows a real bottleneck. If you want to benchmark true cache misses, use **pointer-chasing** (random linked-list traversal), where the CPU cannot predict the next address until it loads the current one. --- #### What Still Kills Performance: Shared Mutable State This is where the latency cliff is real, and where Go developers get burned. When multiple cores read and write the same memory location, even through atomic operations, they trigger the CPU's cache coherence protocol (MESI). Every write on one core must invalidate copies on every other core. The more cores fighting over the same cache line, the worse it gets. I measured this directly with adjacent atomic counters under concurrent load: | Scenario | Latency | |---|---| | Single core, no contention | ~2.2 ns/op | | 8 cores, high contention | ~28.4 ns/op | **A 12× slowdown**, not from algorithmic complexity, but from cores arguing over a 64-byte chunk of memory. This is the real enemy in high-throughput Go services, a shared `var RequestCount int64` or a single global mutex protecting a hot map. ##### Fix 1: Shard your counters Don't use a single shared counter. Map goroutines or connections to buckets: ```go type ShardedCounter struct { counts [128]struct { n int64 _ [56]byte // pad to 64-byte cache line } } func (c *ShardedCounter) Inc(shard int) { atomic.AddInt64(&c.counts[shard%128].n, 1) } ``` Each shard lives on its own cache line. Cores no longer fight. ##### Fix 2: Pad hot struct fields If two fields in a struct are written by different goroutines, put padding between them so they land on separate cache lines: ```go type HotStruct struct { FieldA int64 _ [56]byte // force onto separate cache line FieldB int64 } ``` Without the padding, `FieldA` and `FieldB` share a line. A write to one invalidates the other on every other core. ##### Fix 3: Prefer channels over shared memory Go's channel semantics transfer *ownership*, not just a value. The sending goroutine gives up its reference; the receiver takes over. This is exactly what cache coherence wants, one writer at a time, and it's why idiomatic Go with channels often outperforms lock-heavy designs even when the channel has overhead. --- #### Keep Your Working Set in L2 The 12× contention penalty is dramatic, but the more insidious problem in real services is simply having a working set that's too large. If a request handler touches 10MB of data, a large in-process cache, a bloated context struct, a deep call chain with big allocations, that data spills out of L2 and into L3 or RAM on every request. You won't see a single expensive operation; you'll see uniformly elevated tail latency under load. Practical rules: - **Keep per-connection and per-request structs small.** Prefer slices of IDs over slices of full objects when you don't need all the fields. - **Avoid large in-process caches for hot-path data.** A 50MB LRU cache sounds helpful, but if it exceeds your LLC, you're paying RAM latency for every miss. - **Use `sync.Pool` for short-lived allocations** to reduce GC pressure and improve locality, reused objects tend to stay in cache. --- #### Measuring It Yourself To see cache effects in your own service, the most useful tools are: **`go test -bench`** for microbenchmarks. Run with `-cpu=1,4,8` to see how contention scales: ```bash go test -bench=. -benchmem -cpu=1,4,8 . ``` **Linux `perf`** for hardware counters on production workloads: ```bash perf stat -e cache-misses,cache-references,LLC-load-misses ./your-binary ``` High LLC (Last Level Cache) miss rates, especially above 5–10%, are the clearest signal that your working set is exceeding the cache. That's where to optimize first. I've added a benchmark suite to experiment with these patterns yourself. {{< download file="cache_test.go" label="Download cache_test.go" >}} --- #### Summary Modern hardware prefetchers have largely eliminated the penalties for predictable access patterns. The naive "row vs. column traversal" benchmarks from ten years ago often don't apply anymore. What *does* still matter, and significantly: 1. **Shared mutable state across cores**, cache coherence contention is a 10× problem, not a 2× problem. Shard counters, pad structs, prefer channels. 2. **Working set size**, if your per-request data footprint exceeds L2 (~1–4MB per core), you're paying LLC or RAM latency on every operation. Keep hot structures lean. 3. **Measure before optimizing**, use `perf` to look for LLC misses. A high miss rate is the smoking gun; a low one means your bottleneck is somewhere else entirely. --- ## ClickHouse vs. Postgres: When to Move Your Logs Out of a Relational DB **Date:** 2026-03-30 **URL:** https://lorbic.com/clickhouse-vs-postgres-log-storage/ **Description:** Postgres is the swiss-army knife of databases, but when you hit the 10-million row mark for write-heavy logs, the relational wall becomes real. Here is why we moved our observability stack to ClickHouse. Postgres is the most reliable tool in my stack. It handles users, configurations, and complex relations without breaking a sweat. But databases, like any physical system, have a "design limit". For Postgres, that limit usually appears when you try to use it as a dumping ground for high-velocity logs and metrics. When your `ANALYZE` commands start taking minutes and your indexes consume more RAM than your data, you aren't facing a Postgres bug; you're facing an architectural mismatch. ## The Write-Heavy Wall Relational databases are designed for **OLTP** (Online Transactional Processing). They prioritize consistency, ACID compliance, and point-lookups. Logs, however, are **OLAP** (Online Analytical Processing) data. They are immutable, sequential, and usually queried in broad sweeps rather than by individual IDs. In Postgres, every new log entry must be written to the Write-Ahead Log (WAL) and then inserted into a B-Tree index. As the table grows, the cost of maintaining those B-Trees during high-concurrency writes creates a "latency floor" that you simply cannot drop below. {{< details title="B-Tree vs. LSM-Tree" label="Deep Dive //" >}} **B-Trees (Postgres):** Optimized for reading small amounts of data quickly. Every write requires finding the correct spot in a tree, which can lead to random disk I/O as the tree grows larger than memory. **LSM-Trees (ClickHouse):** Optimized for high-volume writes. It appends data to sorted "parts" in memory and flushes them to disk sequentially. Background "merges" combine these parts later. This turns random I/O into sequential I/O, which is where modern SSDs (and even HDDs) truly shine. {{< /details >}} ### The "Write Amplification" Problem In Postgres, if you have a table with 5 indexes, every single `INSERT` must update those 5 separate B-Trees. This is **Write Amplification**. For a log-heavy system processing 5,000 events/sec, your disk is working 5x harder just to keep the indexes "ready", even if you only query those logs once a week. --- ## Why ClickHouse? Moving logs to ClickHouse isn't just about "faster writes". It's about changing how you think about data storage. ### 1. Columnar Compression In Postgres, data is stored in rows. If you have 50 columns but only query 2 (`timestamp` and `error_code`), Postgres still has to read the whole row from disk. ClickHouse stores each column in its own file. {{< alert type="tip" >}} Because similar data is stored together in columns (e.g., thousands of "200 OK" strings), ClickHouse can use specialized algorithms like **ZSTD** or **Delta encoding** to achieve 10x to 100x compression ratios. {{< /alert >}} ### 2. The Power of Materialized Views One of ClickHouse's "superpowers" is the ability to aggregate data _on ingestion_. Instead of running a heavy `COUNT(*)` query every time you refresh a dashboard, ClickHouse can maintain a running total in a background table as the data arrives using the `SummingMergeTree` or `AggregatingMergeTree` engines. {{< code lang="sql" title="Example: Continuous Aggregation" >}} CREATE MATERIALIZED VIEW hourly_errors_mv ENGINE = SummingMergeTree AS SELECT toStartOfHour(timestamp) as hour, count() as error_count FROM logs WHERE status >= 400 GROUP BY hour; {{< /code >}} --- I don't want to make this sound like a free lunch. Moving to ClickHouse introduces "Operational Tax": 1. **No Real Updates:** ClickHouse is for immutable data. `UPDATE` and `DELETE` commands exist (as `ALTER TABLE ... UPDATE`), but they are heavy, asynchronous background operations. If you need to frequently edit your data, stay with Postgres. 2. **Join Performance:** While ClickHouse _can_ do joins, it isn't its primary strength. It prefers large, "denormalized" tables. 3. **Consistency Models:** Postgres gives you strict ACID. ClickHouse is "eventually consistent" during background merges. For logs, this is fine. For a bank balance, it is a disaster. If your logs table is under 5 million rows and your write rate is low, **stay with Postgres**. The operational simplicity of having one database outweighs the performance gains of ClickHouse. However, if you're seeing any of the following, it's time to move: - `VACUUM` processes are struggling to keep up with dead tuples (bloat). - Your log indexes are larger than the actual log data. - Simple aggregations (like "average response time per hour") are timing out. - You are spending more on RDS storage than on actual compute. The goal is to use the right tool for the job. Postgres remains my "Source of Truth" for application state: users, sessions, and relationships. But for the firehose of observability data, ClickHouse has reclaimed our performance headroom and slashed our storage bills. --- {{< newsletter >}} --- ## The WSL2 Performance Tax: Why Your Go Apps Are Slow on Windows **Date:** 2026-03-29 **URL:** https://lorbic.com/wsl2-performance-tax-go-windows/ **Description:** If you're building Go applications on WSL2 and keeping your source code on the Windows filesystem, you're paying a hidden performance tax on every build. Here is how to reclaim your CPU cycles. WSL2 (Windows Subsystem for Linux) has been a godsend for developers who love Linux tools but need/have a Windows environment. But for Go developers, WSL2 isn't just a "transparent layer". If configured incorrectly, it becomes the bottleneck that can slow down builds by 3x and introduce mysterious latency in networked services. This isn't a failure of WSL2; it's a failure of understanding the **9p boundary**. {{< details title="What is the 9P Boundary?" label="Glossary //" >}} WSL2 is a Virtual Machine. To let that VM see your Windows files, Microsoft uses the **9P protocol** (Plan 9). When you access `/mnt/c/`, you aren't talking to the disk directly; you're acting as a network client. Every file read requires a round-trip across the Hyper-V socket to the Windows host. For Go, which reads thousands of small files during compilation, this latency is catastrophic. {{< /details >}} ## The Performance Cliff: /mnt/c/ The most common mistake I see is developers (including me) keeping their Go projects in `C:\Users\Name\Projects` and accessing them via `/mnt/c/Users/...` inside WSL. When you run `go build` on a project located in the Windows filesystem: 1. Go makes thousands of syscalls to read source files. 2. WSL2 must translate these Linux syscalls into Windows I/O via the **9p protocol**. 3. Each translation adds microseconds of latency. For a project with 50 dependencies, those microseconds compound into seconds. ### The Benchmarkc | Filesystem Location | `go build` Time | I/O Throughput | | :------------------ | :-------------- | :------------- | | Windows (/mnt/c/) | 12.4s | ~40 MB/s | | Linux Native (~) | 3.2s | ~800 MB/s | **The takeaway:** You are paying a 4x build-time tax just for the convenience of using Windows Explorer. --- ## Tweak 1: Move to the Native Filesystem The solution is simple but often ignored: **Clone your code into the Linux home directory (`~/`)**. {{< terminal >}} # BAD cd /mnt/c/Users/vikash/projects/myapp # GOOD cd ~/projects/myapp {{< /terminal >}} If you need to access these files from Windows, use the built-in WSL share: `\\wsl$\Ubuntu\home\user\projects`. ## Tweak 2: The .wslconfig Memory Buffer By default, WSL2 can be greedy with memory but slow to release it. For Go's memory-intensive compilation, you should constrain WSL to prevent Windows from swapping to disk. Create or edit `C:\Users\\.wslconfig`: {{< code lang="ini" file=".wslconfig" >}} [wsl2] memory=8GB # Limits VM memory to 8GB processors=4 # Limits VM to 4 cores {{< /code >}} ## Tweak 3: Windows Defender Exclusion Windows Defender scans every file access. When Go builds, it creates thousands of small temporary files in `/tmp`. Defender sees this as suspicious activity and hooks into every process. **The Fix:** Add an exclusion in Windows Security for the `vmmem` process and your WSL storage location (usually `%USERPROFILE%\AppData\Local\Packages\...\LocalState\ext4.vhdx`). --- ## Conclusion WSL2 is a high-performance environment, but it respects the laws of physics. If you force it to cross the boundary between two disparate filesystems on every build, it will slow down. Move your code to Linux, tune your config, and get back to writing Go. --- {{< newsletter >}} --- ## AI Coding in 2026: Productivity Multipliers vs. Skill Replacements **Date:** 2026-03-28 **URL:** https://lorbic.com/ai-coding-productivity-vs-skill/ **Description:** Two years of coding with AI every day: what actually saves time, what creates hidden debt, and why your fundamentals are your only real defense against hallucinated correctness. "Write a REST API with authentication". I hit enter. Thirty seconds later, Claude spat out 400 lines of perfectly formatted Go code. JWT middleware, password hashing, error handling, even rate limiting. It looked professional. It looked production-ready. It took me three hours to figure out why the token refresh logic had a race condition. This is AI-assisted coding in 2026. It's not magic. It's not a replacement. It's a very fast junior developer that never gets tired, never complains, and confidently makes mistakes you won't catch unless you actually understand the **architectural nuances** of what you're building. I've spent two years using AI coding tools daily. From Aider and Antigravity to Claude Code. I've used them for everything from refactoring legacy monoliths to building new microservices from scratch. I've saved hundreds of hours, and I've lost dozens more debugging "hallucinated correctness". This isn't a tutorial on transformers. This is a forensic report on what actually works, what fails, and how to use these tools without increasing the **cognitive entropy** of your codebase. {{< details title="Probabilistic vs. Deterministic" label="Glossary //" >}} **Probabilistic (AI):** LLMs predict the "most likely" next token based on statistical patterns. They don't "know" if code works; they know it "looks like" code that works. **Deterministic (Engineering):** Compilers and tests follow strict logical rules. A program either satisfies its constraints or it doesn't. Engineering with AI is the art of using a probabilistic tool to generate deterministic results. {{< /details >}} --- ## The mirage of understanding Before we talk about tools, we must address the fundamental truth: **AI coding tools lack grounded understanding.** They are sophisticated pattern-matching engines. They excel at well-trodden paths (CRUD apps, standard algorithms) but struggle with novelty or deep domain logic. When you ask an AI to "write a function that does X", you are essentially performing a high-speed search across billions of lines of code. **The Amazing:** Need boilerplate database migrations? Test scaffolding? API endpoints that follow standard patterns? AI crushes these. It has seen the "correct" way ten thousand times. **The Dangerous:** Need to debug a distributed race condition? Optimize a slow query in a complex join? AI is mediocre at best. It often suggests "plausible but wrong" solutions that require a senior engineer to debunk. {{< alert type="important" >}} **The "Autocomplete" Fallacy:** I stopped thinking of AI as "intelligence" and started thinking of it as "autocomplete on steroids". It suggests the most likely implementation, not the most correct one. {{< /alert >}} --- ## The forensic workflow I've tried most major tools. Here is the stack that actually survived the transition from hype to production. ### The Stack - **Aider + Claude Opus 4.5 API:** My primary tool for surgical edits. It is git-aware and allows me to control the context window manually. - **Gemini CLI:** For high-speed reasoning and massive context handling when bridging multiple architectural layers. - **Cline:** For end-to-end task delegation and autonomous research. - **Antigravity:** For exploratory work where I need the agent to find relevant files itself. - **Requirement.md:** I write down the architecture, goals, and constraints in a Markdown file *before* prompting. This forces me to solve the problem in my head first. ### The Process: A Real-World Example **Task:** Add rate limiting middleware to a Go service. 1. **Specification:** I write a `ratelimit.md` defining the 100 req/min limit, the 429 response, and the Redis backend. 2. **Context Priming:** I add existing middleware to the chat so the AI learns our specific error handling patterns. 3. **Generation:** Aider produces the code in ~45 seconds. 4. **Forensic Review:** I read every single line. I'm not looking for syntax errors (the compiler finds those); I'm looking for **logical drift**. {{< alert type="tip" >}} **The Review Ratio:** AI didn't save me 5 hours. It saved me 4 hours of typing and cost me 1 hour of intense review. The net win is 3 hours, but only if that 1 hour of review is higher quality than the original typing. {{< /alert >}} --- ## Where AI nails it Two years in, I've mapped the "Safe Zones" for AI assistance: 1. **Boilerplate & Patterns:** "Generate CRUD handlers for this model". (95% accuracy) 2. **Code Translation:** "Convert this Python utility to idiomatic Go". (80% accuracy) 3. **Test Scaffolding:** "Generate table-driven tests for this parser". (90% accuracy) 4. **Documentation:** "Add docstrings explaining the concurrency model here". (100% accuracy for description) {{< details title="The Context Boundary" label="Deep Dive //" >}} In physics, the **9P protocol** (used by WSL2) adds a latency tax when crossing the Windows/Linux boundary. In AI development, there is a similar **Context Tax**. The larger your codebase, the more "noise" enters the context window. High-performance AI coding requires **Surgical Context Management**: only showing the AI the exact files it needs to solve the immediate problem. {{< /details >}} --- ## The hallucination minefield This is where the "Senior" in Senior Engineer becomes non-negotiable. ### 1. Confident Misinformation AI confidently invents APIs that don't exist. ```go // AI suggested this for a Redis client: client.IncrementWithExpiry("key", 60) // Reality: This method doesn't exist in 'go-redis'. // You need an atomic Lua script or multi-exec. ``` ### 2. Concurrency Blindness AI writes code that works in a single-threaded test but fails in a high-concurrency production environment. It often "forgets" that maps aren't thread-safe or that `atomic.Add` is required for global counters. {{< alert type="warning" >}} **The "Looks Good, Doesn't Work" Bug:** Syntactically perfect code with logical flaws is the hardest bug to find. AI specializes in this. {{< /alert >}} --- ## Skills that matter now The developer skill set has shifted. If you memorize syntax, you are competing with a machine that has already won. If you understand **Systems Thinking**, you are the machine's pilot. ### What Matters More (Way More) 1. **Code Reading:** You are now a professional reviewer. Can you spot a race condition in 500 lines of generated code? 2. **System Design:** AI can implement a feature, but it can't decide if you need a microservice or a monolith. 3. **Deterministic Validation:** Writing benchmarks and stress tests to prove the AI's probabilistic output is correct. 4. **Precision Communication:** The quality of your output is a direct function of the precision of your prompt. --- ## The force multiplier AI is not replacing developers. Not at least for half a decade, but it is raising the bar for what a developer is expected to deliver. If you understand the underlying mechanics, including memory management, the cache hierarchy, and database trade-offs, AI becomes a high-speed tool that lets you build faster than ever. If you use it to skip learning those things, you are building a house on shifting sand. **The Rule:** Use AI cautiously, verify everything, own what you ship, and never stop learning the fundamentals. --- {{< newsletter >}} --- ## Stop Rebuilding Docker Images on Every Code Change **Date:** 2026-02-15 **URL:** https://lorbic.com/stop-rebuilding-docker-images-runtime-containers/ **Description:** Waiting 2 minutes to see a typo fix? Runtime containers separate your code from your runtime, enabling instant feedback without sacrificing containerized consistency. Mount your code instead of baking it in. Edit files, see changes instantly. No rebuilds, no waiting. I used to rebuild my Docker image every time I fixed a typo. A one-line change meant waiting 2-3 minutes for Docker to rebuild layers, reinstall dependencies, and restart the container. I thought this was just the cost of containerized development. Then I discovered runtime containers. Same isolated environment, same reproducibility, but code changes reflect instantly. No rebuilds. No waiting. The runtime lives in the container, the code lives on the host. This pattern isn't new, but it's criminally underused. Most developers either suffer through slow rebuild cycles or abandon containers for local development entirely. Both approaches miss the point: you can have containerized consistency AND development speed. This guide covers the what, why, and how of runtime containers. When they're perfect, when they're terrible, and how to build production-quality containerized development workflows. ## Why most containerized development is painfully slow Consider a typical Python web application. Your Dockerfile looks like this: {{< code lang="dockerfile" file="Dockerfile" >}} FROM python:3.11-slim WORKDIR /app # Install dependencies COPY requirements.txt . RUN pip install -r requirements.txt # Copy application code COPY . . # Run the application CMD ["python", "app.py"] {{< /code >}} This works. But every code change requires: 1. Rebuild the image (`docker build -t myapp .`) 2. Stop the old container 3. Start a new container from the new image 4. Wait for the application to initialize Even with layer caching, this takes 30 seconds to 2 minutes. Fix a typo, wait 90 seconds. Adjust a log statement, wait 90 seconds. You're spending more time waiting than coding. The standard "solution" is to use bind mounts in development: {{< code lang="bash" >}} docker run -v $(pwd):/app myapp {{< /code >}} But developers rarely do this correctly. They mount code over an image that already has code baked in. The image is still bloated with stale code. Layer caching is still broken by code changes. The workflow is still hybrid and confusing. {{< alert type="important" >}} Runtime containers solve this properly. Build an image with _only_ the runtime and dependencies. Mount your code at runtime. Separate concerns cleanly. {{< /alert >}} ## What is a runtime container? A runtime container is an image that contains everything needed to execute your code, except the code itself. **A runtime container contains:** - The language runtime (Python interpreter, Node.js, JVM, etc.) - System dependencies (libraries, build tools) - Application dependencies (pip packages, npm modules) **A runtime container does NOT contain:** - Your application source code - Configuration files that change frequently - Data or state Your code lives on the host machine. You mount it into the container when you run it. ``` +----------------------------------------------------------+ | Docker Container | | | | +------------------+ +-----------------------+ | | | Runtime & Deps | +---> | Mounted: /app/src | | | | - Python 3.11 | | | (from host filesystem)| | | | - pip packages | | +-----------------------+ | | | - system libs | | | | +------------------+ | +-----------------------+ | | +---> | Mounted: /app/config | | | | (from host filesystem)| | | +-----------------------+ | +----------------------------------------------------------+ ^ | Code lives on host, mounted into container at runtime ``` Created with [asciiflow](https://asciiflow.com?ref=lorbic.com)
This architecture gives you: - **Instant code changes**: Edit files on your host, changes reflect immediately in the container - **Fast rebuilds**: Only rebuild the runtime image when dependencies change, not on every code change - **Familiar workflow**: Use your local IDE, debugger, and tools - **True isolation**: Runtime is containerized, ensuring consistency across team members ## Level 1: Your first runtime container Let's build a runtime container for a Python Flask application. **Traditional approach (what you're probably doing, what I was doing):** {{< code lang="dockerfile" file="Dockerfile" >}} FROM python:3.11-slim WORKDIR /app COPY requirements.txt . RUN pip install -r requirements.txt COPY . . # This bakes your code into the image CMD ["python", "app.py"] {{< /code >}} Every time you change `app.py`, you rebuild the entire image. **Runtime container approach:** {{< code lang="dockerfile" file="Dockerfile.runtime" >}} FROM python:3.11-slim WORKDIR /app # Install dependencies (this rarely changes) COPY requirements.txt . RUN pip install -r requirements.txt # DO NOT COPY SOURCE CODE # Code will be mounted at runtime CMD ["python", "src/app.py"] {{< /code >}} Notice what's missing? No `COPY . .`. Your code isn't baked in. Build the runtime image once: {{< code lang="bash" >}} docker build -f Dockerfile.runtime -t myapp-runtime . {{< /code >}} Run it with your code mounted: {{< code lang="bash" >}} docker run -v $(pwd)/src:/app/src myapp-runtime {{< /code >}} Now edit `src/app.py` on your host machine. If you're using a framework with auto-reload (Flask's debug mode, FastAPI with `--reload`), changes appear instantly. Zero rebuild time. ### Node.js example The same pattern works for any language runtime. {{< code lang="dockerfile" file="Dockerfile.runtime" >}} FROM node:18-slim WORKDIR /app # Install dependencies COPY package.json package-lock.json ./ RUN npm ci --only=production # DO NOT COPY SOURCE CODE CMD ["node", "src/server.js"] {{< /code >}} Run with mounted code and nodemon for hot-reloading: {{< code lang="bash" >}} docker run \ -v $(pwd)/src:/app/src \ -e NODE_ENV=development \ myapp-runtime \ npx nodemon src/server.js {{< /code >}} Edit `src/server.js`, nodemon detects the change, restarts the process. Entire cycle takes milliseconds. ## Level 2: Docker Compose workflows Managing mount paths and environment variables in `docker run` commands gets tedious with more than 2-3 containers. Docker Compose makes managing multiple containers and their configurations easy for development. {{< code lang="yaml" file="docker compose.yml" >}} version: '3.8' services: app: build: context: . dockerfile: Dockerfile.runtime volumes: # Mount source code - ./src:/app/src:ro - ./config:/app/config:ro environment: - FLASK_ENV=development - FLASK_DEBUG=1 ports: - "5000:5000" command: flask run --host=0.0.0.0 db: image: postgres:15-alpine environment: POSTGRES_PASSWORD: devpassword POSTGRES_DB: myapp volumes: - postgres_data:/var/lib/postgresql/data volumes: postgres_data: {{< /code >}} Start the entire stack: {{< code lang="bash" >}} docker compose up {{< /code >}} Your application runs in a container with the Python runtime and dependencies. Your database runs in its own container. Your code is mounted from the host. Edit code, refresh browser. That's it. ### Read-only mounts for safety Notice `:ro` on the volume mounts? This makes them read-only from inside the container. Your application can't accidentally modify source files. For development this isn't strictly necessary, but it's good practice. If your app needs to write files (logs, uploads, generated assets), mount those directories separately as read-write: {{< code lang="yaml" >}} volumes: - ./src:/app/src:ro - ./config:/app/config:ro - ./logs:/app/logs # Read-write for log output - ./uploads:/app/uploads # Read-write for user uploads {{< /code >}} ### Environment-specific configuration Development and production often need different settings. Use environment variables to control behavior: {{< code lang="yaml" file="docker compose.yml" >}} services: app: environment: - FLASK_ENV=development - DATABASE_URL=postgresql://postgres:devpassword@db:5432/myapp - LOG_LEVEL=DEBUG - ENABLE_DEBUGGER=1 {{< /code >}} Or load from an `.env` file: {{< code lang="yaml" >}} services: app: env_file: - .env.development {{< /code >}} ## Level 3: Advanced patterns ### Handling compiled dependencies Some languages need to compile dependencies (C extensions in Python, native modules in Node.js, cli-tools like ffmpeg). These dependencies are platform-specific. If you're on macOS but deploying to Linux, the compiled cli-tools won't work as is. E.g. on ubuntu 22.04, you can't use the latest ffmpeg (>v4) from apt. macOS brew has the latest version.. **Solution:** Compile dependencies inside the container, store them in a named volume. {{< code lang="dockerfile" file="Dockerfile.runtime" >}} FROM python:3.11-slim WORKDIR /app # Install system dependencies for compilation RUN apt-get update && apt-get install -y \ gcc \ && rm -rf /var/lib/apt/lists/\* COPY requirements.txt . RUN pip install -r requirements.txt CMD ["python", "src/app.py"] {{< /code >}} {{< code lang="yaml" file="docker compose.yml" >}} services: app: volumes: - ./src:/app/src:ro # Anonymous volume to prevent host node_modules from overriding container's - /app/node_modules {{< /code >}} The anonymous volume `/app/node_modules` ensures the container uses its own compiled modules, not your host's. ### Multi-stage builds for dev and production You can use a single Dockerfile for both development (runtime container) and production (baked-in code). {{< code lang="dockerfile" file="Dockerfile" >}} FROM python:3.11-slim AS base WORKDIR /app # Install dependencies COPY requirements.txt . RUN pip install -r requirements.txt # Development stage: no code, expects it to be mounted FROM base AS development CMD ["flask", "run", "--host=0.0.0.0", "--reload"] # Production stage: code baked in FROM base AS production COPY . . CMD ["gunicorn", "--bind", "0.0.0.0:8000", "app:app"] {{< /code >}} {{< code lang="yaml" file="docker compose.yml" >}} services: app: build: context: . target: development # Use the development stage volumes: - ./src:/app/src:ro {{< /code >}} Development: `docker compose up` (uses mounted code from development stage) Production: `docker build --target production -t myapp .` (bakes code into production stage) One Dockerfile, two workflows. Clean separation of concerns. ### Debugging inside containers Runtime containers make debugging easier because you're using your local IDE and tools. But sometimes you need to debug inside the container itself. **Python with debugpy:** {{< code lang="dockerfile" file="Dockerfile.runtime" >}} FROM python:3.11-slim WORKDIR /app COPY requirements.txt . RUN pip install -r requirements.txt # Install debugger in development RUN pip install debugpy CMD ["python", "-m", "debugpy", "--listen", "0.0.0.0:5678", "--wait-for-client", "src/app.py"] {{< /code >}} {{< code lang="yaml" file="docker compose.yml" >}} services: app: ports: - "5000:5000" - "5678:5678" # Debugger port {{< /code >}} Connect your IDE's debugger to `localhost:5678`. Set breakpoints in your local files. The debugger works because the code is mounted from the host. ### Watching for dependency changes You update `requirements.txt` but forget to rebuild the runtime image. Your app crashes with "ModuleNotFoundError". This is frustrating. **Solution 1: Document the workflow** Add clear instructions in your README: ```markdown ## Development Workflow - Edit code in `src/` -> changes reflect instantly - Change dependencies in `requirements.txt` -> run `docker compose build app` - Change system packages in Dockerfile -> run `docker compose build app` ``` **Solution 2: Automate with Make** {{< code lang="makefile" file="Makefile" >}} .PHONY: dev rebuild deps dev: docker compose up rebuild: docker compose build docker compose up deps: rebuild # Alias for clarity when you add dependencies {{< /code >}} Usage: - `make dev` - start development - `make deps` - rebuild and restart after changing dependencies **Solution 3: File watchers** Use a file watcher to automatically rebuild when dependency files change: {{< code lang="bash" file="watch-deps.sh" >}} #!/bin/bash # Requires entr: brew install entr (macOS) or apt-get install entr (Linux) ls requirements.txt package.json | entr -r docker compose build {{< /code >}} Run this in a separate terminal. When `requirements.txt` or `package.json` changes, it rebuilds automatically. ## Common mistakes and how to avoid them ### Mistake 1: Mounting too much Don't mount your entire project directory: {{< code lang="yaml" >}} # BAD: Mounts everything, including .git, node_modules, **pycache** volumes: - .:/app {{< /code >}} This clutters the container with development artifacts, breaks package management, and causes permission issues. **Solution:** Mount only what you need. {{< code lang="yaml" >}} # GOOD: Mount only source and config volumes: - ./src:/app/src:ro - ./config:/app/config:ro {{< /code >}} Add a `.dockerignore` file to prevent accidental copies during builds: {{< code lang="dockerignore" file=".dockerignore" >}} **/**pycache** **/_.pyc \*\*/.pytest_cache node_modules .git .env .vscode .idea _.log {{< /code >}} ### Mistake 2: Ignoring file permissions On Linux, files created by the container might be owned by `root`, making them uneditable on the host. **Solution:** Run the container as your user. {{< code lang="bash" >}} docker run -u $(id -u):$(id -g) -v $(pwd)/src:/app/src myapp {{< /code >}} Or set the user in `docker compose.yml`: {{< code lang="yaml" >}} services: app: user: "${UID:-1000}:${GID:-1000}" volumes: - ./src:/app/src {{< /code >}} ### Mistake 3: Forgetting to rebuild when dependencies change You add a new package to `requirements.txt` but forget to rebuild the runtime image. Your app crashes with "ModuleNotFoundError". This is the single most common mistake with runtime containers. The container has an old snapshot of your dependencies. **Solution:** Make rebuilds obvious. Use clear naming conventions, add checks, or automate. Add a check script that compares dependency files: {{< code lang="bash" file="check-deps.sh" >}} #!/bin/bash if [ requirements.txt -nt .docker-built ] || [ ! -f .docker-built ]; then echo "requirements.txt changed since last build" echo "Run: docker compose build" exit 1 fi {{< /code >}} Update your Makefile to track builds: {{< code lang="makefile" >}} .PHONY: dev rebuild dev: .docker-built docker compose up rebuild: docker compose build touch .docker-built docker compose up .docker-built: requirements.txt docker compose build touch .docker-built {{< /code >}} ### Mistake 4: Using runtime containers in CI/CD Your CI pipeline should NOT mount code into containers. It should build complete, testable images that mirror production. {{< code lang="yaml" file=".github/workflows/ci.yml" >}} # BAD: Mounting code in CI - name: Test run: docker run -v $(pwd):/app myapp-runtime pytest # GOOD: Build a complete image for testing - name: Build test image run: docker build --target production -t myapp:test . - name: Test run: docker run myapp:test pytest {{< /code >}} CI should mimic production as closely as possible. Runtime containers are for development velocity, not deployment. ### Mistake 5: Mixing development and production configurations Developers sometimes accidentally deploy development images or use development settings in production. **Solution:** Use clear naming and separate files. ``` Dockerfile # Production image (code baked in) Dockerfile.runtime # Development runtime (no code) docker compose.yml # Development environment docker compose.prod.yml # Production environment ``` Never deploy anything with `.runtime` or `development` in the name. ## When NOT to use runtime containers Runtime containers aren't universally better. They have specific use cases. **Use runtime containers when:** - You're actively developing and iterating on code - You want instant feedback without rebuilds - Your team uses different operating systems - You need consistent development environments **Don't use runtime containers when:** - Building for production (bake code in) - Running CI/CD tests (build complete images) - Code and dependencies change together frequently - You're working on a compiled language with slow compilation (Go, Rust, C++) For compiled languages, the compilation step is the bottleneck, not Docker builds. Runtime containers don't help much there. ## Real-world example: Polyglot microservices Here's a complete setup for a Python API backend + Node.js frontend, both using runtime containers. {{< code lang="yaml" file="docker compose.yml" >}} version: '3.8' services: # Python API backend api: build: context: ./backend dockerfile: Dockerfile.runtime volumes: - ./backend/src:/app/src:ro - ./backend/tests:/app/tests:ro environment: - DATABASE_URL=postgresql://postgres:password@db:5432/myapp - FLASK_ENV=development - FLASK_DEBUG=1 ports: - "5000:5000" depends_on: - db command: flask run --host=0.0.0.0 # Node.js frontend frontend: build: context: ./frontend dockerfile: Dockerfile.runtime volumes: - ./frontend/src:/app/src:ro - ./frontend/public:/app/public:ro - /app/node_modules # Anonymous volume for dependencies environment: - NODE_ENV=development - REACT_APP_API_URL=http://localhost:5000 ports: - "3000:3000" command: npm start # PostgreSQL database db: image: postgres:15-alpine environment: POSTGRES_PASSWORD: password POSTGRES_DB: myapp volumes: - postgres_data:/var/lib/postgresql/data ports: - "5432:5432" volumes: postgres_data: {{< /code >}} **backend/Dockerfile.runtime:** {{< code lang="dockerfile" >}} FROM python:3.11-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt CMD ["flask", "run", "--host=0.0.0.0"] {{< /code >}} **frontend/Dockerfile.runtime:** {{< code lang="dockerfile" >}} FROM node:18-slim WORKDIR /app COPY package\*.json ./ RUN npm ci CMD ["npm", "start"] {{< /code >}} Start everything: {{< code lang="bash" >}} docker compose up {{< /code >}} The workflow: - Edit Python code -> Flask auto-reloads - Edit React code -> Webpack auto-reloads - Add Python dependency -> `docker compose build api && docker compose up` - Add Node dependency -> `docker compose build frontend && docker compose up` Both services have isolated runtimes. Both reload instantly on code changes. Both share the same database. The entire stack runs with one command. ## The mental model The key to using runtime containers effectively is changing how you think about containers. **Traditional Docker thinking:** Container = snapshot of entire application (code + runtime + dependencies) **Runtime container thinking:** Container = runtime environment, code is dynamic This mental changes your development approach: - **Don't rebuild for code changes**: Rebuilds are only for runtime/dependency changes - **Separate concerns**: Runtime stability vs code iteration are different problems - **Development ≠ Production**: Use different strategies for different environments - **Mount, don't copy**: During development, mount everything you're actively changing The goal is to maintain containerized consistency without sacrificing development velocity. You want the isolation and reproducibility of containers with the speed of local development. --- Runtime containers solve a specific problem: containerized development is slow. By separating the runtime from the code, you get the best of both worlds: isolated, reproducible environments and instant feedback loops. This isn't a replacement for production container builds. It's a development accelerator. Use runtime containers to iterate quickly during active development. Build complete, immutable images for testing and deployment. The workflows in this guide are battle-tested patterns from real projects: Python APIs processing thousands of requests, Node.js frontends with hot module replacement, polyglot microservices running in Docker Compose. They work because they respect the separation of concerns: runtime in the container, code on the host, clear boundaries between development and production. If you're rebuilding your Docker image every time you change a line of code, you're doing it wrong. Adopt runtime containers. Your feedback loop will thank you. ## Further Reading - [Docker Volumes vs Bind Mounts](https://docs.docker.com/engine/storage/volumes/?ref=lorbic.com) - Official Docker documentation on storage options - [Multi-Stage Builds](https://docs.docker.com/build/building/multi-stage/?ref=lorbic.com) - How to build development and production images from one Dockerfile - [Developing inside a Container](https://code.visualstudio.com/docs/devcontainers/containers?ref=lorbic.com) - VS Code DevContainers guide - [Best practices for writing Dockerfiles](https://docs.docker.com/build/building/best-practices/?ref=lorbic.com) - Production Dockerfile optimizations - [Docker Compose for Development](https://docs.docker.com/compose/gettingstarted/?ref=lorbic.com) - Multi-container development workflows --- ## Building Production-Ready Background Workers in Python **Date:** 2026-02-01 **URL:** https://lorbic.com/building-production-ready-background-workers-in-python/ **Description:** Your background jobs crash on deployment, fail silently, and corrupt data. Here's how to fix it with worker pools, retry strategies, idempotency, and graceful shutdown. A guide to building production background processing systems that don't break under load. Tested at 20 req/sec. I thought processing audio in the background was simple: spawn a thread, run the script, save the file. Then I hit 200 concurrent requests, and it failed epically. The CPU spiked to full usage because of pydub's processing. The TTS API didn't rate-limit me but, it was horribly slow. Then the jobs started failing. Half the jobs died silently. The other half wrote corrupted files because of race conditions I didn't know existed. And when I deployed a little fix? The deployment killed in-flight jobs, leaving orphaned audio segments scattered across cloud storage directory. This is not a guide about how to spawn a background task. This is about what happens after that, when the simple script becomes production infrastructure, when "it works on my machine" becomes "why did it fail in production and how can I fix it?" Referencing existing production systems was a great way to learn about the tradeoffs and patterns that exist in the space. I found a lot of cool articles. So, I started looking for architectural keywords, patterns, and tradeoffs. And started reading about their implementation in production. Then I built them into the batch audio processing system. Python, FastAPI, asyncio. It takes text, sends it to Google Cloud Text-to-Speech, processes the audio, and stitches hundreds of segments into final output. I wrote down every pattern, every decision, every tradeoff. This is that document (with the important parts). ## Why background processing is harder than it looks Consider a client request to generate audio for 200 text segments. Each segment requires a TTS API call (300-1000ms), post-processing (adding silence, normalizing), upload to cloud storage, progress tracking, final stitching, and webhook notification. Sequentially? That's 4-5 minutes. You can't hold an HTTP connection open that long. The answer is obvious: background processing. Accept the request, return a job ID, process asynchronously. But this creates a cascade of new problems. What happens if a worker crashes mid-job? What if two workers grab the same job? What about rate limits? Retries? What if the database locks under concurrent writes? What if a deployment kills a running job? How do you even debug failures that happen while you're asleep? {{< alert type="important" >}} Background processing isn't hard because the happy path is hard. It's hard because everything else is hard. {{< /alert >}} This blog covers the patterns that solve these problems. ## Level 1: The simple approach Here's how most people start: {{< code lang="python" file="api.py" >}} @app.post("/generate") async def generate_audio(request: AudioRequest): # Bad idea: processing in the request handler for segment in request.segments: audio = await tts_service.generate(segment.text) await storage.upload(audio) return {"status": "done"} {{< /code >}} This fails immediately. The client times out. The request gets killed. Half the segments are processed, half aren't. There's no way to recover. The first fix is obvious: move to background tasks. {{< code lang="python" file="api.py" >}} @app.post("/generate") async def generate_audio(request: AudioRequest): job_id = create_job(request) # Stores job in database with PENDING status asyncio.create_task(process_job(job_id)) # Fire and forget return {"job_id": job_id, "status": "accepted"} {{< /code >}} Better. The client gets a response immediately. But now you have new problems. If the process restarts, all running jobs are lost. There's no persistence. If `process_job` throws an exception, it vanishes silently. If you scale to multiple instances, you have no coordination. If a job takes too long, there's no timeout. This is where I learned and used the real architectural patterns and techniques. ## Level 2: Using a worker pool The fundamental unit of background processing is the **worker pool**: a fixed set of workers pulling jobs from a queue. ``` +-------------------------------------------------------+ | Job Queue | | [Job1] [Job2] [Job3] [Job4] [Job5] [Job6] ... | +--------------------------+----------------------------+ | +------------------+------------------+ v v v +---------+ +---------+ +---------+ | Worker1 | | Worker2 | | Worker3 | +---------+ +---------+ +---------+ ``` Created with [asciiflow](https://asciiflow.com?ref=lorbic.com)
Unlike fire-and-forget tasks, a worker pool provides bounded concurrency, crash recovery through job persistence, and visibility into what's running. The queue can be in-memory (fast, but loses jobs on crash), database-backed (durable, simple), or a dedicated message broker (Kafka, RabbitMQ, SQS). I used SQLite with WAL mode. It's surprisingly capable for moderate workloads, and deployment is trivial. One file, no infrastructure.
### The Worker Loop Here's the skeleton of a worker in Python: {{< code lang="python" file="worker.py" >}} class Worker: def **init**(self, db: Database, semaphore: asyncio.Semaphore): self.db = db self.semaphore = semaphore self.shutdown_event = asyncio.Event() async def run(self): while not self.shutdown_event.is_set(): job = await self.db.claim_next_pending_job() if job is None: await asyncio.sleep(1) # No work, poll again continue async with self.semaphore: # Bound concurrent operations try: await self.process_job(job) await self.db.mark_completed(job.id) except Exception as e: await self.db.mark_failed(job.id, str(e)) async def process_job(self, job): for segment in job.segments: if segment.status == "COMPLETED": continue # Skip already-done segments (resumption) audio = await self.tts_service.generate(segment.text) await self.storage.upload(audio) await self.db.mark_segment_completed(segment.id) {{< /code >}} A few critical details here: The `claim_next_pending_job` must be **atomic**. Two workers should never claim the same job. In SQL, this looks like: {{< code lang="sql" file="claim_job.sql" >}} UPDATE jobs SET status = 'IN_PROGRESS', worker_id = ? WHERE id = ( SELECT id FROM jobs WHERE status = 'PENDING' LIMIT 1 ) RETURNING \*; {{< /code >}} The `semaphore` bounds concurrency. If you fire 1000 API calls at once, you'll exhaust rate limits, connection pools, and most importantly, system resources like CPU and memory. Start with 5-10 concurrent operations and increase based on monitoring. The loop checks for already-completed segments before processing. This enables **resumption**. If the job failed at segment 150, the retry skips segments 1-149. ## Level 3: Making it resilient Failures are not exceptional. They're normal in distributed systems. There are many reasons why things can fail: networks drop, APIs rate-limit, databases lock, servers restart etc. Your system needs to survive all of this. ### Retry with exponential backoff The naive approach to retries: retry immediately. This is wrong. If an API is overloaded, hammering it with immediate retries makes things worse. Exponential backoff spaces out retries. And it allows the system to cool down. {{< code lang="python" file="retry.py" >}} import asyncio import random from functools import wraps def retry_with_backoff(max_retries=5, base_delay=1.0, max_delay=60.0): def decorator(func): @wraps(func) async def wrapper(*args, \*\*kwargs): last_exception = None for attempt in range(max_retries): try: return await func(*args, \*\*kwargs) except TransientError as e: last_exception = e if attempt == max_retries - 1: raise # Exponential backoff with jitter delay = min(base_delay * (2 ** attempt), max_delay) jitter = delay * 0.5 * random.random() await asyncio.sleep(delay + jitter) raise last_exception return wrapper return decorator @retry_with_backoff(max_retries=5) async def call_tts_api(text: str) -> bytes: response = await http_client.post("/synthesize", json={"text": text}) if response.status_code == 429: raise TransientError("Rate limited") if response.status_code >= 500: raise TransientError("Server error") if response.status_code == 400: raise PermanentError("Invalid input") # Don't retry this return response.content {{< /code >}} The **jitter** is crucial. Without it, all clients retry at the same instant after an outage, creating thundering herd spikes. Random jitter spreads the load and prevent repeated failures. For example if 1000 clients all retry at the same instant, they'll all fail and retry at the same instant, creating a cascade of failures. {{< alert type="tip" >}} Retry transient failures. Fail fast on permanent errors. {{< /alert >}} A 400 Bad Request won't magically succeed on retry. A 429 or 503 might. Classify your errors and act accordingly. ### Making operations safe to repeat (Idempotency) What happens if a worker crashes after uploading an audio file but before marking the segment as complete? The retry will upload the same file again. If your upload uses a deterministic key (e.g., `job_123/segment_045.mp3`), this is fine. Uploading the same content to the same key is a no-op. That's **idempotency**. For database updates, use conditional writes: {{< code lang="sql" file="idempotent_update.sql" >}} UPDATE segments SET status = 'COMPLETED' WHERE id = ? AND status = 'IN_PROGRESS'; {{< /code >}} This won't double-complete a segment. If something else already completed it, the update affects zero rows. For external APIs that aren't idempotent, use client-generated request IDs: {{< code lang="python" >}} request*id = f"job*{job*id}\_segment*{segment_id}\_v{attempt}" response = await api.synthesize(text=text, request_id=request_id) {{< /code >}} Many APIs deduplicate by request ID. Check your provider's documentation to implement what's best for your use case. ## Level 4: Production Readiness Your system works. Jobs process. Retries happen. But you're not done. Production means handling deployments, shutdowns, and failures you haven't anticipated yet while developing. ### Graceful Shutdown When you deploy new code, what happens to running jobs? The naive answer: they die. SIGTERM kills the process. Jobs are left in `IN_PROGRESS` forever, or worse, with corrupted output. The correct answer is to implement graceful shutdown. {{< code lang="python" file="shutdown.py" >}} import signal import asyncio class WorkerManager: def **init**(self): self.workers = [] self.active_jobs = set() self.shutdown_requested = False def setup_signal_handlers(self): for sig in (signal.SIGTERM, signal.SIGINT): signal.signal(sig, self._handle_shutdown) def _handle_shutdown(self, signum, frame): self.shutdown_requested = True for worker in self.workers: worker.shutdown_event.set() async def wait_for_shutdown(self, timeout=30): start = asyncio.get_event_loop().time() while self.active_jobs: if asyncio.get_event_loop().time() - start > timeout: # Hard deadline: force exit break await asyncio.sleep(0.1) {{< /code >}} Here the sequence matters: 1. Receive SIGTERM 2. Stop accepting new work 3. Signal workers to stop pulling new jobs 4. Wait for in-progress jobs to finish (with timeout) 5. Flush any pending writes 6. Exit If a job doesn't finish within the timeout, it should be **resumable**. When the new instance starts, it should pick up where the old one left off. ### Structured Logging When something fails, your logs are your only debugging tool. Unstructured logs are useless: {{< terminal title="Bad Logs" >}} Processing job job_123 segment 45 with voice en-US-Standard-A {{< /terminal >}} Structured logs are useful and queryable: {{< code lang="python" file="logging.py" >}} import structlog log = structlog.get_logger() log.info( "segment_processing_start", job_id=job.id, segment_id=segment.id, voice=segment.voice, attempt=attempt_number ) {{< /code >}} This outputs JSON that you can search in your logging platform: {{< terminal title="Good Logs">}} { "event": "segment_processing_start", "job_id": "job_123", "segment_id": "45", "voice": "en-US-Standard-A", "attempt": 1, "timestamp": "2026-02-01T10:30:00Z" } {{< /terminal >}} Log job state transitions, external API calls (with durations), retries, and errors. Don't log sensitive data. Use log levels appropriately (DEBUG for verbose tracing, INFO for milestones, ERROR for failures). ### Health Checks and Startup Validation A service that starts with invalid configuration will fail eventually, usually at the worst time. Validate everything at startup: {{< code lang="python" file="validation.py" >}} async def startup_validation(): # Check secrets exist if not os.getenv("TTS_API_KEY"): raise ConfigurationError("TTS_API_KEY not set") # Check external connectivity try: await tts_client.ping() except Exception as e: raise ConfigurationError(f"Cannot reach TTS API: {e}") # Check database try: await db.execute("SELECT 1") except Exception as e: raise ConfigurationError(f"Database unavailable: {e}") # Check storage try: await storage.check_permissions() except Exception as e: raise ConfigurationError(f"Storage not writable: {e}") {{< /code >}} If validation fails, exit immediately with a clear error message. Don't try to "work around" missing configuration. It will cause confusing failures later. ## The reusable patterns No single pattern is sufficient. It's their combination that produces reliability. **Statelessness** means any worker can pick up any job. State lives in the database, not in memory. **Idempotency** means operations are safe to repeat. Retries can't corrupt data. **Bounded concurrency** means you don't overwhelm downstream services or your own resources. Semaphores control parallelism. **Granular progress tracking** means jobs can resume from where they failed. You don't re-process 149 segments because segment 150 failed. **Graceful shutdown** means deployments don't kill work. In-flight jobs complete or become resumable. **Structured logging** means you can debug failures after the fact. Every significant event is recorded with context. **Startup validation** means configuration errors are caught early. You don't discover missing credentials during processing job, if they were never added in the first place. ## Common mistakes to avoid **Unbounded queues.** Memory grows without limit as queue depth increases. Bound your queues. Apply backpressure (return 503) when full. For internal systems, persist all submitted jobs to the database and process them in the background. Rate limiting and capacity planning can handle excessive request volumes. **Missing timeouts.** Operations hang forever, blocking workers. Timeout everything: API calls, database queries, entire jobs. **Ignoring partial failures.** One failed segment causes the entire job to fail. Track granular status. Report partial success. Enable resumption. **State in memory.** Worker crash loses all in-progress state. Persisting state externally is a must. **Retry storms.** All clients retry simultaneously after an outage (you restart the service). Adding jitter to backoff and using circuit breakers can help. **No graceful shutdown.** Deployments kill jobs, causing failures and orphaned state. Handle SIGTERM, wait for active jobs to finish and drain queues. ## Do you need all this? - **< 10 jobs/hour, quick tasks**: FastAPI BackgroundTasks is fine - **100s of jobs/hour, mission-critical**: Use these patterns - **1000s+ jobs/hour**: Consider dedicated systems (Celery, etc) --- Building production-ready background processing systems requires attention to many details. Worker pools with bounded concurrency prevent resource exhaustion. Retry strategies with exponential backoff and jitter handle transient failures gracefully. Idempotent operations enable safe retries and resumption. Graceful shutdown protects in-flight work during deployments. The audio processing system I built handles thousands of segments daily (tested with 20rps, with 10 segments each and 5 workers). The principles behind it would work equally well for video transcoding, document processing, email sending, or any other batch workload. I learned a lot from this experience and I will be using these principles in my future projects and learn more design patterns. My understanding is, start simple and add complexity only when needed. Measure everything you can. And always design as if your system will fail, because eventually, it will. The question is whether it fails gracefully or just plain fails. ## Further Reading These books and articles helped me a lot to build this system. - For queue design: Chapter 11 on Queues and Streams of [Designing Data-Intensive Applications](https://dataintensive.net/?ref=lorbic.com) by Martin Kleppmann - For retry patterns: Chapter 5 on Stability Patterns of [Release It!](https://pragprog.com/titles/mnee2/release-it-second-edition/?ref=lorbic.com) by Michael Nygard - [FastAPI Background Tasks](https://fastapi.tiangolo.com/tutorial/background-tasks/?ref=lorbic.com) - [structlog for Python](https://www.structlog.org/en/stable/?ref=lorbic.com) - [SQLite WAL Mode Explained](https://www.sqlite.org/wal.html?ref=lorbic.com) --- ## Go Struct Field Alignment **Date:** 2026-01-24 **URL:** https://lorbic.com/go-struct-field-alignment/ **Description:** Your Go structs might be wasting up to 32% of their memory due to invisible padding bytes. This deep dive into struct field alignment reveals how the compiler arranges memory, why field order matters, and provides benchmarks showing real memory savings. Learn the simple reordering rules that can shrink your heap, reduce GC pressure, and improve CPU cache efficiency. You write a struct to represent a database entity. Maybe 10 fields, maybe 20. What could possibly go wrong? Nothing, according to your tests. But somewhere in production, your heap is 30% larger than it should be, your Garbage Collector is working overtime, and your L1 cache is not used properly. The reason? **Invisible padding bytes** silently inflating every instance of your struct. This is the story of struct field alignment: a memory optimization that costs nothing to implement but can significantly improve performance. ## Why Alignment Exists Modern CPUs don't load memory one byte at a time. They operate on **aligned words**, typically 8 bytes on 64-bit systems. When a data type's memory address is not divisible by its alignment requirement, one of two things happens: 1. **Performance penalty**: The CPU performs multiple memory accesses to load the value 2. **Hardware fault**: On some architectures (historically common, now rare), unaligned access causes a crash To prevent this, the Go compiler automatically inserts **padding bytes** between struct fields. The padding ensures each field starts at a memory address that satisfies its alignment requirement. ### The Go Spec on Alignment According to the [Go Language Specification](https://go.dev/ref/spec#Size_and_alignment_guarantees): > For a variable `x` of any type: `unsafe.Alignof(x)` is at least 1. > For a variable `x` of struct type: `unsafe.Alignof(x)` is the largest of all the values `unsafe.Alignof(x.f)` for each field `f` of `x`, but at least 1. This means a struct's alignment is determined by its largest-aligned field. A struct containing an `int64` must be 8-byte aligned, even if its first field is a `bool`. ## Demonstrating the Silent Bloat Consider this innocent-looking struct with fields ordered "logically" (booleans together, then data): ```go type BadStruct struct { IsActive bool // 1 byte + 7 padding ID uint64 // 8 bytes IsVerified bool // 1 byte + 7 padding Name string // 16 bytes (ptr + len) IsAdmin bool // 1 byte + 7 padding Score float64 // 8 bytes // ... more fields } ``` Each 1-byte `bool` followed by an 8-byte type wastes 7 bytes of padding. Let's measure real structs with all common Go types: ```go // BadStruct: Fields interleaved to maximize padding type BadStruct struct { IsActive bool // 1 byte + 7 padding ID uint64 // 8 bytes IsVerified bool // 1 byte + 7 padding Name string // 16 bytes IsAdmin bool // 1 byte + 7 padding Score float64 // 8 bytes IsPremium bool // 1 byte + 3 padding ParentID uint32 // 4 bytes TinyVal int8 // 1 byte + 1 padding SmallVal int16 // 2 bytes + 4 padding Email string // 16 bytes IsDeleted bool // 1 byte + 7 padding Count int64 // 8 bytes IsArchived bool // 1 byte + 3 padding Rating float32 // 4 bytes Status int8 // 1 byte + 7 padding Tags []string // 24 bytes Enabled bool // 1 byte + 7 padding Metadata map[string]string // 8 bytes Ready bool // 1 byte + 7 padding CreatedAt int64 // 8 bytes Done bool // 1 byte + 7 padding UpdatedAt int64 // 8 bytes Callback func() // 8 bytes Flag bool // 1 byte + 7 padding Description string // 16 bytes } ``` Measuring with `unsafe.Sizeof`: ```go fmt.Println(unsafe.Sizeof(BadStruct{})) // Output: 224 bytes ``` **224 bytes.** Let's see how much is wasted. ## The Alignment Rules Go's alignment requirements are architecture-dependent but follow predictable rules on 64-bit systems: | Type | Size | Alignment | Notes | |:-----|:----:|:---------:|:------| | `bool` | 1 | 1 | | | `int8`, `uint8`, `byte` | 1 | 1 | | | `int16`, `uint16` | 2 | 2 | | | `int32`, `uint32`, `float32` | 4 | 4 | | | `int64`, `uint64`, `float64` | 8 | 8 | | | `int`, `uint`, `uintptr` | 8 | 8 | On 64-bit systems | | `string` | 16 | 8 | Header: `{ptr, len}` | | `slice` (`[]T`) | 24 | 8 | Header: `{ptr, len, cap}` | | `map` | 8 | 8 | Pointer to `hmap` | | `func` | 8 | 8 | Pointer | | `interface{}` | 16 | 8 | `{type, data}` | | `pointer` (`*T`) | 8 | 8 | | | `chan` | 8 | 8 | Pointer to `hchan` | {{< alert type="important" >}} Strings and slices are **header types**. They contain pointers to the actual data, not the data itself. A `string` is 16 bytes regardless of its content length. {{< /alert >}} ### The Optimization Rule **Order fields from largest alignment requirement to smallest:** 1. **24-byte types**: Slices (`[]T`) 2. **16-byte types**: Strings, interfaces 3. **8-byte types**: `int64`, `uint64`, `float64`, pointers, maps, funcs, chans 4. **4-byte types**: `int32`, `uint32`, `float32` 5. **2-byte types**: `int16`, `uint16` 6. **1-byte types**: `int8`, `uint8`, `bool`, `byte` Applying this to our struct: {{< code lang="go" wrap="true" >}} // GoodStruct: Fields ordered by alignment (largest to smallest) type GoodStruct struct { // 24-byte: Slices Tags []string // 24 bytes // 16-byte: Strings Name string // 16 bytes Email string // 16 bytes Description string // 16 bytes // 8-byte: int64, uint64, float64, pointers ID uint64 // 8 bytes Count int64 // 8 bytes Score float64 // 8 bytes CreatedAt int64 // 8 bytes UpdatedAt int64 // 8 bytes Metadata map[string]string // 8 bytes (pointer) Callback func() // 8 bytes (pointer) // 4-byte: int32, uint32, float32 ParentID uint32 // 4 bytes Rating float32 // 4 bytes // 2-byte: int16, uint16 SmallVal int16 // 2 bytes // 1-byte: int8, bool (packed together) TinyVal int8 // 1 byte Status int8 // 1 byte IsActive bool // 1 byte IsVerified bool // 1 byte IsAdmin bool // 1 byte IsPremium bool // 1 byte IsDeleted bool // 1 byte IsArchived bool // 1 byte Enabled bool // 1 byte Ready bool // 1 byte Done bool // 1 byte Flag bool // 1 byte + 2 padding (struct alignment) } {{< /code >}} ```go fmt.Println(unsafe.Sizeof(GoodStruct{})) // Output: 152 bytes ``` **152 bytes.** We saved **72 bytes** (32% reduction) with zero code changes, just by reordering fields. ## Measuring the Impact with Benchmarks Here are real benchmarks run on an Apple M2 (darwin/arm64): ### Struct Sizes ```text === TestEntitySizes === BadStruct size: 224 bytes GoodStruct size: 152 bytes Memory saved: 72 bytes (32.1% reduction) ``` ### Allocation Benchmarks ```text goos: darwin goarch: arm64 cpu: Apple M2 BenchmarkBadStruct_Alloc-8 17997783 61.11 ns/op 224 B/op 1 allocs/op BenchmarkGoodStruct_Alloc-8 33894393 31.85 ns/op 160 B/op 1 allocs/op BenchmarkBadStruct_Slice1k-8 95834 12318 ns/op 229377 B/op 1 allocs/op BenchmarkGoodStruct_Slice1k-8 136102 8421 ns/op 155649 B/op 1 allocs/op BenchmarkBadStruct_Slice10k-8 16683 71874 ns/op 2244613 B/op 1 allocs/op BenchmarkGoodStruct_Slice10k-8 20770 60512 ns/op 1523715 B/op 1 allocs/op ``` ### Analysis | Benchmark | Bad | Good | Improvement | |:----------|:----|:-----|:------------| | **Single Alloc** | 61.1 ns, 224 B | 31.9 ns, 160 B | **48% faster** | | **1k Slice** | 12.3 µs, 224 KB | 8.4 µs, 152 KB | **32% faster, 32% less memory** | | **10k Slice** | 71.9 µs, 2.19 MB | 60.5 µs, 1.49 MB | **16% faster, 32% less memory** | {{< alert type="tip" >}} The 32% memory reduction is consistent because it directly reflects the struct size difference. The time improvement comes from less memory to zero-initialize and better cache utilization. {{< /alert >}} ### Run the Benchmarks Yourself {{< download file="struct_alignment_benchmark_test.go" label="Download struct_alignment_benchmark_test.go" >}} {{< terminal title="Running the benchmarks" >}} $ go mod init alignment_test $ go test -v -run TestEntitySizes $ go test -bench=. -benchmem {{< /terminal >}} ## How Padding Works Behind the Scenes Let's trace through the field offsets to understand exactly where padding is inserted: ```text BadStruct Field Offsets: ``` | Field | Offset | Size | Explanation | |:------|:------:|:----:|:------------| | IsActive | 0 | 1 | Starts at 0 | | (padding) | 1 | 7 | ID needs 8-byte alignment | | ID | 8 | 8 | Starts at 8 (divisible by 8) | | IsVerified | 16 | 1 | Starts at 16 | | (padding) | 17 | 7 | Name needs 8-byte alignment | | Name | 24 | 16 | Starts at 24 (divisible by 8) | | IsAdmin | 40 | 1 | | | (padding) | 41 | 7 | Score needs 8-byte alignment | | Score | 48 | 8 | | | IsPremium | 56 | 1 | | | (padding) | 57 | 3 | ParentID needs 4-byte alignment | | ParentID | 60 | 4 | 60 is divisible by 4 | | TinyVal | 64 | 1 | | | (padding) | 65 | 1 | SmallVal needs 2-byte alignment | | SmallVal | 66 | 2 | 66 is divisible by 2 | | (padding) | 68 | 4 | Email needs 8-byte alignment | | Email | 72 | 16 | 72 is divisible by 8 | | ... | | | | You can inspect these offsets programmatically: ```go var s BadStruct fmt.Printf("IsActive offset: %d\n", unsafe.Offsetof(s.IsActive)) // 0 fmt.Printf("ID offset: %d\n", unsafe.Offsetof(s.ID)) // 8 fmt.Printf("IsVerified offset: %d\n", unsafe.Offsetof(s.IsVerified)) // 16 fmt.Printf("Name offset: %d\n", unsafe.Offsetof(s.Name)) // 24 ``` ## Why This Matters Beyond Memory ### 1. Reduced GC Pressure Smaller structs mean: - **Smaller heap** -> Less memory to scan during garbage collection - **Better cache locality** -> GC mark phase runs faster - **Lower allocation churn** -> GC triggers less frequently {{< alert type="tip" >}} **Want to go deeper on GC?** Read my other deep dive: [Don't Take Out the Garbage: Go GC Deep Dive](/posts/dont-take-out-the-garbage-go-gc-deep-dive/). It explains how the mark-and-sweep algorithm works and why large heaps crush performance. {{< /alert >}} If `runtime.scanobject` appears in your CPU profiles, you have GC pressure. Shrinking struct sizes directly reduces this cost. ### 2. CPU Cache Efficiency The L1 data cache line is typically 64 bytes. Compact structs fit more instances per cache line: ```text 64-byte cache line capacity: - BadStruct (224 bytes): 0.28 structs per line - GoodStruct (152 bytes): 0.42 structs per line ``` When iterating over a slice of structs, compact layouts mean fewer cache misses and better hardware prefetching. ### 3. Network/Disk Efficiency If you use binary serialization (protobuf, msgpack, gob), struct layout can affect: - Network payload sizes - Memory-mapped file efficiency - Serialization/deserialization speed ## Detecting Alignment Issues ### Tool: `fieldalignment` The Go team provides an official analyzer: {{< terminal title="Using fieldalignment" wrap="true" >}} $ go install golang.org/x/tools/go/analysis/passes/fieldalignment/cmd/fieldalignment@latest $ fieldalignment ./... $ fieldalignment -fix ./... {{< /terminal >}} Example output: ```text entity/user.go:15:6: struct of size 224 could be 152 (order fields by alignment) entity/post.go:42:6: struct of size 120 could be 104 (order fields by alignment) ``` ### Manual Inspection Use `unsafe.Sizeof` and `unsafe.Offsetof` to audit critical structs: ```go import "unsafe" func auditStruct() { var e Entity fmt.Printf("Total size: %d bytes\n", unsafe.Sizeof(e)) fmt.Printf("Field1 offset: %d\n", unsafe.Offsetof(e.Field1)) fmt.Printf("Field2 offset: %d\n", unsafe.Offsetof(e.Field2)) // Check for gaps between (offset + size) and next offset } ``` ## Pitfalls and Caveats ### 1. Auto-fix Can Break Binary Compatibility {{< alert type="caution" >}} The `fieldalignment -fix` tool reorders fields. If your code depends on specific field ordering for binary serialization, memory-mapped files, or CGO interop, this will break things silently. {{< /alert >}} Always review changes before applying them. For CGO structs, field order must often match C struct definitions exactly. ### 2. JSON/YAML Unmarshalling Is Unaffected The common fear that reordering fields breaks JSON parsing is **unfounded**. The `encoding/json` package uses reflection and struct tags to map JSON keys to fields: ```go type User struct { Age int `json:"age"` // Field order in memory Name string `json:"name"` // ≠ key order in JSON } // JSON: {"name":"Vikash","age":40} // Works identically regardless of struct field order ``` ### 3. Don't Micro-optimize Everything Alignment optimization matters when: - The struct is instantiated **thousands or millions of times** - The struct appears in **hot paths** (request handlers, tight loops) - You're seeing **GC pressure** (`runtime.scanobject` > 5% CPU) - Memory is a **constraint** (embedded systems, large caches) For structs used sparingly, readability may be more valuable than a few bytes. ### 4. Struct Tail Padding Even optimally ordered structs may have tail padding. The struct's total size must be a multiple of its alignment: ```go type Example struct { A int64 // 8 bytes B bool // 1 byte + 7 padding = 16 total } ``` This ensures arrays of structs maintain proper alignment for each element. ## Quick Reference Checklist When designing or refactoring structs: 1. **Order by alignment**: 24-byte -> 16-byte -> 8-byte -> 4-byte -> 2-byte -> 1-byte 2. **Pack bools together**: Place all `bool` fields at the end 3. **Verify with `unsafe.Sizeof`**: Confirm your optimizations work 4. **Run `fieldalignment`**: Catch what you miss, but review before applying 5. **Profile first**: Only optimize structs that matter ```go // Quick verification import "unsafe" func checkSize[T any]() { var x T fmt.Printf("Type: %T, Size: %d bytes, Align: %d\n", x, unsafe.Sizeof(x), unsafe.Alignof(x)) } ``` ## Summary | Concept | Key Point | |:--------|:----------| | **Problem** | CPU alignment requirements force compilers to insert padding | | **Symptom** | Structs consume more memory than the sum of their fields | | **Solution** | Order fields from largest to smallest alignment requirement | | **Benefit** | 20-40% typical memory reduction, better cache usage, lower GC pressure | | **Caveat** | Don't auto-fix binary-serialized or CGO structs without proper review and testing | Field alignment is the kind of optimization that costs nothing to implement correctly. Once you internalize the ordering rules, writing compact structs becomes second nature. Your heap will be smaller, your GC will run faster, and your CPU cache will thank you. ## Further Reading 1. **[Go Language Specification: Size and Alignment](https://go.dev/ref/spec#Size_and_alignment_guarantees)** - Go spec 2. **[fieldalignment analyzer source](https://cs.opensource.google/go/x/tools/+/master:go/analysis/passes/fieldalignment/)** - Understand how the tool works 3. **[A Guide to the Go Garbage Collector](https://go.dev/doc/gc-guide)** - Why heap size matters 4. **[unsafe package documentation](https://pkg.go.dev/unsafe)** - `Sizeof`, `Offsetof`, `Alignof` --- ## I Added Session Management to Aider **Date:** 2026-01-15 **URL:** https://lorbic.com/aider-session-management/ **Description:** Aider is my favorite AI coding assistant. It lives in the terminal, commits directly to Git, and actually understands my codebase. But it was missing one thing: the ability to save and restore chat sessions. So I built it. The PR never got merged, but I've been running my fork for months now. Here's the story. I've been using [aider](https://aider.chat) as my primary AI coding assistant for a while now. It's the one tool that actually fits my workflow: terminal-based, Git-native, and it works directly on my local files. No copy-pasting into web forms. No context windows that forget everything. And not waiting for agents to keep thinking. But it was missing one thing that drove me crazy. ## The Problem: Context Evaporates Here's the scenario. I'm deep into a complex refactor. I've added 15 files to the chat, built up tons of context with the model, and we're making real progress. Then one of these happens: - I need to step away for a meeting - I merge a big feature branch (need to change branches, aider detects repo changes and resets, loosing all context) - My terminal session dies - I accidentally hit `/clear` Poof. All that context is gone. The conversation history, the files I'd carefully curated, everything. Starting over every time was killing my momentum. I kept thinking: *tmux has sessions. Why doesn't aider?* So I built it. ## The `/session` Command I added a full session management system to aider. Five simple commands: ```bash # Save your current session /session save my-feature-work # List all saved sessions /session list # Peek inside a session without loading it /session view my-feature-work # Load a session (clears current chat and restores the saved one) /session load my-feature-work # Delete a session you don't need anymore /session delete my-feature-work ``` That's it. Now I can save context before stepping away, after hitting a good milestone, or just because I'm paranoid about losing progress. ## How It Works When you run `/session save`, here's what happens: 1. A `.aider/sessions/` directory gets created in your repo root 2. The current state is captured: chat history, editable files, read-only files 3. Everything gets bundled into a JSON file with metadata (timestamp, aider version) Loading is the reverse, read the JSON, reset state, restore everything. The sessions are project-specific (stored in each repo), so you don't get cross-contamination between projects. And since it's just JSON files, you can even version control them if you want. I put the implementation in `aider/commands.py` and added a full test suite in `tests/test_session.py`. The tests spin up a temporary Git repo and run all the subcommands. ## The PR Situation Here's the honest part: **my PR never got merged into upstream aider**. I'm not entirely sure why. Maybe the maintainers are busy with other things, or maybe upstream is not in active development. These things happen in open source. Thanks to it being open source, I can still fork and build my own version to use it daily. But here's the thing: **I've been running my fork daily for months now**. Check it out: [github.com/vk4s/aider](https://github.com/vk4s/aider) The feature works great. I use it constantly. Every time I'm about to do something risky or need to step away, I hit `/session save`. It's muscle memory now. ## Why I'm Still Happy About It Even without the merge, this contribution taught me a lot: 1. **I got to dive deep into aider's codebase** - Understanding how it manages state, handles commands, and integrates with Git was fascinating. 2. **I scratched my own itch** - The best open source contributions come from actual pain points. This wasn't a theoretical improvement; it was something I really needed. 3. **I use it every day** - The feature exists. It works. I benefit from it. That's a win. 4. **Forking is fine** - There's sometimes stigma around maintaining a fork, but if the upstream isn't taking your changes and the feature is valuable to you, why not? Keep your fork in sync with upstream, apply your patches, and move on. I'm leaving my pr open in the upstream and when it's merged I'll use the upstream version. (: ## If You Want Session Management If you're an aider user who wants this feature: 1. Clone my fork: `git clone https://github.com/vk4s/aider` 2. Install it: `pip install -e .` 3. Use `/session save|load|list|view|delete` Or just grab the relevant commits and cherry-pick them into your own setup. --- Contributing to open source isn't always about getting merged. Sometimes it's about solving your own problems, learning a codebase, and sharing the solution for others who might want it too. The code is there. The feature works. And I'm still waiting for the day I can delete my fork because it landed upstream. Until then, `/session save peace-of-mind` is my new best friend. --- PS: I wrote this about 4 months ago, and a lot has changed, we have claude-code, antigravity. But aider is still my favorite AI coding assistant for working with monoliths. --- ## ClickHouse as a Vector Database **Date:** 2026-01-14 **URL:** https://lorbic.com/clickhouse-as-a-vector-database/ **Description:** Vector databases are everywhere now. Pinecone, Weaviate, Milvus - the list goes on. But what if you're already running ClickHouse and don't want yet another database to manage? Turns out, ClickHouse can do vectors too. This post explores what vector databases actually are, why ClickHouse might be a surprisingly good choice, and how to get semantic search running with plain SQL. Everyone's building AI apps now. And every AI app needs a place to stash embeddings. The instant your data grows beyond "fits in memory", you need a vector database. Or do you? If you're already running ClickHouse for analytics, here's some good news: you might not need another database. ClickHouse can hold its own as a vector store. Not because it was built for it, but because it's built to be fast at everything. Let's dig in. ## What Even Is a Vector Database? To understand what a vector database is, we need to start with what vectors are. In the context of machine learning and AI, a vector is a numerical representation of data - often used to represent text, images, audio, or even user behavior. These are not arbitrary numbers; they are structured in such a way that their relative distances capture some idea of *similarity*. For instance, in the world of natural language processing (NLP), embeddings are generated from text using models like BERT or OpenAI's Ada. These embeddings transform language into dense vectors of floating point numbers, often 384 or 768 dimensions long. Words or phrases with similar meanings will have embeddings that are closer together in vector space, while dissimilar ones will be far apart. Here's what that looks like in practice, if we plot words in a simplified 2D vector space, semantically similar words cluster together: > *Vector space: cat and kitten are neighbors, toaster is not invited.* ![Vector space: cat and kitten are neighbors, toaster is not invited](vector-space.png) A vector database, then, is a database that is optimized to store these high-dimensional vectors and perform efficient similarity searches on them. The core operation is typically the k-nearest neighbors (KNN) search: given a query vector, return the most similar vectors (and hence documents, images, etc.) from the database. Unlike traditional databases (e.g. MySQL) which rely on B-trees and hash indexes for exact matches or range queries, vector databases use specialized indexing structures like HNSW (Hierarchical Navigable Small World graphs), IVF (Inverted File Index - cool stuff), and PQ (Product Quantization). These indexes allow for approximate nearest neighbor (ANN) search, which trades a bit of accuracy for massive speed-ups in high-dimensional spaces. Some popular purpose-built vector databases are Pinecone, Weaviate, Milvus, and Faiss (from Meta). However, the recent evolution in OLAP systems like ClickHouse has opened the door to hybrid approaches, combining their analytical power with vector search at big-scale. ## Why ClickHouse? ClickHouse isn't a vector database per se. It's a lightning-fast OLAP column-store database originally developed at Yandex for real-time analytics. It's built for high-throughput aggregations over massive datasets, with features like compression, parallel processing, and late materialization. ### So why use ClickHouse for vectors? **First**, it supports arrays and fixed-size numeric types, including `Array(Float32)`, which makes it possible to store embedding vectors natively. **Second**, ClickHouse has recently introduced native support for vector indexes, including HNSW and flat indexes. This means you can create an ANN index directly on an embedding column and query it using standard SQL syntax. This native support makes it viable to unify analytical and semantic queries in a single system, no need to juggle a separate vector store and an analytics database. **Third**, ClickHouse's scalability and cost-efficiency make it attractive for production workloads. If you're already using ClickHouse for logs, metrics, or analytics, extending it for semantic search can be both convenient and economical. In short: fewer moving parts, same fast queries, one less thing to sync. ## Example with ClickHouse The foundation of most ClickHouse tables is the MergeTree engine. It provides partitioning, ordering, indexing, and background merges, ideal for write-heavy and analytical use cases. To store embeddings, you create a table with a column of type `Array(Float32)`. Here's a simplified schema: ```sql CREATE TABLE content_transcript_segment ( segment_id String, text String, start_time Float64, end_time Float64, embeddings Array(Float32), embedding_dimensions UInt64 ) ENGINE = MergeTree ORDER BY segment_id; ``` Starting from ClickHouse v24.6, you can add a vector index like this: ```sql ALTER TABLE content_transcript_segment ADD INDEX vec_idx embeddings TYPE HNSW('metric_type=cosine') GRANULARITY 1; ``` This creates a cosine similarity-based HNSW index over the embedding vectors. The `GRANULARITY` defines how frequently the index is updated in data parts, a tradeoff between index precision and performance. After creating the index, you must materialize it to populate it: ```sql ALTER TABLE content_transcript_segment MATERIALIZE INDEX vec_idx; ``` From there, queries are straightforward. ## Semantic Search in SQL Once your embeddings and index are in place, querying becomes remarkably ergonomic. Let's say you have a query embedding from an external model, a 384-dimensional `Array(Float32)` value. You can use SQL to find the most similar segments: ```sql SELECT segment_id, text, cosineDistance(embeddings, [your_query_vector]) AS score FROM content_transcript_segment ORDER BY score ASC LIMIT 5; ``` You can even wrap this in a view or a UDF to expose semantic search as a high-level API. This simplicity is what makes ClickHouse compelling. You stay in SQL. You use familiar tooling. You don't need to manage and sync a separate vector database. It's just another analytical query, but instead of filtering on keywords, you're querying on meaning. ## Pain Points and Learnings Nothing's free. Here's what to watch out for: 1. **Indexing Costs**: HNSW indexes aren't free. Adding or materializing the index can be resource-intensive. Batch writes are better than high-frequency single inserts. 2. **Embedding Compatibility**: Your application must ensure all embeddings are of the same dimensionality, dtype (`Float32`), and normalization (e.g., unit length for cosine distance). 3. **Debugging Vectors**: It's easy to make mistakes in preprocessing or vector normalization. Simple metrics like `L2Norm`, `dotProduct`, or histograms can help debug anomalies in your embeddings. 4. **Query Limits**: For now, only certain similarity metrics are supported (`cosineDistance`, `l2Distance`, etc.), and the query vector has to be a literal, not dynamically fetched from a `JOIN` or subquery. Workarounds include injecting the vector at query time from the application layer. 5. **Index Evolution**: As the schema evolves, you may need to rebuild or rematerialize indexes. Something to plan for in CI/CD pipelines. ## What's Next This journey is far from over. There are open questions around live updates, incremental index maintenance, hybrid filtering (semantic + keyword), and ranking models. I'm exploring whether hybrid scoring functions can be created, for example, combining vector similarity with popularity scores or recency. Another area of interest is applying semantic clustering to group text segments automatically, then using these groups to power navigation in long-form content. I'm also experimenting with embedding compression techniques and hashing for storage optimization. High-dimensional vectors can get expensive, and every byte matters at scale. --- ClickHouse isn't a plug-and-play vector database, but it gives you some pretty sharp tools. If you understand the mechanics, how indexes work, how embeddings behave, how OLAP engines think, you can build a system that handles millions of vectors, performs blazing-fast searches, and answers questions that once required a fleet of infrastructure. ClickHouse is the AMD of OLAP databases. Best price-performance ratio. The real value of these tools lie in the kind of questions we can now ask. Go ahead, [sign up for their trial account](https://clickhouse.cloud/?ref=lorbic.com) and build something cool. ## Further Reading - **HNSW Indexes** (many sources because one is never enough): - [Supabase - HNSW Indexes](https://supabase.com/docs/guides/ai/vector-indexes/hnsw-indexes?ref=lorbic.com) - [Pinecone - HNSW](https://www.pinecone.io/learn/series/faiss/hnsw/?ref=lorbic.com) - [Neon - Understanding Vector Search and HNSW](https://neon.com/blog/understanding-vector-search-and-hnsw-index-with-pgvector?ref=lorbic.com) - [ClickHouse - HackerNews Vector Search (Cool story)](https://clickhouse.com/docs/getting-started/example-datasets/hackernews-vector-search-dataset?ref=lorbic.com) - **ClickHouse Vector Search**: - [ClickHouse - Vector Search Documentation](https://clickhouse.com/docs/engines/table-engines/mergetree-family/annindexes?ref=lorbic.com) - [ClickHouse Blog - Vector Search with ClickHouse](https://clickhouse.com/blog/vector-search-clickhouse-p1?ref=lorbic.com) - [ClickHouse - Distance Functions](https://clickhouse.com/docs/sql-reference/functions/distance-functions?ref=lorbic.com) - **Embeddings and Vector Fundamentals**: - [OpenAI - Embeddings Guide (Use this to generate embeddings)](https://platform.openai.com/docs/guides/embeddings?ref=lorbic.com) - [Hugging Face - Sentence Transformers](https://huggingface.co/docs/hub/en/sentence-transformers?ref=lorbic.com) - [Jay Alammar - The Illustrated Word2Vec (Great intro)](https://jalammar.github.io/illustrated-word2vec/?ref=lorbic.com) - **ANN Algorithms Deep Dive**: - [Faiss Wiki - Index Structures](https://github.com/facebookresearch/faiss/wiki?ref=lorbic.com) - [ANN Benchmarks - Compare Algorithm Performance](https://ann-benchmarks.com/index.html?ref=lorbic.com) --- ## Memory Mechanics In Go - Stack vs Heap **Date:** 2026-01-12 **URL:** https://lorbic.com/memory-mechanics-stack-vs-heap-in-go/ **Description:** When thinking about performance, it's easy to focus on Big O notation. But in Go, the difference between the Stack and the Heap is often the difference between a service that scales and one that chokes on GC pauses. This post explores escape analysis, the "Pointer Myth", and why passing by value is often 40x faster than passing by pointer. We often talk about "fast" code in terms of Big O notation or algorithmic complexity. But in systems programming languages like Go, "fast" is often a function of _where_ your data lives in memory. When optimizing for high throughput, efficient loops and database indexes are only part of the story. Eventually, you have to talk about the Stack and the Heap. Understanding the difference isn't just trivia. It is the difference between a service that hums along at 100k OPS with flat latency, and one that chokes on Garbage Collection (GC) pauses every few seconds. This post explores the mechanics of Go's memory management, why the Stack is your friend, and why "using pointers for performance" is often a lie we tell ourselves. ## The Two Worlds: Stack vs Heap Memory in your Go program is effectively divided into two zones. They aren't physically different RAM chips, but they are managed with completely different strategies. ### The Stack: The Fast Lane Every Goroutine in Go gets its own stack. It starts small (2KB) and grows/shrinks dynamically. Think of the stack like a scratchpad. When a function is called, it gets a slice of this scratchpad (a stack frame) to write its variables. When the function returns, that slice is simply marked as "free" by moving a pointer. - **Allocation Cost:** One CPU instruction (subtracting from the stack pointer). - **Cleanup Cost:** Zero (adding to the stack pointer). - **Access:** Extremely fast (L1/L2 cache locality). Technically, Go uses continuous stacks. When a goroutine runs out of stack space, the runtime allocates a new, larger segment and copies the existing stack to it. This "stack copying" is the only time stack memory incurs a significant cost, but it happens rarely compared to function calls. ### The Heap: The Shared Chaos Anything that cannot fit on the stack, or needs to live longer than the function that created it, goes to the Heap. The Heap is a massive pool of shared memory. - **Allocation Cost:** Expensive. The runtime (`runtime.mallocgc`) must find a free block of the right size, potentially hold locks, and update metadata. - **Cleanup Cost:** Very Expensive. This is the domain of the Garbage Collector. It has to scan, mark, and eventually sweep this memory. - **Access:** Slower (pointer chasing, less cache friendly). The golden rule of Go performance: **If it's on the Stack, you don't pay for it. If it's on the Heap, you pay tax on every GC cycle.** ## Escape Analysis: The Compiler's Decision So, how does Go decide? You don't have `malloc` or `free`. You have the compiler. Go uses a technique called **Escape Analysis** to determine the lifetime of a variable. If the compiler can prove that a variable _never_ leaves the function scope, it allocates it on the stack. If the variable "escapes" (e.g., is returned to a caller, stored in a global variable, or sent to a channel), it generally must be moved to the Heap. ### Seeing It In Action You can see this decision-making process by using the `-gcflags` flag. ```go package main type User struct { ID int Name string } func createUserV1() User { u := User{ID: 1, Name: "Vikash"} return u // Returns a VALUE } func createUserV2() *User { u := User{ID: 2, Name: "Vikash"} return &u // Returns a POINTER } func main() { _ = createUserV1() _ = createUserV2() } ``` Compile with analysis enabled: ```bash $ go build -gcflags="-m" main.go ./main.go:14:2: moved to heap: u ``` - `createUserV1`: Returns `User` by value. The data is copied to the caller's stack frame. `u` stays on the stack. **Fast.** - `createUserV2`: Returns `&u`. The caller needs to access variables created inside `createUserV2` _after_ the function returns. If `u` were on the stack, it would be overwritten by the next function call. The compiler _must_ move `u` to the heap. **Slow.** ## Hidden Escape Routes Returning a pointer is the obvious escape route. But there are subtle ones that catch even experienced engineers. ### 1. Interfaces (Dynamic Dispatch) When you assign a concrete value to an interface, the runtime often needs to store the type information along with the data. If the compiler cannot deduce the type at compile time, or if the method call involves dynamic dispatch, the value often escapes. ```go func Log(v interface{}) { fmt.Println(v) // 'v' escapes to heap because fmt.Println takes interface{} } ``` This is why passing huge structs to `log.Println` or `json.Marshal` kills performance, it forces them onto the heap. ### 2. Slices with Dynamic Size A slice on the stack must have a known size at compile time. If the size is determined by a variable, it often escapes. ```go func makeSpace(n int) { x := make([]byte, n) // Escapes to heap } func makeFixed() { x := make([]byte, 64) // Stays on stack } ``` ### 3. Closure Capture If a closure (anonymous function) references a variable from the outer scope, and that closure is passed around, the closed-over variable escapes. ## Benchmarking the Cost Let's prove the "Pointer Myth" wrong with data. We will compare passing a small struct (64 bytes) by value vs. by reference. ```go // Benchmark Code type Config struct { ID int Name string Data [50]byte // Padding to make it ~64 bytes } //go:noinline func byValue(c Config) int { return c.ID } //go:noinline func byPointer(c *Config) int { return c.ID } ``` **Results:** ```text BenchmarkByValue-10 1000000000 0.30 ns/op 0 B/op 0 allocs/op BenchmarkByPointer-10 100000000 12.50 ns/op 64 B/op 1 allocs/op ``` - **Pass by Value:** 0.3ns. Zero allocations. The CPU simply copies the registers. - **Pass by Pointer:** 12.5ns. One allocation. - **Difference:** 40x slower. Why? Because `byPointer` creates a heap allocation (in this specific microbenchmark setup where the pointer effectively escapes or the compiler decides to alloc). In real-world code, even if it doesn't always alloc, the _pressure_ on the GC adds up. ## Practical Advice for Optimization When optimizing: 1. **Default to "Pass by Value".** Current CPUs can copy 64 bytes faster than you can blink. Do not use pointers just to avoid copying small structs. 2. **Watch your Interfaces.** Heavy use of `interface{}` in hot paths (like middleware or data ingest loops) is a common source of hidden allocations. 3. **Pre-allocate Slices.** Use `make([]T, 0, capacity)` where `capacity` is a constant if possible, or at least a calculated max, to avoid resizing allocations. 4. **Profile `mallocgc`.** If `runtime.mallocgc` is more than 5% of your CPU profile, you have a churn problem. Use `go build -gcflags="-m"` to find the leaks. ## Summary - **Stack** = Fast, local, free. - **Heap** = Slow, shared, taxed by GC. - **Pointers** = Cause heap allocations. Use them for _semantics_ (I need to modify this), not for _performance_ (unless the struct is > 2KB). Memory management in Go is about empathy for the runtime. If you treat the Garbage Collector like a colleague who is already overworked, you'll naturally write code that stays on the stack, keeps the heap clean, and scales effortlessly. ## Further Reading 1. **[Language Mechanics On Escape Analysis](https://www.ardanlabs.com/blog/2017/05/language-mechanics-on-escape-analysis.html)** – Ardan Labs (William Kennedy) - _The definitive 4-part series on how the Go compiler makes allocation decisions._ 2. **[A Guide to the Go Garbage Collector](https://go.dev/doc/gc-guide)** – Official Go Documentation - _Deep dive into the tri-color mark-and-sweep algorithm and tuning `GOGC`._ 3. **[Go SliceTricks](https://go.dev/wiki/SliceTricks)** – Go Wiki - _Efficiency patterns for manipulating slices without unnecessary allocations._ --- ## OLTP vs OLAP - Why You Need Two Databases **Date:** 2026-01-11 **URL:** https://lorbic.com/oltp-vs-olap-why-you-need-two-databases/ **Description:** "The database that runs your app cannot be the database that analyzes your app". It's a hard lesson learned at scale. Early on, Postgres does it all. But as you hit massive scale, your analytics queries start killing your login APIs. This post breaks down the physics of Row-oriented (Couchbase) vs Column-oriented (ClickHouse) databases, and how to bridge them using Change Data Capture (CDC) for a robust, lag-free architecture. At a recent ClickHouse conference in New Delhi, I attended this Saturday. There were many interesting sessions, but one that stood out was a talk on multi-tenant analytics at scale. I was reminded of a fundamental truth in backend engineering: "The database that runs your app cannot be the database that analyzes your app". Early in a startup's life, we shove everything into one database. User profiles, session data, logs, and analytics events all live in the same Postgres, Mongo, or Couchbase instance. It works, until it throttles. Eventually, your analytics queries start creating backpressure on your login API. That is when you realize you have two distinct problems pretending to be one. You need to split your world into OLTP (Online Transaction Processing) and OLAP (Online Analytical Processing). In my stack, this split is represented by two heavyweights: **Couchbase** and **ClickHouse**. > **Note:** While this post focuses on Couchbase and ClickHouse because they are part of my stack, these principles are universal. > * **OLTP Examples:** Postgres, MySQL, MongoDB, DynamoDB. > * **OLAP Examples:** Snowflake, BigQuery, Redshift, DuckDB. > > The physics of Row vs Column remains the same regardless of the brand. ## The fundamental Divergence: Row vs Column The difference isn't just branding. It is physics. ### Couchbase (The OLTP Workhorse) Couchbase is designed for **Transactions**. It is a document store that excels at "Key-Value" lookups. Imagine you are fetching a User Profile. You want all the data for *one specific user* (Name, Email, Preferences, LastLogin). In a Row-Oriented database (and for the sake of simplicity, a JSON Document is just a fancy Row), this data is stored contiguously on disk. * **The "Row" Concept:** Whether it is a PostgreSQL Tuple or a Couchbase Document, the principle is the same: all fields for one ID are stored together. * **Read Pattern:** Fetch row `user:123`. The disk head jumps to one spot and reads the whole block. **Fast.** * **Write Pattern:** Update `user:123`. The DB locks that specific row/document, updates it, and moves on. **Fast.** ### ClickHouse (The OLAP Beast) ClickHouse is designed for **Analytics**. It is a Column-Oriented store. Imagine you need to calculate the average age of 50 million users. You don't care about their names, emails, or session IDs. * **The "Column" Concept:** ClickHouse stores every `age` value for every user together on disk. All `names` are in a separate physical location. * **Read Pattern:** To calculate an average, the system only reads the `age` file. It skips 90% of the data it doesn't need. **Blazing Fast for Aggregations.** * **Write Pattern:** Updating a single record is "expensive" because it involves touching many different files. ClickHouse is optimized for appending massive batches of immutable data. **High Throughput.** ### See It In Action Let's say you have `users.json` (1TB of Data). **Query:** `SELECT AVG(age) FROM users` * **OLTP (Row/Doc):** 1. Load Block 1 (User A). 2. Parse JSON/Row. 3. Extract `age`. 4. Discard rest. 5. Repeat for 1 billion users. * **Result:** Massive I/O waste. Your disk is reading "Name","Email","Bio" just to throw it away. * **OLAP (Column):** 1. Load `age.bin` (Just the integers). 2. Send array to CPU. 3. **Vectorized Processing (SIMD):** The CPU adds 4 or 8 integers *in a single instruction* (AVX-512). I'll write about SIMD in a future blog. 4. Zero branching, zero parsing overhead. * **Result:** 100x faster execution. This is why ClickHouse can scan billions of rows per second on a modern laptop. ## The Architecture of the Split So, how do you use both without going crazy? You need a bridge. At this scale of our example, we use Couchbase for the "Live" path and ClickHouse for the "Retrospective" path. 1. **The Live Path (Couchbase):** * User logs in? Hit Couchbase. * User posts a comment? Write to Couchbase. * **Requirement:** Sub-millisecond latency, high concurrency, strong consistency (for the user). 2. **The Bridge (CDC - The "Real" Engineering):** * **The Amateur Move (Dual Writes):** ```go // Don't do this db.SaveUser(user) clickhouse.InsertUser(user) // What if this fails? Now your analytics is drifting. ``` * **The Pro Move (Change Data Capture or CDC):** We treat the database log (WAL or DCP Stream) as the source of truth. * User updates profile -> Couchbase commits to disk -> Couchbase emits event. * **Kafka/Redpanda** captures this event. * **ClickHouse Connector** consumes the batch and inserts it. * **Why this matters:** If ClickHouse goes down for maintenance, Kafka buffers the events. When ClickHouse comes back up, it drains the buffer. **Zero data loss. Zero impact on the Login API.** 3. **The Retrospective Path (ClickHouse):** * Product Manager wants to know: "How many users logged in from iOS vs Android last month?" * Dashboard query hits ClickHouse. * **Requirement:** Throughput. Scanning millions of rows in milliseconds. Latency can be 200ms-500ms, but it must handle massive volume. ## Why Not Just Use Postgres for Everything? "But Postgres has columnar indexes now!" or "What about TimescaleDB?" Yes, tools are converging. Postgres is the swiss-army knife of databases, and for 90% of use cases, it is enough. But at high scale, general-purpose tools hit hard limits where **specialization wins**. ### 1. The WAL Bottleneck (Ingestion) Postgres guarantees ACID compliance for every transaction. This means every insert must be written to the **Write Ahead Log (WAL)** before it is visible. * **Postgres:** High write volume = High WAL contention. If you try to insert 100k events/sec into Postgres, your WAL becomes the choke point. * **ClickHouse:** Uses `MergeTree` engines. Data is written to parts in memory and flushed to disk as immutable files. There is no traditional "WAL" for every row. It is designed to swallow firehoses of log data. ### 2. The Maintenance Tax (Vacuum vs Merges) * **Postgres:** Updates are "Copy on Write". Old versions of rows are kept around (dead tuples). You rely on `AUTOVACUUM` to clean them up. At terabyte scale, Vacuuming is a heavy background process that eats I/O. * **ClickHouse:** Updates are rare. It relies on **background merges** to combine data parts. This is efficient for write-once-read-many (WORM) patterns typical of analytics. ### 3. Resource Isolation If you run a heavy analytical query (`GROUP BY` over 1 billion rows) on your primary Postgres instance: * It poisons the page cache (evicting hot OLTP usage data). * It maxes out CPU. * **Result:** Your user login times out because the DB is busy calculating "Average Age". By splitting them, you ensure that your **Analysts' dashboard queries never bring down the Checkout page.** --- The key to a scalable architecture is knowing the tool's purpose. * **OLTP (Couchbase):** Optimized for the **individual**. "Who is User X?" * **OLAP (ClickHouse):** Optimized for the **aggregate**. "What are users doing?" Don't force one to do the other's job. Your on-call rotation (and your latency SLOs) will thank you. --- ## Further Reading 1. **[OLTP vs. OLAP](https://clickhouse.com/resources/engineering/oltp-vs-olap)** – Clickhouse Engineering * *An overview of the fundamental differences between transaction processing and analytical processing.* 2. **[ClickHouse MergeTree Family](https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree)** – ClickHouse Docs * *Deep dive into the LSM-tree inspired architecture that powers high-speed analytics.* 3. **[PostgreSQL and ClickHouse for Analytics](https://clickhouse.com/comparison/postgresql)** – ClickHouse * *A benevolent comparison helping you decide when to stick with Postgres and when to introduce ClickHouse.* 4. **[Couchbase Architecture Overview](https://docs.couchbase.com/server/current/learn/architecture-overview.html)** – Couchbase Docs * *Understanding the memory-first, shared-nothing architecture of a modern document store.* --- ## Go GC Deep Dive: How to Reduce Latency and Allocation Pressure in Production **Date:** 2026-01-07 **URL:** https://lorbic.com/dont-take-out-the-garbage-go-gc-deep-dive/ **Description:** "Why is our service slow?" "I don't know, the heap is only 200MB". "But we're allocating... wait, how much?" "12 terabytes". "...in how long?" "30 seconds profile". That's when we realized: we weren't running a service. We were running a garbage factory that occasionally served API requests. The Go garbage collector was heroically trying to clean up our mess, and we were blaming it for not cleaning fast enough. This deep dive into GC internals, profiling tools, and production war stories will teach you how to stop fighting the garbage collector and start working with it. In the world of high-throughput backend services, we often obsess over the usual suspects of performance: database indexing, network latency, and algorithmic complexity. But recently, while debugging our core gateway service (`backend-gw`), I encountered a bottleneck that defied standard logic. The service was **CPU-bound**, yet active heap usage was surprisingly low (~200MB). P99 latency was spiking at random intervals, but database queries were returning in milliseconds. The culprit was not the business logic. It was **memory management**. I was effectively running a Denial-of-Service attack on my own runtime. This post is a detailed breakdown of the Go Garbage Collector (GC). I'll explain how the GC *actually* works, dissect its specific phases (including the dreaded "Stop The World"), and show you how to use the Go Trace tool to identify when your application is losing the battle against allocation churn. ## Part 1: Finding the Smoking Gun My investigation began with a standard CPU profile. The service processes JSON documents from Couchbase and serves them via HTTP. I expected the CPU time to be dominated by business rules or I/O handling. Instead, the profile (`cpu_top_cumulative.txt`) showed this: ```text Showing nodes accounting for 29.17s, 72.56% of 40.20s total flat flat% sum% cum cum% 0.02s 0.05% ... 19.04s 47.36% couchbaseDB.GetWithCAS 0 0% ... 18.40s 45.77% encoding/json.Unmarshal 0.64s 1.59% ... 13.89s 34.55% encoding/json.(*decodeState).object ... 0.48s 1.19% ... 1.94s 4.83% runtime.scanobject 0.02s 0.05% ... 1.81s 4.50% runtime.gcDrain ``` I had **two major bottlenecks** consuming roughly equal CPU time: 1. **Database operations** (47.36% cumulative) - Couchbase fetch and I/O 2. **JSON deserialization** (45.77% cumulative) - Standard library unmarshalling But crucially, functions like `runtime.scanobject` and `runtime.gcDrain` were consuming significant cycles. These are internal GC methods. While only 4.83% of CPU was directly in GC functions, this was a sign of a deeper problem. The allocation profile (`allocs_top_cumulative.txt`) confirmed my suspicion. We weren't leaking memory, we were "churning" it. Allocation profile shows cumulative allocations since last deployment (4 hours in this case). ```text Showing nodes accounting for 11054705.39MB, 89.25% of 12386152.87MB total 2981421.22MB 24.07% github.com/go-kit/log.With 216546.74MB 1.75% encoding/json.Unmarshal ``` ### Understanding the Gap: Heap vs. Allocations Here's the critical insight that I had missed. The heap profile showed: ```text Type: inuse_space Showing nodes accounting for 188.98MB, 92.83% of 203.58MB total ``` Let that sink in: - **Active Heap:** 203 MB (memory in use at snapshot time) - **Total Allocations:** 12,386 GB (cumulative garbage created since last deployment) - **Churn Ratio:** 62,000:1 (cumulative allocations / active heap) We were generating **twelve terabytes** of cumulative garbage while maintaining only 200MB of live data. This meant that for every byte we needed, we were creating 62,000 bytes of garbage that the GC had to clean up. **This is allocation churn**, and it's the silent killer. The GC must scan and mark every reachable object during collection, and when you're creating millions of temporary objects, the GC spends more time cleaning than the application spends doing actual work. We were generating this garbage using standard library JSON parsers and an inefficient logging strategy. To understand why this crushed our CPU, I had to look under the hood of the Go Runtime. Then I started reading about the Go GC. I had some familiarity with Python's GC. So, I started reading countless articles, reading Go docs and reading the source code. After about 25 hours, I finally got some clarity on how the GC works and why the backend was so slow. ## Part 2: The Architecture of the Go GC The Go GC is a **Non-Generational, Concurrent, Tri-Color Mark-and-Sweep** collector. That is a mouthful. Let's break down the implications: 1. **Non-Generational:** Unlike Java or Python, Go does not separate objects into "Young" and "Old" generations. It treats the heap as one singular space. This simplifies the runtime but means the GC must scan *everything* that is live during every cycle. 2. **Concurrent:** The GC runs *alongside* your application code (except during brief Stop-The-World phases). It does not pause your app for the entire duration of the collection. 3. **Mark-and-Sweep:** It traverses the object graph to find live objects ("Mark") and reclaims the rest ("Sweep"). ### The Object Graph You must visualize your application's memory as a directed graph. * **Nodes:** Objects (structs, strings, slices). * **Edges:** Pointers (references). The GC starts at the **Roots** (global variables, stack frames of active goroutines) and follows the pointers. Anything it can reach is "Live". Anything it cannot reach is "Garbage". ## Part 3: The Lifecycle of a GC Cycle (The 4 Phases) A GC cycle is a carefully choreographed dance between your code (the "Mutator") and the GC background workers. It consists of four specific phases. ### Phase 1: Sweep Termination (Stop-The-World) The cycle begins here. The GC must ensure that the *previous* cycle's cleanup is 100% complete before starting a new one. * **What happens:** The runtime pauses **all** application goroutines (Stop-The-World, or STW). It finishes any remaining sweeping of memory spans. * **Latency Impact:** In modern Go versions (1.18+), this is typically sub-millisecond. However, if your heap is highly fragmented or massive, this pause can stretch, causing "jitter" in your p99 latency. ### Phase 2: Concurrent Mark (The Heavy Lifter) This is the longest phase, accounting for the bulk of the GC-related CPU usage I saw in my profiles. The world resumes, and the GC runs in the background (consuming ~25% of CPU capacity by default). This phase uses the **Tri-Color Invariant**: 1. **White:** Candidate for collection (not yet visited). 2. **Grey:** Alive, but its children (pointers) have not been scanned. 3. **Black:** Alive, and fully scanned. The GC picks a Grey object, scans its memory for pointers (`runtime.scanobject`), marks the referenced objects Grey, and turns the original object Black. **The Bottleneck:** The cost of this phase is determined by **Pointer Density**, not just object size. ```text Cost = (BytesScanned * C_1) + (PointerCount * C_2) ``` In my profiling data, I saw `runtime.scanobject` consuming 4.83% of CPU. This is because `encoding/json` creates a graph full of `interface{}` and pointer wrappers. I was forcing the GC to traverse millions of tiny edges. ### Phase 3: Mark Termination (Stop-The-World) Once the Concurrent Mark is done, the GC must ensure no Grey objects remain. * **What happens:** A second Stop-The-World (STW) pause. The GC performs final housekeeping, flushes caches, and calculates the target heap size for the *next* cycle. * **Latency Impact:** This pause is generally longer than Sweep Termination. Its duration scales with the number of active goroutines and the complexity of the root set. ### Phase 4: Concurrent Sweep The STW pause ends. The GC now knows that any object still marked "White" is garbage. * **What happens:** The GC does not explicitly "delete" every object immediately. Instead, as your application requests *new* memory (via `mallocgc`), the runtime lazily reclaims the White space. * **Resource Usage:** This happens largely off the critical path, but high churn can cause fragmentation here. ## Part 4: Analyzing the Trace – The "G Block" and Mark Assists This is where the theory meets the reality of debugging. When you run `go tool trace trace.out`, you are presented with a timeline view. ### What is a "G Block"? In the trace tool, you will see rows labeled **Proc 0, Proc 1, ...** (representing logical Processors/Cores). The colorful bars inside these rows are **G Blocks** (Goroutine execution blocks). A "G Block" represents a specific Goroutine running on a specific Processor for a duration of time. * **Green:** User code running normally. * **Blue:** Networking/IO. * **Yellow/Red:** GC related delays or synchronization waiting. ### The Latency Killer: Mark Assists The Go GC has a "Pacing" algorithm. It aims to prevent the heap from growing out of control before the GC cycle finishes. **The Concept:** Imagine your application is allocating memory (creating garbage) faster than the background GC workers can clean it up. The runtime realizes it is losing the race. To prevent an Out-Of-Memory (OOM) crash, it begins to "tax" your application. **The Mechanism:** When a Goroutine attempts to allocate memory (`runtime.mallocgc`), the runtime checks the current GC credit. If the Goroutine is in "debt" (it has allocated too much relative to the scan progress), the runtime **hijacks** the Goroutine. Instead of running your business logic, your Goroutine is forced to switch into **Mark Assist** mode. It must scan a portion of the heap (doing the GC's job) to "pay off its debt" before it is allowed to perform the allocation. ### Interpreting the Trace If you see this in your trace, you have an allocation problem: 1. Look at the **"G" row** (Goroutine analysis). 2. Look for the state labels. You want to see **"Running"**. 3. If you see large chunks labeled **"Mark Assist"** or time spent in **"GCWaiting"** states, your CPU is not serving users; it is fighting the heap. In my specific case, my API handlers were spending significant time in Mark Assist because `log.With` and `json.Unmarshal` were flooding the heap with pointers. ## Part 5: The Dual Problem - JSON and Logging ### The JSON Unmarshalling Problem While I can't optimize the database (network and disk I/O are physical constraints), I *can* optimize how I process the data once it arrives. The standard library's `encoding/json` package is general-purpose and safe, but it's not optimized for high-throughput scenarios. The issue is **how** it allocates memory: - Creates many small allocations for each field - Uses `interface{}` extensively, forcing heap allocations - Generates intermediate objects during parsing - Creates pointer-heavy object graphs ### The Logging Problem (The Silent Memory Hog) Looking deeper at the allocation profile: ```text 2981421.22MB 24.07% github.com/go-kit/log.With 464404.86MB 3.75% backend-gw/routes/rctx.LoggerWithFn ``` The go-kit logger was allocating **2.9 TB of memory** - nearly 24% of all allocations! This is because `log.With()` creates a new logger instance every time it's called, and we are calling it in HTTP middleware and every function for every single request. Every log line with context creates: - A new logger instance - Copies of all context key-value pairs - String conversions for values - Interface boxing for non-string types When you're handling thousands of requests per second, this adds up catastrophically. **Why This Compounds the GC Problem:** These allocations don't just cost memory, they create millions of objects that the GC must scan during the mark phase. Even though each logger instance is short-lived, the GC must still prove it's garbage by scanning the entire object graph. ## Part 6: The Fix and The Benchmarks I replaced `encoding/json` with `goccy/go-json`, a drop-in replacement that's heavily optimized for performance. The impact was visible in micro-benchmarks immediately. ### Benchmark Setup ```go func BenchmarkUnmarshal_Std_UserSE(b *testing.B) { data := []byte(`{"userId":"..".,"sessions":[...],...}`) // Representative data b.ReportAllocs() b.ResetTimer() for i := 0; i < b.N; i++ { var result UserSE if err := json.Unmarshal(data, &result); err != nil { b.Fatal(err) } } } func BenchmarkUnmarshal_Goccy_UserSE(b *testing.B) { data := []byte(`{"userId":"..".,"sessions":[...],...}`) b.ReportAllocs() b.ResetTimer() for i := 0; i < b.N; i++ { var result UserSE if err := goccyjson.Unmarshal(data, &result); err != nil { b.Fatal(err) } } } ``` ### Benchmark Results: The "Pointer Chasing" Proof I benchmarked the unmarshalling of several of our critical database document types on Apple M2: | Struct Type | Library | Time (ns/op) | Allocs/op | Bytes/op | Speedup | Alloc Reduction | |:---|:---|---:|---:|---:|---:|---:| | **UserSE** | encoding/json | 127,135 | **589** | 33,625 | - | - | | | goccy/go-json | 27,118 | **273** | 32,637 | 4.7x | 53.7% | | **UserEE** | encoding/json | 133,181 | **347** | 29,745 | - | - | | | goccy/go-json | 25,736 | **154** | 36,283 | 5.2x | 55.6% | | **User** | encoding/json | 4,819 | **22** | 2,392 | - | - | | | goccy/go-json | 1,095 | **9** | 2,545 | 4.4x | 59.1% | | **Post** | encoding/json | 6,685 | **18** | 1,184 | - | - | | | goccy/go-json | 1,621 | **11** | 1,689 | 4.1x | 38.9% | | **UserENR** | encoding/json | 10,345 | **44** | 1,704 | - | - | | | goccy/go-json | 2,286 | **16** | 2,533 | 4.5x | 63.6% | **The Analysis:** Across all document types, I saw: - **Average speedup:** 4.6x faster unmarshalling - **Average allocation reduction:** 54% fewer allocations - **Bytes per operation:** Roughly similar (slight increase in some cases) The byte count (`B/op`) remained similar or slightly higher in some cases. **This is the key insight.** The GC does not care about the total bytes allocated as much as it cares about the **number of objects** (allocations) it must track. When unmarshalling `UserSE`: - Standard lib: 589 separate allocations = 589 objects to scan - goccy/go-json: 273 allocations = 273 objects to scan This change effectively **cut the work of `runtime.scanobject` by more than half** for these hot paths. ### Why goccy/go-json Is Faster The performance difference comes from: 1. **Reduced intermediate allocations** - Directly writes to destination structs 2. **Optimized pointer handling** - Minimizes pointer chains in the object graph 3. **Better memory layout** - Fewer separate heap allocations 4. **SIMD optimizations** - Uses CPU vector instructions where available The slight increase in bytes/op for some structs is acceptable because it reduces allocation count, the metric that actually drives GC pressure. ### Next Steps: Logging Optimization My next phase of optimization is to replace the go-kit logger with a zero-allocation alternative like zerolog or zap, and to restructure the logging middleware to reduce the quantity and improve the quality of logs. Based on my profiling data showing 2.9TB allocated by `log.With()`, I anticipate this will: - Eliminate ~600GB per hour of memory churn - Free up an additional 15-20% of CPU cycles - Further reduce GC pressure ## Architectural takeaways ### For Senior Engineers: Rethink "Clean Code" We often prefer interfaces, wrappers, and layers of abstraction for "clean architecture". In Go, every interface wrapper that escapes to the heap is a node in the graph. * **The Trade-off:** Be wary of libraries that accept `interface{}` for everything (like standard `log` or `json`). * **Structure:** Design structs to keep pointers contiguous. The GC scans memory faster when pointers are grouped at the start or end of a struct, rather than interleaved with scalars (int/bool), due to how bitmap marking works. ### For Junior Engineers: Value vs. Pointer * **Pointers are not free.** Passing a pointer to a function might save a memory copy, but it might burden the GC if that pointer escapes to the heap. * **Pre-allocate Slices.** Use `make([]T, 0, capacity)` if you know the size. Appending to a slice causes resizing, which creates a *new* backing array and leaves the old one as garbage. ### Red Flags: Is Your Application GC-Bound? Watch for these warning signs in your profiles: 1. **Allocation/Heap Ratio > 1000:1** - You're churning far more than you're using 2. **`runtime.scanobject` > 5% of CPU** - GC is spending significant time scanning 3. **Many small allocations** - Objects < 1KB in hot paths create pointer-chasing 4. **`interface{}` in unmarshal targets** - Forces heap allocation for every value 5. **Logger allocations in middleware** - Creating context on every request ### General Optimization Strategies 1. **Use `sync.Pool` for frequently allocated objects** - Reuse instead of allocating 2. **Prefer value types over pointers** - When objects are small (<100 bytes) 3. **Consider binary protocols** - protobuf or msgpack have less allocation overhead 4. **Batch allocations** - Pre-allocate slices to known capacity 5. **Profile before and after** - Always measure; intuition fails at scale ## Conclusion The Go Garbage Collector is a marvel of engineering, prioritizing low latency over high throughput. However, it obeys the laws of physics. If you create millions of objects, the GC must touch millions of objects. By analyzing my profiles and traces, I realized the service wasn't slow, it was just distracted. The CPU was spending its time: - Fetching data from Couchbase (necessary I/O) - Deserializing JSON with inefficient allocations (optimizable) - Creating logger instances for every request (optimizable) - Running the GC to clean up the resulting garbage (consequence) By addressing the two optimizable factors, switching to `goccy/go-json` and planning to fix the logging, I'm reclaiming significant CPU capacity. I reduced allocation counts by 50%+, which directly reduces the work that `runtime.scanobject` must perform during each GC cycle. **The lesson:** In high-performance Go services, allocation count matters more than allocation size. The GC must scan the object graph, and every object is a node. Minimize the nodes, and you minimize the GC's work. Understanding your memory allocation patterns through profiling is not optional, it's essential for building services that scale. **Reference Documentation:** * [A Guide to the Go Garbage Collector (Official)](https://tip.golang.org/doc/gc-guide) * [Go Tool Trace User Guide](https://go.dev/doc/diagnostics#trace) * [Go Memory Model](https://go.dev/ref/mem) * [goccy/go-json on GitHub](https://github.com/goccy/go-json) **Profiling Commands Used:** ```bash # CPU profiling go test -cpuprofile=cpu.prof -bench=. ./... go tool pprof -top cpu.prof # Memory allocation profiling go test -memprofile=mem.prof -bench=. ./... go tool pprof -alloc_space -top mem.prof # Heap profiling go tool pprof -inuse_space -top mem.prof # Generating trace go test -trace=trace.out -bench=. ./... go tool trace trace.out ``` **Collect profiling data from production systems:** ```bash # 1. Collect the raw data { # 0. CPU Profile: Identifies exactly which line is creating the most garbage. curl -L -u pprofuser:pwd "http://backend-gw/debug/pprof/profile?seconds=30" -o cpu.pprof & # 3. Execution Trace: Captures a 10-second timeline of every event (GC, Network, Scheduler). curl -L -u pprofuser:pwd "http://backend-gw/debug/pprof/trace?seconds=15" -o trace.out & sleep 2 # 1. Heap Profile: Identifies exactly which line is creating the most garbage. curl -L -u pprofuser:pwd "http://backend-gw/debug/pprof/heap" -o heap.pprof # 2. Goroutine Profile: Shows what all concurrent workers are doing at this exact moment. curl -L -u pprofuser:pwd "http://backend-gw/debug/pprof/goroutine" -o goroutine.pprof # Block Profile: Shows where code is waiting for Channels or Network. curl -L -u pprofuser:pwd "http://backend-gw/debug/pprof/block" -o block.pprof # Mutex Profile: Shows which locks are causing contention. curl -L -u pprofuser:pwd "http://backend-gw/debug/pprof/mutex" -o mutex.pprof # Allocation profiles curl -L -u pprofuser:pwd "http://backend-gw/debug/pprof/allocs" -o allocs.pprof wait echo "All 30s/15s profiles collected." } # 2. Convert the data into readable format (you can use web version as well) { go tool pprof -top -cum cpu.pprof > cpu_top_cumulative.txt go tool pprof -tree cpu.pprof >> cpu_tree.txt go tool pprof -top -cum heap.pprof > heap_top_cumulative.txt go tool pprof -tree heap.pprof >> heap_tree.txt go tool pprof -top -cum allocs.pprof > allocs_top_cumulative.txt go tool pprof -tree allocs.pprof >> allocs_tree.txt go tool pprof -top -cum block.pprof > block_top_cumulative.txt go tool pprof -tree block.pprof >> block_tree.txt go tool pprof -top -cum mutex.pprof > mutex_top_cumulative.txt go tool pprof -tree mutex.pprof >> mutex_tree.txt go tool pprof -top goroutine.pprof >> goroutines.txt go tool trace -pprof=sync trace.out > trace_sync.pprof go tool pprof -top trace_sync.pprof > trace_summary.txt } ``` Then you can open these text files in any text editor to view the profiles. You can also upload these to an LLM to get insights. I used Claude Opus 4.5 for correlating these prifiles with my p90 and p99 metrics and keeping them open on the side to read and cross-reference. While you are reading the profiles you should also look at allocation patterns in your code and make notes of them to see if you can optimize them. **Note:** If you use goccy/go-json, test your code extensively. It is a drop-in replacement for encoding/json as claimed, but it has some issues parsing multibyte characters in some languages like Hindi. PS: Just deployed to prod, and p90 and p99 are looking much better. Next: I will be testing the bytedance/sonic library and see how it's JIT and SIMD features compare to goccy/go-json. And maybe I will try to build a simple transcoder to learn more about GC, SIMD, JIT and systems programming. I am liking this a lot. Until next time. --- ## Introducing CouchLens: A Query Analysis Tool for Couchbase **Date:** 2025-12-28 **URL:** https://lorbic.com/couchlens-couchbase-query-analysis-tool/ **Description:** CouchLens is a browser-based tool for analyzing Couchbase N1QL query performance. It parses system tables, extracts execution plans, and generates insights to help database administrators find performance bottlenecks without sending data to external servers. This post explains what it does, how to use it, and what features are coming. CouchLens is a client-side web application for analyzing Couchbase N1QL query performance. You feed it JSON exports from `system:completed_requests`, `system:indexes`, and schema inference results. It parses execution plans, computes metrics, detects inefficiencies, and generates a report showing where your queries are slow. Everything runs in the browser. No data leaves your machine. The tool is a Progressive Web App, so you can install it and use it offline. The goal is to give database administrators and developers a way to understand query behavior without writing custom scripts or debugging raw JSON. ### Why I Built This I work for an organization that relies heavily on Couchbase. Although my background was primarily in traditional SQL databases (MySQL, MariaDB, PostgreSQL) and occasionally MongoDB, being introduced to Couchbase here was a revelation. I fell in love with the capability of running SQL on JSON via SQL++ (formerly N1QL). This enthusiasm led me to dive deep into Couchbase's internals. I constantly read about how the query planner, indexing strategies, cost-based optimizer, and storage engines function. Concepts I explored in {{< link href="/what-couchbase-taught-me-about-system-thinking" text="What Couchbase Taught Me About System Thinking" target="_blank" >}} I built this tool as a way to understand these concepts better and to share my learnings. **You can give it a try at {{< link href="https://couchlens.lorbic.com/#input?ref=lorbic.com" text="couchlens.lorbic.com" target="_blank" >}}.** Looking ahead, I am working on an index generation engine. This engine will look at your document schema and query patterns to suggest covering indexes or identify inefficiencies in existing ones. These features are currently in development and will eventually be integrated into CouchLens. This post describes what CouchLens does now (version 0.3 alpha), how it works, and what features are planned. --- ### What It Does CouchLens provides eight main views organized into tabs: #### Dashboard Charts showing query throughput over time, average durations, and statement type distribution. Use this to identify periods of high load or performance degradation. ![CouchLens Dashboard displaying query throughput charts and duration metrics](couchlens.lorbic.com_dashboard.webp) #### Insights Automated detection of performance issues. The insight engine scans execution plans and flags queries that match 11 predefined patterns: **Index Issues:** - Inefficient index scans (scanned 50k+ rows but returned less than 10%) - Pagination over-fetch (ORDER BY + LIMIT + OFFSET scanning 10k+ items) - Primary index usage (full bucket scans) **Pattern Analysis:** - SELECT \* usage - LIKE with leading wildcards - Missing WHERE clauses - Slow USE KEYS operations (taking more than 20ms) **Resource Issues:** - High memory usage (more than 10MB per query) - High kernel time (more than 10ms, indicating OS scheduling pressure) - Slow parse or plan phases (more than 5ms) **Performance:** - Long-running queries (more than 1 second) - Large result sets (more than 20MB) - Concurrent query conflicts (heuristic based on fetch latency, index scan throughput, and kernel time ratio) Each insight includes affected query counts, severity (critical/warning/info), and recommendations. ![CouchLens Insights tab highlighting performance issues and recommendations](couchlens.lorbic.com_insights.webp) #### Analysis Statistical breakdown of query performance: p50, p95, p99 latencies, min/max/average durations, and performance grouped by statement type (SELECT, INSERT, UPDATE, etc.). Lists the top 10 slowest queries. ![CouchLens Analysis tab showing query performance statistics including latency percentiles and slowest queries](couchlens.lorbic.com_analysis.webp) #### Queries A sortable, searchable table of every query in the dataset. Sort by duration, result count, or payload size. Full-text search to find specific SQL patterns. Case-insensitive filtering. #### Flow A Sankey diagram showing which queries use which indexes. Visualizes index usage patterns. Helps identify underutilized indexes or queries hitting primary indexes. Includes text filters to include or exclude specific index names. ![CouchLens Flow diagram visualizing the relationship between queries and indexes](couchlens.lorbic.com_flow.webp) #### Indexes A filterable table of index definitions with search by name or definition, status badges (online/building/offline), and copy-to-clipboard support. #### Schema Hierarchical view of buckets, scopes, and collections with type analysis for each field. Detects mixed-type fields (where a field is a number in some documents and a string in others) and highlights date/time fields and unique identifiers. #### Report Selectively generate a printable report combining multiple sections (Dashboard, Insights, Analysis, etc.) into a single view. Export to PDF via browser print. ![CouchLens Report view for generating printable summaries](couchlens.lorbic.com_report.webp) The tool also includes integrated documentation (Getting Started, SQL Queries reference, User Guide, and Changelog) accessible from the Help tab. --- ### How to Use It **Collect Data** Run these queries in Couchbase Query Workbench: ```sql -- Completed Requests SELECT *, meta().plan FROM system:completed_requests; ``` ```sql -- Indexes SELECT s.name, s.id, s.metadata, s.state, s.num_replica, CONCAT("CREATE INDEX ", s.name, " ON ", k, ks, p, w, ";") AS indexString FROM system:indexes AS s LET bid = CONCAT("", s.bucket_id, ""), sid = CONCAT("", s.scope_id, ""), kid = CONCAT("", s.keyspace_id, ""), k = NVL2(bid, CONCAT2(".", bid, sid, kid), kid), ks = CASE WHEN s.is_primary THEN "" ELSE "(" || CONCAT2(",", s.index_key) || ")" END, w = CASE WHEN s.condition IS NOT NULL THEN " WHERE " || REPLACE(s.condition, '"', "'") ELSE "" END, p = CASE WHEN s.`partition` IS NOT NULL THEN " PARTITION BY " || s.`partition` ELSE "" END; ``` ```sql -- Schema (per collection) INFER `bucket`.`scope`.`collection` WITH {"sample_size": 10000}; ``` Export each result as JSON. **Load Data** Open CouchLens. Navigate to the Input tab. Upload the JSON files or paste JSON directly. Click Analyze. Parsing typically completes in under a second for 10,000 queries. ![CouchLens Input tab for uploading JSON data files](couchlens.lorbic.com_input.webp) **Explore** Navigate through tabs to review metrics, insights, schema, and individual queries. Use insights to prioritize optimization work. Generate reports for documentation or sharing. --- ### Design Decisions CouchLens uses Svelte 5 and TypeScript. Svelte provides reactive state management with a small runtime. TypeScript ensures type safety during data transformations. All parsing and analysis runs client-side in the browser's JavaScript runtime. The tool does not connect to Couchbase directly. This avoids security concerns and deployment complexity. You export data once and analyze locally. For continuous monitoring, collect new exports periodically. Charts use Chart.js with zooming and panning enabled. The flow diagram uses a custom Sankey implementation. The insight engine is rule-based. Each insight category has thresholds and pattern matchers defined in `src/lib/insights.ts`. Thresholds are currently hardcoded but tunable by editing the source. The app is a Progressive Web App. Once installed, it works offline. The service worker caches assets for fast loading. --- ### Current Limitations CouchLens is version 0.3 alpha. Known limitations: - **No Historical Tracking**: Each analysis session is independent. Tracking performance over time requires manual export and comparison. - **No Auto-Refresh**: You must manually re-export and re-upload data to see new queries. - **Fixed Thresholds**: Insight thresholds are hardcoded. Different workloads may need different definitions of "slow" or "inefficient". - **Limited Correlation**: The tool does not correlate queries with external events (deployments, traffic spikes, configuration changes). - **No Background Monitoring**: There is no alerting when new issues appear. --- ### Roadmap Future versions will focus on making CouchLens more useful for ongoing performance work. These plans are not committed and may change based on my availability and requirement. #### Planned Features **Index Recommendations** A rule-based recommendation engine that suggests specific indexes for queries scanning too many rows or using primary indexes. The engine will parse query predicates and suggest covering indexes or compound indexes based on filter patterns. **Custom Thresholds** User-configurable thresholds for each insight category. Allow users to define what "slow" or "inefficient" means for their workload. **Query Profiling Wizard** A guided workflow that takes a slow query, generates explanations of execution plan phases, and suggests specific optimizations (add index, rewrite WHERE clause, use covering index, etc.). **Historical Comparison** Store past analysis results (in browser local storage or exported files) and compare performance across time periods. Show which queries degraded, improved, or appeared/disappeared. **Direct Couchbase Integration (Optional)** An optional mode that connects directly to Couchbase and pulls fresh data on demand. For users who prefer live data over manual exports. This would be opt-in. **Alerting** Background checks that notify when p99 latency crosses thresholds or when new inefficient queries appear in fresh datasets. **Export Reports to File** Generate standalone HTML or PDF reports summarizing insights and metrics for sharing or archiving. Priority will be given to index recommendations and custom thresholds as these directly address current pain points. --- ### Operational Cost Right now CouchLens runs entirely in the browser. There is no backend to maintain, no database to configure, no infrastructure to deploy. The queries you run against `system:completed_requests` and `system:indexes` do have a cost on the Couchbase cluster. `system:completed_requests` is bounded by the `completed-threshold` setting (default 1000 queries). Fetching this data is a single query. `system:indexes` typically contains hundreds of indexes, not thousands, so fetching is cheap. Schema inference with `INFER` samples documents and can be expensive on large collections. Use `sample_size` to control cost. Once you have JSON exports, analysis completes in under a second for 10,000 queries on a modern browser. --- ### Comparison to Other Tools Couchbase provides the Query Workbench, which shows `EXPLAIN` and `PROFILE` for individual queries. The Couchbase Server UI shows cluster-level metrics. CouchLens fits between these tools. It provides batch analysis across many queries and automates pattern detection that would otherwise require manual inspection of each query. For real-time monitoring with dashboards and alerting, use Prometheus, Grafana, and the Couchbase exporter. CouchLens is for ad-hoc analysis and optimization work, not 24/7 monitoring. Although it can become one in future. For application-level tracing (tracking queries from application code through to database execution), integrate OpenTelemetry or a similar distributed tracing framework. CouchLens focuses only on the database layer. --- ### Technical Details The codebase structure: ``` src/ ├── components/ # Svelte UI components (dashboard, insights, etc.) ├── lib/ │ ├── parsing.ts # Query log parsing, execution plan extraction │ ├── insights.ts # Insight detection rules and thresholds │ ├── schema.ts # Schema type analysis │ ├── aggregation.ts # Data aggregation for charts │ ├── types.ts # TypeScript type definitions │ └── stores.ts # Svelte reactive state stores └── App.svelte # Main application with routing ``` Parsing logic in `lib/parsing.ts` extracts fields from `system:completed_requests`, computes derived metrics (elapsed time in milliseconds, statement type, indexes used), and normalizes timestamps. Insight detection in `lib/insights.ts` scans the parsed dataset and returns structured results with affected query IDs, severity, and recommendations. Chart aggregation in `lib/aggregation.ts` groups queries by time intervals (minute, hour, day) and computes throughput and average durations per bucket. Schema analysis in `lib/schema.ts` recursively walks the `INFER` output and detects type inconsistencies and special field types (dates, UUIDs, etc.). --- ### Conclusion CouchLens is a tool for parsing Couchbase query logs and finding inefficiencies. It runs in the browser, requires no backend, and works offline. It automates detection of common performance issues and provides a structured way to explore query behavior across large datasets. The tool is not a replacement for `EXPLAIN` or `PROFILE`. It is a complement. Use CouchLens to find which queries need investigation, then use `EXPLAIN` and `PROFILE` to diagnose the specific problem and validate fixes. Future versions will add index recommendations, customizable thresholds, and historical tracking. If you manage Couchbase clusters and spend time optimizing N1QL queries, try CouchLens. Load your query logs and see what it finds. Further readings: - https://docs.couchbase.com/server/current/n1ql/n1ql-manage/monitoring-n1ql-query.html - https://docs.couchbase.com/server/current/n1ql/n1ql-language-reference/explain.html - https://docs.couchbase.com/server/current/n1ql/n1ql-language-reference/query-hints.html --- ## What Couchbase Taught Me About System Thinking **Date:** 2025-11-18 **URL:** https://lorbic.com/what-couchbase-taught-me-about-system-thinking/ **Description:** Working with Couchbase for the past year and a half has been more than just learning a database. It has been an exercise in system thinking: seeing how indexes, queries, consistency, and durability interact, and how small design choices ripple through performance and reliability. This essay is my personal notebook from that journey. It is technically dense because Couchbase demands precision, but it is also reflective because the lessons extend beyond one database. Understanding how array indexes multiply entries, why compound index order matters, or how query consistency flags change latency is not just about Couchbase, it is about learning to think in terms of systems, trade‑offs, and consequences. ## Couchbase Internals: Indexes, Queries, Consistency, and Performance ### Introduction Over the last few years I've lived deep inside backend systems, and for the past year and a half Couchbase has been my daily companion. Working with Go services that depend on Couchbase taught me that the real lessons aren't in the marketing slides or quick‑start guides. They're in the internals: how indexes are built, how queries are planned, how consistency flags change the story, and how durability levels quietly decide whether your system survives a failure or not. This post is my attempt to capture those lessons in one place. It's not a tutorial or a sales pitch. It's a personal forensic walk through the parts of Couchbase that I had to understand to keep my systems stable. I'll show you the indexes I built, the queries I ran, and the mistakes I made. Think of it as a long journal entry written for other engineers who want to see what Couchbase looks like when you peel back the surface. This post is a detailed, technical exploration of the internal mechanics that determine Couchbase behavior in production. It assumes you have practical familiarity with JSON document databases, basic distributed systems concepts, and a working knowledge of Go and N1QL. The goal is to explain cause and effect: why an index behaves the way it does, why a query plan chooses a document fetch instead of an index seek, and how durability choices translate into latency. This post is built on practical examples. Each index example below is followed by what the index stores, how queries use it, and the operational cost you must pay to maintain it. --- ### Architecture and Data Model Couchbase is a set of cooperating services. - The **Data Service** implements the key-value interface, persistence, and replica mechanics. - The **Index Service** builds and serves Global Secondary Indexes (GSIs). - The **Query Service** parses N1QL, plans execution, and orchestrates distributed operations. Documents are assigned to logical shards called **vBuckets**. A hashing function maps document keys to vBuckets, and the cluster maintains a vBucket map that SDKs use for direct routing of key-value operations. Rebalancing moves vBuckets across nodes, streaming data and per-vBucket sequence metadata. Two storage engines are important: **Couchstore** and **Magma**. - **Couchstore** behaves like a compacting, append-oriented structure with B-tree-like lookup characteristics. - **Magma** behaves like a Log-Structured Merge-Tree (LSM) store, optimized for large datasets and high write throughput. This choice affects **write amplification** - a term describing how a single database write (like an update) can cause multiple I/O operations on disk as data is compacted or moved. In production, you must watch compaction metrics and index backfill durations, as these operations reveal whether the chosen engine fits your workload. Document shape matters. Couchbase stores JSON documents with metadata such as CAS, expiration, and optional extended attributes. Updates use a **copy-on-write** model from the client's perspective, meaning the entire document is typically sent for any change. - **Large or "hot" documents** (frequently updated) increase write amplification and amplify index maintenance cost when indexed fields change. - **Embedding** many logically independent items in a single document gives read locality (fast reads) at the cost of heavier writes and larger fetch units. - **Splitting** independent items across documents reduces write amplification and allows finer-grained sharding by vBucket. --- ### Indexing Deep Dive with Examples A **Global Secondary Index (GSI)** maps keys derived from document fields to document identifiers. Indexes significantly change read cost and raise write cost proportionally to the number and size of indexed fields. - **Array elements** create multiple index entries per document. - **Compound indexes** create keys ordered alphabetically or numerically (a "lexographical" order). - **Partial indexes** restrict entries to only those documents that match a specific predicate (a `WHERE` clause). - **Covering indexes** store all fields required by a query _inside_ the index itself, so a query can be answered without fetching the full document. This is achieved by making all filtered and projected fields part of the index key. Below are detailed, concrete index examples, each followed by its semantics, storage, query use, and operational trade-offs. #### Example 1 - Primary Index, Full Bucket Scans ```sql CREATE PRIMARY INDEX ON `bucket`; ``` - **Semantics and Use:** The primary index provides a mapping of every document ID, allowing N1QL to perform full bucket scans for ad-hoc queries. This is useful for development but is rarely acceptable for production queries on large datasets. - **Operational Cost:** Full scans scale linearly with document count and cause heavy I/O. Keep this index only if you explicitly need ad-hoc scanning capabilities. #### Example 2 - Single Field Secondary Index ```sql CREATE INDEX idx_user_age ON `users`(age); ``` - **What it Stores:** For each document containing `age`, the index stores the `age` value and a pointer to the document ID. - **Query Usage:** A predicate `WHERE age = 30` becomes an index seek on `idx_user_age`. If the query returns only `age` and the document ID, the index can serve the request without a document fetch. If the query projects other fields (e.g., `SELECT name...`), a Fetch is required. - **Operational Cost:** Each mutation that changes `age` must update the index. The cost is proportional to the write rate and the index size. #### Example 3 - Compound Index and Left-Prefix Semantics ```sql CREATE INDEX idx_user_age_name ON `users`(age, name); ``` - **What it Stores:** Tuples of `(age, name)` mapped to document IDs. Keys are ordered by `age` first, then `name`. - **Query Implications:** This demonstrates **left-prefix semantics**. Queries must use the indexed fields from left to right. - `WHERE age = 30` **can** use the index (a left-prefix seek). - `WHERE age = 30 AND name = "Alice"` **can** use the index (a full compound seek). - `WHERE name = "Alice"` **cannot** use this index for a seek, because `age` is the leading component and is not included in the query. - **Operational Guidance:** Place the most selective or most commonly filtered field first, especially when using equality predicates. #### Example 4 - Compound Index with Range Predicates ```sql CREATE INDEX idx_user_country_age ON `users`(country, age); ``` - **Query Patterns:** - `WHERE country = "IN" AND age BETWEEN 25 AND 35` benefits from this index. It performs an efficient seek on `country = "IN"` and then a contiguous range scan for the `age`. - **Design Rule:** For compound indexes mixing equality and range predicates, place the equality columns _before_ the range columns. #### Example 5 - Array Index, Distinct Semantics ```sql CREATE INDEX idx_posts_tags_distinct ON `posts`(DISTINCT ARRAY t FOR t IN tags END); ``` - **What it Stores:** For each document, the index stores an entry for each _unique_ element `t` in the `tags` array. If a document has `tags: ["go", "go"]`, `DISTINCT` ensures only one index entry for "go" is created for that document. - **Query Usage:** ```sql SELECT META().id FROM posts WHERE ANY t IN tags SATISFIES t = "go" END; ``` The query becomes an index seek for the key "go". The engine returns one entry per matching document. - **Operational Cost:** Cost is proportional to the total number of _unique_ array elements across all documents. #### Example 6 - Array Index without DISTINCT ```sql CREATE INDEX idx_posts_tags ON `posts`(ARRAY t FOR t IN tags END); ``` - **What it Stores:** An entry for _every_ array element, including duplicates. A document with `tags: ["go", "go"]` will produce two index entries for "go". - **Query Semantics:** Index scans return multiple entries per document if multiple elements match. The query engine must then deduplicate these results, which adds CPU cost. #### Example 7 - Compound Index that Includes an Array Element ```sql CREATE INDEX idx_user_age_tag ON `users`(age, DISTINCT ARRAY t FOR t IN tags END); ``` - **What it Stores:** Tuples such as `(age, tag)` where `tag` iterates over the distinct tags in each document. - **Example Query:** ```sql SELECT u.id, t FROM users u UNNEST u.tags t WHERE u.age BETWEEN 25 AND 35 AND t = "k8s"; ``` - **How the Planner Can Use It:** The index supports an `age` range seek and a specific `tag` match simultaneously. - **Operational Impact:** This index is more selective but also larger, as it stores a tuple for every `(age, tag)` pair. #### Example 8 - Partial Index to Reduce Index Size ```sql CREATE INDEX idx_active_users ON `users`(signup_date) WHERE status = "active"; ``` - **What it Stores:** Only documents with `status = "active"` appear in the index. - **Use Case:** If the majority of queries target active users, this index saves significant space and increases selectivity. - **Operational Cost:** Updates that change `status` (e.g., from "pending" to "active") require an index entry insertion or deletion. Partial indexes work best when the predicate defines a stable, frequently queried subset. #### Example 9 - Covering Index ```sql CREATE INDEX idx_cover_profile ON `users`(last_login, email, name); ``` - **What it Stores:** The index stores a composite key of `(last_login, email, name)` mapped to document IDs. - **Query Example:** ```sql SELECT name, email FROM users WHERE last_login > "2025-01-01"; ``` - **Execution:** The query can be satisfied entirely by the index. It seeks on `last_login` (the leading key) and then projects the `name` and `email` values directly from the index keys, avoiding a full document Fetch. - **Trade-off:** Adding fields to the index key increases index size and write cost, but it can dramatically reduce query latency for high-frequency, read-heavy queries. #### Example 10 - Indexes for JOIN Probes - **Index Definitions:** ```sql CREATE INDEX idx_orders_userid ON `orders`(user_id); CREATE INDEX idx_users_id ON `users`(id); ``` - **Query:** ```sql SELECT o.*, u.* FROM orders o JOIN users u ON o.user_id = u.id WHERE o.date > "2025-01-01"; ``` - **Planner Choices:** 1. **Nested Loop:** If the planner expects few orders for the date range, it will scan `orders` and then "probe" the `users` table using `idx_users_id` for each `user_id`. Each probe is an efficient index lookup. 2. **Hash Join:** If the planner expects many orders, it may choose a hash join and build an in-memory hash table of one side, provided memory permits. - **Diagnostic Action:** Use `EXPLAIN` and `PROFILE` to compare estimated sizes against real row counts. If estimates are wrong, consider improving statistics or rewriting the query. #### Example 11 - Composite Index with Function-Based Keys ```sql CREATE INDEX idx_lower_email ON `users`(LOWER(email)); ``` - **Use Case:** When queries request case-insensitive email lookups (e.g., `WHERE LOWER(email) = "..."`), indexing the computed value removes the need to fetch and compute during query time. - **Operational Caveat:** Index maintenance requires computing the function on every mutation that might change the input field. #### Example 12 - Index Including Geospatial or Numeric Projections ```sql CREATE INDEX idx_locations ON `places`(dist_lat, dist_lon, name); ``` - **Index Use:** When queries perform bounding box or distance filters (e.g., on `dist_lat` and `dist_lon`) and also need to return the `name`, placing all fields in the index key allows the index to cover the query. - **Design Considerations:** Be mindful of precision and range bucket sizes, which affect selectivity. --- ### Query Planning, Cardinality Estimation, and Execution N1QL queries are transformed into physical plans formed from operators such as `IndexScan`, `Fetch`, `Filter`, `NestedLoopJoin`, and `HashJoin`. The planner relies heavily on statistics. The most fragile part of this process is **cardinality estimation** - the planner's _guess_ at how many documents will match a filter. It often assumes columns are independent (e..g., `city` and `job_title` are unrelated). When columns _are_ correlated (e.g., "San Francisco" and "Software Engineer"), the planner's guesses can be wildly wrong, leading to poor join choices. #### Statistics: Collection and Consequences Statistics (distinct counts, histograms) are collected during index builds and via the `COLLECT STATS` or `ANALYZE` operations. **Stale statistics** (out-of-date information) create wrong cost estimates. If your data changes often (**data drift**) or has **heavy skew** (an uneven distribution of values), you must refresh statistics more frequently or design targeted indexes to reduce dependence on inaccurate estimates. #### Recognizing Bad Plans A common diagnostic pattern is to run `EXPLAIN` to see the _estimated_ operator row counts, then run `PROFILE` to run the query and see the _actual_ row counts and operator timing. When you see a large discrepancy between estimated and actual rows at a particular operator, the logical fix is one or more of these actions: - Tune or refresh statistics for the indexes involved. - Create smaller, targeted indexes that expose correlation between fields. - Rewrite the query so its selectivity is more explicit (e.g., move predicates into a sub-select). - Add covering index projections to avoid expensive Fetch operations. - Use hints to force an index or join strategy while you diagnose the root cause. #### Join Strategies: Pick the Right Tool - A **Nested Loop Join** is efficient when the "outer" side of the join is small; the join then probes the "inner" side by its index. - A **Hash Join** suits large datasets where both sides can be materialized within memory constraints. Wrong planner estimates often cause the planner to pick a nested loop join against a large outer set. This results in millions of index probes and thousands of Fetches, revealing the need for better statistics or query refactoring. --- ### Consistency, Durability, and Mutation Ordering Couchbase exposes query consistency and durability options. Consistency controls whether a query must reflect recent mutations. Durability controls whether a mutation is acknowledged only after being replicated or persisted. #### Query Consistency Options The common modes are `NOT_BOUNDED`, `REQUEST_PLUS`, and `STATEMENT_PLUS`. - **`NOT_BOUNDED`:** This is the fastest. It does no index synchronization, and queries may return slightly stale results (i.e., not reflecting writes made milliseconds ago). - **`REQUEST_PLUS`:** Guarantees that a query observes mutations that _that specific client_ performed prior to the query. It uses mutation tokens to achieve this. - **`STATEMENT_PLUS`:** Similar to `REQUEST_PLUS`, but applies when coordinating multiple statements in a single logical sequence. #### Mechanics Behind REQUEST_PLUS When an SDK performs a mutation, it can ask for a **mutation token**. This token contains the vBucket ID and a sequence number for that write. The client then attaches this token to a subsequent query. The index service checks its local sequence numbers for that vBucket and will pause (or block) the query until its index has processed mutations up to the requested sequence. This targeted synchronization is far more efficient than forcing a global index refresh. #### Durability Levels and Costs Durability levels such as `NONE`, `MAJORITY`, and `MAJORITY_AND_PERSIST` control how many replicas must acknowledge a mutation. - Use `MAJORITY` when you need to survive single-node failures without data loss. - Use `MAJORITY_AND_PERSIST` when you require on-disk durability across a majority of nodes. - Each level increases write latency and affects throughput. For high-throughput write paths, prefer `NONE` or asynchronous replication if your application can tolerate it. #### CAS and Optimistic Concurrency **CAS (Compare-And-Swap)** tokens permit optimistic concurrency control. 1. Read a document, and get its unique CAS value. 2. Perform a conditional write _with_ that CAS value. 3. If the CAS has changed (meaning someone else modified the document in the meantime), the write fails. 4. Your application must then retry the read-modify-write cycle or reconcile the conflict. CAS works well in low-contention patterns but induces retry cycles under "hot key" contention. #### XDCR and Conflict Resolution **XDCR (Cross-Datacenter Replication)** uses metadata to resolve conflicts according to a set policy (e.g., last-writer-wins). Active-active designs require application-level awareness of conflict semantics, because a simple metadata-based resolution may not match your domain-level business rules. --- ### Performance Engineering, Operations, and Experiments Performance engineering must be based on data. Observability is the first necessity. Monitor indexer memory, index build progress, index disk usage, KV get and mutate latencies, **resident ratio** (the percentage of active data held in RAM), compaction throughput, disk IOPS, and CPU per service. #### Compaction and Storage Engine Effects Compaction is a background process. Couchstore compaction rewrites files to remove **tombstones** (markers for deleted data) and obsolete versions. Magma compaction (LSM-based) produces different I/O patterns. On large clusters, compaction or index backfills can saturate I/O and cause tail latency spikes. Schedule heavy maintenance windows and test compaction settings under load. #### Hot Keys and Sharding A **hot key** (a single document or small set of keys receiving disproportionate writes) produces tail latency and write contention. To mitigate this, shard the logical state across multiple documents and use deterministic sharding schemes or application-level throttles for heavy producers. #### Go SDK Tuning and Client Behavior The Go SDK offers connection pooling, retry behavior, and Durability options. Tune connection pool sizes, pipeline depth, and maximum in-flight requests to match cluster capacity. Use mutation tokens to apply `REQUEST_PLUS` only on paths that require it. Avoid aggressive client-side retries, which amplify server load. #### Reproducible Experiments Below are experiment outlines that isolate variables to reveal index behavior. **Experiment A - Array Cardinality Effect** - **Data:** 100k documents with `tags` arrays of sizes 1, 10, 100, and 1000 (in separate runs). - **Index:** ```sql CREATE INDEX idx_posts_tags_distinct ON `posts`(DISTINCT ARRAY t FOR t IN tags END); ``` - **Measure:** Index disk size, build time, per-mutation index update latency, and query latency for `ANY t IN tags SATISFIES t = "X" END`. - **Expected Outcome:** Index size scales with the total number of unique array elements. Per-mutation cost increases with array size. Queries for rare tags remain efficient, but index build and maintenance cost become the bottleneck. **Experiment B - Cardinality Skew and Join Planning** - **Data:** Two 1-million-document datasets. Dataset 1 has a uniform distribution on field `x`. Dataset 2 has 99% of documents with `x = "A"`. - **Indexes:** ```sql CREATE INDEX idx_items_x ON `items`(x); CREATE INDEX idx_other_itemkey ON `other`(item_key); ``` - **Query:** ```sql SELECT i.*, o.* FROM items i JOIN other o ON i.key = o.item_key WHERE i.x = "somevalue"; ``` - **Measure:** `EXPLAIN` and `PROFILE` outputs, operator row counts, join choice, and end-to-end latency. - **Expected Outcome:** The skewed dataset will reveal planner misestimation, likely causing bad nested loop joins with many probes. **Experiment C - Durability Latency Trade-off** - **Operation:** Upsert a document with Durability `NONE`, `MAJORITY`, and `MAJORITY_AND_PERSIST`. - **Measure:** p50, p95, and p99 write latency and throughput under identical concurrency. - **Expected Outcome:** Write latency and throughput degrade as the durability level increases. #### Minimal Go Harness Snippet for Mutation Latency Collect Couchbase server metrics from the REST API and combine them with client timings from a harness like this. ```go package main import ( "context" "fmt" "time" "github.com/couchbase/gocb/v2" ) func main() { cluster, err := gocb.Connect("couchbase://localhost", gocb.ClusterOptions{ Username: "Administrator", Password: "password", }) if err != nil { panic(err) } bucket := cluster.Bucket("test") coll := bucket.DefaultCollection() ctx := context.Background() start := time.Now() _, err = coll.Upsert("docKey", map[string]interface{}{"name": "test"}, &gocb.UpsertOptions{ DurabilityLevel: gocb.DurabilityLevelMajority, Timeout: 5 * time.Second, }) elapsed := time.Since(start) if err != nil { fmt.Println("upsert error", err) } else { fmt.Println("upsert ok", elapsed) } _ = ctx } ``` --- ### Remediation Patterns for Common Failures When `PROFILE` shows that an `IndexScan` returns far more rows than estimated and a subsequent `Fetch` consumes large resources, use this ordered approach: 1. **Check covering potential.** Add the projected fields to the end of the index key list (e.g., `CREATE INDEX... ON \`bucket\`(filter_field, projected_field1, projected_field2)\`) if the index size increase is acceptable. This is often the biggest win. 2. **Refresh index statistics** so that the optimizer uses recent histograms. 3. **Create partial indexes** for hot predicates (e.g., `WHERE type="widget"`) to improve selectivity. 4. **Rewrite the query** to make selectivity explicit or to push predicates earlier in the plan. 5. **Apply hints** to force an index or join strategy while you iterate on diagnostics. --- ### Conclusion and Practical Checklist Index design and data modeling are not separate tasks; they are deeply connected. Each index you add is a measured trade-off between read latency and write cost. Indexes are concrete structures that occupy memory and disk and must be updated on every mutation that touches indexed fields. #### Checklist - Design compound indexes with equality fields first, then range fields. - Avoid indexing very large arrays. If arrays are needed, consider splitting high-cardinality members into separate documents and indexing the link key. - Use `DISTINCT` in array indexes when duplicate elements are possible and you prefer deduplicated index entries. - Favor partial indexes when queries target a narrow and stable subset of documents. - Create covering indexes by adding projected fields to the index key list to avoid Fetches for frequent read paths, after evaluating the index size impact. - Use `REQUEST_PLUS` with mutation tokens only where correctness demands immediate visibility of recent writes. - Refresh statistics when data distribution changes, especially when you see poor plan choices. - Monitor indexer memory, index build times, compaction load, and resident ratio continuously. - Reproduce issues with controlled experiments that vary a single factor at a time. Understanding Couchbase internals changes how you model data and design queries. Indexes are costly tools that, when used precisely, produce dramatic latency improvements. When misused, they are a primary source of unpredictable resource usage. Read plans, analyze `EXPLAIN` and `PROFILE`, design targeted indexes, and measure constantly. I should admit that I've skipped over some details here and fast‑forwarded through others. Couchbase internals are too broad to cover in one sitting, and I wanted this essay to stay readable rather than drown in every operator and statistic. In particular, I haven't gone step by step through query plans or shown how to interpret every line of EXPLAIN and PROFILE. That deserves its own dedicated post. My plan is to write another piece where I take one index and one query, dissect the query plan in detail, and show how to improve the index design based on what the planner is telling you. So treat this essay as the map. The microscope will come later. {{< highlight >}} If you want a practical tool to help with this analysis, I built {{< link href="/couchlens-couchbase-query-analysis-tool" text="CouchLens" target="_blank" >}}, a browser-based query analysis tool for Couchbase. It parses `system:completed_requests` and `system:indexes`, detects inefficient index scans, primary index usage, and other performance issues, and generates insights to help you find which queries need attention. Everything runs locally in your browser, no data leaves your machine. {{< /highlight >}} Further readings: - https://docs.couchbase.com/server/current/indexes/indexing-overview.html - https://docs.couchbase.com/server/current/learn/buckets-memory-and-storage/vbuckets.html - https://docs.couchbase.com/server/current/learn/data/durability.html - https://support.couchbase.com/hc/en-us/articles/23629925229851-Completed-Requests-basics - https://github.com/aakash-advait/couchbase-sync-tool - if you need to sync couchbase nodes. For local setup etc. --- ## Understanding Private-Public Key Encryption **Date:** 2025-11-06 **URL:** https://lorbic.com/private-public-key-encryption/ **Description:** Understanding Private-Public Key Encryption - The Backbone of Modern Security. In today's digital world, where secure communication, authentication, and data integrity are non-negotiable, **private-public key encryption** (also called *asymmetric cryptography*) plays a foundational role. From HTTPS in your browser to SSH logins, cryptocurrency wallets, and email encryption, this elegant cryptographic system enables trust without prior shared secrets. In this blog, we'll dive into: - The core concepts behind asymmetric encryption - How it differs from symmetric encryption - The mathematics (intuitively explained) behind popular algorithms like RSA and ECC - Real-world examples and code snippets - Common use cases and pitfalls Whether you're a curious beginner or a seasoned developer brushing up on fundamentals, you'll find something valuable here. ## A Quick Comparison of Symmetric vs. Asymmetric Encryption Before we dive into public-key crypto, let's contrast it with its older sibling: **symmetric encryption**. | Feature | Symmetric Encryption | Asymmetric Encryption | |------------------------|-------------------------------|-----------------------------------| | Keys | One shared secret key | Two keys: public + private | | Speed | Fast | Slower (computationally heavy) | | Key Distribution | Hard (needs secure channel) | Easy (public key can be shared) | | Use Cases | Bulk data encryption (e.g., AES) | Key exchange, digital signatures | **Symmetric encryption** (like AES) is great for encrypting large amounts of data quickly, but you need a way to **securely share the secret key** beforehand. That's where **asymmetric encryption** shines: it solves the *key distribution problem*. ## Two Keys, One Mathematical Bond In asymmetric cryptography: - The **public key** is shared openly. - The **private key** is kept secret. - Data encrypted with one key can **only** be decrypted with the other. This creates two powerful capabilities: 1. **Confidentiality**: Encrypt with someone's *public key* → only they can decrypt with their *private key*. 2. **Authentication & Integrity**: Sign data with your *private key* → anyone can verify it using your *public key*. > **Crucial Insight**: The public key *cannot* be used to derive the private key, even though they're mathematically linked. This is the "hard problem" that makes the system secure. ## How It Works ### Example 1: Sending a Secure Message Alice wants to send a confidential message to Bob. 1. Bob generates a key pair: - Public key: `Bob_pub` - Private key: `Bob_priv` (kept secret) 2. Bob shares `Bob_pub` with Alice (e.g., via email, website, key server). 3. Alice encrypts her message using `Bob_pub`. 4. Bob receives the ciphertext and decrypts it with `Bob_priv`. Even if Eve intercepts the message and has `Bob_pub`, she **cannot** decrypt it, because the private key is required. ### Example 2: Digital Signatures Now, Alice wants to prove a message truly came from her. 1. Alice hashes her message → `H = SHA256("Hello Bob!")` 2. She encrypts the hash with her *private key*: `Sig = Encrypt(H, Alice_priv)` 3. She sends both the message and `Sig` to Bob. 4. Bob: - Hashes the received message → `H'` - Decrypts `Sig` using `Alice_pub` → gets `H` - Checks if `H == H'` If they match, the message is **authentic** and **unchanged**. > Note: In practice, we don't encrypt the whole message, just its hash (for efficiency and security). ## Under the Hood: RSA and ECC Two major algorithms power most public-key cryptography today: ### 1. RSA (Rivest–Shamir–Adleman) **Based on**: The difficulty of factoring large prime numbers. #### Key Generation (Simplified): 1. Choose two large primes: `p` and `q` 2. Compute `n = p * q` 3. Compute Euler's totient: `φ(n) = (p-1)(q-1)` 4. Choose public exponent `e` (commonly 65537) such that `1 < e < φ(n)` and `gcd(e, φ(n)) = 1` 5. Compute private exponent `d` such that `(d * e) mod φ(n) = 1` - Public key: `(n, e)` - Private key: `(n, d)` **Encryption**: `ciphertext = plaintext^e mod n` **Decryption**: `plaintext = ciphertext^d mod n` > RSA requires large keys (2048+ bits) for security today. #### Python Example (using `cryptography` library): ```python from cryptography.hazmat.primitives.asymmetric import rsa, padding from cryptography.hazmat.primitives import hashes from cryptography.hazmat.primitives import serialization # Generate keys private_key = rsa.generate_private_key(public_exponent=65537, key_size=2048) public_key = private_key.public_key() # We are writing the keys to files for demonstration purposes with open("public_key.pem", "wb") as f: f.write(public_key.public_bytes( encoding=serialization.Encoding.PEM, format=serialization.PublicFormat.SubjectPublicKeyInfo )) with open("private_key.pem", "wb") as f: f.write(private_key.private_bytes( encoding=serialization.Encoding.PEM, format=serialization.PrivateFormat.PKCS8, encryption_algorithm=serialization.NoEncryption() )) # Encrypt message = b"Secret message" ciphertext = public_key.encrypt( message, padding.OAEP( mgf=padding.MGF1(algorithm=hashes.SHA256()), algorithm=hashes.SHA256(), label=None ) ) # Decrypt plaintext = private_key.decrypt( ciphertext, padding.OAEP( mgf=padding.MGF1(algorithm=hashes.SHA256()), algorithm=hashes.SHA256(), label=None ) ) print(plaintext) # b"Secret message" ``` ### 2. ECC (Elliptic Curve Cryptography) **Based on**: The algebraic structure of elliptic curves over finite fields. A type of math involving curved lines and numbers that reset after a certain point, designed so that going forward is easy, but going backward without a secret key is practically impossible. - Offers **same security as RSA** with **much smaller keys** (e.g., 256-bit ECC ≈ 3072-bit RSA). - Faster and more efficient, ideal for mobile and embedded systems. Common curves: `secp256k1` (used in Bitcoin), `Curve25519` (used in Signal, SSH). #### ECC Key Generation (Conceptual): 1. Choose a standard elliptic curve (e.g., `secp256r1`) 2. Pick a random private key `d` (a large integer) 3. Compute public key `Q = d * G`, where `G` is a known base point on the curve The security relies on the **Elliptic Curve Discrete Logarithm Problem (ECDLP)**: given `Q` and `G`, it's computationally infeasible to find `d`. > ECC is now the **preferred choice** for new systems (TLS 1.3, blockchain, etc.). ## Real-World Use Cases ### 1. **TLS/SSL (HTTPS)** - Your browser uses the server's public key to establish a secure session. - Asymmetric crypto exchanges a symmetric key (e.g., AES) for efficient data transfer. ### 2. **SSH Authentication** - Your SSH client proves identity using a private key; the server verifies with your public key. ### 3. **PGP/GPG Email Encryption** - Encrypt emails with recipient's public key; sign with your private key. ### 4. **Cryptocurrencies** - Bitcoin wallet addresses are derived from public keys. - Transactions are signed with private keys to prove ownership. ## Common Pitfalls & Best Practices | Mistake | Why It's Bad | Best Practice | |--------|-------------|---------------| | Reusing RSA keys for encryption + signing | Increases attack surface | Use separate keys or follow PKCS standards | | Using small key sizes (e.g., 1024-bit RSA) | Vulnerable to factorization | Use ≥2048-bit RSA or 256-bit ECC | | Not using padding (e.g., textbook RSA) | Vulnerable to chosen-ciphertext attacks | Always use OAEP (encryption) or PSS (signing) | | Hardcoding private keys in code | Massive security risk | Store in secure vaults (e.g., HashiCorp Vault, AWS KMS) | --- Private-public key encryption is not just a cryptographic curiosity, it's the **trust engine** of the internet. By elegantly solving the key distribution problem, it enables secure communication between strangers across an insecure network. While the math behind RSA and ECC can seem daunting, modern libraries abstract away the complexity. As a developer, your job isn't to implement these algorithms from scratch, but to **use them correctly**, respect key hygiene, and understand their limits. As you explore steganography or other low-level projects (like embedding data in media), remember: **encryption protects content, steganography hides its existence**, and the two can even work together! ## Further Reading - [PKCS Standards](https://en.wikipedia.org/wiki/PKCS?ref=lorbic.com) - [NIST Guidelines on Key Management](https://csrc.nist.gov/publications/detail/sp/800-57-part-1/rev-5/final?ref=lorbic.com) - [Serious Cryptography by Jean-Philippe Aumasson](https://nostarch.com/serious-cryptography-2nd-edition?ref=lorbic.com) - [Cloudflare's ECC explanation](https://blog.cloudflare.com/a-relatively-easy-to-understand-primer-on-elliptic-curve-cryptography/?ref=lorbic.com) --- **Got questions or want a deep dive into a specific algorithm?** Leave a comment below! Happy encrypting! --- ## The 5-Minute Refactor **Date:** 2025-10-28 **URL:** https://lorbic.com/the-5-minute-refactor/ **Description:** Learn how to refactor your code in 5 minutes. ## The 5-Minute Refactoring Guide As experienced software engineers, we often face a dilemma: our codebases, like all physical systems, trend toward entropy. The gap between "getting it done" and "getting it right" grows, leading to sluggish feature delivery and inevitable technical debt. The solution isn't a massive, heroic rewrite; it's the disciplined, humble practice of **Kaizen**, or continuous improvement. For Golang engineers, this translates to the **5-Minute Refactor**: a daily commitment to making **one tiny, tangible, quality improvement** to any code you touch. This practice leverages Go's philosophy of simplicity to prevent decay and sharpen your engineering judgment, all in less time than it takes to make coffee. ## Why 5 Minutes Works (The Deep Engineering Principles) This micro-habit is effective because it adheres to core principles that govern high-performing systems: 1. **Combats Entropy:** The 5-Minute Refactor is your consistent, small-energy input to counteract the natural disorder that makes codebases harder to maintain over time. 2. **Improves Flow (The First Way of DevOps):** Every small refactor removes a cognitive obstacle (like a confusing variable name or a nested block). Removing friction speeds up feature delivery. 3. **Enhances Judgment:** By constantly pausing to ask, "How can I make this single line better?" you train your eye to spot code smells, which is the definition of true engineering expertise. 4. **Fosters Humility:** It instills a sense of craftsmanship and shared responsibility for the codebase, moving the culture away from merely "shipping code" to "owning quality". ## The Practical "How-To" in Golang The goal is to choose **ONE** of these improvements when you open a file, then commit the change. The focus is on embracing Go idioms for simplicity, readability, and explicit design. ### 1\. Simplify Error Handling: The Early Return Go code famously suffers from deeply nested error checks. The Kaizen approach is to flatten the logic using the **Guard Clause** or **Early Return** pattern, ensuring the "happy path" (the successful outcome) is easy to follow. ##### Before (Deep Nesting) ```go func loadUser(id int) (*User, error) { if id > 0 { user, err := db.fetch(id) if err == nil { if user.Active { return user, nil } else { return nil, errors.New("user is inactive") } } else { return nil, err } } return nil, errors.New("invalid ID provided") } ``` ##### After (Flat Flow) ```go func loadUser(id int) (*User, error) { // Guard 1: Input validation returns immediately if id <= 0 { return nil, errors.New("invalid ID provided") } // Guard 2: I/O error check returns immediately user, err := db.fetch(id) if err != nil { return nil, fmt.Errorf("fetch failed: %w", err) } // Guard 3: Business logic check returns immediately if !user.Active { return nil, errors.New("user is inactive") } // The clear 'Happy Path' follows after all guards return user, nil } ``` **Why it helps:** Reduces cognitive load by keeping the core logic clear of error branches, making it significantly easier to read and reason about. ### 2\. Extract Logic: The Pure Function Refactor This improvement separates pure business logic (calculations) from side-effecting I/O logic (HTTP, DB calls), improving testability and clarity. ##### Before (Mixed Concerns in a Handler) ```go func handler(w http.ResponseWriter, r *http.Request) { // ... reading request, validation ... total := 0 for _, item := range items { // Business logic (calculation) mixed with I/O concerns total += item.Price + (item.Price * config.TaxRate) } // ... writing total to response ... } ``` ##### After (Extracted Pure Logic) ```go // Extracted Pure Function: easily unit testable, no side effects func calculateTotal(items []Item, taxRate float64) int { total := 0 for _, item := range items { total += item.Price + int(float64(item.Price) * taxRate) } return total } // The handler now only focuses on I/O and orchestration func handler(w http.ResponseWriter, r *http.Request) { // ... reading request, validation ... total := calculateTotal(items, config.TaxRate) // Call the pure function // ... writing total to response ... } ``` **Why it helps:** The core business logic is now isolated and **unit-testable**, improving quality and maintainability by adhering to the single responsibility principle. ### 3\. Improve Clarity by Using Clearer Names This simple refactor replaces an overly abbreviated receiver name with one that clearly conveys the context within the function body, especially in more complex methods. ##### Before (Too Generic Receiver) ```go type CacheService struct { data map[string]string metrics *MetricsCollector } // What is 'c'? It forces the reader to pause and remember the type. func (c *CacheService) Get(key string) (string, error) { // ... complex logic using c.data, c.metrics, logging ... val, ok := c.data[key] c.metrics.Increment("cache_hit") // ... return val, nil } ``` ##### After (Clearer Role) ```go type CacheService struct { data map[string]string metrics *MetricsCollector } // 'cache' clearly refers to the cache service instance, improving readability. func (cache *CacheService) Get(key string) (string, error) { // The scope of 'cache' is immediately clear throughout the method body val, ok := cache.data[key] cache.metrics.Increment("cache_hit") // ... return val, nil } ``` **Why it helps:** You communicate intent, reduce ambiguity, and make the method body easier to parse by clearly referencing the service instance. ### 4\. Enhance Type Safety: Custom Domain Types This refactor uses Go's type system to embed domain meaning into primitive types (`int`, `string`), preventing accidental misassignment of IDs or values and leading to compile-time checks for logical errors. ##### Before (Ambiguous Primitives) ```go // Both User IDs and Product IDs might be just 'int' func deleteRecord(id int) error { // ... logic to delete a record ... } // A call might accidentally pass the wrong type of ID, leading to a silent bug userID := 1001 productID := 2005 deleteRecord(userID) // Correct intent deleteRecord(productID) // Potential bug if deleteRecord expects a UserID! ``` ##### After (Type Safety Kaizen) ```go // Define custom types for domain clarity type UserID int type ProductID int // The function signature now clearly dictates the required input type func deleteUser(id UserID) error { // ... logic to delete a user ... fmt.Printf("Deleting user with ID: %d\n", id) return nil } // The compiler now prevents mistakes: deleteUser(ProductID(2005)) would be a compile-time error! userID := UserID(1001) productID := ProductID(2005) deleteUser(userID) // deleteUser(productID) // This line would cause a compiler error, catching a bug early! ``` **Why it helps:** This is Kaizen for **type safety**. The compiler now enforces domain rules, preventing an entire class of runtime errors by catching them at compile-time. ### 5. Tidy Up: Remove Dead Code & Redundant Variables Clutter, such as unused code, commented-out sections, or unnecessary intermediate variables, adds cognitive load. Removing it makes the actual working code stand out. ##### Before (Unnecessary Variable and Comment) ```go func HashData(data string) string { // The data needs to be converted to a byte slice // This comment just restates what the code does, adding no value. dataBytes := []byte(data) // Return the hashed value return util.hash(dataBytes) } ``` ##### After (Concise and Direct) The intermediate variable is inlined; the redundant comment is removed. The code is now more direct and less noisy. ```go func HashData(data string) string { return util.hash([]byte(data)) } ``` **Why it helps:** Go values conciseness. Eliminating code noise ensures engineers focus their attention only on lines that contain actual logic or important business context. ### The Kaizen Mindset The 5-Minute Refactor is about **consistency, not intensity**. Your goal is to make a tiny deposit into the quality bank every day. Don't wait for permission or a dedicated task. When you open a file for any reason, ask yourself: > **What is the smallest, safest, most impactful quality improvement I can make in this file in the next five minutes?** By adopting this mindset, you turn every code session into a micro-learning experience, steadily transforming your codebase and your engineering capabilities. This humble, daily discipline is how you truly become a humble engineer. Start today. Your future self (and your teammates) will thank you. --- ## I Built My Own Google Drive **Date:** 2025-10-01 **URL:** https://lorbic.com/i-built-my-own-google-drive/ **Description:** My journey to build my own cloud storage from scratch with nextcloudflare and the lessons I learned along the way ## Why Bother? Whenever I want to download a folder from Google Drive, it starts zipping it for what feels like minutes, and then it downloads the whole zip. There's no incremental download, and I can't add it as a network drive for obvious reasons. These limitations are frustrating, but they also got me thinking: what if I could have a solution that's more flexible, more customizable, and truly mine? The natural instinct for a systems engineer is: "Fine, I'll build my own NAS". But then you remember that a NAS at home means a mini data center: constant uptime, power backup, static IPs, UPS, maybe even an inverter if you live in India. Suddenly, the "simple" idea of owning your storage becomes a logistics project. I already had a cloud VM sitting idle. Something I'd spun up in 2022 for experiments. So I thought: what if I just turn that into my "own Google Drive"? Attach a 100–200 GB volume, mount it, run Nextcloud, and I'd have a private, scalable "Google Drive" that exists entirely under my control. The only catch: I didn't want to just make it work. I wanted to learn something new in the process, specifically about Cloudflare's networking stack. So instead of exposing ports and managing SSL myself, I decided to route everything through Cloudflare Tunnel. That meant I could have a nice domain like `lorbic.com` sitting in front of my VM, with Cloudflare handling SSL, DDoS protection, and tunneling, no public IP exposure at all. There are obviously easier paths. Hetzner's Storage Box. Dropbox. Even a hosted Nextcloud provider. But that's not ownership; that's delegation. I wanted to see my own storage service boot up from scratch, because there's a quiet kind of satisfaction in watching the pieces come alive and realizing, this is mine. ## Setup Everything revolved around Docker. I wrote a clean little docker-compose.yml with four containers: MariaDB, Redis, Nextcloud, and a Cron worker. No external dependencies, no magic scripts. Just containers and mounted volumes. Here's the core of it: ```yaml services: db: image: mariadb:12.0.2 restart: always command: > --transaction-isolation=READ-COMMITTED --binlog-format=ROW --innodb-buffer-pool-size=1024M volumes: - /srv/nextcloud/db:/var/lib/mysql environment: - MYSQL_ROOT_PASSWORD=... - MYSQL_DATABASE=nextcloud - MYSQL_USER=nextcloud - MYSQL_PASSWORD=... redis: image: redis:7-alpine restart: always command: ["redis-server", "--maxmemory", "256mb", "--maxmemory-policy", "allkeys-lru"] volumes: - /srv/nextcloud/redis:/data app: image: nextcloud:32.0.0-apache restart: always depends_on: [db, redis] ports: - 8080:80 environment: - MYSQL_HOST=db - REDIS_HOST=redis - PHP_MEMORY_LIMIT=1024M volumes: - /srv/nextcloud/app:/var/www/html - /srv/nextcloud/data:/var/www/html/data cron: image: nextcloud:32.0.0-apache depends_on: [db, redis] entrypoint: /cron.sh volumes: - /srv/nextcloud/app:/var/www/html - /srv/nextcloud/data:/var/www/html/data ``` Simple, clean and repeatable. Except it didn't work. The Nextcloud container couldn't reach the internet. Not even a curl google.com from inside the container worked. I went down every rabbit hole: DNS settings, IPTables, VCN routes, even Cloudflare Tunnel's network binding. Nothing changed. Three hours gone. Then I noticed something: Docker was installed via Snap. This server was old, probably an artifact of some experiment from two years ago. Snap had quietly sandboxed Docker in a way that broke outbound connectivity for containers. Once I ripped out Snap and reinstalled Docker from apt. Everything started working like magic. One of those moments where you simultaneously feel relieved and slightly betrayed by your past self. ## The Cloudflare Tunnel Next step: make it accessible to the outside world. Instead of opening port 8080 (or reverse proxying it) and managing SSL certificates (although it can be done with certbot), I ran a Cloudflare Tunnel using cloudflared. My config.yml looked like this: ```yaml tunnel: ingress: - hostname: lorbic.com service: http://localhost:8080 - service: http_status:404 ``` Newer version of cloudflared does not rely on the `credential-file` based config. It needs you to directly use the run command. So, be mindful of the version you're using when setting up the tunnel. That's it. Cloudflare handles DNS, SSL, and traffic routing. My VM stays hidden behind the tunnel. No open ports. No certbot. No firewall headache. From my perspective, it's almost unfair how simple it is. The tunnel connected within seconds, and I could access https://lorbic.com from anywhere in the world. The Nextcloud setup wizard appeared. I connected it to the MariaDB container, configured Redis, and within minutes had my own private cloud running securely behind Cloudflare. ### Some Issues Everything looked good until I tried to log in from the Android and Mac Nextcloud apps. It threw a cryptic error: "The polling URL does not start with HTTPS despite the login URL starting with HTTPS". Basically, the app was paranoid, and rightfully so. If the polling endpoint (used during login) isn't HTTPS, it refuses to proceed for security reasons. After some investigation, I realized I had misconfigured the hostname in the Nextcloud setup. I was using the http version in my domain. So, while my login started with HTTPS (https://lorbic.com), the polling went to the tunnel URL, which was HTTP. Fixing the Nextcloud config to use the HTTPS version of the domain and adding some header related config solved it. Change it here: file: `nextcloud/app/config/config.php` ```php 'trusted_domains' => array ( 0 => 'lorbic.com', 1 => 'localhost', 2 => '.cfargotunnel.com', ), 'overwrite.cli.url' => 'https://lorbic.com', 'overwriteprotocol' => 'https', 'trusted_proxies' => ['127.0.0.1', '::1'], 'forwarded_for_headers' => ['HTTP_X_FORWARDED_FOR'], ``` ### The Warp Problem Then came Cloudflare Warp, the VPN. I like Warp because it's secure and integrates beautifully with the Cloudflare network. But here's the irony: when I connected Warp on my phone, https://lorbic.com stopped working. Warp, being part of the same Cloudflare ecosystem, was actually routing my requests differently, through Cloudflare's internal network rather than over the open internet. And since I had configured the tunnel for external access only, Warp essentially short-circuited the route. The workaround was simple: add a local override or exclude that domain from Warp's routing. That way, Warp would stop being too clever for its own good. _There are proper ways to setup Cloudflare Tunnel to work with Warp, but that's a topic for another post_. ## What I Learned Beyond the technical triumph, this little project taught me how powerful modern abstractions have become. Cloudflare Tunnel abstracts away networking in the same way Docker abstracts away system configuration. Together, they turn the once-daunting process of self-hosting into something playful, something you can do in an evening with coffee and a terminal. I also ended up writing a simple cron job to sync some important directories from the cloud volume to an S3 bucket. It's not elegant, but it's effective, a redundancy layer for my "private Google Drive". More importantly, I now have a personal cloud that I control. No ads. No quota nags. No "Your storage is almost full". Just a small, efficient, self-contained service that does exactly what I want, and nothing I didn't agree to. There's something oddly satisfying about this kind of engineering. Not because it saves money, but because it gives back a sense of ownership, that feeling of knowing your system, line by line, container by container. When I log into lorbic.com, it doesn't feel like another SaaS dashboard. It feels like home. **PS:** This post is a work in progress. I'll add more details as I go. And there are some simpler and much cheaper ways to get cloud storage than self-hosting Nextcloud. For example, you can use Hetzner Storage Box, which is a great option for a small, private cloud storage solution. Note: The domain `lorbic.com` is just a placeholder. --- --- ## Introducing Quark **Date:** 2025-09-23 **URL:** https://lorbic.com/introducing-quark/ **Description:** A Minimal Note-Taking System for Thoughts, Ideas, Questions, and Actions ## Quark: A Minimal Note-Taking System A few months ago, I was in a meeting where the conversation was moving at lightning speed. Updates, critical decisions, all flying around the room. I opened my laptop, ready to take notes in Notion, but within minutes I was lost. Too many clicks, too much formatting, and many annoying boxes. By the time I found the right template. I had already missed half of the discussion. That was the moment I realized: most note-taking systems are built for after the fact for polishing, arranging, and linking. But when you're in the middle of the storm of thoughts. You don't need polish; you need to write. ## Why I Like Quark I've tried the big tools like Notion, Obsidian, Roam. They're powerful, no doubt. For people who enjoy configuring, linking, and fiddling with templates, they can be great. But I don't like them that much. For me, they get in the way. Instead of writing, I spend a lot of time formatting, arranging, and configuring the tools. Instead of writing. What I wanted was something simpler: a method that lets me write immediately. Whether I'm in a meeting, reading a book, or listening to a lecture, I need to capture what's happening **now**. Refinement can be done later. With Quark, I don't lose important points. I don't pad my notes with fluff. I just write what matters, in plain text. That's the beauty: Quark works in a terminal, in Sublime, or on a piece of paper. That's also how I write all my blogs and notes. With pen and paper first, then simple markdown files. No databases, no custom dashboards, no unnecessary friction. Just thinking, written down. ## What is Quark? Quark is a minimal, universal note-taking system based on three atomic symbols: - `-` → Idea / Statement - `!` → Action / Task - `?` → Question / Doubt That's it. Three prefixes, plus a bit of indentation, and you have a system that works across meetings, research notes, books, and even journaling. Why the name? In physics, _quarks_ are the fundamental particles that make up matter. Small, essential and universal. Quark notes are the same: tiny, fast to write, and the building blocks of larger understanding. ## The Core Syntax Here's all you need to remember: | Symbol | Meaning | Example | | ------ | ----------------- | --------------------------------- | | - | Statement / Idea | `- Revenue grew 12% this quarter` | | ! | Action / Decision | `! Send the draft by Friday` | | ? | Question / Doubt | `? Why did conversions drop?` | Indentation creates hierarchy. Nothing else required. ## Lists and Structure Quark supports both unordered and ordered notes. **Unordered (default):** ``` - Item one - Item two - Sub-item A - Sub-item B ``` **Ordered (when sequence matters):** ``` 1. Step one 2. Step two 1. Sub-step ``` **Quick flow (sequence without numbering):** ``` > Start > Middle > End ``` ## Modifiers (Optional Power-ups) You don't need these to use Quark. But they add expressive punch without complexity. | Symbol | Use Case | Example | | ------ | -------------------- | --------------------------------------- | | \* | Highlight / Emphasis | `- * Critical concept: entropy` | | → | Link / Cause-effect | `- Pressure ↑ → Volume ↓ (Boyle's Law)` | | > | Quote | `> "Knowledge is power." - Bacon` | | #tag | Topic / Label | `- Distributed systems #scalability` | | % | Personal note | `% This connects to my project` | ## Real-World Examples **Meeting Notes** ``` - Marketing update - Campaign CTR up 15% ? Why drop in conversions despite CTR? ! Check landing page load speed ``` **Reading Notes** ``` - Plato: Allegory of the Cave - Prisoners see shadows = illusion - Sun = truth * Core idea: reality ≠ appearances ? Compare with Advaita concept of maya ``` **Learning (Programming)** ``` - Goroutines are lightweight threads in Go - Managed by Go scheduler - Not 1:1 with OS threads ? What is max concurrency limit? ! Write demo with 1000 goroutines ``` ## Why Quark Works 1. **Speed** You're never stuck choosing a format. Just `- ! ?`. Everything else is optional. 2. **Universality** Works on paper, in a terminal, or in your favorite editor. No special software needed. 3. **Scalability** Hierarchy comes from indentation. Tags and symbols keep meaning intact even across hundreds of notes. 4. **Searchability** In text editors, you can instantly jump to all actions (`/^!`) or all questions (`/^?`). In a notebook, the symbols stand out visually. ## The Philosophy of Quark Quark is deliberately kept small. It resists the temptation of sprawling frameworks. The Quark philosophy is simple: - Notes are atoms of thought. - Keep them fast to capture and easy to scan. - Don't let the medium (software, notebook, template) become the bottleneck. You can always refine Quark notes later into essays, reports, or flashcards. But the raw capture stays lightweight and universal. ### Quick Workflow Summary 1. **Capture fast**: `- ! ?` with optional `* → # % >` 2. **Stream by context**: Reading, meetings, learning 3. **Tag and link lightly**: Only when meaningful 4. **Review regularly**: Refine, consolidate, prioritize 5. **Act on `!` and investigate `?`**: Close the loop on actions and knowledge ## How Quark Compares to Other Note-Taking Systems You might wonder: why not just use an existing method like Cornell, Zettelkasten, or Bullet Journal? Each has its strengths, but here's why Quark stays leaner: - **Cornell Notes** Great for structured lecture review, but the rigid split (cues, notes, summary) slows you down in real time. Quark keeps capture fast and lets you organize later. - **Zettelkasten** Powerful for long-term knowledge building, but it requires discipline to use effectively. IDs, cross-links, atomic notes. Quark skips the overhead. You can always turn Quark notes into Zettels later if you want. - **Bullet Journal** A creative, flexible paper system, but often drifts into over-customization and decoration. Quark removes the artistry by using just symbols and indentation. In short: those systems are great for refinement. Quark is built for capture. The fastest way to not lose ideas in the moment. ## Conclusion Note-taking doesn't need to be complicated. With Quark, you only need three prefixes and a bit of indentation. The rest is mental freedom. **Quark Rule:** If you can type `- ! ?`, you can capture the world. **Download:** {{< link href="/media/uploads/quark_cheatcheet_by_lorbic.com.pdf" text="The Quark Cheat Sheet" target="_blank" >}} --- --- ## Why Your Password Habits Will Get You Hacked **Date:** 2025-08-13 **URL:** https://lorbic.com/password-security-habits/ **Description:** Practical, unsentimental steps to lock down your passwords, devices, and online identity before someone else owns them. # The Real Cost of Digital Laziness We've built our entire lives on top of fragile passwords. Most people treat them like they treat dental checkups, ignore it until something hurts. By then, it's too late. The numbers are not on your side. Every password you've ever created has probably been leaked in some breach you've never heard about. It doesn't matter that the breach was from a forgotten forum you joined in 2012; attackers don't forget. They run automated "credential stuffing" attacks across every major platform, testing your old passwords against your bank, your email, your cloud storage. One lazy reuse, and they're in. The weak link is always human behavior. We think we're clever with "Password123!" or by swapping 'E' for '3', patterns a brute-force script can guess in milliseconds. We reuse passwords because we tell ourselves, "Nobody would bother hacking me". That's wrong. Nobody is hacking *you*. They're hacking *everybody*, at scale, without even knowing who you are. Security is not paranoia; it's hygiene. You wouldn't leave your apartment door wide open just because you've never been robbed. So why leave your digital life exposed? A compromised email account isn't just a privacy leak, it's a skeleton key to reset every other account you own. **What works?** - Unique passwords for every account. Non-negotiable. - Password managers to store them. The "I don't trust password managers" crowd is already trusting their brain, which is worse. - Two-factor authentication on critical accounts (email, bank, primary cloud). And no, SMS OTP is not enough. - Regular breach checks using services like [HaveIBeenPwned](https://haveibeenpwned.com) to see if your credentials have been exposed. Digital security is boring, until it becomes catastrophic. The same way seatbelts don't make you a better driver, strong passwords won't make you invincible. But they drastically reduce the damage when the inevitable collision happens. Most hacks aren't cinematic scenes of hoodie-wearing geniuses bypassing firewalls. They're just someone logging in with your password, because you gave it to them years ago and never thought twice. You are either disciplined about security, or you're gambling with your entire online identity. There's no middle ground. ## A Checklist to Lock Down Your Digital Life *(No motivation speeches. Just do it.)* **1. Secure Your Email.** - Your email is the control tower of your digital identity. - If it's weak, every "secure" account you own is just one password reset away from theft. - Set a strong password, enable 2FA, and set recovery settings. **2. Get a Good Password Manager.** - Bitwarden, 1Password, or KeePass. - Stop using your brain as storage; it's a terrible database. - Never write down your passwords in your diary. **3. Enable 2FA.** - Authenticator apps (Aegis, Authy, Google Authenticator) or hardware keys (YubiKey). - SMS OTP should be the last resort. - Enable passkeys if the website supports it. Passkeys replace passwords with cryptographic keys tied to your device, making phishing almost impossible. **4. Use Unique Passwords.** - If one account is compromised, it shouldn't unlock the rest of your life. - Use password generators like Bitwarden's password generator. - Prefer passphrases over passwords. **5. Check for Breaches.** - Use [haveibeenpwned.com](https://haveibeenpwned.com) - If your password is in a breach, assume it's public and change it now. **6. Update Your Devices.** - Phone, laptop, browser extensions, outdated software is an open window. **7. Remove Accounts You Don't Use.** - Old, abandoned accounts are breach magnets. Shut them down. **8. Prevention Best Practices.** - Change passwords when account breached or suspected compromised. - For high-value accounts like bank, cloud service etc; rotate passwords regularly. - For banking and other sensitive logins, use a clean browser profile with no extensions, or incognito mode. - Avoid logging into critical accounts on public or shared computers, they may have keyloggers or malware. ## Social Engineering & Phishing Vectors Even the most secure device can be compromised if the attacker convinces you to willingly install or permit malicious code. Common tactics include: 1. Fake App Updates: Pop-ups claiming "Your WhatsApp is out of date, download now" that lead to malicious APKs. 2. Lookalike Apps: Apps in third-party stores with names and icons mimicking legitimate software. 3. Malicious Links: SMS, WhatsApp, or email links that lead to credential phishing sites or drive-by downloads. 4. Fake System Alerts: In-browser alerts saying "Your phone is infected" to prompt a bogus cleaner app download. 5. Compromised QR Codes: Publicly posted QR codes (cafés, events) that link to APKs or phishing pages instead of expected URLs. 6. Public Wi-Fi: The public wi-fi could be setup to decrypt your traffic. It could be an evil-twin access point. Mitigation: - Install only from Google Play or trusted app stores. - Avoid clicking links from unsolicited messages. - Verify app publisher names and permissions before installing. - Use a scanner like VirusTotal Mobile to check suspicious APKs before opening. - Use a trusted VPN (ProtonVPN, MullvadVPN, IVPN). Avoid the VPNs advertised by YouTubers. - Know that VPNs don't make you anonymous, they only encrypt your traffic. - You can host your own VPN with Wireguard, OpenVPN or PI-VPN. - Avoid all free VPNs at all times. If you're not paying for the service, your data is the product. The only widely recommended exception is ProtonVPN's free tier, which is subsidized by paid users and has a transparent privacy policy. A flowchart of a social engineering attack: ![Flowchart of a social engineering attack. Picture by Vikash from Lorbic.com](/images/uploads/android_malware_kill_chain.png) ## Trusted Tools for Digital Security *(No sponsorships. Just what works.)* **Password Managers** - [Bitwarden](https://bitwarden.com) (Free + open-source, cloud sync, self-hostable) - [1Password](https://1password.com) (Paid, polished UI, family plans) - [KeePassXC](https://keepassxc.org) (Free, open-source, offline storage) **2FA / MFA Apps** - [Aegis Authenticator](https://getaegis.app) (Free, encrypted backups, Android) - [Authy](https://authy.com) (Cross-device sync, encrypted, consider disabling cloud-sync if you want maximum control.) - [YubiKey](https://www.yubico.com) (Hardware key, phishing-resistant) **Breach Checkers** - [Have I Been Pwned](https://haveibeenpwned.com) (Email & password leak lookup) - [Firefox Monitor](https://monitor.firefox.com) (Automated breach alerts) **Teach yourself** - [Techlore's resources page](https://techlore.tech/resources/?ref=lorbic.com) (Learning Resource) --- If you skip these steps, you're not being "laid-back about security". You're just volunteering to be the easy target in a very crowded shooting gallery. --- ## Stealth VPN with Outline over HTTPS **Date:** 2025-08-12 **URL:** https://lorbic.com/outline-vpn-over-https/ **Description:** Running Outline VPN over HTTPS (port 443) to slip past censorship. Learn how to blend in with HTTPS traffic and what it actually takes to stay invisible. ## What We Are Doing If you are somewhere that blocks VPNs through deep packet inspection (DPI), SNI filtering, or crude port blocking, the goal is to make your VPN traffic look as boring as possible. The quick win is to put it on port 443, the same port used by HTTPS. That is the lifeline of the modern web, so most firewalls are reluctant to block it. But think of it as changing your outfit but not your walk: just putting Shadowsocks (which Outline runs under the hood) on port 443 does not make it indistinguishable from real HTTPS. To a modern DPI system, it still has a unique fingerprint. If you want to disappear in plain sight, you need to wrap it in actual TLS or another disguise layer. ## Why Port 443 Matters and Its Limits Port 443 is the default for encrypted web traffic. Blocking it would break Gmail, YouTube, Facebook, banking apps, and much more. Using 443 for your VPN helps against simple port-based blocks. It does not stop advanced DPI, which can still detect Shadowsocks' handshake patterns even inside encryption. Think of it as changing your outfit but not your walk. Observers may not see your face, but they still recognize the way you move. ## How Outline Works Outline is a friendly manager and server wrapper for Shadowsocks. It runs a SOCKS5 proxy with strong AEAD encryption and offers an API for generating and revoking access keys. It does **not** speak TLS natively. To mimic HTTPS convincingly, you must add a plugin or a front-end reverse proxy that handles TLS and passes traffic inside a valid protocol like WebSocket. ## 1. Provision Your Server Choose a VPS provider that is not hostile to privacy. Good options include Hetzner, DigitalOcean, and Vultr. The Oracle free tier works if you enjoy tinkering, but it can be unreliable. Stick to Ubuntu 20.04 or 22.04 for compatibility. Open only these inbound ports for now: ``` 443/tcp # VPN traffic disguised as HTTPS 53/udp # DNS (for fallback only, see DNS notes below) ``` ## 2. Install Outline Server Run the official installer: ```bash curl -sS https://getoutline.org/install.sh | sudo bash ``` This will: - Install Docker if needed - Set up the Outline Manager service - Give you an API URL and fingerprint for connecting Manage the server using the [Outline Manager](https://getoutline.org) desktop app. ## 3. Put It on Port 443 By default, Outline listens on a high random port. To change it to 443, you need to modify the Docker Compose configuration directly, as editing `access.txt` won't change the listening port of the Shadowsocks server managed by Outline. The `access.txt` file is for access keys, not server configuration. Edit the Docker Compose mapping to expose port 443 and ensure the Outline service inside the container listens on port 443. First, access your server via SSH and open the `docker-compose.yml` file: ```bash sudo nano /opt/outline/docker-compose.yml ``` Locate the `ports` section for the `shadowsocks` service and ensure it maps the external port 443 to the internal container port 443. It might look something like this initially, with a high random port: ```yaml ports: - "8080:8080" # Example, your random port will be different ``` Change it to: ```yaml ports: - "443:443" ``` Additionally, you might need to ensure the Shadowsocks configuration within the Docker container is also set to listen on port 443. While Outline typically handles this when it sets up the service, if you encounter issues, you may need to check Outline's internal configuration files (though this is less common for simple port changes). For most setups, just changing the `docker-compose.yml` ports mapping is sufficient. Restart: ```bash docker-compose -f /opt/outline/docker-compose.yml down docker-compose -f /opt/outline/docker-compose.yml up -d ``` This gets you the "minimal camouflage" setup. It helps against basic censorship but still looks like Shadowsocks to advanced DPI. ## 4. HTTPS (Recommended) Running Outline over HTTPS significantly enhances security and can help bypass certain network restrictions. This involves setting up a reverse proxy (like Nginx) to handle SSL/TLS encryption and forward traffic to your Outline server. First, ensure you have a domain name (e.g., `vpn.example.com`) pointed to your server's IP address. ### Install Nginx and Certbot ```bash sudo apt update sudo apt install nginx certbot python3-certbot-nginx -y ``` ### Obtain an SSL/TLS Certificate Use Certbot to get a free SSL/TLS certificate from Let's Encrypt for your domain. Replace `vpn.example.com` with your actual domain. ```bash sudo certbot --nginx -d vpn.example.com ``` Follow the prompts. Certbot will automatically configure Nginx to use the certificate and set up automatic renewals. ### Configure Nginx as a Reverse Proxy Create a new Nginx configuration file for your domain: ```bash sudo nano /etc/nginx/sites-available/vpn.example.com ``` Add the following configuration. This assumes your Outline server is listening on `localhost:443` (as configured in the previous step). If you changed Outline's port, adjust `proxy_pass` accordingly. ```nginx server { listen 80; server_name vpn.example.com; return 301 https://$host$request_uri; } server { listen 443 ssl; server_name vpn.example.com; ssl_certificate /etc/letsencrypt/live/vpn.example.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/vpn.example.com/privkey.pem; ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers 'TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384'; ssl_prefer_server_ciphers off; location / { proxy_pass http://127.0.0.1:443; # Or your Outline's listening address and port proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_buffering off; proxy_max_temp_file_size 0; } } ``` ### Enable the Nginx Configuration Create a symbolic link to enable the new configuration and then test Nginx syntax and restart. ```bash sudo ln -s /etc/nginx/sites-available/vpn.example.com /etc/nginx/sites-enabled/ sudo nginx -t sudo systemctl restart nginx ``` With this setup, Nginx handles the HTTPS connection, decrypts the traffic, and forwards it to your Outline server, which can now listen on its internal port (e.g., 443) without directly managing SSL/TLS certificates. ## 5. Cloudflare Routing Cloudflare does not proxy raw TCP over 443 unless you pay for [Spectrum](https://www.cloudflare.com/products/cloudflare-spectrum/). The free plan works for HTTP, HTTPS, and WebSocket. Therefore, to route your Outline VPN traffic through Cloudflare's free tier, your Outline client must use a WebSocket plugin. Configure your Nginx (as described in the HTTPS section) to proxy WebSocket traffic to your Outline Shadowsocks server. This involves adding a `location` block in your Nginx configuration to handle a specific WebSocket path (e.g., `/your_websocket_path`) and proxy it to your Outline server's internal address and port. On the client side, ensure your Shadowsocks client is configured to use a WebSocket plugin (like [v2ray-plugin](https://github.com/shadowsocks/v2ray-plugin)) and the specified WebSocket path. For example, with Shadowsocks-Rust, your client configuration might look like this: ```json { "server": "your_domain.com", "server_port": 443, "password": "your_password", "method": "chacha20-ietf-poly1305", "plugin": "v2ray-plugin", "plugin_opts": "tls;host=your_domain.com;path=/your_websocket_path" } ``` This setup makes your VPN traffic appear as regular web traffic to Cloudflare, further enhancing obfuscation and bypassing advanced deep packet inspection (DPI). ## 6. Connect Your Client Download the Outline client for your platform: - [Android](https://play.google.com/store/apps/details?id=org.outline.android.client) - [iOS](https://apps.apple.com/us/app/outline-app/id1356177741) - [Windows, Mac, Linux](https://getoutline.org) Paste in your access key from the Outline Manager. If you switch from raw IP to a domain for TLS, you must regenerate or edit the key to match the new hostname. ## 7. Test Your Stealth Testing is more than "it connects". Use these: - [gfw.report](https://gfw.report) to check if the handshake is blocked in censorship regions - Wireshark or [mitmproxy](https://mitmproxy.org) to inspect packet patterns Minimal camouflage will still show a Shadowsocks signature. Strong camouflage should be indistinguishable from normal HTTPS. > **DNS warning:** Many censorship regimes hijack or block UDP 53. For better stealth, configure DNS-over-HTTPS (DoH) or DNS-over-TLS (DoT) on your device or in the Outline client. ## Active Probing Defense In some places, censors will connect to your IP on 443 and try to speak Shadowsocks. If your server replies, they blacklist you. Plugins like [Cloak](https://github.com/cbeuw/Cloak) or nginx fronting can drop these connections unless they match a known pattern. ## Summary - Port 443 is a good first step, but on its own it only beats basic port blocking. - For serious stealth, add a TLS or WebSocket transport with a valid certificate. - Cloudflare routing only works if traffic is inside HTTP or WebSocket. - Protect against active probing in high-risk environments. ## References and Further Reading - [Outline Server GitHub](https://github.com/Jigsaw-Code/outline-server) - [Shadowsocks Plugins](https://shadowsocks.org/doc/plugins.html) - [v2ray-plugin](https://github.com/shadowsocks/v2ray-plugin) - [Cloak](https://github.com/cbeuw/Cloak) - [Let's Encrypt](https://letsencrypt.org/) - [gfw.report](https://gfw.report) If you only need to get around crude blocks, changing Outline to port 443 might be enough. If you are under active, sophisticated censorship, you will need to dress it up in TLS and maybe hide it behind something bigger like Cloudflare. I will try this idea and write another blog detailing the journey. The more it looks and acts like normal HTTPS, the longer it will survive. --- ## How to Resolve Huge Git Merge Conflicts Without Losing Your Mind **Date:** 2025-07-17 **URL:** https://lorbic.com/resolving-huge-merge-conflicts/ **Description:** A no-fluff, step-by-step guide for software engineers dealing with massive Git merge conflicts during migrations or long-lived branches. Includes real anecdotes and ASCII diagrams. **How to Resolve Huge Git Merge Conflicts Without Losing Your Mind** If you've ever been knee-deep in a long-lived branch merge and thought, _"this can't be what version control was meant for"_, you're not alone. Git gives us powerful tools, but resolving large merge conflicts, especially during major migrations or rewrites can feel like surgery without anesthesia. In this post, we'll walk through how to approach and resolve massive merge conflicts systematically. Not with hand-wavy advice, but with actual steps, real mental models, and clear visual understanding. Let's say you're working on a legacy codebase, maybe something ancient like Django 1.10, and you're helping move it to Django 4.2. That's what I had to do once. The problem? Two long-running branches with divergent histories. One branch added features while the other refactored the core framework. Two months of drift meant the merge base was ancient history. When we finally attempted the merge, Git didn't just report conflicts; it presented a battlefield. Time to toss a coin to your engineer. Here's what we saw: 

*--*--*--*--*--*-- Feature Branch
 \
  *--*--*--*--*--*--*--*-- Main (Updated Django)
Trying to merge led to a wall of conflicts across dozens of files. Looks like rain. Here's how to approach this situation methodically. ### 1. Stop. Don't Start Fixing Right Away. Before typing `git merge`, take a moment. Understand the nature and scope of the divergence. - **See commits exclusive to each branch:** - `git log main..feature-branch --oneline` (commits in `feature-branch` but not `main`) - `git log feature-branch..main --oneline` (commits in `main` but not `feature-branch`) - **See the total diff:** - `git diff main...feature-branch` (three dots) shows all changes on `feature-branch` since the common ancestor. This is the work you are trying to merge in. This preliminary investigation tells you whether you're dealing with a skirmish or a war. Hmm. ### 2. Visualize the Conflict Use: ```bash git log --graph --oneline --all --decorate ``` Or: ```bash gitk --all ``` Or use a GUI tool like **gitk**, **Sublime Merge**, or your IDE's Git history viewer. These help you find the _merge base_: the common ancestor commit from which the two branches diverged. You can find it on the command line too: ```bash git merge-base main feature-branch ``` Every conflict is a story about how two different developers modified the same starting point (`BASE`). Understanding that `BASE` commit is key. ### 3. Create a Temporary Merge Playground Never fight the monster directly on the main branch. ```bash git checkout -b conflict-playground feature-branch git merge main ``` Now you're in a temporary space to resolve without pressure. ### 4. Know What You're Resolving Use a 3-way diff tool. For example, `meld`, `vscode`, or `sublime-merge` diff view. A 3-way diff tool is non-negotiable. Configure it first: ```bash git config --global merge.tool meld # Or vscode, p4merge, etc. ``` Then run `git mergetool` to open the first conflicted file. It will show you three versions: - **`LOCAL`**: Your version (`HEAD`, i.e., `main`). - **`REMOTE`**: Their version (the one you're merging in, i.e., `feature-branch`). - **`BASE`**: The common ancestor. The state of the file before either branch made changes. Your job is not to choose between `LOCAL` and `REMOTE`. It's to integrate the _intent_ of both changes into the `BASE` to create the final, `MERGED` result. ### 5. Triage: Divide and Conquer List all conflicts to see what you're up against: `git status --short | grep "^U"` Then, group them into categories: 1. **File-level conflicts**: One branch deleted a file that the other modified (`DU` or `UD`), or both added the same file (`AA`). Resolve these first with `git rm` or `git add`. 2. **Trivial content conflicts** (`UU`): Whitespace, comments. Easy wins. 3. **Structural conflicts** (`UU`): A class was renamed in `main` while a method was added to it in the feature. The tool will show a huge conflict, but the resolution is often straightforward: apply the feature's logic to the newly renamed code. 4. **Logical conflicts** (`UU`): Both branches modified the same piece of logic in incompatible ways. These require the most thought and careful testing. Tackle them in that order. Steel for humans, silver for monsters and planning for complex merge-conflict. Gaining momentum is half the battle. ### 6. Use `git merge --abort` When Overwhelmed Made a mess? Damn it. No problem. ```bash git merge --abort ``` Back to a clean slate. You can retry as many times as needed. ### 7. Use Commands to Resolve Files Quickly For some files, you know one side is definitively correct. Don't open the mergetool; use a command. - To accept your version (from `main`): `git checkout --ours path/to/file.js` - To accept their version (from `feature-branch`): `git checkout --theirs path/to/file.js` This is extremely efficient for structural refactors where you know the `main` branch's changes (e.g., a file move or rename) are the correct "canvas" to paint the feature changes onto. After checking out the correct base version, you can manually re-apply the other side's changes. For manual resolution, open the file and look for the conflict markers. Edit the code to combine the sections, remove the `<<<<<<<`, `=======`, and `>>>>>>>` markers, and then `git add path/to/file.js`. ### 8. Run Tests Aggressively After Each Batch After every 5–10 files, run your tests. Don't wait till you've resolved 100 conflicts to see what broke. Think of it as taking a potion before the fight, not after you're already bleeding out. ### 9. Use `git rerere` to Avoid Redoing Work If you abort and retry a merge, `rerere` (Reuse Recorded Resolution) is your best friend. ```bash git config --global rerere.enabled true ``` Once enabled, the first time you resolve a hunk of a conflict and `git add` the file, Git records the "before" (the conflict state) and "after" (your resolution) in the `.git/rr-cache` directory. If you abort and later encounter the exact same conflict, Git will automatically apply your recorded resolution. This is a massive time-saver. You can even commit the `rr-cache` directory to share conflict resolutions with your team. *⚠️ Note: `rerere` works best when the conflicting hunks are *identical* to the previous conflict. If the context or line numbers shift slightly, Git might not auto-resolve them. It's useful, but not magic.* ### 10. Distinguish a Strategy from an Option Sometimes you want to favor one branch's changes during a merge. There's a sharp tool for this, but you must know the difference between a strategy and an option. - **The `ours` strategy**: `git merge -s ours feature-branch` This is a blunt instrument. It creates a merge commit that includes `feature-branch` as a parent, but **discards all of its changes entirely**. The resulting code is identical to `main`. This is useful for marking a dead-end feature as "merged" to clean up branch history, but it doesn't integrate any code. - **The `-X ours` or `-X theirs` option**: `git merge -X ours feature-branch` This is far more useful for complex merges. It tells the standard `recursive` merge strategy to attempt a normal merge, but for any hunk that has a conflict, it automatically resolves it by choosing _your_ side (`-X ours`) or _their_ side (`-X theirs`). This is powerful for refactoring merges where you know one side's changes are globally more important. ### 11. ASCII Summary: Merging Chaos into Order 

Before:
Feature:   A -- B -- C -- D
                  \
Main:              X -- Y -- Z
Conflict explosion: merging D and Z Resolution Strategy: 1. Visualize 2. Playground Branch 3. Triage by File Type 4. Resolve in Order (Trivial → Structural → Logical) 5. Test Often --- That old Django migration? It took me three solid days to resolve the merge. I didn't use any fancy tools, just `git mergetool`, `sublime-merge` diff viewers, and manual review. What saved me wasn't skill; it was method. Large merge conflicts are not bugs. They're symptoms of drift. And you don't fight drift by panic-merging. You fight it with discipline, snapshots, and composure. Git won't make it easy. But it gives you enough rope to build a bridge or hang yourself. Your call. ### Resources Worth Reading 1. [Git Branching - Basic Branching and Merging](https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging) 2. [Advanced Merging](https://git-scm.com/book/en/v2/Git-Tools-Advanced-Merging) 3. [git-mergetool Documentation](https://git-scm.com/docs/git-mergetool) ### Git Conflict Glossary (Quick Reference) - **BASE**: The common ancestor of the two branches being merged. - **LOCAL**: Your current branch (the one you're merging _into_). - **REMOTE**: The branch you're merging _from_. - **ours**: Refers to LOCAL in conflict resolution. - **theirs**: Refers to REMOTE in conflict resolution. - **Merge strategy -s recursive**: Git's default merge strategy that tries to automatically resolve conflicts using a three-way merge. **If you've got a merge war story, I'd like to hear it. Or better yet, avoid one, by rebasing early and often. Don't let your codebase become a cautionary tale.** --- --- ## What Facebook's Memcache Taught Me About Systems Thinking **Date:** 2025-07-07 **URL:** https://lorbic.com/scaling-memcache-facebook/ **Description:** A deep dive into Facebook's memcache architecture and the hard lessons it teaches about real-world system design, performance, and failure. What Facebook's Memcache Taught Me About Systems Thinking > "The probability of reading transient stale data is a tunable parameter". - Scaling Memcache at Facebook (NSDI, 2013) There's a moment in every engineer's life when a seemingly simple component like a cache, suddenly becomes the most complex piece in the stack. For me, that moment arrived reading Facebook's paper on scaling Memcache. I didn't expect a key-value store to challenge my understanding of systems architecture. But this paper wasn't about cache keys or TTLs. It was about what happens when infrastructure hits the limits of scale, physics, and human reliability. What follows isn't a summary. It's a set of systems insights that stayed with me, principles that go deeper than code and that I now see everywhere. #### 1. Keep the Core Dumb. Push Complexity Outward. Facebook's memcached servers are intentionally brain-dead. They don't coordinate. They don't know each other exists. There's no replication, gossip protocol, or consensus layer. Instead, all intelligence like request routing, error handling, retries, batching is pushed into the client. The client might be an embedded library, or a proxy called mcrouter, which looks like a memcached server but simply forwards requests based on consistent hashing and configuration state. At first, this seems backwards. Wouldn't smart servers reduce coordination overhead? Wouldn't a central cache service simplify consistency? But at their scale, the opposite holds true. By keeping memcached dumb and stateless: - They can deploy and upgrade it independently. - Failure recovery becomes local: just restart or replace the box. - There's no server-side coordination failure to debug at 2 a.m. The lesson here is architectural, not technical: > Complexity doesn't vanish. But if you push it to the edge into clients it becomes easier to version, test, and evolve. Dumb, fast, and replaceable beats smart and brittle. #### 2. Systems Thinking Begins Where Feature Thinking Ends Most of us reach for a cache to "improve performance". That's a feature mindset. But Facebook engineers were solving something else: survival at scale. Example: they used UDP for GET requests. Not TCP. UDP doesn't guarantee delivery or order. Why use it? Because: - It has less overhead per connection. - It avoids memory-hungry TCP state on the server. - It allows each thread to talk directly to memcached with minimal coordination. Yes, they dropped 0.25% of packets. They didn't retry. They just treated those as cache misses and moved on. At a billion QPS, a few dropped packets don't matter. A congested server does. Another example: leases. A lease is a token that a client gets on a cache miss. Only that client is allowed to re-populate the value. Everyone else waits briefly. Why? Because without this, when a hot key is evicted, thousands of clients might simultaneously hit the database to fetch and repopulate it. That's called a thundering herd. You don't want that during peak load. Leases rate-limit writes back into cache. They also prevent stale sets, cases where two clients race to set different values for the same key, and the loser overwrites the winner. These are not elegant solutions. They are pragmatic design decisions. They're what you build when you're no longer solving "how do I cache this" but "how do I keep the backend alive". > Systems thinking asks not "what's ideal", but "what fails gracefully, what scales, and what I can fix under pressure". #### 3. Consistency Is a Dial We're taught to think of consistency as sacred: the database is right, and the cache must reflect it. But Facebook intentionally treats consistency as a performance lever. Their memcache infrastructure accepts that cached data can be stale, duplicated, or missing, as long as the user experience doesn't suffer. How do they do it? They treat the cache as discardable. The database is the source of truth. Cache can be evicted, expired, or wrong, as long as the database eventually corrects it. They even allow slightly stale values to serve requests after deletes. Instead of immediately purging deleted keys, they move them to a short-lived "recently deleted" buffer. If a client requests that key again soon, it might get a _marked-as-stale_ value and proceed anyway, especially if the data is not user-critical. In globally distributed systems, they use remote markers. If a region is lagging in replication, a client can mark a key as _recently deleted_ in a remote region. Other clients, on seeing the marker, are redirected to the master region for fresh data. This isn't strong consistency. It's controlled inconsistency. And it works. Once you realize that consistency can be tuned, like latency or throughput, you start seeing your cache as a system of probabilities, not guarantees. #### 4. Don't Just Cache for Speed, Cache for Economics Facebook's memcache isn't a single cache. It's a collection of memory pools, each tuned to a different access pattern. Why? Because not all keys are equal. Some are: - Accessed frequently but cheap to miss (e.g., user online status). - Rarely accessed but expensive to compute (e.g., machine learning model outputs). - Frequently updated and high-churn (e.g., news feed orderings). If you put these into one common pool, the result is negative interference: - Hot, volatile keys evict long-lived but valuable data. - Rare items get flushed before they're reused. - Hit rates drop, and backend load increases. So they created segregated pools: - A wildcard pool for general use. - An app-specific pool for temporary data. - A replicated pool for hot data served from multiple servers. - A regional pool for large, rarely accessed items shared across clusters. Each pool can be tuned, size, eviction policy, replication strategy, without hurting the others. In other words, they cache not just for speed, but for cost-efficiency. The question isn't "how do I cache this?" It's "what is the cost of a miss, and is that cost worth the memory?" #### 5. Failure Is Not the Exception At Facebook's scale, servers fail constantly. Network partitions happen. Machines go dark. It's not an incident, it's Tuesday. The naive approach on cache failure is to rehash keys onto the remaining servers. But in their system, key access is skewed. A single key might account for 20% of a server's load. Rehashing it elsewhere could overload the new server and create a cascade failure. Their solution? *Gutter pools*. Gutter servers are a small fraction (~1%) of memcached boxes. When a regular server fails, clients don't retry the same key elsewhere. They redirect the request to the Gutter pool. If the Gutter doesn't have the value, it fetches from the DB and inserts it, temporarily. These entries expire quickly. The point isn't to serve 100% correct data. The point is to absorb the shock. To give the system a soft place to land while the real servers recover. This is a lesson in system realism. Don't try to prevent all failure. Design for graceful degradation. #### 6. Performance Lives Most performance advice stops at averages: reduce latency, increase QPS. But real systems hurt in the tail latencies, the 95th, 99th, 99.9th percentiles. Facebook found: - A popular page triggers hundreds of memcache GETs. - A slow server or delayed response creates fan-out latency. - Incast congestion can flood switches when too many responses arrive simultaneously. To fix this, they: - Used batching to reduce round-trips. - Employed sliding windows to limit outstanding requests. - Tuned window size to balance between underutilization and congestion. They also studied memory fragmentation and introduced: - Adaptive slab rebalancing: reclaim slabs from underused classes and give them to those under memory pressure. - Transient item cache: proactively purge expired items to prevent wasted memory. All of this tuned not just throughput, but predictability. Not just average speed, but worst-case resilience. Tail latency isn't a technical problem. It's an experience problem. It's what the user actually feels. #### 7. Systems Thinking as Infrastructure Literacy Reading this paper made me realize that systems thinking is not optional. It's the difference between building something that works and something that survives. Facebook's memcache is full of trade-offs I wouldn't have guessed: - Serving stale data to save a DB. - Using UDP with dropped packets on purpose. - Letting cache invalidations lag, on purpose, to keep systems flowing. This is the kind of engineering that comes from pressure, not planning. From watching things break and learning what actually matters. I came away not just with design ideas, but with a new lens on how to build systems. It questions purity, tolerates imperfection, and optimizes for the system, not the module. I don't need a billion QPS to apply that. ---- References: [1] [Scaling Memcache at Facebook ](https://research.facebook.com/publications/scaling-memcache-at-facebook/) --- ## Bitmasking In Go **Date:** 2025-04-17 **URL:** https://lorbic.com/bitmasking/ **Description:** Bitmasking is one of those computer science tricks that feels like wizardry, until you realize it's just some clever shifting and binary math. This blog explores the idea, shows how we use it in Go, and why it's surprisingly useful when working with databases like Couchbase. # Bitmasking in Go Bitmasking is one of those computer science tricks that feels like wizardry, until you realize it's just some clever shifting and binary math. This blog explores the idea, shows how we use it in Go, and why it's surprisingly useful when working with databases like Couchbase. --- ## What's Bitmasking? A **bitmask** is just an integer where each **bit** (0 or 1) represents a flag or state. Instead of storing multiple booleans in a slice or map, you cram them into a single `int`. Fast to compute, fast to store, and great for indexing. Let's get into what actually happens. --- ## How Bitmasking Actually Works Let's say you want to represent a few features: | Feature Name | Bit Index | | ------------ | --------- | | Feature A | 0 | | Feature B | 1 | | Feature C | 2 | | Feature D | 3 | We don't store `[true, false, true, false]`. We store a single number. ### Encoding To encode which bits are **on**, we use the left-shift operator `1 << n`. This is how we turn a bit position into a power of two. For example: ```go 1 << 0 // = 1 = 00000001 1 << 1 // = 2 = 00000010 1 << 2 // = 4 = 00000100 1 << 3 // = 8 = 00001000 ``` To combine them, we use **bitwise OR** (`|=`): ```go mask := 0 mask |= 1 << 1 // enable Feature B → mask = 2 mask |= 1 << 3 // enable Feature D → mask = 10 ``` So: ```go Encode([]int{1, 3}) = 10 // Binary: 00001010 ``` ### Decoding To get the list of enabled features back, we use **bitwise AND** (`&`) to check each bit one by one: ```go for i := 0; i < 32; i++ { if mask & (1 << i) != 0 { fmt.Println("Bit", i, "is set") } } ``` Why does this work? Because `(mask & (1 << i))` will only be non-zero **if** bit `i` is set in `mask`. For example: ```go mask := 10 // 00001010 1 << 3 = 8 // 00001000 10 & 8 = 8 // Bit 3 is set 1 << 1 = 2 // 00000010 10 & 2 = 2 // Bit 1 is set 1 << 2 = 4 // 00000100 10 & 4 = 0 // Bit 2 is NOT set ``` ### Toggle a Bit To flip a bit (on → off, or off → on), use **bitwise XOR** (`^`): ```go mask := 10 // 00001010 mask ^= 1 << 3 // Bit 3 was 1 → now 0 → mask becomes 2 ``` ### Summary of Operators | Operator | Meaning | Use Case | | -------- | ------------------- | ------------------- | | `1 << n` | Bit at position `n` | Build mask | | `\|=` | Bitwise OR | Set a bit | | `&` | Bitwise AND | Check if bit is set | | `^=` | Bitwise XOR | Toggle bit (flip) | --- ## The Go Bitmask Package Here's the package I wrote with these concepts built in: ```go // Encode returns a bitmask with bits set for each ID in the input slice. // Each ID is converted using: 1 << id = 2^id func Encode(ids []int) int { var mask int for _, id := range ids { mask |= 1 << id } return mask } // Decode returns which bits are set in the mask. func Decode(mask int) []int { var ids []int for bit := 0; mask != 0; bit++ { if mask&1 == 1 { ids = append(ids, bit) } mask >>= 1 } return ids } // HasBit returns true if the bit at position id is set. func HasBit(mask int, id int) bool { return (mask & (1 << id)) != 0 } // ToggleBit flips the bit at position id. func ToggleBit(mask int, id int) int { return mask ^ (1 << id) } ``` --- ## Why Use Bitmasking in Couchbase? Let's take the example of **feature flags** or **user roles** stored in Couchbase. ### Option A: Store as Arrays (Bad) ```json { "user_id": 123, "roles": [1, 3, 5] } ``` Problems: - Indexing arrays is expensive. Each value creates a separate index entry. - Couchbase will use `DISTINCT SCAN`, which can be heavy. - Filtering with `WHERE ANY r IN roles SATISFIES r = 3 END` is slow under load. ### Option B: Store as Bitmask (Good) ```json { "user_id": 123, "role_mask": 42 } ``` - `42 = 00101010` → roles 1, 3, and 5 are set. - Use a simple numeric index: ```sql CREATE INDEX idx_role_mask ON users(role_mask); ``` - Filter with: ```sql SELECT meta().id FROM users WHERE BITAND(role_mask, 8) = 8; ``` This checks if role 3 is enabled (since `1 << 3 = 8`). Fast, lean, and no complex joins or array scans. --- ## When to Use Bitmasking Good for: - Feature flags - User roles/permissions - Categorical flags with a known upper limit (< 64 ideally) - Couchbase performance under large load Not great for: - Huge dynamic lists - Highly descriptive data (bitmasks are not self-documenting) - Frequent schema changes (bit positions are hard-coded) --- Bitmasking isn't magic; it's just integers in disguise. When used well, it's a powerful way to compress and manipulate data for faster storage and filtering, especially in database systems like Couchbase. If you're tracking a small set of flags or categories, think in powers of two and let your bits do the work. --- ## Docker Volume Backup: The 1-Line Command You Should Be Using (Instead of tar) **Date:** 2025-04-13 **URL:** https://lorbic.com/how-to-backup-and-restore-docker-volumes/ **Description:** When working with Docker, persistent data is often stored in volumes. Unlike container filesystems, volumes survive restarts and recreations. But what if you need to move this data from one machine to another? Here's a quick and reliable way to back up a Docker volume on one computer and restore it on another. # How to Backup and Restore Docker Volumes Between Machines When working with Docker, persistent data is often stored in volumes. Unlike container filesystems, volumes survive restarts and recreations. But what if you need to move this data from one machine to another? Here's a quick and reliable way to back up a Docker volume on one computer and restore it on another. I find this use case very often when I'm working on a new project that changes the database schema and I need to maintain existing code with old schema. To keep both the new project and the existing code working, I create a new volume from my backup, attach it to my docker-compose database service and work on the new project. When I need to go back to the existing code for a bug fix or enhancement, it is a one line change in the docker-compose. ## Step 1: Back Up the Docker Volume Let's say you have a volume named db_data that you want to back up. Run this command from the terminal in your project directory: ```sh docker run --rm \ -v db_data:/source \ -v $(pwd):/backup \ busybox \ tar czf /backup/db_data.tar.gz -C /source . ``` What it does? - `--rm`: Automatically removes the container after it finishes. - `-v db_data:/source`: Mounts the source Docker volume. - `-v $(pwd):/backup`: Mounts your current directory to write the backup file. - `busybox`: A tiny Linux container that includes tar. - `tar czf`: Creates a compressed tarball of the volume contents. This will create a file called `db_data.tar.gz` in your current directory. ## Step 2: Check the Backup Contents Want to double-check what's inside the archive? Use: ```sh tar -tzf db_data.tar.gz | head ``` This lists the first few files inside the backup, so you know it's not empty or corrupt. ## Step 3: Move the Backup to the New Machine Transfer `db_data.tar.gz` to the new machine. You can use scp, ftp, a USB drive, cloud storage, AirDrop, whatever works for you. Example: ```sh scp db_data.tar.gz v@mac2:\~/Downloads ``` ## Step 4: Restore the Docker Volume On the new machine, create an empty volume with the same name: ```sh docker volume create db_data ``` Now restore the backup into that volume: ```sh docker run --rm \ -v db_data:/restore \ -v \~/Downloads:/backup \ busybox \ tar xzf /backup/db_data.tar.gz -C /restore ``` What it does? - `-v db_data:/restore`: Mounts the target volume to write data into. - `-v ~/Downloads:/backup`: Mounts the directory where the tarball lives. - `tar xzf`: Extracts the archive into the volume. ## Step 5: Verify the Restore Option 1: Use Docker Desktop, Go to Volumes and explore `db_data` visually. Option 2: Inspect from the terminal: ```sh docker run --rm -v db_data:/data busybox ls /data ``` This lets you see the contents directly from the CLI. ## Bonus: Use the `dvbackup` Script Source code: [github.com/vk4s/docker-volume-backup](https://github.com/vk4s/docker-volume-backup) If you find yourself doing this often, you might want to use the `dvbackup` script (I have created it), which simplifies the entire process: 1. Install the script: (always check the contents of any script before running it in your system). ```sh curl -O https://raw.githubusercontent.com/vk4s/docker-volume-backup/main/dvbackup chmod +x dvbackup ./dvbackup install ``` 2. Backup volumes: ```sh dvbackup backup ./backups db_data postgres_data ``` 3. Restore volumes (with same names): ```sh dvbackup restore ./backups db_data postgres_data ``` 4. Restore with new names: ```sh dvbackup restore ./backups new_db=db_data new_pg=postgres_data ``` The script provides following features: - Automatic validation of volume names - Safe handling (won't overwrite existing volumes) - Support for multiple volumes in one command - Cross-platform compatibility (macOS, Linux, Windows Git Bash) It's a great tool for developers who frequently need to move Docker volumes between machines or create backups of their data. ## Docker Volume Cheatsheet ### Basic Volume Operations ```bash # Create a new volume docker volume create my_volume # List all volumes docker volume ls # Inspect a volume docker volume inspect my_volume # Remove a volume docker volume rm my_volume # Remove all unused volumes docker volume prune ``` ### Using Volumes with Containers ```bash # Mount a volume to a container docker run -v my_volume:/data my_image # Mount a volume with read-only access docker run -v my_volume:/data:ro my_image # Mount a specific directory as a volume docker run -v /host/path:/container/path my_image # Use a named volume with docker-compose # volumes: # my_volume: # name: my_volume ``` ### Volume Backup and Restore Checkout `dvbackup` above. ### Volume Management ```bash # Check volume disk usage docker system df -v # Backup all volumes in a project docker-compose down dvbackup backup ./backups $(docker volume ls -q --filter name=project_name) # Migrate volumes between hosts dvbackup backup ./backups volume_name # Transfer backup file to new host dvbackup restore ./backups volume_name ``` ### Common Volume Patterns ```bash # Database volumes docker volume create postgres_data docker run -v postgres_data:/var/lib/postgresql/data postgres # Application data docker volume create app_data docker run -v app_data:/app/data my_app # Configuration volumes docker volume create app_config docker run -v app_config:/app/config my_app # Cache volumes docker volume create app_cache docker run -v app_cache:/app/cache my_app ``` --- ## Testing Types **Date:** 2025-03-11 **URL:** https://lorbic.com/testing-types/ **Description:** Understanding different types of software testing is crucial for delivering a reliable application. Smoke Testing checks basic stability after a new build, while Sanity Testing verifies specific bug fixes or minor updates. Functional Testing ensures that features work as expected based on business requirements. Regression Testing prevents new changes from breaking existing functionality. End-to-End (E2E) Testing simulates real-world user workflows, and Performance Testing checks system speed, load handling capacity, and responsiveness. Implementing these (some or all) test types helps maintain software quality and prevent critical failures. Understanding different types of software testing is crucial for delivering a reliable application. Smoke Testing checks basic stability after a new build, while Sanity Testing verifies specific bug fixes or minor updates. Functional Testing ensures that features work as expected based on business requirements. Regression Testing prevents new changes from breaking existing functionality. End-to-End (E2E) Testing simulates real-world user workflows, and Performance Testing checks system speed, load handling capacity, and responsiveness. Implementing these (some or all) test types helps maintain software quality and prevent critical failures. Here's a basic explanation of each test type with simple examples: ## 1. Smoke Testing Purpose: Ensures that the core functionality of the application is working before deeper testing. It acts as a basic stability check after a build or deployment. Example: • Checking if the application launches successfully. • Verifying if a user can log in with valid credentials. • Ensuring the homepage loads without errors. 💡 Analogy: Like checking if a car starts before going on a long drive. ## 2. Sanity Testing Purpose: A quick check of specific functionality after minor changes or bug fixes to ensure they work as expected. It is narrower than smoke testing. Example: • Verifying if a fixed issue (e.g., password reset functionality) works correctly. • Ensuring a new button added to the UI works as expected. • Checking if a database update reflects correctly in the UI. 💡 Analogy: Like fixing a broken switch in a house and ensuring only that switch works, without checking all the lights. ## 3. Functional Testing Purpose: Tests whether the application behaves as expected according to business requirements. It focuses on what the system does. Example: • Testing a shopping cart to see if adding/removing items works correctly. • Checking if an email is sent when a user registers. • Ensuring a user can apply a discount coupon and see the correct price. 💡 Analogy: Like testing if a vending machine dispenses the correct snack when the right button is pressed. ## 4. Regression Testing Purpose: Ensures that recent code changes do not break existing functionality. It is done after bug fixes, updates, or new feature additions. Example: • After updating the login process, testing if previous login methods (Google, Facebook) still work. • Ensuring a new feature in a banking app does not break existing transactions. • After fixing a checkout bug, verifying if other payment options still work. 💡 Analogy: Like fixing a broken pipe in a house and checking if all other plumbing still works. ## 5. End-to-End (E2E) Testing Purpose: Tests the entire user journey across multiple integrated systems, mimicking real-world usage. Example: • Verifying if a user can register, search for a product, add it to the cart, make a payment, and receive an order confirmation email. • Testing if a banking app allows fund transfers, sends notifications, and updates transaction history correctly. Testing. 💡 Analogy: Like checking if a restaurant experience flows smoothly: from ordering food to payment and receiving a bill. ## 6. Performance Testing Purpose: Checks how the system performs under different loads, ensuring speed, scalability, and responsiveness. Example: • Measuring if a website loads within 2 seconds under normal traffic. • Checking if a mobile app can handle 10,000 users at the same time. • Verifying if a database query retrieves results within an acceptable time limit. 💡 Analogy: Like stress-testing a bridge to see how much weight it can handle before it collapses. --- ## Tools I Use on My Mac **Date:** 2025-03-10 **URL:** https://lorbic.com/tools-i-use-on-my-mac/ **Description:** # Tools I Use on My Mac As a new Mac user, it has been a tough journey getting used to limitations of macOS compare to linux based os. I rely on a set of carefully chosen tools to improve productivity, streamline workflows, and make my overall experience less frustrating. Here's a look at top 20 essential apps I use daily: ## System & UI Enhancements ### 1. Scroll Reverser Allows me to reverse the scrolling direction for trackpads and mice independently, making navigation more intuitive. ### 2. BeardedSpice A must-have for controlling media playback on websites and apps using my keyboard's media keys. It can control Spotify. ### 3. KDE Connect Seamlessly integrates my Mac with my Android phone, enabling file transfers, notifications, and remote input control. ### 4. Flux Automatically adjusts my screen's color temperature based on the time of day, reducing eye strain and improving sleep. ### 5. Shottr A lightweight and fast screenshot tool that includes built-in annotation and OCR capabilities. ### 6. Maccy A clipboard manager that keeps track of copied text and allows quick retrieval of previous copies. ### 7. GPG Keychain Essential for managing GPG encryption keys for secure email communication and file encryption. ### 8. Music Decoy Helps me keep block apple music from launching whenever I connect my earphones/headphones. What a stupid idea to auto launch it. ### 9. AltTab Brings a Windows-style Alt+Tab app switcher to macOS, making app switching faster and more convenient. ### 10. Rectangle A powerful window management tool that lets me quickly arrange and snap windows using keyboard shortcuts. ### 11. Monitor Control Gives me precise control over external monitor brightness and volume directly from macOS. ## Development & Productivity ### 12. Docker Enables me to run containerized applications, making development and deployment more efficient. ### 13. iTerm2 A feature-packed terminal replacement that enhances productivity with split panes, custom profiles, and better performance. ### 14. Obsidian My go-to knowledge management tool for organizing notes, ideas, and projects in a Markdown-based system. ### 15. Postman A great API testing tool that helps me debug and develop RESTful services with ease. ### 16. JetBrains Fleet/GoLand/PyCharm A suit of powerful and modern IDEs from JetBrains that provide world's best coding experience. ### 17. VS Code Well, we all know what it is. Anyways, it's a highly customizable and extensible code editor that I use for various programming tasks. ### 18. Sublime Merge A fast and powerful Git GUI that makes managing repositories much smoother. ### 19. Scrcpy A fantastic open-source tool for mirroring and controlling my Android phone from my Mac. ### 20. ChatGPT I use ChatGPT for brainstorming, coding assistance, and getting quick explanations on technical topics. These tools have significantly improved my daily workflow, making my Mac experience more efficient and enjoyable. If you use any of these or have other recommendations, do let me know! --- ## Understanding T and *T method receivers in Go **Date:** 2025-02-23 **URL:** https://lorbic.com/method-receivers-in-go/ **Description:** In Go, method receivers determine whether a method acts on a copy of a value or a reference to it. This choice isn't just about performance; it affects correctness and behavior, especially when dealing with synchronization primitives (mutex, wait group, etc), slices, and embedded types. In Go, method receivers determine whether a method acts on a copy of a value or a reference to it. This choice isn't just about performance; it affects correctness and behavior, especially when dealing with synchronization primitives (mutex, wait group, etc), slices, and embedded types. In this post I explore when to use T (a value receiver) vs. \*T (a pointer receiver) and why, in most cases, pointer receivers should be the default and preferred. --- #### Understanding T and \*T In Go, every type T has an associated pointer type \*T. Taking the address of a T variable results in a \*T: ```go type T struct { a int b bool } var t T // t is of type T var p = &t // p is of type *T ``` T and \*T are distinct types. You can't substitute \*T where T is expected and vice versa. #### Declaring Methods on Types Go allows you to declare methods on any type you define in your package. That means you can attach methods to both T and \*T: ```go func (t T) ValueMethod() { /* operates on a copy */ } func (t *T) PointerMethod() { /* operates on the original */ } ``` Now the billion dollar question: When should a method use a pointer receiver? #### When to Use a Pointer Receiver ##### 1. The method modifies the receiver. If your method changes the state of T, you must use \*T, or modifications won't persist. ```go func (p *Person) Rename(name string) { p.Name = name } ``` ##### 2. The type is large. Passing large structs by value incurs unnecessary copying. ```go type BigStruct struct { Data [1024]byte } func (b BigStruct) BadMethod() { /* Copies 1024 bytes */ } func (b *BigStruct) GoodMethod() { /* Uses reference of the 1024 bytes */ } ``` ##### 3. The type contains synchronization primitives (e.g., sync.Mutex). Copying a sync.Mutex defeats it purpose of maintaining the state. ```go type Counter struct { mu sync.Mutex val int } func (c *Counter) Increment() { c.mu.Lock() defer c.mu.Unlock() c.val++ } ``` Declaring Increment on Counter (not \*Counter) would copy mu, making it useless. ##### 4. The type is meant to be used via pointers. Some types naturally work as pointers, such as linked list nodes. ##### 5. The type is embedded in another struct. If a type is meant to be embedded, methods must use pointer receivers to avoid unintended copies. Copies introduce performance penalty. ```go type Stats struct { a, b, c Counter } func (s Stats) Sum() int { return s.a.Increment() + s.b.Increment() + s.c.Increment() // Copies `Counter`, breaks mutex } ``` #### When To Use a Value Receiver? A value receiver is fine if: - The method does not modify the receiver. - The type is small and simple (like int, bool, or struct{X, Y int}). - The type is inherently immutable. For example: ```go type Point struct { X, Y int } func (p Point) Distance(q Point) float64 { dx, dy := float64(q.X-p.X), float64(q.Y-p.Y) return math.Sqrt(dx*dx + dy*dy) } ``` Here, Distance doesn't modify Point, and copying two ints is negligible. #### Should Methods That Don't Mutate Use a Pointer Receiver? Yes, in most cases. Even if a method doesn't modify the receiver, using \*T avoids unintended copies and maintains consistency across your codebase. The exceptions: - Small, immutable types (e.g., Point). - Types that should be copied intentionally, like function options or temporary configurations. Basically: - Default to using \*T (reference receiver) unless you have a good reason not to. - Use T (value receiver) only when your type is small and meant to be copied. - Never use T (value receiver) if the type has a sync.Mutex, a slice, or is embedded in another struct. By following these guidelines, you avoid subtle bugs and unintended performance penalties in your Go code. Cheers!! Happy coding. --- ## Analysis Paralysis in Engineering Teams **Date:** 2025-02-10 **URL:** https://lorbic.com/analysis-paralysis/ **Description:** In engineering, progress is key. However, sometimes teams get stuck in endless discussions, over-planning, and constant changes, delaying actual work. This situation is called "Analysis Paralysis." It happens when people focus too much on making perfect decisions instead of moving forward with practical solutions. Analysis Paralysis in Engineering ![An image representing "Analysis Paralysis" in engineering. It shows a team stuck in decision-making and hesitant to take action.](/images/uploads/image.jpeg) In engineering, progress is key. However, sometimes teams get stuck in endless discussions, over-planning, and constant changes, delaying actual work. This situation is called "Analysis Paralysis." It happens when people focus too much on making perfect decisions instead of moving forward with practical solutions. One common cause of analysis paralysis is fear of failure. Engineers and managers want to avoid mistakes, so they keep analyzing every possible outcome. While planning is important, overthinking can waste time and resources. Another reason is complex decision-making. If too many people are involved, or if there are too many options, teams struggle to decide and keep delaying action. For example, imagine a software development team working on a new app. They spend weeks debating the best database technology: SQL or NoSQL. They read articles, compare benchmarks, and discuss performance differences. Meanwhile, no actual coding happens. Instead of choosing one and adjusting later if needed, they waste valuable time in discussions. The solution to analysis paralysis is to set clear deadlines for decisions and follow the "good enough" principle. Instead of chasing perfection, engineers should pick a reasonable option and refine it along the way. Agile development, for instance, focuses on building a minimum viable product (MVP) and improving it step by step rather than perfecting everything upfront. In short, analysis paralysis slows down innovation. While careful planning is necessary, taking action is just as important. Engineers must find the right balance: analyzing enough to make informed decisions but not so much that progress stops. --- ## Go Clean Code Guidelines **Date:** 2024-09-22 **URL:** https://lorbic.com/go-clean-code-guidelines/ **Description:** When it comes to writing clean, maintainable code, there are a few fundamental rules that can help improve the overall structure and quality of your codebase. As engineers, our goal should be to keep things simple, clear, and scalable. With this in mind, here are some guidelines which prioritize code readability, functional clarity, and the overall maintainability of a project. These guidelines are based on principles from clean code, SOLID, and functional programming while emphasizing simplicity over unnecessary complexity. ### Go Clean Code Guidelines for Code Quality and Maintainability. When it comes to writing clean, maintainable code, there are a few fundamental rules that can help improve the overall structure and quality of your codebase. As engineers, our goal should be to keep things simple, clear, and scalable. With this in mind, here are some guidelines which prioritize code readability, functional clarity, and the overall maintainability of a project. These guidelines are based on principles from clean code, SOLID, and functional programming while emphasizing simplicity over unnecessary complexity. --- ### Rule 1: Function Naming Clarity Your function names must clearly indicate what they do. Avoid vague names or overly complex abstractions. Each function should serve a single purpose, and its name should reflect this responsibility. #### Example: Instead of a vague name like `ProcessData`: ```go // Bad func ProcessData() { // Complex logic for processing some data } ``` Use something more descriptive like `ConvertOrderToInvoice`: ```go // Good func ConvertOrderToInvoice(order Order) Invoice { // Convert an Order object into an Invoice } ``` Here, the function name precisely conveys what the function does, making it easier for developers to follow the code. ### Rule 2: No Magic Numbers or Strings Avoid using magic numbers or strings directly in your code. Instead, define constants with meaningful names to improve readability and reduce errors. #### Example: ```go // Bad if user.Role == "admin" { // Grant access } ``` Replace magic strings with constants: ```go // Good const AdminRole = "admin" if user.Role == AdminRole { // Grant access } ``` This makes the code more readable and easier to maintain. If the role name changes in the future, you only need to update it in one place. ### Rule 3: Use Clear and Consistent Variable Names Variable names should be descriptive enough to indicate their purpose. Do not abbreviate in a way that makes it hard for others to understand the meaning of the variable. #### Example: ```go // Bad v := 10 // What is 'v'? ``` Instead, use meaningful names: ```go // Good maxAllowedItems := 10 ``` With clear variable names, other developers can quickly understand what the variable represents. ### Rule 4: Keep Functions Small Each function should perform one task and do it well. If a function is doing multiple things, break it down into smaller functions. This improves code readability and makes testing easier. #### Example: ```go // Bad func HandleOrder(order Order) Invoice { // Generating invoice, many lines of code // Save invoice to database, many lines of code // Send notification to customer, many lines of code return invoice } ``` The function is doing too many things. Instead, split it into smaller functions: ```go // Good func HandleOrder(order Order) Invoice { invoice := generateInvoice(order) persistInvoice(invoice) notifyCustomer(invoice) return invoice } func persistInvoice(invoice Invoice) { // Save invoice to the database } func notifyCustomer(invoice Invoice) { // Send a notification to the customer } ``` This makes each function's responsibility clearer, and it is easier to test and maintain. ### Rule 5: Error Handling Should Be Explicit Always check for errors and handle them explicitly. Do not assume the absence of an error. The code should either handle the error gracefully or propagate it back to the caller. #### Example: ```go // Bad result, err := getUserDetails(userID) process(result) // No error handling! ``` Instead, always handle or propagate errors: ```go // Good result, err := getUserDetails(userID) if err != nil { logError(err) return nil, err } process(result) ``` Proper error handling ensures that bugs and edge cases are caught early and makes the code more reliable. ### Rule 6: Use Structs for Complex Data If a function accepts multiple parameters of related data, bundle them into a struct. This makes the function signature cleaner and more flexible. #### Example: ```go // Bad func CreateUser(name string, age int, address string, email string) User { // Create a new user } ``` Instead, use a struct for the user's details: ```go // Good type UserDetails struct { Name string Age int Address string Email string } func CreateUser(details UserDetails) User { // Create a new user } ``` This approach makes the code easier to read and modify in the future, especially if more fields are added to the user details. ### Rule 7: Use Consistent Struct Field Order Maintain consistency in the order of struct fields. Group similar fields together to enhance readability. #### Example: ```go // Bad type User struct { Address string Age int Email string Name string } ``` Instead, group fields logically: ```go // Good type User struct { Name string Age int Address string Email string } ``` This way, when you or others revisit the code, it's easier to understand the structure of the object. ### Rule 8: Inline Commenting for Complex Logic Where necessary, provide inline comments explaining the purpose of complex logic or algorithms. Avoid obvious comments, but make sure to clarify non-obvious code. #### Example: ```go // Bad // Add two numbers total := a + b ``` Instead, explain complex logic: ```go // Good // Apply discount if the customer is a premium user and their purchase exceeds $100 if customer.IsPremium && order.Total > 100 { applyDiscount(order) } ``` The inline comment explains the logic behind applying the discount, which may not be immediately apparent from the code itself. ### Rule 9: Group Related Code into Logical Blocks Separate blocks of code that serve different purposes with blank lines. This improves readability and helps group related logic together. #### Example: ```go // Bad func ProcessOrder(order Order) { validateOrder(order) applyDiscount(order) finalizePayment(order) sendReceipt(order) } ``` Instead, separate the code into logical sections: ```go // Good func ProcessOrder(order Order) { validateOrder(order) applyDiscount(order) finalizePayment(order) sendReceipt(order) } ``` This makes the code visually easier to follow and highlights the different stages of processing an order. ### Rule 10: Function Arguments vs. Return Types Make sure function names clearly reflect their return types and arguments. This prevents confusion and makes the function's purpose obvious to the caller. #### Example: ```go // Bad func FetchData(input string) bool { // Returning a boolean from a 'Fetch' function is misleading } ``` Instead, update the name to reflect the return type: ```go // Good func IsDataAvailable(input string) bool { // Now it's clear that the function returns a boolean } ``` #### This helps maintain consistency in your codebase and makes the function's role immediately clear. By following these guidelines, you can ensure that your codebase remains clean, efficient, and easier to navigate for you and your team. Keep function names clear, avoid magic numbers, handle errors explicitly, and group your code logically for the best results. Happy coding! --- ## SOLID and Functional Programming Principles in Go **Date:** 2024-09-22 **URL:** https://lorbic.com/solid-and-functional-programming-principles-in-go/ **Description:** SOLID and functional programming principles explained and implemented in Go Let's go through the **SOLID principles** and **functional programming principles** that can apply to your Go codebase. I'll provide simple Go examples along with brief explanations to show how each principle can improve code quality. --- ### **SOLID Principles** 1. **S**ingle Responsibility Principle (SRP) - **Description:** A class or function should have only one reason to change, meaning it should only have one job or responsibility. **Example:** ```go // Good: Separate responsibilities into two functions type EmailSender struct{} func (e *EmailSender) SendEmail(to, subject, body string) { // logic for sending email } func formatEmailBody(template string, data map[string]string) string { // logic for formatting the email body return "" } // Avoid: Single function doing multiple jobs func sendFormattedEmail(to, subject, template string, data map[string]string) { body := formatEmailBody(template, data) // email sending logic mixed with formatting } ``` --- 2. **O**pen/Closed Principle (OCP) - **Description:** Software entities (classes, modules, functions) should be open for extension but closed for modification. **Example:** ```go // Good: Extend functionality via interfaces type PaymentProcessor interface { ProcessPayment(amount float64) error } type PayPalProcessor struct{} func (p PayPalProcessor) ProcessPayment(amount float64) error { // PayPal payment processing logic return nil } type CreditCardProcessor struct{} func (c CreditCardProcessor) ProcessPayment(amount float64) error { // Credit card processing logic return nil } // Avoid: Modifying existing functions to add new payment methods func processPayment(method string, amount float64) error { if method == "paypal" { // PayPal logic } else if method == "creditcard" { // Credit card logic } return nil } ``` --- 3. **L**iskov Substitution Principle (LSP) - **Description:** Objects should be replaceable by their subtypes without altering the correctness of the program. **Example:** ```go // Good: Subtypes maintain expected behavior type Animal interface { Speak() string } type Dog struct{} func (d Dog) Speak() string { return "Woof!" } type Cat struct{} func (c Cat) Speak() string { return "Meow!" } func makeAnimalSpeak(a Animal) { fmt.Println(a.Speak()) } // Avoid: Subtypes behaving unpredictably type SilentDog struct{} func (s SilentDog) Speak() string { return "" // Violates expectations of the Speak method } ``` --- 4. **I**nterface Segregation Principle (ISP) - **Description:** No client should be forced to depend on methods it doesn't use. Create small, specific interfaces instead of one large interface. **Example:** ```go // Good: Segregated interfaces type Reader interface { Read() string } type Writer interface { Write(data string) error } // Avoid: Fat interface forcing implementations to use unused methods type ReadWriteCloser interface { Read() string Write(data string) error Close() error } // If a class only needs reading, it should not be forced to implement Write and Close. ``` --- 5. **D**ependency Inversion Principle (DIP) - **Description:** High-level modules should not depend on low-level modules. Both should depend on abstractions (interfaces). **Example:** ```go // Good: High-level module depends on abstraction type NotificationService struct { sender EmailSender } func NewNotificationService(sender EmailSender) *NotificationService { return &NotificationService{sender: sender} } func (n *NotificationService) Notify(message string) { n.sender.SendEmail("user@example.com", "Notification", message) } // Avoid: High-level module directly depending on a low-level module func sendNotification() { sender := &EmailSender{} // tight coupling with EmailSender sender.SendEmail("user@example.com", "Notification", "Hello!") } ``` --- ### **Functional Programming Principles** 1. **First-Class and Higher-Order Functions** - **Description:** Functions are treated as first-class citizens and can be passed around as arguments, returned from other functions, or assigned to variables. **Example:** ```go // Higher-order function that accepts another function as a parameter func applyDiscount(price float64, discountFunc func(float64) float64) float64 { return discountFunc(price) } func tenPercentDiscount(price float64) float64 { return price * 0.9 } func main() { price := 100.0 discountedPrice := applyDiscount(price, tenPercentDiscount) fmt.Println(discountedPrice) // 90.0 } ``` --- 2. **Pure Functions** - **Description:** A pure function's output is determined only by its input values, without observable side effects. **Example:** ```go // Good: Pure function (no side effects) func add(a, b int) int { return a + b } // Avoid: Impure function (with side effects) var counter int func addWithSideEffect(a, b int) int { counter++ // changes external state return a + b } ``` --- 3. **Immutability** - **Description:** Data should not be modified after it is created. Instead, create a new copy of the data when changes are needed. **Example:** ```go // Good: Immutability by creating a new slice func addToSlice(slice []int, value int) []int { newSlice := append(slice, value) return newSlice } // Avoid: Modifying the original slice func modifySlice(slice []int, value int) { slice = append(slice, value) } ``` --- 4. **Recursion Over Loops** - **Description:** Functional programming often favors recursion over iterative loops, especially when solving problems like list processing. **Example:** ```go // Good: Recursion func factorial(n int) int { if n == 0 { return 1 } return n * factorial(n-1) } // Avoid: Iterative approach (loops) func iterativeFactorial(n int) int { result := 1 for i := n; i > 0; i-- { result *= i } return result } ``` --- 5. **Function Composition** - **Description:** Combine simple functions to build more complex ones, allowing for clean and reusable logic. **Example:** ```go func double(x int) int { return x * 2 } func square(x int) int { return x * x } func compose(f, g func(int) int) func(int) int { return func(x int) int { return f(g(x)) // Apply g, then f } } func main() { doubleSquare := compose(double, square) fmt.Println(doubleSquare(3)) // Output: 18 (3^2 * 2) } ``` --- These principles can help keep your code clean, modular, and testable while using both object-oriented and functional paradigms. --- ## 10 Essential Tips for Beginners Starting in IT **Date:** 2024-02-23 **URL:** https://lorbic.com/essential-tips-for-beginners-starting-in-it/ **Description:** In the ever-evolving world of IT, prioritize mastering core concepts over chasing trends. Embrace hands-on learning through projects, stay curious, seek guidance when needed, and remember that problem-solving and persistence are key to success in this dynamic field. ![A girl sitting in a room studying and thinking. Generated with AI by Vikash from Lorbic.com](/images/uploads/10-essential-tips-for-beginners-starting-in-it.jpg) Are you new to IT? Feeling a bit overwhelmed? Don't worry, I've got you covered! In this article, I'll share 10 essential tips to help you get started on your journey in Information Technology. From mastering the basics to gaining hands-on experience and staying curious, these tips will set you on the right path. Let's dive in and start the IT adventure! 1. **Focus on Fundamentals:** Instead of trying to understand every buzzword out there or the latest tech trend, focus on building a strong foundation in core concepts such as networking, operating systems, databases, and programming languages. 2. **Hands-on Practice:** Don't just rely on theory. Get your hands dirty by working on projects, setting up environments, and troubleshooting issues. Practical experience is invaluable in IT. I can tell you from personal experience, it doesn't matter what you know or which course you have completed, only thing that matters is if you can convert that Jira ticket into some sweet .py files 😉. 3. **Learn by Doing:** Regularly work on projects and real-world scenarios. Whether it's setting up a home lab, create open-source projects (Invite your internet friends to add features). Practical experience is the most effective way to understand complex stuff, eg. like how that payment is processed in the eCommerce website and how you can do it on your own. Create imaginary scenarios like there is some XYZ that sells car parts and you are responsible for all tech things, now how will you design, build, host a website where you will sell those parts. Some projects can be simple 1 hour coding, some will take many months to complete, once you are addicted you can't go back. **YOU NEED THAT ADDICTION** in this distracted, shorts-oriented world. 4. **Stay Curious:** The field of IT is rapidly changing (new JS and CSS frameworks being born every week), so it's essential to stay curious and keep learning. Be open to exploring new technologies, tools, and methodologies to expand your knowledge and skills continuously. Like the AI things going on right now. But don't distract from your path. 5. **Need Help? ask for it:** Don't hesitate to reach out to more experienced professionals for guidance or mentorship. Not everyone you ask will help you, but if you ask 10 people at least 1 may help you. As we say in Hindi *"*जहाँ अहंकार होता है, वहाँ ज्ञान नहीं होता", which translates to "Where there is pride, there is no wisdom" in English. 6. **Stay Updated but Don't Chase Every Trend:** Stay informed about industry trends and advancements, but don't feel pressured to chase every new technology or buzzword. Instead, focus on understanding the underlying principles and how they apply to your work. You can't master everything. Focus on the things you care and are useful. 7. **Practice Problem-Solving:** Our industry always revolves around solving complex problems and troubleshooting issues. Strengthen your problem-solving skills by tackling challenges, breaking down problems into smaller steps, and systematically finding solutions. I'm not talking about mindlessly doing DSA questions. There are other problems that need solutions. Such as, how will you modernize and deploy a decade old monolith into cloud which is so goddamn fragile, you touch it, it breaks! 8. **Document and read docs:** Keep track of your learning journey by documenting your progress, experiences, and lessons learned. Reflecting on your achievements and challenges can help remember your understanding and identify areas for improvement. For this build your own blog. 9. **Stay Patient and Persistent:** Learning IT is challenging, and you will encounter setbacks along the way. Stay patient, be persistent, and don't get discouraged by obstacles. Every challenge is an opportunity to learn and grow. Back in college, when I was learning Django and working on my first project there was an issue with my implementation which took over a week to find it and fix it. 10. **Enjoy the Journey:** Remember to enjoy the learning process and celebrate your successes, no matter how small. IT is a fascinating and rewarding field, and embracing the journey of continuous learning can lead to fulfilling and meaningful career opportunities. Thinking something? Write it down in the comments. Have fun. Happy learning. --- ## NGINX: Building from Source and Installation on Linux (Updated) **Date:** 2024-02-02 **URL:** https://lorbic.com/build-nginx-from-source/ **Description:** In this blog we are going to build NGINX from source code. And we will configure NGINX's settings, paths, and add or remove modules. [NGINX](https://en.wikipedia.org/wiki/Nginx) is a web server that can also be used as a reverse proxy, load balancer, mail proxy and HTTP cache. NGINX is a free and open source software licensed under _2-clause BSD_.\ You can easily download and use NGINX in Windows, macOS, Linux and other operating systems [link](https://nginx.org/en/download.html) read NGINX installation instructions from the [official docs](https://nginx.org/en/docs/install.html) and install on your operating system for free. But, in this blog we are going to build NGINX from source code. And we will configure NGINX's settings, paths, and add or remove modules. So let's get started… ## Preparation We'll use Ubuntu 20.04 LTS as our operating system, though these instructions are applicable to other Linux-based systems as well. ### 1. Download the source code: Start by visiting the NGINX download page and choose either the stable release or the mainline version for experimental features. For example, you can download nginx-1.21.4. Link: [NGINX download page](https://nginx.org/en/download.html). \ Note: You can use `wget` or `curl` to download. ```shell wget https://nginx.org/download/nginx-1.21.4.tar.gz ``` After downloading, extract the tar file using: ```shell tar -zxvf nginx-1.21.4.tar.gz ``` Now cd into `nginx-` directory. ### 2. Install Required Tools: Ensure you have the necessary build tools. On Ubuntu, use `apt` package manager: ```shell sudo apt install build-essential ``` Additionally, install required dependencies and libraries: ```shell sudo apt install libpcre3 libpcre3-dev zlib1g zlib1g-dev libssl-dev ``` ### 3. Configure NGINX for build Check all configuration options: `sudo ./configure --help` Refer to the NGINX documentation's module reference section for details on modules. Configure installation paths and modules as needed.\ Follow the link: [nginx docs > module reference section](https://nginx.org/en/docs/) Configure Installation: ```shell sudo ./configure \ --sbin-path=/usr/bin/nginx \ --conf-path=/etc/nginx/nginx.conf \ --error-log-path=/var/log/nginx/error.log \ --http-log-path=/var/log/nginx/access.log \ --pid-path=/var/run/nginx.pid \ --with-pcre \ --with-http_ssl_module \ --with-http_v2_module \ --with-http_gunzip_module ``` > –sbin-path=/usr/bin/nginx\ > –conf-path=/etc/nginx/nginx.conf\ > –error-log-path=/var/log/nginx/error.log\ > –http-log-path=/var/log/nginx/access.log\ > –pid-path=/var/run/nginx.pid\ > –with-pcre --with-http_ssl_module\ > –with-http_v2_module\ > –with-http_gunzip_module ### 4. Compile and Install the Application: Compile NGINX by running: ```shell make # Run below command to install NGINX make install ``` ### 5. Restart NGINX and Configure as a Service: Reload NGINX to apply changes: `nginx -s reload` **If you encounter an error related to the nginx.pid file missing, follow these steps:** > nginx: \[error] open() "/var/run/nginx.pid" failed (2: No such file or directory) **5.1. Add Nginx as service**: Add NGINX as a service by creating a systemd config file. Create or modify `/lib/systemd/system/nginx.service` with appropriate paths:\ Visit: [nginx systemd config](https://www.nginx.com/resources/wiki/start/topics/examples/systemd/) and create the file.\ _`--/lib/systemd/system/nginx.service--`_\ Change _pid_ and _sbin_ path as set in step 3. ```/lib/systemd/system/nginx.service [Unit] Description=The NGINX HTTP and reverse proxy server After=syslog.target network-online.target remote-fs.target nss-lookup.target Wants=network-online.target [Service] Type=forking PIDFile=/var/run/nginx.pid ExecStartPre=/usr/bin/nginx -t ExecStart=/usr/bin/nginx ExecReload=/usr/bin/nginx -s reload ExecStop=/bin/kill -s QUIT $MAINPID PrivateTmp=true [Install] WantedBy=multi-user.target ``` **5.2. Start NGINX using systemd:** ```sh sudo systemctl start nginx ``` **5.3. Enable NGINX to start on boot:** ```sh sudo systemctl enable nginx ``` That's all you need to do to compile and run NGINX on Ubuntu.\ If you get any error or suggestion please leave a comment. Happy NGINX-ing! 🤟 --- ## How to Install Docker on Another Drive in Windows (D: Drive Guide) **Date:** 2024-01-27 **URL:** https://lorbic.com/docker-custom-installation-location-windows/ **Description:** Is your C: drive full? Learn how to install Docker Desktop in a custom location (like a D: drive) on Windows 10 or 11 using CLI flags for WSL2 data and container roots. While installing Docker on Windows typically forces it onto your **C: drive**, many developers run out of space as images and containers grow. This guide shows you how to bypass the GUI and use CLI flags to install Docker Desktop in a custom location (e.g., `D:\Docker`) while also moving the heavy WSL2 data roots. **TL;DR:** To save your `C:` drive from Docker's growing appetite, bypass the GUI installer. Open PowerShell as Admin and run: `start /w "" "Docker Desktop Installer.exe" install --accept-license --installation-dir="D:\Docker" --wsl-default-data-root="D:\Docker\wsl"` --- ### Requirements for Custom Installation To ensure a smooth migration to another drive, verify you meet these criteria: - **OS:** Windows 10 or 11 (with Hyper-V and WSL 2 enabled). - **Installer:** The latest Docker Desktop `.exe` [downloaded here](https://docs.docker.com/desktop/install/windows-install/). - **Permissions:** Administrator access. - **Storage:** A secondary drive (D:, E:, etc.) with at least 64GB of free space (SSD recommended for performance). ### Preparation 1. **Create Folders:** Create your target directory. For example: `D:\Containers`. 2. **Cleanup:** If you have an existing Docker installation, back up your volumes and uninstall it first. 3. **Stop Services:** Open PowerShell as admin and run `sc.exe stop docker` to ensure no locks are active. ### Recommended Directory Structure For a clean setup, use a structure like this on your secondary drive: ```text D:\CONTAINERS ├───docker (App files) └───wsl (Linux container data) └───windows-data (Windows Container Data) ``` ### Step-by-Step Installation via CLI The GUI installer doesn't offer a "Browse" button for the path. You must run it from the command line (PowerShell or CMD) with specific flags. **The Complete Command:** ```powershell start /w "" "Docker Desktop Installer.exe" install --accept-license \ --installation-dir="D:\Containers\docker" \ --wsl-default-data-root="D:\Containers\wsl" \ --windows-containers-default-data-root="D:\Containers\windows-data" ``` #### Flag Explanations: - `--installation-dir`: Moves the actual Docker application binaries (usually `C:\Program Files\Docker\Docker`). - `--wsl-default-data-root`: **The most important flag.** This moves the massive `ext4.vhdx` files where your Linux containers actually live. - `--windows-containers-default-data-root`: Moves the data root for Windows-native containers. _Note: Replace `D:\Containers` with your actual drive letter._ ### Post-Installation & Verification 1. **Restart your computer** to ensure all registry changes and WSL paths are initialized. 2. **Verify paths:** Open PowerShell and run: ```powershell docker info | select-string "Docker Root Dir" ``` 3. **Check WSL:** Verify the data is being stored in your new location by checking the folder size of your `wsl` directory after pulling a few large images. ### Why move Docker to another drive? - **Performance:** Offloading I/O to a dedicated SSD can prevent OS lag during heavy builds. - **Stability:** Prevents "Disk Full" errors that can crash Windows when Docker pulls large image layers. - **Organization:** Keeps your system drive lean and focused strictly on the OS. --- **Stop fighting your tools.** If you found this useful, join the **Dispatch**—a weekly deep dive into the mechanics of high-performance systems, Go internals, and devops architecture. {{< newsletter >}} --- ## Build a URL redirector application with Cloudflare Workers and Cloudflare D1 **Date:** 2024-01-07 **URL:** https://lorbic.com/build-a-url-redirector-application-with-cloudflare-workers-and-cloudflare-d1/ **Description:** Discover the power of Cloudflare Workers and D1 database as we guide you through building a robust URL redirector application on LorbiC.com. Shorten and customize your links effortlessly, optimizing user experience and website performance. Learn step-by-step with prerequisites, database setup, and code implementation. Unlock the potential of Cloudflare's edge computing and revolutionize URL handling for seamless redirections. Elevate your web application's performance with this comprehensive guide at Lorbic.com. ![Lorbic.com - Cloudflare Workers and D1 database used to build a simple url redirector app.](/images/uploads/lorbic.com-cloudflare-workers-d1-featured-image-2x.jpg "Clopudflare Workers and D1 database - lorbic.com") In today's fast-paced digital world, where attention spans are short, optimizing website performance and user experience is crucial. Imagine asking someone to visit a lengthy URL () – their reaction might be a disappointed face (🙁). Typing such URLs can lead to errors, resulting in users landing on a dreaded 404 page. To address this, we'll build a simple and powerful solution – a URL redirector application using Cloudflare Workers and D1 database. This application will not only shorten your links but also provide a seamless and error-free redirection feature.\ \ For the sake of simplicity, we'll keep the set of features to a minimum and add whatever necessary at some later time. ### Main features: * Create a short code for a URL. * Redirect users to the original link. ### Understanding Cloudflare Workers: Cloudflare Workers allow developers to run code at the edge of the Cloudflare network, enabling the execution of custom logic for HTTP requests. This allows for enhanced performance and reduced latency, as the code runs closer to the end-users. That's what Workers are, basically intercepting requests before they reach your origin server. This allows for lightning-fast redirects based on URL paths, query parameters, headers, and even cookies. You can put workers between **USER** and **ORIGIN SERVER.** Then handle the request accordingly. #### **Here's how Workers handle redirects:** * **Interception:** The Worker intercepts incoming requests based on your chosen route configuration. * **Rule Matching:** The Worker analyzes the request against your predefined rules, stored in JavaScript code, or you can connect Cloudflare KV, Cloudflare D1 or other database via Cloudflare Hyperdrive and get the configuration. * **Redirection Magic:** If a rule matches, the Worker constructs a redirect response, specifying the new target URL and status code (e.g., 301 for permanent move, 302 for temporary). * **Speedy Delivery:** The redirect response is instantly delivered back to the user, bypassing the origin server for optimal performance. - - - ### Prerequisites: Before diving into the implementation, ensure you have the following: * Node JS * A Cloudflare account. * A domain configured with Cloudflare. * Cloudflare Wrangler CLI (check [here](https://developers.cloudflare.com/workers/wrangler/install-and-update/)) * Familiarity with JavaScript, Node.js, and ExpressJS (we'll use HonoJS, similar to ExpressJS). * Install Hono js. ### Create Cloudflare D1 Database Login to Cloudflare Wrangler CLI by running the following command and allow Cloudflare permissions. ```javascript wrangler login ``` ##### Create a D1 database Run the command below to create a database. Change with your actual database name. ```shell wrangler d1 create ``` This returns following response, with details of your database. \ Paste these into your `wrangler.toml` ```toml [[d1_databases]] binding = "DB" # i.e. available in your Worker on env.DB database_name = "" database_id = "5283redacted23-4a04-456b-af3f-redacted" ``` ##### Create database schema We will keep it very simple with just 1 table and 3 columns in the table. Here is the command to create the database table. **Note:** you'll have to paste the database configuration in **wrangler.toml** and have to be inside the project directory. ```shell wrangler d1 execute DB \ --command "CREATE TABLE IF NOT EXISTS urls (short_code TEXT PRIMARY KEY, created_at INTEGER, url TEXT);" ``` **Columns:** * short_code: A string column to store the short code associated with the URL. * url: A string column to store the full URL. * created_at: An integer column to store the Unix timestamp representing when the URL was created. Check your database by running the following command: ```shell wrangler d1 execute DB \ --command "select * from urls" ``` ### Create Cloudflare Worker Go to [this page](https://developers.cloudflare.com/workers/get-started/guide/) and create the Cloudflare worker. Or simple install Wrangler CLI and run following command to initialize a project. Change **``** with actual name. (I've chosen **lrbc).** ```shell npx wrangler init ``` Select, "hello world" example in the list and chose other options accordingly. Now go to the **lrbc** directory (your project directory). ```shell cd lrbc ``` Now open the `wrangler.toml` file in the root of your directory. Add the following code to connect your Cloudflare D1 database. If you do not have it set-up check the **Create Cloudflare D1 Database** of this blog. Replace the name of your database and database id with actual values. The binding is how you will refer to this configuration in your worker code. ```toml [[d1_databases]] binding = "DB" database_name = "" database_id = "" ``` ##### Install Honojs: Install hono js by running following code `npm install --save hono`. ### The Code: Next, open the `src/index.js` file and add the following code. It uses Hono JS. Hono is a tiny and super fast framework for creating web application and it is very easy to integrate with Cloudflare Workers. It's feels native. ```javascript import { Hono } from 'hono'; const app = new Hono(); app.get('/:code', async (c) => { try { const code = c.req.param('code'); console.log(code); const { results } = await c.env.DB.prepare('SELECT url FROM urls WHERE short_code = ?').bind(code).all(); if (results.length > 0) { const websiteUrl = results[0].url; console.log('Redirecting to ' + websiteUrl); return c.redirect(websiteUrl); } else { console.log('Short code not found'); return c.json({ status: 404, error: 'Short code not found' }); } } catch (e) { console.log('Error fetching URL:', e); return c.json({ status: 500, error: 'Failed to fetch URL' }); } }); export default app; ``` The provided code is written in JavaScript and uses the Hono library. Let's break down its key components: 1. ##### **Route Handling:** The code defines a route using the 'app.get' method, which responds to incoming HTTP GET requests with a specific parameter in the URL. ```javascript app.get('/:code', async (c) => { // Code logic goes here }); ``` 2. ##### **Database Query:** Inside the route handler, the code attempts to fetch the full URL associated with the provided short code from a database. Notice, the **DB** in the code **c.env.DB** . This is the binding we've defined in **wrangler.toml** file. Here **c** is the context of Hono. Everything related request and response is stored in this object. ```javascript const code = c.req.param('code'); const { results } = await c.env.DB.prepare('SELECT url FROM urls WHERE short_code = ?').bind(code).all(); ``` 3. ##### URL Redirection or Error Handling: Based on the query results, the code decides whether to redirect the user to the corresponding website URL or return a JSON response with an error message. ```javascript if (results.length > 0) { // Redirect to the full website URL return c.redirect(websiteUrl); } else { // Short code not found, return a JSON response return c.json({ status: 404, error: 'Short code not found' }); } ``` 4. ##### Error Handling: The code includes basic error handling, logging any encountered errors and returning an appropriate JSON response. You can check these logs in the Workers dashboard. And by running `wrangler tail` ```javascript } catch (e) { console.log('Error fetching URL:', e); return c.json({ status: 500, error: 'Failed to fetch URL' }); } ``` ### Deploy your worker: You can deploy your worker by running just one command `wrangler deploy`. It will create a ***.workers.dev*** URL. ```shell wrangler deploy ``` ### Create Short URLs: You can either create the URLs by running a following code and directly inserting them into your D1 database. ```javascript app.post('/store', async (c) => { try { // get the long url from json payload in request body const { url } = await c.req.json(); // Generate a random id of length = NANOID_SIZE const shortCode = nanoid(); const created_at = Date.now(); console.log(shortCode); // Validation if (!shortCode || !url) { return c.json({status: 400, error: 'Missing short code or URL' }); } // Save the url and const { results } = await c.env.DB.prepare("INSERT INTO urls (short_code, url, created_at) VALUES (?, ?, ?)").bind(shortCode, url, created_at).run(); return c.json({ message: `URL stored successfully. short: ${shortCode}, url: ${url}` }); } catch (e) { console.error('Error storing URL:', e); return c.json({ status: 500, error: 'Failed to store URL. There is some error or the code already exists.' }); } }); ``` ### What's Next * Create a GitHub repository and store this project there. * Extend the functionality by adding the short URL creation feature. * Integrate this worker (project) into your existing website and create a share functionality, so when someone clicks the share button they get a short URL. * Extend the database schema and add a hash of URL, so creating existing URLs will return the short existing code instead of creating new code, reducing the number of duplicate URLs. If you have any question, write it down in the comments below. Your response really matters 😅. Share it with your friends. Have a nice Sunday. Cheers ✌ --- ## Setting Up Llama 2 with Docker on Your Gaming Laptop Using Ollama **Date:** 2023-12-02 **URL:** https://lorbic.com/setting-up-llama-2-with-docker-on-your-gaming-laptop-using-ollama/ **Description:** Effortlessly set up Llama 2 on your gaming laptop using Docker and Ollama, with a user-friendly interface and local data storage for privacy. This article is simple guide on setting up Llama 2, a powerful large language model, on your gaming laptop using Docker and the Ollama platform. Ollama makes it very easy to run many LLMs locally. In this detailed article, I'll cover each step, such as Ollama installation and ease of interaction through a Chat GPT-like UI. Before we go into the technical details, let's understand what a large language model is. ![Llama working on a laptop. ](/images/uploads/llama-computer.webp "Llama working on a laptop") ## What is a Large Language Model? Large language models, like Llama 2, Code Llama are advanced artificial intelligence algorithms designed to understand and generate human-like language. These models are trained on massive datasets (like thousands of books), learning the fundamentals of language structure, context, and syntax. Llama 2, in particular, is a versatile language model capable of various natural language processing tasks, making it a valuable tool for developers, researchers, and enthusiasts. And it is open source. So, we can use it for free. ## Prerequisites: Before diving into the installation process, let's ensure your system meets the necessary prerequisites: * **Minimum RAM:** 16 GB (8 GB is fine but may result in slower performance) * **Docker:** Install the latest version of Docker on your machine. If you don't have it, visit Docker's official website for detailed installation instructions. * **For Windows Users:** Install the latest version of Windows Subsystem for Linux (WSL 2). (I couldn't get the GPU to work, I'll learn more and keep this blog updated). * **For Linux Users:** Ensure you have an Nvidia GPU. Note that you may need the Nvidia Container Toolkit\[3] for GPU support. ## Installing Docker: If you haven't installed Docker yet, follow these steps: Download and Install Docker: 1. Visit [Docker's official website](https://www.docker.com/get-started). 2. Choose the right version for your operating system. 3. Follow the installation instructions provided. ## Setting Up Ollama with Docker: Now, let's set up the Ollama Docker container for Llama 2: #### 1. Install the Ollama Docker Container: ```shell docker run -d --gpus=all -v ${PWD}:/root/.ollama -p 11434:11434 --name ollama ollama/ollama ``` * This command downloads the Ollama Docker image and creates a container named "ollama", exposing Ollama's APIs on port 11434. * For Linux users, the option --gpus=all enables GPU support. #### 2. Run the Llama 2 Model: ```shell docker exec -it ollama ollama-run Llama2 ``` * Execute this command to run Llama 2 inside the Ollama Docker container. ## Installing Chat GPT-like UI: Enhance your user experience by installing the Ollama-webui container: ```shell docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway --name ollama-webui --restart always ghcr.io/ollama-webui/ollama-webui:main ``` * This command creates a container named "ollama-webui" and exports it to localhost:3000. ## Accessing Your Chat GPT: With everything set up, access your **Chat GPT** from your browser at http://localhost:3000. Enjoy the chatting with your AI in a user-friendly interface with all your data stored locally for maximum privacy. ![Llama 2 running locally on Ollama](/images/uploads/ollama.jpg "Llama 2 running locally on Ollama | Lorbic.com") ## Additional Tips and Considerations: Disclaimer: Keep in mind that the response time of Llama 2 may vary depending on your machine's capabilities. You can also install other models. Find a model from [Ollama Library](https://ollama.ai/library) and run it inside the a container just like we did with Llama 2. ## Reference Links: * [Stackoverflow - Mount a drive](https://stackoverflow.com/a/41489151/11175853) * [Docker Official Website](https://docs.docker.com/engine/install/) * [Ollama Official Website](https://ollama.ai/) * [](https://ollama.ai/)[Ollama Models Library](https://ollama.ai/library) * [Nvidia Container Toolkit Installation Guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installation) * [Ollama Webui GitHub Repository](https://github.com/ollama-webui/ollama-webui) This complete-ish guide provides a step-by-step approach to setting up and running Llama 2 (and other LLMs) on your gaming laptop using Docker and the Ollama platform. If you encounter any issues or have suggestions, feel free to leave a comment. \ \ Have an awesome day! --- ## AWS - Host Static Website With S3 + Cloudfront + Route53 **Date:** 2023-05-07 **URL:** https://lorbic.com/aws-host-static-website-with-s3--cloudfront--route53/ **Description:** In this blog learn how to host a static website on AWS with S3, Cloudfront and Route53. Enable HTTP/2, HTTP/3 and reditect insecure http to secure https. Set the bucket to private. In this blog I'll discuss how to host a static website on AWS with S3, CloudFront and Route 53. We will secure our website with SSL certificate generated from Certificate Manger, enable HTTP/2 and HTTP/3 and redirect insecure http traffic to https version of the website. We will also make our bucket private and add policy to allow access from CloudFront. ![Picture of a Motor Buggy](/images/uploads/museums-victoria-zg0if6dcns0-unsplash_2.webp "Picture of a Motor Buggy") In a later blog I will also setup GitHub actions to automatically deploy the website when we create a new post. ### GOALS: 1. Create S3 bucket. 2. Create CloudFront distribution. 3. Point CloudFront to s3 bucket and restrict bucket access to CloudFront only. 4. Add your domain (optional). ***NOTE: Replace \`example.org\` with your actual domain name.*** # Create S3 bucket Open AWS console and got to S3 dashboard. Then create a new bucket with name as \`example.org\`.\ Go to **Properties** tab and enable **Bucket Versioning.** By enabling bucket versioning you will not lose previous version of your assets (static files).\ Go to **Permissions** tab and enable **Block public access (bucket settings)**, this will prevent access to the bucket from the public. Remember, goal #2 we will use CloudFront to deliver the assets. # Create CloudFront distribution Now we will create the CloudFront distribution and point it to our S3 bucket.\ In *CloudFront Dashboard* create a new CloudFront distribution.\ **Origin > Origin Domain,** select your S3 bucket.\ **Origin > Origin Access,** Select **origin access control settings** and select **Create Control Settings** and fill the following details. > *Name:* leave it default or update\ > *Description (Optional):* add a description\ > *Signing Behavior:* Sign requests\ > *Origin Type:* S3 Then click create and select this newly created settings from the **origin access control** drop-down.\ (We will need to update our bucket's policy to allow CloudFront access to the bucket. We will do this later.) In the **Web Application Firewall (WAF),** select `Do not enable security protections`\ At the very bottom, in the *Settings section* set **Default root object** to `index.html`. If we do not this we will see a sweet *access denied error.* Because we are allowing access to all the files inside our bucket **YOUR_BUCKET_NAME/** but not the bucket itself. So when we visit CloudFront distribution URL (e.g. `https://dfi9s7i5typ2c.cloudfront.net/`) CloudFront will not know what does `/` means. And when we set the default root object, CloudFront will know `/` means `/index.html` and it will load the index.html file. Leave everything else as default, scroll down to bottom and click **Create Distribution.** **Note: It will take 5-10 minutes to fully deploy the CloudFront distribution.** # Configure bucket access Open S3 Dashboard, go to **Permissions** tab. Select **Edit** on the *bucket policy* section and paste the following policy in the text block. This will allow CloudFront to access the objects while keeping the bucket private. Replace the **YOUR_BUCKET_NAME**, **YOUR_ACCOUNT_ID** and **YOUR_CLOUDFRONT_DISTRIBUTION_ID** placeholders with correct values and save the policy. ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowCloudFrontServicePrincipalReadOnly", "Effect": "Allow", "Principal": { "Service": "cloudfront.amazonaws.com" }, "Action": "s3:GetObject", "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*", "Condition": { "StringEquals": { "AWS:SourceArn": "arn:aws:cloudfront::YOUR_ACCOUNT_ID:distribution/YOUR_CLOUDFRONT_DISTRIBUTION_ID" } } } ] } ``` That's it, your website is live at your CloudFront distribution URL. You can find the URL on the home page of your distribution.\ If you are seeing some DNS error, then wait for some time.\ **Note: It will take 5-10 minutes to fully deploy the CloudFront distribution.** Now, we will add a custom domain in our distribution. # Custom Domain (Optional) ## Add Domain to Route 53 To complete this step you will need an existing domain or need to purchase a new one.\ Go to Route 53 dashboard and create a hosted zone (it will incur charges) and add your custom domain. Then go to your domain registrar and update the nameserver records, this will make Route 53 your DNS manager.\ It will take up to 24 hours to fully propagate the DNS. You can check the status on . ## Create SSL certificate Open **Certificate Manager**, request a new certificate. Add your domain name and update your DNS records to validate the ownership. Keep in mind to add both the root domain (example.org) and the wildcard domain (*.example.org). ## Add the domain to CloudFront To add the domain, go to CloudFront Dashboard and select **Edit** under **General > Settings.** In the *Settings* section, update `Alternate domain name (CNAME)` and add your domain name. In `Custom SSL certificate` select your newly created SSL certificate. \ To point this domain to CloudFront. Open to your domain settings (*in R*oute *53*), click on **Create Record** and create an `A` record. Toggle the **Alias** button and in the *Route Traffic To* select your CloudFront distribution.\ In the same section enable HTTP/2 and HTTP3. After some time (5-10 minutes) your website will be available at your domain address.\ \ Thanks for reading.\ If you have any questions, leave a comment below or drop an email at vikash\[at]lorbic\[dot]com.\ \ Have a nice day. --- ## Simplifying Search Activation: A Quick Guide with JavaScript **Date:** 2023-04-01 **URL:** https://lorbic.com/simplifying-search-activation-a-quick-guide-with-javascript/ **Description:** Learn how to enhance user experience by activating the search input field with a simple key press on your website. This quick guide provides a code snippet and easy-to-follow steps. Learn how to enhance user experience by activating the search input field with a simple key press on your website. This quick guide provides a code snippet and easy-to-follow steps. ### 1. HTML Markup for Search Input: ```html
    • ``` ### 2. JavaScript Functionality: Activate the search input field by pressing a key (in this example, the '/' key). Modify the key binding according to your preferences. ```javascript // app.js (app.min.js) function handleSearchActivation(event, searchInput, searchButton) { let KEY_TO_ACTIVATE_SEARCH = '/'; if (event.key === KEY_TO_ACTIVATE_SEARCH) { searchInput.focus(); searchInput.value = ""; searchButton.click(); console.log("Pressed '/' to activate search."); } } window.onload = function () { console.log("Document loaded..."); let searchInput = document.querySelector("#search-input"); let searchButton = document.querySelector("#search-button"); document.addEventListener("keyup", (event) => { handleSearchActivation(event, searchInput, searchButton); }); }; ``` ### 3. Link the Script File in Your HTML: Include the JavaScript file in the head or footer of your HTML. Alternatively, copy and paste the code into an existing JS file to reduce file requests. ```html ... ... ``` ### 4. Customize UI: You can also tailor the UI of the search input to match your preferences.\ \ Now, users can effortlessly activate the search input by pressing a key and start typing. You can also add a submit button and press 'Enter' to initiate the search.\ \ Thanks for Simplifying Your User's Experience! --- ## Setting Up a PostgreSQL Container: A Quick Guide **Date:** 2023-04-01 **URL:** https://lorbic.com/setting-up-a-postgresql-container-a-quick-guide/ **Description:** Learn the simple and fast process of creating a PostgreSQL container using Docker. This guide provides both the command and a detailed explanation of the container and PostgreSQL database configuration. Learn the simple and fast process of creating a PostgreSQL container using Docker. This guide provides both the command and a detailed explanation of the container and PostgreSQL database configuration. ## 1. Create the PostgreSQL Container: ```shell docker run -d --name postgres \ -e POSTGRES_PASSWORD=RunningFlying2 \ -e PGDATA=/var/lib/postgresql/data/pgdata \ -v postgres_data:/var/lib/postgresql/data \ -p 5432:5432 \ postgres:latest ``` ## **2. Explanation:** The above command initiates the creation of a container with the latest version of PostgreSQL. Here's a breakdown of the parameters used: `-d: Runs the container in detached mode.` `--name postgres: Assigns the name "postgres" to the container.` `-e POSTGRES_PASSWORD=RunningFlying2: Sets the PostgreSQL user password to "RunningFlying2".` `-e PGDATA=/var/lib/postgresql/data/pgdata: Specifies the PostgreSQL data directory.` `-v postgres_data:/var/lib/postgresql/data: Creates a Docker volume named "postgres_data" for persistent data storage.` `-p 5432:5432: Maps the container's 5432 port to the host machine's 5432 port for external access.` `postgres:latest: Pulls and uses the latest version of the PostgreSQL image.` ## 3. Technical Details: ``` Container Name: postgres Docker Volume: postgres_data Host: 127.0.0.1 Port: 5432 User: postgres Password: RunningFlying2 ``` You have successfully set up a PostgreSQL container, ready to handle your database needs. The provided technical details ensure you can seamlessly connect to this container from your development or production environment. **Explore More:** Feel free to explore additional PostgreSQL configurations and features to tailor the container to your specific requirements. > **Note:** For enhanced security, consider changing default passwords in production environments. Have an amazing day. --- ## Learn Dart Programming **Date:** 2021-05-12 **URL:** https://lorbic.com/learn-dart-programming/ **Description:** Learn basics of Dart programming language with code examples and explainations. Good Evening, Here are the Hands-on notes for Dart Programming Language. I created these notes when I was learning Dart Language. So, these notes might help you too. You can use these notes as a quick reference to Syntax and Demo. I've covered almost every fundamental concept of Dart in this article. ![](https://images.unsplash.com/photo-1576504164753-8fd338cd450b?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=727&q=80) Photo by Birmingham Museums Trust --- # 1. Basics ## 1.1 The Main Function Every Dart program starts execution from `main()` functions. ### The syntax of main function _We use this version when we want to use command line arguments._ ```dart void main(List args){ // do your stuff here } ``` Or simple version _Generally we use this version of main._ ```dart void main(){ // do your stuff here } ``` ### Demo ```dart void main() { // say hello world print("Hello World!"); // print some numbers, integer expression print(1 * 2 + 34 - 12 / 5); // print double print(2.53); // boolean print(true); } ``` --- ## 1.2 Data Types Literals in Dart. ```dart // literals in Dart "Dan"; // String literal 23; // int literal 45.6; // double literal true; // bool literal ``` There are 4 built-in data types in Dart. | SN | Data Type | Description | | --- | --- | --- | | 1. | int | Integer value | | 2. | double | Floating point value | | 3. | String | String value | | 4. | bool | `true` , `false` value | ### Syntax creating variables in Dart **Use explicit way to declare the variable.** `typename variable_name;` Or initialize while creating. `typename variable_name = initial_value;` ```dart int intVariable; int anotherIntVariable = 111; String name = "Vikas"; double pi = 3.14; bool isdeveloper = true; ``` **Use implicit way to declare the variable.** Use `var` to create any type of variable. `var variable_name;` Or initialize while creating. `var variable_name = intial_value`;\` ### Demo **Note: _In Dart all the data types are objects and their initial value is `null`._** Create the variables ```dart // numbers: int int score = 90; // store hexadecimal value int hexValue = 0xEAEAEA; // numbers: double double length = 24.6; double exponentValue = 1.4e5; // String String name = "Dave"; // boolean bool going = false; // use var var name2 = "Vikas"; // String var age = 20; // int var score2 = 45.6; // double var isHere = true; // bool var exp = 2.5e6; // exponential (double) var hex2 = 0xEC43AB; // hex (int) var data; // will have null ``` Print the values ```dart print(score); //int print(hexValue); //int print(length); //double print(exponentValue);//double print(going); //bool print(name); //String print(name2); //String print(age); //int print(score2); //int print(isHere); //bool print(exp); //double print(hex2); //int print(data); //null value ``` --- ## 1.3 String and String Interpolation Ways to define a String variable ```dart // ways to define string String s1 = 'Vikas'; // with single quote String s2 = "Dave"; // with double quote String s3 = "It's Easy"; // can use both String s4 = 'It\'s Easy'; // excape sequence String s5 = 'This is a very very long string ' 'and exceeding the line.'; // long string // auto concatenation of strings ``` String Interpolation ```dart // Normal Concatenation String name = "Vikas"; String msg = name + ", We are heading to the west"; // Interpolation String msg2 = "$name We are heading to the west"; String msglen = "Length of msg = ${msg2.length}"; // Print print(msg2); print(msglen); ``` We can interpolate any data type with String in `print(...)` using `$varname`. ```dart int BoxLength = 123; double BoxWeight = 4522.4545; print("Length of box = $BoxLength \nWeight of Box = $BoxWeight"); ``` --- ## 1.4 Constant and Final Variables In dart we can create constants using -> final -> const 1. **final vs const:** - `final` variable can only be set once and it is initialized when accessed - `const` is internally final in nature but it is a `compile-time constant` -> i.e. it is initialized during compilation 2. Instance variable can be final but cannot be constant 3. If you want a Constant class level then make it `static const` . ### Demo ```dart void main() { // final final city = "Toronto"; // or // final String city = "Toronto"; // city = "NY"; // Error: Can't assign to the final variable 'city'. const PI = 3.1415; const double gravity = 9.8; } ``` Class level constant ```dart class Circle { final color = "RED"; // OK // const PI = 3.1415; // Not OK static const PI = 3.1415; // OK } ``` **Only static fields can be declared as `const`.** **Try declaring the field as `final`, or adding the keyword `static`.** --- # 2. Control Statements These statements are use to control the execution flow of the program and add various logics to the program. ## 2.1 If - Else Statements Simple condition checking. **Syntax** #### IF alone ```dart if(condition){ // block 1 // execute the code } ``` When the condition is `true` the code inside `block 1` gets executed. Otherwise nothing happens to the code inside `block 1`. #### IF and ELSE together ```dart if(condition){ // block 1 // execute the code } else{ // block 2 // execute the code } ``` #### IF-ELSE ladder ```dart if(condition){ // block 1 // execute the code } else if(condition2){ // block 2 // execute the code } else if(condition3){ // block 3 // execute the code } . . . else if(condition n){ // block n // execute the code } else{ // block last // execute the code } ``` When the condition is `true` the code inside the respective `block` gets executed. Otherwise `block last` is executed. ### Demo ```dart void main() { var salary = 250000; if (salary > 200000) { print("You, can get the Credit Card"); } else if (salary > 150000) { print("You can apply for the Credit Card"); } else { print("You cannot get the Credit Card"); } } ``` Check code inside `01 basics/05if_else.dart`. --- ## 2.2 Conditional Expression Conditional expression is a simple alternative to `if-else`. There are two conditional expressions 1. **`condition ? expr1 : expr2;`** returns `expr1` if condition is true else returns `expr2` ```dart int a = 12, b = 45; print("Small Number = ${(a < b) ? a : b}"); int c = (a < b) ? a : b; print("Small Number = $c); ``` 2. **`expr1 ?? expr2`** if `expr1` is not-null, returns its value; otherwise, evaluates and returns the value of `expr2` ```dart int x, y = 345; // x = null; print("Value = ${x ?? y}"); int z = x ?? y; print(z); ``` --- ## 2.3 Switch Case If we have constant conditions like ids as `1, 2, 3, 4, 5` Instead using many `if-else` statements we can use `switch-case` statement. The case can be a constant `int` or a constant `String`. #### Syntax ```dart switch(variable){ case 1: // code break; // this is must case 2: // another code break; default: // will execute if no case matches } ``` `break` is must to stop execution after every `case`. ### Demo With integer cases. ```dart // Integer (int) var option = 2; switch (option) { case 1: print("Option 1"); break; case 2: print("Option 2"); break; default: print("Invalid option"); } ``` With String cases. ```dart String name = "Andy"; switch (name) { case "Andy": print("Hello, Andy"); break; case "Dave": print("Hello, Dave"); break; default: print("No name"); } ``` --- # 3. Looping We can repeat the statements using looping. There are 3 loops in Dart : - For - While - Do ...While ## 3.1 For loop 1. The loop first initializes the `iteration_var` 2. then checks `breaking_condition` 3. then executes the `Statements` and 4. increments or decrements the `iteration_var` and 5. 2,3,4, are repeated until `breaking_condition` remains `true`. ### Syntax ```dart for(iteration_var; breaking_condition; increment/decrement) { // Statements } ``` ### Demo ```dart // print even nums for (var i = 1; i <= 10; i++) { if (i % 2 == 0){ print(i); } } ``` ### 3.1.1 For ... in Loop #### Syntax `for(var varname in IterableList){....}` #### Demo ```dart // for ..in loop List names = ["Dave", "Andy", "Vikas", "Mark"]; for (var name in names) { print(name); } ``` This loop is very useful when using List and other data containers. ### 3.1.2 Variation in for we can omit the `initial`, `breaking` and `incr/decr` statements. ```dart for (; x < 20; x++) { print(x); } for(;;x++){ print(x); if(x==10){ break; } } for(var x = 0;;){ x++; print(x); if(x==10){ break; } } ``` --- ## 3.2 While Loop While loop is an `entry controlled` loop. The code inside `{...}` of while loop will execute only when the condition is `true`. ### Syntax `while(condition){...}` Here we declare a condition or breaking condition for the loop. See the demo ### Demo ```dart // while loop // print 0...9 var k = 0; // this is must while (k < 10) { print(k); k++; } ``` --- ## 3.3 Do ...While Loop Do ...while Loop is an `exit controlled` loop. Here the `Statement` will run for the first time regardless the `condition` is `true` or `false`. Then it will check the `condition` if it is `true` the the loop will continue execution otherwise it will jump out of the loop. ### Syntax `do{...} while(conditon);` Here the semicolon(`;`) is must. ### Demo ```dart // print 0...9 // do ..while loop int x = 0; do { print(x); x++; } while (x < 10); ``` ### While vs Do...While Both work in similar manner. But, `do...while` will execute at least once and `while` won't. --- ## 3.4 Break and Continue These are two loop control statements. Which are used to _forcefully control_ the flow of loop execution. ### Break `break` is used to stop the execution, and jump out of the loop. #### Demo ```dart // break after printing 5 for(int i = 0; i < 10; i++){ print(i); if(i==5){ break; } } // run and observe // break -> stop execution of loop for (int i = 0; i < 5; i++) { for (int j = 0; j < 5; j++) { print("$i $j"); if (i == 2 && j == 2) { break; // will break inner loop } } } ``` **We have already used `break` in `switch...case` program above.** ### Continue `continue` is used to skip current iteration. #### Demo ```dart // continue -> skip an iteraion in a loop for (var i = 0; i < 10; i++) { if (i == 5) { // skip when i==5 continue; } print(i); } ``` --- ## 3.5 Label Label is a new concept in Dart (we have seen it in `C` language with its buddy `goto`). This is not a magical spell from _Hogwarts_ that will do amazing magical things. It is just a way to make loop more controlled than before. We can name a block of code using label. Take a look at below code snippet. ### Demo ```dart print("i j"); outerLoop: for (int i = 0; i < 5; i++) { // ignore: unused_label innerloop: for (int j = 0; j < 5; j++) { print("$i $j"); if (i == 2 && j == 2) { break outerLoop; // will break outerloop } } } ``` Try doing this and observe the result of `i j`. ```dart print("i j"); outerLoop: for (int i = 0; i < 5; i++) { // ignore: unused_label innerloop: for (int j = 0; j < 5; j++) { print("$i $j"); if (i == 2 && j == 2) { break innerLoop; // will break innerloop } } } ``` --- # 4. Functions Functions are used to group the code statements that do something. e.g. You want to calculate the area of a rectangle then the formula will be `area = length * width`. You can do it where you want to use the `area`. _But, if you want to double the area and add 20 to the result then what ? _ _Will you write all the code again and again or find some convenient solution._ So, the solution is using functions. Your code will be ```dart var area = length * width; area = area * 2; area = area + 20; ``` Now you are able to use the final value of `area`. So, let's declare a simple function that says `Hello World!` to the user (after learning the simple function we will continue with our `area` calculation) ```dart void sayHello(){ print("Hello World!"); // That's common saying hello to the world // let's say 'Hello Dart, Welcome to Earth' print("Hello Dart, Welcome to Earth"); } ``` ## 4.1 Breakdown of function (Syntax) **`void`** : It is the return type as you know Dart is a statically typed language like Java, C++ and C. This tells the compiler that we will return a value of the given type `(int, double, String, bool, other user defined types)`. Here `void` tells that the function will not return any value. **`sayHello`** : It is the name of the function. It can be anything but, should be relevant to the work of the function like: `multiplyBy2(), makeHalf(), doubleThevalue(), reduceThree(), calculateSI() etc.` **`()`** : It is called parameter. we can pass values that we want to work on. Like a `name` that should be printed instead of `Dart` or `World` in the above example. Let's assume we want to say Hello to you and we don't know your name then we will pass a `String` called `name` and interpolate it with the word `Hello` and print it. Look at the code below.. ```dart void sayHello(String name){ print("Hello, $name"); } void main(){ sayHello("Vikas"); // Hello, Vikas sayHello("Dart"); // Hello, Dart sayHello("World"); // Hello, World } ``` Copy the above code and run it. You can also return the string. ```dart String sayHello(String name){ return "Hello, $name"; // don't forget to change the return type } void main(){ print(sayHello("Vikas")); // Hello, Vikas print(sayHello("Dart")); // Hello, Dart print(sayHello("World")); // Hello, World } ``` Now, let's work on our `area` example: ```dart int smartAreaOfRectange(int length, int width) { int area = length * width; area = area * 2; area = area + 20; return area; } ``` Put this code in your file and call it in the main function as we've done in `sayHello(...)` function. Another Example: ```dart int doubleIt(int x) { return x * 2 + 17; } void main() { for (int i = 1; i <= 10; i++) { print(doubleIt(i)); } } ``` --- ## 4.2 Function Expression (Fat Arrow `=>`) The function expression or _Fat Arrow `=>`_ is used to make the function definition shorter and simpler. **It can only be used when there is only one statement in the function.** Like , double the value ```dart int doubleIt(int x) => x*2; ``` we don't use the `return` keyword here. So , ```dart int doubleIt(int x) => return x*2; ``` will be wrong. --- ## 4.3 Types of function parameters PARAMETERS: - Required - Optional: - Positional - Named - Default: - Positional - Named ### 4.3.1 Required Parameters These are the required parameters that cannot be omitted. ```dart // required parameters void requiredParameters(String c1, String c2, String c3) { print("Required Parameters"); print("$c1, $c2, $c3"); } void main() { // required parameters requiredParameters("NY", "Toronto", "San Diego"); } ``` ### 4.3.2 Optional Parameters We can omit passing these parameters. Blank value is considered as `null`. #### 4.3.2.1 Optional Positional Parameters We pass the value based on the portion of the parameter in the function. These parameters are surrounded by brackets `[ ... ]`. ```dart // optional positional parameters void optionalPositionalParameters(String c1, String c2, [String c3]) { print("Optional Positional Parameters"); print("$c1, $c2, $c3"); } void main() { // optional positional parameters optionalPositionalParameters("NY", "Toronto"); // 2 values optionalPositionalParameters("NY", "Toronto", "San Diego"); // 3 values } ``` #### 4.3.2.2 Optional Named Parameters we can give the name of parameter while passing the value. These parameters are surrounded by braces `{ ... }`. ```dart // optional named parameters void optionalNamedParameters(String c1, String c2, {String c3}) { print("Optional Named Parameters"); print("$c1, $c2, $c3"); } void main() { // optional named parameters optionalNamedParameters("NY", "Toronto"); optionalNamedParameters("NY", "Toronto", c3: "San Diego"); } ``` #### 4.3.2.3 Default Parameters These are the `positional and named` parameters with default value. **Default Positional** We pass the default value of the Positional Parameter while defining the function `[String name = "Dart", int id = 1111]`. ```dart // default positional parameters void defaultPositionalParameters(String c1, String c2, [String c3 = "Mumbai"]) { print("deafault positional Parameters"); print("$c1, $c2, $c3"); } void main() { // default positional defaultPositionalParameters("NY", "Toronto"); defaultPositionalParameters("NY", "Toronto", "San Diego"); } ``` **Default Named** We pass the default value of the Named Parameter while defining the function `{String name = "Dart", int id = 1111}`. ```dart // default named parameters void defaultNamedParameters(String c1, String c2, {String c3 = "Mumbai"}) { print("default Named Parameters"); print("$c1, $c2, $c3"); } void main() { // default named defaultNamedParameters("NY", "Toronto"); defaultNamedParameters("NY", "Toronto", c3: "San Diego"); } ``` --- # 5. Exception Handling _What are exceptions ?_ An **Exception** is a runtime error which occurs when the program is running due to some error that the compiler was unable to detect. If an exception occurs the program crashes. If not handled may cause serious issues. Below table showing some exceptions in Dart
    | S.N. | Exception | Description | | --- | --- | --- | | 1. | DeferredLoadException | Thrown when a deferred library fails to load. | | 2. | FormatException | Exception thrown when a string or some other data does not have an expected format and cannot be parsed or processed. | | 3. | IntegerDivisionByZeroException | Thrown when a number is divided by zero. | | 4. | IOException | Base class for all Input-Output related exceptions. | | 5. | IsolateSpawnException | Thrown when an isolate cannot be created. | | 6. | Timeout | Thrown when a scheduled timeout happens while waiting for an async result. |
    ## 5.1 Example of Exception Simplest example will be dividing a number by _Zero_. ```dart void main(){ // integer division int result = 12 ~/ 4; // OK int badResult = 15 ~/ 0; // NOT OK } ``` So, the line `double badResult = 15 / 0;` raises an `IntegerDivisionByZeroException` exception. ## 5.2 Handling of Exception We can handle the exceptions by surrounding the exception occurring code inside a `try` block. And handle the exception using `on`, `catch`, and `finally` blocks. ### 5.2.1 Try Block This is the place where we write an exception occurring statement. #### Demo ```dart try { int answer = 12 ~/ 0; print("Division = $answer"); } ``` If the statement inside `try` block raises an exception the program will not crash this time instead it will throw the exception out of the `try` block which we'll have to catch. ### 5.2.2 On Block We can use `on` to catch the exception _if we know the thrown exception._ ```dart // CASE1: Handle with "on" // use on when we know the exception try { int answer = 12 ~/ 0; print("Division = $answer"); } on IntegerDivisionByZeroException { print("Cannot divide by zero"); } ``` Here we know the thrown exception `IntegerDivisionByZeroException`. ### 5.2.3 Catch Block If we don't know the exception we can simply catch it using `catch` block. ```dart // CASE2: Handle with "catch" // use catch(e) when we don't know the exception try { int answer = 12 ~/ 0; print("Division = $answer"); } catch (e) { print("The exception : $e"); } ``` Here the unknown exception is successfully handled. ### 5.2.4 Stack Trace We can use the Stack Trace to find the events that threw the exception. ```dart // CASE3: using STACK TRACE to know the events occured // before exception was thrown try { int answer = 12 ~/ 0; print("Division = $answer"); } catch (e, stackTraceObject) { print("The exception : $e"); print("Stack Trace:\n$stackTraceObject"); } ``` ### 5.2.5 Finally Block This Block is always executed regardless the occurance of the exception. **No exception:** ```dart // CASE4: finally clause try { int answer = 12 ~/ 4; print("Division = $answer"); } catch (e, stackTraceObject) { print("The exception : $e"); print("Stack Trace:\n$stackTraceObject"); } finally { print("This will always execute No exception)"); } ``` **With Exception:** ```dart // CASE4: finally clause try { int answer = 12 ~/ 0; print("Division = $answer"); } catch (e, stackTraceObject) { print("The exception : $e"); print("Stack Trace:\n$stackTraceObject"); } finally { print("This will always execute (With exception)"); } ``` ## 5.3 Custom Exception **Q. Can we have our iwn exceptions ?** **A. Yes, we can define our own exception class using the following method.** First create a class with `YourCustomExceptionName` and _implement_\* the built-in `Exception` class. Then _override_ the `errorMessage()` method to display "Your custom error message". \*_We will learn about Object Oriented Programming later in the article._ #### Demo ```dart // custom exception class class DepositAmountLessThanZeroException implements Exception { String errorMessage() { return "Ammount cannot be less than 0"; } } ``` Let's use the custom exception class. Create a function which throws the exception. ```dart void depositMoney(int amount) { if (amount < 0) { throw new DepositAmountLessThanZeroException(); } else { print("Successful, Deposited $amount"); } } ``` Now call the function in the try block. ```dart void main() { // OK try { depositMoney(1500); } catch (e, s) { print(e.errorMessage()); print(s); } // OK try { depositMoney(0); } catch (e, s) { print(e.errorMessage()); print(s); } // Exception try { depositMoney(-1233); } catch (e, s) { print(e.errorMessage()); print(s); } } ``` That's all from my side on basic exception handling. We will discuss the topic in detail later in another blog. Thanks for reading. 🙏 If you liked it feel free to share with your dear ones.💗 And tell me what do you think in the comment box.💬 Stay tuned. --- ## Types of Pointers in C/C++ **Date:** 2021-05-12 **URL:** https://lorbic.com/types-of-pointers-in-c/c/ **Description:** ![Photo of Hyacinth macaw on Unsplash - Village Programmer](https://images.unsplash.com/photo-1509116453669-0baa67ef9073?ixid=MnwxMjA3fDB8MHxzZWFyY2h8Mnx8aHlhY2ludGglMjBtYWNhd3xlbnwwfHwwfHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=500&q=60) Photo of _Hyacinth macaw by_ Roi Dimor on [Unsplash](https://unsplash.com/photos/AEQ-166CWY8) --- # Types of Pointers in C / C++ ## 1. Null Pointer It is a pointer pointing to nothing. **NULL** pointer points to the base address of the segment. `-EXAMPLE-` ```c int * ptr = (int) * 0; float * fptr = (float) * 0; double * dptr = (double) * 0; char * chptr = (char) * 0; ``` ##### Other ways of initializing NULL pointer ```c int * ptr = NULL; char * chptr = '\0'; ``` > #### NULL also means 0 in macro > > ```c > #define NULL 0 > ``` --- ## 2. Dangling Pointer A pointer pointing to the memory address of any variable (or object) which has been deleted from memory. When a pointer points to a deleted memory address, the pointer is called as a _dangling pointer_. `-EXAMPLE-` ```cpp Person p = new Person(); Person * pptr = &p; delete p; /* here pointer pptr (* pptr) is still pointing to the Person object which has been deleted. */ ``` --- ## 3. Generic Pointer _void_ pointer is known as generic pointer. It can point to **_any type_** of data. `-EXAMPLE-` ```cpp void * ptr; ``` #### Points to remember - We can't de-reference a generic pointer. - We can find the size of pointer using `sizeof()` operator. - These pointers can hold any type of pointer such as char, int, float, structure, array, object etc without any typecasting. - Any type of pointer can hold generic pointer without typecasting. - Generic pointer can be used to implement dynamic datatype. --- ## 4. Wild Pointer A pointer which has not been initialized is called as _wild pointer_. `-EXAMPLE-` ```cpp int * ptr; ``` --- ## 5. Complex Pointers These are pointers pointing to derived data-types. - Pointer to array - Pointer to function - Pointer to structure / union - Pointer to object - Multilevel pointers --- ## 6. near / far / huge Pointers These pointer modifiers were used in 1980's-90's before 32 bit architecture. **_near_** : a 16 bit pointer that can address any byte in a 64k segment. **_far_** : a 32 bit pointer that contains a segment and an offset. **_huge_** : a 32 bit pointer in which the segment is "normalized" so that no two far pointers point to the same address unless they have the same value. You can read this stack overflow answer for more information. --- Thanks for reading. 🙏 If you liked it feel free to share with your dear ones.💗 And tell me what do you think in the comment box.💬 Stay tuned. --- --- ## Avoid Memory Leaks In C++ **Date:** 2021-05-04 **URL:** https://lorbic.com/avoid-memory-leaks-in-c/ **Description:** **Instructions** Things You'll Need - Proficiency in C++ - C++ compiler - Debugger and other investigative software tools # Part 1 Understand the operator basics. The C++ operator `new` allocates heap memory. The `delete` operator frees heap memory. For every `new`, you should use a `delete` so that you free the same memory you allocated: ```cpp char* str = new char [30]; // Allocate 30 bytes to house a string. delete [] str; // Clear those 30 bytes and make str point nowhere. ``` # Part 2 Reallocate memory only if you've deleted. In the code below, `str` acquires a new address with the second allocation. The first address is lost irretrievably, and so are the 30 bytes that it pointed to. Now they're impossible to free, and you have a memory leak: ```cpp char* str = new char [30]; // Give str a memory address. // delete [] str; // Remove the first comment marking in this line to correct. str = new char [60]; /* Give str another memory address with the first one gone forever.*/ delete [] str; // This deletes the 60 bytes, not just the first 30. ``` # Part 3 Watch those pointer assignments. Every dynamic variable (allocated memory on the heap) needs to be associated with a pointer. When a dynamic variable becomes disassociated from its pointer(s), it becomes impossible to erase. Again, this results in a memory leak: ```cpp char* str1 = new char [30]; char* str2 = new char [40]; strcpy(str1, "Memory leak"); str2 = str1; // Bad! Now the 40 bytes are impossible to free. delete [] str2; // This deletes the 30 bytes. delete [] str1; // Possible access violation. What a disaster! ``` # Part 4 Be careful with local pointers. A pointer you declare in a function is allocated on the stack, but the dynamic variable it points to is allocated on the heap. If you don't delete it, it will persist after the program exits from the function: ```cpp void Leak(int x){ char* p = new char [x]; // delete [] p; // Remove the first comment marking to correct. } ``` # Part 5 Pay attention to the square braces after "delete". Use `delete` by itself to free a single object. Use `delete []` with square brackets to free a heap array. Don't do something like this: ```cpp char* one = new char; delete [] one; // Wrong char* many = new char [30]; delete many; // Wrong! ``` --- _**Keep Learining, Keep Practicing. :) :)**_ --- That's All. 😅 Thanks for reading. 🙏 If you liked it feel free to share with your dear ones.💗 And tell me what do you think in the comment box.💬 Stay tuned with CS111. --- ## Operators in Bash Shell Programming **Date:** 2021-05-04 **URL:** https://lorbic.com/operators-in-bash-shell-programming/ **Description:** ![A Picture of Pluto by NASA featured on Village Programmer](https://images.unsplash.com/photo-1614314107768-6018061b5b72?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=1334&q=80) Photo by [NASA](https://unsplash.com/photos/-5V6VZxSQRo) on Unsplash --- If you want to learn bash bash scripting please read this article. [Learn complete Linux bash (shell) scripting in one article](/post/learn-complete-linux-bash-scripting-in-one-article/) Operators help us perform various types of operations such as addition, multiplication etc. There are following types of operators present in bash: - Arithmetic, - Relational, - Boolean, - File Test, - String Test ## 1. Arithmetic Operators All the arithmetic operators present in bash are discussed below with examples. These operators work with integers. #### 1.1 Addition Operator It's a **`+`** that adds operands together. Shown in the below example: ```bash $x=20 $y=30 a=`expr $x + $y` echo $a ``` This will display: > 50 ##### NOTE: > 1. The **`expr `** command evaluates the expression. > 2. There is `` before and after the **`+`** operator . #### 1.2 Subtraction Operator It's a **`-`** that subtracts second operand from first operand. Shown in the below example: ```bash $x=40 $y=30 a=`expr $x - $y` echo $a ``` This will display: > 10 #### 1.3 Multiplication Operator It's a **`*`** that multplies operands together. Shown in the below example: ```bash $x=2 $y=3 a=`expr $x * $y` echo $a ``` This will display: > 6 #### 1.4 Division Operator It's a **`/`** that performs division operation. Shown in the below example: -Example 1- ```bash $x=200 $y=50 a=`expr $x / $y` echo $a ``` This will display: > 4 -Example 2- ```bash $x=20 $y=3 a=`expr $x / $y` echo $a ``` This will display: > 6 #### 1.5 Modulus Operator It's a **`%`** that performs modulus operation and returns the remainder. Shown in the below example: ```bash $x=10 $y=3 a=`expr $x % $y` echo $a ``` This will display: > 1 #### 1.6 Equality Check Operator It's a **`==`** that performs equality check operation and returns 1 if both sides are equal otherwise returns 0. Shown in the below example: -Example 1- ```bash $x=10 $y=10 a=`expr $x == $y` echo $a ``` This will display: > 1 -Example 2- ```bash $x=10 $y=5 a=`expr $x == $y` echo $a ``` This will display: > 0 #### 1.7 Not Equal Check Operator It's a **`!=`** that performs equality check operation and returns 0 if both sides are equal otherwise returns 1. Shown in the below example: -Example 1- ```bash $x=10 $y=10 a=`expr $x == $y` echo $a ``` This will display: > 0 -Example 2- ```bash $x=10 $y=5 a=`expr $x == $y` echo $a ``` This will display: > 1 ## 2. Relational Operators Relational operators are used to perform comparison among given operands. ### -eq (Equals to operator) This checks whether the operands are equal. This operator is used to perform comparison in the `if` condition. ```bash a=10 b=10 if [ $a -eq $b ] then echo "Equal" else echo "Not Equal" fi ``` > Equal ### -ne (Not equals to operator) This checks whether the operands are equal. This operator is used to perform comparison in the `if` condition. ```bash a=10 b=20 if [ $a -ne $b ] then echo "Not Equal" else echo "Equal" fi ``` > Not Equal ### -gt (Greater than operator) This checks whether the first operand is greater than the second operand. This operator is used to perform comparison in the `if` condition. ```bash a=40 b=20 if [ $a -gt $b ] then echo "$a is greater than $b" else echo "$a is not greater than $b" fi ``` > 40 is greater than 20 ### -lt (Less than operator) This checks whether the first operand is less than the second operand. This operator is used to perform comparison in the `if` condition. ```bash a=20 b=40 if [ $a -lt $b ] then echo "$a is less than $b" else echo "$a is not less than $b" fi ``` > 20 is greater than 40 ### -ge (Greater than or equals to operator) This checks whether the first operand is greater than or equals to the second operand. This operator is used to perform comparison in the `if` condition. ```bash a=40 b=20 if [ $a -ge $b ] then echo "$a is greater than or equals to $b" else echo "$a is not greater than $b" fi ``` > 40 is greater than or equals to 20 ### -le (Less than or equals to operator) This checks whether the first operand is greater than or equals to the second operand. This operator is used to perform comparison in the `if` condition. ```bash a=40 b=200 if [ $a -le $b ] then echo "$a is less than or equals to $b" else echo "$a is not less than $b" fi ``` > 40 is less than or equals to 200 ## 3. Boolean Operators ### ! (Not operator) This changes `true` into `false` and vice versa. We have used this operator in _Not equal operator_ under arithmetic operators. Shown in the below example: -Example 1- ```bash $x=10 $y=10 a=`expr $x == $y` echo $a ``` This will display: > 0 -Example 2- ```bash $x=10 $y=5 a=`expr $x == $y` echo $a ``` This will display: > 1 ### -o (OR operator) Checks whether one of the operands is true and executes the statements. ```bash a=40 b=20 c=10 if [ $a -gt $b -o $c -gt $b] then echo "$a or $c is greater than $b" else echo "$a is not less than $b" fi ``` > 40 or 10 is greater than 20 ### -a (AND operator) Checks whether all of the operands are true and executes the statements. ```bash a=40 b=20 c=100 if [ $a -gt $b -a $c -gt $b] then echo "$a and $c are greater than $b" else echo "$a is not less than $b" fi ``` > 40 and 100 are greater than 20 ## 4. String Test Operators These operators are used to perform string related comparisons. ### = (Equality Operator) Check whether two strings are equal. ```bash user="VIKAS" defaultUser="VIKAS" if [ $user = $defaultUser ] then echo "User is allowed" else echo "User is not allowed" fi ``` > User is allowed ### != (Not equal operator) Check whether two strings are not equal. ```bash user="VIKAS" defaultUser="DAVE" if [ $user != $defaultUser ] then echo "User is not allowed" else echo "User is allowed" fi ``` > User is not allowed ### -z (Zero length string) Checks whether the string is zero length. ```bash s="" if [ -z $s ] then echo "String is zero length" else echo "String is non zero length" fi ``` > String is zero length ### -n (Non zero length string) Checks whether the string is empty. ```bash s="Village Programmer" if [ -n $s ] then echo "String is non zero length" else echo "String is zero length" fi ``` > String is non zero length ### Empty string check ```bash s="" if [ $s ] then echo "String is not empty" else echo "String is empty" fi ``` > String is not empty --- _**Keep Learining, Keep Practicing. :) :)**_ --- That's All. 😅 Thanks for reading. 🙏 If you liked it feel free to share with your dear ones.💗 And tell me what do you think in the comment box.💬 Stay tuned with CS111. This post was originally posted on [Village Programmer](https://villageprogrammer.blogspot.com) --- ## Learn Complete Linux Bash Scripting in One Article **Date:** 2021-05-04 **URL:** https://lorbic.com/learn-complete-linux-bash-scripting-in-one-article/ **Description:** ![](https://images.pexels.com/photos/745266/pexels-photo-745266.jpeg?auto=compress&cs=tinysrgb&dpr=2&h=650&w=940) -- Photo by **[Iván Rivero](https://www.pexels.com/@osho?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels)** from **[Pexels](https://www.pexels.com/photo/tractor-with-trailer-under-cloudy-skies-during-day-745266/?utm_content=attributionCopyText&utm_medium=referral&utm_source=pexels)** In this article I'll be discussing about linux bash scripting (shell scripting) and I will cover every fundamental concept that you need to get started with bash scripting under linux environment. Bash scripting is a critical skill for every programmer. First of all you will need a **linux machine** to execute these scripts. You can use any of the below techniques to get a linux machine: - Install a linux distribution - Install WSL - Install Cygwin or MSYS - Installing _git for bash_ will provide a bash prompt but this prompt is not a complete linux environment I'll suggest you to install **WSL2** with **Ubuntu-2020.04** (_same is installed on my machine_). This one article will cover all the basics and some advanced concepts about bash (shell) scripting, so read it completely. ## Very Basics ### 1. Hello World Every programmer start by writing a program to print _'Hello World'_ on the console. Create a file named `hello.sh` and open it with your favourite text editor (notepad, sublime, atom, vscode, nano, vim, etc). Write the following code into the file and save. ```bash echo "Hello World :)" ``` Now run the script by using following command ```sh $ bash ./hello.sh ``` This will print > Hello World :) ### 2. Comments We can write comments by using `#` symbol. ```bash # This is a comment and will not be executed # The below command will print "Hello World" to the console echo "Hello World" ``` > Hello World ### 3. What is shebang (#! /bin/bash) The **shebang is used** if you run the script directly as an executable file (for example with the command ./script.sh ). In this case it tells the operating system which executable to run. It's not required and has no effect if you for example write `bash ./script.sh` or source the script. We write the below line at the top of every script `#!/bin/bash` or `#!/usr/bin/env bash` ##### Hello World with shebang ```bash #!/usr/bin/env bash # hello.sh echo "Hello World :)" ``` > Hello World :) ### 4. Variables Creating variables in bash script is as simple as ```bash name="Vikash" ``` **NOTE:** Remember there is no space around _=_ operator. ##### Hello World with variables -- `hello2.sh` -- ```bash #!/usr/bin/env bash msg="Hello World from a variable" echo $msg ``` `$ ./hello2.sh` > Hello World from a variable ### 5. Command line arguments We can pass arguments while executing the script and use them in the script. A simple example is below. -- `params.sh`-- ```bash #!/usr/bin/env bash name=$1 website=$2 echo "Hello I am $name" echo "My website is $website" ``` `$ ./params.sh "Vikash" "Villageprogrammer.blogspot.com"` > Hello I am Vikash > My website is Villageprogrammer | Parameter Name | Description | | -------------- | ------------------------ | | $0 | Script name and path | | $1 | First argument | | $2 - $9 | Second to ninth argument | | ${10} - ${...} | More than tenth argument | You can execute a standard linux command and store the result in the variable. -- `params2.sh`-- ```bash #!/usr/bin/env bash today=$(date) directory=$(pwd) echo "Today is : $today" echo "Directory : $directory" ``` `$ ./params2.sh` > Today is : Thu Feb 18 17:05:56 IST 202 > Directory : /mnt/d/Xplore-Training/UNIX/bash-scripting/scripting ## Controls ### 1. If statement The syntax of if statement is similar to other programming languages like _C, C++_. Syntax ```bash if [ condition ] then # write your code here fi ``` Example: --`if.sh `-- ```bash #!/usr/bin/env bash num=10 if [ $num -lt 20 ] then echo "$num is less than 20" fi ``` `$ if.sh` > 10 is less than 20 You can combine parameter and variable logic with if to write complex scripts. ### 2. if-else Syntax ```bash if [ condition ] then # write your code here else # write your code here fi ``` Example: --`if-else.sh `-- ```bash #!/usr/bin/env bash num=10 if [ $num -eq 50 ] then echo "$num is equal to 50" else echo "$num is not equal to 50" fi ``` `$ ./if-else.sh` > 10 is not equal to 50 ### 3. if-elif-else Syntax ```bash if [ condition ] then # write your code here elif [ condition ] then # write your code here fi ``` Example: --`if-elif-else.sh `-- ```bash #!/usr/bin/env bash num=10 if [ $num -eq 20 ] then echo "$num is equal to 20" elif [ $num -lt 20 ] then echo "$num is less than 20" else echo "$num is greater than 20" fi ``` `$ ./if-elif-else.sh` > 10 is less than 20 ## Looping Repetition of a code block can be done using while loop and for loop. ### 1. while loop Syntax ```bash while [ condition ] do # write your code here done ``` Example -- `while.sh`-- ```bash count=1 while [ $count -lt 5 ] do echo $count ((count++)) done ``` `$ ./while.sh` > 1 > 2 > 3 > 4 > 5 ### 2. for loop We can pass an array of elements from command line and use them in the script by creating an array **`array=$@`** Syntax: ```bash for num in $container do echo $num done ``` Example: --`for.sh`-- ```bash #!/usr/bin/env bash nums=$@ for num in $nums do echo $num done ``` `$ ./for.sh 1 2 3 ` > 1 > 2 > 3 ### 3. break and continue --`break-continue.sh`-- ```bash #!/usr/bin/env bash # use '@' for entering more than one inputs (simply an array) # Pass the values from command line arguments names=$@ # use of break for name in $names do if [ $name = "Andy" ] then echo "End found" break else echo "Hello, $name" fi done echo "for loop terminated" # use of continue for name in $names do if [ $name = "Andy" ] then echo "Skip found" continue else echo "Hello, $name" fi done echo "for loop terminated" exit 0 ``` `$ break-continue.sh "Vikas" "Andy" "Mark"` > Hello, Vikas > End Found > for \ loop terminated > Hello, Vikas > Skip found > Hello, Mark > for \ loop terminated ## Working with environment variables We can access any of the environment variable from the bash script. --`vars.sh`-- ```bash #!/usr/bin/env bash # environment variables echo "The path is : $PATH" echo "The terminal is : $TERM" echo "The editor is : $EDITOR" # -z checks for empty if [ -z $EDITOR ] then echo "The Editor variable is empty" fi exit 0 ``` `$ ./vars.sh` > The path is : \ > The terminal is : xterm-256color > The editor is : > The Editor variable is empty ##### Environment variables and their meaning HOME : user's home directory PATH : executable binary's path HOSTNAME : hostname of the machine SHELL : shell that is being used USER : user of this session TERM : type of command line terminal that is being used ##### write a program to print user's name, computer name, and home directory ```bash echo "$USER@$HOSTNAME in [$HOME]" ``` > vikas@DAVE in [/home/vikas] ## Functions in bash There are two ways to create functions in _bash_ : #### 1. Function declaration syntax with _function_ keyword -`function1.sh`- ```bash #!/usr/bin/env bash function greeting(){ echo "Hello" } ``` #### 2. Function declaration without keyword -`function2.sh`- ```bash #!/usr/bin/env bash greeting(){ echo "Hello" } ``` #### 3. Calling the function To call the function we just need to use the name of the function, we do not use parentheses as we use in other programming languages (e.g. C, C++, Java, Python, etc.). -`function1.sh`- ```bash #!/usr/bin/env bash # declare the function function greeting(){ echo "Hello" } # calling the function greeting ``` **`$ bash ./function1.sh`** > Hello #### 4. What about passing parameters ? It's a two step process 1. **Access the parameter value** To access the parameter use : _`$1`_: for first parameter, _`$2`_: for second parameter, ... , _`$n`_: for nth parameter. You can either create a local variable and save the value by _`local variable=$n`_ or directly access the value by _`$n`_ (n is the parameter number 1, 2, 3, etc). 2. **Pass the parameter value** To pass the value write the value after the function name while calling the function (seperate by space). `add 2 3` #### 5. Complete example -`function.sh`- ```bash #!/usr/bin/env bash function add(){ local num1=$1 local num2=$2 sum=`expr $num1 + $num2` echo "Sum=$sum" } # call the function add 2 3 # 5 add 4 5 # 9 add -10 30 # 20 # or use command line arguments (uncomment if used) # add $1 $2 ``` **`$ bash ./function.sh`** > 5 > 9 > 20 ## Working with prompt message You can write a message when asking the user for input. It can be done by using **`read`** command. -`user-prompt.sh`- ```bash #!/usr/bin/env bash read "Hey, What's your name ?" name echo "Hello, $name" ``` **`$ bash ./user-prompt.sh`** > Hey, What's your name ? > Vikash > Hello, Vikash If you want the input to be in the same line use **`-p`**.. -`user-prompt2.sh`- ```bash #!/usr/bin/env bash read -p "Hey, What's your name ?" name echo "Hello, $name" ``` **`$ bash ./user-prompt2.sh`** > Hey, What's your name ? Vikash > Hello, Vikash --- _Now you have enough understanding to get started with bash (shell) scripting. Go ahead an read a book or two on bash scripting and you will be good at it._ **_Keep Learining, Keep Practicing. :) :)_** --- That's All. 😅 Thanks for reading. 🙏 If you liked it feel free to share with your dear ones.💗 And tell me what do you think in the comment box.💬 Stay tuned with CS111. This post was originally posted on [Village Programmer](https://villageprogrammer.blogspot.com) --- ## Access Wsl Filesystem From Windows 10 File Explorer **Date:** 2021-05-04 **URL:** https://lorbic.com/access-wsl-filesystem-from-windows-10-file-explorer/ **Description:** ![](https://images.unsplash.com/photo-1442120108414-42e7ea50d0b5?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw=&ixlib=rb-1.2.1&auto=format&fit=crop&w=818&q=80) If you want to get Linux environment in your Windows system there are some solutions that provide the opportunity.\ For example MYSYS, CYGWIN and a Virtual Machine (If you want complete linux machine) or Multiple Boot.\ But some of them aren't much powerful and some are resources hungry like Virtual Machine.\ So, the solution is **WSL (Windows Subsystem for Linux)**.\ You can learn how to setup WSL from [here](https://docs.microsoft.com/en-us/windows/wsl/install-win10#manual-installation-steps). In this article I'll be sharing how to access WSL file system from Windows 10 file explorer.\ I am assuming you have completely setup WSL and is able to access it through command prompt/powershell. ### Step 1 Open Windows Explorer\ Then type `\\wsl$` in the path location as shown in the below picture and hut enter. ![](https://i.imgur.com/sHbW2MZ.png) ### Step 2 Now a new window will open having your WSL linux machine name.![](https://i.imgur.com/9czE5vN.png) ### Step 3 Now right click on the icon shown in the above image (Name may not be the same)![](https://i.imgur.com/qAexGU9.png) ### Step 4 Now Select `Map Network Drive` option.\ A new diaglog box will open, As shown in the below picture.![](https://i.imgur.com/PXs7nwn.png) Now choose a drive letter and click **Finish**. ### Step 5 You will have the Linux machine file system listed under Network Storage.![](https://i.imgur.com/ASl6mIy.png) ### Conclusion You can view, rename, delete, copy/paste the files as you do with normal windows files.\ But do not delete any of the files you don't know. If you do that it may break the WSL System. That's all from my side. Thanks for reading. 🙏\ If you liked it feel free to share with your dear ones.💗\ And tell me what do you think in the comment box.💬 Stay tuned with CS111. This post was originally posted on [Village Programmer](https://villageprogrammer.blogspot.com)