Linux crontab is one of the most elegant pieces of software ever written for single-host automation. It is simple, clear, and has kept Unix systems running reliably since 1975.

The problem starts when we take a single-host tool and deploy it across a multi-node cloud setup.

In Relay, a multi-tenant AI API gateway system design, background jobs power core operations: every top of the hour, a job rolls up raw API usage tokens into tenant billing metrics; every 15 minutes, another job scans for expired API keys and purges them from cache; every 30 seconds, a health checker pings upstream LLM provider endpoints.

When an application runs on a single server, crontab works perfectly. But when running a distributed gateway across multiple application nodes and availability zones, running uncoordinated local cron loops causes immediate failures:

1. Duplicate Execution: If three nodes run the same cron schedule, all three wake up at 00:00:00 UTC, pull the same billing job, and bill tenants three times for the exact same usage.
2. Thundering Herds: Dozens of nodes running SELECT * FROM jobs WHERE next_run_at <= NOW() at the exact same millisecond exhaust database connection pools and cause lock wait timeouts.

Why This Post Exists (The Central Idea)

When teams face this challenge, engineering advice usually splits into two extremes:

• The Overkill Path: Deploy heavy external distributed orchestrators (like Temporal, Airflow, or a ZooKeeper/Etcd cluster). This adds complex setup, new infrastructure dependencies, and steep learning curves for teams that simply need reliable scheduled tasks.
• The Naïve Path: Write custom Redis key polling or uncoordinated database loops that suffer from thundering herds, race conditions, and lost jobs when nodes crash.

The thesis of this post is that there is a pragmatic middle ground: you can build a zero-dependency, fault-tolerant, horizontally scalable distributed scheduler using two building blocks already in your stack: PostgreSQL (SKIP LOCKED + partial indexes) and Go (chan struct{} counting semaphores).

We will design Pulse, a distributed job scheduling engine built for Relay. This post breaks down Pulse step by step: defining what a job scheduler is, what “distributed” means in this context, how PostgreSQL SKIP LOCKED guarantees atomic locks, how hash ring sharding avoids central bottlenecks, and how to write production Go code that survives node crashes mid-execution.

What Is a Distributed Job Scheduler?

Before looking at database queries or Go code, we need to clearly define the problem space.

1. Task Queue vs. Job Scheduler

Engineers often mix up task queues and job schedulers. They are two different tools:

• Task Queue (Reactive / Push-Driven): Pushes work as fast as possible when triggered by an application event. An API endpoint receives a request, enqueues a message to RabbitMQ or SQS, and a worker consumes it immediately. The driver is an external event.
• Job Scheduler (Temporal / Time-Driven): Evaluates schedules over time. It maintains schedule rules (such as cron expressions like 0 * * * *), continuously monitors wall-clock time, computes target execution deadlines (next_run_at), and triggers work when current time reaches or exceeds next_run_at. The driver is time itself.

1Task Queue:    [User Request] ---> (Enqueue) ---> [Queue] ---> (Execute Now)
2Job Scheduler: [Cron Rule]     ---> (Clock Poll: current_time >= next_run_at) ---> [Dispatch] ---> (Execute)

2. What Does “Distributed” Mean for a Scheduler?

Running a single-node scheduler (like robfig/cron in a Go binary or a Linux crontab) is simple: one process owns the clock, the schedule list, and the worker threads in local memory.

A distributed job scheduler runs across N independent nodes (for example, 5 Go worker containers across 3 AWS availability zones). This creates three hard problems to solve:

1. Clock Drift Across Nodes: Physical server clocks are never perfectly in sync. Even with NTP (Network Time Protocol), clock skew between Node A and Node B typically ranges from 5ms to 50ms. If Node A’s clock is 20ms ahead of Node B, Node A will see current_time >= next_run_at first. The system must work correctly no matter which node’s clock triggers first.
2. Preventing Double Work Without a Single Master Node: If 10 scheduler nodes poll for runnable jobs, how do we guarantee that exactly one node claims job J for execution timestamp T, without routing every single request through a single master server?
3. Handling Node Crashes and Lost Networks: If Node A claims job J, sets its state to running, and immediately suffers an OOM crash, kernel panic, or network partition, the system must detect Node A’s death, release the lock, and re-schedule job J on Node B without causing double execution.

The Core Architecture of Pulse

To solve these problems, Pulse splits work across three main components:

1. Job Store: A relational database table storing schedule definitions, next execution timestamps, locks, and status flags. This serves as the single source of truth for time state.
2. Poller / Scheduler Loop: A cluster of stateless Go nodes that poll owned database shards, claim due jobs atomically, recalculate next_run_at, and dispatch payloads.
3. Execution Workers: Worker pools that run the actual business logic (calling webhooks, querying ClickHouse, purging caches) with strict context timeouts.

Database Schema: Designing for Concurrent State

A distributed scheduler needs database storage that stays safe even if every server crashes. If all Pulse nodes crash at the same time, no scheduled jobs can be lost.

Here is the schema we will use for Pulse in PostgreSQL:

 1CREATE TYPE job_status AS ENUM ('idle', 'running', 'failed', 'disabled');
 2
 3CREATE TABLE scheduled_jobs (
 4    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
 5    tenant_id       TEXT NOT NULL,
 6    name            TEXT NOT NULL,
 7    cron_expr       TEXT NOT NULL, -- e.g., '0 * * * *'
 8    payload         JSONB NOT NULL DEFAULT '{}'::jsonb,
 9    
10    -- Execution State
11    status          job_status NOT NULL DEFAULT 'idle',
12    shard_id        INT NOT NULL DEFAULT 0,
13    
14    -- Scheduling Timestamps
15    last_run_at     TIMESTAMPTZ,
16    next_run_at     TIMESTAMPTZ NOT NULL,
17    
18    -- Lease & Lock Control
19    locked_at       TIMESTAMPTZ,
20    locked_by       TEXT,
21    lease_timeout   INTERVAL NOT NULL DEFAULT INTERVAL '5 minutes',
22    
23    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
24    updated_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()
25);
26
27-- Critical Index for the Hot Polling Path
28CREATE INDEX idx_scheduled_jobs_poll 
29ON scheduled_jobs (shard_id, next_run_at) 
30WHERE status = 'idle';

Notice the partial index definition: WHERE status = 'idle'.

In a system with 500,000 scheduled jobs, only a tiny fraction (perhaps 50 to 500) will be in the idle status with next_run_at <= NOW() at any given second. By filtering the index specifically to idle rows, the index tree stays minimal in memory, keeping index lookups and fetch latency under 1ms.

High-Throughput Polling with SKIP LOCKED

The biggest mistake developers make with a database scheduler is simple row locking:

1-- DO NOT DO THIS IN PRODUCTION
2SELECT * FROM scheduled_jobs 
3WHERE next_run_at <= NOW() AND status = 'idle' 
4FOR UPDATE;

When multiple scheduler nodes run this query at once, Node 1 locks matching rows. Node 2 and Node 3 do not skip those rows; instead, they block and wait for Node 1 to finish its transaction. This causes lock contention, context switching, and query timeouts.

PostgreSQL 9.5 introduced FOR UPDATE SKIP LOCKED. When a node executes a query with SKIP LOCKED, PostgreSQL instantly skips any rows currently locked by another transaction and returns only available rows.

Here is how Pulse claims a batch of executable jobs atomically:

 1WITH runnable_jobs AS (
 2    SELECT id 
 3    FROM scheduled_jobs
 4    WHERE shard_id = ANY($1) 
 5      AND status = 'idle' 
 6      AND next_run_at <= NOW()
 7    ORDER BY next_run_at ASC
 8    LIMIT $2
 9    FOR UPDATE SKIP LOCKED
10)
11UPDATE scheduled_jobs j
12SET 
13    status = 'running',
14    locked_at = NOW(),
15    locked_by = $3,
16    updated_at = NOW()
17FROM runnable_jobs r
18WHERE j.id = r.id
19RETURNING j.id, j.tenant_id, j.name, j.cron_expr, j.payload, j.next_run_at, j.shard_id;

When PostgreSQL evaluates a standard SELECT ... FOR UPDATE query:

1. It scans the table or index for matching tuples.
2. For every matching row, it inspects the tuple header (xmin/xmax) and the lock manager memory structure to see if an active transaction holds an exclusive row-level lock.
3. Without SKIP LOCKED: If Row 1 is locked by Worker Node A, Worker Node B is placed into a lock wait queue. Node B blocks execution until Node A commits or rolls back its transaction. If 10 scheduler nodes poll simultaneously, 9 nodes freeze until the leader finishes.
4. With SKIP LOCKED: When Node B sees Row 1 is locked by Node A, it bypasses Row 1 immediately with zero wait time and moves to Row 2. If Row 2 is unlocked, Node B claims Row 2 and returns it.

If all candidate rows in a batch are currently locked by other workers, SKIP LOCKED returns an empty result set instantly (0ms latency) rather than blocking. The poller cleanly finishes its tick and sleeps until the next interval.

This single atomic query gives us three important guarantees:

• Zero Double-Execution: Two pollers will never receive the same job ID.
• Non-Blocking Operation: Pollers do not wait on each other. If Node 1 claims jobs 1 through 10, Node 2 instantly receives jobs 11 through 20.
• Minimal Transaction Window: The lock is held only for the microsecond duration of the batch UPDATE statement.

Shard Partitioning Across Scheduler Replica Nodes

While SKIP LOCKED stops lock waiting, having 20 nodes scan the full table over and over still wastes database CPU.

We can partition jobs across nodes using a Shard Hash Ring.

Every job is assigned a shard_id upon creation (hash(job_id) % total_shards).

When a Pulse node boots up, it registers itself with a coordinator (or claims a set of integer shard IDs via advisory locks in PostgreSQL). Node A polls shards [0, 1], while Node B polls shards [2, 3].

This design reduces database query overhead from O(N) pollers scanning everything to O(1) owned shard lookups per node.

Building the Scheduler Engine in Go

Let us translate these principles into a clean, complete Go implementation.

The Cron Parser & Next Run Resolver

Pulse relies on cron parsing to update a job’s next_run_at timestamp immediately after claiming it.

 1package scheduler
 2
 3import (
 4	"context"
 5	"database/sql"
 6	"encoding/json"
 7	"fmt"
 8	"log"
 9	"sync"
10	"time"
11
12	"github.com/robfig/cron/v3"
13)
14
15// Job represents a scheduled task stored in the database.
16type Job struct {
17	ID          string          `json:"id"`
18	TenantID    string          `json:"tenant_id"`
19	Name        string          `json:"name"`
20	CronExpr    string          `json:"cron_expr"`
21	Payload     json.RawMessage `json:"payload"`
22	NextRunAt   time.Time       `json:"next_run_at"`
23	ShardID     int             `json:"shard_id"`
24}
25
26// Scheduler Engine orchestrates polling and dispatching.
27type Engine struct {
28	db         *sql.DB
29	nodeID     string
30	cronParser cron.Parser
31	workerSem  chan struct{} // Semaphore for concurrency backpressure
32	wg         sync.WaitGroup
33	quit       chan struct{}
34}
35
36func NewEngine(db *sql.DB, nodeID string, maxConcurrentWorkers int) *Engine {
37	return &Engine{
38		db:         db,
39		nodeID:     nodeID,
40		cronParser: cron.NewParser(cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow),
41		workerSem:  make(chan struct{}, maxConcurrentWorkers),
42		quit:       make(chan struct{}),
43	}
44}

The Atomic Polling and Claim Loop

Here is how the engine polls owned shards and dispatches jobs to workers with backpressure:

 1// ClaimBatch fetches and locks pending jobs for owned shards using SKIP LOCKED.
 2func (e *Engine) ClaimBatch(ctx context.Context, shardIDs []int, batchSize int) ([]Job, error) {
 3	tx, err := e.db.BeginTx(ctx, nil)
 4	if err != nil {
 5		return nil, fmt.Errorf("begin tx: %w", err)
 6	}
 7	defer tx.Rollback()
 8
 9	query := `
10		WITH runnable AS (
11			SELECT id 
12			FROM scheduled_jobs
13			WHERE shard_id = ANY($1) 
14			  AND status = 'idle' 
15			  AND next_run_at <= NOW()
16			ORDER BY next_run_at ASC
17			LIMIT $2
18			FOR UPDATE SKIP LOCKED
19		)
20		UPDATE scheduled_jobs j
21		SET 
22			status = 'running',
23			locked_at = NOW(),
24			locked_by = $3,
25			updated_at = NOW()
26		FROM runnable r
27		WHERE j.id = r.id
28		RETURNING j.id, j.tenant_id, j.name, j.cron_expr, j.payload, j.next_run_at, j.shard_id;
29	`
30
31	rows, err := tx.QueryContext(ctx, query, shardIDs, batchSize, e.nodeID)
32	if err != nil {
33		return nil, fmt.Errorf("query claim batch: %w", err)
34	}
35	defer rows.Close()
36
37	var jobs []Job
38	for rows.Next() {
39		var j Job
40		if err := rows.Scan(&j.ID, &j.TenantID, &j.Name, &j.CronExpr, &j.Payload, &j.NextRunAt, &j.ShardID); err != nil {
41			return nil, fmt.Errorf("scan job: %w", err)
42		}
43		jobs = append(jobs, j)
44	}
45
46	if err := tx.Commit(); err != nil {
47		return nil, fmt.Errorf("commit tx: %w", err)
48	}
49
50	return jobs, nil
51}

Worker Dispatch with Backpressure

When a node claims 50 jobs, sending all 50 to memory simultaneously can crash the node if memory or CPU is tight. We enforce worker concurrency using a Go channel semaphore:

 1func (e *Engine) Start(ctx context.Context, shardIDs []int, pollInterval time.Duration) {
 2	ticker := time.NewTicker(pollInterval)
 3	defer ticker.Stop()
 4
 5	for {
 6		select {
 7		case <-ctx.Done():
 8			log.Println("[Pulse] Shutdown signal received. Waiting for active workers...")
 9			e.wg.Wait()
10			return
11		case <-ticker.C:
12			jobs, err := e.ClaimBatch(ctx, shardIDs, 25)
13			if err != nil {
14				log.Printf("[Pulse] Error claiming batch: %v", err)
15				continue
16			}
17
18			for _, job := range jobs {
19				// Acquire semaphore slot with context cancellation support (Backpressure)
20				select {
21				case e.workerSem <- struct{}{}:
22				case <-ctx.Done():
23					log.Println("[Pulse] Context cancelled while acquiring worker semaphore slot")
24					e.wg.Wait()
25					return
26				}
27
28				e.wg.Add(1)
29				go func(j Job) {
30					defer func() {
31						<-e.workerSem
32						e.wg.Done()
33					}()
34					e.executeJob(ctx, j)
35				}(job)
36			}
37		}
38	}
39}

In Go, chan struct{} with a fixed buffer capacity (e.g., make(chan struct{}, 10)) acts as a zero-allocation counting semaphore:

1. Acquiring a slot: select { case workerSem <- struct{}{}: default: ... } sends an empty struct into the channel buffer. If the buffer is full (10 active goroutines), this operation blocks the polling loop, preventing new goroutines from being spawned until a slot is freed.
2. Releasing a slot: Inside defer func() { <-workerSem }(), reading from the channel frees a slot in the buffer, allowing the main loop to proceed and process the next job.

This pattern enforces backpressure: the scheduler automatically slows down batch claiming if existing jobs take longer to process than expected, preventing OOM (Out Of Memory) crashes.

Updating Next Execution & Releasing Locks

Once execution succeeds, Pulse schedules the next run based on its cron expression and transitions status back to idle:

 1func (e *Engine) executeJob(parentCtx context.Context, j Job) {
 2	ctx, cancel := context.WithTimeout(parentCtx, 2*time.Minute)
 3	defer cancel()
 4
 5	// 1. Perform actual business work
 6	execErr := e.runHandler(ctx, j)
 7
 8	// 2. Calculate next schedule run timestamp relative to completion time
 9	sched, err := e.cronParser.Parse(j.CronExpr)
10	if err != nil {
11		log.Printf("[Pulse] Invalid cron expression for job %s: %v", j.ID, err)
12		e.markFailed(parentCtx, j.ID, err.Error())
13		return
14	}
15	nextRun := sched.Next(time.Now())
16
17	// 3. Update job record state atomically
18	if execErr != nil {
19		log.Printf("[Pulse] Job %s failed: %v", j.ID, execErr)
20		e.markFailed(parentCtx, j.ID, execErr.Error())
21		return
22	}
23
24	query := `
25		UPDATE scheduled_jobs
26		SET 
27			status = 'idle',
28			last_run_at = NOW(),
29			next_run_at = $1,
30			locked_at = NULL,
31			locked_by = NULL,
32			updated_at = NOW()
33		WHERE id = $2;
34	`
35	if _, err := e.db.ExecContext(parentCtx, query, nextRun, j.ID); err != nil {
36		log.Printf("[Pulse] Error completing job %s: %v", j.ID, err)
37	}
38}
39
40func (e *Engine) runHandler(ctx context.Context, j Job) error {
41	// Dispatch based on job payload name (e.g. usage-rollup, key-cleanup)
42	timer := time.NewTimer(150 * time.Millisecond)
43	defer timer.Stop()
44
45	select {
46	case <-ctx.Done():
47		return ctx.Err()
48	case <-timer.C:
49		return nil
50	}
51}
52
53func (e *Engine) markFailed(ctx context.Context, jobID string, reason string) {
54	query := `
55		UPDATE scheduled_jobs
56		SET 
57			status = 'failed',
58			locked_at = NULL,
59			locked_by = NULL,
60			updated_at = NOW()
61		WHERE id = $1;
62	`
63	if _, err := e.db.ExecContext(ctx, query, jobID); err != nil {
64		log.Printf("[Pulse] Error marking job %s as failed: %v", jobID, err)
65	}
66}

Surviving Edge Cases and Disasters

Systems break in unexpected ways. Here is how to handle the most common failures:

Case 1: The Scheduler Node Crashes Mid-Job

Suppose Node A claims a billing aggregation job, sets status = 'running', and starts execution. Mid-way through, Node A suffers an OOM (Out Of Memory) crash.

The job is left stuck in status = 'running' forever unless cleaned up.

Solution: Dead-Man Lease Sweeper We can run a lightweight background sweeper query every 60 seconds on the leader node:

1-- Recover jobs whose lease expired (node died mid-execution)
2UPDATE scheduled_jobs
3SET 
4    status = 'idle',
5    locked_at = NULL,
6    locked_by = NULL,
7    updated_at = NOW()
8WHERE status = 'running' 
9  AND locked_at < NOW() - lease_timeout;

If Node A dies, its held jobs are reset back to idle after lease_timeout (e.g., 5 minutes) and automatically picked up by surviving nodes.

Case 2: What Happens During System Downtime? (Missed Runs)

If the scheduling cluster goes down for 2 hours during a database upgrade, 120 minute-interval jobs will be past their next_run_at time when nodes boot back up.

Do you execute all 120 missed runs sequentially (Backfill), or do you skip straight to the current time?

In this system design, we configure this per job type:

• Billing Aggregations: Must BACKFILL. Missing a run means unbilled customer revenue.
• Cache Pruning / Health Checks: Must SKIP. Running 120 consecutive cache purges provides zero value and overloads your Redis cluster.

In Go, skip logic is simple:

 1func CalculateNextRun(cronExpr string, lastNextRun time.Time, catchUpPolicy string) (time.Time, error) {
 2	sched, err := cron.ParseStandard(cronExpr)
 3	if err != nil {
 4		return time.Time{}, fmt.Errorf("parse cron expression: %w", err)
 5	}
 6	now := time.Now()
 7
 8	if catchUpPolicy == "SKIP" && lastNextRun.Before(now) {
 9		// Advance next_run to the upcoming valid time slot relative to NOW
10		return sched.Next(now), nil
11	}
12
13	// Advance relative to previous scheduled time (Backfill step-by-step)
14	return sched.Next(lastNextRun), nil
15}

Summary & Architectural Checklist

Building a reliable distributed job scheduler means moving away from local timers and sharing state safely.

When building or reviewing your scheduler architecture, verify these core properties:

By pairing PostgreSQL’s concurrency primitives with Go’s lightweight channel mechanics, this architecture can execute millions of scheduled tasks across a multi-tenant gateway with zero duplicate runs and sub-millisecond polling latency.