Queue & Distributed Job Processing

Katal features a production-grade background queue and job processing system with zero external dependencies. It supports an in-memory driver for development, a native Redis Streams distributed driver for multi-node production clusters, and a 5-field UNIX Cron Scheduler.


1. Architecture

flowchart TD
    App["Application / Controllers"] -->|enqueue| Store["QueueStore Interface"]
    Store -->|Development| Memory["InMemoryQueue"]
    Store -->|Production| Redis["RedisStreamsQueueStore (RESP Socket)"]
    Worker["QueueWorker"] -->|poll / XREADGROUP| Store
    Worker -->|success| Ack["Complete & Acknowledge"]
    Worker -->|max attempts reached| DLQ["Dead-Letter Queue (${stream}:dlq)"]
    Scheduler["QueueScheduler"] -->|UNIX Cron & Intervals| Store
Component Responsibility
QueueStore Abstract storage contract (enqueue, dequeue, complete, fail, get)
InMemoryQueue In-memory store for local testing and lightweight services
RedisStreamsQueueStore High-throughput distributed queue driver powered by native RESP TCP sockets
QueueWorker Worker engine with concurrency control, polling backoff, and handler execution
QueueScheduler 5-field UNIX Cron and interval scheduler with overlap prevention

2. Distributed Redis Streams Driver

For multi-node production clusters, Katal provides a native Redis Streams queue store that communicates directly over TCP sockets without third-party packages (ioredis or redis are not needed):

Enabling Redis Streams

import { Application } from "katal";

const app = new Application({ port: 3000 });

// Switch application queue to production Redis Streams
app.useRedisQueue({
    host: "127.0.0.1",
    port: 6379,
    streamName: "katal:jobs:main",
    group: "workers",          // Consumer group
    consumer: "worker-node-1", // Unique consumer ID
    password: process.env.REDIS_PASSWORD,
});

Redis Streams Capabilities:

  • Consumer Groups (XREADGROUP): Workloads are distributed across worker nodes without race conditions or duplicated jobs.
  • Atomic Acknowledgements (XACK & XDEL): Processed jobs are durably acknowledged and removed from the active stream.
  • Delayed Jobs via Sorted Sets: Delayed and retried jobs are stored in a sorted set (ZADD) and atomically promoted using Lua scripting.
  • Stalled Worker Recovery (XAUTOCLAIM): If a worker process crashes or loses network connectivity mid-execution, surviving workers automatically claim pending stalled jobs.
  • Dead-Letter Queue (DLQ): Jobs exceeding maxAttempts are durably routed to ${streamName}:dlq with full error messages and timestamps.

3. Enqueueing Jobs

Basic Enqueue

const queue = app.getQueue();

await queue.enqueue("send-welcome-email", {
    userId: "usr_42",
    email: "alice@example.com",
});

With Exponential Backoff & Jitter

await queue.enqueue("process-payment", { orderId: "ord_99" }, {
    maxAttempts: 5,
    retry: {
        strategy: "exponential", // "none" | "fixed" | "exponential"
        delayMs: 500,
        maxDelayMs: 30_000,
        jitter: true,            // Adds ±20% randomness to prevent stampedes
        jitterFactor: 0.2,
    },
});

Delayed Delivery

// Deliver 60 seconds into the future
await queue.enqueue("send-reminder", { appointmentId: "apt_1" }, {
    delayMs: 60_000,
});

4. Processing Jobs with QueueWorker

Register handler functions and start the worker supervision loop:

import { QueueWorker } from "katal";

const worker = new QueueWorker(app.getQueue(), {
    pollIntervalMs: 50,  // Check interval when idle
    concurrency: 4,      // Max concurrent jobs per worker instance
});

worker.register<{ userId: string; email: string }>("send-welcome-email", async (job) => {
    console.log(`Sending welcome email to ${job.payload.email}`);
    // Process work...
});

worker.register<{ orderId: string }>("process-payment", async (job) => {
    // If an error is thrown, the worker automatically applies the configured retry backoff
    await paymentGateway.charge(job.payload.orderId);
});

// Start worker
worker.start();

// Graceful shutdown on SIGTERM / SIGINT
process.on("SIGTERM", async () => {
    await worker.stop();
});

5. UNIX Cron & Job Scheduling

Katal features a built-in 5-field UNIX Cron Scheduler matching standard crontab syntax:

$$\text{minute}\quad\text{hour}\quad\text{dayOfMonth}\quad\text{month}\quad\text{dayOfWeek}$$

import { QueueScheduler } from "katal";

const scheduler = new QueueScheduler(app.getQueue());

// 1. Run every midnight with overlap prevention
scheduler.everyCron("generate-daily-invoices", {}, "0 0 * * *", {
    withoutOverlapping: true,
});

// 2. High-frequency cron: every 15 minutes during business hours
scheduler.everyCron("sync-inventory", {}, "*/15 9-17 * * 1-5");

// 3. Human-readable @every syntax
scheduler.everyCron("heartbeat-metrics", {}, "@every 30s");

// 4. One-shot delayed job
await scheduler.scheduleOnce("deactivate-trial", { accountId: "acc_1" }, 86_400_000);

// Stop recurring schedules
scheduler.stopAll();

6. Inspecting Dead-Letter Queues (DLQ)

When jobs exceed their configured retry limits, you can inspect and replay the DLQ:

const redisStore = app.getQueue() as RedisStreamsQueueStore;

const deadJobs = await redisStore.getDlq();
for (const dead of deadJobs) {
    console.error(`Failed job ${dead.id} (${dead.name}): ${dead.error}`);
}

See Also