Queue & Distributed Job Processing¶
Katal features a production-grade background queue and job processing system with zero external dependencies. It supports an in-memory driver for development, a native Redis Streams distributed driver for multi-node production clusters, and a 5-field UNIX Cron Scheduler.
1. Architecture¶
flowchart TD
App["Application / Controllers"] -->|enqueue| Store["QueueStore Interface"]
Store -->|Development| Memory["InMemoryQueue"]
Store -->|Production| Redis["RedisStreamsQueueStore (RESP Socket)"]
Worker["QueueWorker"] -->|poll / XREADGROUP| Store
Worker -->|success| Ack["Complete & Acknowledge"]
Worker -->|max attempts reached| DLQ["Dead-Letter Queue (${stream}:dlq)"]
Scheduler["QueueScheduler"] -->|UNIX Cron & Intervals| Store
| Component | Responsibility |
|---|---|
QueueStore |
Abstract storage contract (enqueue, dequeue, complete, fail, get) |
InMemoryQueue |
In-memory store for local testing and lightweight services |
RedisStreamsQueueStore |
High-throughput distributed queue driver powered by native RESP TCP sockets |
QueueWorker |
Worker engine with concurrency control, polling backoff, and handler execution |
QueueScheduler |
5-field UNIX Cron and interval scheduler with overlap prevention |
2. Distributed Redis Streams Driver¶
For multi-node production clusters, Katal provides a native Redis Streams queue store that communicates directly over TCP sockets without third-party packages (ioredis or redis are not needed):
Enabling Redis Streams¶
import { Application } from "katal";
const app = new Application({ port: 3000 });
// Switch application queue to production Redis Streams
app.useRedisQueue({
host: "127.0.0.1",
port: 6379,
streamName: "katal:jobs:main",
group: "workers", // Consumer group
consumer: "worker-node-1", // Unique consumer ID
password: process.env.REDIS_PASSWORD,
});
Redis Streams Capabilities:¶
- Consumer Groups (
XREADGROUP): Workloads are distributed across worker nodes without race conditions or duplicated jobs. - Atomic Acknowledgements (
XACK&XDEL): Processed jobs are durably acknowledged and removed from the active stream. - Delayed Jobs via Sorted Sets: Delayed and retried jobs are stored in a sorted set (
ZADD) and atomically promoted using Lua scripting. - Stalled Worker Recovery (
XAUTOCLAIM): If a worker process crashes or loses network connectivity mid-execution, surviving workers automatically claim pending stalled jobs. - Dead-Letter Queue (DLQ): Jobs exceeding
maxAttemptsare durably routed to${streamName}:dlqwith full error messages and timestamps.
3. Enqueueing Jobs¶
Basic Enqueue¶
const queue = app.getQueue();
await queue.enqueue("send-welcome-email", {
userId: "usr_42",
email: "alice@example.com",
});
With Exponential Backoff & Jitter¶
await queue.enqueue("process-payment", { orderId: "ord_99" }, {
maxAttempts: 5,
retry: {
strategy: "exponential", // "none" | "fixed" | "exponential"
delayMs: 500,
maxDelayMs: 30_000,
jitter: true, // Adds ±20% randomness to prevent stampedes
jitterFactor: 0.2,
},
});
Delayed Delivery¶
// Deliver 60 seconds into the future
await queue.enqueue("send-reminder", { appointmentId: "apt_1" }, {
delayMs: 60_000,
});
4. Processing Jobs with QueueWorker¶
Register handler functions and start the worker supervision loop:
import { QueueWorker } from "katal";
const worker = new QueueWorker(app.getQueue(), {
pollIntervalMs: 50, // Check interval when idle
concurrency: 4, // Max concurrent jobs per worker instance
});
worker.register<{ userId: string; email: string }>("send-welcome-email", async (job) => {
console.log(`Sending welcome email to ${job.payload.email}`);
// Process work...
});
worker.register<{ orderId: string }>("process-payment", async (job) => {
// If an error is thrown, the worker automatically applies the configured retry backoff
await paymentGateway.charge(job.payload.orderId);
});
// Start worker
worker.start();
// Graceful shutdown on SIGTERM / SIGINT
process.on("SIGTERM", async () => {
await worker.stop();
});
5. UNIX Cron & Job Scheduling¶
Katal features a built-in 5-field UNIX Cron Scheduler matching standard crontab syntax:
$$\text{minute}\quad\text{hour}\quad\text{dayOfMonth}\quad\text{month}\quad\text{dayOfWeek}$$
import { QueueScheduler } from "katal";
const scheduler = new QueueScheduler(app.getQueue());
// 1. Run every midnight with overlap prevention
scheduler.everyCron("generate-daily-invoices", {}, "0 0 * * *", {
withoutOverlapping: true,
});
// 2. High-frequency cron: every 15 minutes during business hours
scheduler.everyCron("sync-inventory", {}, "*/15 9-17 * * 1-5");
// 3. Human-readable @every syntax
scheduler.everyCron("heartbeat-metrics", {}, "@every 30s");
// 4. One-shot delayed job
await scheduler.scheduleOnce("deactivate-trial", { accountId: "acc_1" }, 86_400_000);
// Stop recurring schedules
scheduler.stopAll();
6. Inspecting Dead-Letter Queues (DLQ)¶
When jobs exceed their configured retry limits, you can inspect and replay the DLQ:
const redisStore = app.getQueue() as RedisStreamsQueueStore;
const deadJobs = await redisStore.getDlq();
for (const dead of deadJobs) {
console.error(`Failed job ${dead.id} (${dead.name}): ${dead.error}`);
}