Building Two Async Job Systems From Scratch: Lessons in Reliability, Scheduling, and Knowing When to Stop

Most backend systems eventually converge on the same pattern: something enqueues work, something else picks it up and does it, and someone wants to know if it succeeded. I built two systems around this pattern recently — a Go-based distributed job execution platform (go-distributed-worker) and a Python job search automation pipeline (job_search_automation). They share the same conceptual skeleton but solve very different problems, and the differences reveal more about system design than either project does alone.

The core problem they both solve

Both systems exist because certain work shouldn’t happen synchronously. Submitting an HTTP request and waiting for a response is fine for reading a database row. It’s not fine when the work takes variable time, needs to survive crashes, or should happen on a schedule. The Go platform handles this for arbitrary job types — send emails, transform data, call webhooks. The Python pipeline handles it for a specific workflow — aggregate job listings, generate cover letters, produce a spreadsheet. Same structural problem, different stakes and scale.

The Go platform: what production-grade actually requires

go-distributed-worker started as “what would it take to build a simplified Celery in Go” and ended up with fifteen distinct features. The gap between “enqueue something and have a worker pick it up” and “enqueue something and have it reliably execute, survive failures, support retries, and scale horizontally” is enormous.

Worker registration and heartbeats

The obvious part is workers pulling jobs from a queue. The part that took real design thinking was liveness detection. Workers register themselves on startup and send periodic heartbeats to Redis with a TTL. If a heartbeat expires, the scheduler knows that worker is dead and can recover its in-flight jobs. This is the difference between “the job disappeared into a dead worker” and “the job gets re-queued and executed by someone else.”

Job leasing, not just queuing

When a worker picks up a job, it doesn’t remove it from the queue — it leases it with a visibility timeout. If the worker completes the job, the lease is released. If the worker crashes, the lease expires and the job becomes visible to other workers again. This prevents duplicate execution without requiring distributed transactions.

The kill-worker demo

The most convincing demonstration of this architecture is the kill-worker scenario: spin up three workers, submit ten thousand jobs, kill one worker mid-execution. The heartbeat expires, leases are recovered, jobs are re-queued, and the remaining workers continue processing. No duplicate execution, no lost jobs. That’s the kind of reliability that separates a demo from a system you’d actually run in production.

The Python pipeline: reliability at a different scale

job_search_automation is a fundamentally different beast. It doesn’t need distributed workers or Redis-backed priority queues. It needs to aggregate job listings from a few sources, generate cover letters via LLM calls, and produce an Excel spreadsheet. The reliability requirements are lower — if the script crashes, you rerun it — but the design decisions are no less interesting.

Multi-provider LLM abstraction

The pipeline supports six LLM providers (Anthropic, OpenAI, Gemini, DeepSeek, OpenCode, Ollama) through a single call_llm() function. The dispatch logic lives entirely in llm_api.py. Any step can be pinned to a different provider via environment variables:

COVER_LETTER_PROVIDER=ollama        # batch drafting, free
CONNECTION_NOTE_PROVIDER=anthropic  # personalized outreach, paid

This matters more than it sounds like. When one provider rate-limits you mid-batch, having the ability to route individual steps to a different provider without touching generator code is a real operational advantage. The practical reason to split providers isn’t quality — it’s cost per token and avoiding a single point of failure.

The no-auto-submit decision

The original ask was a tool that fills forms and clicks submit on your behalf. I stopped short, deliberately:

1. Most application forms are one-off React components per company — there’s no reliable universal submit target to automate against.
2. Where direct POST submissions are technically possible, doing it without reading each cover letter first risks sending something wrong at scale.
3. On LinkedIn and especially, automated submission is the fastest way to get an account flagged.

The pipeline generates drafts. You review and submit manually. The automation is in the aggregation and drafting, not in the submission.

Local LLM for zero-cost runs

Running Ollama locally with a model like llama3.1:8b gives you fully offline, zero-cost cover letter drafting. For someone running this daily, the cost math matters — a cloud provider might cost a few dollars per batch, but the cumulative cost over months of job searching adds up. More importantly, keeping job search data off third-party servers is a real privacy consideration when you’re dealing with company names and job descriptions you’re not publicly associated with yet.

Where they converge

Both systems share a few structural decisions that reflect real-world async system design:

Idempotency. The Go platform uses idempotency keys to prevent duplicate job submission. The Python pipeline tracks processed jobs in an Excel spreadsheet and skips them on re-run. Same principle — don’t do the same work twice — at different scales.

Visibility into state. The Go platform exposes Prometheus metrics and OpenTelemetry traces. The Python pipeline writes status columns to a spreadsheet. Both let you answer “what happened” without grepping logs.

Graceful degradation. The Go platform has dead-letter queues for failed jobs. The Python pipeline logs skipped slugs and moves on. Neither system halts entirely because one unit of work failed.

What I’d do differently

If I rebuilt the Go platform, I’d question whether Redis is the right queue backend at all. For a system designed to survive worker failures, a log-based queue (Kafka, NATS JetStream) gives you replayability that Redis sorted sets don’t. Redis was the right choice for a learning project — the operational simplicity is real — but the “what if I need to replay the last hour of jobs” scenario is where it falls short.

If I rebuilt the Python pipeline, I’d add a simple SQLite database instead of the Excel spreadsheet for state tracking. The xlsx file works, but concurrent access (running the CLI and the Streamlit UI simultaneously) occasionally causes file lock issues that a database wouldn’t have.

The actual takeaway

Building reliable async systems isn’t about picking the right framework. It’s about answering a series of questions: what happens when a worker crashes mid-job? What happens when a job needs to retry? What happens when you need to know the state of a specific job without reconstructing it from logs? The Go platform answers these questions with heartbeats, leases, and priority queues. The Python pipeline answers them with simpler mechanisms because the stakes are lower. The right level of engineering complexity matches the actual failure modes you’re trying to survive, not the ones that sound impressive in a design doc.