About Services Tools Blog Contact us

Automation Job Queue

A to-do list for slow automation work, handed to a free worker so the web page responds straight away.

  • Scheduled Workers & Sync
  • Professional services
  • Scheduled worker
  • Product
  • Local/private
  • 2026

The problem

Rebuilding search indexes and other slow jobs time out when they run while a page loads. So jobs were started by hand on whichever computer happened to be free, and a long task blocked the next one. A failure disappeared instead of staying visible.

What it does

Each job is added to the list with a type, priority, details and an optional duplicate check, then a signal file wakes a background worker. Jobs move from waiting to picked up to done, with a count of attempts and the last error. A person can pause the list, and the procedures themselves stay separate.

Outcomes

  • Slow work no longer times out the page.
  • A failed job stays visible with its error.
  • Duplicate jobs collapse on the duplicate key.
  • The next job waits instead of stacking on a busy machine.

Simplified mockup

Interactive preview with sample data. Nothing is sent to a server.

Job queue

Queued
4
Completed
0
Failed
0
Workers free
2
Open jobs 1 duplicate collapsed onto 1042
JobTypePriStatusDetail
1042 Embedding rebuild 20 Queued dedupe emb-catalogue-oct
1043 Board pack render 40 Queued 24 pages, house template
1044 Address file clean 80 Queued 640 rows waiting
1045 Catalogue thumbnails 90 Queued 180 images waiting

Workers

Worker A
Free
Last claim 1041, address clean, yesterday
Worker B
Free
Last claim 1038, thumbnail pass, yesterday
How a job leaves the page
  1. The page inserts the row and returns.
  2. The trigger file wakes a backend process.
  3. A free worker claims the next row and records the start.
  4. A busy machine does not take a second job on top.

Enqueue

New job The request returns before the work runs
Already collapsed

A second embedding rebuild used the same dedupe key and updated job 1042 instead of inserting another row.

How it works

  1. Add a job with its type, details, priority and an optional duplicate check.
  2. Return the web page at once.
  3. Wake a worker with the signal file.
  4. Hand the next job to a free worker and record the start.
  5. On success, store the result. On failure, keep the error on the job.
  6. Let a person pause the list.

Pipeline

flowchart LR
  Web["Web request"] --> Enqueue["Queue row"]
  Enqueue --> Wake["Wake worker"]
  Wake --> Claim["Claim"]
  Claim --> Done["Result"]
  Claim --> Fail["Last error"]
  Pause["Pause"] --> Enqueue