Article
Background Jobs, Queues and Cron for Small Web Apps
Imagine a customer submits an order on your web app. Behind the scenes, the app saves the order, sends a confirmation email, generates a PDF invoice, notifies the warehouse and pushes data to an accounting system. If all of that happens before the page responds, the customer stares at a spinner, and if the accounting API is slow or down, the whole order may fail.
Background jobs solve this. The app does the essential work immediately, responds to the user, and hands slower tasks to a separate worker. Scheduled jobs, often called cron jobs, handle work that should run at set times. This guide explains both and how to use them safely in small web apps.
What Belongs in a Background Job
Move work out of the request when it is slow, unreliable or not needed for the immediate response. Common examples:
- Sending emails, SMS or push notifications
- Generating PDFs, invoices and reports
- Resizing or optimising uploaded images. See our guide to image optimisation for why this matters.
- Calling third-party APIs such as accounting, shipping or CRM systems
- Importing or exporting large CSV files
- Processing incoming webhooks that need heavier work
Keep in the request anything the user must see right away, such as validation results and confirmation that their order was saved.
How Queues Work
A queue is a list of jobs waiting to be processed. Your web app pushes a job, such as "send order confirmation for order 123", into the queue. One or more worker processes run separately, take jobs from the queue and do the work.
Most popular frameworks include queue systems, for example Laravel queues, Django with Celery or other task libraries, and Node.js libraries backed by Redis. Queues can be stored in Redis, a database table or a managed cloud queue service. For a small app, a database-backed queue is often enough to start; Redis or a managed queue helps as volume grows.
Scheduled Jobs and Cron
Some work is not triggered by a user but by time. Examples include nightly database backups, daily sales summary emails, clearing expired carts, renewing subscriptions and syncing stock levels with a marketplace every hour.
On Linux servers, cron runs commands on a schedule. Many frameworks provide a scheduler where you define tasks in code and a single cron entry runs the scheduler every minute. That keeps your schedule version-controlled with the rest of the app instead of hidden in a server setting someone forgets during a migration.
Tip: make every scheduled task log when it starts, finishes and fails, and alert someone if an important job does not run. Silent cron failures can go unnoticed for weeks, such as backups that quietly stopped.
Make Jobs Safe to Retry
Jobs fail: an API times out, a server restarts, a bug slips through. A good job system retries failed jobs, which means your jobs must be safe to run more than once. This property is called idempotency.
- Check before acting: before sending an invoice email, check whether it was already sent.
- Use unique keys: when calling external APIs that support idempotency keys, use them so retries do not create duplicates.
- Limit retries and back off: retry a few times with increasing delays, then move the job to a failed list for review.
- Keep jobs small: pass IDs rather than large data, and load fresh data inside the job.
- Set timeouts: a job stuck forever blocks a worker.
The same thinking applies to incoming webhooks from payment gateways and marketplaces; see our guide on payment gateway integration for related patterns.
Running Workers in Production
Workers are long-running processes, so they need supervision. Use a process manager or your platform's worker feature to start them on boot and restart them if they crash. Restart workers after each deploy so they run the new code. Monitor the queue length: if jobs pile up, add workers or investigate slow tasks. On shared hosting where long-running workers are not allowed, a scheduled task that processes a batch of queued jobs every minute can be a workable compromise.
Finally, give your team visibility. A simple dashboard, many queue libraries include one, showing pending, running and failed jobs makes problems obvious at a glance. When a customer says they never received an invoice, being able to search for the job and see exactly what happened turns a long investigation into a two-minute answer.
FAQ
Do small apps really need queues?
Not always on day one. But as soon as you send emails or call external APIs during requests, moving them to background jobs usually makes the app faster and more reliable.
What happens to jobs if the server restarts?
With a persistent queue such as a database table or Redis configured for durability, jobs remain queued and workers pick them up after restart.
How do I see which jobs failed?
Most queue systems record failed jobs with the error. Review them regularly and set up alerts for important failures.
Background jobs and scheduled tasks are what make a web app feel fast while still doing a lot of work. Start with your slowest request, move its heavy parts into a job and build the habits of logging, retries and monitoring from the beginning.