Overview
Good practice in software development is to delegate as much heavy work as possible to background jobs to avoid blocking the main execution of your application that can be a web app, mobile app, or desktop.
Send email notifications it’s the typical scenario where you should execute it in background.
More scenarios
- Image processing
- Data aggregation / migration / conversion
- Push notifications
Some platforms offer cheaper CPU running in background so you can save money in addition to user experience.
Why is it important
Imagine several users making requests to your server that lasts more than 30 seconds or one minute. Your web app will get slow soon because HTTP connections are not infinite.
Queue several jobs it’s pretty easy but what it’s no easy is to process them one by one or in batches, set states, retry if some of them fail, etc.
This is a common problem so you shouldn’t implement a solution from scratch
Better Queue
From many solutions we have in Node JS, better-queue module is a good one.
Better Queue is designed to be simple to set up but still let you do complex things.
From better-queue:
By default, it uses an in-memory queue but to configure a persistent queue with Redis or MySQL is pretty easy because there are drivers written for better-queue.
More features
- Persistent (and extendable) storage
- Batched processing
- Prioritize tasks
- Merge/filter tasks
- Progress events (with ETA!)
- Fine-tuned timing controls
- Retry on fail
- Concurrent batch processing
- Task statistics (average completion time, failure rate and peak queue size)
