Introduction

A good practice in software development is to delegate as much heavy work as possible to background jobs to avoid blocking the main execution of your application that can be a web app, mobile app or desktop.

Send email notifications it’s the typical scenario where you should execute it in background.

More scenarios

• Image processing
• Data aggregation / migration / conversion
• Push notifications

What else do you think?

Some platforms offer cheaper CPU running in background so you can save money in addition to user experience.

Why is it important?

Imagine several users making requests to you server that last more than 30 seconds or one minute. Your web app will get slow soon because HTTP connection are not infinite.

Queue several jobs it’s pretty easy but what it’s no easy is to process them one by one or in batches, set states, retry if some of them fail, etc.

This is a common problem so you shouldn’t implement a solution from scratch

Better Queue

From many solutions we have in Node JS, better-queue module is a good one.

Better Queue is designed to be simple to set up but still let you do complex things.

From better-queue

By default it uses an in-memory queue but to configure a persistent queue with Redis or MySQL is pretty easy because there are drivers written for better-queue.

More features

• Persistent (and extendable) storage
• Batched processing
• Prioritize tasks
• Merge/filter tasks
• Progress events (with ETA!)
• Fine-tuned timing controls
• Retry on fail
• Concurrent batch processing
• Task statistics (average completion time, failure rate and peak queue size)
• … and more!

Photo by Hal Gatewood on Unsplash