DataQueueAbout
DataQueue is an open source lightweight job queue for Node.js/TypeScript projects, backed by PostgreSQL. It lets you easily schedule, process, and manage background jobs. It's ideal for serverless environments like Vercel, AWS Lambda, and more.
Features
- Simple API for adding and processing jobs
- Strong typing for job types and payloads, preventing you from adding jobs with the wrong payload and ensuring handlers receive the correct type
- Works in serverless environments
- Supports job priorities, scheduling, canceling, and retries
- Reclaims stuck jobs: No job will remain in the
processingstate indefinitely - Cleans up old jobs: Keeps only jobs from the last xxx days
Who is this for?
This package is for you if all of the following apply:
| ☁️ | You deploy web apps to serverless platforms like Vercel, AWS Lambda, etc. |
| 📝 | You use TypeScript |
| ⚡ | You want your app to stay fast and responsive by offloading heavy tasks to the background |
| 💾 | You use PostgreSQL as your database |
| 🤷♂️ | You don't want to set up or maintain another queue system like Redis |
| 💸 | You're on a budget and want to avoid paying for a job queue service or running your own server |
Why PostgreSQL?
Many people use Redis for job queues, but adding another tool to your stack can increase costs and maintenance. If you already use PostgreSQL, it makes sense to use it for job queues, thanks to SKIP LOCKED.
The update process in DataQueue uses FOR UPDATE SKIP LOCKED to avoid race conditions and improve performance. If two jobs are scheduled at the same time, one will skip any jobs that are already being processed and work on other available jobs instead. This lets multiple workers handle different jobs at once without waiting or causing conflicts, making PostgreSQL a great choice for job queues and similar tasks.