Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Running this exact implementation with 47M jobs processed and counting. SKIP LOCKED is great for VACUUM, and having durable storage with indexes make otherwise expensive patterns like delayed jobs, retries, status updates, "at least once", etc. really easy to implement.


Do you have some idea of how many jobs per minute or hour do you have? Just want to compare with what we have on Redis at work.

Do you also have any idea on the concurrency? How many workers you have pulling from Postgres.

I’ve used this approach before (ages ago) when Redis wasn’t even a thing, though not at high throughout requirements.


I'm sure Redis is much faster than an RDBMS w/ all the ACID features turned on. The biggest concern I always have with Redis is simply overwhelming the in-memory storage limits when someone wants to do process a large number of good-sized messages at an inconvenient time. #tradeoffs


I’ve seen it used for up to 1000 jobs per second with concurrency of 3-12


I recently did the same thing but without LISTEN/NOTIFY and with a partial index. Pushed about 700 jobs/sec with 1000 workers (via pgbouncer).




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: