Postiz Files

We have had some problems with concurrency for a hefty load (primarily for Postiz cloud).

Here are the changes we have implemented: - Workers can now reach a maximum of 300 concurrent jobs. - Bottleneck JS timeout (expiration) is set to 60 seconds for a single HTTP request; if it fails, it returns to the user "Something is wrong with ${identifier}" - Max Stalled jobs changed to 10. - Cron for missing jobs - fixed.

Ideology behind the change: Until recently, Postiz added posts to the queues and processed them without any concurrency, which works great with open-source instances that are not automated.

However, the problem was the restrictions of social media platforms - what if I want to post 10 posts to Instagram at the same time? If Instagram has a concurrency limit per second, what is the maximum number of requests that I can make?

Instagram will throw you off with a rate limit error. So we needed a better solution to make "http" requests, wait between multiple workers. So only X requests could run concurrently, and wait for the other requests.

Since you may want to run multiple workers simultaneously, the solution must be Redis-supported and not limited to local execution. So libraries like p-queue would not be sufficient.

Another thought we have is to move to bullmq-pro, which would make concurrency of the entire job (not HTTP request) The main problem was that it costs money, so it's hard to push it into the open-source, although it would probably be the easiest solution.

Bullmq-pro allows you to create concurrent jobs based on groups - so a group can be, for example, "LinkedIn" or "Instagram". So, as promised, only one concurrent job will be run at a time on the social platform.

In any case, I hope these transparent updates provide you with some clarity.

Full Changelog: https://github.com/gitroomhq/postiz-app/compare/v2.5.2...v2.4.3