ruby on rails - Define delayed_jobs dependencies
I am using
delayed_jobs that for some of them have dependencies that need to be executed after the current one and with a small delay. This delay comes from rate limitting of the API I am querying.
Here are my jobs constraints:
1s 1s A1 --------> A2 --------> A3
1s 1s B1 --------> B2 --------> B3
In the end, I would like this process to take ~2s with something like:
1s 1s A1 -> B1 -----> B2 -> A2 ------> A3 -> B3
1s 1s A1 -> B1 -----> A2 -> B2 ------> A3 -> B3
or whatever order which respects my first constraints, but with dependencies and waiting.
I am currently building an app for Shopify. An app for Shopify is rate limitted by 2 API calls/s per shop.
But the webhooks coming from shopify that send me jobs to do can be way more than 2 per second per shop so I need to use jobs to handle them. Then each job actually send two request to their API.
I could use a second queue and use the
after hook to fetch the next job to do for the shop and add it to the jobs table with
run_at: 1.seconds.from_now. However, I don't really like the idea, because I'll have to store the
perform arguments in the table instead of just calling a
perform_later method. Also, I really imagine that this isn't the first use case where dependencies are needed with delayed_job, so I'm sure there is a better approach to it that I just couldn't find.
Any idea would really be appreciated!
I am not at all clear what you're trying to do, but a good common pattern that has worked for me most of the time (many millions of times with a few exceptions)...
when you get a webhook call, create the job (if you have not already seen that webhook call), and return 200 status as fast as possible. That keeps Shopify happy and stops them sending you the same hook over again.
MonkeyPatch ActiveResource to inspect the credit left header number in your App, so that you can easily go into a 500ms wait if the calls left are zero. Works a charm... for 2/second limit.
Didn't find the answer?
Our community is visited by hundreds of Shopify development professionals every day. Ask your question and get a quick answer for free.
Find the answer in similar questions on our website.
Write quick answer
Do you know the answer to this question? Write a quick response to it. With your help, we will make our community stronger.