Hey Lorna, good read!
Here’s a trick I’ve found to achieve something similar without using workers. In one of my apps, I also need to call some webhooks and given the nature of these webhooks they usually take forever to complete so doing them synchronously is definitely not an option. Also maintaining a set of concurrent workers for this app was not an option either as it would require some supervisord (or similar) which we didn’t want to maintain. What I found that works surprisingly well is the following setup:
You PHP code sends an SNS notification to a topic. There is a lambda function that is subscribed to this topic that makes the actual HTTP calls and in turn publishes the results to another SNS topic that our app is subscribed to (for processing the output of the HTTP calls).
The key thing here is that the initial SNS call take two-digit milliseconds to complete and you can hammer the SNS endpoint massively. In our load tests we ran thousands of concurrent calls to SNS (the AWS PHP SDK handles concurrency for you) and there NEVER were any delays. In this setup it doesn’t matter if your code has 1 or 1000 webhooks to process; it’s always going to be very fast.
A sweet side benefit is that if the webhook fails (some transient HTTP burp, for example), SNS will retry the call for you automatically for as many times as you want with configurable retry attempts and retry-wait (like linear/exponential backoff). Also Amazon will limit the concurrency depending on how your Lambda function and VPC are configured so that’s also something you don’t have to worry about.
This way you get speed when “queuing” the webhooks, you get the output of the HTTP calls, you get automatic retries, and you never had to set up and maintain any queues or daemons.