Webhooks are essential to many SaaS integrations, enabling apps to communicate and share data in real time. Their simplicity, HTTP POST requests to a specified URL, has made them the tool of choice for teams that need to efficiently and quickly transfer data.
We are visiting today with Justin Hipple, Co-founder and Chief Software Architect here at Prismatic. Since Justin recently completed an internal webhooks project, the topic is fresh on his mind.
Bru: Let's dive right in. Webhooks are known for being real-time, so let's start with the "when." When should my system trigger webhooks?
Justin: The purpose of sending webhooks is to notify interested parties of changes in the system, ideally in real time. This leads to a much more efficient architecture, as it prevents interested parties from having to run many expensive queries against your API. So the short answer is: "Whenever there is a relevant state change." This is a relatively small subset of the total set of state changes, and you'll be most focused on the most important entities in your data model. For example, if your application is an issue-tracking system, you would likely want to emit an "issue modified" webhook event when an issue's title is changed, but maybe not when someone adds themselves as a subscriber to issue updates.
Our customers can subscribe to these events. Most of these correspond to CRUD actions (other than read) performed against the primary data entities in Prismatic. Additional events are emitted when other important non-CRUD events occur. Other systems will likely have important domain-specific events beyond the regular CRUD events.
Bru: Got it. If I'm building webhook functionality into my app, what does a good webhook architecture look like?
Justin: You should design your architecture so that webhook event emission is as lightweight as possible, without negatively impacting the performance of the business logic. Most of the code is likely to be colocated with business logic in the API rather than in client code such as your frontend, though there may be exceptional cases. You will almost certainly want to ensure the webhook emission code is outside any database transactions, as it is generally poor practice to issue additional network requests or other expensive operations while a database transaction is open.
The central piece of infrastructure you'll need is a queue. Webhook event payloads will be added to the queue as state changes occur, and then one or more queue consumers will pull items from the queue and make the actual network requests to forward the payloads to their intended recipients. We use AWS SQS for our webhooks, but it could be Kafka, RabbitMQ, Redis, or any number of other tools.
You may want to consider whether the queue needs strict FIFO behavior, what sort of delivery semantics you want (at least once, exactly once, etc.), if deduplication is a relevant factor, and whether your queue will be responsible for deduplication or if that will be handled elsewhere in your application or by the receivers.
Bru: Now that we have the architecture, what about the data? How do I determine which data I should send via webhooks?
Justin: This will depend on several factors. You will want to send data that downstream consumers are interested in, so they can accomplish their tasks without having to immediately re-query your API for additional data. Re-querying may still be occasionally necessary, but you should minimize it. You will want to avoid sending irrelevant data. You should also minimize sending data that is not immediately available in the context of the event-emitting code, as gathering it can be quite expensive. You will also need to consider security and compliance frameworks, to avoid sending data that violates any data residency policies your app must adhere to.
Bru: So, I can send a broad range of data. What does a collection of that data (aka payload) look like?
Justin: Payloads should be as uniform as possible. They should use a standard envelope that always includes a set of required fields with defined data types. The envelope encapsulates a dynamic set of fields specific to the event being emitted. These fields should be as uniform in format as possible (same field naming conventions, etc.) and use reasonable data types that are standard across all emissions of a particular event type.
Bru: Are there some things I can do to keep that payload secure? What's the benefit of using HMAC or something similar?
Justin: The benefit is that receivers can verify that webhooks originated in your system. Interested parties will, by design, expose an endpoint in their system so that your system can send webhook events to it. Unfortunately, any other system can also send events to their system if that endpoint is discovered. It could be extremely problematic if an attacker were to discover an unsecured endpoint, so using verification (such as HMAC) is a good idea.
The Prismatic webhook system uses HMAC-SHA256 and includes it in the x-webhook-signature header, which is a common approach to securing webhooks.
Bru: Since nothing works perfectly all the time, how should I handle it when an endpoint fails?
Justin: Several choices range widely in terms of technical difficulty. The easiest option is to fail and move on. The receiver won't receive that event, and that may be OK for some applications.
Another option is to implement a retry policy. A common policy is to use a maximum number of retries with some delay between them. Consider using exponential backoff, where the retry interval increases exponentially. This is usually a good strategy, as it handles most common transient failures in distributed networks and is very simple to implement, since most HTTP clients provide good support for it. The Prismatic webhook system retries three times with exponentially increasing delays. Prismatic also uses an automatic cooldown system: endpoints that have failed repeatedly within a given timeframe are temporarily ignored to avoid wasting system resources by delivering payloads to systems that may be experiencing downtime. Sending is automatically restarted after five minutes.
The last option is to re-queue the event. This is by far the most complex solution, and how complex it gets really depends on other factors like whether FIFO is critical, your delivery policy (at least once/exactly once), etc. There are many failure modes and corner cases to consider here. It's best avoided unless your application absolutely requires it.
Bru: OK. When I send a webhook, what kinds of responses should I expect?
Justin: You determine the response that you need. You will want to make it simple for downstream consumers to implement, however. If you are using at-least-once delivery and don't care about retries, then you don't even need to wait for a response. That is unlikely to be the case in a real production application, though. Generally, you'll want an HTTP 2XX response, most often just HTTP 200 (OK). This indicates that the request was received and processed successfully.
Depending on your policies, you should retry 4XX and 5XX responses. Of particular note is 429 (Slow Down). This indicates the receiver cannot keep up with the volume of events it's receiving, and you need to slow down. To handle this scenario, an exponential backoff retry policy is a good strategy. Consider including a value in the response as well, a form of challenge code, so that you can know for sure the receiving system was the intended one and that processing was successful, but this may not always be necessary.
Bru: Speaking of retries, this next question ties into that topic. How important is idempotency when working with webhooks?
That depends on the type of application and what kind of events you are emitting. A simple example might be a financial application. If you were to send an event every time the balance changed, something like { amountChanged: 100.23}, { amountChanged: -29.40 }, etc., then it would be essential for any consumer trying to maintain a running balance that you send each event exactly once (that is, idempotently), with no duplicates or failures. However, this is obviously impractical; in reality, you would likely include the resulting balance in the payload, thereby obviating the need for strict idempotency.
Additionally, it's a good idea to uniquely identify each webhook payload by generating a UUID and including it in the payload envelope. This way, receivers can easily implement their own additional deduplication logic as needed, thereby achieving the benefits of idempotency even if the underlying events aren't truly idempotent.
Bru: Final question for today. Why should I implement webhooks in my system if I want to do integrations?
Justin: Webhooks have become something of an unofficial standard for programmatically exchanging data in near real time, and their ubiquity makes them an excellent choice for implementing integrations. They help keep systems decoupled by imposing a minimal set of requirements, most of which can be easily implemented with modern tooling and infrastructure. They also reduce the burden on both sending and receiving systems by minimizing the need to run bespoke, often expensive, queries against APIs. And they help keep integration code "out towards the edges" of applications, rather than co-mingling it with and tightly coupling it to complex business logic.
Bru: And that brings us to the end of today's discussion on webhooks. Thank you, Justin.
Check out our docs for details on how Prismatic enables you to work with webhooks for integrations.
