Integrating with Google APIs: Tips and Tricks, Part 2

Last time, we did an overview of the Google API ecosystem. This time, we'll continue our discussion of Google APIs with Taylor Reece, a Developer Advocate at Prismatic. But we'll narrow our focus to Gmail and Google Drive because Google has included a fair amount of complexity with webhooks and notifications for these APIs, complexities that integration devs need to understand before they start building.

Both of these APIs are heavily used by our customers – B2B SaaS companies that build integrations between their products and the other apps their customers use. Fortunately for our customers, we have connectors for these APIs (Google Drive and Gmail) that remove much of this complexity for their developers. Whether you are using Prismatic, or building integrations from scratch, we trust this discussion will jumpstart your work with these APIs.

Bru: All right. Taylor, what can we expect from this discussion on the Gmail and Google Drive APIs?

Taylor: Let's compare and contrast these two APIs, specifically regarding how the apps notify you when a change has occurred. We touched on this in Part 1, but now I'd like to drill down into the details.

Bru: Will this have have applicability beyond these two APIs, or is this functionality unique to Gmail and Google Drive?

Taylor: Gmail and Google Drive implement webhooks in novel ways, but the overall concepts apply to many APIs inside and outside the Google ecosystem. Regardless of the implementation details, to build an event-driven integration, you need to be able to configure an app to notify you when certain events occur (like an email is received or a file is updated).

Bru: Understood. Let's start with Gmail since it seems to be the more complex of the two.

Taylor: Certainly. Things are definitely more complicated with Gmail and webhooks. Gmail doesn't send you notifications directly but instead goes through another Google Product, Google Cloud Pub/Sub, which brokers data from Gmail to your app.

Let's talk about what it takes to set up that pipeline.

As a first step, you'll need a Google Developer Account to create a Google Cloud Platform project with the Pub/Sub API enabled. Then, within that project, you'll set up a topic. And here's where it can get tricky. If you set up a single topic, then notifications from all of your users' Gmail accounts will be published to that topic (and you'll need to route those messages properly within your integration). But, if you set up each user with a different topic, you risk coming up against a Pub/Sub topic quota.

Bru: Sounds like you probably want to plan things out a bit here.

Taylor: It wouldn't hurt. Quotas are pretty large (like 10,000 topics in a single Pub/Sub project), so it's not something everyone will need to be concerned with. But user counts often grow a lot faster than you expect.

Bru: All right, back on the main track. Once you've set up the Pub/Sub topic, what's next?

Taylor: Next, we need to set up a subscription to the topic. Subscriptions can either operate as a push or pull. That is, you can either query Pub/Sub for notifications, or Pub/Sub can push notifications to your app. To make an integration more real-time, you're better off with a push subscription.

The last thing you need to do in your Pub/Sub configuration is to ensure that Gmail has privileges to publish to the topic (or topics) you've set up in Pub/Sub. Complete instructions for doing that via Identify and Access Management (IAM) may be found here.

Bru: OK. It sounds like we have the Pub/Sub pipeline in place. What happens on the Gmail side?

Taylor: Once you have your Pub/Sub topic, subscription, and permissions in place, you'll use your customer's credentials to call the Gmail API watch() function, passing Gmail your Pub/Sub topic endpoint. Once that's called, your customer's Gmail account will notify the topic (which will notify you) when emails are received, deleted, etc.

Gmail's webhook notifications are atypical. Your endpoint receives a message saying, "Something happened with john.doe@gmail.com's account." The message doesn't tell you what happened, just that something occurred.

That's where Gmail's historyId comes into play. When you call watch(), you'll receive a historyId to help you keep track of events you have and haven't processed yet. When you receive the "something happened" message, it's your job to request a list of changes from Gmail since the last time you got a list of changes (using that historyId). Then, you need to take note of a new historyId for the next time you get a "something happened" message.

Another thing you need to watch for that isn't very common in SaaS apps is webhook expiration. In Gmail, watch() expires after 7 days, so you'll need to renew the watch function every week, or Gmail will stop sending notifications to the Pub/Sub topic.

The last novel thing that's worth bringing up is credentials. To maintain a Gmail webhook pipeline, you'll use two sets of credentials – your credentials for configuring Google Pub/Sub, and your customer's Gmail authentication (likely via OAuth 2.0) for Gmail configuration.

Bru: Now that we've gone over the details for setting up webhooks for Gmail let's talk about the corresponding process for Google Drive.

Taylor: Sure. Google Drive is definitely the simpler of the two, though it follows a similar pattern with notifications and event history.

We don't need to fiddle with Google Cloud Pub/Sub for this one. Google Drive handles its own notifications to your app. To enable event notifications, you'll set up what Google calls a notification channel (an outgoing webhook) by invoking the file.watch or changes.watch methods, depending on whether you need notifications about a single file or an entire folder or drive. Once you've created the notification channel, an endpoint you specified will receive notifications when a file changes or when the contents of a folder or drive change.

Bru: That sounds like what we did with Gmail, where we get a notification first, right?

Taylor: Yes. And like Gmail notifications, Google Drive notifications are succinct. That is, the notification only says "Stuff changed," but not "Here's exactly what the change is." Again, as with Gmail, we need to perform another set of steps to get the actual changes.

When you create a webhook, you also need to retrieve an initial token via the changes.getStartPageTokenAPI endpoint. Then, whenever you get a "stuff changed" notification, you can call the changes.list method with the token you have. This will get the changes, though if many changes have occurred, you may need to page through them using a nextPageToken. Then, after you've processed all of the changes, you need to hold on to your newStartPageToken until the next time you get a "stuff changed" notification.

Bru: Do Google Drive webhooks expire like Gmail webhooks?

Taylor: Yeah, they're the same in that regard. A webhook for Google Drive can last for up to 7 days (for file changes), and 1 day (for folder or drive changes). But, if you don't set the expiration property in your *.watch request, it expires after 1 hour by default.

There's a critical difference between Gmail and Google Drive webhook configurations to watch out for. When you renew the webhook in Google Drive, you actually create a different webhook (not the same ID). So, depending on when you call the *.watch endpoint, you may have a gap in your notifications or even receive duplicate notifications. Google Drive doesn't do anything to let you know that the webhook will expire. It just stops working, and you must time your renewal accordingly.

Bru: Are there some specific things Prismatic does to make using webhooks with Gmail and Google Drive easier?

Taylor: Absolutely. With both Gmail and Google Drive, Prismatic allows you to run initial subscription actions immediately when an integration is deployed to a customer as part of a deploy flow, and you can run subsequent renewal actions on a consistent schedule after that. The webhook setup flows for both Gmail and Google Drive are pretty simple – just a couple of built-in actions – and once they've run, you can be assured that they'll renew consistently on a schedule. Here are a couple of example integrations from Prismatic's GitHub repo that template out exactly how you can maintain webhooks for both Google Drive and Gmail with the least amount of additional effort:

• https://github.com/prismatic-io/examples/blob/main/integrations/gmail-with-webhooks.yml

• https://github.com/prismatic-io/examples/blob/main/integrations/google-drive-webhooks.yml

Bru: Thank you, Taylor. And that brings us to the end of Part 2 of using Google APIs for integrations.

Schedule a demo to see how Prismatic makes it easier to build integrations connecting your app to Gmail and Google Drive, and then deploy and manage them at scale, while providing an in-app integration UX for your customers via the white label marketplace and embedded designer.