Handling Binary Files in Prismatic Integrations

In previous blog posts we've covered the flow of structured data (JSON, XML, and the like) through steps of Prismatic integrations. Data that flow through integrations aren't always structured, though. Many integrations require processing of unstructured binary files like images, PDFs, videos, and audio recordings.

Today I'd like to talk about handling binary files in Prismatic integrations and custom components. Let's assemble a binary-data-heavy integration and new custom component together, and examine how binary data flows between an integration's actions.

What We'll Build

For today's integration, suppose our fictitious software company "Progix" has a security module that automatically takes photos of people who visit their customers' rocket launch sites. Progix has been asked to extend their security module so that timestamps are automatically added to these security photos, and the timestamped photos are placed into their customers' Dropbox accounts. Our integration will be pretty simple, involving three steps:

1. First, we will invoke our integration by its webhook URL whenever a visitor's photo is taken. Our webhook payload will include the photo that we want to save.
2. Second, we'll "process" the image by printing a timestamp on the top of the photo so there's a visible record of when the visitor entered the launch site. We'll do this with a custom component using the JIMP (JavaScript Image Manipulation Program) library.
3. Finally, we'll write the resulting image with timestamp to a folder in one of our customers' Dropbox accounts.

Uploading a Binary File as Part of a Webhook Invocation

We can invoke instances of integrations one of two ways: we can either set up our instances to run on a schedule (e.g. "every 15 minutes"), or we can invoke instances from any third party client through the instances' webhook URLs. The latter option is handy if we want to run instances irregularly, like every time a visitor enters a rocket launch site, or if we want to invoke our integration with some data (say, an image file or some JSON payload). We can configure Progix's security module to call out to a Prismatic webhook URL whenever a visitor enters a rocket launch site.

We can pass in structured data to an integration via webhook in JSON format. Here's an example of passing in data as part of a POST request using curl:

curl --data '{"foo": "bar", "baz": "buz"}' \
 --request POST \
 --header "Content-Type: application/json" \
 https://hooks.prismatic.io/trigger/EXAMPLEabc123

If we want to pass in a binary file instead, we can use the --data-binary flag to upload a local file:

curl --data-binary "@/path/to/my-image.png" \
 --request POST \
 --header "Content-Type: image/png" \
 https://hooks.prismatic.io/trigger/EXAMPLEabc123

Note that we're using curl as an example here: you can invoke an integration with whatever HTTP request library for whatever language you like. Python, NodeJS, C#, etc., all support passing a binary payload to an HTTP endpoint.

Once we post our image we can see within our trigger that binary data was received, which can now be used in subsequent steps by referencing the trigger step's results:

Accessing Binary Data in a Custom Component

Now that we have binary data being passed into our integration, let's write a custom component that takes that binary data as input and does something with it. In our case, we'll take in an image of a rocket launch site visitor and add a timestamp to the top of the image.

The full code for our component is available on GitHub. Let's focus on a couple of pieces:

const imageInput = input({
    key: "image",
    label: "Image Data",
    type: "data",
    required: true,
    comments: "A data buffer for an open image file",
});

The one and only input for our custom component is of type data (as opposed to string, like we've used previously). When we use the data type, the component will be passed a parameter that is a JavaScript object in the form:

{
  "data": Buffer,
  "contentType": String
}

The data property is a file buffer, and contentType reveals what type of file it is (like image/png). Most any library that handles binary files (like AWS, Azure, Dropbox, etc) handles Buffer parameters. The image manipulation library that we're going to use, JIMP, does, too.

The code for our action will be relatively short:

export const timestampAction = action({
    key: "timestampImage",
    display: {
        label: "Timestamp Image",
        description: "Add a timestamp to an image",
    },
    perform: async (context, { image }) => {
        assert(image.contentType.startsWith("image/"));
        const jimpInstance = await Jimp.read(image.data);
        const font = await Jimp.loadFont(fontUrl);
        const timestamp = new Date();
        jimpInstance.print(font, 0, 0, timestamp.toISOString());
        const imageWithTimestamp = await jimpInstance.getBufferAsync(
            Jimp.MIME_PNG
        );
        return {
            data: imageWithTimestamp,
            contentType: Jimp.MIME_PNG,
        };
    },
    inputs: [imageInput],
});

Note that the perform function takes an image parameter (which is one of those data / contentType objects). We then assert() that the file passed in is some sort of image by looking at image.contentType. After that we can grab our file's contents (image.data) and throw it into a JIMP constructor so we can start manipulating the image.

From there, we pull down a font, write some text on top of our image, and get ready to write the image back out as a file Buffer:

const font = await Jimp.loadFont(fontUrl);
const timestamp = new Date();
jimpInstance.print(font, 0, 0, timestamp.toISOString());
const imageWithTimestamp = await jimpInstance.getBufferAsync(Jimp.MIME_PNG);

Let's take a moment to look at the return value of our component. It should have a familiar form:

return {
    data: imageWithTimestamp,
    contentType: Jimp.MIME_PNG,
};

We're returning an image from our component, so it makes sense that our return value takes the same form as our input object. We return an object that contains a data property of type Buffer, and a contentType property that will read "image/png", since we're returning a PNG. If we run a test of our integration so far, we see that the image is being properly rendered after timestamp step:

Passing Data Into File Storage Provider Components

At this point, we've uploaded a photo of a person to our integration and fed the image through a custom component. Now it's time to upload the resulting photo with timestamp to a customer's Dropbox account. We can see that the Add Timestamp to Image step yielded a binary file (specifically, a PNG). We can use that output with a Dropbox - Upload File step by referencing the previous step's results:

Running our integration once again all the way through, we can see that our binary image file (with timestamp!) ended up in our Dropbox account as we'd expect it to:

Now all we need to do is configure and deploy instances of our integration to customers. Then, they'll be able to upload photos of visitors to their unique instance webhook URLs and have timestamped photos saved to their security teams' Dropbox accounts.

Further Reading

Today we wrote an integration that accepted and manipulated image files and saved the processed files to Dropbox. Prismatic integrations aren't limited to images, though. Anything you can do in JavaScript, you can do in a Prismatic integration with custom components. A quick search of "PDF" on npmjs.com yielded 2700+ packages related to PDF manipulation, and there are dozens (if not, hundreds) of video and audio manipulation libraries written in JavaScript.

For additional information on writing custom components, check out our docs article. Please reach out to us if you'd like to chat about integrations - we'd love to discuss your integration scenario and what we can do to help!