Handling Large Files

If your integration moves large files between your app and a partner app, you may run into a few runner limitations:

• The payload you can send to a webhook URL is limited to about 6MB
• Your webhook request must complete within 30 seconds
• Your flow can run for a maximum of 15 minutes
• The runner is allocated 1GB of RAM

A full list of runner limits can be found here. When sending large files through a flow, those files may exceed upload size and time limits, or may require more memory that is available (resulting in an out-of-memory error).

There are a few strategies you can use to handle large files in your Prismatic integration.

Upload files directly to a file storage system

If you and your partner app both use a file storage system like Amazon S3 or Dropbox, you can upload files directly to that system. This can be done using the file storage system's API, where you can request a temporary or presigned URL that allows you to upload a file directly from your app to the file storage system.

Upload files directly to Amazon S3

To upload a file directly to your customer's Amazon S3 bucket, you can use the Generate Presigned URL action from the Amazon S3 component.

If you'd like your flow to return a presigned upload URL whenever it is invoked:

1. Ensure that the trigger has a Response Type of Synchronous
2. Ensure that the Generate Presigned URL action is the last action of your flow.

If those things are true, when your app calls an instance's flow's webhook URL, it will receive a response with the results of the Generate Presigned URL action (the presigned URL as a string) as the body of the response. You can use the returned presigned URL to upload a file directly to Amazon S3 through an HTTP PUT request:

# Fetch the presigned URL from the webhook response and remove double-quotes
$ curl 'https://hooks.dev.prismatic-dev.io/trigger/SW5zdEXAMPLE==' --location | tr -d '"'

https://example-bucket.s3.us-west-2.amazonaws.com/my-file.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20240221%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240221T191715Z&X-Amz-Expires=3600&X-Amz-Signature=82a604673c3fffc2671b2dd7c7a86036af67693509ba0d01f172ef0b1f84fb20&X-Amz-SignedHeaders=host&x-id=PutObject

# Use the presigned URL to upload a file directly to Amazon S3
$ curl --request PUT \
    --upload-file ./my-example-file.png \
    https://example-bucket.s3.us-west-2.amazonaws.com/my-file.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20240221%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240221T191715Z&X-Amz-Expires=3600&X-Amz-Signature=82a604673c3fffc2671b2dd7c7a86036af67693509ba0d01f172ef0b1f84fb20&X-Amz-SignedHeaders=host&x-id=PutObject

Upload files directly to Dropbox

Similar to presigned URLs for Amazon S3, you can use the Generate Temporary Upload Link action from the Dropbox component to upload a file directly to Dropbox.

Provided that you have a synchronous trigger and the Generate Temporary Upload Link action is the last action in your flow, the presigned URL will be returned to the caller of your webhook.

# Invoke a flow that generates a presigned URL, and parse the URL from the JSON response
$ curl 'https://hooks.dev.prismatic-dev.io/trigger/SW5zdEXAMPLE==' --location | jq -r .result.link

https://content.dropboxapi.com/apitul/1/ExAmPlE

# Use the presigned URL to upload a file directly to Dropbox
$ curl --request POST \
    --upload-file ./my-example-file.png \
    --headers "content-type: application/octet-stream" \
     https://content.dropboxapi.com/apitul/1/ExAmPlE

Note that Dropbox expects a POST request (unlike Amazon S3, which expects a PUT request) and requires a content-type header with a value of application/octet-stream.

Query an instance's connection config variable and upload a file directly to a third-party

If you need to upload a file to a third-party service that does not support presigned upload URLs, but your customer provided connection information for the third-party service when they configured an instance of your integration, you can query the Prismatic API for the instance's connection config variables and use those credentials to upload the file directly to the third-party service.

To query an instance's config variables, you can query for an instance object's configVariables property - particularly their inputs.nodes.value fields:

Query instance connection config variables

query getAcmeConnectionInfo($myInstanceId:ID!) {
    instance(id: $myInstanceId) {
      configVariables {
        nodes {
          id
          requiredConfigVariable {
            key
          }
          meta
          inputs {
            nodes {
              name
              value
            }
          }
        }
      }
    }
  }

Query Variables

{
  "myInstanceId": "SW5zdGFuY2U6NDY2MGQ4MDgtYjUwZS00NDdhLThhZmQtOWU0NzAxMzJkZThk"
}

Try It Out ❯

This query will return a result like this:

{
  "data": {
    "instance": {
      "configVariables": {
        "nodes": [
          {
            "id": "SW5zdGFuY2VDb25maWdWYXJpYWJsZTo3MzkzYjU0YS1kNWMxLTQzYjEtOTM3ZS1iNTM0ZDhiYTA1NzA=",
            "requiredConfigVariable": {
              "key": "My String Config Variable"
            },
            "meta": null,
            "inputs": {
              "nodes": []
            }
          },
          {
            "id": "SW5zdGFuY2VDb25maWdWYXJpYWJsZTo1OTY1ZjEwMy0xNGIyLTRmZWItYmI4My1hZWI4NmViNmRhYmI=",
            "requiredConfigVariable": {
              "key": "Acme Inc Connection"
            },
            "meta": null,
            "inputs": {
              "nodes": [
                {
                  "name": "password",
                  "value": "my-pass"
                },
                {
                  "name": "username",
                  "value": "my-user"
                }
              ]
            }
          }
        ]
      }
    }
  }
}

From the result you can extract connection information (like a username and password or API key) and use those credentials to make an API request directly to the file storage system. If the file storage system uses OAuth 2.0, your customer's API key will be present in the config variable's meta property.

For more information on query the Prismatic API, see the Prismatic API documentation.

Instruct Dropbox to download a file from a URL

If you have a file that is publically available at a certain URL, you can use the Save from URL action from the Dropbox component to instruct Dropbox to download the file from the internet. Provide the URL where the file is located, and Dropbox will download the file and save it to your specified path in the user's Dropbox account.

Pass a reference to a file via webhook

If you have a file that is too large to upload directly via webhook, but is small enough that the 1GB of memory available to the runner can handle it, you can pass a reference to the file to the runner via a webhook request. This reference could be a URL where the file is located, or a unique identifier that the runner can use to download the file from a file storage system. Your flow can use the HTTP - GET action or a comperable FTP or SFTP action to download the file from the URL, or use the file storage system's API to download the file using the unique identifier, and can then process the file as if it were uploaded directly to the flow via webhook request.

Stream data using a custom action

If you need to load a file from one place, process its data, and store the file in another location, you can build a custom component and leverage NodeJS streams to process small portions of the file at a time. See Handling Large Files in Custom Components for some examples.