Building the S3 Component

The goal of this quickstart is to examine the Amazon S3 component to get you accustomed to developing more advanced components that have multiple actions, handle credentials through connections, and handle non-string binary data input fields.

Amazon S3 component source code is referenced throughout this quickstart.

Prerequisites

You should walk through the component Getting Started Guide and Writing Custom Components prior to this quickstart to get a sense of project set up, and the action, input, connection and component objects that make up a component project.

Reusable inputs

input definitions can be written in an action code block. For example, the Slack component is comprised of a single action, and inputs are defined within that action.

slack/src/index.ts
action({
  /*...*/
  inputs: {
    message: {
      label: "Message",
      placeholder: "Message to send",
      type: "string",
      required: true,
      comments: "The message to send to a Slack channel.",
      example: "Hello World",
    },
    /*...*/
  },
});

However, for sufficiently complex components, you will likely have multiple actions that rely on the same input. The S3 component, for example, has multiple actions that take the name of an S3 bucket as input. Defining the input field once, and referencing it by multiple actions results in cleaner, more concise code.

inputs.ts
import { input } from "@prismatic-io/spectral";
export const bucket = input({
  label: "Bucket Name",
  placeholder: "Name of an S3 Bucket",
  type: "string",
  required: true,
  comments:
    "An Amazon S3 'bucket' is a container where files are stored. You can create a bucket from within the AWS console. Bucket names contain only letters, numbers, and dashes.",
  example: "my-s3-bucket-abc123",
});

actions.ts
import { bucket } from "./inputs";
import { action } from "@prismatic-io/spectral";

const deleteObject = action({
  inputs: { awsRegion, accessKey: accessKeyInput, bucket, objectKey },
});

const getObject = action({
  inputs: { awsRegion, accessKey: accessKeyInput, bucket, objectKey },
});

const listObjects = action({
  inputs: { awsRegion, accessKey: accessKeyInput, bucket, prefix },
});

Reusable helper functions

S3 actions require an Amazon S3 client, imported from the AWS SDK. Rather than wrapping the client creation logic into each action, authorization and client creation are placed in a distinct file, auth.ts, and referenced by the actions.

auth.ts
import { S3, STS } from "aws-sdk";
import { Connection, ConnectionError, util } from "@prismatic-io/spectral";

export const createS3Client = async (accessKey: Connection, region: string) => {
  const credentials: STS.Types.ClientConfiguration = {
    accessKeyId: util.types.toString(accessKey.fields.accessKeyId).trim(),
    secretAccessKey: util.types
      .toString(accessKey.fields.secretAccessKey)
      .trim(),
    region,
  };

  // Verify credentials are valid with STS.getCallerIdentity()
  const sts = new STS(credentials);
  try {
    await sts.getCallerIdentity({}).promise();
  } catch (err) {
    throw new ConnectionError(
      accessKey,
      `Invalid AWS Credentials have been configured. This is sometimes caused by missing characters from a copy/paste. Original AWS error message: ${
        (err as Error).message
      }`
    );
  }

  return new S3(credentials);
};

actions.ts
import { createS3Client } from "./auth";

const listObjects = action({
  perform: async (context, { awsRegion, accessKey, bucket, prefix }) => {
    const s3 = createS3Client(accessKey, util.types.toString(awsRegion));
  },
});

const getObject = action({
  perform: async (context, { awsRegion, accessKey, bucket, objectKey }) => {
    const s3 = createS3Client(accessKey, util.types.toString(awsRegion));
  },
});

Handling authentication

To make Amazon S3 API calls, an access_key_id and secret_access_key key pair are presented to AWS's API. So, we'll create a connection that has those two fields:

import { connection } from "@prismatic-io/spectral";

export const accessKeySecretPair = connection({
  key: "apiKeySecret",
  display: {
    label: "AWS API Key and Secret",
    description: "AWS API Key and Secret",
  },
  inputs: {
    accessKeyId: {
      label: "Access Key ID",
      placeholder: "Access Key ID",
      type: "string",
      required: true,
      shown: true,
      comments: "An AWS IAM Access Key ID",
      example: "AKIAIOSFODNN7EXAMPLE",
    },
    secretAccessKey: {
      label: "Secret Access Key",
      placeholder: "Secret Access Key",
      type: "password",
      required: true,
      shown: true,
      comments: "An AWS IAM Secret Access Key",
      example: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    },
  },
});

export default [accessKeySecretPair];

Then, we'll let our component know that it should support the accessKeySecretPair connection:

index.ts
import { component } from "@prismatic-io/spectral";
import { actions } from "./actions";
import connections from "./connections";

export default component({
  key: "aws-s3",
  documentationUrl: "https://prismatic.io/docs/components/aws-s3/",
  public: true,
  display: {
    label: "Amazon S3",
    description: "Manage files within an Amazon (AWS) S3 bucket",
    iconPath: "icon.png",
    category: "Data Platforms",
  },
  connections,
  actions,
});

This connection will now be available to any action that takes an input of type connection (which is all of our actions).

Binary data

For most components, passing text or JSON as inputs and outputs is sufficient. The Slack component, for example, will only ever post string messages.

The Amazon S3 component, on the other hand, might fetch or write non-string data, like images, videos, or PDFs.

Binary data as outputs

The S3 getObject action reads a file (text or binary data) from an S3 bucket and returns its contents as an action output.

actions.ts
/*
  Implementation of the S3 GetObject API
  https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
*/
const getObject = action({
  display: {
    label: "Get Object",
    description: "Get the contents of an object",
  },
  perform: async (context, { awsRegion, accessKey, bucket, objectKey }) => {
    const s3 = await createS3Client(accessKey, util.types.toString(awsRegion));
    const getObjectParameters = {
      Bucket: util.types.toString(bucket),
      Key: util.types.toString(objectKey),
    };
    const response = await s3.getObject(getObjectParameters).promise();
    return {
      data: response.Body as Buffer,
      contentType: response.ContentType,
    };
  },
  inputs: { awsRegion, accessKey: accessKeyInput, bucket, objectKey },
  examplePayload: {
    data: Buffer.from("Example"),
    contentType: "application/octet",
  },
});

note

Note that in addition to the file's contents, the return contains contentType. That might be image/png, etc., depending on the type of the file.

Binary data as inputs

The S3 putObject action writes either plain text or a binary file to an S3 bucket. S3 extracts contentType from the file that is uploaded.

Our input is defined with type: "data" to indicate that it might contain binary data from a previous step.

inputs.ts
export const fileContents = input({
  label: "File Contents",
  placeholder: "Output data from previous step, or a string, to write",
  type: "data",
  required: true,
  comments:
    "The contents to write to a file. This can be a string of text, it can be binary data (like an image or PDF) that was generated in a previous step.",
  example: "My File Contents",
});

If plain text is entered in the fileContents input field, a string is passed into the perform function. If a reference to binary data is passed in (for example, from an HTTP GET action's output), fileContents will be an object with .data and .contentType properties.

The S3 putObject action takes advantage of Spectral's util.types.toData() function to turn fileContents into a type that the AWS SDK expects.

actions.ts
/*
  Implementation of the S3 PutObject API
  https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html
*/
const putObjectOutput: { data: S3.Types.PutObjectOutput } = {
  data: { ETag: "Example Tag", VersionId: "Example Version Id" },
};
const putObject = action({
  display: {
    label: "Put Object",
    description: "Write an object to S3",
  },
  perform: async (
    context,
    { awsRegion, accessKey, bucket, fileContents, objectKey, tagging }
  ) => {
    const s3 = await createS3Client(accessKey, util.types.toString(awsRegion));
    const { data, contentType } = util.types.toData(fileContents);
    const tags = querystring.encode(
      (tagging || []).reduce(
        (acc, { key, value }) => ({ ...acc, [key]: value }),
        {}
      )
    );
    const putParameters: S3.PutObjectRequest = {
      Bucket: util.types.toString(bucket),
      Key: util.types.toString(objectKey),
      Body: data,
      ContentType: contentType,
      Tagging: tags,
    };
    const response = await s3.putObject(putParameters).promise();
    return {
      data: response,
    };
  },
  inputs: {
    awsRegion,
    accessKey: accessKeyInput,
    bucket,
    fileContents,
    objectKey,
    tagging,
  },
  examplePayload: putObjectOutput,
});

Prerequisites​

Reusable inputs​

Reusable helper functions​

Handling authentication​

Binary data​

Binary data as outputs​

Binary data as inputs​