Skip to main content

AWS Glue Component

Perform data transformation through AWS Glue

Component key: aws-glue

Description#

AWS Glue is a serverless data integration service from Amazon Web Services. The AWS Glue component allows you to interact with jobs, triggers, and crawlers in your AWS Glue account.

Authorization Notes#

An AWS IAM access key pair is required to interact with AWS Glue. Make sure that the key pair you generate in AWS has proper permissions to the AWS Glue resources you want to access. Read more about Glue IAM actions in the AWS docs.

AWS Glue Connections#

AWS Glue Access Key and Secret#

InputKeyNotesExample

Access Key ID

string
/ Required
accessKeyIdAn AWS IAM Access Key IDAKIAIOSFODNN7EXAMPLE

Secret Access Key

string
/ Required
secretAccessKeyAn AWS IAM Secret Access KeywJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Actions#

Get Job Run#

Retrieves the metadata for a given job run. | key: getJobRun

InputKeyDefaultNotesExample

Connection Input

connection
/ Required
awsConnection

AWS Region

string
/ Required
awsRegionus-east-1AWS provides services in multiple regions, like us-west-2 or eu-east-1. AWS region indicates the region in which your DynamoDb instance is hostedus-east-1

Name

string
/ Required
nameProvide a string value for the name (NOT the ARN).

Run Id

string
/ Required
Value List
runIdProvide a string value for the run Id.

Output Example Payload#

{  "data": {    "JobRun": ""  }}

List Crawlers#

List Crawlers available in AWS Glue | key: listCrawlers

InputKeyDefaultNotesExample

Connection Input

connection
/ Required
awsConnection

AWS Region

string
/ Required
awsRegionus-east-1AWS provides services in multiple regions, like us-west-2 or eu-east-1. AWS region indicates the region in which your DynamoDb instance is hostedus-east-1

Marker

string
markerSpecify the pagination token that's returned by a previous request to retrieve the next page of resultslslTXFcbLQKkb0vP9Kgh5hy0Y0OnC7Z9ZPHPwPmMnxSk3eiDRMkct7D8E

Max Items

string
maxItemsProvide an integer value for the maximum amount of items that will be returned. Provide a value from 1 to 50.20

Output Example Payload#

{  "data": {    "NextToken": "",    "CrawlerNames": [      "crawler-1",      "crawler-2"    ]  }}

List Jobs#

List job schemas available in AWS Glue | key: listJobs

InputKeyDefaultNotesExample

Connection Input

connection
/ Required
awsConnection

AWS Region

string
/ Required
awsRegionus-east-1AWS provides services in multiple regions, like us-west-2 or eu-east-1. AWS region indicates the region in which your DynamoDb instance is hostedus-east-1

Marker

string
markerSpecify the pagination token that's returned by a previous request to retrieve the next page of resultslslTXFcbLQKkb0vP9Kgh5hy0Y0OnC7Z9ZPHPwPmMnxSk3eiDRMkct7D8E

Max Items

string
maxItemsProvide an integer value for the maximum amount of items that will be returned. Provide a value from 1 to 50.20

Output Example Payload#

{  "data": {    "JobNames": [      "job1",      "job2"    ],    "NextToken": ""  }}

List Triggers#

List the names of all triggers in the account | key: listTriggers

InputKeyDefaultNotesExample

Connection Input

connection
/ Required
awsConnection

AWS Region

string
/ Required
awsRegionus-east-1AWS provides services in multiple regions, like us-west-2 or eu-east-1. AWS region indicates the region in which your DynamoDb instance is hostedus-east-1

Marker

string
markerSpecify the pagination token that's returned by a previous request to retrieve the next page of resultslslTXFcbLQKkb0vP9Kgh5hy0Y0OnC7Z9ZPHPwPmMnxSk3eiDRMkct7D8E

Max Items

string
maxItemsProvide an integer value for the maximum amount of items that will be returned. Provide a value from 1 to 50.20

Output Example Payload#

{  "data": {    "NextToken": "",    "TriggerNames": [      "trigger-1",      "trigger-2"    ]  }}

Start Crawler#

Starts an existing crawler in AWS Glue. | key: startCrawler

InputKeyDefaultNotesExample

Connection Input

connection
/ Required
awsConnection

AWS Region

string
/ Required
awsRegionus-east-1AWS provides services in multiple regions, like us-west-2 or eu-east-1. AWS region indicates the region in which your DynamoDb instance is hostedus-east-1

Name

string
/ Required
nameProvide a string value for the name (NOT the ARN).

Output Example Payload#

{  "data": {    "Name": "exampleCrawlerName"  }}

Start Job Run#

Starts a job run using a AWS Glue job definition. | key: startJobRun

InputKeyDefaultNotesExample

args

string
Key Value List
argsOptional key value parameters to pass into a job.

Connection Input

connection
/ Required
awsConnection

AWS Region

string
/ Required
awsRegionus-east-1AWS provides services in multiple regions, like us-west-2 or eu-east-1. AWS region indicates the region in which your DynamoDb instance is hostedus-east-1

Allocated Capacity

string
capacityProvide an optional value that represents the amount of memory to allocate to the job.2056

Name

string
/ Required
nameProvide a string value for the name (NOT the ARN).

Security Configuration

string
securityProvide a string value that represents the security method for the job.

Output Example Payload#

{  "data": {    "Name": "exampleJobRunName"  }}

Start Trigger#

Starts an existing trigger in AWS Glue. | key: startTrigger

InputKeyDefaultNotesExample

Connection Input

connection
/ Required
awsConnection

AWS Region

string
/ Required
awsRegionus-east-1AWS provides services in multiple regions, like us-west-2 or eu-east-1. AWS region indicates the region in which your DynamoDb instance is hostedus-east-1

Name

string
/ Required
nameProvide a string value for the name (NOT the ARN).

Output Example Payload#

{  "data": {    "Name": "exampleTriggerName"  }}

Stop Crawler#

If the specified crawler is running, stops the crawl | key: stopCrawler

InputKeyDefaultNotesExample

Connection Input

connection
/ Required
awsConnection

AWS Region

string
/ Required
awsRegionus-east-1AWS provides services in multiple regions, like us-west-2 or eu-east-1. AWS region indicates the region in which your DynamoDb instance is hostedus-east-1

Name

string
/ Required
nameProvide a string value for the name (NOT the ARN).

Output Example Payload#

{  "data": {    "Name": "exampleCrawlerName"  }}

Stop Job Run#

Stops one or more job runs for a specified job definition | key: stopJobRun

InputKeyDefaultNotesExample

Connection Input

connection
/ Required
awsConnection

AWS Region

string
/ Required
awsRegionus-east-1AWS provides services in multiple regions, like us-west-2 or eu-east-1. AWS region indicates the region in which your DynamoDb instance is hostedus-east-1

Job Run Ids

string
/ Required
Value List
jobRunIdsProvide a list of job run Ids

Name

string
/ Required
nameProvide a string value for the name (NOT the ARN).

Output Example Payload#

{  "data": {    "SuccessfulSubmissions": [      ""    ],    "Errors": [      ""    ]  }}

Stop trigger#

Stops a specified trigger | key: stopTrigger

InputKeyDefaultNotesExample

Connection Input

connection
/ Required
awsConnection

AWS Region

string
/ Required
awsRegionus-east-1AWS provides services in multiple regions, like us-west-2 or eu-east-1. AWS region indicates the region in which your DynamoDb instance is hostedus-east-1

Name

string
/ Required
nameProvide a string value for the name (NOT the ARN).

Output Example Payload#

{  "data": {    "Name": "exampleTriggerName"  }}