Skip to main content

AWS Glue Component

Manage AWS Glue crawlers, jobs and triggers

Component key: aws-glue

Description#

AWS Glue is a serverless data integration service from Amazon Web Services. The AWS Glue component allows you to interact with jobs, triggers, and crawlers in your AWS Glue account.

AWS Glue Connections#

AWS Glue Access Key and Secret#

An AWS IAM access key pair is required to interact with AWS Glue. Make sure that the key pair you generate in AWS has proper permissions to the AWS Glue resources you want to access. Read more about Glue IAM actions in the AWS docs.

InputNotesExample
Input
Access Key ID
string
/ Required
Notes
An AWS IAM Access Key ID
Example
AKIAIOSFODNN7EXAMPLE
Input
Secret Access Key
password
/ Required
Notes
An AWS IAM Secret Access Key
Example
wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Actions#

Get Job Run#

Retrieves the metadata for a given job run. | key: getJobRun

InputDefaultNotesExample
Input
Connection
connection
/ Required
Default
 
Notes
 
Example
 
Input
AWS Region
string
/ Required
Default
us-east-1
Notes
AWS provides services in multiple regions, like us-west-2 or eu-east-1. AWS region indicates the region in which your DynamoDb instance is hosted
Example
us-east-1
Input
Name
string
/ Required
Default
 
Notes
Provide a string value for the name (NOT the ARN).
Example
 
Input
Run Id
string
/ Required
Value List
Default
 
Notes
Provide a string value for the run Id.
Example
 

Output Example Payload#

{  "data": {    "JobRun": ""  }}

List Crawlers#

List Crawlers available in AWS Glue | key: listCrawlers

InputDefaultNotesExample
Input
Connection
connection
/ Required
Default
 
Notes
 
Example
 
Input
AWS Region
string
/ Required
Default
us-east-1
Notes
AWS provides services in multiple regions, like us-west-2 or eu-east-1. AWS region indicates the region in which your DynamoDb instance is hosted
Example
us-east-1
Input
Marker
string
Default
 
Notes
Specify the pagination token that's returned by a previous request to retrieve the next page of results
Example
lslTXFcbLQKkb0vP9Kgh5hy0Y0OnC7Z9ZPHPwPmMnxSk3eiDRMkct7D8E
Input
Max Items
string
Default
 
Notes
Provide an integer value for the maximum amount of items that will be returned. Provide a value from 1 to 50.
Example
20

Output Example Payload#

{  "data": {    "NextToken": "",    "CrawlerNames": [      "crawler-1",      "crawler-2"    ]  }}

List Jobs#

List job schemas available in AWS Glue | key: listJobs

InputDefaultNotesExample
Input
Connection
connection
/ Required
Default
 
Notes
 
Example
 
Input
AWS Region
string
/ Required
Default
us-east-1
Notes
AWS provides services in multiple regions, like us-west-2 or eu-east-1. AWS region indicates the region in which your DynamoDb instance is hosted
Example
us-east-1
Input
Marker
string
Default
 
Notes
Specify the pagination token that's returned by a previous request to retrieve the next page of results
Example
lslTXFcbLQKkb0vP9Kgh5hy0Y0OnC7Z9ZPHPwPmMnxSk3eiDRMkct7D8E
Input
Max Items
string
Default
 
Notes
Provide an integer value for the maximum amount of items that will be returned. Provide a value from 1 to 50.
Example
20

Output Example Payload#

{  "data": {    "JobNames": [      "job1",      "job2"    ],    "NextToken": ""  }}

List Triggers#

List the names of all triggers in the account. | key: listTriggers

InputDefaultNotesExample
Input
Connection
connection
/ Required
Default
 
Notes
 
Example
 
Input
AWS Region
string
/ Required
Default
us-east-1
Notes
AWS provides services in multiple regions, like us-west-2 or eu-east-1. AWS region indicates the region in which your DynamoDb instance is hosted
Example
us-east-1
Input
Marker
string
Default
 
Notes
Specify the pagination token that's returned by a previous request to retrieve the next page of results
Example
lslTXFcbLQKkb0vP9Kgh5hy0Y0OnC7Z9ZPHPwPmMnxSk3eiDRMkct7D8E
Input
Max Items
string
Default
 
Notes
Provide an integer value for the maximum amount of items that will be returned. Provide a value from 1 to 50.
Example
20

Output Example Payload#

{  "data": {    "NextToken": "",    "TriggerNames": [      "trigger-1",      "trigger-2"    ]  }}

Start Crawler#

Starts an existing crawler in AWS Glue. | key: startCrawler

InputDefaultNotesExample
Input
Connection
connection
/ Required
Default
 
Notes
 
Example
 
Input
AWS Region
string
/ Required
Default
us-east-1
Notes
AWS provides services in multiple regions, like us-west-2 or eu-east-1. AWS region indicates the region in which your DynamoDb instance is hosted
Example
us-east-1
Input
Name
string
/ Required
Default
 
Notes
Provide a string value for the name (NOT the ARN).
Example
 

Output Example Payload#

{  "data": {    "Name": "exampleCrawlerName"  }}

Start Job Run#

Starts a job run using a AWS Glue job definition. | key: startJobRun

InputDefaultNotesExample
Input
args
string
Key Value List
Default
 
Notes
Optional key value parameters to pass into a job.
Example
 
Input
Connection
connection
/ Required
Default
 
Notes
 
Example
 
Input
AWS Region
string
/ Required
Default
us-east-1
Notes
AWS provides services in multiple regions, like us-west-2 or eu-east-1. AWS region indicates the region in which your DynamoDb instance is hosted
Example
us-east-1
Input
Allocated Capacity
string
Default
 
Notes
The number of AWS Glue data processing units (DPUs) that can be allocated when this job runs. If this is omitted, Glue will use the default number of DPUs configured for your job.
Example
10
Input
Name
string
/ Required
Default
 
Notes
Provide a string value for the name (NOT the ARN).
Example
 
Input
Security Configuration
string
Default
 
Notes
The name of the SecurityConfiguration structure to be used with this job. This can be left blank if you do not have a security configuration.
Example
 

Output Example Payload#

{  "data": {    "Name": "exampleJobRunName"  }}

Start Trigger#

Starts an existing trigger in AWS Glue. | key: startTrigger

InputDefaultNotesExample
Input
Connection
connection
/ Required
Default
 
Notes
 
Example
 
Input
AWS Region
string
/ Required
Default
us-east-1
Notes
AWS provides services in multiple regions, like us-west-2 or eu-east-1. AWS region indicates the region in which your DynamoDb instance is hosted
Example
us-east-1
Input
Name
string
/ Required
Default
 
Notes
Provide a string value for the name (NOT the ARN).
Example
 

Output Example Payload#

{  "data": {    "Name": "exampleTriggerName"  }}

Stop Crawler#

If the specified crawler is running, stops the crawl | key: stopCrawler

InputDefaultNotesExample
Input
Connection
connection
/ Required
Default
 
Notes
 
Example
 
Input
AWS Region
string
/ Required
Default
us-east-1
Notes
AWS provides services in multiple regions, like us-west-2 or eu-east-1. AWS region indicates the region in which your DynamoDb instance is hosted
Example
us-east-1
Input
Name
string
/ Required
Default
 
Notes
Provide a string value for the name (NOT the ARN).
Example
 

Output Example Payload#

{  "data": {    "Name": "exampleCrawlerName"  }}

Stop Job Run#

Stops one or more job runs for a specified job definition | key: stopJobRun

InputDefaultNotesExample
Input
Connection
connection
/ Required
Default
 
Notes
 
Example
 
Input
AWS Region
string
/ Required
Default
us-east-1
Notes
AWS provides services in multiple regions, like us-west-2 or eu-east-1. AWS region indicates the region in which your DynamoDb instance is hosted
Example
us-east-1
Input
Job Run Ids
string
/ Required
Value List
Default
 
Notes
Provide a list of job run Ids
Example
 
Input
Name
string
/ Required
Default
 
Notes
Provide a string value for the name (NOT the ARN).
Example
 

Output Example Payload#

{  "data": {    "SuccessfulSubmissions": [      ""    ],    "Errors": [      ""    ]  }}

Stop trigger#

Stops a specified trigger | key: stopTrigger

InputDefaultNotesExample
Input
Connection
connection
/ Required
Default
 
Notes
 
Example
 
Input
AWS Region
string
/ Required
Default
us-east-1
Notes
AWS provides services in multiple regions, like us-west-2 or eu-east-1. AWS region indicates the region in which your DynamoDb instance is hosted
Example
us-east-1
Input
Name
string
/ Required
Default
 
Notes
Provide a string value for the name (NOT the ARN).
Example
 

Output Example Payload#

{  "data": {    "Name": "exampleTriggerName"  }}