Streaming Logs to Google Cloud
Streaming logs to Google Cloud Logging requires a service account and mapping Prismatic's severity levels to Google's (WARNING
vs warn
).
The recommended approach is to send logs to a Google Cloud Function, which then writes to Cloud Logging.
-
In Google Cloud IAM, create a new service account. Grant it the Logging Admin role via the IAM dashboard.
-
Create a Google Cloud Function and assign the service account. Add @google-cloud/logging as a dependency in
package.json
:{
"dependencies": {
"@google-cloud/functions-framework": "^3.0.0",
"@google-cloud/logging": "11.2.0"
}
}Its
index.js
can read something like this (replacePROJECT_ID
):const functions = require("@google-cloud/functions-framework");
const { Logging } = require("@google-cloud/logging");
const PROJECT_ID = "INSERT YOUR PROJECT ID HERE";
const LOG_NAME = "prismatic";
const SEVERITY_MAP = {
debug: "DEBUG",
info: "INFO",
warn: "WARNING",
error: "ERROR",
};
functions.http("helloHttp", async (req, res) => {
// Creates a client
const logging = new Logging({ projectId: PROJECT_ID });
// Selects the log to write to
const log = logging.log(LOG_NAME);
const { timestamp, severity, message, ...rest } = req.body;
// Labels must be strings
const labels = Object.entries(rest).reduce(
(acc, [key, value]) => ({
[key]: value === null ? "" : `${value}`,
...acc,
}),
{},
);
// The metadata associated with the entry
const metadata = {
resource: { type: "global" },
severity: SEVERITY_MAP[severity],
timestamp,
labels,
};
// Prepares a log entry
const entry = log.entry(metadata, message);
await log.write(entry);
res.send({ success: true });
}); -
Deploy the function and note its URL (e.g.,
https://us-central1-your-project-12345.cloudfunctions.net/your-function
). -
In Prismatic, configure an external log stream to point to your function's URL. Ensure the timestamp is sent in ISO format. Use the following payload template:
{
"message": {{ message }},
"timestamp": {{ timestamp_iso }},
"severity": {{ severity }},
"instance": {{ instanceName }},
"customer": {{ customerExternalId }},
"integration": {{ integrationName }},
"isTestExecution": {{ isTestExecution }},
"flow": {{ flowName }},
"step": {{ stepName }},
"executionId": {{ executionId }},
"instanceId": {{ instanceId }},
"flowConfigId": {{ flowConfigId }},
"integrationId": {{ integrationId }},
"flowId": {{ flowId }},
"executionErrorStepName": {{ executionErrorStepName }},
"duration": {{ durationMS }},
"succeeded": {{ succeeded }},
"errorMessage": {{ errorMessage }},
"retryAttempt": {{ retryAttemptNumber }},
"retryForExecutionId": {{ retryForExecutionId }}
}
