Skip to main content

Streaming Logs to Google Cloud

Streaming logs to Google Cloud Logging requires a service account and mapping Prismatic's severity levels to Google's (WARNING vs warn). The recommended approach is to send logs to a Google Cloud Function, which then writes to Cloud Logging.

  1. In Google Cloud IAM, create a new service account. Grant it the Logging Admin role via the IAM dashboard.

  2. Create a Google Cloud Function and assign the service account. Add @google-cloud/logging as a dependency in package.json:

    {
    "dependencies": {
    "@google-cloud/functions-framework": "^3.0.0",
    "@google-cloud/logging": "11.2.0"
    }
    }

    Its index.js can read something like this (replace PROJECT_ID):

    const functions = require("@google-cloud/functions-framework");
    const { Logging } = require("@google-cloud/logging");

    const PROJECT_ID = "INSERT YOUR PROJECT ID HERE";
    const LOG_NAME = "prismatic";

    const SEVERITY_MAP = {
    debug: "DEBUG",
    info: "INFO",
    warn: "WARNING",
    error: "ERROR",
    };

    functions.http("helloHttp", async (req, res) => {
    // Creates a client
    const logging = new Logging({ projectId: PROJECT_ID });

    // Selects the log to write to
    const log = logging.log(LOG_NAME);

    const { timestamp, severity, message, ...rest } = req.body;

    // Labels must be strings
    const labels = Object.entries(rest).reduce(
    (acc, [key, value]) => ({
    [key]: value === null ? "" : `${value}`,
    ...acc,
    }),
    {},
    );

    // The metadata associated with the entry
    const metadata = {
    resource: { type: "global" },
    severity: SEVERITY_MAP[severity],
    timestamp,
    labels,
    };

    // Prepares a log entry
    const entry = log.entry(metadata, message);

    await log.write(entry);

    res.send({ success: true });
    });
  3. Deploy the function and note its URL (e.g., https://us-central1-your-project-12345.cloudfunctions.net/your-function).

  4. In Prismatic, configure an external log stream to point to your function's URL. Ensure the timestamp is sent in ISO format. Use the following payload template:

     {
    "message": {{ message }},
    "timestamp": {{ timestamp_iso }},
    "severity": {{ severity }},
    "instance": {{ instanceName }},
    "customer": {{ customerExternalId }},
    "integration": {{ integrationName }},
    "isTestExecution": {{ isTestExecution }},
    "flow": {{ flowName }},
    "step": {{ stepName }},
    "executionId": {{ executionId }},
    "instanceId": {{ instanceId }},
    "flowConfigId": {{ flowConfigId }},
    "integrationId": {{ integrationId }},
    "flowId": {{ flowId }},
    "executionErrorStepName": {{ executionErrorStepName }},
    "duration": {{ durationMS }},
    "succeeded": {{ succeeded }},
    "errorMessage": {{ errorMessage }},
    "retryAttempt": {{ retryAttemptNumber }},
    "retryForExecutionId": {{ retryForExecutionId }}
    }