Streaming Logs to Google Cloud

Writing logs to Google's Cloud Logging service requires a service account, and requires severity levels that are slightly different than Prismatic's severity levels (e.g. WARNING vs warn). The best way to stream Prismatic logs to Google's Cloud Logging service is to send the requests to a Google Cloud Function which will write the log.

1. Log in to service accounts and create a new service account. Then, visit the IAM dashboard and grant the service account the role of Logging Admin.

2. Create a new Google Cloud Function. Assign the service account you created to this cloud function. Its package.json should include a dependency on @google-cloud/logging

   {
     "dependencies": {
       "@google-cloud/functions-framework": "^3.0.0",
       "@google-cloud/logging": "11.2.0"
     }
   }

   Its index.js can read something like this (replace PROJECT_ID):

   const functions = require("@google-cloud/functions-framework");
   const { Logging } = require("@google-cloud/logging");
   
   const PROJECT_ID = "INSERT YOUR PROJECT ID HERE";
   const LOG_NAME = "prismatic";
   
   const SEVERITY_MAP = {
     debug: "DEBUG",
     info: "INFO",
     warn: "WARNING",
     error: "ERROR",
   };
   
   functions.http("helloHttp", async (req, res) => {
     // Creates a client
     const logging = new Logging({ projectId: PROJECT_ID });
   
     // Selects the log to write to
     const log = logging.log(LOG_NAME);
   
     const { timestamp, severity, message, ...rest } = req.body;
   
     // Labels must be strings
     const labels = Object.entries(rest).reduce(
       (acc, [key, value]) => ({
         [key]: value === null ? "" : `${value}`,
         ...acc,
       }),
       {},
     );
   
     // The metadata associated with the entry
     const metadata = {
       resource: { type: "global" },
       severity: SEVERITY_MAP[severity],
       timestamp,
       labels,
     };
   
     // Prepares a log entry
     const entry = log.entry(metadata, message);
   
     await log.write(entry);
   
     res.send({ success: true });
   });

3. Publish your cloud function. Take note of its URL. It'll be something like https://us-central1-your-project-12345.cloudfunctions.net/your-function

4. Log in to Prismatic and configure an external log stream to point to your function's URL. Be sure to send the timestamp in ISO format. Your Payload template can read

    {
      "message": {{ message }},
      "timestamp": {{ timestamp_iso }},
      "severity": {{ severity }},
      "instance": {{ instanceName }},
      "customer": {{ customerExternalId }},
      "integration": {{ integrationName }},
      "isTestExecution": {{ isTestExecution }},
      "flow": {{ flowName }},
      "step": {{ stepName }},
      "executionId": {{ executionId }},
      "instanceId": {{ instanceId }},
      "flowConfigId": {{ flowConfigId }},
      "integrationId": {{ integrationId }},
      "flowId": {{ flowId }},
      "executionErrorStepName": {{ executionErrorStepName }},
      "duration": {{ durationMS }},
      "succeeded": {{ succeeded }},
      "errorMessage": {{ errorMessage }},
      "retryAttempt": {{ retryAttemptNumber }},
      "retryForExecutionId": {{ retryForExecutionId }}
    }