Log forwarding from Google Cloud Platform

Google Cloud Platform (GCP) logs provide a comprehensive view of your cloud environment. Forward logs from GCP to Site24x7 AppLogs to analyze and monitor log data from your GCP resources.

Log forwarding from GCP involves the following steps:

Google Cloud services, such as Google Compute Engine, Google Kubernetes Engine (GKE), Google App Engine, Google Cloud Functions, Google Cloud Pub/Sub, and other services write logs to Google Cloud Logging.
Configuring a log routing sink publishes log entries to a Google Cloud Pub/Sub topic.
Deploying our Dataflow flex template to the Google Cloud Platform creates a dataflow job that will read logs from the topic created and export processed log data to Site24x7.

Prerequisites

The logged-in user should have owner-level permissions for the project.

Steps to forward GCP logs

Follow the below steps to forward GCP logs to AppLogs:

Creating resources (a Pub/Sub Topic and a Log Routing Sink)
Running a Dataflow Flex Template in GCP

Creating resources

Follow the steps below to:

Create a Pub/Sub topic
Create a subscription for it
Create a log routing sink to forward logs
Enable dataflow project

Create a Pub/Sub topic:

Run the below command and replace TOPIC_NAME with the desired name for your Pub/Sub topic.

gcloud pubsub topics create TOPIC_NAME

Create a subscription:

Run the below command and replace SUBSCRIPTION_NAME with the desired name for your subscription and TOPIC_NAME with the name of the topic you created.

gcloud pubsub subscriptions create SUBSCRIPTION_NAME --topic=TOPIC_NAME

Create a log routing Sink:

Run the below command and replace SINK_NAME with the desired name, PROJECT_ID with your Google Cloud project ID, and TOPIC_NAME with the name of the topic you created.

gcloud logging sinks create SINK_NAME pubsub.googleapis.com/projects/PROJECT_ID/topics/TOPIC_NAME

Note

Configure log filters (--log-filter="LOG_FILTER") if you only want to forward specific types of logs, log levels, or logs from specific resources.
For example, for VPC Flow logs, you can configure log-filter as mentioned below:
gcloud logging sinks create SINK_NAME pubsub.googleapis.com/projects/PROJECT_ID/topics/TOPIC_NAME --log-filter='resource.type="gce_subnetwork"'

Enable dataflow project:

Run the below command and replace PROJECT_ID with the project name.

gcloud services enable dataflow --project PROJECT_ID

Running a Dataflow Flex Template in GCP

Execute the following command to run a dataflow flex template in GCP.

gcloud dataflow flex-template run "TEMPLATE_NAME" --project=PROJECT_ID --template-file-gcs-location=gs://site24x7_applog/dataflow/v1/flex/Cloud_PubSub_to_Site24x7 --region=REGION --parameters "inputSubscription=projects/PROJECT_ID/subscriptions/INPUT_SUBSCRIPTION_NAME,apiKey=API_KEY,site24x7Domain=site24x7.com"