Google Cloud Platform VPC Flow Logs
VPC Flow Logs gives you information on the IP traffic to and from network interfaces within your virtual private cloud (VPC). You can follow the steps in this document to collect VPC flow logs from Google Cloud Platform (GCP) and forward them to Site24x7's AppLogs for monitoring.
Prerequisites
The logged-in user should have owner-level permissions for the project. In other words, the user should have permission to:
- Create a Pub/Sub topic and set its permissions.
- Create and update a Log Router.
- Create a Dataflow job.
Enable VPC Flow Logs for an existing subnet
Follow the steps below to enable VPC Flow Logs for an existing subnet. Refer to Google's official documentation for more information.
- Go to the VPC networks page in the Google Cloud console.
- You can either select the subnet you want to update or select all, then click FLOW LOGS.
- Adjust the Aggregation Interval and Sample rate to manage the logs and data ingestion cost. For example, if you keep a 100% Sample rate with an Aggregation Interval of 5 SEC, then all the entries are kept, resulting in a higher data ingestion cost.
- Click SAVE.
GCP VPC flow log best practice
Aggregation interval
To troubleshoot network connectivity issues or detect security threats in real time, we recommend setting the aggregation interval to 5 seconds. If you only want to analyze network performance or optimize network costs, we recommend setting the aggregation interval to one minute, 5 minutes, or 10 minutes.
Sampling rate
Set the flow sampling rate to 100% (for all logs). This will ensure Site24x7 captures all network traffic, not just a sample.
Log forwarding from GCP
Follow the steps in this document to forward logs from the GCP. Make sure to configure the log filter, as mentioned below, when creating a log routing sink:
gcloud logging sinks create SINK_NAME pubsub.googleapis.com/projects/PROJECT_ID/topics/TOPIC_NAME --log-filter='resource.type="gce_subnetwork"'
Sample log
Below is the sample log syntax for VPC Flow Logs:
{
"insertId": "2s85kofd71z0y",
"jsonPayload": {
"reporter": "SRC",
"src_gke_details": {
"pod": {
"pod_name": "packageserver-df86dcdd-qlpnz",
"pod_namespace": "olm"
},
"cluster": {
"cluster_name": "redis-test",
"cluster_location": "us-central1-a"
},
"service": [
{
"service_name": "packageserver-service",
"service_namespace": "olm"
}
]
},
"src_instance": {
"zone": "us-central1-a",
"region": "us-central1",
"project_id": "zylker-a76a7ass",
"vm_name": "gke-redis-test-default-pool-2f152eb2-53hc"
},
"dest_vpc": {
"project_id": "zylker-a76a7ass",
"vpc_name": "default",
"subnetwork_name": "default"
},
"src_vpc": {
"vpc_name": "default",
"project_id": "zylker-a76a7ass",
"subnetwork_name": "default"
},
"dest_instance": {
"region": "us-central1",
"vm_name": "gke-redis-test-default-pool-2f152eb2-x642",
"project_id": "zylker-a76a7ass",
"zone": "us-central1-a"
},
"dest_gke_details": {
"pod": {
"pod_namespace": "kube-system",
"pod_name": "konnectivity-agent-777f7f84d6-57fgj"
},
"cluster": {
"cluster_name": "redis-test",
"cluster_location": "us-central1-a"
}
},
"packets_sent": "8",
"end_time": "2023-10-11T05:25:47.962287597Z",
"bytes_sent": "1448",
"start_time": "2023-10-11T05:25:47.958517575Z",
"connection": {
"dest_ip": "10.10.0.10",
"protocol": 6,
"dest_port": 11111,
"src_ip": "10.10.0.10",
"src_port": 1111
}
},
"resource": {
"type": "gce_subnetwork",
"labels": {
"subnetwork_name": "default",
"project_id": "zylker-a76a7ass",
"location": "us-central1-a",
"subnetwork_id": "12345678901"
}
},
"timestamp": "2023-10-11T05:25:52.288729877Z",
"logName": "projects/zylker-a76a7ass
logs/compute.googleapis.com%2Fvpc_flows",
"
receiveTimestamp": "2023-10-11T05:25:52.288729877Z"
}
VPC Flow Logs dashboard
Here's a list of the widgets available on the GCP VPC Flow Logs dashboard:
- Total Bytes Transferred
- Average Bytes Transferred
- TotalPackets Sent
- Average Packets Transferred
- Maximum Latency
- Average Latency
- Top Source VMs by Traffic
- Source Address Locations
- Total BytesSent from SourceIP
- Traffic by Subnetwork
- VPC Flows per Protocol by Hour
- Packets Sent Over Time
- Bytes transfers by source and destination IP addresses
- Average Latency of Destination over time
- Destination Address Locations
- Top External Destination Ports by VPC Flows
- Top External IPs by VPC Flows
- Top Destination IPs by Traffic
