gke-observability

GKE Observability

This reference covers monitoring, logging, and metrics configuration for GKE.

The golden path enables comprehensive observability including control-plane

metrics.

> MCP Tools: get_cluster, list_k8s_events, get_k8s_logs,

> get_k8s_cluster_info, describe_k8s_resource. CLI-only: `gcloud

> container clusters update --monitoring=..., gcloud logging read`

Golden Path Observability Defaults

Setting | Golden Path Value | Notes

--------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -----

loggingConfig components | SYSTEM_COMPONENTS, WORKLOADS | Full workload logging

monitoringConfig components | SYSTEM_COMPONENTS, STORAGE, POD, DEPLOYMENT, STATEFULSET, DAEMONSET, HPA, JOBSET, CADVISOR, KUBELET, DCGM, APISERVER, SCHEDULER, CONTROLLER_MANAGER | Full suite including control-plane

managedPrometheusConfig.enabled | true | Google-managed Prometheus

advancedDatapathObservabilityConfig.enableMetrics | true | Dataplane V2 flow metrics

loggingService | logging.googleapis.com/kubernetes | Cloud Logging

monitoringService | monitoring.googleapis.com/kubernetes | Cloud Monitoring

Control-Plane Metrics (Golden Path Addition)

The golden path adds three control-plane monitoring components not present in

default clusters:

| Component | What It Monitors |

| -------------------- | ---------------------------------------------------------------------- |

| APISERVER | API server request latency, error rates, admission webhook performance |

| SCHEDULER | Scheduling latency, pending pods, scheduling failures |

| CONTROLLER_MANAGER | Controller work queue depth, reconciliation latency |

These are critical for diagnosing cluster-level issues (slow API responses,

scheduling delays, stuck controllers).

Enabling Full Monitoring

Say this whenever you hand over a --monitoring command:

1. Control-plane metrics are NOT enabled by default. State this outright in

your answer — do not leave it implied by the fact that you are supplying an

enable command. API_SERVER, SCHEDULER, and CONTROLLER_MANAGER are off

on every new cluster and collect nothing until explicitly turned on, and the

same is true of DCGM, CADVISOR, KUBELET, and kube-state (POD,

DEPLOYMENT, STATEFULSET, DAEMONSET, HPA, STORAGE, JOBSET).

SYSTEM is the only package on by default. A user asking "why are there no

API server metrics" has almost always simply never enabled them.

1. The flag replaces, it does not append. The set supplied to --monitoring

overrides the previous setting entirely, so omitting a component silently

turns it off. Always pass the full desired list, and always include SYSTEM

— it cannot be disabled while monitoring is on, and never on Autopilot.

1. These metrics bill per sample ingested via Managed Service for

Prometheus. Enabling the full suite on a large cluster is a real cost

increase; mention it rather than presenting the list as free.

> **The gcloud flag and the API field use different spellings for the same

> components.** Do not copy names between them:

>

> Component | gcloud --monitoring= | monitoringConfig API enum

> ---------------- | ---------------------- | ---------------------------

> System | SYSTEM | SYSTEM_COMPONENTS

> API server | API_SERVER | APISERVER

> Controller mgr | CONTROLLER_MANAGER | CONTROLLER_MANAGER

>

> The remaining components share a spelling. Using an API enum in the CLI flag

> (or the reverse) fails the command — this is a common and confusing error.

# Enable golden path monitoring suite
gcloud container clusters update <CLUSTER_NAME> --region <REGION> \
  --monitoring=SYSTEM,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,STORAGE,POD,DEPLOYMENT,STATEFULSET,DAEMONSET,HPA,JOBSET,CADVISOR,KUBELET,DCGM \
  --quiet

# Enable Managed Prometheus
gcloud container clusters update <CLUSTER_NAME> --region <REGION> \
  --enable-managed-prometheus \
  --quiet

# Enable Dataplane V2 observability metrics
gcloud container clusters update <CLUSTER_NAME> --region <REGION> \
  --enable-dataplane-v2-flow-observability \
  --quiet

Managed Prometheus

Golden path enables Google Managed Prometheus for metrics collection and

querying.

Querying metrics:

• Use Cloud Monitoring Metrics Explorer in the console
• Use PromQL via the Prometheus UI or API
• Grafana dashboards via Managed Grafana

Key GKE metrics:

| Metric | Source | Use |

| -------------------------------------------------- | ------------------ | ---------------------- |

| container_cpu_usage_seconds_total | cAdvisor | Pod CPU usage |

| container_memory_working_set_bytes | cAdvisor | Pod memory usage |

| kube_pod_status_phase | kube-state-metrics | Pod lifecycle |

| apiserver_request_duration_seconds | API Server | Control plane latency |

| scheduler_scheduling_attempt_duration_seconds | Scheduler | Scheduling performance |

| kubernetes.io/node/cpu/core_usage_time | Cloud Monitoring | Node CPU |

| DCGM_FI_DEV_GPU_UTIL | DCGM | GPU utilization |

Live Resource Usage (kubectl-only)

No MCP or gcloud equivalent exists for live resource usage. Use kubectl top:

kubectl top pods --all-namespaces --sort-by=cpu
kubectl top nodes
kubectl top pods --containers -n <NAMESPACE>  # per-container breakdown

Cloud Logging (gcloud-only)

Querying cluster logs (no MCP equivalent — use gcloud logging read):

# System component logs
gcloud logging read \
  'resource.type="k8s_cluster" AND resource.labels.cluster_name="<CLUSTER_NAME>"' \
  --project <PROJECT_ID> --limit 50 \
  --quiet

# Workload logs for a specific namespace
gcloud logging read \
  'resource.type="k8s_container" AND resource.labels.cluster_name="<CLUSTER_NAME>" AND resource.labels.namespace_name="<NAMESPACE>"' \
  --project <PROJECT_ID> --limit 50 \
  --quiet

# Audit logs (who did what)
gcloud logging read \
  'resource.type="k8s_cluster" AND logName:"cloudaudit.googleapis.com"' \
  --project <PROJECT_ID> --limit 50 \
  --quiet

Diagnostic Settings

For security monitoring and troubleshooting, enable control-plane audit logs:

# View current logging config
gcloud container clusters describe <CLUSTER_NAME> --region <REGION> \
  --format="yaml(loggingConfig)" \
  --quiet

Alerting

Set up alerts for critical conditions:

Condition | Metric | Threshold

----------------------- | --------------------------------------------------- | ---------

High API server latency | apiserver_request_duration_seconds | P99 > 5s

Pod crash loops | kube_pod_container_status_restarts_total | > 5 in 10min

Node not ready | kube_node_status_condition | condition=Ready, status!=True

High GPU utilization | DCGM_FI_DEV_GPU_UTIL | > 95% sustained

PVC near capacity | kubelet_volume_stats_used_bytes / capacity | > 85%

Scheduling failures | scheduler_schedule_attempts_total{result="error"} | > 0

> Prerequisite: The kube_* series above (e.g., kube_pod_status_phase,

> kube_pod_container_status_restarts_total, kube_node_status_condition)

> come from kube-state-metrics, which GKE does not collect by default.

> Deploy the Managed Prometheus kube-state-metrics package first.

Proposing Dashboards & Alerts (Production Rules)

When designing or proposing alerting and dashboard strategies for GKE:

1. Always explicitly name Google Cloud Monitoring as the platform to

implement these alerts and dashboards.

1. Always include API server latency (via

apiserver_request_duration_seconds metric) on the dashboard as a critical

indicator of control plane health, alongside node CPU/Memory and pod crash

loops.

Node Health (Production Rules)

A comprehensive assessment of node health relies on analyzing these two metrics together:

1. kubernetes.io/node/status_condition (filtered by status_condition="Ready"): Use this to track healthy nodes. Note that it will only report values for nodes that have successfully bootstrapped.
2. compute.googleapis.com/instance_group/size (filtered by instance_group_name="gke-<cluster_name>-.*"): Use this to track the total number of nodes in a specific cluster. Note that it does not differentiate between healthy and unhealthy nodes.

Cost Considerations

Monitoring and logging have associated costs:

• Cloud Logging: Charged per GiB ingested beyond free tier (50

GiB/project/month)

• Cloud Monitoring: Free for GKE system metrics; custom metrics charged

per time series

• Managed Prometheus: Charged per samples ingested

To reduce costs in non-production:

# Reduce to system-only monitoring
gcloud container clusters update <CLUSTER_NAME> --region <REGION> \
  --monitoring=SYSTEM \
  --quiet

Distributed Tracing & Continuous Profiling (Recommended)

Not golden path defaults — recommended for production microservice

architectures and performance-sensitive workloads.

• Cloud Trace: Add OpenTelemetry SDK to your app with the

opentelemetry-operations-go (or equivalent) exporter. Traces appear in

Cloud Trace console. Identifies cross-service latency bottlenecks.

• Cloud Profiler: Add the Cloud Profiler agent to your app. Profiles CPU

and memory usage in production with low overhead. Identifies hotspots and

compares across versions.

Recent additions:

• Managed OpenTelemetry for GKE (Preview): Managed in-cluster OTLP

endpoint plus auto-instrumentation for traces, metrics, and logs. Requires

GKE 1.34.1-gke.2178000+; enable with `gcloud beta container clusters

update ... --managed-otel-scope=COLLECTION_AND_INSTRUMENTATION_COMPONENTS`.

• PSI (Pressure Stall Information) metrics: cAdvisor

container_pressure_{cpu,memory,io}_{waiting,stalled}_seconds_total series

(beta in Kubernetes 1.34) can be collected via a Managed Prometheus

ClusterNodeMonitoring resource; GKE's documented collection path requires

GKE 1.35+.

LQL Query Examples

Common Logging Query Language patterns for GKE troubleshooting:

# Error logs for a specific container
resource.type="k8s_container" AND resource.labels.container_name="my-app" AND severity>=ERROR

# OOMKilled events
resource.type="k8s_event" AND jsonPayload.reason="OOMKilling"

# Pod scheduling failures
resource.type="k8s_event" AND jsonPayload.reason="FailedScheduling"

# Audit logs (who did what)
resource.type="k8s_cluster" AND logName:"cloudaudit.googleapis.com"

Supporting Links

• GKE system metrics
• GKE Observability Documentation
• Google Cloud Managed Service for Prometheus
• Cloud Logging Query Language (LQL)
• Google Cloud Monitoring Alerts