gke-ai-troubleshooting-tpu-dynamic-slices-monitoring

GKE TPU Dynamic Slices Monitoring & Management

Monitors the status of TPU Slice custom resources, troubleshoots provisioning

failures, validates workload manifests on dynamic slices, and performs cleanups.

Prerequisites

• Cloud Logging enabled for the project.
• kubectl and gcloud CLIs configured to access the GKE cluster.

Diagnostic Workflow

Step 0: Context Acquisition & Time Window Definition

Gather project, cluster, and slice context using cluster tools or the following

parameters:

• Project ID: {project_id} (e.g., my-gcp-project)
• Cluster Name: {cluster_name} (e.g., tpu-cluster)
• Region/Zone: {location} (e.g., us-central1-a)
• Slice Name: {slice_name} (e.g., test-slice)
• Issue Time: {timestamp} (Optional; default to the last 30 minutes

window [T - 30m] to [T + 30m])

--------------------------------------------------------------------------------

Step 1: Describe the Slice Custom Resource [Low Risk]

When asked to inspect, troubleshoot, or check a slice status, immediately execute kubectl describe slice {slice_name} using available cluster tools to perform the inspection. Parse the resulting Status.Conditions output against the condition table below to diagnose the exact state and provide concrete recommendations.

• Command:

    kubectl describe slice {slice_name}

State & Reason Analysis

Analyze the Status.Conditions (especially Type: Ready and its Reason and

Status):

| Lifecycle State / Reason | Meaning | Recommended Action |

| :--- | :--- | :--- |

| SliceNotCreated | GKE Slice Controller is initializing the slice and performing resource checks. | Wait a few minutes and re-check slice status. |

| SliceCreationFailed | Prerequisites validation failed (e.g., selected nodes don't exist, nodes are already used by another slice, or the topology doesn't match the number of partitions). | Verify selected nodes exist, are unallocated, and topology matches partition count. |

| ACTIVATING | GKE is actively forming and provisioning the TPU slice. | Monitor node provisioning. |

| ACTIVE | The TPU slice is successfully formed and ready to host workloads. | Proceed to deploy or check workloads. |

| ACTIVE_DEGRADED | The slice is usable, but one or more sub-blocks are degraded. | Monitor workload logs for interconnect or device errors. Check faulty node VMs. |

| FAILED | GKE failed to form the TPU slice (e.g., selected nodes are not part of the same reservation block). | Ensure all selected nodes belong to the same reservation block. |

| DEACTIVATING | The slice is dismantling (triggered by user deletion or a critical systemic failure). | Wait for dismantling to finish, or patch finalizers if stuck. |

| INCOMPLETE | The terminal phase before the Slice CR is deleted from the cluster. | No action required; the resource will be removed shortly. |

Provisioning Failure Troubleshooting Checklist

When investigating slice creation or provisioning failures (SliceCreationFailed or FAILED), perform the following verification steps:

1. Node Existence & Allocation Check: Verify that the selected TPU nodes exist in the cluster and are not already allocated to another slice (kubectl get nodes -l cloud.google.com/gke-tpu-slice, kubectl get slice -A).
2. Topology Alignment: Confirm that the partition count matches the requested topology dimensions (e.g. topology 2x2 requires 4 nodes).
3. Reservation Block Alignment Check: Confirm that all selected TPU nodes belong to the same reservation and reservation block.

--------------------------------------------------------------------------------

Step 2: Verify Workload Specification [Low Risk]

Ensure workload manifests are configured correctly to target the dynamic slice.

1. Single-Slice Workload Requirements

Check that the Pod template contains the following annotations and selectors:

• Annotations:
• cloud.google.com/gke-tpu-slice-topology: "{topology}" (e.g.,

"4x4x4")

• NodeSelector:
• cloud.google.com/gke-tpu-topology: "{topology}" (e.g., "4x4x4")
• cloud.google.com/gke-tpu-accelerator: "{accelerator_type}" (e.g.,

"tpu7x")

• cloud.google.com/gke-tpu-slice: "{slice_name}" (e.g., "test-slice")

2. Multi-Slice (JobSet) Workload Requirements

If deploying a multi-slice JobSet, verify:

• JobSet Annotation:
• `alpha.jobset.sigs.k8s.io/exclusive-topology:

cloud.google.com/gke-tpu-slice`

• Pod Template Annotations:
• cloud.google.com/gke-tpu-slice-topology: "{topology}"
• Pod Template NodeSelector:
• cloud.google.com/gke-tpu-topology: "{topology}"
• cloud.google.com/gke-tpu-accelerator: "{accelerator_type}"
• *Note: Do NOT manually specify cloud.google.com/gke-tpu-slice in the

nodeSelector; JobSet handles slice assignment automatically.*

--------------------------------------------------------------------------------

Resolution & Management Workflow

Resolution 1: Force Delete a Stuck Slice [High Risk]

If a slice is stuck in DEACTIVATING or deletion hangs indefinitely due to stuck finalizers:

1. Identify Cause: Explain that finalizers on the slice resource (metadata.finalizers) are preventing Kubernetes from completing resource deletion.
2. Propose Resolution: Propose removing finalizers from the metadata path (/metadata/finalizers) using a JSON patch operation:

    kubectl patch slice {slice_name} --type json -p='[{"op": "remove", "path": "/metadata/finalizers"}]'

1. Provide Warning: Explicitly warn the user that removing finalizers bypasses standard controller dismantling and may leave underlying VM, network, or accelerator resources uncleaned or orphaned.
2. CRITICAL SAFETY MANDATE: The response MUST explicitly ask the user for confirmation (e.g. "Removing finalizers on /metadata/finalizers via JSON patch is a high-risk operation that may leave orphaned resources. Do you confirm you want to apply this patch to slice {slice_name}?") and pause for user confirmation before applying or executing the patch.

--------------------------------------------------------------------------------

Resolution 2: Disable and Clean Up Slice Controller [High Risk]

If dynamic slicing needs to be disabled:

1. Check for existing Slices:

    kubectl get slice -A

Ensure all slices are deleted before disabling the controller.

1. Disable Slice Controller via gcloud:

    gcloud container clusters update {cluster_name} \
        --location={location} \
        --no-enable-slice-controller

1. Delete the Slice CRD:

    kubectl delete crd slices.accelerator.gke.io

1. Clean up Node Labels: Remove GKE TPU Slice labels from all nodes in the

cluster:

    kubectl label nodes --all cloud.google.com/gke-tpu-slice- cloud.google.com/gke-tpu-slice-topology-

• Safety Rule: Propose the exact commands and confirm before executing

disabling or destructive cleanup steps.