Kubernetes Operations Manual

Fix the cluster and namespace first, then narrow problems along Pod → EndpointSlice → Service → Ingress. Status fields, probes, connectivity, and application correctness provide different evidence.

Task	Start here
Confirm the target	Context and resources
Pod not ready or restarting	Events and logs
Running but unreachable	Traffic and configuration
Rollout, rollback, debug	Verify changes

Pin context and namespace

# List contexts with cluster, user, and namespace
kubectl config get-contexts

The star marks the current choice; current-context prints only its name. Replace your-context and default below with verified values. Each command specifies them explicitly so sections do not depend on earlier shell variables.

# Inspect workloads, entry points, and claims in the namespace
kubectl --context your-context -n default get deploy,svc,ingress,pvc

These show configuration at different layers. Query Pods below for placement. For NotFound check names/scope; for Forbidden check authorization.

# Get Pod names, readiness, restarts, IPs, and nodes
kubectl --context your-context -n default get pods -o wide

-o wide adds IP and node columns. Obtain your-pod from this list and verify container name app in describe’s Containers section.

# Query cluster-scoped node state
kubectl --context your-context get nodes -o wide

Ready does not prove every workload is schedulable; check requests, taints, and affinity.

# Check Pod-read authorization for this identity and scope
kubectl --context your-context -n default auth can-i get pods

yes/no answers authorization for this verb/resource, not API availability or other operations.

Replace example context, namespace, and resource names with verified targets. Listing Secrets does not require exporting their contents. Impersonation via --as also requires authorization.

Pod status locates a stage, not a root cause

Read events and per-container state first

# Inspect scheduling events, probes, mounts, and container state
kubectl --context your-context -n default describe pod your-pod

Distinguish State from Last State and record exit codes, reasons, event times, and counts. Events expire; absence is not proof of no failure.

# Read the current container instance’s last 100 log lines
kubectl --context your-context -n default logs your-pod -c app --tail=100

-c selects a container, not a Pod. Specify it for multi-container Pods; add -f to follow, ending with Ctrl-C.

# Read logs from the previous terminated instance
kubectl --context your-context -n default logs your-pod -c app --previous --tail=100

Use this for restart loops. Missing prior instances or retained logs can produce an error; restarting again is not a remedy.

# Review namespace events ordered by creation time
kubectl --context your-context -n default get events --sort-by=.metadata.creationTimestamp

This is not a timeline of every repeated occurrence; correlate Last Seen and counts in describe. Use the symptom table below.

Symptom	Inspect
Pending / FailedScheduling	Events: requests, quota, affinity, taints, PVC binding; not only idle CPU
ImagePullBackOff	Image/tag, registry access, credentials and rate limits from the node
CrashLoopBackOff	Exit code, lastState, previous logs, startup/liveness probes; this is restart backoff
Running / 0/1 Ready	Readiness probes/gates and individual containers; Running is not ready to serve

Match configuration fields to traffic

Preparing the visual

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
    - name: http
      port: 80
      targetPort: 8080

This Service selects Pods labeled app: web in its namespace and targets port 8080. The application must actually listen on a reachable Pod address. containerPort does not open a port and is not required for an integer targetPort; a named targetPort must resolve to a matching named port.

Check selectors, endpoints, then ports

The YAML above is a complete Service example, not a Pod. If saved as service.yaml, validate/apply through the release workflow.

# Find Pods matching the Service selector
kubectl --context your-context -n default get pods -l app=web --show-labels

If empty, check labels and namespace before changing ports.

# Inspect web backend addresses, ports, and readiness
kubectl --context your-context -n default get endpointslice -l kubernetes.io/service-name=web -o yaml

Inspect endpoints.addresses, conditions.ready, and ports.port across all slices. Existing endpoints still require network-policy, CNI, and listener checks.

# Verify the Service selector, port, and targetPort
kubectl --context your-context -n default describe service web

Clients use Service port; the application listens at targetPort. Named target ports must resolve to a matching Pod port name.

Port-forward tests only its own path

# Open a local debugging port on the kubectl client
kubectl --context your-context -n default port-forward service/web 8080:80

Keep this terminal running; curl http://127.0.0.1:8080/ from another client terminal. Ctrl-C stops forwarding. Reconnect if the selected Pod ends.

# Test the Service path from the real client Pod
kubectl --context your-context -n default exec client-pod -c app -- curl -v http://web.default.svc.cluster.local/

Replace client-pod/app with the intended permitted client containing curl; adjust namespace and cluster domain. This tests DNS and Service access with that client identity. Test Ingress separately from outside.

Empty EndpointSlices suggest checking selectors and namespace; non-ready endpoints require readiness diagnosis. Port-forward uses the API path to a selected Pod and does not test ClusterIP routing, cluster DNS, or Ingress. Test Service DNS/ClusterIP from an appropriate cluster client, then inspect Ingress host/path/backend, controller, and external entry separately.

Verify scope for rollout and debugging

Review, validate against the API, then apply

# Compare the reviewed manifest with live API objects
kubectl --context your-context -n default diff -f manifest.yaml

manifest.yaml is the change file. Check embedded namespaces and cluster-scoped objects. Exit 0 means no difference, 1 means differences; handle other errors first.

# Ask the API to validate/default without persisting objects
kubectl --context your-context -n default apply --dry-run=server -f manifest.yaml

This calls the API/admission. Fix failures; success does not prove image pulls or application health.

# Persist the reviewed and validated declarations
kubectl --context your-context -n default apply -f manifest.yaml

Check created/configured/unchanged objects against the intended change before watching rollout.

# Wait for Deployment rollout completion or timeout
kubectl --context your-context -n default rollout status deployment/web --timeout=120s

On timeout inspect new ReplicaSets, events, and logs. On success test requests. A wait timeout does not automatically roll back.

Inspect the revision before rollback

# List available Deployment template revisions
kubectl --context your-context -n default rollout history deployment/web

Inspect a candidate with rollout history deployment/web --revision=2, replacing 2 with the verified revision.

# Restore the reviewed workload-template revision
kubectl --context your-context -n default rollout undo deployment/web --to-revision=2

Observe rollout status, Pods, and requests again. Database state, external systems, and independent ConfigMap contents are not restored.

A diff exit code of 1 normally means differences exist. Server dry-run contacts API/admission. Verify actual requests after rollout. rollout undo restores workload template history, not databases; rollout restart creates new Pods without changing image-tag contents.

Use exec when tools exist in the container

# Start an interactive shell in the chosen container
kubectl --context your-context -n default exec -it your-pod -c app -- sh

After -- comes the container command; -i keeps input and -t allocates a terminal. This starts another process, not a new main process. Exiting does not delete the Pod.

Choose an ephemeral container for minimal images

# Add an ephemeral debugging container to the existing Pod
kubectl --context your-context -n default debug -it your-pod --image=busybox:1.37 --target=app

This is an alternative to exec, subject to permissions/policy. Use a project-approved image. Its record remains in the Pod after exit; the ephemeral-container entry cannot be removed independently.

Specify file-copy direction and container

# Copy a diagnostic file to the client directory
kubectl --context your-context -n default cp your-pod:/tmp/report.txt ./report.txt -c app

Avoid overwriting a needed destination and verify content/size. Reversing source and destination writes to the container. cp normally requires tar inside; inspect errors first.

kubectl debug adds an ephemeral container to the existing Pod, subject to policy; runtime support controls target process visibility. kubectl run creates a separate Pod with potentially different network policy. kubectl cp generally needs tar in the container. Save evidence before deleting a controller-managed Pod that will be recreated.

Debug Services · Debug Pods · EndpointSlices