Kubernetes for Developers: The 20% You Actually Need

Kubernetes has roughly 70 built-in object kinds and a kubectl with more subcommands than git. You will use about eight of them in a normal week. The rest are platform-team concerns, niche workloads, or features you should be paying a managed service to handle for you.

I've shipped apps to Kubernetes since 1.9, back when you still hand-rolled RBAC and prayed over your kubelet certs. The single most useful thing I can tell a full-stack developer is this: you do not need to understand the control plane to deploy services confidently. You need one mental model and a small inventory of objects. This post is that inventory, plus the part nobody writes about — when to not use Kubernetes at all.

The mental model: desired state and reconciliation

Everything in Kubernetes reduces to one loop. You declare the state you want in YAML. A controller continuously compares that desired state to the actual state of the cluster and takes action to close the gap. Forever.

You say "I want 3 replicas of this container." A node dies and you're down to 2. The Deployment controller notices the gap and schedules a new Pod. You never told it to do that — you only ever told it the target. This is the difference between Kubernetes and a deploy script. A script runs once and stops caring. Kubernetes never stops caring.

Once this clicks, the entire API makes sense. Every object is a record of intent. kubectl apply doesn't "do" anything imperatively — it writes your intent to etcd, and controllers reconcile toward it. When something is wrong, you are almost always looking at a gap between what you declared and what the cluster could actually achieve.

The objects you actually use

Pod, Deployment, Service, Ingress

A Pod is one or more containers that share a network namespace and lifecycle. You rarely create Pods directly — they're the unit controllers manage on your behalf.

A Deployment manages a set of identical Pods, handles rolling updates, and gives you rollback. This is where 90% of your stateless services live.

A Service is a stable virtual IP and DNS name in front of a changing set of Pods. Pods are cattle with rotating IPs; the Service is the fixed address other things talk to.

An Ingress routes external HTTP traffic to Services based on host and path. It's your L7 router.

Here's a complete, working set for a typical web API. This is roughly what I deploy for real, minus the org-specific labels:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
  labels: { app: orders-api }
spec:
  replicas: 3
  selector:
    matchLabels: { app: orders-api }
  template:
    metadata:
      labels: { app: orders-api }
    spec:
      containers:
        - name: orders-api
          image: registry.example.com/orders-api:1.8.2
          ports:
            - containerPort: 8080
          envFrom:
            - configMapRef: { name: orders-config }
            - secretRef: { name: orders-secrets }
          resources:
            requests: { cpu: "100m", memory: "256Mi" }
            limits: { memory: "512Mi" }
          readinessProbe:
            httpGet: { path: /healthz/ready, port: 8080 }
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet: { path: /healthz/live, port: 8080 }
            initialDelaySeconds: 15
            periodSeconds: 20
---
apiVersion: v1
kind: Service
metadata:
  name: orders-api
spec:
  selector: { app: orders-api }
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: orders-api
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts: [api.example.com]
      secretName: orders-api-tls
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /orders
            pathType: Prefix
            backend:
              service:
                name: orders-api
                port: { number: 80 }

That's a production-shaped deployment in one file. Note the chain: Ingress → Service (port 80) → Pod (targetPort 8080). The selector on the Service is the glue — it matches the Pod labels, nothing else. Mismatched labels here is the single most common "my Service returns no endpoints" bug. When that happens, run kubectl get endpoints orders-api; an empty list means your selector matches zero Pods.

ConfigMap and Secret

ConfigMap holds non-sensitive config as key-value pairs. Secret holds sensitive data. Both inject into Pods as env vars (shown above with envFrom) or mounted files.

The honest truth about Secrets: by default they are base64-encoded, not encrypted, in etcd. Anyone with get secret RBAC or etcd access reads them in plaintext. For real workloads you want encryption at rest enabled on the API server (EncryptionConfiguration), or better, an external store — External Secrets Operator pulling from AWS Secrets Manager or HashiCorp Vault. Treat the native Secret as a delivery mechanism, not a vault.

kubectl create configmap orders-config \
  --from-literal=LOG_LEVEL=info \
  --from-literal=MAX_BATCH=200
 
kubectl create secret generic orders-secrets \
  --from-literal=DATABASE_URL='postgres://app:****@db:5432/orders'

Namespace

A Namespace is a soft boundary for naming, RBAC, and resource quotas. Use one per team or per environment-on-a-shared-cluster. Don't overthink it: dev, staging, prod is a fine start.

HorizontalPodAutoscaler

The HPA scales replica count based on observed metrics. CPU is the default; in practice I autoscale on a custom metric (requests-per-second or queue depth) far more often, because CPU is a poor proxy for I/O-bound services.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: orders-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300

The HPA needs metrics-server installed and CPU requests set on your Pods — utilization is computed as a percentage of the request. No request, no autoscaling. That stabilizationWindowSeconds matters: without it, autoscalers flap, churning Pods on every traffic wobble. Five minutes on scale-down is a sane default.

Probes, requests, and limits: the three things that cause real outages

These three settings cause more 3 a.m. pages than any networking config.

Readiness vs liveness. A readiness probe failing pulls the Pod out of the Service load balancer — no traffic, but the Pod keeps running. A liveness probe failing kills and restarts the container. Mixing these up is catastrophic. I've watched a team point a liveness probe at an endpoint that hit the database; the database got slow, every liveness check timed out, Kubernetes killed every Pod simultaneously, and a transient slowdown became a total outage. Liveness should answer "is this process wedged?" and nothing else. Readiness answers "can I serve traffic right now?" — that's where dependency checks belong.

Requests vs limits. A request is what the scheduler reserves; it's how Kubernetes decides which node has room. A limit is the hard ceiling. For memory there is no throttling — exceed the limit and the kernel OOMKills your container. You'll see it as exit code 137 and OOMKilled in the Pod status.

My production rule, learned the hard way:

Omitting CPU limits is the spicy take, but it's the right one for most latency-sensitive services. CPU is compressible — under contention you get throttled, not killed — and a tight CPU limit silently adds tail latency even when the node has idle cores. Memory is the opposite: incompressible, so you must cap it, and you want request ≈ limit so the scheduler doesn't overcommit a node into an OOM cascade.

kubectl: the commands that solve 95% of problems

You don't need to memorize kubectl. You need these.

# What's actually running, and is it healthy?
kubectl get pods -n prod
kubectl get pods -n prod -o wide        # adds node + IP
 
# Why is this Pod unhappy? READ THE EVENTS AT THE BOTTOM.
kubectl describe pod orders-api-7d9f8-x4k2 -n prod
 
# Logs — current and previous (after a crash/restart)
kubectl logs orders-api-7d9f8-x4k2 -n prod
kubectl logs orders-api-7d9f8-x4k2 -n prod --previous
kubectl logs -f -l app=orders-api -n prod --tail=100   # stream all replicas
 
# Get a shell inside a running container
kubectl exec -it orders-api-7d9f8-x4k2 -n prod -- sh
 
# Test a Service from inside the cluster without exposing it
kubectl port-forward svc/orders-api 8080:80 -n prod
 
# Did my Service find any Pods?
kubectl get endpoints orders-api -n prod
 
# Roll back a bad deploy
kubectl rollout status deployment/orders-api -n prod
kubectl rollout undo deployment/orders-api -n prod

kubectl describe pod is the most underused command in the toolkit. The Events section at the bottom tells you almost everything: FailedScheduling (no node has your requested resources), ImagePullBackOff (registry auth or a typo'd tag), CrashLoopBackOff (your container exits on startup — go read logs --previous), OOMKilled (raise the memory limit or fix the leak). Read events before you theorize.

When you should not use Kubernetes

Here's the part that gets me into arguments. Most apps don't need Kubernetes yet, and adopting it early is a tax you pay forever. Kubernetes solves problems you have at scale — many independently deployed services, multi-team ownership, bin-packing across a fleet, sophisticated rollout strategies. If you have one app and three engineers, it solves problems you don't have while creating ones you do: cluster upgrades, CNI debugging, RBAC, an entire YAML supply chain.

A single VM with Docker Compose or systemd units, restored from a Terraform-managed snapshot, will outperform a Kubernetes cluster on cost, latency, and the number of things that can wake you up — right up until you genuinely need orchestration across many machines and teams. The clearest buy signal for Kubernetes is organizational, not technical: when coordinating deploys across teams becomes the bottleneck, the declarative model and per-namespace isolation start earning their keep. Before that, you're paying for a fleet operating system to run a side project.

The managed control plane (EKS, GKE, AKS) removes the hardest 30% — you're not patching etcd. It does not remove the conceptual surface area, the upgrade cadence, or the YAML. Don't let "managed" trick you into thinking it's free.

The checklist

Before you ship a service to Kubernetes, confirm:

• Do you actually need k8s? If a PaaS or one VM fits, use that. Revisit when org scale demands it.
• Memory request and limit set (request ≈ limit); CPU request set, CPU limit usually omitted.
• Readiness probe checks dependencies; liveness probe checks only process health. Never the reverse.
• Service selector matches Pod labels — verify with kubectl get endpoints.
• Secrets come from an external store or at minimum etcd encryption-at-rest is on.
• HPA has metrics-server and CPU requests, plus a scaleDown stabilization window.
• You can recite the debug loop from memory: get pods → describe pod (read Events) → logs --previous.

Learn these eight objects and seven commands and you can operate confidently on any cluster. Everything beyond that is a problem you'll know you have when you have it — and most of you won't for a long time.

Further reading

• Kubernetes documentation — kubernetes.io/docs
• OWASP Kubernetes Top Ten — owasp.org
• External Secrets Operator — external-secrets.io