Building Scalable Cloud Infrastructure with Kubernetes

Modern platforms must absorb traffic spikes, recover from failures, and keep costs under control. Kubernetes (K8s) delivers this through automated deployment, scaling, and self-healing of containerized workloads across clouds and on-prem.

What “scalable” really means on K8s

• Elastic horizontally: add/remove pod replicas fast (seconds), and nodes dynamically (minutes).
• Resilient by design: pods are rescheduled on healthy nodes; rollouts can be paused/rolled back.
• Cost-aware: right-size CPU/memory, bin-pack pods, mix instance types/spot capacity.
• Team-safe: namespaced quotas, network boundaries, and RBAC keep multi-tenant clusters orderly.

---

Core building blocks

1) Workload primitives

• Deployment: stateless services with rolling updates.
• StatefulSet: ordered, persistent identity + stable storage (e.g., MongoDB, RabbitMQ).
• DaemonSet: per-node agents (logging, metrics, CNI, service mesh).
• Job/CronJob: batch and scheduled tasks.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  strategy:
    rollingUpdate: { maxUnavailable: 0, maxSurge: 1 }
  selector: { matchLabels: { app: api } }
  template:
    metadata: { labels: { app: api } }
    spec:
      containers:
        - name: api
          image: ghcr.io/org/api:1.4.2
          ports: [{ containerPort: 3000 }]
          readinessProbe:
            httpGet: { path: /healthz, port: 3000 }
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /livez, port: 3000 }
            periodSeconds: 10
          resources:
            requests: { cpu: "250m", memory: "256Mi" }
            limits:   { cpu: "500m", memory: "512Mi" }
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector: { matchLabels: { app: api } }

Notes:

• readinessProbe gates traffic until a pod is truly ready.
• topologySpreadConstraints keep replicas balanced across zones.

2) Service discovery & ingress

• ClusterIP for internal traffic, LoadBalancer for external, Ingress/Gateway API for L7 routing.

apiVersion: v1
kind: Service
metadata: { name: api-svc }
spec:
  selector: { app: api }
  ports: [{ name: http, port: 80, targetPort: 3000 }]
  type: ClusterIP

3) Autoscaling

• HPA (Horizontal Pod Autoscaler): scales replicas by CPU/memory or custom metrics.
• VPA (optional): recommends/sets requests/limits based on usage histories.
• Cluster Autoscaler: adds/removes nodes to fit pending pods.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: api-hpa }
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 15
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 70 }

For queue-driven workloads (e.g., RabbitMQ, Kafka, SQS), consider KEDA to scale on lag/queue depth.

4) Reliability guards

• PodDisruptionBudget (PDB): keep N pods available during voluntary disruptions.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: api-pdb }
spec:
  minAvailable: 2
  selector: { matchLabels: { app: api } }

• Anti-affinity & spread: avoid single-node or single-AZ concentration.
• Health-first rollouts: maxUnavailable: 0 for zero-downtime updates; use Argo Rollouts for canary.

5) Multi-tenancy & quotas

• Namespaces + ResourceQuota + LimitRange prevent noisy neighbors.

apiVersion: v1
kind: ResourceQuota
metadata: { name: team-a-quota, namespace: team-a }
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
    pods: "200"

---

Observability (see what matters)

• Metrics: Prometheus (scrapes kube-state-metrics, cAdvisor, app /metrics). Dashboards via Grafana.
• Logs: Fluent Bit/Vector → OpenSearch/ELK/CloudWatch/Stackdriver.
• Tracing: OpenTelemetry SDKs → Jaeger/Tempo/X-Ray.
• Alerting: Alertmanager (SLOs, burn-rates), plus kube events (CrashLoopBackOff, ImagePullBackOff).

Golden signals to track: latency, traffic, errors, saturation—as well as queue lag and DB connection pool utilization for backends.

---

Security (shift left, lock down)

• Least privilege with RBAC; group service accounts by app/namespace.
• Pod Security Standards (Baseline/Restricted) via admission controls.
• NetworkPolicies to default-deny and allow explicit flows (e.g., API → Redis only).
• Secret management: Kubernetes Secrets + envelope encryption; consider external KMS.
• Supply-chain: signed images (Sigstore/cosign), SBOMs, registry allow-lists, and admission policy (Kyverno/OPA Gatekeeper).
• Node/Kernel hardening: managed OS, read-only root FS, drop capabilities.

---

Data & storage

• CSI for cloud volumes; define StorageClass for SSD/HDD tiers.

• For high write throughput or HA databases, prefer managed services where possible; if self-hosting:

  • Use StatefulSets, anti-affinity, zonal PVs, and readiness gates.
  • Back up with Velero (cluster state + PV snapshots) and workload-native backups.

---

Delivery strategies (safe speed)

• GitOps: Argo CD or Flux tracks your desired state from Git (single source of truth).
• Progressive delivery: canary/blue-green with Argo Rollouts; automate promotion on metrics.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: platform-api }
spec:
  destination: { namespace: platform, server: https://kubernetes.default.svc }
  source:
    repoURL: https://github.com/org/platform-manifests
    targetRevision: main
    path: services/api
  syncPolicy:
    automated: { prune: true, selfHeal: true }

---

Cost optimization

• Requests/limits discipline: right-size from real usage; avoid setting only limits.
• Workload bin-packing: separate node pools (general, compute-optimized, memory-optimized).
• Spot/preemptible: mix for stateless workloads behind PDBs.
• Idle controls: scale to zero for dev/preview; cron wake-ups for batch.

---

Reference cluster design (multi-AZ)

• 3+ control-plane nodes (managed by EKS/GKE/AKS if possible).
• Node pools per profile: web/api, jobs, data, system/ingress/mesh.
• CNI: Cilium or Calico with NetworkPolicies.
• Ingress: NGINX or Gateway API controller.
• Service mesh (optional): Linkerd/Istio for mTLS, retries, canaries at L7.

---

Rollout plan (pragmatic)

1. Foundations

• Choose managed K8s; enable 3 AZs, set up node pools and autoscaler.
• Install core add-ons: metrics-server, CNI, Ingress/Gateway, ExternalDNS (if needed), cert-manager.

2. Platform ops

• Add Prometheus/Grafana, Loki/OpenSearch, Alertmanager, Velero.
• Enforce Pod Security + NetworkPolicies; bootstrap RBAC roles.

3. App onboarding

• Containerize services with health probes.
• Define Deployment/Service/HPA/PDB per service.
• Add CI to build/push images; add CD (Argo CD) to sync manifests.

4. Progressive delivery

• Introduce canary for high-risk services; automate rollback on error budgets.

5. Hardening & scale

• Tune quotas/limits; introduce VPA recommendations; add KEDA where queues exist.
• Add synthetic checks and SLOs; review dashboards & on-call playbooks.

---

Example: Scaling a queue consumer with KEDA (RabbitMQ)

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: orders-consumer }
spec:
  scaleTargetRef:
    kind: Deployment
    name: orders-consumer
  pollingInterval: 10
  minReplicaCount: 1
  maxReplicaCount: 50
  triggers:
    - type: rabbitmq
      metadata:
        hostFromEnv: RABBITMQ_CONN
        queueName: orders
        queueLength: "100"   # 1 replica per 100 messages

---

Operational checklist (prod-ready)

• Health probes + graceful shutdown (terminationGracePeriodSeconds).
• HPA + PDB + topology spread.
• Namespaces, quotas, limit ranges.
• NetworkPolicies default-deny.
• Centralized logs/metrics/traces + actionable alerts.
• Regular backups & disaster-recovery drills.
• GitOps with protected branches and image signing.
• Runbooks for incident response and rollbacks.

---

Common pitfalls

• No requests set → unpredictable bin-packing and throttling.
• Single-AZ PVs → outage risk; pin StatefulSets to zones matching PVs.
• Over-eager limits → CPU throttling; start with requests + generous/no limits.
• Ignoring readiness → 502s during rollouts; always gate traffic on readiness.
• One giant node pool → fragmentation; split by workload class.

---

Conclusion

Kubernetes gives you the control plane for elastic, resilient, and cost-efficient platforms. Combine good primitives (HPA/PDB/spread), strong ops (observability, GitOps, backups), and sound security (RBAC, Pod Security, NetworkPolicies) to scale confidently—whether you’re serving bursty web traffic, streaming events, or running data pipelines.