Running Kubernetes in Production
Kubernetes has become the industry standard for container orchestration, but the gap between a working development cluster and a production-grade deployment is significant. This guide covers the essential practices for running Kubernetes reliably in production environments, drawing from real-world experience managing clusters that serve millions of requests daily.
Cluster Architecture and Setup
Production Kubernetes clusters require careful planning around high availability, network topology, and node sizing. Key architectural decisions include:
- Managed vs. self-managed -- services like Amazon EKS, Google GKE, and Azure AKS eliminate the operational burden of managing the control plane, and are recommended for most organizations
- Multi-AZ deployment with nodes distributed across at least three availability zones for fault tolerance
- Node pool segmentation separating workloads by resource requirements -- compute-intensive services on CPU-optimized instances, memory-heavy databases on memory-optimized instances
- Cluster autoscaler configuration with appropriate minimum and maximum node counts and scale-down delays to prevent thrashing
Deployment Strategies
Production deployments demand zero-downtime strategies that minimize risk. Rolling updates are the default in Kubernetes, but teams should also implement:
- Blue-green deployments for major version releases where instant rollback capability is critical
- Canary deployments that route a small percentage of traffic to new versions before full rollout
- Pod Disruption Budgets ensuring that voluntary disruptions like node drains never reduce available replicas below a safe threshold
- Readiness and liveness probes configured with appropriate thresholds to prevent traffic from reaching unhealthy pods
Resource Management
Proper resource requests and limits are fundamental to cluster stability. Every production pod should define CPU and memory requests that reflect actual usage, ensuring the scheduler places pods on nodes with sufficient capacity. Memory limits prevent individual pods from consuming all available memory on a node, which would trigger the OOM killer and affect neighboring workloads.
Namespace-level ResourceQuotas and LimitRanges provide guardrails that prevent any single team or application from monopolizing cluster resources.
Observability and Monitoring
Production Kubernetes clusters require comprehensive observability across three pillars:
- Metrics collected by Prometheus and visualized in Grafana, covering cluster health, node utilization, pod resource consumption, and application-specific KPIs
- Logging aggregated through a centralized stack such as the EFK (Elasticsearch, Fluentd, Kibana) stack or Loki, with structured JSON logging from all applications
- Tracing using OpenTelemetry to track requests across microservices, identifying latency bottlenecks and error propagation paths
Alerting should be configured around symptoms rather than causes — alert on elevated error rates and increased latency, not on individual pod restarts.
Security Best Practices
Kubernetes security requires a defense-in-depth approach. Essential measures include enforcing Pod Security Standards to prevent privileged containers, implementing Network Policies to restrict inter-pod communication to only what is necessary, scanning container images for vulnerabilities in the CI/CD pipeline, rotating secrets regularly using tools like External Secrets Operator or HashiCorp Vault, and enabling audit logging on the API server to track all administrative actions.
Continuous Delivery with GitOps
GitOps workflows using tools like ArgoCD or Flux ensure that the desired cluster state is always defined in version-controlled manifests. Every change to the production environment flows through a pull request, is reviewed by peers, and is automatically reconciled by the GitOps controller. This approach provides a complete audit trail, enables easy rollbacks by reverting commits, and eliminates configuration drift between environments.
Share this article