Skip to content

Production Considerations

This section documents production-grade hardening applied to this cluster. This deployment simulates a real production deployment — every design decision is held to production standards.

Why This Section Exists

Running Kubernetes is easy. Running it the way it should run in production requires deliberate, additional work beyond "it works". This section captures:

  • Gaps identified — things that would fail or be unacceptable in production if left unaddressed
  • Decisions made — why the production-correct approach was chosen even when a simpler alternative existed
  • Known deviations — where a current environment or tooling constraint forces a compromise, with the production-correct alternative documented
Production Standards Applied in This Deployment
Standard Applied Notes
High-availability replicas ✅ GitLab 2-pod, PostgreSQL 3-node, Vault 3-node, Valkey 3-node
Pod anti-affinity ✅ All multi-replica workloads spread across nodes
RWX storage for multi-pod ✅ GitLab on CephFS, not RWO ceph-block
Shared session state ✅ GitLab sessions/cache/queue in Valkey, not in-memory
Secrets management ✅ HashiCorp Vault + VSO — no secrets in Git or env vars
GitOps ✅ All workload config in Git, deployed via ArgoCD
Internal CA + TLS ✅ cert-manager with internal CA for all ingress
Pod Disruption Budgets ✅ See PDB page
Resource limits & requests ✅ § Resource Limits — LimitRange/ResourceQuota per namespace, QoS classes
Backup strategy ✅ § Backup & DR — Velero + Vault/etcd/GitLab snapshots to Ceph S3
Network policies ✅ § Network Policies — default-deny + explicit allows (Canal/Calico)
Observability (metrics/alerts) ✅ § Observability — Prometheus + Alertmanager + Loki
Known Deviations From Production

These are constraints imposed by the current environment or tooling. Each has the production-correct alternative documented.

Deviation Reason Production-Correct Alternative
HTTP Helm registry ArgoCD v3 has known OCI repo bugs OCI registry once ArgoCD OCI support stabilises
Self-signed internal CA No public DNS / public cert Public CA (Let's Encrypt) or corporate PKI on real infra
preferred anti-affinity on GitLab Lab has limited nodes; required would block scheduling requiredDuringScheduling + topologySpreadConstraints on 5+ node prod cluster
Single-replica for some components Resource constraints Scale ArgoCD repo-server, cert-manager webhook, pgpool to 2+
3-node Raft clusters (Vault) Sized to current footprint 5 members — see below
Raft / Quorum Cluster Sizing — Use 5, Not 3

Every consensus-based clustered app must run an odd number of members, and production should use 5, not 3. This applies to Vault today and to all upcoming Raft/quorum apps (Hatchet, etcd, Consul, RabbitMQ quorum queues, Patroni, NATS).

Members Quorum Failures tolerated Verdict
3 2 1 Minimum — zero headroom during a rolling upgrade
5 3 2 Production — survives a maintenance roll and an unplanned failure simultaneously

Why not 3 in production: a rolling upgrade intentionally takes one member down → you are at the 2/3 quorum edge with no margin; a single additional hiccup (OOM, drain, network blip) loses quorum → cluster goes read-only/unavailable. 5 members give you the margin to roll and tolerate a failure at the same time.

Mandatory companions for any Raft cluster (production):

  • Odd size (3/5/7) — never even (4 tolerates the same as 3 at higher cost + split-brain risk).
  • requiredDuringScheduling pod anti-affinity — no two members on one node, so a single node loss never removes 2 voters.
  • PodDisruptionBudget minAvailable: <quorum> (e.g. 3 for a 5-node cluster) — node drains can't break consensus.
  • OnDelete update strategy + one-at-a-time rolls — never blind kubectl rollout restart.
  • Persistent per-member storage (Ceph RBD / SSD), never emptyDir.

➡️ Full operational detail (quorum math, the safe rolling procedure, auto-unseal/bootstrap dependencies) is in KB → Raft / Quorum StatefulSet Clusters.

Resource Limits & Requests

Every container must declare requests and limits. Without them the scheduler can't bin-pack safely, a single workload can starve a node, and you have no QoS guarantees.

  • requests = what the scheduler reserves. Set it to the workload's real steady-state usage (measure with kubectl top / Prometheus), not a guess. Under-requesting causes overcommit and OOM cascades; over-requesting wastes capacity.
  • limits = the hard cap. For memory, limit should survive transient spikes (e.g. JVM/Rails boot eager-load — see the GitLab webservice 3Gi limit). Exceeding the memory limit = OOMKill. For CPU, prefer no limit (or a generous one) on latency-sensitive apps — CPU limits cause throttling; rely on requests for fairness.
  • QoS classes: requests == limits → Guaranteed (evicted last); requests < limits → Burstable; none → BestEffort (evicted first — never use in prod).

Enforce it cluster-wide:

# A LimitRange per namespace — default requests/limits + a floor/ceiling
apiVersion: v1
kind: LimitRange
metadata: { name: defaults, namespace: <ns> }
spec:
  limits:
    - type: Container
      default:        { cpu: "500m", memory: 256Mi }   # applied if a container omits limits
      defaultRequest: { cpu: "100m", memory: 128Mi }   # applied if it omits requests
      max:            { cpu: "4",    memory: 8Gi }
---
# A ResourceQuota per tenant namespace — caps total consumption
apiVersion: v1
kind: ResourceQuota
metadata: { name: tenant-quota, namespace: <ns> }
spec:
  hard:
    requests.cpu: "8"
    requests.memory: 16Gi
    limits.cpu: "16"
    limits.memory: 32Gi
    pods: "50"

Production-correct additions: VPA in recommendation mode to right-size requests from real usage, and requiredDuringScheduling PodTopologySpread so requests are honoured across zones.

Backup & Disaster Recovery

A workload is not production-ready until a restore has been tested. Cover four layers:

Layer Tool What it protects Restore test
Cluster objects + PVs Velero (with the CSI snapshot plugin → Ceph RBD/CephFS snapshots) Namespaces, manifests, and the data on PVCs velero restore into a scratch namespace
etcd (control plane) RKE2 built-in snapshots (rke2 etcd-snapshot) to S3 (Ceph RGW) The entire cluster state Restore on a rebuilt control plane
Vault vault operator raft snapshot save (scheduled CronJob → Ceph S3) All secrets + auth config raft snapshot restore into a test Vault
GitLab gitlab-backup / gitlab-rake to the gitlab-backups Ceph bucket Repos, DB, registry metadata Restore into a staging GitLab

Rules: backups go to object storage off the workload's own disk (Ceph S3 here; a different failure domain in real prod); keep a retention policy (e.g. 7 daily / 4 weekly); and automate a periodic restore drill — an untested backup is a hope, not a backup. Note: GitOps means most manifests are already recoverable from Git — backups are for stateful data (PVs, Vault, GitLab DB/registry, etcd).

Network Policies

The cluster is currently flat — any pod can reach any other pod. Production requires a default-deny posture with explicit allows (zero-trust east-west).

# 1. Default-deny ALL ingress + egress in a namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: <ns> }
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
# 2. Then allow only what's needed, e.g. app → its DB, and DNS egress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-dns, namespace: <ns> }
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } } }]
      ports: [{ protocol: UDP, port: 53 }, { protocol: TCP, port: 53 }]

The cluster's Canal (Calico) CNI enforces NetworkPolicy. Production-correct: per-namespace default-deny, explicit app→dependency allows, restrict egress to known CIDRs, and use Calico GlobalNetworkPolicy for cluster-wide baseline rules. Keep the policies in cluster-addons/infra/netpol/ so they're GitOps-managed (an apps/netpol/app.yaml dir-type app — see App of Apps).

Observability (metrics, logs, alerts)

You cannot operate what you cannot see. The production stack is kube-prometheus-stack (Prometheus + Alertmanager + Grafana) plus log aggregation.

  • Metrics: Prometheus scrapes via ServiceMonitor/PodMonitor CRDs; each workload exposes /metrics. Grafana dashboards for cluster, Ceph, Vault, GitLab, ArgoCD.
  • Alerts (Alertmanager): the non-negotiable set —
  • Node: NodeMemoryPressure, NodeDiskPressure, KubeNodeNotReady
  • Workloads: KubePodCrashLooping, KubePodNotReady, KubeDeploymentReplicasMismatch
  • Quorum apps: Vault sealed / Raft peer lost; etcd no-leader — page immediately
  • Storage: Ceph HEALTH_WARN/HEALTH_ERR, OSD down, PG degraded, near-full
  • Certs: cert-manager certificate expiring < 14 days
  • GitOps: ArgoCD app OutOfSync/Degraded > 15 min
  • Logs: Loki + promtail (or Fluent Bit → object storage), so pod logs survive pod death — essential for the post-incident reviews this platform needs (kubectl logs is gone once the pod is).
  • Capacity: alert on requests vs allocatable (this cluster repeatedly hit Insufficient memory — an alert at 85% reserved would have caught it before pods went Pending).

Deploy it the same way as everything else — a monitoring App-of-Apps under the modgpt project.

Hardening Checklist
# Item Status Reference
01 Pod Disruption Budgets ✅ Done prod-01-pdb.md
02 Resource Limits & Requests ✅ Documented § Resource Limits — apply LimitRange/ResourceQuota per namespace
03 Backup & DR (Velero + Vault/etcd/GitLab snapshots) ✅ Documented § Backup & DR — automate + test restores
04 Network Policies (default-deny) ✅ Documented § Network Policies — GitOps in infra/netpol/
05 Observability (Prometheus + Alertmanager + Loki) ✅ Documented § Observability
06 Raft clusters sized to 5 (odd) ✅ Documented § Raft Sizing + KB
07 OCI Helm Registry (when ArgoCD OCI matures) ⏸️ Deferred Blocked by ArgoCD v3.4.2 double-encoding — troubleshooting → ArgoCD OCI 403
08 Scale single-replica components to 2+ ⏸️ Deferred ArgoCD repo-server (done: 2), cert-manager webhook (2), pgpool → 2
09 Harbor container registry ✅ Done Deployed (2 replicas); all container images mirrored here

Status legend

✅ Done — implemented and running. ✅ Documented — production approach written + ready to apply. ⏸️ Deferred — blocked by a tooling constraint, with the trigger to revisit noted.