Pod Disruption Budgets¶

What Is a PDB and Why Does It Matter?

A PodDisruptionBudget tells Kubernetes the minimum number of pods for a workload that must remain available during a voluntary disruption — a node drain, a cluster upgrade, a kubectl drain for maintenance.

Without a PDB, Kubernetes can evict every pod of a workload simultaneously during a node drain. For a 3-node quorum system like Vault, losing all pods at once means data unavailability or split-brain. For a 2-pod service like the GitLab webservice, it means total downtime.

Without PDB:
  kubectl drain node1
    → evicts webservice pod A (was on node1)
    → evicts webservice pod B (also on node1 — anti-affinity was "preferred" not "required")
    → GitLab is 0/2 — total outage during drain

With PDB (maxUnavailable: 1):
  kubectl drain node1
    → evicts pod A
    → tries to evict pod B → PDB blocks: "only 1 available, max unavailable is 1"
    → drain waits until pod A is rescheduled and Ready elsewhere
    → then evicts pod B
    → GitLab always has ≥ 1 pod running

PDBs protect against voluntary disruptions only. They cannot protect against node crashes (involuntary) — that is what replicas + anti-affinity handle.

PDB Design Rules

Quorum-based systems (odd number of replicas)¶

minAvailable = floor(replicas / 2) + 1     (equivalently maxUnavailable = floor(replicas/2))

Replicas	minAvailable	maxUnavailable	Quorum safe?
3	2	1	✅ 2/3 maintain quorum
5	3	2	✅ 3/5 maintain quorum

Applies to: Vault HA (3-node Raft) and every future Raft/quorum app (Hatchet, etcd, …) — see KB → Raft Clusters and Prod → Raft Sizing.

Non-quorum Deployments (2+ replicas)¶

maxUnavailable: 1     (or minAvailable: 1)

At least one pod must serve traffic at all times. Applies to: GitLab webservice/sidekiq/etc., ArgoCD repo-server & server, cert-manager controller & webhook, Reloader, VSO.

Single-replica workloads¶

A PDB with minAvailable: 1 on a single-replica pod blocks node drains forever — the pod can never be evicted without violating the budget. The correct fix is to scale to 2+, not to add a PDB.

Never put minAvailable: 1 on a replicas: 1 workload

It prevents kubectl drain from completing, which blocks cluster upgrades. Scale to 2 first, then add the PDB. The one accepted exception here is the ArgoCD application-controller (intentional singleton — see inventory).

Workload Inventory (current)

Workload	Namespace	Replicas	PDB	Source
GitLab — webservice, sidekiq, gitaly, kas, registry, gitlab-shell	gitlab	chart-set	`maxUnavailable: 1` (per component)	GitLab Helm chart (automatic)
Vault (HA Raft)	vault	3	`maxUnavailable: 1` (keep 2/3)	Vault Helm chart via `values/vault/vault-stg.yaml`
`argocd-repo-server`	argocd	2	`minAvailable: 1`	cluster-addons `infra/pdb/`
`argocd-server`	argocd	2	`minAvailable: 1`	cluster-addons `infra/pdb/`
`argocd-application-controller`	argocd	1	no PDB	Intentional singleton (sharding not enabled)
`cert-manager` controller	cert-manager	2	`minAvailable: 1`	cluster-addons `infra/pdb/`
`cert-manager-webhook`	cert-manager	2	`minAvailable: 1`	cluster-addons `infra/pdb/`
`cert-manager-cainjector`	cert-manager	1	none	Not in the critical path
`reloader`	reloader	2 (HA)	`minAvailable: 1`	cluster-addons `infra/pdb/`
`vso-controller`	vault-secrets-operator	2	`minAvailable: 1`	cluster-addons `infra/pdb/`

How PDBs Are Managed Here

Three sources, by workload type:

Source	Workloads	Where
1. Chart-provided	GitLab (all components), Vault	The component's own Helm values — upgrade-safe, no extra manifests
2. cluster-addons standalone	ArgoCD, cert-manager, reloader, VSO	`cluster-addons/infra/pdb/.yaml`, deployed by the `modgpt.cluster-addons.pdb`* app (the dir-type `apps/pdb/app.yaml`)
3. Scale-then-PDB	the single-replica components above	values bumped to `replicas: 2` so a `minAvailable: 1` PDB is meaningful

Implementation

1. Chart-provided PDBs¶

GitLab — the chart sets a PDB per component automatically (you'll see gitlab-webservice-default, gitlab-sidekiq-…, gitlab-gitaly, gitlab-kas, gitlab-registry-…, gitlab-gitlab-shell, each maxUnavailable: 1). No action needed beyond running ≥ 2 replicas where you want HA.

Vault — set in values/vault/vault-stg.yaml (already applied):

server:
  podDisruptionBudget:
    maxUnavailable: 1     # 3 nodes − 1 = 2 remaining = Raft quorum maintained

2. Standalone PDBs via cluster-addons (`infra/pdb/`)¶

These live in the GitLab cluster-addons repo and are deployed by the pdb app — a dir-type apps/pdb/app.yaml that recurses infra/pdb/:

cluster-addons/
├── apps/pdb/app.yaml            # type: dir, dirPath: infra/pdb  → modgpt.cluster-addons.pdb
└── infra/pdb/
    ├── argocd-pdb.yaml          # argocd-repo-server-pdb + argocd-server-pdb
    ├── cert-manager-pdb.yaml    # cert-manager-pdb + cert-manager-webhook-pdb
    ├── reloader-pdb.yaml        # reloader-pdb
    └── vso-pdb.yaml             # vso-controller-pdb

Example (infra/pdb/argocd-pdb.yaml):

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: argocd-repo-server-pdb, namespace: argocd }
spec:
  minAvailable: 1
  selector:
    matchLabels: { app.kubernetes.io/name: argocd-repo-server }
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: argocd-server-pdb, namespace: argocd }
spec:
  minAvailable: 1
  selector:
    matchLabels: { app.kubernetes.io/name: argocd-server }

Add/change a PDB = edit infra/pdb/, commit, push to GitLab — ArgoCD applies it via modgpt.cluster-addons.pdb within ~3 minutes:

cd ~/cluster-addons          # origin → gitlab.cl1.sq4.aegis.internal/modgpt/cluster-addons.git
git add infra/pdb/ && git commit -m "chore: update PDBs" && git push

argocd-application-controller has no PDB

It is an intentional singleton StatefulSet (ArgoCD v3 sharding not enabled). A PDB on a single-replica StatefulSet would block node drains. Leave it without a PDB and accept brief controller downtime during a drain — a known limitation of the single-shard design.

3. Scale single-replica → 2+ (done, kept permanent in values)¶

A PDB only protects what already has redundancy. These were scaled to 2 in their values (so ArgoCD self-heal keeps them there — never a one-off kubectl scale):

Component	Set in
ArgoCD repo-server & server (`replicas: 2`)	`values/argocd/argocd-stg.yaml`
cert-manager webhook (`webhook.replicaCount: 2`)	`values/cert-manager/cert-manager-stg.yaml`
Reloader (`reloader.deployment.replicas: 2` + `enableHA`)	`values/reloader/reloader-stg.yaml`
VSO (`controller.replicas: 2`)	`values/vso/vso-stg.yaml`

Always scale in the values file, not with kubectl scale

A live kubectl scale is ephemeral — ArgoCD self-heal reverts it to whatever the values say. Bump the replica count in cluster-addons/values/<svc>/ and push.

Verification

kubectl get pdb -A

Expected (current platform):

NAMESPACE              NAME                        MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS
argocd                 argocd-repo-server-pdb      1              N/A              1
argocd                 argocd-server-pdb           1              N/A              1
cert-manager           cert-manager-pdb            1              N/A              1
cert-manager           cert-manager-webhook-pdb    1              N/A              1
gitlab                 gitlab-gitaly               N/A            1                1
gitlab                 gitlab-gitlab-shell         N/A            1                1
gitlab                 gitlab-kas                  N/A            1                1
gitlab                 gitlab-registry-v1          N/A            1                1
gitlab                 gitlab-sidekiq-all-in-1-v1  N/A            1                1
gitlab                 gitlab-webservice-default   N/A            1                1
reloader               reloader-pdb                1              N/A              1
vault                  vault                       N/A            1                1
vault-secrets-operator vso-controller-pdb          1              N/A              1

ALLOWED DISRUPTIONS = 0 on a healthy cluster usually means all replicas are currently on the node being drained (PDB working correctly — the drain waits for rescheduling). ALLOWED DISRUPTIONS = 0 with a crashing pod means the minimum genuinely isn't met — fix the pod before draining.

Drain tests¶

# Dry-run — shows what would be evicted vs blocked
kubectl drain node1 --ignore-daemonsets --delete-emptydir-data --dry-run=client
# Look for: "evicting pod …" vs "cannot evict pod as it would violate the pod's disruption budget"

# Real drain on a worker (PDBs control eviction order)
kubectl cordon node3
kubectl drain node3 --ignore-daemonsets --delete-emptydir-data --timeout=300s
kubectl get pods -A | grep -vE "Running|Completed"   # nothing critical down
kubectl uncordon node3

Stateful workloads need more than a PDB during a drain

A drain that evicts a Vault pod is fine if the PDB keeps quorum — but the recreated pod must re-join Raft and auto-unseal. For Raft apps, drain one node at a time and verify rejoin between drains. See KB → Raft Clusters.

Common Issues

Symptom	Cause	Fix
`drain` hangs indefinitely	PDB `minAvailable`/`maxUnavailable` can't be satisfied — a pod is already unhealthy	Fix the unhealthy pod first, then retry the drain
`ALLOWED DISRUPTIONS: 0` on a healthy cluster	All replicas are on the node being drained	PDB working correctly — the drain waits for rescheduling
Drain blocked by a `gitlab-*` PDB during a crash-loop	A GitLab component is `0/1` so its PDB has no budget	Stabilise the component (or temporarily remove its PDB) before draining — see troubleshooting → VM Node Disk Pressure recovery
PDB selector matches nothing	Label selector doesn't match pod labels	`kubectl get pods -n <ns> --show-labels` and fix the selector
Replica count reverts after a manual scale	ArgoCD self-heal restores the values-file count	Set `replicas: 2` in `cluster-addons/values/<svc>/` and push — don't `kubectl scale`