HashiCorp Vault — Root Vault (Manual)¶

Two-Step Deployment

Vault is deployed in two deliberate steps:

Step	What	How	Page
1 — Root Vault	HA Raft 5-node Vault. Only job: Transit auto-unseal for the main cluster	Manual Helm + kubectl	This page
2 — Main Vault	Full HA cluster serving application secrets	ArgoCD App of Apps	App of Apps →

Root Vault is intentionally not managed by ArgoCD. If ArgoCD tried to sync Root Vault and accidentally deleted it, the main Vault cluster would be permanently sealed. Manual ownership = deliberate protection.

What Vault Provides

Feature	Detail
Secrets management	Store and tightly control access to tokens, passwords, certificates, API keys
Dynamic secrets	Generate short-lived credentials on-demand for databases, cloud providers
Encryption as a service	Encrypt/decrypt data without storing it
PKI / TLS	Issue and renew certificates automatically
Kubernetes integration	Inject secrets into pods via sidecar agent or CSI driver

Full Architecture¶

┌─────────────────────────────────────────────────────────┐
│  Bastion                                                │
│  helm install vault-root   (manual, this page)          │
└────────────────────┬────────────────────────────────────┘
                     ↓ deploys
┌─────────────────────────────────────────────────────────┐
│  Root Vault  (namespace: vault-root)                    │
│  HA Raft · 5 nodes · Transit secrets engine             │
│  Only job: unseal the main Vault cluster                │
└────────────────────┬────────────────────────────────────┘
                     ↓ Transit auto-unseal
┌─────────────────────────────────────────────────────────┐
│  Main Vault  (namespace: vault)          ← ArgoCD owns  │
│  HA Raft · 3 nodes · vault.cl1.sq4.aegis.internal                    │
│  Vault Agent Injector → pod sidecars                    │
└─────────────────────────────────────────────────────────┘
                     ↑
┌─────────────────────────────────────────────────────────┐
│  ArgoCD App of Apps  (15-app-of-apps.md)                │
│  Also manages: NVIDIA Device Plugin, Reloader, ...      │
└─────────────────────────────────────────────────────────┘

Root Vault runs in HA Raft (5 nodes) — Transit secrets engine, no UI, no ingress. Main Vault (managed by ArgoCD) runs in HA Raft with 3 nodes and auto-unseals via Root Vault's Transit engine.

Deployment Mode — Sizing & Trade-offs

Understanding the trade-offs before you deploy:

Mode Comparison¶

	HA Raft 5-node (this deployment)	HA Raft (production)
Vault nodes	5	3 or 5
Storage	Raft distributed across 5 PVCs	Raft distributed across all nodes
Pod failure	Leader election — another node takes over in seconds	Same
Node failure	Tolerates 2 simultaneous failures (quorum = 3 of 5)	3-node: 1 failure · 5-node: 2 failures
Unseal on restart	Manual (3 of 5 key shares per pod)	Manual OR Auto-unseal
Kubernetes spread	`requiredDuringScheduling` anti-affinity across 5 nodes	Same
PodDisruptionBudget	`maxUnavailable: 1`	Required

What Production-Grade Requires¶

1. HA Raft — 3 or 5 nodes

server:
  ha:
    enabled: true
    replicas: 3          # always odd: 3 or 5
    raft:
      enabled: true
      setNodeId: true    # stable node IDs survive pod restarts
      config: |
        ui = true
        listener "tcp" { ... }
        storage "raft" {
          path    = "/vault/data"
          retry_join {
            leader_api_addr = "https://vault-0.vault-internal:8200"
          }
          retry_join {
            leader_api_addr = "https://vault-1.vault-internal:8200"
          }
          retry_join {
            leader_api_addr = "https://vault-2.vault-internal:8200"
          }
        }

2. Auto-Unseal — critical for production

Why manual unseal is unacceptable in production¶

Vault uses Shamir's Secret Sharing to protect the master encryption key. On initialization, the key is split into N shares (here: 5) and a threshold number (here: 3) must be combined to reconstruct it. Vault never stores the master key — without the reconstructed key, it cannot decrypt its own storage.

Every time a Vault pod restarts (upgrade, OOM kill, node eviction, VM reboot), it starts in a sealed state. Until unsealed:

All reads and writes to Vault are rejected with 503 Service Unavailable
VSO cannot render secrets → application pods that depend on Vault secrets fail to start
ArgoCD sync fails for apps awaiting rendered secrets
Main Vault cannot unseal → full platform outage if all pods restart simultaneously

In production with automated rolling updates, an operator manually running three vault operator unseal commands is not a viable recovery path.

Auto-unseal options¶

Option	Suitable for	Requires	Production preference
Cloud KMS (AWS KMS / Azure Key Vault / GCP KMS)	Cloud or hybrid	Cloud account, internet or private VPC endpoint to KMS	⭐⭐⭐ Best for cloud
HSM (PKCS#11)	Air-gap, enterprise, regulated	Physical HSM (e.g. Thales Luna, AWS CloudHSM)	⭐⭐⭐ Best for on-prem regulated
Vault Transit	Air-gap, OSS, no HSM	A second standalone "root" Vault	⭐⭐ Pragmatic air-gap choice
Manual (this deployment)	Non-production only	Operator runs unseal after every restart	❌ Not for production

How Vault Transit auto-unseal works (this deployment's implementation)¶

Transit auto-unseal replaces Shamir key entry with an API call. Instead of a human providing 3 key shares, the Vault process itself calls a Transit encryption key on a trusted "root" Vault to unwrap the master key on startup.

The mechanism — step by step¶

┌─────────────────────────────────────────────────────────┐
│  ROOT VAULT (vault-root namespace)                       │
│  HA Raft · 5 pods · Transit secrets engine              │
│  Sealed by: 5 Shamir shares (threshold 3) — manual      │
│  Only job: serve Transit encrypt/decrypt requests        │
└────────────────────┬────────────────────────────────────┘
                     │  Transit API (in-cluster HTTP)
                     │  http://vault-root.vault-root.svc:8200
                     ▼
┌─────────────────────────────────────────────────────────┐
│  MAIN VAULT (vault namespace)                            │
│  HA Raft · 3 pods · Full secrets platform               │
│  Auto-unseal: calls Root Vault Transit on every start   │
└─────────────────────────────────────────────────────────┘

Initialization (one-time): 1. Root Vault is initialized and manually unsealed (human provides 3 of 5 Shamir shares) 2. Transit secrets engine is enabled on Root Vault: vault secrets enable transit 3. An encryption key is created: vault write -f transit/keys/vault-unseal 4. Main Vault is configured with the Transit stanza (see values below) 5. Main Vault is initialized — instead of creating Shamir shares for the master key, it wraps the master key with the Transit encryption key and stores the ciphertext

Every pod restart (automated): 1. Main Vault pod starts sealed 2. The seal "transit" stanza in Vault config fires automatically 3. Main Vault calls: POST http://vault-root.vault-root.svc:8200/v1/transit/decrypt/vault-unseal 4. Root Vault decrypts the wrapped master key and returns the plaintext 5. Main Vault uses the plaintext master key to decrypt its Raft storage 6. Main Vault is now unsealed and serving — no human intervention

What still requires manual unseal: - Root Vault itself — if all 5 root pods restart simultaneously, Root Vault is sealed and Main Vault cannot auto-unseal. Root Vault is therefore sized at 5 pods (tolerates 2 simultaneous failures while keeping quorum) and is treated as a critical infrastructure component that should never need a full restart in normal operations.

Main Vault Transit seal configuration¶

# values/vault/vault-stg.yaml (relevant section)
server:
  ha:
    config: |
      seal "transit" {
        address         = "http://vault-root.vault-root.svc.cluster.local:8200"
        token           = "<root-vault-token>"   # stored in Vault, rendered by VSO
        key_name        = "vault-unseal"
        mount_path      = "transit/"
        tls_skip_verify = "true"                 # internal cluster comms, no TLS
      }

Security properties¶

Property	Detail
Master key never stored in plaintext	Wrapped ciphertext stored in Raft; only Root Vault can decrypt it
Root Vault token scope	Read-only access to `transit/encrypt/vault-unseal` and `transit/decrypt/vault-unseal` only — minimal privilege
Root Vault token compromise	Attacker can unseal Main Vault but cannot read secrets — unsealing only gives access to the wrapped key, not the data
Root Vault availability	5-pod HA Raft — tolerates 2 simultaneous pod failures
Key rotation	`vault write -f transit/keys/vault-unseal/rotate` — Root Vault re-wraps the master key; no Main Vault restart needed

Failure modes¶

Scenario	Effect	Recovery
1–2 Root Vault pods restart	Root Vault stays available (3/5 quorum) — Main Vault unaffected	None needed
All Root Vault pods restart	Root Vault sealed → Main Vault cannot auto-unseal on its next restart	Manually unseal Root Vault (3 of 5 Shamir shares from `~/vault-root-init.json`)
Root Vault token expires	Main Vault cannot unseal on restart	Rotate token; update VSO-rendered secret
Main Vault pod restart (normal)	Auto-unseals via Transit in ~5 seconds	None needed — fully automated

Why not Cloud KMS here?¶

Cloud KMS is the cleanest solution for cloud-hosted clusters — no second Vault to manage, no Shamir shares for the unseal key, and the KMS is an externally-managed HA service. However this environment is fully air-gapped (no internet, no cloud account). Vault Transit is the correct OSS substitute for air-gapped on-prem deployments.

Production recommendation (x86, cloud-connected): use AWS KMS or Azure Key Vault auto-unseal. The Vault config stanza changes to:

seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "alias/vault-unseal"
}

No Root Vault instance needed. KMS is HA by design and requires no manual unsealing.

3. Anti-affinity — spread pods across physical nodes

Without this, Kubernetes may schedule all 3 Vault pods on the same node. If that node fails, the whole cluster goes down.

server:
  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: vault
              component: server
          topologyKey: kubernetes.io/hostname

4. PodDisruptionBudget — prevent simultaneous eviction

Without a PDB, a kubectl drain during a node upgrade can evict all Vault pods at once, losing quorum.

server:
  podDisruptionBudget:
    maxUnavailable: 1   # at most 1 pod down at a time

5. Audit Logging

vault audit enable file file_path=/vault/logs/audit.log

Ship to a SIEM (Splunk, Elastic, etc.) for compliance. Vault logs every request and response — critical for secrets access auditing.

6. Resource Limits and Requests

server:
  resources:
    requests:
      memory: 256Mi
      cpu: 250m
    limits:
      memory: 512Mi
      cpu: 500m

Production HA Values Skeleton¶

server:
  image:
    repository: harbor.cl1.sq4.aegis.internal/dockerhub/hashicorp/vault
    tag: "1.21.2"

  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: true

  dataStorage:
    storageClass: ceph-block
    size: 10Gi

  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: vault
              component: server
          topologyKey: kubernetes.io/hostname

  podDisruptionBudget:
    maxUnavailable: 1

  # Auto-unseal via AWS KMS (example)
  # extraEnvironmentVars:
  #   VAULT_SEAL_TYPE: awskms
  #   VAULT_AWSKMS_SEAL_KEY_ID: alias/vault-unseal

injector:
  enabled: true
  image:
    repository: harbor.cl1.sq4.aegis.internal/dockerhub/hashicorp/vault-k8s
    tag: "1.7.2"

ui:
  enabled: true

vault-root now runs HA Raft (3 nodes) — migration complete

vault-root was scaled from standalone → 3-pod HA Raft → 5-pod HA Raft in May 2026. Quorum: 3 of 5 — tolerates 2 simultaneous pod failures before Transit is unavailable. See Day-2 — Scale vault-root to HA for the runbook (kept for reference and re-use on fresh clusters).

Prerequisites

kubectl get nodes        # all Ready
kubectl get storageclass # ceph-block present
curl -sk https://gitlab.cl1.sq4.aegis.internal/-/health  # GitLab reachable
kubectl get pods -n cert-manager | grep "1/1"  # cert-manager Running
helm version             # v3.x

Phase 1 — TLS Certificates

cert-manager must be deployed before this step — see cert-manager (Internal CA).

Two Vault namespaces exist in this environment:

Namespace	Service	Ingress	TLS cert needed?
`vault-root`	Root Vault (Transit unseal)	❌ internal only	❌ No — `tls_disable = 1`
`vault`	Main Vault HA	✅ `vault.cl1.sq4.aegis.internal`	✅ Yes — `vault-tls` Secret

Root Vault has TLS disabled — it is only reachable inside the cluster and never exposed externally. No certificate is required.

Main Vault needs a certificate issued before ArgoCD deploys it — the ingress requires the vault-tls Secret to exist in the vault namespace at sync time.

Step 1.1 — Create Main Vault Namespace¶

kubectl create namespace vault

Step 1.2 — Issue vault-tls Certificate¶

kubectl apply -f - << 'EOF'
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: vault-tls
  namespace: vault
spec:
  secretName: vault-tls
  issuerRef:
    name: internal-ca-issuer
    kind: ClusterIssuer
  duration: 8760h
  renewBefore: 720h
  dnsNames:
    - vault.cl1.sq4.aegis.internal
EOF

Verify the certificate is issued:

kubectl get certificate -n vault vault-tls
# Expected: READY = True

kubectl get secret -n vault vault-tls
# Expected: TYPE = kubernetes.io/tls

vault-tls ready

The vault-tls Secret now exists in the vault namespace. ArgoCD can deploy Main Vault without any TLS errors.

Phase 2 — Mirror the Vault chart & images

Vault's chart and images are brought into the air-gapped registries with the standard bastion mirror workflow — see How-To → Mirror Images & Charts. Chart/image versions come from helm search repo hashicorp/vault and helm show values.

Artifact	Version	Destination
`hashicorp/vault` Helm chart	`0.32.0`	GitLab Helm registry — `https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/<PID>/packages/helm/stable`
`hashicorp/vault` image	`1.21.2`	Harbor — `harbor.cl1.sq4.aegis.internal/dockerhub/hashicorp/vault`
`hashicorp/vault-k8s` image	`1.7.2`	Harbor — `harbor.cl1.sq4.aegis.internal/dockerhub/hashicorp/vault-k8s`

Credentials are read from Vault (secret/gitlab/migration-token for GitLab) and ~/harbor-images/harbor-robot.json for Harbor — never hardcoded.

Phase 3 — Deploy Root Vault (Manual)

Root Vault is a minimal HA Raft instance (5 nodes). No UI, no ingress, no agent injector — just the Transit secrets engine accessible inside the cluster.

Step 3.1 — Create Values File¶

mkdir -p ~/cluster-addons/values/vault-root

cat > ~/cluster-addons/values/vault-root/vault-root.yaml << 'EOF'
server:
  image:
    repository: harbor.cl1.sq4.aegis.internal/dockerhub/hashicorp/vault
    tag: "1.21.2"          # ← replace with your VAULT_TAG

  standalone:
    enabled: false

  ha:
    enabled: true
    replicas: 5            # odd number — quorum = 3/5, tolerates 2 simultaneous failures
    raft:
      enabled: true
      setNodeId: true
      config: |
        ui = false

        listener "tcp" {
          tls_disable = 1
          address     = "[::]:8200"
        }

        storage "raft" {
          path = "/vault/data"
          retry_join { leader_api_addr = "http://vault-root-0.vault-root-internal:8200" }
          retry_join { leader_api_addr = "http://vault-root-1.vault-root-internal:8200" }
          retry_join { leader_api_addr = "http://vault-root-2.vault-root-internal:8200" }
          retry_join { leader_api_addr = "http://vault-root-3.vault-root-internal:8200" }
          retry_join { leader_api_addr = "http://vault-root-4.vault-root-internal:8200" }
        }

        service_registration "kubernetes" {}

        disable_mlock = true

  dataStorage:
    enabled: true
    size: 5Gi
    storageClass: ceph-block
    accessMode: ReadWriteOnce

  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: vault
              component: server
          topologyKey: kubernetes.io/hostname

  tolerations:
    - key: storage
      operator: Equal
      value: ceph
      effect: NoSchedule

  podDisruptionBudget:
    maxUnavailable: 1

  updateStrategyType: "OnDelete"

  service:
    type: ClusterIP       # internal only — main Vault calls it via the Service

injector:
  enabled: false          # no sidecar injector needed for root vault

ui:
  enabled: false          # operators use kubectl exec only

global:
  imagePullSecrets:
    - name: harbor-pull-secret
EOF

cat ~/cluster-addons/values/vault-root/vault-root.yaml   # verify

Step 3.2 — Deploy Root Vault from GitLab Helm HTTP¶

kubectl create namespace vault-root

GITLAB_HELM_REPO="https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/<PID>/packages/helm/stable"

helm install vault-root \
  vault \
  --repo ${GITLAB_HELM_REPO} \
  --version 0.32.0 \
  -n vault-root \
  -f ~/cluster-addons/values/vault-root/vault-root.yaml \
  --insecure-skip-tls-verify \
  --wait --timeout 5m

Watch pods:

kubectl get pods -n vault-root -w

Expected — pod Running but sealed (normal):

NAME             READY   STATUS
vault-root-0     0/1     Running   ← sealed until initialised

Phase 4 — Initialise Root Vault and Enable Transit

Step 4.1 — Initialize Vault¶

Initialization generates the master key shares and the initial root token. Run exactly once.

kubectl exec -n vault-root vault-root-0 -- vault operator init \
  -key-shares=5 \
  -key-threshold=3 \
  -format=json > ~/vault-root-init.json

cat ~/vault-root-init.json

Save vault-init.json — you cannot recover these keys

vault-init.json contains:

5 unseal keys — any 3 of the 5 are required to unseal Vault after every restart
Initial root token — used to configure Vault and create admin accounts

Back this file up immediately. Store it outside the cluster (not in a Kubernetes secret). If you lose all unseal keys, Vault data is permanently inaccessible — there is no recovery path.

Back it up to your organisation's secured offline secrets store immediately (not a Kubernetes Secret, not ~/.bashrc). Restrict access to the platform operators who hold unseal responsibility.

Step 4.2 — Unseal Root Vault¶

Vault was initialized with 5 key shares, threshold 3 — you must run the unseal command 3 times with 3 different keys. Each run shows Unseal Progress advancing until sealed = false.

# Extract all three unseal keys
ROOT_UNSEAL_0=$(cat ~/vault-root-init.json | python3 -c \
  "import sys,json; d=json.load(sys.stdin); print(d['unseal_keys_b64'][0])")
ROOT_UNSEAL_1=$(cat ~/vault-root-init.json | python3 -c \
  "import sys,json; d=json.load(sys.stdin); print(d['unseal_keys_b64'][1])")
ROOT_UNSEAL_2=$(cat ~/vault-root-init.json | python3 -c \
  "import sys,json; d=json.load(sys.stdin); print(d['unseal_keys_b64'][2])")

# Run all three unseal commands
kubectl exec -n vault-root vault-root-0 -- vault operator unseal ${ROOT_UNSEAL_0}
kubectl exec -n vault-root vault-root-0 -- vault operator unseal ${ROOT_UNSEAL_1}
kubectl exec -n vault-root vault-root-0 -- vault operator unseal ${ROOT_UNSEAL_2}

After the 3rd key, expected output:

Sealed          false    ← unsealed
Unseal Progress 0/3

Verify pod goes Ready:

kubectl get pods -n vault-root
# NAME             READY   STATUS
# vault-root-0     1/1     Running

Vault must be unsealed after every pod restart

Vault seals itself every time the pod restarts (crash, upgrade, node failure). Before running any vault commands — including Step 4.3 below — confirm Vault is unsealed:

kubectl exec -n vault-root vault-root-0 -- vault status | grep Sealed
# Sealed: false  ← OK to proceed
# Sealed: true   ← run the 3 unseal commands above first

Step 4.3 — Enable Transit Secrets Engine¶

Vault must be unsealed before this step

If you see Code: 503 — Vault is sealed, go back and run Step 4.2 unseal commands first.

ROOT_TOKEN=$(cat ~/vault-root-init.json | python3 -c \
  "import sys,json; d=json.load(sys.stdin); print(d['root_token'])")

# Login
kubectl exec -n vault-root vault-root-0 -- vault login ${ROOT_TOKEN}

# Enable Transit
kubectl exec -n vault-root vault-root-0 -- vault secrets enable transit

# Create the encryption key that will protect Main Vault's master key
kubectl exec -n vault-root vault-root-0 -- vault write -f transit/keys/vault-unseal

Step 4.4 — Create a Restricted Transit Token¶

The main Vault cluster needs a token to call root Vault's Transit engine. Grant it the minimum permissions — encrypt and decrypt the unseal key only.

Heredoc does not work with kubectl exec

The << 'POLICY' heredoc syntax is not passed through kubectl exec stdin correctly — Vault receives an empty body and returns 'policy' parameter not supplied or empty. Use sh -c to run the heredoc inside the container instead.

# Write the policy — note: sh -c runs the heredoc inside the container
kubectl exec -n vault-root vault-root-0 -- sh -c \
  'vault policy write vault-unseal - << EOF
path "transit/encrypt/vault-unseal"   { capabilities = ["update"] }
path "transit/decrypt/vault-unseal"   { capabilities = ["update"] }
EOF'

# Verify policy was created
kubectl exec -n vault-root vault-root-0 -- vault policy read vault-unseal

Expected:

path "transit/encrypt/vault-unseal"   { capabilities = ["update"] }
path "transit/decrypt/vault-unseal"   { capabilities = ["update"] }

# Create a long-lived token with that policy
TRANSIT_TOKEN=$(kubectl exec -n vault-root vault-root-0 -- \
  vault token create \
    -policy=vault-unseal \
    -period=8760h \
    -format=json | \
  python3 -c "import sys,json; print(json.load(sys.stdin)['auth']['client_token'])")

echo "Transit token: ${TRANSIT_TOKEN}"
echo ${TRANSIT_TOKEN} > ~/vault-transit-token.txt   # save — needed for main Vault config

Step 4.5 — Get Root Vault Internal Service Address¶

kubectl get svc -n vault-root vault-root
# NAME         TYPE        CLUSTER-IP    PORT(S)
# vault-root   ClusterIP   10.96.x.x    8200/TCP

# DNS name (use this in main Vault config — stable across pod restarts)
echo "http://vault-root.vault-root.svc.cluster.local:8200"

Summary — Root Vault

Item	Value
Namespace	`vault-root`
Mode	HA Raft — 5 nodes
Storage	`ceph-block` PVC (5 GiB)
Helm chart	`vault` (GitLab Helm registry)
Service	`vault-root.vault-root.svc.cluster.local:8200` (internal only)
Init file	`~/vault-root-init.json` — back up to Mac immediately
Transit token	`~/vault-transit-token.txt` — needed for main Vault

Root Vault is ready

Root Vault is unsealed and serving Transit requests. The Transit token is saved. Next step → App of Apps — deploy main Vault + all other cluster add-ons via ArgoCD.

Root Vault after a single pod restart (HA behaviour)

With 5-pod HA, if one vault-root pod restarts it becomes sealed — but the other 4 pods keep serving Transit requests via the Service. Main Vault continues to auto-unseal normally. Unseal the restarted pod at your convenience:

# unseal the restarted pod (3 of 5 keys)
for key in $KEY1 $KEY2 $KEY3; do
  kubectl exec -n vault-root vault-root-0 -- vault operator unseal $key
done

The cluster only becomes unavailable if 3+ pods are sealed simultaneously (quorum lost — need 3 of 5). Two simultaneous restarts are tolerated. See Day-2 — Scale vault-root to HA for the migration runbook.

Day-2 — Scale vault-root to HA (3 → 5 pods)

Why: vault-root was originally deployed as a single pod (standalone: true). A pod restart seals the Transit engine → main Vault cannot auto-unseal → cascading outage. Scaling to 5-pod Raft HA (via 3 pods first) means 2 simultaneous pod restarts still leave 3/5 pods serving Transit — the cluster stays available. 4-node clusters are skipped (even numbers have the same quorum as N-1).

How it works: the main Vault's seal "transit" config points at the Kubernetes Service (vault-root.vault-root.svc.cluster.local:8200), not a specific pod. With 3 pods, the Service routes Transit requests to any unsealed pod automatically.

Important: vault-root-0 will NOT restart automatically (OnDelete)

The StatefulSet uses updateStrategyType: OnDelete. vault-root-0 keeps running with the old config until you manually delete it. The scale-up (adding vault-root-1 and vault-root-2) causes no disruption to vault-root-0 or the main Vault cluster.

Step 1 — Apply the HA values (helm upgrade)¶

The values file at ~/cluster-addons/values/vault-root/vault-root.yaml has been updated to ha: enabled: true, replicas: 5 (with all 5 retry_join entries and a storage=ceph:NoSchedule toleration so pods can spread across all 7 nodes). Apply with a dry-run first:

GITLAB_HELM_REPO="https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/<PID>/packages/helm/stable"
# dry-run to confirm no surprises
helm upgrade vault-root vault --repo ${GITLAB_HELM_REPO} \
  -n vault-root \
  --version 0.32.0 \
  -f ~/cluster-addons/values/vault-root/vault-root.yaml \
  --dry-run 2>&1 | grep -E "UPGRADE|replicas|ha:"

If the dry-run looks correct, apply:

helm upgrade vault-root vault --repo ${GITLAB_HELM_REPO} \
  -n vault-root \
  --version 0.32.0 \
  -f ~/cluster-addons/values/vault-root/vault-root.yaml

Step 2 — Verify new pods are created (NOT vault-root-0 restart)¶

kubectl get pods -n vault-root -w
# Expected: vault-root-0 stays Running (1/1)
#           vault-root-1 and vault-root-2 appear as new pods (0/1 — sealed, expected)

vault-root-1 and vault-root-2 start sealed. This is correct — they haven't been unsealed yet. vault-root-0 continues serving Transit uninterrupted.

Step 3 — Join vault-root-1 to the cluster¶

# 1. join to the existing raft cluster (vault-root-0 is the leader)
kubectl exec -n vault-root vault-root-1 -- vault operator raft join \
  http://vault-root-0.vault-root-internal:8200

# 2. unseal vault-root-1 (3 of 5 keys from ~/vault-root-init.json)
KEYS=$(python3 -c "import json; k=json.load(open('vault-root-init.json'))['unseal_keys_b64']; print(' '.join(k[:3]))")
for key in $KEYS; do
  kubectl exec -n vault-root vault-root-1 -- vault operator unseal $key
done

# confirm it joined and is unsealed
kubectl exec -n vault-root vault-root-1 -- vault status 2>&1 | grep -E "Sealed|Mode"
# Expected: Sealed=false, Mode=standby

Step 4 — Join vault-root-2 to the cluster¶

kubectl exec -n vault-root vault-root-2 -- vault operator raft join \
  http://vault-root-0.vault-root-internal:8200

for key in $KEYS; do
  kubectl exec -n vault-root vault-root-2 -- vault operator unseal $key
done

kubectl exec -n vault-root vault-root-2 -- vault status 2>&1 | grep -E "Sealed|Mode"
# Expected: Sealed=false, Mode=standby

Step 5 — Verify raft cluster health¶

# need vault token — use root token from vault-root-init.json
ROOT_TOKEN=$(python3 -c "import json; print(json.load(open('vault-root-init.json'))['root_token'])")
kubectl exec -n vault-root vault-root-0 -- \
  env VAULT_TOKEN=$ROOT_TOKEN vault operator raft list-peers
# Expected: 3 peers, all alive, vault-root-0 is leader

Step 6 — Verify main Vault is still unsealed¶

kubectl get pods -n vault
# Expected: vault-0, vault-1, vault-2 all 1/1 Running (unchanged)
for p in vault-0 vault-1 vault-2; do
  echo -n "$p: "; kubectl exec -n vault $p -- vault status 2>/dev/null | grep "^Sealed"
done
# Expected: all Sealed=false

Step 7 — Commit the updated values¶

cd ~/cluster-addons
git add values/vault-root/vault-root.yaml
git -c http.sslVerify=false commit -m "feat(vault-root): scale to 5-pod HA Raft (tolerate 2 simultaneous failures)"
git -c http.sslVerify=false push origin main

Verification after migration¶

Check	Expected
`kubectl get pods -n vault-root`	`vault-root-0/1/2` all `1/1 Running`
`vault operator raft list-peers` (on vault-root-0)	3 peers, all alive
`kubectl get pods -n vault`	`vault-0/1/2` all `1/1 Running`, `Sealed=false`
Main Vault serves secrets	`vault kv get secret/gitlab/migration-token` works

What happens now when vault-root-0 restarts¶

Before (SPOF):   vault-root-0 restarts → sealed → Transit unavailable
                 → main Vault cannot auto-unseal → cascading outage

After (5-pod HA): vault-root-0 restarts → sealed
                 → Service routes Transit to any of vault-root-1..4
                 → main Vault auto-unseals normally ✅
                 → 2 simultaneous restarts still leave 3/5 serving ✅
                 → only 3+ simultaneous failures require human intervention

Common Issues

Symptom	Cause	Fix
`vault-0` stays `0/1 Running`	Vault is sealed — readiness probe fails until unsealed	Run Phase 3 initialize + unseal steps
`helm install` 504 / 502	GitLab registry slow/unavailable	Retry; check the in-cluster GitLab pods and its registry — see Troubleshooting
`ImagePullBackOff`	Image not present in Harbor	Re-mirror the image to Harbor (Phase 2)
Browser shows 404 on `vault.cl1.sq4.aegis.internal`	Wrong ingress schema — use `hostname:` not `hosts:`	Fix values file: `hostname: vault.cl1.sq4.aegis.internal` (see ArgoCD install page for details)
`dial tcp: lookup vault.cl1.sq4.aegis.internal`	DNS not resolving on the client	Ensure BIND9 resolves `vault.cl1.sq4.aegis.internal` → ingress VIP `100.115.2.100`
`Error initializing: Vault is already initialized`	`vault operator init` run more than once	Do not re-init — use saved `vault-init.json` unseal keys
Vault sealed after restart	Expected behaviour — no auto-unseal configured	Unseal manually with 3 key shares from `vault-init.json`

Knowledge Base

This section explains the concepts behind Vault's security model. Read this once — it answers the "why" behind every operational decision.

KB-1 — Why Vault Seals and Unseals¶

Vault's data on disk is always encrypted using a master key. When Vault starts, that master key is NOT on disk — it only exists in memory after unsealing.

Pod starts
  → Vault reads encrypted data from disk
  → Vault has NO master key in memory
  → Status: SEALED — refuses all requests

Operator provides 3 of 5 key shares
  → Vault reconstructs master key (Shamir's Secret Sharing algorithm)
  → Master key loaded into memory only
  → Status: UNSEALED — serving requests

Pod restarts (crash / upgrade / node failure)
  → Master key gone from memory
  → Status: SEALED again — must unseal again

Why this design? If an attacker steals the physical disk or the PVC snapshot, the data is encrypted and useless. The master key was never written anywhere — it only existed in RAM. This is the foundation of Vault's security model.

When you initialise Vault with -key-shares=5 -key-threshold=3, Vault uses Shamir's Secret Sharing to split the master key into 5 pieces:

Master Key  →  Split into 5 key shares
                Share 1 → Operator A
                Share 2 → Operator B
                Share 3 → Operator C
                Share 4 → Operator D
                Share 5 → Operator E

To unseal: any 3 of the 5 shares are enough to reconstruct the master key.

This means: - No single person holds the full key (principle of least privilege) - 2 people compromised = still cannot unseal (attacker needs 3) - 2 people unavailable = the other 3 can still unseal (operational resilience)

The numbers are configurable. A stricter setup uses key-shares=7 key-threshold=4.

KB-3 — Why Manual Unseal Breaks Production¶

In a non-production environment, running three unseal commands after a restart is acceptable. In production it fails:

Scenario	Impact without auto-unseal
Pod OOM-killed at 3am	Vault down — on-call engineer paged to type key shares
Rolling node upgrade	Every Vault pod restarts in sequence — each one needs manual unseal
HA leader pod crashes	New leader elected but also sealed — cluster loses quorum
Key holder on holiday	No one can unseal — production secrets unreachable for hours
Key shares in a password manager	Distributing to multiple engineers creates secret leakage risk

The core problem: manual unseal requires a human with a secret at exactly the moment the pod restarts — which is precisely when humans are least available (automated recovery, overnight failures, cascading restarts).

KB-4 — How Auto-Unseal Works¶

Auto-unseal replaces the human-with-key-shares step with a call to an external cryptographic service (KMS or HSM):

First initialisation:
  Vault generates master key
  → sends master key to KMS: "encrypt this"
  → KMS returns: encrypted_master_key (a blob, useless without KMS)
  → Vault stores encrypted_master_key on disk
  → Vault does NOT generate Shamir key shares

Every restart:
  Vault reads encrypted_master_key from disk
  → sends blob to KMS: "decrypt this"
  → KMS returns: master_key
  → Vault loads master_key into memory
  → UNSEALED — no human involved, completes in milliseconds

The actual master key never leaves the KMS. Vault only ever sees it decrypted in RAM.

KB-5 — Auto-Unseal Options Compared¶

Option	Air-gap	Cost	Complexity	Vault edition
Cloud KMS (AWS/Azure/GCP)	❌ Needs outbound HTTPS to cloud	~$1/month	Low	OSS
HSM (PKCS#11)	✅	$10k–$100k hardware	High	Enterprise only
Vault Transit	✅	Free	Medium	OSS
Manual (Shamir)	✅	Free	None	OSS

For this deployment's production upgrade path: Vault Transit — fully air-gapped, works with OSS Vault, costs nothing extra.

KB-6 — Vault Transit Auto-Unseal Explained¶

Vault Transit is Vault's own encryption-as-a-service feature. A dedicated "root" Vault acts as the KMS for the main HA cluster. Root Vault is intentionally not managed by ArgoCD — losing it would permanently seal the main cluster.

┌───────────────────────────────────────────────────────┐
│  Root Vault  (5-node HA Raft)                         │
│  namespace: vault-root                                │
│  vault-root-0..4 · 5 unique nodes · quorum = 3/5     │
│  Transit secrets engine: ENABLED                      │
│  Encryption key: "vault-unseal"                       │
│  Shamir seal: 3 of 5 keys required per pod            │
│  Service: vault-root.vault-root.svc.cluster.local     │
│  Only job: encrypt/decrypt the main Vault master key  │
└──────────────────┬────────────────────────────────────┘
                   │  HTTP (internal only, no TLS)
                   │  "decrypt my master key blob"
                   │  → any unsealed vault-root pod serves it
                   ▼
┌───────────────────────────────────────────────────────┐
│  Main Vault  (3-node HA Raft cluster)                 │
│  namespace: vault                                     │
│  Serves all application secrets                       │
│  Auto-unseals on every pod restart                    │
│  Calls vault-root Service at startup                  │
└───────────────────────────────────────────────────────┘

How it survives a vault-root pod restart: The main Vault seal "transit" config points at the Kubernetes Service (vault-root.vault-root.svc.cluster.local), not a specific pod. With 5-pod HA:

1 vault-root pod restarts → 4/5 remain unsealed → Service routes to them → main Vault unaffected
2 simultaneous restarts → 3/5 remain → still quorum, main Vault unaffected
3+ simultaneous restarts → quorum lost → main Vault cannot auto-unseal → manual intervention required

Security properties: - Root Vault does not know any of main Vault's secrets — it only decrypts one encrypted blob (the master key wrapper) - An attacker who compromises main Vault's disk still needs Root Vault's Transit key to decrypt the master key - Root Vault's Transit key is itself protected by Root Vault's Shamir seal — 3 of 5 shares required - Root Vault is intentionally manual (not ArgoCD) — a bad ArgoCD sync cannot accidentally destroy it

Day-2 — Transit Auto-Unseal Setup

Upgrade this deployment from manual unseal to Transit auto-unseal. Prerequisites: standalone Vault deployed and unsealed (Phase 1–3 above).

D2-1 — Deploy Root Vault¶

Deploy a minimal standalone Vault in a separate namespace (vault-root). This instance needs no ingress — only the main Vault cluster talks to it internally.

# Values file already created in Phase 3 Step 3.1
# Re-deploy using cluster-addons values

kubectl create namespace vault-root

helm install vault-root \
  vault \
  --repo ${GITLAB_HELM_REPO} \
  --version 0.32.0 \
  -n vault-root \
  -f ~/cluster-addons/values/vault-root/vault-root.yaml \
  --insecure-skip-tls-verify \
  --wait --timeout 5m

D2-2 — Initialise and Unseal Root Vault¶

# Initialise with 5-of-3 Shamir — matches production deployment (see Phase 4.1)
kubectl exec -n vault-root vault-root-0 -- vault operator init \
  -key-shares=5 \
  -key-threshold=3 \
  -format=json > ~/vault-root-init.json

# Extract 3 unseal keys and root token
ROOT_UNSEAL_0=$(cat ~/vault-root-init.json | python3 -c \
  "import sys,json; d=json.load(sys.stdin); print(d['unseal_keys_b64'][0])")
ROOT_UNSEAL_1=$(cat ~/vault-root-init.json | python3 -c \
  "import sys,json; d=json.load(sys.stdin); print(d['unseal_keys_b64'][1])")
ROOT_UNSEAL_2=$(cat ~/vault-root-init.json | python3 -c \
  "import sys,json; d=json.load(sys.stdin); print(d['unseal_keys_b64'][2])")
ROOT_TOKEN=$(cat ~/vault-root-init.json | python3 -c \
  "import sys,json; d=json.load(sys.stdin); print(d['root_token'])")

kubectl exec -n vault-root vault-root-0 -- vault operator unseal ${ROOT_UNSEAL_0}
kubectl exec -n vault-root vault-root-0 -- vault operator unseal ${ROOT_UNSEAL_1}
kubectl exec -n vault-root vault-root-0 -- vault operator unseal ${ROOT_UNSEAL_2}

# Back up ~/vault-root-init.json to your secured offline secrets store immediately

D2-3 — Enable Transit and Create Unseal Key¶

# Login to root vault
kubectl exec -n vault-root vault-root-0 -- vault login ${ROOT_TOKEN}

# Enable Transit secrets engine
kubectl exec -n vault-root vault-root-0 -- vault secrets enable transit

# Create the encryption key that will protect main Vault's master key
kubectl exec -n vault-root vault-root-0 -- vault write -f transit/keys/vault-unseal

# Create a restricted policy — only allowed to use the unseal key
kubectl exec -n vault-root vault-root-0 -- vault policy write vault-unseal - << 'POLICY'
path "transit/encrypt/vault-unseal"   { capabilities = ["update"] }
path "transit/decrypt/vault-unseal"   { capabilities = ["update"] }
POLICY

# Create a token with that policy (this goes into main Vault's config)
TRANSIT_TOKEN=$(kubectl exec -n vault-root vault-root-0 -- \
  vault token create -policy=vault-unseal -format=json | \
  python3 -c "import sys,json; print(json.load(sys.stdin)['auth']['client_token'])")

echo "Transit token: ${TRANSIT_TOKEN}"   # save this

D2-4 — Get Root Vault's Internal Service Address¶

kubectl get svc -n vault-root
# Note the ClusterIP — e.g. 10.96.x.x
# OR use the DNS name: http://vault-root.vault-root.svc.cluster.local:8200

D2-5 — Migrate Main Vault to Transit Auto-Unseal¶

This requires a Vault migration — plan for downtime

Changing the seal type requires running vault operator migrate. Schedule a maintenance window. Take a snapshot first.

# Snapshot current Vault data
kubectl exec -n vault vault-0 -- vault operator raft snapshot save /tmp/vault-backup.snap
kubectl cp vault/vault-0:/tmp/vault-backup.snap ~/vault-backup.snap

Update vault-stg.yaml — add the Transit seal block:

server:
  standalone:
    enabled: true
    config: |
      ui = true

      listener "tcp" {
        tls_disable = 1
        address     = "[::]:8200"
        cluster_address = "[::]:8201"
      }

      storage "raft" {
        path = "/vault/data"
      }

      seal "transit" {
        address         = "http://vault-root.vault-root.svc.cluster.local:8200"
        token           = "REPLACE_WITH_TRANSIT_TOKEN"
        disable_renewal = "false"
        key_name        = "vault-unseal"
        mount_path      = "transit/"
      }

      service_registration "kubernetes" {}

Run the migration:

# Exec into vault-0 and run migrate
kubectl exec -n vault vault-0 -- vault operator migrate \
  -config /vault/config/extraconfig-from-values.hcl

# Apply new values
helm upgrade vault \
  vault \
  --repo ${GITLAB_HELM_REPO} \
  --version 0.32.0 \
  -n vault \
  -f ~/cluster-addons/values/vault/vault-stg.yaml \
  --insecure-skip-tls-verify

D2-6 — Verify Auto-Unseal¶

# Restart vault-0 — should unseal automatically
kubectl delete pod -n vault vault-0

# Watch — should go 0/1 Running → 1/1 Running without manual unseal
kubectl get pods -n vault -w

# Confirm
kubectl exec -n vault vault-0 -- vault status | grep -E "Sealed|Storage Type"
# Sealed: false
# Storage Type: raft

Auto-unseal working

Pod restarted and reached 1/1 Running without any vault operator unseal commands. The main cluster now auto-unseals via root Vault Transit on every restart.