Vault Authentication — UserPass + Kubernetes¶

Prerequisites

Main Vault HA deployed and initialized — Main Vault HA
All 3 vault pods 1/1 Running
Root token available at ~/vault-main-init.json

Order matters — do NOT revoke root token until Step 4 is confirmed working

The sequence is:

Enable UserPass → create admin user
Enable Kubernetes auth → configure it
Test both logins work
Only then → revoke root token

Phase 1 — Set Up Environment

Step 1.1 — Install Vault CLI on Bastion¶

# Download vault CLI (amd64) — match the version running in the cluster
wget -q https://releases.hashicorp.com/vault/1.21.2/vault_1.21.2_linux_amd64.zip
unzip vault_1.21.2_linux_amd64.zip
sudo mv vault /usr/local/bin/
rm vault_1.21.2_linux_amd64.zip

# Verify
vault version

Expected:

Vault v1.21.2

Step 1.2 — Set Environment and Authenticate¶

export VAULT_ADDR="https://vault.cl1.sq4.aegis.internal"
export VAULT_SKIP_VERIFY=true

# Persist across sessions
echo 'export VAULT_ADDR="https://vault.cl1.sq4.aegis.internal"' >> ~/.bashrc
echo 'export VAULT_SKIP_VERIFY=true' >> ~/.bashrc

# Unset VAULT_TOKEN if set — it overrides the token helper and causes 403 errors
unset VAULT_TOKEN

# Authenticate with vault-admin (UserPass)
# Root token was set up in Phase 2 and revoked in Phase 5 — do NOT use it here
vault login -method=userpass \
  username=vault-admin \
  password="VaultAdmin123!"

# Verify connection
vault status

Do not use VAULT_TOKEN=root_token after Phase 5

Once the root token is revoked in Phase 5, setting VAULT_TOKEN from vault-main-init.json will result in:

* permission denied
* invalid token

Always use unset VAULT_TOKEN then vault login -method=userpass for all Day-2 operations.

Expected — Sealed: false, HA Mode: active on one pod.

Phase 2 — Enable UserPass Auth

Step 2.1 — Enable the Auth Method¶

vault auth enable userpass

Expected:

Success! Enabled userpass auth method at: userpass/

Step 2.2 — Create an Admin Policy¶

vault policy write admin - << 'EOF'
# Admin policy — full access to everything
path "*" {
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
EOF

Expected:

Success! Uploaded policy: admin

Step 2.3 — Create the Admin User¶

vault write auth/userpass/users/vault-admin \
  password="VaultAdmin123!" \
  policies="admin"

Expected:

Success! Data written to: auth/userpass/users/vault-admin

# Login with username and password — get back a token
vault login -method=userpass \
  username=vault-admin \
  password="VaultAdmin123!"

Expected:

Success! You are now logged in. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  hvs.XXXXXXXXXXXX
token_duration         768h
token_renewable        true
token_policies         ["admin" "default"]

✅ UserPass works. UI login at https://vault.cl1.sq4.aegis.internal → Method: Username → vault-admin / VaultAdmin123!

Phase 3 — Enable Kubernetes Auth

Kubernetes auth lets pods authenticate to Vault using their ServiceAccount token — no hardcoded credentials anywhere.

Step 3.1 — Create a ServiceAccount for Vault Token Review¶

Vault needs a Kubernetes ServiceAccount with permission to validate other ServiceAccount tokens (the system:auth-delegator ClusterRole).

# Create the ServiceAccount in the vault namespace
kubectl create serviceaccount vault-auth -n vault

# Bind it to system:auth-delegator so Vault can validate K8s tokens
kubectl create clusterrolebinding vault-auth-tokenreview \
  --clusterrole=system:auth-delegator \
  --serviceaccount=vault:vault-auth

Step 3.2 — Create a Long-Lived Token for the ServiceAccount¶

In Kubernetes 1.24+, ServiceAccount tokens are not auto-mounted in Secrets. Create one explicitly:

kubectl apply -f - << 'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: vault-auth-token
  namespace: vault
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
EOF

Wait a moment for Kubernetes to populate the token:

kubectl get secret vault-auth-token -n vault

Step 3.3 — Configure Vault Kubernetes Auth¶

# Get the Kubernetes API server URL
KUBE_HOST=$(kubectl config view --raw --minify --flatten \
  -o jsonpath='{.clusters[].cluster.server}')
echo "Kubernetes API: ${KUBE_HOST}"

# Get the ServiceAccount JWT token
VAULT_SA_TOKEN=$(kubectl get secret vault-auth-token -n vault \
  -o jsonpath='{.data.token}' | base64 -d)

# Get the Kubernetes CA certificate
KUBE_CA_CERT=$(kubectl get secret vault-auth-token -n vault \
  -o jsonpath='{.data.ca\.crt}' | base64 -d)

# Enable Kubernetes auth
vault auth enable kubernetes

# Configure it
vault write auth/kubernetes/config \
  kubernetes_host="${KUBE_HOST}" \
  kubernetes_ca_cert="${KUBE_CA_CERT}" \
  token_reviewer_jwt="${VAULT_SA_TOKEN}"

Expected:

Success! Data written to: auth/kubernetes/config

Step 3.4 — Create a Kubernetes Auth Role (Example)¶

A role maps a Kubernetes ServiceAccount in a specific namespace to a Vault policy. Create one example role for testing:

# Create a read-only policy for apps
vault policy write app-readonly - << 'EOF'
path "secret/data/*" {
  capabilities = ["read", "list"]
}
EOF

# Create a role — any pod using the "default" SA in "default" namespace gets app-readonly
vault write auth/kubernetes/role/demo-app \
  bound_service_account_names=default \
  bound_service_account_namespaces=default \
  policies=app-readonly \
  ttl=1h

Expected:

Success! Data written to: auth/kubernetes/role/demo-app

Audience warning is expected

WARNING! Role demo-app does not have an audience configured.

This warning is informational only — the role still works. An audience adds extra JWT claim verification (e.g. restricts the token to only authenticate to Vault). For production, consider adding:

vault write auth/kubernetes/role/demo-app \
  bound_service_account_names=default \
  bound_service_account_namespaces=default \
  policies=app-readonly \
  audience=vault \
  ttl=1h

Ignore the warning.

Binding to a different namespace or multiple namespaces

The bound_service_account_namespaces field controls which namespace(s) the role applies to. Change it to match wherever your app's ServiceAccount actually lives:

# Single specific namespace
vault write auth/kubernetes/role/my-app \
  bound_service_account_names=my-app-sa \
  bound_service_account_namespaces=my-app-namespace \
  policies=app-readonly \
  ttl=1h

# Multiple specific namespaces
vault write auth/kubernetes/role/my-app \
  bound_service_account_names=my-app-sa \
  bound_service_account_namespaces=staging,production \
  policies=app-readonly \
  ttl=1h

# Any namespace — wildcard (use carefully in production)
vault write auth/kubernetes/role/my-app \
  bound_service_account_names=my-app-sa \
  bound_service_account_namespaces="*" \
  policies=app-readonly \
  ttl=1h

Field	Meaning
`bound_service_account_names`	Which ServiceAccount name(s) are allowed
`bound_service_account_namespaces`	Which namespace(s) that SA must be in
Both must match	A pod is only authenticated if both SA name and namespace match

Step 3.5 — Test Kubernetes Auth from a Pod¶

kubectl run vault-test --image=alpine --restart=Never --rm -it \
  -- sh -c '
    apk add -q curl
    KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
    curl -sk \
      --request POST \
      --data "{\"jwt\": \"${KUBE_TOKEN}\", \"role\": \"demo-app\"}" \
      https://vault.cl1.sq4.aegis.internal/v1/auth/kubernetes/login \
      | grep -o "\"client_token\":\"[^\"]*\""
  '

Expected:

"client_token":"hvs.XXXXXXXXXXXX"

✅ Kubernetes auth works — the pod authenticated using only its ServiceAccount token.

Phase 4 — Verify Both Auth Methods

# List all enabled auth methods
vault auth list

Expected:

Path           Type          Description
----           ----          -----------
kubernetes/    kubernetes    n/a
token/         token         token based credentials
userpass/      userpass      n/a

Phase 5 — Revoke the Root Token

Only run this after confirming UserPass login works in Phase 2.4

Once the root token is revoked it cannot be recovered. A new root token can only be generated using the 3-of-5 recovery keys from ~/vault-main-init.json.

# Get the root token
ROOT_TOKEN=$(cat ~/vault-main-init.json | \
  python3 -c "import sys,json; print(json.load(sys.stdin)['root_token'])")

# Re-authenticate as vault-admin (UserPass) so we have a valid session after revocation
vault login -method=userpass \
  username=vault-admin \
  password="VaultAdmin123!"

# Revoke the root token
vault token revoke ${ROOT_TOKEN}

Expected:

Success! Revoked token (if it existed)

Verify the root token no longer works:

VAULT_TOKEN=${ROOT_TOKEN} vault token lookup

Expected:

Error looking up token: Code: 403. Errors:
* permission denied

✅ Root token revoked. Only vault-admin (UserPass) and Kubernetes SA tokens can now authenticate.

Day-2 — Adding Auth for a New Application

For every new application that needs Vault secrets:

# Always start with this — unset stale token, login with userpass
unset VAULT_TOKEN
vault login -method=userpass username=vault-admin password="VaultAdmin123!"

# 1. Create a policy scoped to the app's secret paths
vault policy write my-app - << 'EOF'
path "secret/data/my-app/*" {
  capabilities = ["read"]
}
EOF

# 2. Create a Kubernetes auth role for the app's ServiceAccount
vault write auth/kubernetes/role/my-app \
  bound_service_account_names=my-app-sa \
  bound_service_account_namespaces=my-app-namespace \
  policies=my-app \
  ttl=1h

# 3. The app pod uses vault-agent sidecar — no SDK needed
# Add these annotations to the Deployment:
#   vault.hashicorp.com/agent-inject: "true"
#   vault.hashicorp.com/role: "my-app"
#   vault.hashicorp.com/agent-inject-secret-config: "secret/data/my-app/config"

Summary

Auth Method	Who uses it	Credentials
UserPass	Human operators	`vault-admin` / `VaultAdmin123!`
Kubernetes	Pods / applications	ServiceAccount token (auto-mounted)
~~Token (root)~~	~~Initial setup only~~	~~Revoked~~

Common Issues

Symptom	Cause	Fix
`permission denied` after root token revoked	Current session used root token	Re-login: `vault login -method=userpass username=vault-admin password=...`
`Kubernetes auth role not found`	Role not created for this SA	Run Step 3.4 for the app's SA + namespace
`could not load Kubernetes configuration`	Wrong `kubernetes_host` or expired SA token	Re-run Step 3.3 with fresh token
Need root access again	Emergency — root token was revoked	Use recovery keys: `vault operator generate-root` with 3-of-5 recovery keys

ESO Vault Policy Reference¶

ESO (External Secrets Operator) uses a single Kubernetes auth role (eso) that bundles two policies. Every time you add a new Vault path for a service secret, check which policy it belongs to and add it there.

The two policies and their role¶

auth/kubernetes/role/eso
  ├── eso-read       — read-only; platform/shared paths outside ask/*
  └── eso-ask-push   — read+write; per-service paths under ask/*

ESO gets the union of both — it can read every path covered by either policy.

`eso-read` — current paths¶

Path	Added for
`secret/data/argocd/*`	ArgoCD repo/cluster credentials
`secret/data/apps/*`	Generic app secrets
`secret/data/harbor/*`	Harbor registry credentials
`secret/data/foundry`	Foundry LiteLLM master key
`secret/data/core-service/*`	core-service storage (S3 keys, added 2026-06-29)
`secret/metadata/*`	Metadata reads for all paths

`eso-ask-push` — current paths¶

Path	Added for
`secret/data/ask/postgres/*`	PostgreSQL credentials
`secret/data/ask/hatchet/*`	Hatchet worker token
`secret/data/ask/core-service/*`	core-service app secrets
`secret/data/ask/assistant-service/*`	assistant-service secrets
`secret/data/ask/knowledge-management-service/*`	KMS secrets
`secret/data/ask/frontend/*`	Frontend secrets

Why two policies?

eso-ask-push was created to support ESO PushSecret (write back to Vault) for ask/* service paths. eso-read is pure read-only for shared/platform paths that live outside the ask/ namespace. Keeping them separate means platform secrets can never be overwritten by a PushSecret operation.

Adding a new secret path — decision tree¶

New Vault secret path?
        │
        ├── Under ask/<service>/*  →  add to eso-ask-push
        │
        └── Outside ask/*          →  add to eso-read

After editing vault-bootstrap-config.yaml in Git, always apply the policy live immediately (don't wait for ArgoCD — the ExternalSecret will 403 until the policy is updated):

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal

# Re-apply eso-read (example — paste the full current policy)
vault policy write eso-read - << 'POLICY'
path "secret/data/argocd/*"        { capabilities = ["read"] }
path "secret/data/apps/*"          { capabilities = ["read"] }
path "secret/data/harbor/*"        { capabilities = ["read"] }
path "secret/data/foundry"         { capabilities = ["read"] }
path "secret/data/core-service/*"  { capabilities = ["read"] }
path "secret/metadata/*"           { capabilities = ["read", "list"] }
POLICY

vault policy read eso-read   # verify

Vault Authentication — UserPass + Kubernetes¶

Step 1.1 — Install Vault CLI on Bastion¶

Step 1.2 — Set Environment and Authenticate¶

Step 2.1 — Enable the Auth Method¶

Step 2.2 — Create an Admin Policy¶

Step 2.3 — Create the Admin User¶

Step 2.4 — Test UserPass Login¶

Step 3.1 — Create a ServiceAccount for Vault Token Review¶

Step 3.2 — Create a Long-Lived Token for the ServiceAccount¶

Step 3.3 — Configure Vault Kubernetes Auth¶

Step 3.4 — Create a Kubernetes Auth Role (Example)¶

Step 3.5 — Test Kubernetes Auth from a Pod¶

ESO Vault Policy Reference¶

The two policies and their role¶

eso-read — current paths¶

eso-ask-push — current paths¶

Adding a new secret path — decision tree¶

`eso-read` — current paths¶

`eso-ask-push` — current paths¶