Upgrade External Secrets Operator¶

This runbook upgrades ESO from any 0.x version to v2.6.0 (chart 2.6.0).

Pipeline: mirror image → push Helm chart → update Git manifests → apply CRDs manually → sync ArgoCD.

Background — what changed between 0.14.4 and 2.6.0¶

Resource	Old apiVersion	New apiVersion
`ExternalSecret`	`external-secrets.io/v1beta1`	`external-secrets.io/v1`
`ClusterSecretStore`	`external-secrets.io/v1beta1`	`external-secrets.io/v1`
`ClusterExternalSecret`	`external-secrets.io/v1beta1`	`external-secrets.io/v1`
`PushSecret`	`external-secrets.io/v1alpha1`	`external-secrets.io/v1alpha1` (unchanged)
`ClusterGenerator`	`generators.external-secrets.io/v1alpha1`	`generators.external-secrets.io/v1alpha1` (unchanged)

ArgoCD drift if you skip the manifest update

From ESO v0.16.x onward, the admission webhook auto-converts v1beta1 resources to v1 on the cluster. If your Git manifests still say v1beta1, ArgoCD sees a diff on every sync and never reaches Synced. Update all manifests to v1 before syncing the new chart.

PushSecret + generatorRef bug in 0.14.4

ESO v0.14.4 has a bug where PushSecret.selector.generatorRef passes raw generated bytes to a Vault HTTP response parser → invalid character 'X' looking for beginning of value. This is fixed in v2.6.0. If you disabled PushSecrets as a workaround, re-enable them after the upgrade (see Re-enable PushSecrets below).

Pre-requisites — ArgoCD Application changes¶

Before syncing, add ServerSideApply=true to the ESO ArgoCD Application's syncOptions. ESO v2.6.0 CRDs have large OpenAPI validation schemas that exceed the 262 KB annotation limit used by client-side apply. Without SSA, every CRD sync fails with:

CustomResourceDefinition "clustersecretstores.external-secrets.io" is invalid:
metadata.annotations: Too long: may not be more than 262144 bytes

# environment/auh1/dev/02-argocd-application/cluster-addons/external-secrets.yaml
syncPolicy:
  syncOptions:
    - CreateNamespace=true
    - ServerSideApply=true    # required for large CRD schemas

Also add ignoreDifferences to suppress the persistent ClusterExternalSecret OutOfSync caused by server-side field normalization:

ignoreDifferences:
  - group: external-secrets.io
    kind: ClusterExternalSecret
    jqPathExpressions:
      - .metadata.namespace

Step 1 — Mirror the container image to Harbor¶

Run on bastion1 (needs internet via proxy):

HARBOR_USER=$(jq -r '.auths["harbor.cl1.sq4.aegis.internal"].username' ~/harbor-images/harbor-robot.json)
HARBOR_PASS=$(jq -r '.auths["harbor.cl1.sq4.aegis.internal"].password' ~/harbor-images/harbor-robot.json)

skopeo copy \
  --src-no-creds \
  --dest-creds "${HARBOR_USER}:${HARBOR_PASS}" \
  --dest-tls-verify=false \
  docker://ghcr.io/external-secrets/external-secrets:v2.6.0 \
  docker://harbor.cl1.sq4.aegis.internal/ghcr/external-secrets/external-secrets:v2.6.0

Step 2 — Mirror chart to Harbor, push to GitLab¶

Update scripts/harbor-mirror/helm-charts/mirror-charts.yaml and scripts/harbor-mirror/helm-charts/gitlab-charts.yaml with version 2.6.0, then run on bastion1:

# Mirror chart to Harbor OCI
cd ~/harbor-mirror
python3 core42-mirror-charts.py -c helm-charts/mirror-charts.yaml --only external-secrets

# Push chart from Harbor to GitLab HTTP registry
python3 helm-charts/gitlab-push-charts.py -c helm-charts/gitlab-charts.yaml --only external-secrets

The gitlab-push-charts.py script auto-fetches the GitLab token from Vault if GITLAB_TOKEN is not set — just ensure vault login is active first (export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal).

Step 3 — Update Git manifests¶

File	Change
`applications/cluster-addons/external-secrets/values-mod-auh1-dev-ask.yaml`	Image tag `v0.14.4` → `v2.6.0` (controller, webhook, certController)
`environment/.../external-secrets.yaml`	Chart `targetRevision` `0.14.4` → `2.6.0`; add `ServerSideApply=true`; add `ignoreDifferences`
`applications/cluster-addons/external-secrets/manifests/vault-backend-css.yaml`	`v1beta1` → `v1`
`applications/cluster-addons/external-secrets/manifests/harbor-pull-secret-ces.yaml`	`v1beta1` → `v1`
`applications/ask-db/*/core-pg-credentials.yaml` etc.	All ExternalSecrets `v1beta1` → `v1`

Push to GitLab (git push origin main) so ArgoCD can detect the update.

Step 4 — Apply CRDs manually (required)¶

ArgoCD cannot bootstrap large CRD upgrades automatically

Even with ServerSideApply=true, ArgoCD performs a dry-run validation before applying. If the cluster still has old CRDs, the dry-run for v1 resources fails and the entire sync aborts — including the CRD update. CRDs must be applied manually first.

The ESO chart stores CRDs in templates/crds/ using Helm template syntax (not the crds/ directory). helm show crds returns nothing. Use helm template instead:

# On bastion1 — chart is already in the mirror cache
helm template eso /opt/data/ask/ask_helm_charts/external-secrets-2.6.0.tgz \
  --set installCRDs=true | \
  python3 -c "
import sys, yaml
docs = [d for d in yaml.safe_load_all(sys.stdin)
        if d and d.get('kind') == 'CustomResourceDefinition']
print('\n---\n'.join(yaml.dump(d) for d in docs))
" | kubectl apply --server-side --force-conflicts -f -

Wait for the two key CRDs to be established:

kubectl wait crd clustersecretstores.external-secrets.io \
  externalsecrets.external-secrets.io \
  --for=condition=Established --timeout=60s

Step 5 — Sync ArgoCD¶

argocd app sync external-secrets --grpc-web --assumeYes

# Watch rollout
kubectl -n external-secrets rollout status \
  deploy/external-secrets \
  deploy/external-secrets-webhook \
  deploy/external-secrets-cert-controller

Confirm all pods are running:

kubectl -n external-secrets get pods

After controller restart, verify no errors in logs:

kubectl -n external-secrets logs deploy/external-secrets --tail=20

Expected: "Starting Controller" lines for each controller type, no no matches for kind errors.

Step 6 — Restart controller (post-CRD-delete recovery)¶

If you force-deleted a CRD during troubleshooting and ArgoCD recreated it, the ESO controller's informer cache will be stale. Restart it:

kubectl -n external-secrets rollout restart deploy/external-secrets
kubectl -n external-secrets rollout status deploy/external-secrets

Step 7 — Sync dependent apps and verify¶

argocd app sync \
  mod-auh1-dev-ask.ask.postgresql.core \
  mod-auh1-dev-ask.ask.postgresql.hatchet \
  mod-auh1-dev-ask.ask.postgresql.knowledge \
  --grpc-web --assumeYes

# Verify all ExternalSecrets are synced
kubectl -n ask get externalsecret -o wide

All should show SecretSynced / True.

Re-enable PushSecrets post-upgrade¶

ESO v0.14.4 had a bug where PushSecret.selector.generatorRef + Vault provider crashed with invalid character 'X'. This is fixed in v2.6.0.

To re-enable, restore PushSecret resources in the credential files with updatePolicy: IfNotExists — this guarantees existing Vault secrets are never overwritten:

apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: core-service-db-push
  namespace: ask
  annotations:
    argocd.argoproj.io/sync-wave: "0"   # runs before ExternalSecrets (wave 1)
spec:
  refreshInterval: 6h
  updatePolicy: IfNotExists             # safe: skips if Vault path already has a value
  secretStoreRefs:
    - name: vault-backend
      kind: ClusterSecretStore
  selector:
    generatorRef:
      apiVersion: generators.external-secrets.io/v1alpha1
      kind: ClusterGenerator
      name: password-alphanumeric-24
  data:
    - match:
        secretKey: password
        remoteRef:
          remoteKey: ask/postgres/core-service--DATABASE--PASSWORD

ExternalSecrets must be at sync-wave: "1" so they run after PushSecrets have written the password to Vault.

Fix persistent OutOfSync on ExternalSecret / PushSecret¶

ESO mutates resources after ArgoCD applies them:

What ESO does	Result
Controller adds `metadata.finalizers`	ArgoCD desired state has no finalizer → OutOfSync
Webhook defaults `spec.target.deletionPolicy: Retain`	ArgoCD desired state has no field → OutOfSync

Fix: add ignoreDifferences to each ArgoCD Application that manages ESO resources:

# applications/ask-db/apps/core-pg-app.yaml (and hatchet, knowledge)
ignoreDifferences:
  - group: external-secrets.io
    kind: ExternalSecret
    jsonPointers:
      - /metadata/finalizers
      - /spec/target/deletionPolicy
  - group: external-secrets.io
    kind: PushSecret
    jsonPointers:
      - /metadata/finalizers

Troubleshooting¶

Symptom	Cause	Fix
`Too long: may not be more than 262144 bytes`	CRD schema exceeds kubectl annotation limit	Add `ServerSideApply=true` to Application syncOptions
`no matches for kind "ExternalSecret" in version "external-secrets.io/v1"`	Controller running v2.6.0 but CRDs still v1beta1	Apply CRDs manually (Step 4)
`conversion webhook for v1beta1 failed: server could not find the requested resource`	CRD is pending deletion (stuck finalizer)	See below — CRD stuck in deletion
`helm show crds` returns nothing	CRDs are in `templates/crds/` not `crds/`	Use `helm template --set installCRDs=true` + python filter
ArgoCD dry-run fails with `version v1 not found`	CRD not yet updated when ArgoCD runs dry-run	Apply CRDs manually first; trigger re-sync after `kubectl wait --for=condition=Established`
ExternalSecret / PushSecret always OutOfSync	ESO adds finalizers + webhook defaults fields	Add `ignoreDifferences` for `/metadata/finalizers` and `/spec/target/deletionPolicy`
Controller logs: `failed to watch *v1.ClusterSecretStore` after CRD delete/recreate	Informer cache stale	`kubectl -n external-secrets rollout restart deploy/external-secrets`
`ClusterExternalSecret` always OutOfSync, diff is empty	SSA field ownership ghost	Add `ignoreDifferences` with `jqPathExpressions: [.metadata.namespace]`
`PushSecret: error unmarshalling vault secret: invalid JSON format`	Brand-new Vault path returns `{"errors":[]}` (404) which ESO cannot parse as a KV v2 secret — happens on first run of any PushSecret whose Vault path has never been written	Seed the path manually: `vault kv put secret/<path> password="$(openssl rand -hex 16)"` then force-sync the PushSecret

CRD stuck in deletion (`deletionTimestamp` set for hours)¶

This happens when kubectl delete crd <name> is run but CR instances still exist (the customresourcecleanup.apiextensions.k8s.io finalizer blocks deletion).

# 1. Delete all instances of the CRD
kubectl delete clustersecretstore --all

# 2. If CRD is still stuck, force-remove the finalizer
kubectl patch crd clustersecretstores.external-secrets.io \
  -p '{"metadata":{"finalizers":[]}}' --type=merge

# 3. Trigger ArgoCD sync — it will recreate the CRD from the chart
argocd app sync external-secrets --grpc-web --assumeYes

# 4. Wait for CRD to be re-established
kubectl wait crd clustersecretstores.external-secrets.io \
  --for=condition=Established --timeout=60s