Graceful Full Shutdown for Hardware Maintenance (Node/VM-Level)¶

Runbook for a planned node/VM-level hardware update (RAM/vCPU/disk resize) affecting the mod-auh1-dev-ask cluster. Every workload on the affected nodes needs to stop cleanly before the nodes go down, rather than being forcibly killed. This is a full planned-outage shutdown — downtime is acceptable/expected — optimized for "stop cleanly in dependency order, come back up the same way in reverse," not for zero-downtime node draining.

Related: Release Strategy (post-restart verification checklist), Pod Disruption Budgets.

Execution record — hardware-maintenance window¶

Environment is back up

A full-stack hardware-maintenance shutdown was performed and the environment brought back online. This section records what was actually done; the steps further down are the procedure that was followed for the application tier.

Item	Detail
Date / window	TBD — fill in
Scope	Application tier (AI71) → Kubernetes (RKE2) → infrastructure (KVM VMs + Ceph)
Sequence	App tier quiesced via the runbook below → infra team powered down K8s nodes, VMs, and Ceph → power-on in reverse order
System updates	OS patched during the window — Rocky Linux 9.7 → 9.8 (confirmed on all nodes)
Operators	TBD

Confirmed post-restart state (verified 2026-07-05):

All 17 nodes Ready — 3 masters (.51–.53), 7 CPU workers (.54–.60), 7 GPU (.14–.20)
RKE2 v1.33.1+rke2r1 (Kubernetes 1.33), containerd 2.0.5-k3s1, Rocky Linux 9.8
DNS, image provenance (Harbor ask/), and the API VIP all green via verify-docs.sh
Incidents during the window: TBD — record any here.

Scope — this runbook covers the application tier

The steps below quiesce the ask / hatchet application stack (ArgoCD pause → scale workloads → CNPG hibernate) so nothing is force-killed. Shutting down Kubernetes itself, the KVM VMs, Ceph, and OS patching is an infrastructure-layer task (bare-metal / Ansible), sequenced after the app tier is quiesced and before power-off; bring-up is the exact reverse.

Post-maintenance doc updates¶

After each maintenance window, reconcile the docs to the new reality:

[x] OS → Rocky Linux 9.8, RKE2 v1.33.1, containerd 2.0.5 (done 2026-07-05)
[ ] Models & aliases — re-list from Foundry (Model Aliases §1); update served-models / alias tables, capability matrix, and LLM → config mappings
[ ] Re-run verify-docs.sh and reconcile any drift
[ ] Confirm image digests unchanged (no silent redeploys) — else update the digest map
[ ] Refresh the Resource Inventory snapshot date + node/pod usage
[ ] Confirm Ceph still Reef v18 and healthy (ceph version, ceph -s)
[ ] Finalize the daytona1 access list (was TBD)

Why not just `kubectl drain` the nodes?¶

PDBs only block voluntary evictions (kubectl drain), not a Deployment's own desired-replica count change. A direct kubectl scale --replicas=0 sidesteps PDB concerns entirely and gives full control over ordering — which matters here because we want dependents stopped before their dependencies (app tier before Hatchet, Hatchet before its database), not an eviction order chosen by the node drain process.

Two things that will silently undo a manual shutdown¶

ArgoCD self-heal. Every Application here runs Sync Policy: Automated (Prune). A plain kubectl scale drifts from Git and gets reverted on the next reconcile. Disable sync on every affected Application before scaling anything.
HorizontalPodAutoscalers. core-service has one confirmed. Standard K8s HPAs cannot target minReplicas: 0 — the HPA controller will keep fighting a manual scale-to-0 even with ArgoCD paused. Delete the HPA object (ArgoCD recreates it from Git once sync resumes).

Step 1 — Disable ArgoCD auto-sync¶

ARGOCD_PASSWORD=$(kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath='{.data.password}' | base64 -d)
argocd login argocd.cl1.sq4.aegis.internal --username admin --password "${ARGOCD_PASSWORD}" --insecure --grpc-web

argocd app list | grep "mod-auh1-dev-ask.ask"

Real app list is bigger than expected, and argocd app set doesn't work here

Confirmed 2026-07-02 against the live cluster:

argocd app set <app> --sync-policy none fails on every app with FATA[0000] Source position should be specified and must be greater than 0 for applications with multiple sources — every Application here has 2+ sources (chart repo + devops repo), and this ArgoCD version's CLI categorically requires --source-position for app set on multi-source apps, even for app-level (non-source) fields like sync policy. Use kubectl patch on the Application resource directly instead — it bypasses this entirely.
There are more Applications than the original plan assumed: core-service-secrets (parallel to hatchet-secrets), and the 3 CNPG Postgres clusters are themselves ArgoCD Applications — mod-auh1-dev-ask.ask.postgresql.core, .hatchet, .knowledge — not just plain StatefulSets to hibernate directly.
Nested app-of-apps self-heal, THREE levels deep, not two. The 3 postgresql.* children are managed by mod-auh1-dev-ask.ask.postgresql. Patching the children alone didn't stick — the parent immediately reverted them. Patching mod-auh1-dev-ask.ask (the app-of-apps for the whole ask environment) also didn't stick on its own — confirmed it silently reverted back to map[prune:true] a short time after being patched, having shown <none> moments earlier (don't trust a single check — verify a second time after a delay). The actual cause: there's a third, higher-level root — mod-auh1-dev-ask (environment/auh1/dev/02-argocd-application/_root, no dot suffix) — which manages mod-auh1-dev-ask.ask itself, alongside cluster-addons and observability. All three levels must be paused, outermost included, or the chain silently re-heals from the top down:
```
kubectl patch application mod-auh1-dev-ask -n argocd --type merge -p '{"spec":{"syncPolicy":null}}'
kubectl patch application mod-auh1-dev-ask.ask -n argocd --type merge -p '{"spec":{"syncPolicy":null}}'
kubectl patch application mod-auh1-dev-ask.ask.postgresql -n argocd --type merge -p '{"spec":{"syncPolicy":null}}'
```
Find the full app-of-apps hierarchy for any environment with:
```
kubectl get application -n argocd -o custom-columns='NAME:.metadata.name,SOURCE_PATH:.spec.source.path,SOURCES:.spec.sources[*].path'
```
and look for _root/parent-style source paths pointing at directories of other Application manifests, not just Helm chart paths.
Multi-line for loops failed when pasted into the bastion's interactive shell (bash: syntax error near unexpected token 'do') — likely no bracketed-paste support, so each line executes independently rather than as one continued statement. Use flat, individual one-line commands instead of loops when running commands interactively here.

Working version, accounting for all of the above:

# leaf apps
kubectl patch application mod-auh1-dev-ask.ask.core-service -n argocd --type merge -p '{"spec":{"syncPolicy":null}}'
kubectl patch application mod-auh1-dev-ask.ask.core-service-secrets -n argocd --type merge -p '{"spec":{"syncPolicy":null}}'
kubectl patch application mod-auh1-dev-ask.ask.assistant-service -n argocd --type merge -p '{"spec":{"syncPolicy":null}}'
kubectl patch application mod-auh1-dev-ask.ask.knowledge-management-service -n argocd --type merge -p '{"spec":{"syncPolicy":null}}'
kubectl patch application mod-auh1-dev-ask.ask.hatchet -n argocd --type merge -p '{"spec":{"syncPolicy":null}}'
kubectl patch application mod-auh1-dev-ask.ask.frontend -n argocd --type merge -p '{"spec":{"syncPolicy":null}}'

# all THREE root/app-of-apps levels — must be paused BEFORE their children, or the children get
# reverted from the top down (outermost root first, since it manages the middle one)
kubectl patch application mod-auh1-dev-ask -n argocd --type merge -p '{"spec":{"syncPolicy":null}}'
kubectl patch application mod-auh1-dev-ask.ask -n argocd --type merge -p '{"spec":{"syncPolicy":null}}'
kubectl patch application mod-auh1-dev-ask.ask.postgresql -n argocd --type merge -p '{"spec":{"syncPolicy":null}}'

# CNPG children — only stick once their parent (above) is paused
kubectl patch application mod-auh1-dev-ask.ask.postgresql.core -n argocd --type merge -p '{"spec":{"syncPolicy":null}}'
kubectl patch application mod-auh1-dev-ask.ask.postgresql.hatchet -n argocd --type merge -p '{"spec":{"syncPolicy":null}}'
kubectl patch application mod-auh1-dev-ask.ask.postgresql.knowledge -n argocd --type merge -p '{"spec":{"syncPolicy":null}}'

# confirm — check this more than once a few seconds apart, since reverts can lag
kubectl get application -n argocd -o custom-columns='NAME:.metadata.name,AUTOMATED:.spec.syncPolicy.automated' | grep "mod-auh1-dev-ask.ask"

Every row should show <none> — including the two root apps this time, unlike the original plan which assumed they were safe to leave alone.

Step 2 — Delete HPAs¶

kubectl get hpa -n ask

Confirmed 2026-07-02: core-service does NOT have an HPA — assistant-service does

The original plan assumed core-service had one (based on an ArgoCD resource-tree listing seen earlier). The actual live state was 3 HPAs, all on assistant-service components:

kubectl delete hpa -n ask assistant-service assistant-service-intelligence assistant-service-tooling

Always run kubectl get hpa -n ask fresh and delete whatever it actually shows — don't trust an assumption carried over from an earlier, unrelated check.

Step 3 — Frontend (stop new traffic first)¶

kubectl -n ask scale deploy/frontend --replicas=0
kubectl -n ask get pods -l app.kubernetes.io/instance=frontend

Step 4 — App-tier API + workers¶

kubectl -n ask scale deploy/core-service --replicas=0

kubectl -n ask scale deploy/assistant-service --replicas=0
kubectl -n ask scale deploy/assistant-service-intelligence --replicas=0
kubectl -n ask scale deploy/assistant-service-tooling --replicas=0
kubectl -n ask scale deploy/assistant-service-generic-worker --replicas=0
# assistant-service-notifications-worker: skip — disabled (enabled: false)

kubectl -n ask scale deploy/knowledge-management-service --replicas=0
kubectl -n ask scale deploy/knowledge-management-service-generic-worker --replicas=0
kubectl -n ask scale deploy/knowledge-management-service-ingestion-worker --replicas=0
kubectl -n ask scale deploy/knowledge-management-service-live-ingestion-worker --replicas=0

kubectl -n ask get pods -l app.kubernetes.io/instance=core-service
kubectl -n ask get pods -l app.kubernetes.io/instance=assistant-service
kubectl -n ask get pods -l app.kubernetes.io/instance=knowledge-management-service

Step 5 — Zitadel¶

kubectl -n ask scale deploy/core-service-zitadel --replicas=0
kubectl -n ask scale deploy/core-service-zitadel-login --replicas=0

Step 6 — Hatchet (own namespace)¶

kubectl -n hatchet get deploy,statefulset   # confirm exact names before scaling
kubectl -n hatchet scale deploy/hatchet-api --replicas=0
kubectl -n hatchet scale deploy/hatchet-engine --replicas=0
kubectl -n hatchet scale deploy/hatchet-frontend --replicas=0
kubectl -n hatchet scale statefulset/hatchet-rabbitmq --replicas=0

Step 7 — Redis, Weaviate, and the Hatchet PgBouncer pooler¶

kubectl -n ask scale statefulset/assistant-service-redis-master --replicas=0
kubectl -n ask scale statefulset/weaviate --replicas=0
kubectl -n ask scale deploy/hatchet-pg-pooler --replicas=0

Weaviate and the pooler were missing from the original scope entirely

Neither was in the original plan. Weaviate is KMS's vector database (a StatefulSet in ask) — easy to miss since it isn't referenced by name anywhere in the app-tier Deployments, only used internally by KMS. hatchet-pg-pooler is CNPG's PgBouncer connection pooler for hatchet-pg — leaving it running while hatchet-pg gets hibernated in Step 8 would just cause it to sit there failing to connect, not harmful but not clean. Re-run kubectl -n ask get deploy,statefulset -o wide yourself before assuming this list (or the plan's original one) is complete for your environment — new workloads get added over time.

Most of the app tier was found already at 0 replicas when we got here

By the time Steps 2–7 were reached, core-service, assistant-service (+ workers), KMS (+ workers), frontend, Zitadel, and all of Hatchet (api/engine/frontend/RabbitMQ) were already scaled to 0/0 — presumably from earlier, separate action outside this specific sequence. Confirmed via kubectl get deploy -n ask -o wide and kubectl -n hatchet get deploy,statefulset -o wide. Don't assume a workload still needs scaling down just because it's listed in this runbook — check its actual current state first.

Step 8 — CNPG Postgres clusters (last — test one first)¶

Do not kubectl scale the underlying StatefulSet directly — the CNPG operator reconciles it back the same way ArgoCD would. Use CNPG's declarative hibernation instead.

Verified 2026-07-02: the kubectl cnpg plugin is NOT installed on this bastion

kubectl cnpg hibernate on <cluster> -n ask fails with error: unknown command "cnpg" for "kubectl" — go straight to the annotation fallback, don't waste time on the plugin form. Also confirmed: hibernation is not instant. Right after annotating, pods still show Running for a short window (same lag observed on all 3 clusters) before CNPG's operator actually reconciles and terminates them — re-check after a brief pause rather than assuming the annotation didn't work. kubectl get cluster continues to report Cluster in healthy state even fully hibernated with zero pods — that's just CNPG's generic status text, not a sign hibernation failed; trust the pod/PVC check, not the STATUS column.

kubectl annotate cluster knowledge-pg -n ask cnpg.io/hibernation=on --overwrite
kubectl annotate cluster core-pg -n ask cnpg.io/hibernation=on --overwrite
kubectl annotate cluster hatchet-pg -n ask cnpg.io/hibernation=on --overwrite

# wait a short moment, then confirm all three
kubectl -n ask get pods -l cnpg.io/cluster=knowledge-pg
kubectl -n ask get pods -l cnpg.io/cluster=core-pg
kubectl -n ask get pods -l cnpg.io/cluster=hatchet-pg
kubectl -n ask get pvc -l cnpg.io/cluster=knowledge-pg
kubectl -n ask get pvc -l cnpg.io/cluster=core-pg
kubectl -n ask get pvc -l cnpg.io/cluster=hatchet-pg

Confirmed complete 2026-07-02: all three clusters hibernated (zero pods each), all PVCs intact.

At every step above, verify pods are actually gone before moving to the next — don't pipeline blindly.

Restart procedure (after hardware work — reverse order)¶

# 1. CNPG clusters back first
kubectl cnpg hibernate off knowledge-pg -n ask
kubectl cnpg hibernate off core-pg -n ask
kubectl cnpg hibernate off hatchet-pg -n ask
kubectl -n ask get cluster knowledge-pg core-pg hatchet-pg   # wait for each to report healthy

# 2. Redis, Weaviate, and the Hatchet PgBouncer pooler
kubectl -n ask scale statefulset/assistant-service-redis-master --replicas=1
kubectl -n ask scale statefulset/weaviate --replicas=1
kubectl -n ask scale deploy/hatchet-pg-pooler --replicas=2

# 3. Hatchet — RabbitMQ first, let it fully rejoin quorum before engine/api/frontend
kubectl -n hatchet scale statefulset/hatchet-rabbitmq --replicas=3
kubectl -n hatchet get pods -l app.kubernetes.io/name=rabbitmq -w   # wait for all 3 Running/Ready
kubectl -n hatchet scale deploy/hatchet-engine --replicas=2
kubectl -n hatchet scale deploy/hatchet-api --replicas=2
kubectl -n hatchet scale deploy/hatchet-frontend --replicas=2

# 4. Zitadel
kubectl -n ask scale deploy/core-service-zitadel --replicas=1
kubectl -n ask scale deploy/core-service-zitadel-login --replicas=1

# 5. App tier + workers
kubectl -n ask scale deploy/core-service --replicas=2
kubectl -n ask scale deploy/assistant-service --replicas=2
kubectl -n ask scale deploy/assistant-service-intelligence --replicas=1
kubectl -n ask scale deploy/assistant-service-tooling --replicas=1
kubectl -n ask scale deploy/assistant-service-generic-worker --replicas=1
kubectl -n ask scale deploy/knowledge-management-service --replicas=1
kubectl -n ask scale deploy/knowledge-management-service-generic-worker --replicas=1
kubectl -n ask scale deploy/knowledge-management-service-ingestion-worker --replicas=1
kubectl -n ask scale deploy/knowledge-management-service-live-ingestion-worker --replicas=1

# 6. Frontend
kubectl -n ask scale deploy/frontend --replicas=1

# 7. Re-enable ArgoCD auto-sync — do NOT hand-reconstruct every leaf app's syncPolicy individually.
#    Our shutdown patch (spec.syncPolicy: null) wiped the WHOLE syncPolicy object per app, not just
#    "automated" — including syncOptions (CreateNamespace, ServerSideApply) and, for hatchet,
#    managedNamespaceMetadata (the PSA labels its setup jobs need to run privileged). Hand-patching
#    each leaf app risks silently dropping those. Instead, restore only the 3 root/app-of-apps
#    levels (matching what was actually observed before shutdown: {"automated":{"prune":true}} for
#    all three), outermost first, then sync from the top down — the app-of-apps pattern re-applies
#    every child Application's FULL original manifest from Git as part of that sync, restoring
#    syncOptions/managedNamespaceMetadata automatically without needing to touch any leaf app directly.
kubectl patch application mod-auh1-dev-ask -n argocd --type merge -p '{"spec":{"syncPolicy":{"automated":{"prune":true}}}}'
kubectl patch application mod-auh1-dev-ask.ask -n argocd --type merge -p '{"spec":{"syncPolicy":{"automated":{"prune":true}}}}'
kubectl patch application mod-auh1-dev-ask.ask.postgresql -n argocd --type merge -p '{"spec":{"syncPolicy":{"automated":{"prune":true}}}}'

Confirmed 2026-07-03: argocd CLI may not be installed on every bastion

bastion2 (unlike bastion1) doesn't have the argocd binary at all. Use kubectl for everything instead — both to force an immediate refresh and to force an actual sync operation (equivalent to argocd app sync, without needing the CLI):

# force a refresh (re-evaluate diff) — NOT the same as a sync, may not be enough on its own
kubectl patch application <app> -n argocd --type merge -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'

# force an actual sync operation
kubectl patch application <app> -n argocd --type merge -p '{"operation":{"sync":{"revision":"HEAD","prune":true}}}'

Patching the 3 roots' syncPolicy cascaded correctly to every leaf app except one (hatchet stayed <none> after the others had already restored) — patch stragglers individually rather than assuming the cascade got everything. Confirm with:

kubectl get application -n argocd -o custom-columns='NAME:.metadata.name,AUTOMATED:.spec.syncPolicy.automated' | grep "mod-auh1-dev-ask"

After the cascade, every leaf app will show OutOfSync (not Synced) for a while — that's expected, since our manual restart replica counts differ from Git's declared values. Force a sync per leaf app the same way if you don't want to wait for the next automatic reconcile.

Don't panic-abort a prune just because an ExternalSecret shows 'DELETE'

Mid-restart, one Application's live sync-diff view showed a DELETE action against knowledge-management-service-external-secret — alarming, since deleting it would break KMS's Vault secret sync. Investigated and confirmed nothing was actually at risk: - The resource's argocd.argoproj.io/tracking-id annotation was correct (matched the current Application) — ruling out the stale-ownership scenario from app-of-apps → ApplicationSet migration. - helm template with current committed values confirmed the chart still renders this resource normally — it wasn't missing from the desired manifest. - The actual cause: a manual force-sync=<timestamp> annotation (added earlier, to force ESO to refresh) isn't part of the Helm chart's template, so ArgoCD's diff sees it as an unexpected extra field and flags the whole resource OutOfSync — which a sync-preview panel can render as if the resource itself were being deleted, when really only that one out-of-band annotation would be reverted. - Confirmed after syncing anyway: the ExternalSecret survived, stayed SecretSynced: True, and kept refreshing normally.

Before approving any prune that looks like it targets a resource you know is real and in-use, check in this order: (1) tracking-id matches the current app, (2) helm template with current values still renders it, (3) the resource's actual health/sync status (kubectl get externalsecret ...) — not just the sync-preview panel. Only proceed once all three check out.

Once every leaf app shows Synced/Healthy (or OutOfSync purely on cosmetic replica-count/annotation drift you've verified is safe), run the post-release verification checklist (Hatchet worker steady-state, Foundry smoke tests, KMS S3 round-trip, end-to-end API call) to confirm everything actually came back healthy, not just Running.

Open items to confirm before executing¶

Exact ArgoCD Application name for frontend — verify with argocd app list before Step 1.
The restart procedure's replica counts are best-guesses from observed state, not confirmed Git-declared values — either check each service's actual replicaCount first, or trust Step 7's ArgoCD sync to correct them immediately rather than relying on the manual numbers.
Confirm the kubectl cnpg plugin is installed on bastion, or use the raw annotation fallback.
Get an explicit go/no-go between each numbered step during actual execution rather than running the whole sequence unattended — the blast radius is a full platform outage.