Release Strategy — core-service, assistant-service, KMS¶

End-to-end runbook for shipping a new build of core-service, assistant-service, or knowledge-management-service into mod-auh1-dev-ask. Written after the 2026-07-01/02 rollout, which surfaced most of the gotchas below the hard way — treat this as the checklist that would have caught them earlier.

Related docs: Mirror Images & Charts (mirroring mechanics + rollback), Assistant Service Integration Tests (detailed test commands).

Why this is more involved than "bump the tag"¶

As of 2026-07-02 all three services are pinned by image digest, not a floating staging tag — see Mirror Images & Charts → Resolving a PP-Reported Commit for why (imagePullPolicy: IfNotPresent + a reused tag name meant ArgoCD never saw a diff and kubelet could silently keep serving stale content). That fixed the deploy-doesn't-happen problem, but it also means every release now has more steps than editing one tag.

Step-by-step release process¶

1. Identify the target build. Usually a commit SHA reported as "PP-stable" (e.g. via Slack). Slack often glues the version tag directly onto the SHA with no separator — split by counting back 40 hex characters (a git SHA is always 40 hex chars).

2. Resolve it to a verified digest. Never trust a hand-typed or previously-recorded digest — confirm it fresh against ACR:

cd ~/harbor-images
CREDS="$(jq -r '.username' acr-creds.json):$(jq -r '.password' acr-creds.json)"
export https_proxy=http://10.96.137.37:3128   # ACR is proxy-only from bastion
REG=ai71uaenprodask71acr01.azurecr.io

skopeo list-tags --creds="$CREDS" docker://$REG/<image-name> | grep -i <short-sha>
skopeo inspect --creds="$CREDS" "docker://$REG/<image-name>:<full-40-char-sha>" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['Digest'])"

Two things that will trip you up here (both hit during this rollout): - The ACR source ref has no project prefix — it's {registry}/{name}, not {registry}/ask/{name}. - skopeo inspect on a multi-arch manifest returns one "Digest" line per architecture if you grep for it naively — only the JSON-parsed top-level Digest field is correct.

3. Update mirror-config.yaml (on the Mac, 096_RKE/scripts/harbor-mirror/ — local-only, not in GitLab; copy it to the bastion, don't push). Add the new digest with a date tag, demote the previous one to staging-prev:

- { name: <image>, source: acr, project: ask, tags: ["staging", "<date>"], digest: "sha256:<new>" }
- { name: <image>, source: acr, project: ask, tags: ["staging-prev", "<old-date>"], digest: "sha256:<old>" }

4. Sync mirror-config.yaml to bastion. This file only lives on the Mac — bastion has its own copy at ~/harbor-images/mirror-config.yaml that does not update automatically. Diff them before mirroring, or you'll mirror against a stale config.

5. Mirror it from bastion.

cd ~/harbor-images
./core42-mirror-image.py --only <image-name> --dry-run
./core42-mirror-image.py --only <image-name>

Confirm the mirror script itself is up to date

skip_existing used to check only whether the destination tag existed in Harbor, not whether it matched the configured digest — meaning re-pinning an already-existing tag name (which is every routine release, since staging is always reused) silently no-op'd with no error. Fixed 2026-07-02 (local commit b1e24f4) to compare digests and re-copy on mismatch. That fix is also Mac-only until copied to bastion's ~/harbor-images/core42-mirror-image.py — check bastion has it before trusting a mirror run.

If the dry-run reports [SKIP] ... already in Harbor for a digest you know is new, don't trust it — verify by pulling the image reference directly (see the migration-job ImagePullBackOff incident this exact bug caused).

6. Update the devops overlay's image.digest. This is the step that's easy to forget now that tags are pinned by digest — updating mirror-config.yaml alone does not change what's deployed. Each service's mod-ask-devops/applications/<service>/values-mod-auh1-dev-ask.yaml has an image.digest field that must be bumped to match:

image:
  tag: "staging"        # label only, unused once digest is set
  digest: "sha256:<new digest>"

7. Commit and push the chart repo (if template changes were needed) and the devops overlay.

8. Sync ArgoCD. Session tokens expire often — always re-login first:

ARGOCD_PASSWORD=$(kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath='{.data.password}' | base64 -d)
argocd login argocd.cl1.sq4.aegis.internal --username admin --password "${ARGOCD_PASSWORD}" --insecure --grpc-web
argocd app sync mod-auh1-dev-ask.ask.<service> --grpc-web

9. Verify the digest is actually live — don't trust "Synced/Healthy" alone:

kubectl -n ask get pods -l app.kubernetes.io/instance=<service> \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\t"}{.status.containerStatuses[*].imageID}{"\n"}{end}'

10. Check for internal changes in the new build before assuming it's a clean rollout. A digest bump can change more than the app's behavior — it changed KMS's worker entrypoints entirely (generic_worker.py → a consolidated singleton_worker.py + WorkerType arg) and broke all 3 worker Deployments with python: can't open file ... No such file or directory, even though the digest pin itself worked correctly. Before trusting a rollout:

kubectl -n ask exec <a-healthy-pod-from-the-new-build> -- find /app -iname "*worker*.py"   # or whatever's relevant

Compare against what the chart's command: fields expect. If they've diverged, the chart's values.yaml needs updating to match the new build's actual entrypoints — same pattern as the KMS fix (ask-knowledge-management-service@90a44e2).

11. Restart the Deployment, not just the migration Job, if there are ConfigMap-driven config changes (model catalogs, TOML config, anything mounted rather than passed as env). ArgoCD syncing a ConfigMap does not guarantee the running pod picks it up: - Kubernetes normally propagates ConfigMap volume updates to the mounted file within ~60s without a restart — but if the app caches the parsed file in memory (e.g. @lru_cache on the load function, as core-service's model_provider/core/parser.py does), only a fresh process picks up the change. - Restarting the migration Job or an unrelated worker does nothing for this — you need to restart the actual Deployment that reads the ConfigMap:

kubectl -n ask rollout restart deploy/<service>
kubectl -n ask rollout status deploy/<service>

Watch for decoy files at similar paths

While debugging this, we spent a while reading /app/src/model_provider/models/models.yml — the file baked into the image — and couldn't figure out why ArgoCD's sync (confirmed correct) never showed up there. The app actually reads APP__MODEL_CONFIG_DIR (/app/config/models/models.yml, the real ConfigMap mount) — a completely different path with the same filename. Before concluding "the sync isn't working," check the app's own config-path env var and confirm which file is genuinely being read:

kubectl -n ask exec deploy/<service> -- env | grep -i CONFIG_DIR
kubectl -n ask exec deploy/<service> -- find / -iname "<config-file>.yml" 2>/dev/null

Hatchet client token — when it breaks and how to fix it¶

The HATCHET__CLIENT_TOKEN each service holds is signed against keys stored in a hatchet-config K8s secret, generated once by Hatchet's setup job (hatchet-admin k8s quickstart). This is not a normal per-release concern — it only breaks when hatchet-config itself gets regenerated, which happens on: - Moving Hatchet to a new namespace (hit this 2026-07-01 during the ask → hatchet namespace migration) - A Hatchet chart redeploy/upgrade that re-runs the setup job fresh

Symptom: workers crash-loop or log failed to register workflow: <name> with grpc._channel._InactiveRpcError: ... UNAUTHENTICATED ... invalid auth token.

Fix — pull the current token from the live hatchet-config and push it to every consumer:

TOK=$(kubectl get secret -n hatchet hatchet-client-config -o jsonpath='{.data.HATCHET_CLIENT_TOKEN}' | base64 -d)

for svc in core-service assistant-service knowledge-management-service; do
  vault kv patch secret/ask/${svc}/auh1-dev HATCHET__CLIENT_TOKEN="$TOK"
done

# ExternalSecret CR names ≠ the target Secret name — find the real name first:
kubectl get externalsecret -n ask | grep -E "core-service|assistant-service|knowledge-management"

kubectl annotate externalsecret -n ask <service>-external-secret force-sync="$(date +%s)" --overwrite
kubectl rollout restart deploy -n ask -l app.kubernetes.io/instance=<service>

Do this proactively for all three services whenever Hatchet's hatchet-config changes — don't wait for each one to crash and surface the same error independently (which is what happened this round: KMS and assistant-service both hit it, hours apart, from the same root cause).

Post-release verification checklist¶

Run these after every release, in rough order of "cheapest to check" first:

Pods actually on the new digest — the imageID check from Step 9 above.
Migration Job succeeded on the new build (kubectl -n ask get job <service>-migrations).
Hatchet worker steady-state — don't grep for the boot banner on a pod that's been up a while (it scrolls off); look for actual step-run activity instead:
```
kubectl -n ask logs deploy/<service>-<worker> --tail=40 | grep -iE "step run|finished step|🪓"
```
Foundry/LLM smoke tests (chat, embeddings, rerank) — see integration-tests.md §5. Always inject the key fresh from the Secret in the test, not the pod's captured env — a Vault key rotation isn't visible to a running pod until it restarts.
KMS S3 round-trip (PUT/GET/LIST/DELETE against Ceph RGW) — 40-kms-s3-wiring.md Step 5B.
Tool/KB functionality — actually trigger a tool call through the real UI while tailing the tooling pod's logs, rather than just checking reachability. Reachability checks proved insufficient on their own to catch the model-catalog issue below.
Model catalog + org-enablement, if any model config changed. This one has two independent layers that both must be correct, and checking only one gives a false negative:
The model exists in the catalog (modelConfig.models.models in the devops overlay) with an id matching exactly what the service is configured to use.
The model is enabled for the org — a separate DB row in core_service.organization_models (org_id, model_id). A catalog entry alone is not sufficient; the frontend's dropdown only shows models enabled for the current org. Query directly to check:
```
kubectl -n ask exec deploy/core-service -- python3 -c "
import asyncio, asyncpg, os
async def main():
    conn = await asyncpg.connect(host='core-pg-rw.ask.svc.cluster.local', port=5432,
        user='core-service', password=os.environ['DATABASE__PASSWORD'], database='core-service')
    rows = await conn.fetch(\"SELECT org_id, model_id, is_default_model FROM core_service.organization_models WHERE model_id = \$1\", '<model-id>')
    for r in rows: print(dict(r))
    await conn.close()
asyncio.run(main())
"
```
Enabling is done via POST /model/enable (@require_org_admin), body {"model_id": "<id>", "default_model_id": "<id>"} — needs a real user JWT, which (as of this writing) is not obtainable from the browser's Network tab, since the admin Model Settings page fetches server-side (Next.js Server Component calling core-service directly, never exposed to the client). Getting a usable JWT for this endpoint is still an open problem as of 2026-07-02 — see the in-progress investigation into calling the app's own OrganizationModelRepository.bulk_enable_organization_models directly as a workaround.

Known gotchas — quick reference¶

Symptom	Cause	Fix
`ImagePullBackOff` / `manifest not found` on a digest you just mirrored	`skip_existing` checked tag existence, not digest match — silently skipped the re-copy	`skopeo copy --preserve-digests` directly, bypassing the script; or use the fixed script (`b1e24f4`+, once synced to bastion)
`argocd` commands fail with `token has invalid claims: token is expired`	Session expired (happens often in this environment)	Re-run `argocd login` before every sync
ExternalSecret force-sync annotate fails with `not found`	The ExternalSecret CR's own name ≠ the target Secret name it produces (`<service>-external-secret` vs `<service>-secret`)	`kubectl get externalsecret -n ask \\| grep <service>` to find the real name
Config change synced by ArgoCD, but pod still shows old values	App caches the parsed config in memory (`@lru_cache` or similar), or you're reading a decoy file at a similar path	Restart the Deployment; verify the actual config-path env var (`APP__*_CONFIG_DIR`) before assuming sync failed
Worker crashes with `can't open file .../<script>.py` after a digest bump	New build changed internal script layout/entrypoints	`find /app -iname "worker.py"` (or similar) on the new image; update chart `command:` to match
`invalid auth token` from a Hatchet worker	`hatchet-config` was regenerated (namespace move, Hatchet redeploy) — old tokens signed against old keys	Pull fresh token from `hatchet-client-config`, patch into every consumer's Vault path, force ESO resync, restart