How to Onboard a Service (air-gap GitOps playbook)¶

The repeatable, end-to-end procedure for bringing a new service onto the air-gapped cluster, distilled from the core-service / assistant-service / KMS / frontend deployments. Worked examples: KMS (37–40), Frontend (41).

The order matters — it's dependency-driven

Most of our deploy failures came from doing a step before its prerequisite. The sequence below is arranged so each step's inputs already exist. The big ones: render before you seed (rendering tells you the exact secret keys), and ESO policy + secret before sync (or ESO 403s and the pod CrashLoops with no secret).

The 13 steps at a glance¶

#	Step	Where
0	Discover the service + write the deploy doc first	Mac (read)
1	Sanitize the chart repo (helm/ only)	Mac
2	Vendor sub-charts (and patch if needed)	Mac
3	Write the env values (air-gap overrides)	Mac
4	Render locally (`helm template`) — reveals exact images + secret keys	Mac
5	Mirror the images to Harbor	bastion
6	Add/update the ESO policy for the new Vault path + sync vault	bastion
7	Seed the Vault secret (all required keys + placeholders)	bastion
8	Create the ArgoCD Application manifest + devops overlay	Mac
9	Push the chart repo + devops	Mac (VPN)
10	Sync the ArgoCD app + validate sync/health	bastion
11	Troubleshoot + record each issue	both
12	Test — functional + lazy scenarios; hardcoded audit	bastion
13	Update the docs + memory	Mac

0. Discover & document first¶

Before touching anything, read the upstream chart and source. You are looking for the things that break air-gap silently:

Images — every image the chart pulls (app + sub-charts + init containers, e.g. weaviate's alpine).
Sub-charts — vendored .tgz or repo deps (Postgres, Redis, Weaviate…).
Required config that crashes if missing — Dynaconf @format {env[...]} (KMS) or a zod schema (frontend app-environment.server.ts). List every key — including for dependencies you won't deploy (they still need placeholder values).
Config model / layers — which model/URL/endpoint comes from L1 TOML (we control), L2 baked catalog, or L3 hardcoded source. See Model References.
Cloud/external hardcodes — run the hardcoded audit grep over src/.
Lazy dependencies — what fails only when used (S3, LLM calls, KMS, voice…). See lazy matrix.
External pre-engagement deps — S3 user, Foundry key, DB cluster, DNS, certs (request early).

Then write the deploy doc (docs/docs/NN-<service>-deploy.md) — the process requires the doc before execution. Rebuild + restart mkdocs after every doc change.

1. Sanitize the chart repo¶

Copy helm/ only into modgpt/mod-ask-<service> (strip src/, CI, app code). This is what ArgoCD pulls. See Repo Sanitization.

2. Vendor sub-charts (and patch if needed)¶

Air-gap can't pull charts at sync time — vendor them into helm/charts/:

cd helm && helm dependency update     # or drop the .tgz in charts/

If a sub-chart needs an air-gap fix that values can't express (e.g. weaviate's CLUSTER_ADVERTISE_ADDR from status.podIP), unpack the .tgz into charts/<name>/ and patch the template — unpacking keeps the patch reviewable in Git. (See KMS §2 memberlist.)

3. Write the env values (air-gap overrides)¶

helm/environments/values-mod-auh1-dev-ask.yaml, baselined on the PP env. Air-gap musts:

Images → harbor.cl1.sq4.aegis.internal/...; imagePullSecrets: [harbor-pull-secret].
Node → nodeSelector: node.kubernetes.io/role: ask71 (+ matching toleration).
Service → ClusterIP; Ingress → className: nginx, cert-manager.io/cluster-issuer: internal-ca-issuer, host under *.mod.auh1.dev.dir.
Models / URLs → override every cloud default to a served model / in-cluster URL (L1 config).
Secrets → ESO externalSecret block (provider vault, dataFrom: extract from the service's Vault path).
Env selector → set it (e.g. KMS APP__ENV) or the override file is ignored.
PSA → no privileged containers under baseline (disable privileged init containers; tune on nodes).

4. Render locally — the validation gate¶

cd helm && helm template <release> . -f environments/values-mod-auh1-dev-ask.yaml >/dev/null && echo OK
# extract the two things you need next:
helm template <release> . -f environments/values-mod-auh1-dev-ask.yaml | grep -E "image: "        # → mirror list
helm template <release> . -f environments/values-mod-auh1-dev-ask.yaml | grep -iE "secretKeyRef|@format|extract" # → secret keys

Rendering must be clean before you mirror or seed — it tells you the exact image list and the exact secret keys.

5. Mirror the images (bastion)¶

Image mirroring runs from the bastion, not the Mac (the proxy that reaches ACR is cluster-internal). On the Mac, edit scripts/harbor-mirror/mirror-config.yaml (add entries, no <digest> placeholder); on the bastion run:

cd ~/harbor-images && ./core42-mirror-image.py --only <name-substring>

Match Harbor paths to the values exactly (incl. nested repos like ask/semitechnologies/weaviate). See [feedback: mirror on bastion]. ⚠️ don't forget sub-chart + init-container images (alpine, etc.).

6. Add/update the ESO policy — BEFORE sync¶

ESO authenticates to Vault via a role whose policy eso-ask-push lists each ask/<service>/* path explicitly. A new service path is not covered → ESO gets 403 and the secret never syncs → the pod CrashLoops. Add it in devops applications/cluster-addons/vault/manifests/vault-bootstrap-config.yaml:

path "secret/data/ask/<service>/*"     { capabilities = ["create", "update", "read"] }
path "secret/metadata/ask/<service>/*" { capabilities = ["read", "list"] }

then apply: argocd app sync vault --grpc-web → verify vault policy read eso-ask-push | grep <service>.

7. Seed the Vault secret¶

Seed every required key from step 4 — including placeholders for dependencies you won't deploy (so @format/zod validation passes):

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal VAULT_SKIP_VERIFY=true
vault kv put secret/ask/<service>/auh1-dev KEY1="..." KEY2="$(openssl rand -hex 32)" ...

Rules: real values where you have them; shared values (e.g. CORE_SERVICE__JWT_SECRET) copied from the owning service's Vault path; placeholders ≥32 chars; never short literals. Cross-service keys must match (e.g. Weaviate AUTHENTICATION_APIKEY_ALLOWED_KEYS = client VECTOR_DB__API_KEY). External creds (S3, Foundry) are patched in when the owning team delivers — see pre-engagement reqs.

8. Create the ArgoCD Application + overlay¶

Fill the stub in devops environment/auh1/dev/02-argocd-application/ask/argocd-application-<service>.yaml (multi-source: chart repo path: helm + $values overlay from devops; project: mod-auh1-dev-ask.ask; namespace: ask; syncOptions: CreateNamespace=true, ServerSideApply=true). Add ignoreDifferences for the ESO ExternalSecret (managedFieldsManagers: [external-secrets]). Create the applications/<service>/values-mod-auh1-dev-ask.yaml overlay (image tag).

9. Push (VPN)¶

cd modgpt/mod-ask-<service> && git push -u origin main   # new repo: push creates it
cd modgpt/mod-ask-devops    && git push

10. Sync & validate¶

kubectl annotate application -n argocd mod-auh1-dev-ask.ask.<service> argocd.argoproj.io/refresh=hard --overwrite
argocd app sync mod-auh1-dev-ask.ask.<service> --grpc-web 2>/dev/null || true
kubectl get application -n argocd mod-auh1-dev-ask.ask.<service> -o custom-columns='SYNC:.status.sync.status,HEALTH:.status.health.status'
kubectl get pods -n ask | grep -i <service>

11. Troubleshoot & record¶

Expect issues; record each in troubleshooting.md. The recurring ones we hit:

Symptom	Cause	Fix
ExternalSecret `SecretSyncedError` 403	ESO policy missing the path	step 6
Pod CrashLoop, `DynaconfFormatError` / `ZodError: Required`	a required `@format`/zod key missing	seed it (placeholder ok)
`ImagePullBackOff`	image (or init-container image) not mirrored / wrong Harbor path	mirror it; match path to values
`FailedCreate … violates PodSecurity "baseline": privileged`	privileged (init) container vs namespace PSA	disable it; set node-level sysctl
`could not init cluster state / No private IP`	service on non-RFC1918 (100.x) pod CIDR	set advertise addr from `status.podIP`
App `OutOfSync` forever on ExternalSecret	ESO mutates it post-apply	`ignoreDifferences` + `managedFieldsManagers`
`InvalidAccessKeyId` on S3	placeholder/missing external creds	patch real creds when team delivers (lazy — not a deploy blocker)
ArgoCD `Synced` but config change absent	edited a shadowed valueFile — the `$values` overlay (last valueFile) replaces the chart env file's `configToml`	edit the last file in `spec.sources[].helm.valueFiles` (often the devops overlay); confirm with a dummy marker grep on the configmap
Config change in configmap but not in pod	`subPath` mount — no live update	`kubectl rollout restart deploy/<svc>` (subPath needs a real restart)
New commit pushed but ArgoCD applies old render	`force-sync` reuses the git cache	`argocd.argoproj.io/refresh=hard` then sync
Org/auth 500 `Foundry is not configured` (core-service)	eager `get_foundry_client()` needs `[foundry]` (5 fields)	add `[foundry]` + `FOUNDRY_ADMIN_KEY` (single-underscore — `FOUNDRY__` shadows the Dynaconf section); see troubleshooting

12. Test — functional + lazy¶

Functional: each hard dependency reachable (DB, Redis, Hatchet, Weaviate, backends) — see the service's integration tests.
Lazy: the dependencies that pass health but fail only when used (S3, LLM calls, KMS, voice). Run the Lazy Dependency Test Matrix. Remember: a gateway test (e.g. /v1/rerank works) ≠ the app uses it — check the config layer.
Hardcoded audit: re-run the air-gap hardcoded audit grep on the new repo.

Test inside the pod with httpx, not curl

The app images ship no curl. Use python3 -c "import httpx; ..." (and kubectl exec -i for heredocs — without -i, stdin isn't forwarded and the script runs empty).

13. Update the docs + memory¶

Service deploy page; cross-link from LLM → Service Config Mappings, the lazy matrix, the hardcoded audit, and troubleshooting.
Update the deployment-state tracker and write a memory note of the fix chain (so the next onboarding starts ahead).

Quick dependency map (why this order)¶

flowchart TB
    D[0 discover + doc] --> S[1-3 chart + values]
    S --> R[4 render]
    R --> M[5 mirror images]
    R --> P[6 ESO policy]
    P --> SE[7 seed secret]
    R --> A[8 argocd app]
    M --> PUSH[9 push]
    SE --> PUSH
    A --> PUSH
    PUSH --> SY[10 sync + validate]
    SY --> T[11 troubleshoot/record]
    T --> TE[12 test functional + lazy]
    TE --> DOC[13 update docs]

Render (4) feeds both the mirror list (5) and the secret keys (6/7). Policy (6) must precede seed (7) and both must precede sync (10), or ESO 403s and the pod has no secret.