ASK Assistant Service — Deployment Guide (Air-Gapped)

Deployment plan for ask-assistant-service on the air-gapped Core42 cluster. Unlike core-service (a single API + Zitadel), assistant-service is a multi-service package with several new infrastructure dependencies (vector DB, LLM gateway, optional MLflow). Read this whole page before starting — several dependencies are not yet deployed.

What this service is

Four FastAPI apps in one Python package + an MCP gateway + workers, split from a monolith:

Workload	Port	Image	Purpose
assistants (main)	8002	assistant-service	assistants, conversations, triggers, webhooks; owns the DB (Alembic)
intelligence	8003	reuses main image (cmd)	RAG, agents, LangGraph orchestration
tooling	8001	reuses main image (cmd)	MCP tools (gmail/calendar/KB), OAuth, connectors
mcp-gateway	8080	assistant-service-mcp-gateway	per-user MCP server gateway
generic-worker / notifications-worker	—	reuses main image	Hatchet async + Redis-stream notifications

Layer rule: assistants → intelligence → tooling → shared. Intelligence/tooling/workers reuse the main image (command/args select the app); only the gateway is a separate image.

---

1. Prerequisites

• [x] core-service deployed + healthy — assistant-service calls it (core_service.url, services_base_url) and shares its encryption key (see Secrets). This is a hard dependency.
• [x] Cluster add-ons running: ArgoCD, ESO, Stakater Reloader, CNPG operator, cert-manager, Vault
• [x] Worker nodes labelled node.kubernetes.io/role=ask71 (the chart's nodeSelector convention)
• [x] Harbor + GitLab reachable; air-gap mirror workflow available (scripts/harbor-mirror)
• [x] Vault unsealed; you can vault kv put secret/ask/assistant-service/auh1-dev/*
• [x] An ask-namespace Harbor pull secret (harbor-pull-secret) — same as core-service
• [x] DNS / ingress host plan: api.ask.mod.auh1.dev.dir (or a dedicated assistant-service.ask...)

New dependencies that must exist first (see §2) — in plain terms

These don't exist from the core-service deployment and must be stood up:

• Qdrant — a vector database: stores text as numeric "embeddings" and finds similar items by meaning. The intelligence layer uses it for agent memory + RAG. Required.
• Foundry / LLM endpoint — an OpenAI-compatible gateway (litellm-style) that proxies to the real LLMs (Claude/GPT + embedding + reranker). Every AI call goes through it. Required.
• Redis with RediSearch + ReJSON modules — plain Redis is a cache; RediSearch adds search and ReJSON adds JSON storage. The chart loads both module files, so the normal bitnami/redis image won't start. Needs a Redis-Stack image. Required.
• MLflow — records AI orchestration runs for debugging/telemetry. Optional (disable to start).

Daytona (sandboxed code execution) is out of scope — not being deployed. Disable the code-executor tool and omit the daytona config block.

---

2. Dependencies

2a. Already deployed (reuse)

Dependency	In-cluster target	Notes
core-service	http://core-service / core-service.ask…	API + IAM proxy; shared encryption key
Zitadel (IAM)	via core-service	no direct Zitadel onboarding for assistant-service
CNPG operator	cnpg-system	provisions the Postgres cluster
Hatchet	hatchet-engine:7070	async workers (token needed)
Harbor / GitLab / Vault / ESO / Reloader	—	platform

2b. Must be deployed/provisioned for assistant-service

Dependency	Why	Status	Action
PostgreSQL (DB assistant-service, schema assistant_service)	main app storage	✅ role exists on core-pg	reuse core-pg (role + password already provisioned); chart postgresInit just CREATE DATABASE. See §6
Redis + RediSearch + ReJSON	chat/memory cache; chart loadmodule redisearch.so/rejson.so	❌ blocker	the bitnami redis image has no modules — must use a Redis-Stack image (see §7 risk)
Qdrant (vector DB)	memory / mem0 feature only (intelligence.qdrant)	⚪ OPTIONAL — omitted for air-gap	not needed for bring-up; omit qdrant + memory_ingestion config. Deploy only for long-term memory — see Qdrant Deployment
Foundry / LLM gateway (foundry.endpoint + reranker)	intelligence LLM calls	🟡 deferred — dummy values for bring-up	app deploys without it; AI features inactive until integrated. See Foundry / LLM Endpoint
Knowledge Management Service (kms.base_url)	tooling knowledge-base MCP	❌ not deployed	separate service (ask-knowledge-management-service)
Embedding model (BAAI/bge-m3, dims 1024) + azure/gpt-5-nano	memory ingestion	⚠️	served via Foundry; confirm models available in the air-gap gateway
MLflow (intelligence.mlflow)	trace logging	optional	can disable; needs INTELLIGENCE__MLFLOW__TOKEN if enabled
~~Daytona~~	code-executor tool	out of scope	not deploying — disable the tool, omit the daytona config block

2c. Deployment topology — subchart vs separate app

Dependency	Topology	Own ArgoCD app?
Redis	subchart (Chart.yaml → redis, condition: redis.enabled) — vendored like zitadel	❌ deploys with assistant-service
Qdrant	external (URL intelligence.qdrant.url) — not in Chart.yaml	✅ its own app
Foundry / LLM	external (URL foundry.endpoint)	✅ own / existing endpoint
MLflow (optional)	external (URL)	✅ own (if used)

Chart.yaml declares only the redis subchart:

- name: redis
  version: 22.0.7
  repository: oci://registry-1.docker.io/bitnamicharts
  condition: redis.enabled

Everything else (Qdrant, Foundry, MLflow) is a separate deployment that assistant-service reaches by URL — so each needs its own chart/app, and only their endpoints + secrets go into our config.

Redis may become a separate app anyway (modules — see §7)

Because the bitnami redis subchart can't easily ship RediSearch/ReJSON, you'll likely either override its image to Redis-Stack or set redis.enabled: false and run Redis-Stack as its own app, pointing redis.host at it. The separate-app route is usually cleaner — decide in §7.

---

3. Images to mirror

Image	Source	Harbor dest	Used by
assistant-service	ACR	ask/assistant-service:staging	main + intelligence + tooling + workers
assistant-service-mcp-gateway	ACR	ask/assistant-service-mcp-gateway:staging	mcp-gateway
redis with modules	—	dockerhub/redis/redis-stack-server (or equivalent)	redis subchart (override)
postgres:16-alpine (only if using postgresInit)	docker.io	dockerhub/postgres:16-alpine	DB-init job
Bitnami redis chart deps (exporter, os-shell)	docker.io	bitnamilegacy/*	redis subchart

The two ACR app images are already in scripts/harbor-mirror/mirror-config.yaml (by digest). See §8 for the additions.

---

4. Secrets & TOML config

assistant-service uses the same pattern as core-service: non-secret config in an inline TOML (envConf.configToml → ConfigMap assistant-service-config-toml → /app/config/override.toml), secrets injected as env vars from assistant-service-secret, referenced in the TOML via @format {env[KEY]}.

4a. Push-secret vs ExternalSecret — what applies here

All secrets are PRE-SEEDED in one Vault path — no PushSecret/generator

The chart's templates/externalsecrets.yaml shipped only azure / aws branches. We add a vault branch (sanitization step) so the chart generates assistant-service-secret itself from a single Vault KV path — cleaner than a hand-written ExternalSecret.

Add to templates/externalsecrets.yaml (after the aws branch):

{{- else if eq $config.provider "vault" }}
# HashiCorp Vault (KV v2) — pull ALL keys from one path into the target secret.
dataFrom:
  - extract:
      key: {{ $config.secretKey }}     # e.g. ask/assistant-service/auh1-dev
{{- end }}

Enable it in the env values:

externalSecrets:
  assistantService:
    create: true
    provider: vault
    secretStore: { kind: ClusterSecretStore, name: vault-backend }
    target: { name: assistant-service-secret }
    secretKey: ask/assistant-service/auh1-dev

Mechanism	Used?	Why
Chart externalsecrets.yaml + vault branch	✅	ESO extract pulls all keys from one Vault path
Separate hand-written ExternalSecret	❌	no longer needed — the chart does it
ESO PushSecret + generator	❌	manual seed instead (air-gap)

Secrets live as ONE multi-key Vault secret (not one path per key) — see §5. One write = the path exists = no KV-v2 brand-new-path 404, and one extract maps it all in.

4b. The config TOML (what's non-secret)

Per-environment configToml wires every dependency (URLs are non-secret, in Git): app.base_url, core_service.url/services_base_url, database.host, foundry.endpoint, storage.cdn_base_url, kms.base_url, intelligence.qdrant.url, intelligence.mlflow.uri, redis.host, assistants.*, notifications_worker.*, image_analysis.model, etc. Secrets inside it are @format {env[KEY]} placeholders only.

4c. Required env vars — the Dynaconf env-switcher gotcha ⚠️

Dynaconf needs the env-switcher var set on every workload, or config import fails at startup with DynaconfFormatError: Dynaconf can't interpolate variable because '<VAR>'. The var name and the chart injection key differ per service — always check env_switcher= in the service's own config module before assuming:

	core-service	assistant-service
Dynaconf env_switcher	APP__ENV (double _)	APP_ENV (single _)
default.toml env =	@format {env[APP__ENV]}	@format {env[APP_ENV]}
Injected via (chart key)	top-level extraEnv	envConf.extraEnv

For assistant-service, the chart injects .Values.envConf.extraEnv into all workloads (deployments, intelligence, tooling, workers, migration). Set there:

envConf:
  extraEnv:
    - name: APP_ENV
      value: "mod-auh1-dev-ask"        # = the [mod-auh1-dev-ask.*] section + @format {env[APP_ENV]}
    - name: APP_ENVIRONMENT
      value: "mod-auh1-dev-ask"        # settings.py env-file selection
    - name: DATABASE__PASSWORD          # NOT in assistant-service-secret — separate CNPG secret
      valueFrom:
        secretKeyRef: { name: assistant-service-db-password, key: password }

Every @format {env[...]} in the baked default.toml must resolve (Dynaconf evaluates them all in the active merged config). For assistant-service:

Var(s)	Source
APP_ENV, APP_ENVIRONMENT, DATABASE__PASSWORD	envConf.extraEnv (above)
REDIS_PASSWORD, AI71_AUTH_STATE_SECRET, CORE_SERVICE_ENCRYPTION_KEY, CORE_SERVICE__JWT_SECRET, IAM__MACHINE_TOKEN, AI71_FOUNDRY_KEY, HATCHET__CLIENT_TOKEN, INTELLIGENCE__MLFLOW__TOKEN	assistant-service-secret (envFrom)
DAYTONA__API_KEY	TOML overrides to @null (Daytona out of scope)
AMPLITUDE_API_KEY, VAR_NAME	commented out in default.toml — not evaluated

4d. Three config systems — and the env vars each service needs ⚠️

assistant-service has three independent config systems that read config differently:

System	Reads from	Used by
Dynaconf Config	the TOML (override.toml) + env	assistants (main) — Hatchet, Redis, DB, foundry URL
pydantic-settings ToolsSettings	env vars ONLY (not the TOML)	tooling service
pydantic-settings AssistantsSettings	env vars ONLY (not the TOML)	main assistants AND intelligence service

A value in the TOML does not reach ToolsSettings or AssistantsSettings — they must also be real env vars. Required fields with no default → must be in envConf.extraEnv:

envConf:
  extraEnv:
    # ToolsSettings (tooling) — required, no default
    - name: AI71_FOUNDRY_ENDPOINT          # ToolsSettings.foundry_base_url (alias)
      value: "http://foundry-litellm.foundry.svc.cluster.local:4000/v1"
    - name: KMS_BASE_URL                    # ToolsSettings.kms_base_url
      value: "http://knowledge-management-service.ask.svc.cluster.local"
    # AssistantsSettings (main assistants + intelligence) — required, no default
    - name: CORE_SERVICE_URL               # AssistantsSettings.core_service_url
      value: "http://core-service.ask.svc.cluster.local/api/v1"

(AI71_FOUNDRY_KEY for ToolsSettings.foundry_api_key comes from assistant-service-secret.)

AssistantsSettings is imported by TWO workloads — main and intelligence

src/shared/config/assistant_settings.py is imported by both src/assistants/app.py (main, port 8000) and src/intelligence/app.py (intelligence, port 8003). Missing CORE_SERVICE_URL crashes BOTH. Since envConf.extraEnv applies to all workloads via the chart, one entry fixes both. If you patch with kubectl set env, target all deployments — not just intelligence:

kubectl set env deploy -n ask \
  -l app.kubernetes.io/instance=assistant-service \
  CORE_SERVICE_URL="http://core-service.ask.svc.cluster.local/api/v1"

How to spot a new "field required" crash: gunicorn worker exits with code 3 during import. Traceback ends in pydantic_settings/main.py → ValidationError: N validation error for <ClassName> — field_name — Field required. The env var name is the field name uppercased (core_service_url → CORE_SERVICE_URL). Add it to envConf.extraEnv and redeploy.

4e. Deferred / dummy / disabled for air-gap

Decisions to bring the app up without the not-yet-available or unreachable dependencies. These let the app boot; the corresponding feature stays inactive (fails only when invoked / never invoked).

Item	Decision	How
Foundry / LLM	dummy endpoint	AI71_FOUNDRY_ENDPOINT = http://foundry-placeholder.foundry.svc.cluster.local/v1 + foundry.endpoint in TOML. AI calls (chat/embeddings/RAG) fail only when invoked — wire real Foundry later.
KMS	URL set, service optional	KMS_BASE_URL points at the (possibly-not-deployed) KMS service; calls are lazy.
Hatchet token	⚠️ REAL token required — NOT a placeholder	the workers build the Hatchet client at import and ClientConfig validates it's a JWT — a random placeholder (rand -hex 32) crashes them at startup. Pull the real JWT from the cluster's hatchet-client-config secret (see below). Also requires HATCHET_CLIENT_HOST_PORT env override + Hatchet engine grpcInsecure=t — see §4e note.
Sentry	disabled	No [app.sentry] block → default.toml sentry = "@null" → config.app.sentry is None → init_sentry() returns early. Sentry.io is unreachable air-gapped — leave the block out; do not add a dsn.
MLflow	disabled	omit intelligence.mlflow block and don't seed INTELLIGENCE__MLFLOW__TOKEN (else partial-config validation error).
Qdrant / memory	omitted	omit qdrant + memory_ingestion config.
Daytona	disabled	daytona.enable_code_execution = false, daytona.api_key = "@null".
Amplitude	off	the amplitude_* keys stay commented in default.toml — not evaluated.

Hatchet gRPC: two requirements for in-cluster workers

Workers use client_tls_strategy = "none" (plaintext gRPC) and connect to the internal service via HATCHET_CLIENT_HOST_PORT=hatchet-engine.ask.svc.cluster.local:7070. This works only if the Hatchet engine is also in plaintext mode:

1. HATCHET_CLIENT_HOST_PORT — set in envConf.extraEnv (already done, §4c). Overrides the external broadcast address embedded in the JWT token so in-cluster pods don't try to connect to hatchet.ask.mod.auh1.dev.dir:443.

2. grpcInsecure: "t" on the Hatchet engine — set in applications/hatchet/values-mod-auh1-dev-ask.yaml (sharedConfig.grpcInsecure: "t"). Without this the engine demands TLS on 7070, drops plaintext clients with Socket closed. The NGINX ingress uses backend-protocol: GRPC (plaintext backend) so the external path is unaffected. See troubleshooting → Hatchet workers: Socket closed (TLS mismatch).

Air-gap rule of thumb for external-SaaS telemetry/integrations

Sentry, Amplitude, MLflow, Daytona, and the GitLab/Linear connectors all reach external SaaS that the air-gapped cluster cannot contact. Leave them off (omit the block / @null / don't seed the token). They're all optional in the schemas, and turning them on would only produce timeouts or validation errors. Wire only the in-cluster dependencies (core-service, core-pg, redis, Hatchet, and eventually Foundry/KMS).

---

5. Secret list + pre-seed commands

Seed all of these as keys of ONE Vault secret before the first sync (the vault branch uses ESO extract, which reads every key from one path). Path: secret/ask/assistant-service/auh1-dev.

DATABASE__PASSWORD is NOT in this list

The DB password already exists (secret assistant-service-db-password, Vault ask/postgres/assistant-service--DATABASE--PASSWORD). Map it into the app via secretKeyRef in the env values — don't duplicate it into the assistant-service Vault path.

Vault key	Purpose	Shared w/ core-service?	Generate / source
REDIS_PASSWORD	Redis auth (chart auth.existingSecretPasswordKey)	—	openssl rand -hex 32
AI71_AUTH_STATE_SECRET	OAuth state signing (connector flows)	—	openssl rand -hex 32
CORE_SERVICE_ENCRYPTION_KEY	Fernet — decrypts shared connector tokens	✅ YES — must equal core-service SECURITY__MASTER_ENCRYPTION_KEY	copy core-service's value
IAM__MACHINE_TOKEN	M2M calls to core-service IAM	⚠️ should be a real core-service machine token (random placeholder until auth is exercised)	placeholder rand -hex 32; reconcile later
CORE_SERVICE__JWT_SECRET	assistant-service's internal JWTs (IAMClient)	❌ NO — core-service has no JWT secret; this is assistant-service-local	openssl rand -hex 32 (final, not a placeholder)
AI71_FOUNDRY_KEY	LLM gateway API key	—	placeholder until Foundry wired (rand -hex 32)
HATCHET__CLIENT_TOKEN	Hatchet async (workers)	—	⚠️ real JWT required — copy from the cluster's hatchet-client-config secret (key HATCHET_CLIENT_TOKEN). A random placeholder crashes the workers. See Hatchet → Getting the worker token

HATCHET__CLIENT_TOKEN is NOT a safe placeholder

Unlike the other deferred secrets, the Hatchet workers build the client at import and the SDK rejects a non-JWT token, crashing the pod. Wire the real token before the workers run:

TOK=$(kubectl get secret -n ask hatchet-client-config -o jsonpath='{.data.HATCHET_CLIENT_TOKEN}' | base64 -d)
vault kv patch secret/ask/assistant-service/auh1-dev HATCHET__CLIENT_TOKEN="$TOK"
kubectl annotate externalsecret -n ask assistant-service-secret force-sync="$(date +%s)" --overwrite
kubectl rollout restart deploy -n ask -l app.kubernetes.io/instance=assistant-service

(Reuses the cluster's existing default-tenant token — same Hatchet instance core-service uses.)

Two seed rules learned the hard way

1. Placeholders must be ≥32 chars. Several fields enforce min_length (e.g. CORE_SERVICE__JWT_SECRET ≥32). Never seed a short literal like "placeholder" — use openssl rand -hex 32 (64 chars). Format-specific keys still need their exact form (ZITADEL_MASTERKEY = 32 bytes, Fernet = urlsafe-b64). See Troubleshooting.
2. Don't seed a token for an optional block you're leaving disabled. Seeding INTELLIGENCE__MLFLOW__TOKEN turned optional intelligence.mlflow (default @null) into a partial dict → 5 "field required" errors for uri/experiment_name/… MLflow is omitted in air-gap, so do NOT seed INTELLIGENCE__MLFLOW__TOKEN — mlflow stays None and validates.

Which secrets are genuinely shared with core-service

Only CORE_SERVICE_ENCRYPTION_KEY must match core-service exactly (Fernet — shared connector token decryption). IAM__MACHINE_TOKEN should become a real core-service machine token when you wire service-to-service auth. CORE_SERVICE__JWT_SECRET is not shared despite its name — it's assistant-service-internal (the IAMClient); core-service has no JWT-signing secret.

QDRANT_* omitted (air-gap)

QDRANT_SECRET / QDRANT_BASIC_PASSWORD are not seeded — the air-gap deployment runs without Qdrant (memory/mem0 off). Add them only if you later deploy Qdrant and enable memory_ingestion.

Omitted in air-gap — external-SaaS connector secrets

GITLAB_WEBHOOK_OAUTH_CLIENT_SECRET and LINEAR_WEBHOOK_OAUTH_CLIENT_SECRET are not needed. They authenticate the OAuth/webhook flow to gitlab.com / api.linear.app — external SaaS that is unreachable from the air-gapped cluster, so those connectors can't function here. The schema marks them Optional everywhere (str | None); they only error if a user actively uses that connector. Omit them — the GitLab/Linear connectors simply won't be offered.

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
CS=secret/ask/core-service/auh1-dev   # to copy the shared encryption key

# ONE secret, all keys (ESO `extract` reads them all). Re-running overwrites — fine.
# DATABASE__PASSWORD is NOT here — it's the existing assistant-service-db-password secret.
# Every placeholder is ≥32 chars (rand -hex 32) to satisfy schema min_length.
vault kv put secret/ask/assistant-service/auh1-dev \
  REDIS_PASSWORD="$(openssl rand -hex 32)" \
  AI71_AUTH_STATE_SECRET="$(openssl rand -hex 32)" \
  CORE_SERVICE_ENCRYPTION_KEY="$(vault kv get -field=value $CS/SECURITY__MASTER_ENCRYPTION_KEY 2>/dev/null || echo PASTE_CORE_SERVICE_FERNET_KEY)" \
  CORE_SERVICE__JWT_SECRET="$(openssl rand -hex 32)" \
  IAM__MACHINE_TOKEN="$(openssl rand -hex 32)" \
  AI71_FOUNDRY_KEY="$(openssl rand -hex 32)" \
  HATCHET__CLIENT_TOKEN="$(openssl rand -hex 32)"
  # INTELLIGENCE__MLFLOW__TOKEN omitted — MLflow off; seeding it forces partial mlflow validation
  # QDRANT_SECRET / QDRANT_BASIC_PASSWORD omitted — air-gap (no Qdrant / memory off)
  # GITLAB/LINEAR webhook OAuth secrets omitted — external SaaS, unreachable in air-gap (Optional in schema)

vault kv get secret/ask/assistant-service/auh1-dev      # verify all keys present

Secrets are managed MANUALLY in Vault (no automation)

There is no ESO PushSecret or generator for assistant-service — every value is seeded and updated by hand in Vault, and ESO (provider: vault, extract) just mirrors the path into the assistant-service-secret K8s secret. So whenever a real value becomes available (real Foundry key, Hatchet token, core-service machine token, etc.), you manually update Vault, then refresh the ExternalSecret and restart the pods.

Update one key (preserve the rest) — use vault kv patch, NOT vault kv put (a full put rewrites everything and regenerates the random values, breaking redis auth):

vault kv patch secret/ask/assistant-service/auh1-dev HATCHET__CLIENT_TOKEN="<real-token>"
# propagate to the cluster:
kubectl annotate externalsecret -n ask assistant-service-secret force-sync="$(date +%s)" --overwrite
kubectl rollout restart deploy -n ask -l app.kubernetes.io/instance=assistant-service

The K8s secret refreshes on the ExternalSecret's refreshInterval (1m) anyway, but the force-sync makes it immediate; the rollout restart is needed because env vars are read at pod start (Reloader covers config/secret changes if enabled, otherwise restart manually).

CORE_SERVICE_ENCRYPTION_KEY must match core-service

assistant-service decrypts connector/OAuth tokens that core-service encrypted with its Fernet SECURITY__MASTER_ENCRYPTION_KEY. If they differ, shared-connector decryption fails at runtime. Copy the exact value — do not generate a new one.

Also add a Vault policy grant for the new path (same as core-service did) — extend eso-ask-push in vault-bootstrap-config.yaml:

path "secret/data/ask/assistant-service/*"     { capabilities = ["create","update","read"] }
path "secret/metadata/ask/assistant-service/*" { capabilities = ["read","list"] }

---

6. Database requirements

Already provisioned — reuse core-pg (no new cluster)

The assistant-service role already exists on the shared core-pg CNPG cluster (applications/ask-db/core/core-pg-cluster.yaml → managed.roles), with its password in Vault ask/postgres/assistant-service--DATABASE--PASSWORD → K8s secret assistant-service-db-password. Don't create a new cluster.

Item	Value
Cluster	core-pg (existing) — database.host = core-pg-rw.ask.svc.cluster.local, port 5432
Role / user	assistant-service — exists (CNPG managed role), password secret assistant-service-db-password
Database	assistant-service — not yet created → the chart postgresInit job must CREATE DATABASE (core-pg comment: "service init jobs handle CREATE DATABASE + GRANT")
Schema	assistant_service — Alembic auto-creates it (no init-job schema step, unlike core-service)
Migrations	chart migration-job (sync-wave 1) and app startup (FastAPI lifespan)

Chart settings:

postgresql:
  enabled: false                 # reuse core-pg, do NOT create a cluster
postgresInit:
  enabled: true                  # creates the assistant-service DB on core-pg
  connection: { host: core-pg-rw.ask.svc.cluster.local, port: "5432", adminUsername: postgres }
  db: { name: assistant-service, user: assistant-service }
  secrets:
    adminPasswordSecretName: core-pg-superuser            # existing
    userPasswordSecretName: assistant-service-db-password  # existing
  job:
    image: harbor.cl1.sq4.aegis.internal/ghcr/cloudnative-pg/postgresql:16.8-bookworm
    pgSslMode: disable

DATABASE__PASSWORD for the app is the same assistant-service-db-password secret — map it via secretKeyRef in the env values (no need to duplicate it into the assistant-service Vault path).

---

7. S3 / object storage requirements

Item	Value
Config	storage.cdn_base_url (public CDN) + tooling storage (src/config/schemas/tooling/storage.py)
Backend	Ceph RGW (s3.cl1.sq4.aegis.internal), same as core-service
Bucket	dedicated ask-assistant-service (avatars, generated assets, uploads)
Creds	RGW S3 user → Vault → assistant-service-secret → STORAGE__* env (mirror the core-service Ceph S3 wiring doc)

Redis modules — use bitnami redis:8.2.1 (no custom image)

The chart loads RediSearch + ReJSON (loadmodule .../redisearch.so / rejson.so).

Redis 8.2.1 bundles the modules — the simple path

bitnamilegacy/redis:8.2.1-debian-12-r0 runs with the chart's loadmodule config because Redis 8.0+ ships RediSearch + RedisJSON in core and bitnami packages the .so files at /opt/bitnami/redis/lib/redis/modules/. So no custom image and no redis.enabled=false — just use the 8.2.1 image (already mirrored to Harbor). This is what values-mod-auh1-dev-ask.yaml does:

redis:
  enabled: true
  global:
    imagePullSecrets: [{ name: harbor-pull-secret }]
    security: { allowInsecureImages: true }   # non-docker.io mirror (bitnami#30850)
  image:
    registry: harbor.cl1.sq4.aegis.internal
    repository: bitnamilegacy/redis
    tag: 8.2.1-debian-12-r0

Validate after deploy: redis-cli MODULE LIST → search + ReJSON.

The custom-image route below is only needed if you're ever pinned to Redis 7.x (modules separate). For this air-gap deployment, ignore it — use 8.2.1.

Fallback: custom image for Redis 7.x (not needed here)

Keep the subchart and override the image with a build = bitnami redis **+ the two `.so` modules copied to the bitnami module path** (a raw swap to `redis/redis-stack-server` breaks the bitnami startup scripts).

1. Build the image (docker/redis-bitnami-modules/Dockerfile):

# modules from the matching redis-stack version (match the redis major: 7.x)
FROM redis/redis-stack-server:7.4.0-v3 AS modules
FROM bitnamilegacy/redis:7.4.0-debian-12-r0
USER root
COPY --from=modules /opt/redis-stack/lib/redisearch.so /opt/bitnami/redis/lib/redis/modules/redisearch.so
COPY --from=modules /opt/redis-stack/lib/rejson.so     /opt/bitnami/redis/lib/redis/modules/rejson.so
USER 1001

Check Redis 8 first — it may bundle the modules already

Redis 8.0+ ships RediSearch + RedisJSON in core, so if you run the bitnami chart's default Redis 8.x image, the modules may already be present — in which case you don't need a custom image at all, only to drop/adjust the loadmodule lines. Verify on a throwaway pod:

kubectl run redis-check --rm -it --image=harbor.cl1.sq4.aegis.internal/bitnamilegacy/redis:8.2.1-debian-12-r0 \
  --restart=Never -- redis-server --version
# start it and check: redis-cli MODULE LIST  → look for "search" / "ReJSON"

If Redis 8.x already lists them → set redis.image to the 8.x tag and remove the loadmodule lines (cleanest). Only build the custom image (above) if you're pinned to Redis 7.x, where the modules are separate. Either way, the .so version must match the Redis server version.

2. Push to Harbor (it's a built image, not a mirror):

docker build -t harbor.cl1.sq4.aegis.internal/ask/redis-bitnami-modules:7.4.0 docker/redis-bitnami-modules/
# (or skopeo/buildah in the air-gap build host) → push to the ask project

3. Override in the assistant-service env values (the commonConfiguration paths already match):

redis:
  enabled: true
  fullnameOverride: assistant-service-redis
  image:
    repository: harbor.cl1.sq4.aegis.internal/ask/redis-bitnami-modules
    tag: "7.4.0"
  architecture: standalone
  commonConfiguration: |
    loadmodule /opt/bitnami/redis/lib/redis/modules/redisearch.so
    loadmodule /opt/bitnami/redis/lib/redis/modules/rejson.so
    appendonly yes
    save ""
  auth:
    existingSecret: assistant-service-secret
    existingSecretPasswordKey: REDIS_PASSWORD

4. Validate after deploy:

kubectl exec -n ask assistant-service-redis-master-0 -- redis-cli -a "$REDIS_PASSWORD" MODULE LIST
# expect entries for "search" (RediSearch) and "ReJSON"

Mirror prerequisites

Both base images must be in Harbor first: redis/redis-stack-server:7.4.0-v3 (added to mirror-config.yaml) and a bitnamilegacy/redis:7.4.0-debian-12-r0 tag — add that tag to the bitnamilegacy block if not present.

---

8. Mirror config updates (what to add)

Applied in scripts/harbor-mirror/:

• mirror-config.yaml (images): app images already present (by digest). Redis = bitnamilegacy/redis:8.2.1-debian-12-r0 (already mirrored — Redis 8 bundles the modules; no Redis-Stack/custom image needed). postgres:16-alpine already present for any DB-init fallback.
• helm-charts/mirror-charts.yaml: bitnami redis 22.0.7 chart listed.
• helm-charts/gitlab-charts.yaml: redis 22.0.7 added to the GitLab push list (so the chart vendors it like zitadel).
• qdrant/qdrant + redis/redis-stack-server were added earlier but are not needed for the air-gap bring-up (Qdrant omitted, Redis 8 has modules) — harmless to leave.

---

9. Deployment order (once dependencies exist)

flowchart TB
    A["Mirror images + redis-stack"] --> B["Seed Vault + policy grant"]
    B --> C["Provision DB (CNPG) + bucket (RGW)"]
    C --> D["Deploy Qdrant + Foundry endpoint"]
    D --> E["Sanitize chart → push to GitLab"]
    E --> F["assistant-service-secret ExternalSecret (Vault)"]
    F --> G["ArgoCD Application (multi-source)"]
    G --> H["Sync: config(0) → pg/redis(0-1) → migration(1) → deployment(2) → postDeploy(3)"]

    style A fill:#1e3a5f,stroke:#4a90d9,color:#fff
    style H fill:#1e5f3a,stroke:#4ad98a,color:#fff

9a. Sanitized repo — ✅ created

modgpt/mod-ask-assistant-service (local, committed). Applied:

• [x] Stripped to helm/ only (no src/, tests, preview templates, cloud env files)
• [x] Images → harbor.cl1.sq4.aegis.internal/ask/assistant-service*
• [x] vault provider branch added to templates/externalsecrets.yaml; env uses provider: vault, secretKey: ask/assistant-service/auh1-dev → chart builds assistant-service-secret
• [x] Chart.yaml redis dep → GitLab HTTP Helm registry (vendor like zitadel)
• [x] helm/environments/values-mod-auh1-dev-ask.yaml — air-gap config (in-cluster DNS, ask71 nodeSelector, OSS models, Qdrant/memory omitted, Foundry deferred, bitnamilegacy/redis:8.2.1)
• [x] DB: postgresql.enabled=false + postgresInit on core-pg
• [x] helm template validated — 23 manifests, Harbor images, vault ExternalSecret, core-pg DB-init

9b. Remaining to deploy

1. ~~Push chart + Application~~ ✅ done
2. Vendor the redis subchart — see §9c (detailed)
3. Seed Vault + policy grant — see §9d
4. Set APP_ENV + DATABASE__PASSWORD via envConf.extraEnv — the Dynaconf env-switcher gotcha (note APP_ENV single-underscore vs core-service's APP__ENV). See §4c.

---

9c. Vendoring the redis subchart (detailed)

Why this is needed. Chart.yaml declares the redis subchart from the GitLab Helm registry. At sync time ArgoCD runs helm dependency build; in an air-gapped cluster it cannot reach the public bitnami OCI registry, and you don't want ArgoCD pulling from the GitLab registry on every sync either. So we vendor the subchart — commit charts/redis-22.0.7.tgz + Chart.lock into the repo. With both present, helm dependency build (ArgoCD's and yours) uses the local copy and never downloads. This is the same pattern used for the zitadel subchart in core-service.

Prerequisite — the bastion must trust the internal CA (so helm/git can reach the GitLab registry over HTTPS). Add the root CA once (Rocky/RHEL):

sudo cp ~/certificates/modgpt-ca.crt /etc/pki/ca-trust/source/anchors/modgpt-ca.crt
sudo update-ca-trust extract
# verify: curl -sSI https://gitlab.cl1.sq4.aegis.internal | head -1   → HTTP/2 302, no x509 error

Step 1 — get the redis chart into the GitLab Helm registry (project 14). If it's not already there, push it from the bastion (the .tgz lives in the chart cache):

cd <harbor-mirror>/helm-charts
python3 gitlab-push-charts.py -c gitlab-charts.yaml --only redis
#   → POST .../packages/helm/api/stable/charts → "201 Created"

(The push uses chart-push-sa creds from Vault; gitlab-charts.yaml already lists redis 22.0.7.)

Step 2 — register the GitLab Helm repo with helm (so helm dependency build can authenticate):

GLTOKEN=$(vault kv get -field=token secret/gitlab/chart-push-sa)
helm repo add gitlab-helm \
  https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/14/packages/helm/stable \
  --username chart-push-sa --password "$GLTOKEN"
helm repo update gitlab-helm

Step 3 — vendor into the chart:

cd ~/ask-assistant-service/helm        # a clone of the sanitized chart repo
mkdir -p charts && cp <cache>/redis-22.0.7.tgz charts/   # seed the local copy (optional but avoids a pull)
helm dependency build

helm dependency build matches Chart.yaml's redis URL to the authed gitlab-helm repo and writes charts/redis-22.0.7.tgz + Chart.lock.

“Unable to get an update from … context deadline exceeded” is harmless

helm dependency build first tries to refresh every helm repo configured on the host. In the air-gapped bastion the public ones (bitnami, argo, cnpg, …) time out — these are non-fatal. The build still completes: Update Complete → Saving 1 charts → Downloading redis from …gitlab…/packages/helm/stable. Redis comes from the internal GitLab registry, so no proxy / internet is required. (Optional: helm repo remove <stale-public-repos> to silence the noise.)

Step 4 — commit the vendored chart (charts/ + Chart.lock are gitignored → force-add):

git add -f charts/redis-22.0.7.tgz Chart.lock
git commit -m "vendor redis 22.0.7 subchart (helm dependency build)"
git push origin main

After this the chart is self-contained — ArgoCD renders it without any registry access.

---

9d. Seed Vault + policy grant

The chart's externalSecrets (vault branch) builds assistant-service-secret by extracting all keys from one Vault path. That path must (a) be readable by ESO and (b) exist before sync.

Policy grant — add to eso-ask-push in applications/cluster-addons/vault/manifests/vault-bootstrap-config.yaml, then sync the vault app:

path "secret/data/ask/assistant-service/*"     { capabilities = ["create","update","read"] }
path "secret/metadata/ask/assistant-service/*" { capabilities = ["read","list"] }

argocd app sync vault --grpc-web
vault policy read eso-ask-push | grep assistant-service   # confirm the grant

Seed the single secret (all keys in one vault kv put — ESO extract reads them all):

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
CS=secret/ask/core-service/auh1-dev
vault kv put secret/ask/assistant-service/auh1-dev \
  REDIS_PASSWORD="$(openssl rand -hex 24)" \
  AI71_AUTH_STATE_SECRET="$(openssl rand -hex 32)" \
  CORE_SERVICE_ENCRYPTION_KEY="$(vault kv get -field=value $CS/SECURITY__MASTER_ENCRYPTION_KEY)" \
  CORE_SERVICE__JWT_SECRET="placeholder" IAM__MACHINE_TOKEN="placeholder" \
  AI71_FOUNDRY_KEY="placeholder" HATCHET__CLIENT_TOKEN="placeholder" \
  INTELLIGENCE__MLFLOW__TOKEN="placeholder"

Then force the ExternalSecret and confirm it materialised:

kubectl annotate externalsecret -n ask assistant-service-secret force-sync="$(date +%s)" --overwrite
kubectl get externalsecret,secret -n ask | grep assistant-service-secret   # → SecretSynced

Two recurring ordering issues (same as core-service)

• configmap "assistant-service-postgres-init-script" not found → the postgres-init ConfigMaps need a PreSync wave earlier than the Job. Set postgresInit.argocd.connectionConfigMapAnnotations to PreSync sync-wave: "-2" and jobAnnotations to PreSync sync-wave: "-1" (done in the env values).
• secret "assistant-service-secret" not found → seed Vault + policy grant (above) before the workloads sync.

---

10. Release strategy (2-source: chart base + devops overlay)

The ArgoCD Application is multi-source, which separates what the app is from which build is deployed — so releases happen in the devops repo without touching the chart repo.

flowchart LR
    subgraph CHART["chart repo (ask-assistant-service) — app team"]
        B["helm/environments/values-mod-auh1-dev-ask.yaml<br/><b>base config</b> — configToml, DB,<br/>redis, secrets wiring, structure"]
    end
    subgraph DEVOPS["devops repo (mod-ask-devops) — ops"]
        O["applications/assistant-service/values-mod-auh1-dev-ask.yaml<br/><b>release overlay</b> — image.tag,<br/>replicas, per-env tuning"]
        A["ArgoCD Application (2-source)"]
    end
    B -->|"Source 1 valueFile"| A
    O -->|"Source 2 \$values valueFile (WINS)"| A
    A --> K["☸️ assistant-service in ask"]

    style CHART fill:#1e3a5f,stroke:#4a90d9,color:#fff
    style DEVOPS fill:#1e5f3a,stroke:#4ad98a,color:#fff
    style K fill:#3a1e5f,stroke:#a94ad9,color:#fff

Helm applies the value files left-to-right, so the overlay wins over the base.

Layer	Repo / file	Changes on	Owner
Base config	chart helm/environments/values-<env>.yaml	chart/config changes (rare)	app team
Release overlay	devops applications/assistant-service/values-<env>.yaml	every release / promotion	ops / GitOps

Ship a release

Edit only the devops overlay — the chart repo stays the stable template:

# in mod-ask-devops
#   applications/assistant-service/values-mod-auh1-dev-ask.yaml
#   image:
#     tag: "staging-<new-sha>"      ← bump here
git commit -am "release(assistant-service): deploy staging-<sha>" && git push
# ArgoCD auto-syncs → rolling update to the new image

Promote across environments

Each environment has its own overlay (values-mod-auh1-dev-ask.yaml, values-...-stg-...yaml, …) with its own image.tag. Promotion = bump the tag in the target env's overlay and commit. (Argo CD Image Updater can automate this tag bump in the overlay file.)

Why 2-source (vs all-in-chart)

Keeping release-managed values (image tags) in the chart repo would couple the deployment lifecycle to the app source. The overlay decouples them: stable chart config in the chart repo, ops-controlled release lever in the devops repo. core-service uses the same pattern.