38 — KMS Config & Model References (air-gap findings)¶

This page records the model references and required config/secrets for the Knowledge Management Service, found while wiring it to the air-gapped Foundry. It is the KMS analogue of the assistant-service Model References page. For the deploy runbook see 37 — KMS Deployment; for Foundry itself see Foundry / LLM Endpoint.

Summary

KMS is single-layer for models (TOML only — no baked user catalog like assistant-service's models.yaml). But the upstream config/default.toml defaults to models the air-gap Foundry doesn't serve, and several required @format {env[...]} keys were missing from the Vault secret (KMS crashes at config load without them). Both are fixed in helm/environments/values-mod-auh1-dev-ask.yaml; the secret keys must be seeded in Vault.

Model references (single layer — TOML)¶

KMS references models only via config/default.toml, overridden per-env by the chart's envConf.configToml. There is no baked model catalog.

Field	Section	Upstream default (not served)	Air-gap override
`embedding_model_id`	`[organization]`	`BAAI/bge-m3`	`qwen/qwen3-embedding-0-6b` (1024-dim)
`vlm_model_name`	`[organization]`	`core42-oicm-auh1-shd/qwen-3-5-9b`	`qwen/qwen3-5-122b-a10b-kms` (KMS-dedicated, vision)
`foundry_endpoint`	`[document_understanding]`	(env-specific)	`http://foundry-litellm.foundry.svc.cluster.local:4000/v1`
`model_provider`	`[document_understanding]`	`foundry`	unchanged ✅
`reranker`	`[retriever]`	`@none`	unchanged (disabled) ✅

embedding_model_id is load-bearing

BAAI/bge-m3 is not served by the air-gap Foundry. If left unoverridden, every document ingestion fails to vectorize → Weaviate stays empty → KMS is functionally dead. It must be a Foundry-served embedding id. qwen/qwen3-embedding-0-6b produces 1024-dim vectors (verified) and that is the dimensionality Weaviate stores.

VLM model decision

vlm_model_name drives visual document understanding (image/PDF extraction). The upstream 9B VLM isn't served. We point it at qwen/qwen3-5-122b-a10b-kms — the KMS-dedicated, vision-capable Foundry instance (keeps ingestion load off the interactive qwen3-5-122b-a10b). If visual extraction isn't needed yet, this path can be disabled instead.

Required `@format {env[...]}` keys — KMS crashes without them¶

default.toml uses Dynaconf @format {env[...]} for these. Any one missing → DynaconfFormatError at config load → CrashLoopBackOff before the app runs. This is independent of the model fixes.

Env var	Source	Status before fix	Resolution
`APP__ENV`	chart `extraEnv`	❌ not set → KMS ignored the mounted override and used `default.toml`	set `APP__ENV=mod-auh1-dev-ask` in `extraEnv`
`HATCHET__CLIENT_TOKEN`	Vault → secret	✅ seeded	—
`DATABASE__PASSWORD`	`extraEnv` secretKeyRef (CNPG)	✅	—
`DOCUMENT_UNDERSTANDING__FOUNDRY_KEY`	Vault → secret	❌ missing	seed = Foundry `PROXY_MASTER_KEY`
`DOCUMENT_UNDERSTANDING__DI_SUBSCRIPTION_KEY`	Vault → secret	❌ missing	Azure DI is unreachable air-gapped → seed a placeholder to satisfy `@format`; rely on the Foundry VLM path
`VECTOR_DB__API_KEY`	Vault → secret	❌ missing	generate; must equal Weaviate's `AUTHENTICATION_APIKEY_ALLOWED_KEYS`
`CORE_SERVICE__JWT_SECRET`	Vault → secret	❌ missing	must equal core-service's value (copy from `core-service-secret`)
`APP__MASTER_ENCRYPTION_KEY`	Vault → secret	❌ missing	generate once (`openssl rand -hex 32`); never rotate (encrypts stored data)

We initially seeded the wrong key names

The first KMS secret seed used SECRET_KEY and API_KEY — neither is referenced by default.toml. They were no-ops. The authoritative list is the @format table above.

Weaviate authentication — apikey is mandatory¶

The KMS Weaviate client always applies Auth.api_key(config.vector_db.api_key) (src/knowledge_management/config/vector_database.py), with no conditional. An anonymous-only Weaviate therefore returns 401. The subchart must enable apikey auth with a key matching VECTOR_DB__API_KEY.

Wired in values-mod-auh1-dev-ask.yaml (weaviate: block):

weaviate:
  authentication:
    anonymous_access:
      enabled: false
  authorization:
    rbac:
      enabled: false      # apikey-authenticated user gets full access (bring-up)
  env:
    AUTHENTICATION_APIKEY_ENABLED: "true"
    AUTHENTICATION_APIKEY_USERS: "kms-admin"
  envSecrets:
    # subchart mounts env AUTHENTICATION_APIKEY_ALLOWED_KEYS from this secret, same key name
    AUTHENTICATION_APIKEY_ALLOWED_KEYS: knowledge-management-service-secret

Single source of truth: seed both VECTOR_DB__API_KEY and AUTHENTICATION_APIKEY_ALLOWED_KEYS in the Vault path to the same value, so the KMS client and the Weaviate server agree.

Seed the Vault secret (complete)¶

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
export VAULT_SKIP_VERIFY=true

# Foundry key — reuse the gateway master key (least-privilege: mint a KMS virtual key later)
FOUNDRY_KEY=$(kubectl get secret -n foundry foundry-secret -o jsonpath='{.data.PROXY_MASTER_KEY}' | base64 -d)

# Shared JWT secret — the source of truth is the ASSISTANT-SERVICE Vault path (NOT core-service-secret,
# which uses Zitadel/PAT auth and has no shared HS secret). assistant-service validates core-service
# tokens with this value, so it is the correct shared secret for KMS too.
JWT_SECRET=$(vault kv get -field=CORE_SERVICE__JWT_SECRET secret/ask/assistant-service/auh1-dev)

# One key value used by BOTH the KMS client and the Weaviate server
WEAVIATE_KEY=$(openssl rand -hex 32)

vault kv patch secret/ask/knowledge-management-service/auh1-dev \
  DOCUMENT_UNDERSTANDING__FOUNDRY_KEY="$FOUNDRY_KEY" \
  DOCUMENT_UNDERSTANDING__DI_SUBSCRIPTION_KEY="airgap-disabled-placeholder" \
  VECTOR_DB__API_KEY="$WEAVIATE_KEY" \
  AUTHENTICATION_APIKEY_ALLOWED_KEYS="$WEAVIATE_KEY" \
  CORE_SERVICE__JWT_SECRET="$JWT_SECRET" \
  APP__MASTER_ENCRYPTION_KEY="$(openssl rand -hex 32)"

# force ESO to resync the k8s secret
kubectl annotate externalsecret -n ask knowledge-management-service-external-secret \
  force-sync=$(date +%s) --overwrite

The JWT secret is NOT in core-service-secret

core-service-secret holds Zitadel/IAM creds (IAM_SERVICE__PAT, APP__CORE_SERVICES_MACHINE_TOKEN, SECURITY__*) — no shared HS JWT secret. The CORE_SERVICE__JWT_SECRET (HS, 64-char) lives in the assistant-service Vault path and is the value all token validators must share. KMS, the assistant-service, and core-service's signer must all use the same value or cross-service tokens fail validation.

APP__MASTER_ENCRYPTION_KEY is permanent

It encrypts data KMS stores. Generate it once and keep it stable — rotating it makes existing encrypted data unreadable. Treat it like a database master key.

Verify after sync¶

# every @format key present in the rendered secret
for k in DOCUMENT_UNDERSTANDING__FOUNDRY_KEY DOCUMENT_UNDERSTANDING__DI_SUBSCRIPTION_KEY \
         VECTOR_DB__API_KEY AUTHENTICATION_APIKEY_ALLOWED_KEYS CORE_SERVICE__JWT_SECRET \
         APP__MASTER_ENCRYPTION_KEY HATCHET__CLIENT_TOKEN; do
  v=$(kubectl get secret -n ask knowledge-management-service-secret -o jsonpath="{.data.$k}" 2>/dev/null)
  echo "$k: $([ -n "$v" ] && echo present || echo MISSING)"
done

# pods should leave CrashLoop once config loads
kubectl get pods -n ask | grep -iE "knowledge|weaviate"

# the model ids KMS will actually call
kubectl exec -n ask deploy/knowledge-management-service -- \
  cat /app/config/mod-auh1-dev-ask.toml | grep -E "embedding_model_id|vlm_model_name|foundry_endpoint"

Each model id printed must appear in Foundry /v1/models (see Foundry → Validate).