Skip to content

39 — KMS Architecture, Integrations & Operations

Deep reference for the Knowledge Management Service: the four workloads and how they relate, every external integration with tests, model usage, S3 document storage, and the two air-gap design decisions (Weaviate PSA and the non-RFC1918 memberlist fix). Companion to 37 — KMS Deployment (runbook) and 38 — KMS Config & Model References (config/secret detail).

1. Weaviate & Pod Security Admission (PSA) — the privileged-init decision

The problem

The ask namespace enforces pod-security.kubernetes.io/enforce: baseline. Weaviate's chart ships a configure-sysctl init container that runs sysctl -w vm.max_map_count=524288, which requires securityContext.privileged: true. Baseline forbids privileged containers, so the API server refuses to create the pod:

FailedCreate: pods "weaviate-0" is forbidden: violates PodSecurity "baseline:latest":
privileged (container "configure-sysctl" must not set securityContext.privileged=true)

vm.max_map_count is a node-level (non-namespaced) sysctl — it cannot be set from a pod securityContext.sysctls under baseline (only a small set of namespaced safe sysctls is allowed). So the chart's only in-pod mechanism is the privileged init container, and that is exactly what PSA blocks. Weaviate needs the value high (default Linux 65530 is below Weaviate's recommended 524288) to memory-map many index segment files.

Options considered

Option What it does Pros Cons Verdict
A. Relax namespace to privileged Set pod-security.kubernetes.io/enforce: privileged on ask (the hatchet precedent, see hatchet deploy) One-line; init container runs as-is Weakens PSA for the entire ask namespace — core-service, assistant-service, KMS API/workers all lose baseline enforcement. Large blast radius, violates least-privilege ❌ Rejected
B. Dedicated weaviate namespace with privileged PSA Move Weaviate to its own namespace that allows privileged Confines the privilege to Weaviate only; independent lifecycle (production-correct per 37) More moving parts: cross-namespace service DNS, separate Application, separate storage/PSA policy ⚠️ Production target
C. Disable the init container + set vm.max_map_count on nodes initContainers.sysctlInitContainer.enabled: false; set the sysctl via node OS config / a node-tuning DaemonSet in a system namespace No privileged pod anywhere; baseline preserved namespace-wide; Weaviate gets the correct value Requires node-level configuration (infra task) outside the chart ✅ Chosen
D. Disable init container, accept node default Same as C but rely on the node's existing vm.max_map_count Simplest; zero privilege If the node default is 65530, large indexes may hit mmap limits under load ✅ Bring-up fallback (= C without the node tuning yet)

Why we chose C (disable init container + node sysctl)

  • Least privilege / smallest blast radius. Options A and B both run a privileged container; A additionally degrades security for every workload in ask. C runs no privileged pod at all, so the strong baseline enforcement that protects core-service and assistant-service stays intact.
  • The sysctl belongs on the node anyway. vm.max_map_count is a kernel/node setting; configuring it via the OS (or a dedicated, clearly-scoped node-tuning DaemonSet in kube-system) is the conventional, auditable way to tune nodes — not by granting every Weaviate pod privileged.
  • No namespace policy churn. We don't have to special-case the ask namespace PSA label in managedNamespaceMetadata, which would otherwise have to be remembered and justified forever.

Implemented in values-mod-auh1-dev-ask.yaml: 

weaviate:
  initContainers:
    sysctlInitContainer:
      enabled: false   # privileged sysctl -w blocked by baseline PSA; set on nodes instead

Node-side follow-up (production)

# per ask71 node (or via a node-tuning DaemonSet / cloud-init):
sysctl -w vm.max_map_count=524288
echo 'vm.max_map_count=524288' | sudo tee /etc/sysctl.d/99-weaviate.conf
Until this is applied, Weaviate runs on the node default and logs a warning — fine for bring-up, recommended before production ingestion volumes.

2. Weaviate memberlist on a non-RFC1918 pod network

The problem

Weaviate clusters via HashiCorp memberlist (gossip). At startup it must choose an advertise address. memberlist's auto-detection (GetPrivateIP) only accepts RFC1918 ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16). This cluster's pod network is CGNAT 100.64.0.0/10 (e.g. 100.x.x.x), which is not RFC1918, so auto-detection finds nothing:

memberlist not created: Failed to get final advertise address:
No private IP address found, and explicit IP not provided
→ could not init cluster state   (CrashLoopBackOff)

What does NOT work

  • CLUSTER_ADVERTISE_INTERFACE=eth0 — Weaviate 1.34 does not read this variable; advertise_addr stayed empty.
  • Pod securityContext.sysctls / interface tricks — irrelevant; memberlist wants an explicit IP.

The fix — provide the pod IP explicitly

The error itself states the escape hatch: "and explicit IP not provided". An explicitly provided advertise address bypasses the RFC1918 check entirely — even a 100.x address is accepted. Weaviate reads it from CLUSTER_ADVERTISE_ADDR. The right value is the pod's own IP, via the downward API.

The subchart's plain env: map can only express literal name: value pairs (no valueFrom), so we unpacked the vendored chart (charts/weaviate-17.7.0.tgz → charts/weaviate/) and patched templates/weaviateStatefulset.yaml:

- name: CLUSTER_ADVERTISE_ADDR
  valueFrom:
    fieldRef:
      fieldPath: status.podIP

Unpacking (vs. re-tarring a binary .tgz) keeps the one-line patch reviewable in Git. After this, advertise_addr shows the 100.x pod IP, memberlist initialises, and Weaviate serves.

Why podIP and not the headless DNS name

memberlist parses AdvertiseAddr as an IP, not a hostname, so the per-pod headless DNS name (weaviate-0.weaviate-headless...) would not work. status.podIP gives the literal IP.

3. The four KMS workloads — functions & relationships

KMS is one chart that renders a FastAPI API plus three Hatchet workers (split by workload class) and a migration job. All share the same image, config (APP__ENV=mod-auh1-dev-ask), and secret.

flowchart TB
    subgraph kms["KMS workloads (ask ns)"]
        API["knowledge-management-service\nFastAPI REST :8000"]
        IW["ingestion-worker\nHatchet"]
        GW["generic-worker\nHatchet"]
        LW["live-ingestion-worker\nHatchet"]
    end
    PG[("knowledge-pg\nCNPG — metadata")]
    WV[("weaviate\nvectors")]
    S3[("Ceph S3\nask-knowledge-management")]
    FND["foundry-litellm\nembeddings + VLM"]
    HCH["hatchet-engine\n:7070"]
    CORE["core-service\nauth/IAM"]
    AST["assistant-service\n(tooling → KMS)"]

    AST -->|search/chat over KB| API
    API -->|auth| CORE
    API -->|enqueue workflows| HCH
    API -->|vector search| WV
    API -->|metadata| PG
    HCH -.dispatch.-> IW & GW & LW
    IW -->|download/parse| S3
    IW -->|embed| FND
    IW -->|upsert vectors| WV
    IW -->|metadata| PG
    LW -->|streaming ingest| S3 & WV & PG
    GW -->|metrics / connector sync| PG
Workload Type Function Hatchet workflows it runs Hard deps
knowledge-management-service FastAPI Deployment (:8000) Public REST API: document upload, knowledge-base search/retrieval, collection mgmt. Authenticates callers, enqueues ingestion/deletion workflows, serves vector search results. Does not itself do heavy ingestion. — (producer/dispatcher) knowledge-pg, Weaviate, core-service, Hatchet
ingestion-worker Hatchet worker Batch document pipeline: pull file from S3 → parse/OCR (VLM document-understanding) → chunk → embed (Foundry) → upsert vectors to Weaviate + metadata to knowledge-pg. Also document deletion. kms_document_ingestion, kms_document_embedding, kms_document_deletion Hatchet, S3, Foundry, Weaviate, knowledge-pg
live-ingestion-worker Hatchet worker Real-time / streaming ingestion (e.g. connector-driven or incremental updates) — same pipeline shape as ingestion but on the live path. kms_live_document_ingestion, kms_live_document_embedding Hatchet, S3, Foundry, Weaviate, knowledge-pg
generic-worker Hatchet worker Housekeeping/async ops not tied to a document: usage metrics publishing and external connection/connector sync. kms_metrics, kms_connection_sync Hatchet, knowledge-pg
(migration Job) Job (sync-wave) alembic upgrade head against knowledge-pg before the app starts. — knowledge-pg

Slot/replica tuning (from [hatchet] config): ingestion 2 workers × 5 slots, live-ingestion 1 × 5, generic × 2 slots.

Producer→consumer split: the API never blocks on ingestion — it writes a row + enqueues a Hatchet workflow and returns. The workers consume from the engine. This is why all four must share the same HATCHET__CLIENT_TOKEN and reach hatchet-engine.ask.svc.cluster.local:7070.

4. Service integrations & tests

The KMS image has no curl — use Python httpx for in-pod HTTP checks (same as assistant-service).

# 0. all workloads Running / migration Completed
kubectl get pods -n ask | grep -iE "knowledge|weaviate"
kubectl get jobs -n ask | grep knowledge   # migration 1/1
# Integration What to verify Test
1 knowledge-pg (metadata) migration ran; schema exists kubectl exec -n ask knowledge-pg-1 -- psql -U postgres -d knowledge_management -c '\dn' → schema knowledge_service; \dt knowledge_service.* lists tables
2 Weaviate (vectors) reachable + apikey auth works see §below
3 Hatchet (workers) all three workers registered kubectl logs -n ask deploy/knowledge-management-service-ingestion-worker --tail=40 \| grep -iE "step run\|started\|runner\|🪓"
4 Foundry (embed/VLM) embedding model served see §5
5 Ceph S3 (documents) bucket reachable + writable see §6
6 core-service (auth) API authorizes requests kubectl exec -n ask deploy/knowledge-management-service -- python3 -c "import httpx;print(httpx.get('http://core-service.ask.svc.cluster.local/api/v1/health/ready',timeout=10).status_code)" → 200
7 API health (ingress) end-to-end routing curl -sk https://api.ask.mod.auh1.dev.dir/<kms-path>/health/readiness (confirm the ingress path)

Weaviate reachability + auth (test 2): 

WV_KEY=$(kubectl get secret -n ask knowledge-management-service-secret \
  -o jsonpath='{.data.VECTOR_DB__API_KEY}' | base64 -d)
# from the KMS API pod — readiness needs NO auth; the meta endpoint with the key proves apikey auth
kubectl exec -n ask deploy/knowledge-management-service -- python3 -c \
"import httpx,os
b='http://weaviate.ask.svc.cluster.local'
print('ready:', httpx.get(b+'/v1/.well-known/ready',timeout=8).status_code)
print('auth :', httpx.get(b+'/v1/meta',headers={'Authorization':'Bearer '+os.environ.get('WV','')},timeout=8).status_code)" \
  WV="$WV_KEY" 2>/dev/null || echo "set WV via env FK pattern if needed"
ready: 200 and auth: 200 = Weaviate up and the apikey matches AUTHENTICATION_APIKEY_ALLOWED_KEYS (both sourced from the same Vault value — see 38).

5. Model usage — how KMS consumes models & how to test

KMS uses Foundry (LiteLLM gateway) for two model roles during ingestion (single-layer TOML config — full detail in 38):

Role Config field Air-gap model Where used
Embeddings [organization] embedding_model_id qwen/qwen3-embedding-0-6b (1024-dim) ingestion + live-ingestion workers embed chunks before upserting to Weaviate
Visual doc understanding [organization] vlm_model_name qwen/qwen3-5-122b-a10b-kms (KMS-dedicated, vision) ingestion worker extracts text/structure from image/PDF pages
Reranker [retriever] reranker @none (disabled) —

Auth: all Foundry calls use DOCUMENT_UNDERSTANDING__FOUNDRY_KEY (= the LiteLLM PROXY_MASTER_KEY) against foundry_endpoint = http://foundry-litellm.foundry.svc.cluster.local:4000/v1.

Test the embedding model from a KMS pod (the dimension must be 1024 to match Weaviate): 

FK=$(kubectl get secret -n ask knowledge-management-service-secret \
  -o jsonpath='{.data.DOCUMENT_UNDERSTANDING__FOUNDRY_KEY}' | base64 -d)
kubectl exec -n ask deploy/knowledge-management-service-ingestion-worker -- env FK="$FK" python3 -c \
"import os,httpx
r=httpx.post('http://foundry-litellm.foundry.svc.cluster.local:4000/v1/embeddings',
  headers={'Authorization':'Bearer '+os.environ['FK']},
  json={'model':'qwen/qwen3-embedding-0-6b','input':'kms embedding test'},timeout=30)
print('status',r.status_code); d=r.json()
print('dims', len(d['data'][0]['embedding']) if 'data' in d else d)"
# expect: status 200, dims 1024   (404 → model id wrong; 401 → DOCUMENT_UNDERSTANDING__FOUNDRY_KEY stale)

Confirm the live config the pod actually loaded (not the baked default): 

kubectl exec -n ask deploy/knowledge-management-service -- \
  cat /app/config/mod-auh1-dev-ask.toml | grep -E "embedding_model_id|vlm_model_name|foundry_endpoint"
# embedding_model_id = "qwen/qwen3-embedding-0-6b"  (NOT BAAI/bge-m3 → that would mean APP__ENV unset)
If you see BAAI/bge-m3, APP__ENV isn't taking effect and KMS loaded default.toml — see 38 → APP__ENV.

6. S3 document storage — usage & test

KMS stores uploaded source documents and processed artifacts in the external Ceph RGW bucket ask-knowledge-management (S3 is external to the cluster — radosgw-admin runs on the Ceph nodes).

Config ([storage] type = "s3", [storage.s3]): 

endpoint_url = "https://s3.cl1.sq4.aegis.internal/"
bucket_name  = "ask-knowledge-management"
region       = "us-east-1"
verify_ssl   = false        # self-signed Ceph RGW cert
access_key_id     = @format {env[STORAGE__S3__ACCESS_KEY_ID]}
secret_access_key = @format {env[STORAGE__S3__SECRET_ACCESS_KEY]}
Credentials are the dedicated ask-kms RGW user (see 37 → Pre-Engagement R1) stored in Vault and surfaced via ESO. The ingestion worker writes objects on document upload; the API and workers read them back during processing.

Test 6a — can a KMS pod write/read/delete an object in the bucket (proves creds + endpoint + RWX): 

AK=$(kubectl get secret -n ask knowledge-management-service-secret -o jsonpath='{.data.STORAGE__S3__ACCESS_KEY_ID}' | base64 -d)
SK=$(kubectl get secret -n ask knowledge-management-service-secret -o jsonpath='{.data.STORAGE__S3__SECRET_ACCESS_KEY}' | base64 -d)
kubectl exec -n ask deploy/knowledge-management-service-ingestion-worker -- env AK="$AK" SK="$SK" python3 - <<'EOF'
import os, boto3, botocore
s3 = boto3.client("s3", endpoint_url="https://s3.cl1.sq4.aegis.internal/",
    aws_access_key_id=os.environ["AK"], aws_secret_access_key=os.environ["SK"],
    region_name="us-east-1", verify=False,
    config=botocore.config.Config(s3={"addressing_style":"path"}))
b="ask-knowledge-management"; k="_healthcheck/kms-write-test.txt"
s3.put_object(Bucket=b, Key=k, Body=b"kms s3 write ok")
print("PUT  ok")
print("GET  ok:", s3.get_object(Bucket=b, Key=k)["Body"].read().decode())
print("LIST ok:", [o["Key"] for o in s3.list_objects_v2(Bucket=b, Prefix="_healthcheck/").get("Contents",[])])
s3.delete_object(Bucket=b, Key=k); print("DEL  ok")
EOF
# all four ok = ask-kms user has GetObject/PutObject/ListBucket/DeleteObject on the bucket.
# 403 AccessDenied → the Ceph user policy is missing a permission; SignatureDoesNotMatch → wrong keys.
(If boto3 isn't in the image, run the same from the bastion with the ask-kms keys, or use aws s3 cp --endpoint-url.)

Test 6b — end-to-end: upload a real document and confirm it lands in S3 + gets ingested: 

# 1. upload via the KMS API (needs a core-service JWT — see assistant-service/integration-tests.md §7b)
JWT="<core-service-jwt>"
curl -sk -X POST https://api.ask.mod.auh1.dev.dir/<kms-upload-path> \
  -H "Authorization: Bearer $JWT" -F "file=@./sample.pdf"

# 2. watch the ingestion worker pick up the Hatchet workflow
kubectl logs -n ask deploy/knowledge-management-service-ingestion-worker --tail=40 \
  | grep -iE "kms_document_ingestion|s3|embed|weaviate|upsert|error"

# 3. confirm the object exists (bastion, with ask-kms keys) and the bucket grew
#    (the Ceph team can also confirm with: radosgw-admin bucket stats --bucket=ask-knowledge-management)
Healthy signs: worker logs show the document downloaded from S3, embedded (1024-dim via Foundry), vectors upserted to Weaviate, and the bucket object count increases.

Cross-references