37 — Knowledge Management Service (KMS) Deployment¶

Knowledge Management Service provides the document ingestion pipeline, vector search (Weaviate), and knowledge-base retrieval that backs ASK Assistant's RAG flows. It runs four workloads: a FastAPI server and three Hatchet workers (ingestion, generic, live-ingestion).

Architecture¶

flowchart TB
    subgraph ask["ask namespace"]
        API["knowledge-management-service\n(FastAPI :8000)"]
        IW["ingestion-worker\n(Hatchet — batch docs)"]
        GW["generic-worker\n(Hatchet — general)"]
        LW["live-ingestion-worker\n(Hatchet — realtime)"]
        MIG["migration-job\n(alembic)"]
        WEV["weaviate\n(vector DB, subchart)"]
    end

    AST["assistant-service\n(tooling — KMS tool calls)"]
    CORE["core-service\n(auth / org context)"]
    HATCHET["hatchet-engine\n(:7070 gRPC plaintext)"]
    PG["knowledge-pg\n(CNPG cluster)"]
    FOUNDRY["foundry-litellm\n(:4000 LLM gateway)"]
    S3["Ceph RGW\ns3.cl1.sq4.aegis.internal"]

    API --> PG
    API --> WEV
    API --> CORE
    IW --> PG
    IW --> WEV
    IW --> HATCHET
    IW --> FOUNDRY
    IW --> S3
    GW --> PG
    GW --> WEV
    GW --> HATCHET
    GW --> FOUNDRY
    LW --> PG
    LW --> WEV
    LW --> HATCHET
    LW --> S3
    MIG --> PG
    AST --> API

    style WEV fill:#1e3a5f,stroke:#4a90d9,color:#fff
    style PG fill:#2d5016,stroke:#6ab04c,color:#fff

Workload roles¶

Workload	Function	Hard dependencies
`knowledge-management-service`	REST API — chat/search/upload endpoints	knowledge-pg, Weaviate, core-service
`ingestion-worker`	Batch document ingestion pipeline (OCR, chunking, embedding → Weaviate)	Hatchet, knowledge-pg, Weaviate, Foundry, Ceph S3
`generic-worker`	General async workflows dispatched by the API	Hatchet, knowledge-pg, Weaviate, Foundry
`live-ingestion-worker`	Real-time / streaming document ingest	Hatchet, knowledge-pg, Weaviate, S3
`migration-job`	`alembic upgrade head` — runs every ArgoCD sync (wave 1)	knowledge-pg

Air-Gap Changes from PP Baseline¶

Concern	PP (on-prem Abu Dhabi)	Air-Gap (mod-auh1-dev)	Production note
Image registry	Azure CR (`ai71uaenprodask71acr01.azurecr.io`)	Harbor (`harbor.cl1.sq4.aegis.internal/ask/knowledge-management-service`)	Harbor is the production-correct alternative for air-gap
PostgreSQL	Bitnami subchart (`postgresql.enabled: true`)	Disabled — reuse `knowledge-pg` CNPG cluster	CNPG is production-correct (HA, point-in-time recovery)
Weaviate	External service in `weaviate` namespace (`weaviate.weaviate.svc.cluster.local`)	Subchart enabled in `ask` namespace	Production deviation: should be in its own namespace with independent lifecycle
ExternalSecrets	Azure Key Vault (`provider: azure`)	Vault KV v2 (`provider: vault`)	Vault is production-correct for air-gap
S3 storage	Azure S3 (`s3.pp.gov.ae`)	Ceph RGW (`s3.cl1.sq4.aegis.internal`)	Ceph RGW is functionally equivalent; requires `verify_ssl: false` for self-signed
Foundry endpoint	External shared LiteLLM (`foundry.core.uaen.ai71services.ai`)	In-cluster LiteLLM (`foundry-litellm.foundry.svc.cluster.local:4000/v1`)	Same pattern as assistant-service
Hatchet	Hatchet in `hatchet` namespace	Hatchet in `ask` namespace — `HATCHET_CLIENT_HOST_PORT` override required	Same fix applied to assistant-service
Autoscaling	HPA enabled (1–25 replicas/worker)	Disabled for bring-up (1 replica each)	Re-enable HPA for production load

Weaviate namespace deviation

PP deploys Weaviate in its own weaviate namespace (separate lifecycle, storage class, admin boundary). For bring-up we use the KMS chart's built-in Weaviate subchart which deploys it into the ask namespace. This is a bring-up convenience — the production-correct approach is a separate namespace + independent ArgoCD Application for Weaviate. When Weaviate is in ask namespace the TOML vector_db host is knowledge-management-service-weaviate.ask.svc.cluster.local (not weaviate.weaviate.svc.cluster.local).

Pre-Engagement Requirements (external teams)¶

Use this as the engagement intake checklist

These are deliverables that must be requested from other teams before KMS can be deployed — they cannot be self-served by the deploying engineer. Collect all of them up front. Each row is a hand-off to a named team with exactly what to ask for and why.

#	Requirement	Request from	What they must provide	Why
R1	Ceph S3 service user	Storage / Ceph team	`uid=ask-kms` with `access_key` + `secret_key`, scoped read/write on bucket `ask-knowledge-management` (`s3:GetObject/PutObject/DeleteObject/ListBucket`). Endpoint `https://s3.cl1.sq4.aegis.internal`.	KMS ingestion workers store/retrieve uploaded documents + processed artifacts in this bucket.
R2	Foundry LLM access	Platform / Foundry team	LiteLLM endpoint `http://foundry-litellm.foundry.svc.cluster.local:4000/v1` + an API key (virtual key preferred over the master key). Confirm the KMS-dedicated model `qwen/qwen3-5-122b-a10b-kms` and embedding model `qwen/qwen3-embedding-0-6b` (1024-dim) are served.	KMS calls Foundry for embeddings + LLM enrichment during ingestion. The `-kms` instance isolates ingestion load from interactive traffic.
R3	Hatchet client token	Platform team (or self-retrieve from cluster)	A valid Hatchet JWT for the `ask` tenant. Retrievable from `hatchet-client-config` secret in-cluster.	KMS's 3 Hatchet workers (ingestion, generic, live-ingestion) register with the engine.
R4	`knowledge-pg` database	DBA / Platform team	A CNPG cluster `knowledge-pg` in the `ask` namespace, reachable at `knowledge-pg-rw.ask.svc.cluster.local`, with a superuser secret so `postgresInit` can create the `knowledge_management` DB + `kms_user` role.	KMS API + migration job + workers persist metadata here.
R5	GitLab repo + ArgoCD project access	Platform / GitOps team	Push access to `ai71/ask71/ask-knowledge-management-service` and the `mod-ask-devops` repo; the `mod-auh1-dev-ask.ask` AppProject must permit the KMS Application.	GitOps delivery of the chart + Application.

Secrets handling

Credentials from R1–R3 are stored in Vault only (secret/ask/knowledge-management-service/auh1-dev) and surfaced to the pods via the External Secrets Operator — never committed to Git or pasted into values files. See Phase 3.

The detailed request text + commands for R1 (Ceph S3 user) are in Phase 2.

Prerequisites (deployer checklist)¶

[ ] R1 Ceph S3 user ask-kms created + keys received (Phase 2)
[ ] R2 Foundry endpoint + key confirmed, KMS + embedding models served
[ ] R3 Hatchet client token retrieved
[ ] R4 knowledge-pg CNPG cluster running in ask namespace
[ ] Hatchet engine running with SERVER_GRPC_INSECURE=t (plaintext gRPC)
[ ] foundry-litellm service reachable in foundry namespace
[ ] core-service running in ask namespace
[ ] Harbor: ask/ask-knowledge-management-service:staging image mirrored (Phase 1)
[ ] Harbor: ask/semitechnologies/weaviate:1.34.0 image mirrored (Phase 1)
[ ] Ceph S3 bucket ask-knowledge-management created (Phase 2)
[ ] Vault secret at secret/ask/knowledge-management-service/auh1-dev seeded (Phase 3)

Phase 1 — Image Mirroring to Harbor¶

Mirroring runs from the bastion via core42-mirror-image.py (proxy ON) — not from the Mac. The Mac VPN only routes to the internal 100.115.x network (Harbor, GitLab); reaching ACR / Docker Hub requires the cluster forward proxy 10.96.137.37:3128, which is only routable from the bastion. On the Mac you only edit scripts/harbor-mirror/mirror-config.yaml; the bastion runs the copy.

1a + 1b. Config entries (in `mirror-config.yaml`)¶

# KMS application image — from ACR, by tag (no digest pin)
- { name: ask-knowledge-management-service, source: acr, project: ask, tags: ["staging"] }

# Weaviate — from Docker Hub. The weaviate subchart uses image.registry + image.repo (NOT
# image.repository), so the Harbor path must be ask/semitechnologies/weaviate to match
# values: weaviate.image.repo = ask/semitechnologies/weaviate
- { name: semitechnologies/weaviate, dest_name: semitechnologies/weaviate, source: docker.io, project: ask, tags: ["1.34.0"] }

Run the mirror (on bastion, proxy ON)¶

cd ~/harbor-images          # bastion copy of the harbor-mirror scripts
./core42-mirror-image.py --only knowledge    # → ask/ask-knowledge-management-service:staging
./core42-mirror-image.py --only weaviate      # → ask/semitechnologies/weaviate:1.34.0
# expect [PUSHED] for each (or [SKIP] already in Harbor on re-runs)

Harbor image paths must match the values file exactly

App image → harbor.cl1.sq4.aegis.internal/ask/ask-knowledge-management-service:staging (note the ask- prefix — the ACR repo is ask-knowledge-management-service).
Weaviate → harbor.cl1.sq4.aegis.internal/ask/semitechnologies/weaviate:1.34.0; the chart builds the ref from image.registry + image.repo, so repo must be the nested ask/semitechnologies/weaviate.

Phase 2 — Ceph S3 Bucket + User¶

Create a dedicated S3 user and bucket for KMS. The pattern mirrors what was done for core-service.

# From a Ceph node or the Rook toolbox pod:
radosgw-admin user create \
  --uid=ask-kms \
  --display-name="ASK Knowledge Management Service"
# → copy access_key + secret_key from the JSON "keys" array

# Create the bucket (from bastion):
AWS_ACCESS_KEY_ID=<access_key> AWS_SECRET_ACCESS_KEY=<secret_key> \
  aws --endpoint-url https://s3.cl1.sq4.aegis.internal --no-verify-ssl \
      s3 mb s3://ask-knowledge-management

# Verify:
AWS_ACCESS_KEY_ID=<access_key> AWS_SECRET_ACCESS_KEY=<secret_key> \
  aws --endpoint-url https://s3.cl1.sq4.aegis.internal --no-verify-ssl \
      s3 ls

Expected output: ask-knowledge-management listed.

Phase 3 — Vault Secret¶

Seed the KMS secret at secret/ask/knowledge-management-service/auh1-dev. All keys are loaded via ESO extract into knowledge-management-service-secret.

# On bastion
export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
vault login   # or use existing token

# Seed the KMS secret (use vault kv patch to preserve any existing keys)
vault kv put secret/ask/knowledge-management-service/auh1-dev \
  HATCHET__CLIENT_TOKEN="<paste-real-jwt-from-hatchet-client-config>" \
  STORAGE__S3__ACCESS_KEY_ID="<ceph-access-key-from-phase-2>" \
  STORAGE__S3__SECRET_ACCESS_KEY="<ceph-secret-key-from-phase-2>" \
  SECRET_KEY="$(openssl rand -hex 32)" \
  API_KEY="$(openssl rand -hex 32)"

Hatchet JWT

Copy the real JWT from the existing hatchet-client-config secret — do NOT generate a placeholder.

kubectl get secret -n ask hatchet-client-config \
  -o jsonpath='{.data.HATCHET_CLIENT_TOKEN}' | base64 -d

The TOML references @format {env[HATCHET__CLIENT_TOKEN]}.

3b. DB password secret¶

The CNPG operator creates the kms_user password secret automatically (via the postgresInit job bootstrap). If needed, create a placeholder and let postgresInit override it:

kubectl create secret generic knowledge-management-service-db-password \
  -n ask \
  --from-literal=password="$(openssl rand -hex 24)" \
  --dry-run=client -o yaml | kubectl apply -f -

The postgresInit job will create the kms_user role in knowledge-pg with this password.

Phase 4 — Sanitized Helm Chart¶

The sanitized repo lives at /Users/tarun.mittal/Documents/modgpt/mod-ask-knowledge-management-service/. It contains only the helm/ directory (no application source code).

4a. ExternalSecrets template — Vault branch¶

The original externalsecrets.yaml template only has azure and aws branches. Add a vault branch (ESO extract with a path key):

# In helm/templates/externalsecrets.yaml — add after the aws branch:
{{- else if eq $config.provider "vault" }}
dataFrom:
  - extract:
      key: {{ $config.secretKey }}
{{- end }}

This is identical in shape to the AWS extract — ESO's vault provider supports the same extract.key syntax pointing to the Vault KV v2 path (without the secret/data/ prefix — ESO adds it automatically when using the vault ClusterSecretStore).

4b. Values file — `values-mod-auh1-dev-ask.yaml`¶

See values-mod-auh1-dev-ask.yaml below (created in Phase 4 of the sanitized repo).

4c. TOML config — `mod-auh1-dev-ask.toml`¶

The TOML replaces PP-specific hostnames and S3 endpoints with in-cluster DNS and Ceph RGW.

Key differences from PP TOML (ai71-pp-auh1-prod-ask.toml):

Key	PP value	Air-gap value
`vector_db.http_host`	`weaviate.weaviate.svc.cluster.local`	`knowledge-management-service-weaviate.ask.svc.cluster.local`
`vector_db.grpc_host`	`weaviate-grpc.weaviate.svc.cluster.local`	`knowledge-management-service-weaviate-grpc.ask.svc.cluster.local`
`storage.s3.endpoint_url`	`https://s3.pp.gov.ae/`	`https://s3.cl1.sq4.aegis.internal/`
`document_understanding.foundry_endpoint`	`http://litellm.foundry.svc.cluster.local/v1`	`http://foundry-litellm.foundry.svc.cluster.local:4000/v1`
`core_service.services_base_url`	`http://core-service.ask.svc.cluster.local`	`http://core-service.ask.svc.cluster.local` (same)

Phase 5 — ArgoCD Application¶

Add KMS to the ask ApplicationSet in mod-ask-devops. KMS follows the same multi-source pattern as other ask services.

# In ask-apps ApplicationSet — add one entry:
- name: knowledge-management-service
  repoURL: https://gitlab.cl1.sq4.aegis.internal/modgpt/mod-ask-knowledge-management-service.git
  targetRevision: main
  path: helm
  valuesFiles:
    - environments/values-mod-auh1-dev-ask.yaml

Or push the chart to GitLab and create the ArgoCD Application via the GitOps repo.

Phase 6 — Verify¶

6a. Pre-flight — all pods Running¶

kubectl get pods -n ask -l app.kubernetes.io/instance=knowledge-management-service -o wide

Expected workloads:

Workload	Expected status
`knowledge-management-service`	Running
`knowledge-management-service-ingestion-worker`	Running
`knowledge-management-service-generic-worker`	Running
`knowledge-management-service-live-ingestion-worker`	Running
`knowledge-management-service-migration` (Job)	Completed

6b. Migration job¶

kubectl get job -n ask | grep knowledge
# expect: knowledge-management-service-migration  1/1  <age>

kubectl logs -n ask job/knowledge-management-service-migration | tail -20
# expect: "Running upgrade..." → "Done"

6c. API health¶

# Direct (from within cluster)
kubectl exec -n ask deploy/knowledge-management-service -- \
  curl -sf http://localhost:8000/health/
# expect: {"status": "ok"} or similar 200

# Via ingress (if enabled)
curl -sk https://api.ask.mod.auh1.dev.dir/ask71/v2/kmsvc/health/

6d. Weaviate connectivity¶

kubectl exec -n ask deploy/knowledge-management-service -- \
  curl -sf http://knowledge-management-service-weaviate:80/v1/.well-known/ready
# expect: {"status": "ok"}

6e. Hatchet worker registration¶

kubectl logs -n ask deploy/knowledge-management-service-ingestion-worker --since=5m | \
  grep -iE "hatchet|worker|runner|connect|error|socket"
# expect: "starting runner..." — same as assistant-service pattern

6f. S3 connectivity¶

kubectl exec -n ask deploy/knowledge-management-service-ingestion-worker -- \
  python3 -c "
import boto3, os
s3 = boto3.client('s3',
    endpoint_url='https://s3.cl1.sq4.aegis.internal',
    aws_access_key_id=os.environ['STORAGE__S3__ACCESS_KEY_ID'],
    aws_secret_access_key=os.environ['STORAGE__S3__SECRET_ACCESS_KEY'],
    verify=False
)
buckets = [b['Name'] for b in s3.list_buckets()['Buckets']]
print('Buckets:', buckets)
assert 'ask-knowledge-management' in buckets, 'bucket missing!'
print('✅ S3 OK')
"

6g. Assistant-service tooling reachability test¶

From the assistant-service-tooling pod (which calls KMS):

kubectl exec -n ask deploy/assistant-service-tooling -- \
  curl -sf http://knowledge-management-service.ask.svc.cluster.local/health/ \
  && echo "✅ KMS reachable from tooling" || echo "❌ KMS unreachable"

Phase 7 — KMS Org Onboarding (one-time, required)¶

KMS maintains its own organisation registry separate from core-service. Until an org is onboarded, every collection create returns 404 organization_not_onboarded. This is a one-time call per org after deployment — it provisions the Weaviate tenant and writes the org's model config to the KMS DB.

What it sets¶

Field	Value (air-gap)	Source
`embedding_model_id`	`qwen/qwen3-embedding-0-6b`	KMS TOML default
`vlm_model_name`	`qwen/qwen3-5-122b-a10b-kms`	KMS TOML default
`blend_factor`	`0.75`	KMS TOML default
`parsing_strategy`	`generic`	KMS TOML default
`chunking_strategy`	`hierarchical`	KMS TOML default
`keyword_extraction`	`false`	KMS TOML default

Omitting all fields in the request body uses the KMS TOML defaults — the correct approach for the air-gap env where all models are set in config rather than per-org.

Command (run from bastion, one-time per org)¶

kubectl exec -i -n ask deploy/assistant-service-intelligence -- env PW="$SUPERPW" python3 - <<'EOF'
import os, httpx

KMS="http://knowledge-management-service.ask.svc.cluster.local"
CS="http://core-service.ask.svc.cluster.local"

tok=httpx.post(CS+"/api/v1/auth/login", json={
    "org_id":"378447542100166005",
    "email":"[email protected]",
    "password":os.environ["PW"]}, timeout=20).json().get("sessionToken")
H={"Authorization":"Bearer "+tok}

# onboard — empty body uses TOML defaults
r=httpx.post(KMS+"/organizations/onboard", headers=H, json={}, timeout=20)
print("onboard", r.status_code, r.text)

# confirm
r=httpx.get(KMS+"/organizations/me", headers=H, timeout=10)
print("org/me", r.status_code, r.text)
EOF

Verified (2026-06-23):

onboard 200
{"embedding_model_id":"qwen/qwen3-embedding-0-6b","blend_factor":0.75,"keyword_extraction":false,
 "parsing_strategy":"generic","chunking_strategy":"hierarchical","pre_retrieval_filter":false,
 "post_retrieval_filter":false,"small_file_max_threshold":2000,"tabular_preview_rows":5,
 "vlm_model_name":"qwen/qwen3-5-122b-a10b-kms"}

org/me 200
{"embedding_model_id":"qwen/qwen3-embedding-0-6b","blend_factor":0.75,"keyword_extraction":false,
 "parsing_strategy":"generic","chunking_strategy":"hierarchical","pre_retrieval_filter":false,
 "post_retrieval_filter":false,"small_file_max_threshold":2000,"tabular_preview_rows":5,
 "vlm_model_name":"qwen/qwen3-5-122b-a10b-kms"}

Must run before any collection or document operation

Calling POST /collections/ without onboarding returns 404 organization_not_onboarded. Onboarding is idempotent — safe to re-run.

Superadmin credentials required

The onboard endpoint requires an org-admin or superadmin session token. The machine PAT (APP__CORE_SERVICES_MACHINE_TOKEN) is not a user session and will be rejected.

Troubleshooting¶

Migration job fails — `relation "knowledge_service.*" does not exist`¶

The alembic migration ran but the schema target doesn't exist. Check that the postgresInit job completed first:

kubectl get job -n ask | grep kms
kubectl logs -n ask job/knowledge-management-service-postgres-init | tail -20

If the init job failed, the database or user wasn't created. Re-run manually:

kubectl exec -n ask knowledge-pg-1 -- \
  psql -U postgres -c "CREATE DATABASE knowledge_management;"
kubectl exec -n ask knowledge-pg-1 -- \
  psql -U postgres -d knowledge_management -c "CREATE USER kms_user WITH PASSWORD '<password>';"
kubectl exec -n ask knowledge-pg-1 -- \
  psql -U postgres -d knowledge_management -c "GRANT ALL ON DATABASE knowledge_management TO kms_user;"

Weaviate not ready — workers crash on start¶

Workers connect to Weaviate at startup. If Weaviate hasn't completed its startup (takes 30–90 s on first boot), workers may fail:

kubectl get pod -n ask -l app=weaviate -o wide
kubectl logs -n ask -l app=weaviate --since=5m | tail -30

Workers have failureThreshold: 180 / periodSeconds: 10 on startupProbe — they retry for 30 min. Wait for Weaviate ready before diagnosing worker crashes.

`HATCHET__CLIENT_TOKEN` missing / invalid JWT¶

Same root cause as assistant-service. Ensure the Vault secret has a real JWT (not a placeholder):

vault kv get -field=HATCHET__CLIENT_TOKEN secret/ask/knowledge-management-service/auh1-dev | \
  python3 -c "import sys,base64,json; parts=sys.stdin.read().strip().split('.'); print(json.loads(base64.b64decode(parts[1]+'==')))"
# expect: JSON with grpc_broadcast_address, iss, aud fields

S3 upload fails — `SSL: CERTIFICATE_VERIFY_FAILED`¶

Set verify_ssl = false in the TOML [storage.s3] section. Ceph RGW uses a self-signed cert in the air-gap env.

`DATABASE__HOST` not set — `can't resolve host` at startup¶

The extraEnv block must inject DATABASE__HOST. Confirm:

kubectl exec -n ask deploy/knowledge-management-service -- env | grep DATABASE__HOST
# expect: DATABASE__HOST=knowledge-pg-rw.ask.svc.cluster.local

If missing, add it to extraEnv in the values file and force-sync.