Service Config & Secrets Inventory¶

Complete inventory of TOML config variables and Vault secrets required for each ASK service deployment on core42 (cl1.sq4.aegis.internal).

Environment Naming Convention¶

Format:  ai71-{project}-{location}-{env}-{app}

PP prod: ai71-pp-auh1-prod-ask       ← reference environment
core42:  ai71-modgpt-auh1-dev-ask    ← this deployment

This name appears in:

TOML section headers: [ai71-modgpt-auh1-dev-ask.app]
configToml → env = "ai71-modgpt-auh1-dev-ask"
configTomlFileMountPath: /app/config/ai71-modgpt-auh1-dev-ask.toml

How Config + Secrets Work¶

config.toml  (ConfigMap — in Git via Helm values)
  ├── Non-sensitive values → hardcoded in TOML
  └── @format {env[VAR]}  → placeholder only

Pod environment variables
  └── from K8s Secret → ExternalSecret (ESO) → Vault KV
        resolved at runtime by @format {env[...]}

Rule: - Change TOML non-sensitive values → update values-cl1.yaml in Git → ArgoCD syncs - Change secrets → update Vault KV → ESO refreshes K8s Secret (within refreshInterval) - Never patch ConfigMap or K8s Secret directly — ArgoCD will revert

core-service¶

Source repo: ask-core-service
Helm chart path: ./helm
Workers: bulk-operations, billing-metering, cron-scheduler, job-run

TOML Config (non-sensitive — update in `values-cl1.yaml`)¶

Section	Key	PP Value	core42 Value
`app`	`env`	`ai71-pp-auh1-prod-ask`	`ai71-modgpt-auh1-dev-ask`
`database`	`host`	`postgresql.ask.svc.cluster.local`	`postgresql.ask.svc.cluster.local`
`database`	`user`	`core-service`	`core-service`
`database`	`port`	`5432`	`5432`
`database`	`name`	`core-service`	`core-service`
`storage`	`type`	`azure`	`s3` (Ceph RGW)
`storage.azure`	`account_name`	`ai71uaensask71sa03`	❌ Remove — use S3
`storage.s3`	`endpoint_url`	—	`https://s3.mod.auh1.dev.dir` (RGW VIP)
`storage.s3`	`bucket_name`	—	`core-service`
`storage.s3`	`region`	—	`us-east-1`
`iam_service`	`base_url`	`http://core-service-zitadel.ask.svc.cluster.local:8080`	✅ same
`iam_service`	`public_base_url`	`https://sso.ask.pp.gov.ae`	`https://sso.cl1.sq4.aegis.internal`
`iam_service`	`default_org_id`	`357511184963905565`	TBD
`foundry`	`base_url`	`http://litellm.foundry.svc.cluster.local:4000`	✅ same
`hatchet`	`client_tls_strategy`	—	`none`
`redis`	`host`	`assistant-service-redis-master.ask.svc.cluster.local`	✅ same
`redis`	`port`	`6379`	`6379`
`redis`	`ssl`	`true`	`false` (internal cluster)
`public`	`core_services_base_url`	`https://api.ask.pp.gov.ae/ask71/v2/svc`	`https://api.cl1.sq4.aegis.internal/ask71/v2/svc`
`knowledge_management_service`	`base_url`	`http://knowledge-management-service.ask.svc.cluster.local`	✅ same
`feature_flags`	`billing_enabled`	`false`	`false`
`feature_flags`	`public_signup_enabled`	`false`	`false`

Vault Secrets (from `@format {env[...]}` in TOML)¶

Store at: secret/cl1/core-service

Env Var	Vault Key	Description
`STORAGE__ACCOUNT_KEY`	`storage_account_key`	⚠️ Azure only — replace with S3 creds for Ceph RGW
`STORAGE__S3__ACCESS_KEY_ID`	`storage_s3_access_key_id`	Ceph RGW access key (new)
`STORAGE__S3__SECRET_ACCESS_KEY`	`storage_s3_secret_access_key`	Ceph RGW secret key (new)
`IAM_SERVICE__PAT`	`iam_service_pat`	Zitadel Personal Access Token
`NOTIFICATIONS__MAILCHIMP_API_KEY`	`notifications_mailchimp_api_key`	Mailchimp API key
`FOUNDRY__API_KEY`	`foundry_api_key`	LiteLLM/Foundry API key
`HATCHET__CLIENT_TOKEN`	`hatchet_client_token`	Hatchet workflow engine token
`REDIS__PASSWORD`	`redis_password`	Redis auth password
`DB_PASSWORD`	`db_password`	PostgreSQL password for core-service user

Helm Values Changes for core42¶

# values-cl1.yaml (add/override these)
image:
  repository: harbor.cl1.sq4.aegis.internal/ask/core-service

imagePullSecrets:
  - name: harbor-pull-secret

ingress:
  hosts:
    - host: api.cl1.sq4.aegis.internal

externalSecrets:
  coreService:
    create: true
    provider: vault
    secretStore:
      kind: ClusterSecretStore
      name: vault-backend
    secretKey: secret/data/cl1/core-service

envConf:
  configTomlFileMountPath: /app/config/ai71-modgpt-auh1-dev-ask.toml
  configToml: |
    [ai71-modgpt-auh1-dev-ask.app]
    env = "ai71-modgpt-auh1-dev-ask"
    # ... update all URLs and storage type to S3/Ceph RGW

assistant-service¶

Source repo: ask-assistant-service
Helm chart path: ./helm
Workers: generic-worker, notifications-worker, intelligence, tooling
Sub-chart: Redis (Bitnami 22.0.7)

TOML Config (non-sensitive — update in `values-cl1.yaml`)¶

Section	Key	PP Value	core42 Value
`app`	`env`	`ai71-pp-auh1-prod-ask`	`ai71-modgpt-auh1-dev-ask`
`app`	`base_url`	`http://api.ask.pp.gov.ae`	`https://api.cl1.sq4.aegis.internal`
`app`	`fastapi_root_path`	`/ask71/v2/astsvc`	`/ask71/v2/astsvc`
`core_service`	`url`	`http://core-service.ask.svc.cluster.local/api/v1`	✅ same
`core_service`	`services_base_url`	`http://core-service.ask.svc.cluster.local`	✅ same
`database`	`host`	`postgresql.ask.svc.cluster.local`	`postgresql.ask.svc.cluster.local`
`database`	`user`	`assistant-service`	`assistant-service`
`foundry`	`endpoint`	`http://litellm.foundry.svc.cluster.local:4000/v1`	✅ same
`storage`	`cdn_base_url`	`https://cdn.ask.pp.gov.ae/static`	`https://cdn.cl1.sq4.aegis.internal/static`
`kms`	`base_url`	`http://knowledge-management-service.ask.svc.cluster.local`	✅ same
`hatchet`	`client_tls_strategy`	`none`	`none`
`hatchet`	`generic_worker_slots`	`5`	`5`
`redis`	`host`	`assistant-service-redis-master.ask.svc.cluster.local`	✅ same
`daytona`	`enable_code_execution`	`false`	`false`
`feature_flags`	`mode`	`offline`	`offline`
`assistants`	`frontend_url`	`https://ask.pp.gov.ae/`	`https://ask.cl1.sq4.aegis.internal/`
`assistants`	`service_url`	`https://api.ask.pp.gov.ae/ask71/v2/astsvc`	`https://api.cl1.sq4.aegis.internal/ask71/v2/astsvc`
`tooling.data_analysis`	`*_model`	`qwen/qwen3-5-122b-a10b`	TBD — depends on Foundry model config

Vault Secrets¶

Store at: secret/cl1/assistant-service

Env Var	Vault Key	Description
`HATCHET__CLIENT_TOKEN`	`hatchet_client_token`	Hatchet workflow engine token
`DB_PASSWORD`	`db_password`	PostgreSQL password for assistant-service user
`REDIS_PASSWORD`	`redis_password`	Redis auth password (from Redis sub-chart)

Helm Values Changes for core42¶

image:
  repository: harbor.cl1.sq4.aegis.internal/ask/assistant-service

imagePullSecrets:
  - name: harbor-pull-secret

redis:
  image:
    registry: harbor.cl1.sq4.aegis.internal
    repository: bitnamilegacy/redis
    tag: 8.2.1-debian-12-r0
  global:
    imagePullSecrets:
      - name: harbor-pull-secret

ingress:
  hosts:
    - host: api.cl1.sq4.aegis.internal

knowledge-management-service (KMS)¶

Source repo: ask-knowledge-management-service
Helm chart path: ./helm
Workers: generic-worker, ingestion-worker, live-ingestion-worker
Sub-charts: PostgreSQL (Bitnami 16.4.1), Weaviate (17.7.0)

TOML Config (non-sensitive — update in `values-cl1.yaml`)¶

Section	Key	PP Value	core42 Value
`core_service`	`services_base_url`	`http://core-service.ask.svc.cluster.local`	✅ same
`database`	`host`	`@format {env[DATABASE__HOST]}`	`@format {env[DATABASE__HOST]}` (injected via extraEnv)
`database`	`port`	`5432`	`5432`
`database`	`name`	`knowledge_management`	`knowledge_management`
`database`	`user`	`kms_user`	`kms_user`
`database`	`db_schema`	`knowledge_service`	`knowledge_service`
`document_understanding`	`foundry_endpoint`	`http://litellm.foundry.svc.cluster.local/v1`	✅ same
`storage`	`type`	`s3`	`s3` ✅ (Ceph RGW)
`storage.s3`	`endpoint_url`	`https://s3.pp.gov.ae/`	`https://s3.mod.auh1.dev.dir` (RGW VIP)
`storage.s3`	`bucket_name`	`ask-knowledge-management`	`ask-knowledge-management`
`storage.s3`	`region`	`us-east-1`	`us-east-1`
`storage.s3`	`verify_ssl`	`false`	`false`
`vector_db`	`http_host`	`weaviate.weaviate.svc.cluster.local`	✅ same
`vector_db`	`http_port`	`80`	`80`
`vector_db`	`grpc_host`	`weaviate-grpc.weaviate.svc.cluster.local`	✅ same

Vault Secrets¶

Store at: secret/cl1/knowledge-management-service

Env Var	Vault Key	Description
`DATABASE__PASSWORD`	`db_password`	PostgreSQL password for kms_user
`STORAGE__S3__ACCESS_KEY_ID`	`storage_s3_access_key_id`	Ceph RGW S3 access key
`STORAGE__S3__SECRET_ACCESS_KEY`	`storage_s3_secret_access_key`	Ceph RGW S3 secret key

Helm Values Changes for core42¶

image:
  repository: harbor.cl1.sq4.aegis.internal/ask/ask-knowledge-management-service

imagePullSecrets:
  - name: harbor-pull-secret

postgresql:
  image:
    registry: harbor.cl1.sq4.aegis.internal
    repository: bitnamilegacy/postgresql
    tag: 17.6.0-debian-12-r4
  primary:
    persistence:
      storageClass: csi-rbd-sc    # Ceph RBD (not longhorn)
      size: 50Gi                   # adjust for this environment
  global:
    imagePullSecrets:
      - name: harbor-pull-secret

# Remove s3-ca-bundle volume — not needed for Ceph RGW HTTP
volumes: []
volumeMounts: []

ingress:
  hosts:
    - host: api.cl1.sq4.aegis.internal

frontend¶

Source repo: ask-frontend
Helm chart path: ./helm
No sub-charts

TOML Config¶

Frontend has no TOML config file — it uses environment variables directly.

Environment Variables¶

Env Var	PP Value	core42 Value
`NODE_TLS_REJECT_UNAUTHORIZED`	`0`	`0` (internal CA not trusted by Node yet)
Backend API URL	configured at build time or runtime	`https://api.cl1.sq4.aegis.internal`

Vault Secrets¶

Frontend has no Vault secrets — it is a stateless Next.js app with no direct DB/S3 access.

Helm Values Changes for core42¶

image:
  repository: harbor.cl1.sq4.aegis.internal/ask/frontend

imagePullSecrets:
  - name: harbor-pull-secret

ingress:
  hosts:
    - host: ask.cl1.sq4.aegis.internal
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: wildcard-tls-secret
      hosts:
        - ask.cl1.sq4.aegis.internal

Vault KV Structure for core42¶

All secrets stored under secret/cl1/<service>/:

secret/
└── cl1/
    ├── core-service/
    │   ├── storage_s3_access_key_id
    │   ├── storage_s3_secret_access_key
    │   ├── iam_service_pat
    │   ├── notifications_mailchimp_api_key
    │   ├── foundry_api_key
    │   ├── hatchet_client_token
    │   ├── redis_password
    │   └── db_password
    ├── assistant-service/
    │   ├── hatchet_client_token
    │   ├── db_password
    │   └── redis_password
    └── knowledge-management-service/
        ├── db_password
        ├── storage_s3_access_key_id
        └── storage_s3_secret_access_key

Populate Vault (run from bastion):

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal

# core-service
vault kv put secret/cl1/core-service \
  storage_s3_access_key_id="<ceph-rgw-access-key>" \
  storage_s3_secret_access_key="<ceph-rgw-secret-key>" \
  iam_service_pat="<zitadel-pat>" \
  notifications_mailchimp_api_key="<mailchimp-key>" \
  foundry_api_key="<litellm-api-key>" \
  hatchet_client_token="<hatchet-token>" \
  redis_password="<redis-password>" \
  db_password="<postgres-password>"

# assistant-service
vault kv put secret/cl1/assistant-service \
  hatchet_client_token="<hatchet-token>" \
  db_password="<postgres-password>" \
  redis_password="<redis-password>"

# knowledge-management-service
vault kv put secret/cl1/knowledge-management-service \
  db_password="<postgres-password>" \
  storage_s3_access_key_id="<ceph-rgw-access-key>" \
  storage_s3_secret_access_key="<ceph-rgw-secret-key>"

unset VAULT_ADDR

Summary Checklist¶

Vault secrets to populate¶

[ ] secret/cl1/core-service — 8 keys
[ ] secret/cl1/assistant-service — 3 keys
[ ] secret/cl1/knowledge-management-service — 3 keys

TOML / values changes per service¶

[ ] core-service — storage: azure → S3/Ceph, URLs, env name
[ ] assistant-service — URLs, Redis image tag, env name
[ ] KMS — S3 endpoint → RGW VIP, PostgreSQL storageClass → csi-rbd-sc
[ ] frontend — image repo, ingress hostname

ExternalSecrets provider¶

[ ] All services: change provider: azure → provider: vault (or use secretKey with Vault path)
[ ] secretStore.name → vault-backend (Phase 7 ClusterSecretStore)

Service Config & Secrets Inventory¶

Environment Naming Convention¶

How Config + Secrets Work¶

core-service¶

TOML Config (non-sensitive — update in values-cl1.yaml)¶

Vault Secrets (from @format {env[...]} in TOML)¶

Helm Values Changes for core42¶

assistant-service¶

TOML Config (non-sensitive — update in values-cl1.yaml)¶

Vault Secrets¶

Helm Values Changes for core42¶

knowledge-management-service (KMS)¶

TOML Config (non-sensitive — update in values-cl1.yaml)¶

Vault Secrets¶

Helm Values Changes for core42¶

frontend¶

TOML Config¶

Environment Variables¶

Vault Secrets¶

Helm Values Changes for core42¶

Vault KV Structure for core42¶

Summary Checklist¶

Vault secrets to populate¶

TOML / values changes per service¶

ExternalSecrets provider¶

TOML Config (non-sensitive — update in `values-cl1.yaml`)¶

Vault Secrets (from `@format {env[...]}` in TOML)¶

TOML Config (non-sensitive — update in `values-cl1.yaml`)¶

TOML Config (non-sensitive — update in `values-cl1.yaml`)¶