PP Environment — Abu Dhabi Production (`ai71-pp-auh1-prod-ask`)¶

On-premises government cloud deployment
Cluster: RKE2 on bare-metal / private cloud, Abu Dhabi (AUH1)
Domain: *.ask.pp.gov.ae
Registry: registry.pp.gov.ae (Harbor, fully air-gapped)
Secrets backend: HashiCorp Vault (in-cluster) — not Azure Key Vault
ArgoCD branch: feat/pp-deployment (devops repo)

Why PP Is Different¶

The PP environment is a sovereign, air-gapped deployment for a government customer. Key differences from all Azure environments:

Concern	Azure environments	PP environment
Kubernetes	AKS (managed)	RKE2 (self-managed, bare-metal)
Container registry	Azure Container Registry	Harbor (`registry.pp.gov.ae`)
Secrets	Azure Key Vault + ESO	HashiCorp Vault + ESO
PostgreSQL	Azure PostgreSQL Flexible Server	Bitnami Helm chart (in-cluster)
Redis	Azure Cache for Redis	Bitnami Redis (in-cluster subchart)
Object storage	Azure Blob	S3-compatible (`s3.pp.gov.ae`)
LLM proxy	AI71 Foundry (cloud)	LiteLLM in-cluster (`litellm.foundry.svc.cluster.local:4000`)
Feature flags	LaunchDarkly / Unleash (cloud)	Offline mode with bootstrap JSON file
Daytona sandboxes	Enabled	Disabled (`enable_code_execution: false`)
TLS termination	Azure App Gateway / ACR	F5 load balancer at edge → ingress-nginx
Image source	Built and pushed by CI to ACR	Mirrored from ACR to Harbor via `mirror-image.sh`
Storage class	Azure managed disks	Longhorn
Extra service	—	Voice Service (`ask-voice-service`)

DNS & Ingress Map¶

All traffic enters via F5 → ingress-nginx. TLS is terminated at the ingress using the ask-pp-gov-ae-cert TLS Secret (private CA).

Hostname	Backend Service	Path	Namespace
`ask.pp.gov.ae`	Frontend	`/`	`ask`
`chat.ask.pp.gov.ae`	Frontend	`/`	`ask`
`admin.ask.pp.gov.ae`	Frontend	`/`	`ask`
`api.ask.pp.gov.ae`	Core Service	`/ask71/v2/svc/(.*)` → `/`	`ask`
`api.ask.pp.gov.ae`	Core Service	`/(.*)` (catch-all)	`ask`
`api.ask.pp.gov.ae`	Assistant Service	`/ask71/v2/astsvc(/\|$)(.*)` → `/`	`ask`
`api.ask.pp.gov.ae`	KMS	`/ask71/v2/kmsvc(/\|$)(.*)` → `/`	`ask`
`sso.ask.pp.gov.ae`	Zitadel (login)	`/ui/v2/login`	`ask`
`sso.ask.pp.gov.ae`	Zitadel (gRPC)	`/` (GRPC backend protocol)	`ask`
`hatchet.ask.pp.gov.ae`	Hatchet API	`/api`	`hatchet`
`hatchet.ask.pp.gov.ae`	Hatchet Frontend	`/`	`hatchet`
`litellm.ask.pp.gov.ae`	LiteLLM	`/`	`foundry`

Note: TLS cert Secret ask-pp-gov-ae-cert must include the full chain (leaf + intermediates). F5 terminates TLS at the edge; internal cluster traffic is plain HTTP. NODE_TLS_REJECT_UNAUTHORIZED=0 is set on the frontend pod because it calls the backend via public HTTPS hostnames signed by the private CA.

ArgoCD App-of-Apps Hierarchy¶

ai71-pp-auh1-prod-ask                    ← Root App
  repoURL: ask-devops  feat/pp-deployment
  path: environments/pp/auh1/prod/02-argocd-application/_root
  namespace: argocd
       │
       ▼
  ai71-pp-auh1-prod-ask.ask              ← App-of-Apps
    path: environments/pp/auh1/prod/02-argocd-application/ask
         │
         ├─ ai71-pp-auh1-prod-ask.ask-core-service
         ├─ ai71-pp-auh1-prod-ask.ask-assistant-service
         ├─ ai71-pp-auh1-prod-ask.ask-frontend
         ├─ ai71-pp-auh1-prod-ask.ask-km-service
         ├─ ai71-pp-auh1-prod-ask.ask-voice-service
         ├─ ai71-pp-auh1-prod-ask.ask-hatchet          ← multi-source (chart: Harbor OCI, values: Git)
         ├─ ai71-pp-auh1-prod-ask.weaviate              ← multi-source
         └─ ai71-pp-auh1-prod-ask.postgresql            ← multi-source

All Application CRs have syncPolicy.automated: { prune: true, selfHeal: true }.

ArgoCD Applications — Summary Table¶

ArgoCD App Name	Source Repo	Branch	Helm Values	Namespace	Pinned Image Tag
`ask-core-service`	`ask-core-service.git`	`staging`	`values-ai71-pp-auh1-prod-ask.yaml`	`ask`	`b4895f50`
`ask-assistant-service`	`ask-assistant-service.git`	`main`	`values-ai71-pp-auh1-prod-ask.yaml`	`ask`	`d1fec387`
`ask-frontend`	`ask-frontend.git`	`feature/gitlab_manual_build`	`values-ai71-pp-auh1-prod-ask.yaml`	`ask`	`staging-139ce0f1-pp-prod`
`ask-km-service`	`ask-knowledge-management-service.git`	`main`	`values-ai71-pp-auh1-prod-ask.yaml` + `ai71-pp-auh1-prod-ask.toml`	`ask`	`88ca403f`
`ask-voice-service`	`ask-voice-service.git`	`main`	`values-ai71-pp-auh1-prod-ask.yaml`	`ask`	`553d1ce0`
`ask-hatchet`	`registry.pp.gov.ae/helm-charts` chart	`0.10.0`	`values-ai71-pp-auh1-prod.yaml` (from devops Git)	`hatchet`	—
`weaviate`	`registry.pp.gov.ae/helm-charts` chart	`17.7.0`	`values-ai71-pp-auh1-prod.yaml` (from devops Git)	`weaviate`	—
`postgresql`	`registry.pp.gov.ae/helm-charts` chart	`15.5.0`	`values-ai71-pp-auh1-prod.yaml` (from devops Git)	`ask`	—

Container Images — PP Registry¶

All application images are mirrored from ACR to Harbor before deployment.

Service	ACR source	Harbor target (`registry.pp.gov.ae`)
Core Service	`ai71uaenprodask71acr01.azurecr.io/core-service`	`ask/ask-core-service`
Core Service Bulk Worker	`ai71uaenprodask71acr01.azurecr.io/core-service-bulk-operations-worker`	`ask/ask-core-service-bulk-operations-worker`
Assistant Service	`ai71uaenprodask71acr01.azurecr.io/assistant-service`	`ask/ask-assistant-service`
Frontend	`ai71uaenprodask71acr01.azurecr.io/frontend`	`ask/frontend-prod`
KMS	`ai71uaenprodask71acr01.azurecr.io/ask-knowledge-management-service`	`ask/ask-knowledge-management-service`

Third-Party Images (mirrored to Harbor)¶

Image	Harbor path	Used by
`python:3.12-slim-trixie`	— (base, build-time only)	All Python services
`node:22-alpine`	— (base, build-time only)	Frontend
`bitnami/postgresql:17.6.0-debian-12-r4`	`bitnamilegacy/postgresql:17.6.0-debian-12-r4`	Hatchet subchart, shared PostgreSQL
`bitnami/postgresql:18.1.0`	`bitnami/postgresql:18.1.0`	KMS subchart
`bitnami/redis:8.2.1-debian-12-r0`	`bitnamilegacy/redis:8.2.1-debian-12-r0`	Assistant Service Redis subchart
`bitnami/rabbitmq:4.1.3-debian-12-r1`	`bitnamilegacy/rabbitmq:4.1.3-debian-12-r1`	Hatchet RabbitMQ subchart
`library/postgres:16-alpine`	`dockerhub/library/postgres:16-alpine`	Hatchet DB init
`weaviate/weaviate:1.34.0`	`dockerhub/weaviate:1.34.0`	Weaviate
`library/alpine:3.20`	`dockerhub/library/alpine:3.20`	Weaviate init container
`hatchet/hatchet-api:v0.71.0`	`ghcr/hatchet/hatchet-api:v0.71.0`	Hatchet API
`hatchet/hatchet-engine:v0.71.0`	`ghcr/hatchet/hatchet-engine:v0.71.0`	Hatchet Engine
`hatchet/hatchet-frontend:v0.71.0`	`ghcr/hatchet/hatchet-frontend:v0.71.0`	Hatchet UI
`hatchet/hatchet-migrate:v0.71.0`	`ghcr/hatchet/hatchet-migrate:v0.71.0`	Hatchet migration job
`hatchet/hatchet-admin:v0.71.0`	`ghcr/hatchet/hatchet-admin:v0.71.0`	Hatchet setup/seed jobs

Secrets Management — HashiCorp Vault¶

Setup¶

Vault is deployed in-cluster (namespace: hashicorp) and configured via Terraform (02-hashicorp-vault-configure/main.tf):

KV v2 engine mounted at kv/
Kubernetes auth backend enabled
ClusterSecretStore named vault-backend pointing to http://vault.hashicorp:8200
external-secrets-role bound to ServiceAccount external-secrets (namespace external-secrets)
ESO policy: read-only on kv/data/ask/prod/*

Vault KV Paths → Kubernetes Secrets¶

Vault KV Path	K8s Secret Name	Namespace	Consumed By
`kv/data/ask/prod/core-service/environment-secrets`	`core-service-secret`	`ask`	Core Service pod
`kv/data/ask/prod/assistant-service/environment-secrets`	`assistant-service-secret`	`ask`	Assistant Service pod
`kv/data/ask/prod/knowledge-management-service/environment-secrets`	`knowledge-management-service-secret`	`ask`	KMS pod
`kv/data/ask/prod/postgres/environment-secrets`	`postgres-secret`	`ask`	Shared PostgreSQL
`kv/data/ask/prod/zitadel/environment-secrets`	`zitadel-secret`	`ask`	Zitadel IAM
`kv/data/ask/prod/weaviate/environment-secrets`	`weaviate-secret`	`weaviate`	Weaviate (API key)
`kv/data/ask/prod/hatchet/environment-secrets`	`hatchet-secret`	`hatchet`	Hatchet (RabbitMQ password etc.)
`kv/data/ask/prod/litellm/environment-variables`	`litellm-secret`	`foundry`	LiteLLM proxy

All ExternalSecret resources use refresh interval 1 minute.

Infrastructure Components¶

Shared PostgreSQL (Bitnami chart `15.5.0`)¶

ArgoCD app: ai71-pp-auh1-prod-ask.postgresql
Namespace: ask
Architecture: Primary + 1 read replica
Storage: 220 Gi primary, 20 Gi replica (default storage class)
Node affinity: pinned to rke2-worker-node-11 (both primary and replica)
Secret: postgres-secret (key: postgres-password)
Tuning: max_connections=200, shared_buffers=4GB, effective_cache_size=12GB, WAL replication enabled

Weaviate (chart `17.7.0`, app image `1.34.0`)¶

ArgoCD app: ai71-pp-auh1-prod-ask.weaviate
Namespace: weaviate
Replicas: 3 (one per node: rke2-worker-node-07/08/09)
Storage: 512 Gi per replica
Auth: API key only (admin user), anonymous access disabled
Node affinity: role=weaviate (required, 1:1 pod-to-node via podAntiAffinity)
API key secret: weaviate-secret key AUTHENTICATION_APIKEY_ALLOWED_KEYS

Hatchet (chart `0.10.0`, app image `v0.71.0`)¶

ArgoCD app: ai71-pp-auh1-prod-ask.ask-hatchet
Namespace: hatchet
URL: http://hatchet-api.hatchet.svc.cluster.local:8080 (internal); hatchet.ask.pp.gov.ae (external)
Includes subcharts: PostgreSQL (Bitnami), RabbitMQ (Bitnami)
PostgreSQL node: pinned to rke2-worker-node-10
RabbitMQ preset: medium (500m CPU, 1Gi RAM)
Engine memory limit: 8 Gi
Secret: hatchet-secret (RabbitMQ password, shared config override)

LiteLLM (in-cluster LLM proxy)¶

Namespace: foundry
Internal URL: http://litellm.foundry.svc.cluster.local:4000
External URL: litellm.ask.pp.gov.ae
Secret: litellm-secret (from Vault kv/data/ask/prod/litellm/environment-variables)
Serves OSS models (openai/gpt-oss-120b) for on-prem LLM inference

Zitadel IAM (v4.5.0, deployed as Core Service subchart)¶

Internal gRPC: http://core-service-zitadel.ask.svc.cluster.local:8080
External SSO: https://sso.ask.pp.gov.ae
Login UI: https://sso.ask.pp.gov.ae/ui/v2/login (Zitadel v4 login app)
Database: PostgreSQL zitadel DB on postgresql.ask.svc.cluster.local
Default org ID: 357511184963905565
TLS: Disabled inside cluster (TLS handled by F5/ingress); ExternalSecure: true for external URL

Node Topology¶

Pods are scheduled to dedicated node pools using nodeSelector + tolerations:

Node Role	`nodeSelector`	Workloads
`ask`	`node.kubernetes.io/role: ask`	All ASK application pods (core, assistant, frontend, kms)
`postgres`	`role: postgres`	Shared PostgreSQL (node-11), Hatchet PostgreSQL (node-10)
`weaviate`	`role: weaviate`	Weaviate pods (nodes 7, 8, 9)

Per-Service PP Configuration¶

Core Service¶

Config	PP Value
`app.env`	`ai71-pp-auh1-prod-ask`
`database.host`	`postgresql.ask.svc.cluster.local`
`database.name`	`core-service`
`database.user`	`core-service`
`iam_service.base_url`	`http://core-service-zitadel.ask.svc.cluster.local:8080`
`iam_service.public_base_url`	`https://sso.ask.pp.gov.ae`
`iam_service.default_org_id`	`357511184963905565`
`foundry.base_url`	`http://litellm.foundry.svc.cluster.local:4000`
`storage.type`	`azure` (using Azure Blob CDN for static assets only)
`storage.azure.account_name`	`ai71uaensask71sa03`
`storage.cdn_base_url`	`https://cdn.ask.stg.ai71services.ai`
`redis.host`	`assistant-service-redis-master.ask.svc.cluster.local`
`redis.ssl`	`true` / `cluster_mode: true`
`feature_flags.billing_enabled`	`false`
`feature_flags.public_signup_enabled`	`false`
`public.core_services_base_url`	`https://api.ask.pp.gov.ae/ask71/v2/svc`
`knowledge_management_service.base_url`	`http://knowledge-management-service.ask.svc.cluster.local`
Workers enabled	`bulkOperationsWorker` (3 replicas); billing/cron/job workers disabled
`image.repository`	`registry.pp.gov.ae/ask/ask-core-service`

Assistant Service¶

Config	PP Value
`app.env`	`ai71-pp-auh1-prod-ask`
`app.base_url`	`http://api.ask.pp.gov.ae`
`app.fastapi_root_path`	`/ask71/v2/astsvc`
`core_service.url`	`http://core-service.ask.svc.cluster.local/api/v1`
`database.host`	`postgresql.ask.svc.cluster.local`
`database.user`	`assistant-service`
`foundry.endpoint`	`http://litellm.foundry.svc.cluster.local:4000/v1`
`foundry.reranker.endpoint`	`http://litellm.foundry.svc.cluster.local:4000/v1/rerank`
`storage.cdn_base_url`	`https://cdn.ask.pp.gov.ae/static`
`kms.base_url`	`http://knowledge-management-service.ask.svc.cluster.local`
`hatchet.client_tls_strategy`	`none`
`hatchet.generic_worker_slots`	`5`
`redis.host`	`assistant-service-redis-master.ask.svc.cluster.local`
`daytona.enable_code_execution`	`false`
`tool_approval.flow_enabled`	`false`
`feature_flags.mode`	`offline` (bootstrap: `./config/feature-flags/pp-prod.json`)
`assistants.default_public_assistant_name`	`mujeeb`
`assistants.frontend_url`	`https://ask.pp.gov.ae/`
`tooling.data_analysis.*_model`	`openai/gpt-oss-120b` (OSS on-prem model)
`image.repository`	`registry.pp.gov.ae/ask/ask-assistant-service`
Redis subchart image	`registry.pp.gov.ae/bitnamilegacy/redis:8.2.1-debian-12-r0`
`postDeployJob.enabled`	`false`

Frontend¶

Config	PP Value
`image.repository`	`registry.pp.gov.ae/ask/frontend-prod`
`replicaCount`	1 (autoscaling: min 3, max 100, CPU target 80%)
`NODE_TLS_REJECT_UNAUTHORIZED`	`0` (private CA cert workaround)
Ingress hosts	`ask.pp.gov.ae`, `chat.ask.pp.gov.ae`, `admin.ask.pp.gov.ae`
TLS secret	`ask-pp-gov-ae-cert`
Liveness/readiness	`/api/health`, initial delay 60s

Knowledge Management Service (KMS)¶

Config	PP Value
`image.repository`	`registry.pp.gov.ae/ask/ask-knowledge-management-service`
`core_service.services_base_url`	`http://core-service.ask.svc.cluster.local`
`database.host`	Dynamic: `{{ .Release.Name }}-postgresql-primary`
`database.name`	`knowledge_management`
`database.user`	`kms_user`
`database.pool_size`	`200`
`document_understanding.di_resource_name`	`azure-document-intelligence-pp-prod`
`document_understanding.foundry_endpoint`	`http://litellm.foundry.svc.cluster.local/v1`
`organization.vlm_model_name`	`Qwen/Qwen3.5-9B`
`storage.type`	`s3` (S3-compatible: `s3.pp.gov.ae`)
`storage.s3.bucket_name`	`ask-knowledge-management`
`storage.s3.endpoint_url`	`https://s3.pp.gov.ae/`
`storage.s3.region`	`us-east-1`
`storage.s3.verify_ssl`	`false`
`vector_db.http_host`	`weaviate.weaviate.svc.cluster.local`
`vector_db.grpc_host`	`weaviate-grpc.weaviate.svc.cluster.local`
S3 CA bundle	Mounted from Secret `s3-ca-bundle` → `/config/s3/s3-cert.pem`
PostgreSQL subchart	enabled — Bitnami `18.1.0` with replication, Longhorn 300 Gi
`ingestionAutoscaling.minReplicas`	`5`

PP Secrets Required in Vault¶

Below is the combined list of secrets that must be populated in HashiCorp Vault before deploying to PP.

`kv/data/ask/prod/core-service/environment-secrets`¶

Key	Description
`DATABASE__PASSWORD`	PostgreSQL password for `core-service` user
`SECURITY__API_KEY_SALT`	API key hashing salt
`SECURITY__MASTER_ENCRYPTION_KEY`	Master encryption key
`IAM_SERVICE__PAT`	Zitadel machine account PAT
`NOTIFICATIONS__MAILCHIMP_API_KEY`	Mailchimp API key
`STORAGE__ACCOUNT_KEY`	Azure Blob account key (CDN static assets)
`STORAGE__SAS_TOKEN`	Azure Blob SAS token
`HATCHET__CLIENT_TOKEN`	Hatchet client token
`REDIS__PASSWORD`	Redis AUTH password
`FOUNDRY__API_KEY`	LiteLLM API key
`APP__CORE_SERVICES_MACHINE_TOKEN`	Machine token for internal job → API calls

`kv/data/ask/prod/assistant-service/environment-secrets`¶

Key	Description
`DATABASE__PASSWORD`	PostgreSQL password for `assistant-service` user
`HATCHET__CLIENT_TOKEN`	Hatchet client token
`CORE_SERVICE__JWT_SECRET`	JWT secret for Core Service calls
`CORE_SERVICE_ENCRYPTION_KEY`	Encryption key for Core Service tokens
`AI71_FOUNDRY_KEY`	LiteLLM / Foundry API key
`AI71_AUTH_STATE_SECRET`	OAuth state encryption secret
`REDIS_PASSWORD`	Redis AUTH password (matches core-service)

`kv/data/ask/prod/knowledge-management-service/environment-secrets`¶

Key	Description
`DATABASE__PASSWORD`	PostgreSQL password for `kms_user`
`APP__MASTER_ENCRYPTION_KEY`	Master encryption key
`CORE_SERVICE__JWT_SECRET`	JWT secret for inter-service auth
`DOCUMENT_UNDERSTANDING__DI_SUBSCRIPTION_KEY`	Azure Document Intelligence key
`DOCUMENT_UNDERSTANDING__FOUNDRY_KEY`	LiteLLM / Foundry key for DU
`DOCUMENT_UNDERSTANDING__KEYWORD_EXTRACTION__NER_TOKEN`	NER model token
`STORAGE__S3__ACCESS_KEY_ID`	S3 access key (`s3.pp.gov.ae`)
`STORAGE__S3__SECRET_ACCESS_KEY`	S3 secret key
`VECTOR_DB__API_KEY`	Weaviate API key

`kv/data/ask/prod/postgres/environment-secrets`¶

Key	Description
`postgres-password`	Shared PostgreSQL admin/user password

`kv/data/ask/prod/zitadel/environment-secrets`¶

Key	Description
`ZITADEL_DATABASE_POSTGRES_USER_PASSWORD`	Zitadel DB user password
`masterkey`	Zitadel master key (must be in `zitadel-secret`)

`kv/data/ask/prod/weaviate/environment-secrets`¶

Key	Description
`AUTHENTICATION_APIKEY_ALLOWED_KEYS`	Weaviate API key (maps to `admin` user)

`kv/data/ask/prod/hatchet/environment-secrets`¶

Key	Description
`RABBITMQ_PASSWORD`	RabbitMQ auth password
(other Hatchet shared config keys)	As required by `hatchet-shared-config` secret

`kv/data/ask/prod/litellm/environment-variables`¶

Key	Description
(LiteLLM config keys)	API keys for OSS model endpoints, master key, etc.

Image Mirroring¶

Before deploying a new image version, mirror it from ACR to Harbor:

# Script: environments/pp/auh1/prod/03-scripts/mirror-image.sh
# Source: ai71uaenprodask71acr01.azurecr.io
# Target: registry.pp.gov.ae/ask

./mirror-image.sh [--dry-run] [--parallel]

Service mappings (configured in script):

assistant-service                      → ask/ask-assistant-service
core-service                           → ask/ask-core-service
core-service-bulk-operations-worker    → ask/ask-core-service-bulk-operations-worker

The script retries up to 3 times per image and writes structured logs to ./logs/mirror-<timestamp>.log.

PP Environment — Abu Dhabi Production (ai71-pp-auh1-prod-ask)¶