ASK DevOps¶

Repo: ask-devops (GitLab: ai71.ai/devops/ask-devops)
Type: Infrastructure-only — no application containers
Default branch: main | Active development branch: develop
PP deployment branch: feat/pp-deployment

Overview¶

The DevOps repo is the control plane for the entire ASK platform. It owns:

Kubernetes manifests and ArgoCD Application / AppProject CRDs
Terraform modules for Azure infrastructure (AKS, ACR, databases, networking, Vault)
Helm values for platform-level components (Hatchet, Weaviate, PostgreSQL, ArgoCD, Reloader)
External Secrets ExternalSecret / ClusterSecretStore manifests
Image mirror scripts for air-gapped environments
Application add-ons (Daytona, Observability stack, Oracle DB, Kyverno policies)

Repository Structure¶

ask-devops/
├── applications/
│   ├── ask/                        ← ASK app-level manifests (ExternalSecrets, Daytona, Hatchet deps, Oracle DB)
│   │   ├── external-secrets/       ← ExternalSecret + PushSecret per service
│   │   ├── daytona/                ← Daytona Helm values + raw manifests
│   │   ├── hatchet/                ← Hatchet dependency values
│   │   ├── oracle-db/              ← Oracle DB StatefulSet + manifests
│   │   └── applicationset-*.yaml  ← ApplicationSet for dev preview environments
│   ├── cluster-addons/             ← Platform add-ons (ArgoCD, ESO, external-dns, Kyverno, namespaces)
│   ├── dependencies/               ← Shared infra dependency values (hatchet, postgresql, weaviate per env)
│   └── observability/              ← Observability stack (SignOz, Elasticsearch, Kibana, Filebeat, Prometheus…)
├── environments/
│   ├── azure/prod/                 ← Azure UAE North Production (Terraform + ArgoCD apps)
│   ├── atrc/uaenorth/prod/         ← ATRC UAE North Production
│   ├── pp/auh1/prod/               ← PP Abu Dhabi Production (on-prem/gov cloud)
│   └── shared/                     ← Shared Terraform config
└── config.yml                      ← Version pin file (Helm chart versions)

Environment Matrix¶

Environment Code	Cloud/DC	Location	Branch	Cluster Context
`azure/prod` (`ai71-uaen-prod`)	Azure	UAE North	`main`	`ai71-uaen-prod-ask71-aks-01-admin`
`atrc/uaenorth/prod` (`ai71-atrc-uaen-prod-ask`)	Azure / ATRC	UAE North	`develop`	ATRC cluster
`pp/auh1/prod` (`ai71-pp-auh1-prod-ask`)	On-prem / gov.ae	Abu Dhabi (AUH1)	`feat/pp-deployment`	PP cluster
`modgpt/auh1/dev` (`ai71-modgpt-auh1-dev-ask`)	On-prem / RKE2	Abu Dhabi (AUH1)	TBD	core42 RKE2 cluster

PP Environment (Abu Dhabi — `pp/auh1/prod`)¶

Source branch: develop → feat/pp-deployment
Cluster domain: ask.pp.gov.ae
Registry: registry.pp.gov.ae (Harbor, air-gapped gov cloud)
Secondary registry: ai71uaenprodask71acr01.azurecr.io (ACR — source for image mirroring)

Infrastructure Stack¶

The PP environment runs on an on-premises / government cloud Kubernetes cluster (not AKS). All cloud-managed services (Azure PostgreSQL, Azure Redis, Azure Blob) are replaced with in-cluster Helm deployments.

Layer	Component	Source
GitOps	ArgoCD	Helm (ingress-nginx)
Secrets	HashiCorp Vault (in-cluster, `http://vault.hashicorp:8200`) + External Secrets Operator	Manual Helm deploy
Ingress	ingress-nginx	Helm
Database	PostgreSQL (Bitnami Helm)	`registry.pp.gov.ae/helm-charts` — `bitnami/postgresql:15.5.0`
Vector DB	Weaviate	`registry.pp.gov.ae/helm-charts` — `weaviate/weaviate:17.7.0`
Job Queue	Hatchet	`registry.pp.gov.ae/helm-charts` — `hatchet/hatchet-stack:0.10.0`
Container Registry	Harbor (`registry.pp.gov.ae`)	On-prem

PP Infrastructure Deployment Order (`01-infrastructure/`)¶

Step	Path	What it does
`00-base/01-hashicorp-vault-deploy`	Vault Helm deploy	Deploys HashiCorp Vault into cluster
`00-base/02-hashicorp-vault-configure`	Terraform (Vault provider)	Mounts KV v2, configures Kubernetes auth, creates `external-secrets-role`, creates `ClusterSecretStore`
`00-base/03-argocd`	`values-argocd.yaml`	ArgoCD Helm values — domain `ask.pp.gov.ae`, path `/argo-cd`, ingress-nginx
`00-base/04-external-secrets`	ESO Helm values	External Secrets Operator
`00-base/05-ingress-nginx`	ingress-nginx Helm	Ingress controller
`01-litellm`	`externalsecret.yaml`	LiteLLM secrets from Vault
`02-weaviate`	`externalsecret.yaml`	Weaviate secrets from Vault
`03-postgres`	`externalsecret.yaml`	PostgreSQL secrets from Vault
`04-hatchet`	`externalsecret.yaml`	Hatchet secrets + RabbitMQ + TLS cert
`05-azure-container-registry`	README (manual)	`imagePullSecret` for ACR (`ai71uaenprodask71acr01`)
`06-harbor-container-registry`	README (manual)	`imagePullSecret` for `registry.pp.gov.ae`
`07-knowledge-management-service`	`externalsecret.yaml`	KMS secrets + S3 CA bundle
`08-zitadel`	`externalsecret.yaml`	Zitadel IAM secrets
`09-core-service`	`externalsecret.yaml`	Core Service secrets
`10-assistant-service`	`externalsecret.yaml`	Assistant Service secrets

PP Secrets Backend — HashiCorp Vault¶

Unlike Azure environments (which use Azure Key Vault + ESO), the PP environment uses HashiCorp Vault as the secrets backend.

Parameter	Value
Vault address	`http://vault.hashicorp:8200`
Auth method	Kubernetes (`external-secrets-role`)
KV mount	`kv` (KV v2)
`ClusterSecretStore` name	`vault-backend`

Vault KV paths:

Service	Vault KV Path
Core Service	`kv/data/ask/prod/core-service/environment-secrets`
Assistant Service	`kv/data/ask/prod/assistant-service/environment-secrets`
Knowledge Management Service	`kv/data/ask/prod/knowledge-management-service/environment-secrets`
PostgreSQL	`kv/data/ask/prod/postgres/environment-secrets`
Weaviate	`kv/data/ask/prod/weaviate/environment-secrets`
Hatchet	`kv/data/ask/prod/hatchet/environment-secrets`

All ExternalSecret resources sync into Kubernetes Secret objects with a 1-minute refresh interval.

PP ArgoCD Application Hierarchy (App of Apps)¶

ai71-pp-auh1-prod-ask            ← Root App (points to _root/ directory)
    │  repoURL: ask-devops  feat/pp-deployment
    │  path: environments/pp/auh1/prod/02-argocd-application/_root
    │
    └─► ai71-pp-auh1-prod-ask.ask   ← App of Apps (points to ask/ directory)
            │  path: environments/pp/auh1/prod/02-argocd-application/ask
            │
            ├─► ai71-pp-auh1-prod-ask.ask-core-service
            ├─► ai71-pp-auh1-prod-ask.ask-assistant-service
            ├─► ai71-pp-auh1-prod-ask.ask-frontend
            ├─► ai71-pp-auh1-prod-ask.ask-km-service
            ├─► ai71-pp-auh1-prod-ask.ask-voice-service
            ├─► ai71-pp-auh1-prod-ask.ask-hatchet
            ├─► ai71-pp-auh1-prod-ask.weaviate
            └─► ai71-pp-auh1-prod-ask.postgresql

All applications use syncPolicy.automated with prune: true and selfHeal: true.

PP ArgoCD Applications Detail¶

ArgoCD App Name	Repo	Branch	Helm Values File	Namespace	Image Tag (pinned)
`ask-core-service`	`ask-core-service.git`	`staging`	`values-ai71-pp-auh1-prod-ask.yaml`	`ask`	`b4895f50`
`ask-assistant-service`	`ask-assistant-service.git`	`main`	`values-ai71-pp-auh1-prod-ask.yaml`	`ask`	`d1fec387`
`ask-frontend`	`ask-frontend.git`	`feature/gitlab_manual_build`	`values-ai71-pp-auh1-prod-ask.yaml`	`ask`	`staging-139ce0f1-pp-prod`
`ask-km-service`	`ask-knowledge-management-service.git`	`main`	`values-ai71-pp-auh1-prod-ask.yaml` + `ai71-pp-auh1-prod-ask.toml`	`ask`	`88ca403f`
`ask-voice-service`	`ask-voice-service.git`	`main`	`values-ai71-pp-auh1-prod-ask.yaml`	`ask`	`553d1ce0`
`ask-hatchet`	`registry.pp.gov.ae/helm-charts`	chart `0.10.0`	`values-ai71-pp-auh1-prod.yaml`	`hatchet`	N/A
`weaviate`	`registry.pp.gov.ae/helm-charts`	chart `17.7.0`	`values-ai71-pp-auh1-prod.yaml`	`weaviate`	N/A
`postgresql`	`registry.pp.gov.ae/helm-charts`	chart `15.5.0`	`values-ai71-pp-auh1-prod.yaml`	`ask`	N/A

Note: Hatchet, Weaviate, and PostgreSQL use a multi-source ArgoCD Application — the Helm chart is pulled from registry.pp.gov.ae/helm-charts (Harbor OCI mirror) and values files are sourced from the ask-devops Git repo via ref: values.

PP Container Registries¶

Registry	Purpose	How images arrive
`registry.pp.gov.ae/ask`	Application images (air-gapped)	Mirrored from ACR using `03-scripts/mirror-image.sh`
`registry.pp.gov.ae/helm-charts`	Helm chart OCI mirror	Pre-mirrored charts (bitnami, weaviate, hatchet)
`ai71uaenprodask71acr01.azurecr.io`	Source registry	CI pushes here; `imagePullSecret` created manually

Image mirror script (environments/pp/auh1/prod/03-scripts/mirror-image.sh):
Mirrors images from ACR to Harbor with retry logic (3 retries), parallel support, and structured logging.

Service mappings:

assistant-service          → registry.pp.gov.ae/ask/ask-assistant-service
core-service               → registry.pp.gov.ae/ask/ask-core-service
core-service-bulk-ops-worker → registry.pp.gov.ae/ask/ask-core-service-bulk-operations-worker

PP ArgoCD Configuration¶

Setting	Value
Domain	`ask.pp.gov.ae`
Path prefix	`/argo-cd`
Ingress class	`nginx`
Admin enabled	`true`
Exec enabled	`true` (terminal in UI)
Reconciliation timeout	`30s`
Repo credentials	Stored as Kubernetes Secrets in `argocd` namespace

ArgoCD repo secrets configured for: - https://gitlab.com/ai71.ai (broad access) - https://gitlab.com/ai71.ai/devops/ask-devops.git - https://gitlab.com/ai71.ai/ask/ask-knowledge-management-service.git - https://gitlab.com/ai71.ai/ask/ask-assistant-service.git - https://gitlab.com/ai71.ai/ask/ask-frontend.git

Azure Production Environment (`azure/prod`)¶

Branch: main
Cluster: AKS ai71-uaen-prod-ask71-aks-01-admin
Registry: ai71uaenprodask71acr01.azurecr.io
Secrets backend: Azure Key Vault + External Secrets Operator

Azure ArgoCD Applications¶

ArgoCD Applications are provisioned via Terraform (kubernetes_manifest resource). Each service has a 02-argocd-app/main.tf.

ArgoCD App	Repo URL	Branch	Helm Path	Namespace
`core-service`	`dev.azure.com/VentureOne/ASK71/_git/core_service`	`main`	`./helm`	`ask71`
`assistant-service`	`dev.azure.com/VentureOne/ASK71/_git/assistant_service`	`main`	`./helm`	`ask71`
`knowledge-management-service`	`dev.azure.com/VentureOne/ASK71/_git/knowledge_management_service`	`main`	`./helm`	`ask71`
`intelligence-service`	`dev.azure.com/VentureOne/ASK71/_git/intelligence_service`	`main`	`./helm`	`ask71`
`frontend`	`dev.azure.com/VentureOne/ASK71/_git/ask71-v2-frontend`	`main`	`./helm`	`ask71`
`iam-service`	`github.com/LocAI1/iam-service`	`helm-chart`	`./helm/v2`	`iam-service`
`weaviate`	Helm chart direct	`17.4.5`	N/A	(platform ns)
`hatchet`	Helm chart direct	`0.9.2`	N/A	`hatchet-stack`
`filebeat`	Helm chart direct	—	N/A	`observability`
`reloader`	Helm chart direct	`1.2.1`	N/A	`reloader`

Image tag is set via ArgoCD parameter image.tag: $ARGOCD_APP_REVISION — it tracks the Git commit SHA automatically.

Azure Infrastructure Components (Terraform)¶

Module	Path	What it provisions
Base resource group	`00-base/00-base`	Azure Resource Group + Terraform state storage account
Log Analytics	`00-base/01-log-analytics`	Azure Log Analytics workspace
Virtual Network	`00-base/02-virtual-network`	VNet + subnets (aks, pe, mgmt, shared, agw, db)
ACR	`00-base/03-acr`	Azure Container Registry
AKS	`00-base/04-aks`	AKS cluster (Kubernetes 1.31)
ArgoCD	`00-base/05-argocd`	ArgoCD Helm release + Workload Identity + Federated credentials
Cloudflare DNS	`00-base/06-cloudflare-dns`	DNS records in `ai71services.ai` zone
Reloader	`00-base/07-reloader`	Stakater Reloader Helm release
Azure Database	`00-base/08-azure-database`	Azure PostgreSQL Flexible Server instances
Storage Account	`00-base/09-storage-account`	Azure Blob Storage
Application Gateway	`00-base/10-application-gateway`	Azure App Gateway (WAF)
Teleport	`00-base/11-teleport-kube-agent`	Teleport Kubernetes agent

Infrastructure Version Pins (`config.yml`)¶

Component	Version	Type
ArgoCD	8.1.2	Helm chart
Stakater Reloader	1.2.1	Helm chart
Weaviate	17.4.5	Helm chart (Azure) / 17.7.0 (PP)
Hatchet	0.9.2	Helm chart (Azure) / 0.10.0 (PP)
PostgreSQL (PP)	15.5.0	Bitnami Helm chart

Observability Stack (develop branch)¶

Deployed in the ai71-uaen-stg-ask71 cluster under namespace observability:

Component	Purpose
SignOz	All-in-one observability (traces, metrics, logs)
Elasticsearch	Log storage
Kibana	Log visualisation
APM Server	Application performance monitoring
Metricbeat	Metrics collection
Prometheus	Metrics scraping
ClickHouse	High-performance analytics DB (for SignOz)
Superset	Data exploration / dashboards
Filebeat	Log shipping

Platform Add-ons (develop branch)¶

External Secrets ClusterGenerators¶

Pre-configured password generators available cluster-wide:

Generator	Length	Complexity
`alphanumeric-16/24/32`	16 / 24 / 32 chars	Letters + numbers
`complex-16/24/32`	16 / 24 / 32 chars	Letters + numbers + symbols
`complex-safe-16`	16 chars	Symbols safe for env vars
`fernet-key`	—	Fernet encryption key (Airflow / Superset)

Kyverno Policies¶

Policy	Effect
`add-opentelemetry-instrumentation`	Auto-inject OTel instrumentation env vars into pods

Oracle DB (dev environment)¶

A StatefulSet-based Oracle Database is deployed in the ask namespace (development only) for connector testing.

Daytona¶

Daytona sandbox environment values and manifests are managed in applications/ask/daytona/. The PP environment has separate manifests under applications/ask/daytona/manifests-pp/ with S3-compatible storage overrides.