ASK Core Service¶

Repo: ask-core-service-development
Language: Python ≥ 3.12
Framework: FastAPI + Uvicorn
Helm Chart: core-service v1.0.0
Port: 8000

Architecture¶

The Core Service is the central API gateway for the ASK platform. It handles user authentication (via Zitadel IAM), subscription/billing, API key management, notifications, and exposes the main REST API consumed by the Frontend and Assistant Service.

┌─────────────────────────────────────────────────────┐
│                  core-service (port 8000)            │
│                   uvicorn workers                    │
├──────────┬──────────────┬──────────────┬────────────┤
│ bulk-ops │  billing-    │    cron-     │  job-run   │
│ worker   │  metering-   │  scheduler-  │  worker    │
│          │  worker      │  worker      │            │
└──────────┴──────────────┴──────────────┴────────────┘
     │              │
     ▼              ▼
  PostgreSQL      Redis
  (external)    (external)

Container Images¶

Multi-Stage Build Pattern¶

All images use a two-stage build:

Stage	Base Image	Purpose
`builder`	`python:3.12`	Install dependencies via `uv sync`, build venv
`runtime`	`python:3.12-slim`	Minimal runtime, copy venv from builder

The PROXY_PREFIX ARG allows routing base image pulls through a registry mirror.

Images Produced¶

Image Name	Dockerfile	Port	Purpose
`core-service`	`Dockerfile`	8000	Main FastAPI application
`core-service-bulk-operations-worker`	`Dockerfile.bulk_operations_worker`	—	Bulk data operation jobs
`core-service-billing-metering-worker`	`Dockerfile.billing_metering_worker`	—	Billing/metering async jobs
`core-service-cron-scheduler-worker`	`Dockerfile.cron_scheduler_worker`	—	Scheduled/cron job runner
`core-service-job-run-worker`	`Dockerfile.job_run_worker`	—	On-demand job execution

All images push to:

ai71uaenprodask71acr01.azurecr.io/core-service[:<suffix>]:<tag>

Build Arguments¶

ARG	Default	Description
`PROXY_PREFIX`	`""`	Prefix prepended to base image names (e.g. `registry-mirror.internal/`)
`GITLAB_USERNAME`	—	Username for private PyPI index authentication

Build secret required: gitlab_password (mounted via --secret id=gitlab_password).

Helm Chart¶

Chart: helm/Chart.yaml

name: core-service
version: "1.0.0"
appVersion: "0.0.1"

Chart Dependencies (Subcharts)¶

Subchart	Version	Source	Condition
`zitadel`	1.0.1 (wraps upstream Zitadel v9.17.1)	`oci://registry.gitlab.com/ai71.ai/devops/devops-helm-charts`	`zitadel.enabled`

Zitadel provides the in-cluster IAM/OIDC service that core-service uses for machine-to-machine auth.

Default Resource Limits¶

resources:
  limits:
    memory: 768Mi
  requests:
    cpu: 300m
    memory: 500Mi

Health Probes¶

Probe	Path	Initial Delay	Period
Liveness	`/api/v1/health/live`	15s	15s
Readiness	`/api/v1/health/ready`	20s	10s

Autoscaling¶

Setting	Value
Min replicas	1
Max replicas	4
CPU target	80%

Environments¶

Environment File	Environment	Cloud/DC	Notes
`values-ai71-uaen-dev-ask71.yaml`	Development	Azure UAE North	Primary dev cluster
`values-ai71-uaen-stg-ask71.yaml`	Staging	Azure UAE North	Pre-prod validation
`values-ai71-mec1-stg-ask71.yaml`	MEC1 Staging	Azure MEC1	Secondary staging
`values-ai71-uaen-prod-mufti.yaml`	Production (Mufti)	Azure UAE North	Production tenant A
`values-ai71-mec1-prod-mufti.yaml`	Production (Mufti MEC1)	Azure MEC1	Production tenant A, MEC1
`values-ai71-pp-auh1-prod-ask.yaml`	PP Production	On-prem Abu Dhabi	Gov cloud, air-gapped, see below

PP Environment — Core Service¶

Full PP environment details: PP Environment page

PP Image¶

registry.pp.gov.ae/ask/ask-core-service:<tag>

imagePullSecrets: harbor-registry-secret

PP Workers Enabled/Disabled¶

Worker	PP Status	Replicas
`bulkOperationsWorker`	✅ Enabled	3
`billingMeteringWorker`	❌ Disabled	—
`cronSchedulerWorker`	❌ Disabled	—
`cronRunWorker`	❌ Disabled	—

Billing and cron workers are disabled because the PP environment does not use Stripe or metered billing.

PP Non-Secret Configuration (TOML)¶

Config Key	PP Value	Notes
`app.env`	`ai71-pp-auh1-prod-ask`	Dynaconf env selector
`database.host`	`postgresql.ask.svc.cluster.local`	In-cluster Bitnami PostgreSQL
`database.port`	`5432`
`database.name`	`core-service`
`database.user`	`core-service`
`iam_service.base_url`	`http://core-service-zitadel.ask.svc.cluster.local:8080`	In-cluster Zitadel gRPC
`iam_service.public_base_url`	`https://sso.ask.pp.gov.ae`	External SSO domain
`iam_service.default_org_id`	`357511184963905565`	PP tenant org
`iam_service.session_lifetime`	`2592000s` (30 days)
`foundry.base_url`	`http://litellm.foundry.svc.cluster.local:4000`	In-cluster LiteLLM
`foundry.default_tpm`	`5000000`
`foundry.default_rpm`	`500`
`storage.type`	`azure`	Azure Blob for CDN static assets only
`storage.azure.account_name`	`ai71uaensask71sa03`
`storage.cdn_base_url`	`https://cdn.ask.stg.ai71services.ai`
`redis.host`	`assistant-service-redis-master.ask.svc.cluster.local`	In-cluster Bitnami Redis
`redis.port`	`6379`
`redis.ssl`	`true`
`redis.cluster_mode`	`true`
`feature_flags.billing_enabled`	`false`	No billing in PP
`feature_flags.public_signup_enabled`	`false`	Invite-only
`public.core_services_base_url`	`https://api.ask.pp.gov.ae/ask71/v2/svc`
`knowledge_management_service.base_url`	`http://knowledge-management-service.ask.svc.cluster.local`

PP Secrets (from Vault `kv/data/ask/prod/core-service/environment-secrets`)¶

Secret Key	Description
`DATABASE__PASSWORD`	PostgreSQL password
`SECURITY__API_KEY_SALT`	API key salt
`SECURITY__MASTER_ENCRYPTION_KEY`	Master encryption key
`IAM_SERVICE__PAT`	Zitadel PAT
`NOTIFICATIONS__MAILCHIMP_API_KEY`	Mailchimp key
`STORAGE__ACCOUNT_KEY`	Azure Blob key (CDN only)
`STORAGE__SAS_TOKEN`	Azure Blob SAS token
`HATCHET__CLIENT_TOKEN`	Hatchet token
`REDIS__PASSWORD`	Redis password
`FOUNDRY__API_KEY`	LiteLLM API key
`APP__CORE_SERVICES_MACHINE_TOKEN`	Internal machine token

PP Ingress¶

Host	Path	Rewrite
`api.ask.pp.gov.ae`	`/ask71/v2/svc/(.*)`	`/$1`
`api.ask.pp.gov.ae`	`/(.*)` (catch-all)	`/$1`

TLS: ask-pp-gov-ae-cert (private CA).

PP Zitadel Subchart¶

Zitadel v4.5.0 is deployed as a subchart with the following PP overrides:

Setting	Value
`zitadel.ExternalDomain`	`sso.ask.pp.gov.ae`
`zitadel.ExternalSecure`	`true`
`zitadel.TLS.Enabled`	`false` (TLS handled by F5/ingress)
Database host	`postgresql.ask.svc.cluster.local:5432`
Database name	`zitadel`
Login UI image tag	`v4.5.0`
Login UI ingress path	`/ui/v2/login`
`masterkeySecretName`	`zitadel-secret`

core42 Environment — Core Service (`ai71-modgpt-auh1-dev-ask`)¶

core42 Images¶

Container	core42 Registry Path
Main API	`harbor.cl1.sq4.aegis.internal/ask/core-service:<git-sha>`
bulk-operations-worker	`harbor.cl1.sq4.aegis.internal/ask/core-service-bulk-operations-worker:<git-sha>`
billing-metering-worker	`harbor.cl1.sq4.aegis.internal/ask/core-service-billing-metering-worker:<git-sha>`
cron-scheduler-worker	`harbor.cl1.sq4.aegis.internal/ask/core-service-cron-scheduler-worker:<git-sha>`
job-run-worker	`harbor.cl1.sq4.aegis.internal/ask/core-service-job-run-worker:<git-sha>`

imagePullSecrets: harbor-pull-secret

core42 TOML Config Changes vs PP¶

Config Key	PP Value	core42 Value
`app.env`	`ai71-pp-auh1-prod-ask`	`ai71-modgpt-auh1-dev-ask`
`storage.type`	`azure`	`s3` (Ceph RGW)
`storage.azure.*`	Azure blob config	Remove — replaced by S3
`storage.s3.endpoint_url`	—	`https://s3.mod.auh1.dev.dir` (RGW VIP)
`storage.s3.bucket_name`	—	`core-service`
`storage.s3.region`	—	`us-east-1`
`iam_service.public_base_url`	`https://sso.ask.pp.gov.ae`	`https://sso.cl1.sq4.aegis.internal`
`iam_service.default_org_id`	`357511184963905565`	TBD — from Zitadel after setup
`redis.ssl`	`true`	`false` (in-cluster, no TLS)
`public.core_services_base_url`	`https://api.ask.pp.gov.ae/ask71/v2/svc`	`https://api.cl1.sq4.aegis.internal/ask71/v2/svc`
All other keys	—	✅ Same as PP

configTomlFileMountPath: /app/config/ai71-modgpt-auh1-dev-ask.toml

core42 Vault Secrets¶

Vault path: secret/cl1/core-service

Env Var	Vault Key	Description
`STORAGE__S3__ACCESS_KEY_ID`	`storage_s3_access_key_id`	Ceph RGW S3 access key
`STORAGE__S3__SECRET_ACCESS_KEY`	`storage_s3_secret_access_key`	Ceph RGW S3 secret key
`IAM_SERVICE__PAT`	`iam_service_pat`	Zitadel PAT
`NOTIFICATIONS__MAILCHIMP_API_KEY`	`notifications_mailchimp_api_key`	Mailchimp key
`FOUNDRY__API_KEY`	`foundry_api_key`	LiteLLM API key
`HATCHET__CLIENT_TOKEN`	`hatchet_client_token`	Hatchet token
`REDIS__PASSWORD`	`redis_password`	Redis password
`DB_PASSWORD`	`db_password`	PostgreSQL password

core42 Ingress¶

Host	Path
`api.cl1.sq4.aegis.internal`	`/ask71/v2/svc/(.*)`
`api.cl1.sq4.aegis.internal`	`/(.*)` (catch-all)

core42 Zitadel Subchart¶

Setting	core42 Value
`zitadel.ExternalDomain`	`sso.cl1.sq4.aegis.internal`
`zitadel.ExternalSecure`	`true`
`zitadel.TLS.Enabled`	`false`
Database host	`postgresql.ask.svc.cluster.local:5432`
Login UI ingress host	`sso.cl1.sq4.aegis.internal`

Configuration Variables¶

Configuration uses Dynaconf with config/default.toml as the base. Environment-specific overrides are in config/override.toml (injected by Helm as a ConfigMap mount).

Non-Secret Config (representative, from dev values)¶

Variable	Example (dev)	Description
`DATABASE__HOST`	`ai71-uaen-dev-ask71-psql-01.postgres.database.azure.com`	PostgreSQL hostname
`DATABASE__PORT`	`5432`	PostgreSQL port
`DATABASE__NAME`	`core_service`	Database name
`REDIS__HOST`	`ai71-uaen-dev-ask71-amr-01.uaenorth.redis.azure.net`	Redis hostname
`REDIS__PORT`	`6380`	Redis port (TLS)
`IAM_SERVICE__HOST`	`zitadel.iam-service.svc.cluster.local`	Zitadel in-cluster host
`IAM_SERVICE__PORT`	`8080`	Zitadel gRPC port
`FOUNDRY__BASE_URL`	`https://foundry.core.uaen.ai71services.ai`	AI71 Foundry LLM proxy
`STORAGE__ACCOUNT_NAME`	`ai71uaendask71sa03`	Azure storage account
`STORAGE__CONTAINER_NAME`	`$web`	Azure blob container
`APP__ENV`	`dev`	Dynaconf environment name

Secrets¶

Secrets are injected via Azure Key Vault + External Secrets Operator (ExternalSecret CRD). Key Vault secret names follow the pattern: {release-name}--{SECTION__KEY}.

Environment Variable	Key Vault Key Suffix	Description
`DATABASE__PASSWORD`	`DATABASE__PASSWORD`	PostgreSQL password
`SECURITY__API_KEY_SALT`	`SECURITY__API_KEY_SALT`	API key hashing salt
`SECURITY__MASTER_ENCRYPTION_KEY`	`SECURITY__MASTER_ENCRYPTION_KEY`	Master key for connector token encryption
`IAM_SERVICE__PAT`	`IAM_SERVICE__PAT`	Zitadel Personal Access Token (machine account)
`APP__CORE_SERVICES_MACHINE_TOKEN`	`APP__CORE_SERVICES_MACHINE_TOKEN`	Internal machine-to-machine auth token
`NOTIFICATIONS__MAILCHIMP_API_KEY`	`NOTIFICATIONS__MAILCHIMP_API_KEY`	Mailchimp transactional email API key
`NOTIFICATIONS__SENDGRID_API_KEY`	`NOTIFICATIONS__SENDGRID_API_KEY`	SendGrid API key
`NOTIFICATIONS__SMTP_PASSWORD`	`NOTIFICATIONS__SMTP_PASSWORD`	SMTP relay password
`STORAGE__ACCOUNT_KEY`	`STORAGE__ACCOUNT_KEY`	Azure Storage account key
`STORAGE__SAS_TOKEN`	`STORAGE__SAS_TOKEN`	Azure Blob SAS token
`HATCHET__CLIENT_TOKEN`	`HATCHET__CLIENT_TOKEN`	Hatchet job queue client token
`BILLING_METERING__AUTUMN_API_KEY`	`BILLING_METERING__AUTUMN_API_KEY`	Autumn billing API key
`BILLING_METERING__BILLING_HATCHET_CLIENT_TOKEN`	`BILLING_METERING__BILLING_HATCHET_CLIENT_TOKEN`	Hatchet token for billing worker
`BILLING_METERING__SERVICE_API_KEY_SALT`	`BILLING_METERING__SERVICE_API_KEY_SALT`	Billing service API key salt
`BILLING_METERING__STRIPE_WEBHOOK_SECRET`	`BILLING_METERING__STRIPE_WEBHOOK_SECRET`	Stripe webhook signing secret
`REDIS__PASSWORD`	`REDIS__PASSWORD`	Redis AUTH password
`SENTRY__DSN`	`SENTRY__DSN`	Sentry error tracking DSN
`FOUNDRY__API_KEY`	`FOUNDRY__API_KEY`	AI71 Foundry API key

ExternalSecret refresh interval: 1 minute. The secretAutoReloadOnUpdate: true Helm value triggers Stakater Reloader to roll pods when the secret changes.

Key Dependencies¶

Python Packages¶

Package	Version	Purpose
`fastapi`	0.135.2	HTTP framework
`uvicorn`	0.34	ASGI server
`sqlalchemy`	2.0.38	ORM (async)
`asyncpg`	0.31	Async PostgreSQL driver
`psycopg2`	2.9.10	Sync PostgreSQL driver
`alembic`	1.18.4	Database migrations
`dynaconf`	3.2.13	Configuration management
`pydantic`	v2	Data validation
`hatchet-sdk`	1.32.0	Async job queue
`iam-client`	2.1.0	Zitadel IAM SDK (private)
`foundry-client`	0.1.2	AI71 Foundry SDK (private)
`storage-lib`	0.2.0	Azure/MinIO storage SDK (private)
`stripe`	8.0.0+	Stripe billing API
`redis`	7.4	Redis client
`sentry-sdk`	2.57	Error tracking
`mailchimp-transactional`	1.4.1	Email notifications
`jinja2`	3.1.6	Email template rendering
`email-validator`	2.0.0	Email address validation
`phonenumbers`	9.0.27	Phone number validation

Private GitLab PyPI Indexes¶

Index Name	Package Provided
`connectors-lib`	External connector integrations
`iam-client`	Zitadel IAM SDK
`logger-sdk`	Structured logging SDK
`foundry-client`	AI71 Foundry LLM proxy SDK
`storage-lib`	Azure Blob / MinIO abstraction

CI/CD Pipeline¶

File: .gitlab-ci.yml
Tool: GitLab CI

Stage	Key Jobs	Description
`validate`	pre-commit, pytest, sonarcloud, keyvault-validation	Code quality and tests
`build`	`docker-acr-build`	Build all 5 Docker images and push to ACR
`helm-release`	`helm-release`, `helm-release-oci-with-acr`	Publish chart to GitLab OCI and ACR

Key CI variables:

Variable	Value
`UV_VERSION`	`0.9.21`
`PYTHON_VERSION`	`3.12`
`CI base image`	`ghcr.io/astral-sh/uv`