Core Service — Config Updates

Side-by-side comparison of every configuration value changed from the upstream values.yaml defaults to the staging deployment (values-ai71-uaen-prod-modgpt.yaml). Grouped by concern so it is clear what changed, what it was before, and why.

---

Images

Key	Original (upstream)	Staging (values-ai71-uaen-prod-modgpt.yaml)	Reason
image.repository	ai71uaenprodask71acr01.azurecr.io/core-service	harbor.cl1.sq4.aegis.internal/ask/core-service	ACR unreachable; air-gap requires internal registry
image.tag	latest	<pinned-tag>	Never use :latest in GitOps — pin to a specific version
image.pullPolicy	IfNotPresent	IfNotPresent	No change
imagePullSecrets	[]	[{name: harbor-pull-secret}]	Private Harbor registry requires pull secret
bulkOperationsWorker.image.repository	…azurecr.io/core-service-bulk-operations-worker	(worker disabled in v1)	Worker off in v1
billingMeteringWorker.image.repository	…azurecr.io/core-service-billing-metering-worker	(worker disabled in v1)	Worker off in v1
cronSchedulerWorker.image.repository	…azurecr.io/core-service-cron-scheduler-worker	(worker disabled in v1)	Worker off in v1
cronRunWorker.image.repository	…azurecr.io/core-service-job-run-worker	(worker disabled in v1)	Worker off in v1

---

Networking

Service

# BEFORE (values.yaml)
service:
  type: LoadBalancer
  port: 80

# AFTER (values-ai71-uaen-prod-modgpt.yaml)
service:
  type: ClusterIP
  port: 80

Why: No cloud load balancer exists in this environment. Traffic enters via the nginx Ingress instead.

---

Ingress

# BEFORE (values.yaml)
ingress:
  enabled: false
  className: ""
  annotations: {}
  hosts:
    - host: chart-example.local
      paths: [{ path: /, pathType: ImplementationSpecific }]
  tls: []

# AFTER (values-ai71-uaen-prod-modgpt.yaml)
ingress:
  enabled: true
  className: rke2-nginx
  annotations:
    cert-manager.io/cluster-issuer: internal-ca-issuer
  hosts:
    - host: core-service.ask.mod.auh1.dev.dir
      paths: [{ path: /, pathType: Prefix }]
  tls:
    - secretName: core-service-tls
      hosts: [core-service.ask.mod.auh1.dev.dir]

Why: Ingress must be enabled for the service to be reachable. Azure's ingress class replaced with RKE2's nginx; cert-manager issues TLS from the internal CA.

---

Node scheduling

# BEFORE (values.yaml)
nodeSelector:
  node.kubernetes.io/role: ask71

tolerations:
  - key: node.kubernetes.io/role
    operator: Equal
    value: ask71
    effect: NoSchedule

# AFTER (values-ai71-uaen-prod-modgpt.yaml)
nodeSelector: {}
tolerations: []

Why: node.kubernetes.io/role: ask71 does not exist on the nodes. Without clearing this, the API pod stays Pending forever.

---

Workers

# BEFORE (values.yaml)
bulkOperationsWorker:
  enabled: true
  replicaCount: 3
  # resources: 4Gi memory per pod, HPA max 20

billingMeteringWorker:
  enabled: true

cronSchedulerWorker:
  enabled: true

cronRunWorker:
  enabled: true

# AFTER (values-ai71-uaen-prod-modgpt.yaml)
bulkOperationsWorker:  { enabled: false }
billingMeteringWorker: { enabled: false }
cronSchedulerWorker:   { enabled: false }
cronRunWorker:         { enabled: false }

Why (v1 scope): Workers require their own images (not yet mirrored) and their resource footprint (3×4Gi for bulk ops alone) needs sizing against available node RAM. Enabled in Phase 11 once images are available and resources are sized.

---

Postgres init

# BEFORE (values.yaml)
postgresInit:
  enabled: false
  connection:
    host: ""
    port: "5432"
    adminUsername: "postgres"
  db:
    name: "core_service"
    user: "core_service"
  argocd:
    enabled: false
  job:
    image: "postgres:15-alpine"

# AFTER (values-ai71-uaen-prod-modgpt.yaml)
postgresInit:
  enabled: true
  namespace: ask
  connection:
    host: ask-pg-rw.ask.svc.cluster.local    # CNPG primary service
    port: "5432"
    adminUsername: postgres
  db:
    name: core-service
    user: core-service
  secrets:
    adminPasswordSecretName: zitadel-psql-secret   # CNPG superuser pw (VSO-rendered)
    adminPasswordKey: db-admin-password
    userPasswordSecretName: core-service-secret
    userPasswordKey: DATABASE__PASSWORD
  job:
    image: harbor.cl1.sq4.aegis.internal/dockerhub/postgres
    tag: "16-alpine"                               # upgraded from 15 to match zitadelPostgresInit
    pgSslMode: disable                             # CNPG internal — SSL not required
  argocd:
    enabled: true
    connectionConfigMapAnnotations:
      argocd.argoproj.io/hook: PreSync
      argocd.argoproj.io/sync-wave: "-2"
      argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
    jobAnnotations:
      argocd.argoproj.io/hook: PreSync
      argocd.argoproj.io/sync-wave: "-1"
      argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded

Why: No Azure Postgres in this environment. CNPG provides ask-pg-rw.ask.svc.cluster.local. ArgoCD hooks wire the init Job into the PreSync phase so the DB exists before workloads start.

---

Storage

# BEFORE (default.toml — baked into image)
[default.storage]
type = "azure"
cdn_base_url = "@none"

# AFTER (values-ai71-uaen-prod-modgpt.yaml — overrides configToml at /app/config/override.toml)
envConf:
  configToml: |
    [prod.storage]
    type = "stub"

Why (v1): No Azure Blob available. Storage is stubbed so the app starts. Phase 11 replaces this with Ceph RGW S3.

---

Zitadel (IAM/SSO)

# BEFORE (values.yaml)
zitadel:
  enabled: false
  zitadel:
    fullnameOverride: ask-core-service-zitadel
    zitadel:
      masterkeySecretName: zitadel-secret
    login:
      fullnameOverride: ask-core-service-zitadel-login
  zitadelPostgresInit:
    enabled: false

# AFTER (values-ai71-uaen-prod-modgpt.yaml)
zitadel:
  enabled: true
  zitadel:
    replicaCount: 1
    tolerations: []
    nodeSelector: {}
    service: { type: ClusterIP, port: 8080 }
    ingress:
      enabled: true
      className: rke2-nginx
      annotations:
        cert-manager.io/cluster-issuer: internal-ca-issuer
        nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
      hosts:
        - host: sso.ask.mod.auh1.dev.dir
          paths: [{ path: /, pathType: Prefix }]
      tls:
        - secretName: zitadel-tls
          hosts: [sso.ask.mod.auh1.dev.dir]
    zitadel:
      masterkeySecretName: zitadel-secret-kv
      configmapConfig:
        ExternalSecure: true
        ExternalDomain: sso.ask.mod.auh1.dev.dir
        TLS: { Enabled: false }
        Database:
          Postgres:
            Host: ask-pg-rw.ask.svc.cluster.local
            Port: 5432
            Database: zitadel
            User:  { Username: zitadel,  SSL: { Mode: disable } }
            Admin: { Username: postgres, SSL: { Mode: disable } }
    env:
      - name: ZITADEL_DATABASE_POSTGRES_USER_PASSWORD
        valueFrom: { secretKeyRef: { name: zitadel-secret-kv, key: DATABASE__PASSWORD } }
      - name: ZITADEL_DATABASE_POSTGRES_ADMIN_PASSWORD
        valueFrom: { secretKeyRef: { name: zitadel-psql-secret, key: db-admin-password } }
      - name: ZITADEL_FIRSTINSTANCE_ORG_HUMAN_PASSWORDCHANGEREQUIRED
        value: "false"
      - name: ZITADEL_FIRSTINSTANCE_ORG_HUMAN_PASSWORD
        valueFrom: { secretKeyRef: { name: zitadel-secret-kv, key: ADMIN_PASSWORD } }
    login:
      enabled: true
      replicaCount: 1
      ingress:
        enabled: true
        className: rke2-nginx
        hosts:
          - host: sso.ask.mod.auh1.dev.dir
            paths: [{ path: /ui/v2/login, pathType: Prefix }]
  zitadelPostgresInit:
    enabled: true
    connection:
      host: ask-pg-rw.ask.svc.cluster.local
      port: "5432"
      adminUsername: postgres
    secrets:
      adminPasswordSecretName: zitadel-psql-secret
      adminPasswordKey: db-admin-password
      userPasswordSecretName: zitadel-secret-kv
      userPasswordKey: DATABASE__PASSWORD
    job:
      image: harbor.cl1.sq4.aegis.internal/dockerhub/postgres
      tag: "16-alpine"
      pgSslMode: disable
    argocd:
      enabled: true
      connectionConfigMapAnnotations:
        argocd.argoproj.io/hook: PreSync
        argocd.argoproj.io/sync-wave: "-2"
        argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
      jobAnnotations:
        argocd.argoproj.io/hook: PreSync
        argocd.argoproj.io/sync-wave: "-1"
        argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded

Key changes: - enabled: false → true (required for auth) - Azure Postgres host → CNPG ask-pg-rw.ask.svc.cluster.local - SSL.Mode: prefer → disable (CNPG internal cluster) - ExternalDomain → sso.ask.mod.auh1.dev.dir - zitadelPostgresInit.enabled: false → true - Azure ingress class → rke2-nginx - Added cert-manager TLS annotation - tolerations cleared (no node.kubernetes.io/role: ask in this environment)

---

Chart.yaml — dependency repository

# BEFORE (upstream — placeholder)
dependencies:
  - name: zitadel
    version: "1.0.1"
    repository: "oci://your-internal-registry.example.com/devops-helm-charts"
    condition: zitadel.enabled

# AFTER (internal OCI registry)
dependencies:
  - name: zitadel
    version: "1.0.1"
    repository: "oci://harbor.cl1.sq4.aegis.internal/modgpt/charts"
    condition: zitadel.enabled

Why: Placeholder URL used for air-gap documentation purposes only — must be replaced with the actual internal registry before helm dependency build can vendor the chart.

---

Migrations — new key (did not exist upstream)

# BEFORE — key did not exist in values.yaml
# (no migration automation in upstream chart)

# AFTER — added to values.yaml (default off)
migrations:
  enabled: false
  argocd:
    enabled: true
    jobAnnotations:
      argocd.argoproj.io/hook: PreSync
      argocd.argoproj.io/sync-wave: "0"
      argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded
  job:
    backoffLimit: 2
    ttlSecondsAfterFinished: 300
    resources:
      requests: { cpu: 100m, memory: 256Mi }
      limits:   { cpu: 500m, memory: 512Mi }

# In values-ai71-uaen-prod-modgpt.yaml — enabled
migrations:
  enabled: true

Why this was added — investigation of upstream migration handling

Before adding this config, we investigated every place migrations could already be triggered in the upstream codebase:

1. Dockerfile — does not run migrations

COPY alembic.ini ./
COPY alembic/ ./alembic/     # ← copied into image
CMD ["uvicorn", "core_service.app:app", "--host", "0.0.0.0", "--port", "8000"]

The CMD starts uvicorn only. There is no ENTRYPOINT wrapper script and no RUN alembic upgrade head step. The alembic files are in the image but nothing executes them at container start.

2. deployment.yaml — no initContainers or lifecycle hooks The Deployment template has no initContainers block and no postStart lifecycle hook. Confirmed by reading the full template — the spec goes directly from serviceAccountName to containers. There is no mechanism to run a command before the main container starts.

3. Helm chart templates — no migration Job or template

helm/templates/
  job-postgres-init.yaml      ← creates DB + role only (no alembic)
  deployment.yaml             ← no initContainers
  *.yaml                      ← no other Job or migration template

job-postgres-init.yaml runs SQL to create the database and application role — it does not run alembic. There was no migration Job template in the upstream chart at all.

4. NOTES.txt — no migration instructions The chart's post-install notes only print the service URL. No mention of running migrations as a post-install step.

5. scripts/apply_migrations.sh — exists but is manual/local only

uv run alembic upgrade head

This script exists for local developer use — it sources .env (a local dev file) and runs alembic via uv. It has no Kubernetes context, no Helm hook, and no mechanism to run inside the cluster.

6. CI/CD pipelines — not applicable to this deployment There is no .gitlab-ci.yml or azure-pipelines.yml at the repo root. The security/ directory contains Azure DevOps pipeline YAML for security scanning only — no deployment or migration pipelines. In the upstream cloud deployment, migrations were presumably run by a pipeline step outside of Helm — either an ADO pipeline calling scripts/apply_migrations.sh or a manual ops step. That pipeline does not exist in our GitOps-only deployment.

Conclusion

The upstream chart has no automated migration mechanism for a Kubernetes/GitOps deployment. Migrations in the cloud environment were run outside Helm — by an ADO pipeline or manually — which is incompatible with a GitOps-only air-gap deployment where there is no external CI/CD pipeline touching the cluster.

A PreSync Helm Job (helm/templates/job-migrations.yaml) was added with a toggle flag so that: - Migrations run automatically on every ArgoCD sync (before workloads start) - The mechanism is self-contained within the Helm chart — no external pipeline required - The toggle (migrations.enabled) allows disabling for emergency rollbacks without modifying the chart

---

Complete values-ai71-uaen-prod-modgpt.yaml (reference)

The full production overlay combining all of the above:

# helm/environments/values-ai71-uaen-prod-modgpt.yaml
# Production environment — air-gap/customer deployment
# NO SECRETS in this file — all from Vault via VSO

extraEnv:
  - name: APP__ENV
    value: "prod"

# Images
image:
  repository: harbor.cl1.sq4.aegis.internal/ask/core-service
  tag: "<pinned-tag>"
  pullPolicy: IfNotPresent

imagePullSecrets:
  - name: harbor-pull-secret

# Scheduling
nodeSelector: {}
tolerations: []

# Networking
service:
  type: ClusterIP
  port: 80

ingress:
  enabled: true
  className: rke2-nginx
  annotations:
    cert-manager.io/cluster-issuer: internal-ca-issuer
  hosts:
    - host: core-service.ask.mod.auh1.dev.dir
      paths: [{ path: /, pathType: Prefix }]
  tls:
    - secretName: core-service-tls
      hosts: [core-service.ask.mod.auh1.dev.dir]

# Workers — off for v1
bulkOperationsWorker:  { enabled: false }
billingMeteringWorker: { enabled: false }
cronSchedulerWorker:   { enabled: false }
cronRunWorker:         { enabled: false }

# Storage — stubbed for v1 (Ceph RGW S3 in Phase 11)
envConf:
  configToml: |
    [prod.storage]
    type = "stub"

# DB init
postgresInit:
  enabled: true
  namespace: ask
  connection:
    host: ask-pg-rw.ask.svc.cluster.local
    port: "5432"
    adminUsername: postgres
  db: { name: core-service, user: core-service }
  secrets:
    adminPasswordSecretName: zitadel-psql-secret
    adminPasswordKey: db-admin-password
    userPasswordSecretName: core-service-secret
    userPasswordKey: DATABASE__PASSWORD
  job:
    image: harbor.cl1.sq4.aegis.internal/dockerhub/postgres
    tag: "16-alpine"
    backoffLimit: 4
    ttlSecondsAfterFinished: 300
    pgSslMode: disable
  argocd:
    enabled: true
    connectionConfigMapAnnotations:
      argocd.argoproj.io/hook: PreSync
      argocd.argoproj.io/sync-wave: "-2"
      argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
    jobAnnotations:
      argocd.argoproj.io/hook: PreSync
      argocd.argoproj.io/sync-wave: "-1"
      argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded

# Migrations
migrations:
  enabled: true

# Zitadel (see full block in airgap-alignment.md)
zitadel:
  enabled: true
  # ... (full block in config above)

---

➡️ Related: Discovery · Sanitization · Air-Gap Alignment · Deployment Runbook