Core Service — Air-Gap Deployment Inventory¶

Every artifact required to deploy core-service + Zitadel into the air-gapped environment. Work through each section before touching the cluster. Nothing should be pulled from the internet at deploy time.

How to use this page

Each section has a status column. Work down the page in order — cluster prerequisites first, then mirror artifacts, then provision secrets. The deployment runbook (28-core-service-deploy.md) references back to this page for each phase.

1. Cluster Prerequisites

Services that must already be running before core-service is deployed. All of these are managed by the cluster-addons ApplicationSet.

Service	Namespace	How to verify	Status
ArgoCD	`argocd`	`kubectl get pods -n argocd`	✅ deployed
cert-manager + `internal-ca-issuer`	`cert-manager`	`kubectl get clusterissuer internal-ca-issuer`	✅ deployed
NGINX Ingress Controller	`kube-system`	`kubectl get ingressclass rke2-nginx`	✅ deployed
Vault Secrets Operator (VSO)	`vault-secrets-operator-system`	`kubectl get pods -n vault-secrets-operator-system`	✅ deployed
Stakater Reloader	`reloader`	`kubectl get pods -n reloader`	✅ deployed
Main Vault (unsealed)	`vault`	`vault kv list secret/ask`	✅ deployed
CloudNativePG operator	`cnpg-system`	`kubectl get deployment -n cnpg-system`	⬜ install first
GitLab (Helm + container registry)	`gitlab`	`helm repo list \\| grep gitlab`	✅ deployed

CNPG operator must be installed before core-service

The chart's postgresInit and zitadelPostgresInit PreSync Jobs create databases against the CNPG-managed ask-pg-rw service. If the CNPG operator and the ask-pg Cluster CR are not running, the PreSync Jobs will fail and the deployment will not proceed.

Add CNPG as a cluster-addon before deploying core-service:

# Mirror CNPG chart + images to GitLab first (see Section 3 + 4 below)
# then add the addon — see how-to/add-addon-to-cluster-addons.md

2. Helm Charts

All Helm charts must be pushed to the GitLab HTTP package registry before ArgoCD or helm install is run.

Chart	Source	Version	Destination	Notes
`core-service`	In-repo (`helm/`)	`1.0.0`	No registry needed — ArgoCD reads the Git repo directly (`localHelm`)	Zitadel subchart must be vendored first (see Section 2a)
`zitadel` (ai71 wrapper)	`oci://registry.gitlab.com/ai71.ai/devops/devops-helm-charts/zitadel`	`1.0.1`	`oci://harbor.cl1.sq4.aegis.internal/modgpt/charts/zitadel` → then `helm dependency build` in `helm/`	Wraps upstream Zitadel v9.17.1
`cloudnative-pg`	`https://cloudnative-pg.io/charts`	latest stable	GitLab HTTP: `https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/<id>/packages/helm/stable`	Cluster add-on — install once per cluster, not per app

2a — Vendor the Zitadel subchart¶

# 1. Pull from upstream (requires internet access — do this on a connected machine)
helm pull oci://registry.gitlab.com/ai71.ai/devops/devops-helm-charts/zitadel --version 1.0.1

# 2. Push to internal OCI registry
helm push zitadel-1.0.1.tgz oci://harbor.cl1.sq4.aegis.internal/modgpt/charts

# 3. Update helm/Chart.yaml (already done in air-gap repo sanitization)
#    repository: "oci://harbor.cl1.sq4.aegis.internal/modgpt/charts"

# 4. Vendor into the chart
cd helm && helm dependency build
# produces: helm/charts/zitadel-1.0.1.tgz + helm/Chart.lock

# 5. Commit the vendored chart
git add helm/charts/ helm/Chart.lock && git commit -m "vendor: zitadel 1.0.1 subchart"
git push

Vendored chart = no registry pull at deploy time

Once helm/charts/zitadel-1.0.1.tgz is committed, ArgoCD renders the chart entirely offline. The Chart.yaml repository URL is only needed during helm dependency build.

3. Container Images — Mirror As-Is

Images that exist upstream and need to be copied to harbor.cl1.sq4.aegis.internal/ask/ via skopeo.

3a — Infrastructure images (required for v1)¶

Image	Source	Destination	Used by	amd64?	Notes
`postgres:16-alpine`	`docker.io/library/postgres:16-alpine`	`.../ask/postgres:16-alpine`	`postgresInit` + `zitadelPostgresInit` Jobs	✅	Official postgres multi-arch image
`zitadel` server	see note ①	`.../ask/zitadel:<ver>`	Zitadel subchart	✅	Tag from vendored chart values
`zitadel-login` UI	see note ①	`.../ask/zitadel-login:<ver>`	Zitadel login subchart	✅	Tag from vendored chart values

① Zitadel image tags

Exact image tags are defined inside the vendored zitadel chart's values.yaml. Run helm show values oci://harbor.cl1.sq4.aegis.internal/modgpt/charts/zitadel --version 1.0.1 | grep -E "repository|tag" after pushing the chart to get the exact tags. The wrapper chart wraps Zitadel v9.17.1 — expect tags in the v9.17.x range.

3b — CNPG operator image¶

Image	Source	Destination	Notes
`cloudnative-pg`	`ghcr.io/cloudnative-pg/cloudnative-pg:<ver>`	`.../ask/cloudnative-pg:<ver>`	Operator controller image — get version from `helm show values cloudnative-pg/cloudnative-pg \\| grep tag`

3c — Docker base images (needed for image builds)¶

These are the base layers pulled during docker build. Must be available either locally or via a PROXY_PREFIX registry mirror. All Dockerfiles accept ARG PROXY_PREFIX="" — set this to your local mirror if needed.

Image	Source	Used as	Stage
`python:3.12`	`docker.io/library/python:3.12`	Builder layer	Stage 1 — all 5 Dockerfiles
`python:3.12-slim`	`docker.io/library/python:3.12-slim`	Runtime layer	Stage 2 — all 5 Dockerfiles

Pull and re-tag before build

docker pull python:3.12 && docker pull python:3.12-slim
# Mirror to GitLab if the build machine is air-gapped:
docker tag python:3.12 harbor.cl1.sq4.aegis.internal/dockerhub/python:3.12
docker push harbor.cl1.sq4.aegis.internal/dockerhub/python:3.12
# Then build with: --build-arg PROXY_PREFIX=harbor.cl1.sq4.aegis.internal/ask/

3d — Helm test image (skip in production)¶

Image	Used by	Notes
`busybox`	`helm test` pod only	Not deployed — only triggered by `helm test`. Skip unless you run `helm test` in CI.

4. Container Images — Build From Source

These images are pulled from their upstream registry for linux/amd64 and mirrored to Harbor via skopeo (matching the x86_64 production node architecture).

All 5 share the same Dockerfile pattern: python:3.12 builder → python:3.12-slim runtime.

Image name	Dockerfile	Target tag	Private indexes required
`core-service`	`Dockerfile`	`harbor.cl1.sq4.aegis.internal/ask/core-service:<ver>`	All 5 (see Section 5)
`core-service-bulk-operations-worker`	`Dockerfile.bulk_operations_worker`	`.../core-service-bulk-operations-worker:<ver>`	All 5
`core-service-billing-metering-worker`	`Dockerfile.billing_metering_worker`	`.../core-service-billing-metering-worker:<ver>`	All 5
`core-service-cron-scheduler-worker`	`Dockerfile.cron_scheduler_worker`	`.../core-service-cron-scheduler-worker:<ver>`	All 5
`core-service-job-run-worker`	`Dockerfile.job_run_worker`	`.../core-service-job-run-worker:<ver>`	All 5

Build command (run from repo root, on a machine with ai71 network access):

# Set your ai71 registry credentials
export REGISTRY_USERNAME=<your-ai71-username>
export REGISTRY_PASSWORD=<your-ai71-registry-password>

# Build + push API image (repeat for each worker Dockerfile)
docker buildx build \
  --platform linux/amd64 \
  --secret id=registry_password,env=REGISTRY_PASSWORD \
  --build-arg REGISTRY_USERNAME=${REGISTRY_USERNAME} \
  -f Dockerfile \
  -t harbor.cl1.sq4.aegis.internal/ask/core-service:<ver> \
  --push \
  .

v1 scope — API image only

For v1 only the core-service (API) image is required. Worker images (bulk-operations, billing-metering, cron-scheduler, job-run) are deferred to Phase 11 when workers are enabled. Build those 4 later.

5. Private Python Packages

These packages are pulled from ai71's internal package feed during docker build (Stage 1). They are baked into the image — once the image is pushed to GitLab, the cluster never calls the private index again.

Package	Env var (username)	Secret env var (password)	Min version
`storage-lib`	`UV_INDEX_STORAGE_LIB_USERNAME`	`UV_INDEX_STORAGE_LIB_PASSWORD`	`>=0.2.0`
`connectors-lib`	`UV_INDEX_CONNECTORS_LIB_USERNAME`	`UV_INDEX_CONNECTORS_LIB_PASSWORD`	`>=0.2.7`
`iam-client`	`UV_INDEX_IAM_CLIENT_USERNAME`	`UV_INDEX_IAM_CLIENT_PASSWORD`	(from lock)
`logger-sdk`	`UV_INDEX_LOGGER_SDK_USERNAME`	`UV_INDEX_LOGGER_SDK_PASSWORD`	(from lock)
`foundry-client`	`UV_INDEX_FOUNDRY_CLIENT_USERNAME`	`UV_INDEX_FOUNDRY_CLIENT_PASSWORD`	`>=0.1.2`

These are build-time only — the cluster does not need access to these indexes at runtime.

Public packages (PyPI) — already in the image

All other dependencies in pyproject.toml (fastapi, uvicorn, sqlalchemy, alembic, hatchet-sdk, redis, psycopg2, httpx, sentry-sdk, stripe, etc.) are pulled from public PyPI during the same build. They are also baked into the image. No PyPI mirror is needed at deploy time.

6. Vault Secrets to Provision

All secrets live under secret/ask/ in the Main Vault (KV v2). VSO VaultStaticSecret CRs render them into Kubernetes Secrets in the ask namespace automatically.

6a — `core-service-secret` (rendered by VSO)¶

Path: secret/ask/core-service

Key	Description	v1 required?
`DATABASE__PASSWORD`	Postgres `core-service` role password	✅
`SECURITY__API_KEY_SALT`	API key hashing salt	✅
`SECURITY__MASTER_ENCRYPTION_KEY`	Connector token encryption key	✅
`IAM_SERVICE__PAT`	Zitadel Personal Access Token (service-to-service)	✅ post-deploy step
`REDIS__PASSWORD`	Redis auth (empty string in this environment — Redis has no auth)	✅ (set `""`)
`APP__CORE_SERVICES_MACHINE_TOKEN`	Inter-service auth token	✅
`FOUNDRY__API_KEY`	Foundry proxy API key	✅
`HATCHET__CLIENT_TOKEN`	Hatchet workflow orchestrator token	✅
`SENTRY__DSN`	Sentry error tracking DSN	⬜ optional (stub with `""`)
`STORAGE__ACCOUNT_KEY`	Object storage key — stubbed in v1	⬜ stub (`""`)
`NOTIFICATIONS__MAILCHIMP_API_KEY`	Email via Mailchimp	⬜ stub (`""`)
`NOTIFICATIONS__SENDGRID_API_KEY`	SMTP fallback	⬜ stub (`""`)
`BILLING_METERING__AUTUMN_API_KEY`	Billing/Autumn API key	⬜ workers off in v1
`BILLING_METERING__BILLING_HATCHET_CLIENT_TOKEN`	Hatchet billing token	⬜ workers off in v1
`BILLING_METERING__SERVICE_API_KEY_SALT`	Billing salt	⬜ workers off in v1
`BILLING_METERING__STRIPE_WEBHOOK_SECRET`	Stripe webhook	⬜ workers off in v1

6b — `zitadel-secret-kv` (rendered by VSO)¶

Path: secret/ask/zitadel

Key	Description	Notes
`masterkey`	Zitadel master encryption key	Random 32+ char string — generate once, never change
`ADMIN_PASSWORD`	Zitadel initial superadmin password	Used by `seedData`; change after first login
`DATABASE__PASSWORD`	Zitadel postgres role password	Must match the password used in `zitadelPostgresInit`

6c — `zitadel-psql-secret` (rendered by VSO)¶

Path: secret/ask/zitadel-psql

Key	Description	Notes
`db-admin-password`	CNPG postgres superuser (`postgres` role) password	Used by BOTH `postgresInit` and `zitadelPostgresInit` admin connections

6d — Provision commands¶

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal

# core-service secrets (v1 minimum)
vault kv put secret/ask/core-service \
  DATABASE__PASSWORD="<generate>" \
  SECURITY__API_KEY_SALT="<generate>" \
  SECURITY__MASTER_ENCRYPTION_KEY="<generate>" \
  IAM_SERVICE__PAT="PLACEHOLDER_SET_AFTER_ZITADEL_BOOTS" \
  REDIS__PASSWORD="" \
  APP__CORE_SERVICES_MACHINE_TOKEN="<generate>" \
  FOUNDRY__API_KEY="<your-foundry-key>" \
  HATCHET__CLIENT_TOKEN="<your-hatchet-token>" \
  SENTRY__DSN="" \
  STORAGE__ACCOUNT_KEY=""

# zitadel secrets
vault kv put secret/ask/zitadel \
  masterkey="$(openssl rand -base64 32)" \
  ADMIN_PASSWORD="<initial-admin-password>" \
  DATABASE__PASSWORD="<generate>"

# CNPG superuser password (shared admin credential for DB init jobs)
vault kv put secret/ask/zitadel-psql \
  db-admin-password="<generate>"

IAM_SERVICE__PAT — chicken-and-egg

The PAT is created inside Zitadel after it boots. Set it to a placeholder now; update it post-deploy (Phase 10). VSO re-syncs and Reloader restarts the API pod automatically.

7. Kubernetes Pre-requisites

Manual steps that must be done before helm install or ArgoCD sync.

7a — Namespace¶

kubectl create namespace ask

7b — Harbor registry pull secret¶

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
ARGOCD_RO=$(vault kv get -field=token secret/gitlab/argocd-ro)

kubectl -n ask create secret docker-registry harbor-pull-secret \
  --docker-server=harbor.cl1.sq4.aegis.internal \
  --docker-username=argocd-ro \
  --docker-password="$ARGOCD_RO"

7c — DNS / `/etc/hosts`¶

On the bastion (and any machine that needs to reach these services):

NODE_IP="100.115.2.100"   # ingress VIP (Frontend VLAN)

sudo tee -a /etc/hosts <<EOF
${NODE_IP}  core-service.ask.mod.auh1.dev.dir
${NODE_IP}  sso.ask.mod.auh1.dev.dir
EOF

7d — CNPG Postgres Cluster CR¶

The ask-pg Cluster must exist and be Ready before the PreSync Jobs run:

# apply to the ask namespace before the first ArgoCD sync
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: ask-pg
  namespace: ask
spec:
  instances: 3
  storage:
    size: 20Gi
    storageClass: ceph-block
  enableSuperuserAccess: true
  superuserSecret:
    name: zitadel-psql-secret    # VSO must have rendered this first

# Verify the cluster is ready before deploying core-service
kubectl get cluster -n ask ask-pg
# Expected: STATUS=Cluster in healthy state
kubectl get svc -n ask ask-pg-rw
# Expected: TYPE=ClusterIP, PORT=5432

8. Skopeo Mirror Commands (reference)

All image mirror operations use the same pattern. Run from the bastion or any machine with access to both the source registry and harbor.cl1.sq4.aegis.internal.

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
GLT=$(vault kv get -field=token secret/gitlab/migration-token)
DEST="harbor.cl1.sq4.aegis.internal/ask"

# postgres
skopeo copy --override-arch amd64 \
  --dest-creds "root:${GLT}" --dest-tls-verify=false \
  docker://docker.io/library/postgres:16-alpine \
  docker://${DEST}/postgres:16-alpine

# python base images (for image builds)
skopeo copy --override-arch amd64 \
  --dest-creds "root:${GLT}" --dest-tls-verify=false \
  docker://docker.io/library/python:3.12 \
  docker://${DEST}/python:3.12

skopeo copy --override-arch amd64 \
  --dest-creds "root:${GLT}" --dest-tls-verify=false \
  docker://docker.io/library/python:3.12-slim \
  docker://${DEST}/python:3.12-slim

# zitadel images — fill in <ver> from vendored chart values
# skopeo copy --override-arch amd64 \
#   --dest-creds "root:${GLT}" --dest-tls-verify=false \
#   docker://ghcr.io/zitadel/zitadel:<ver> \
#   docker://${DEST}/zitadel:<ver>

9. Pre-Deploy Checklist

Run this before triggering the first ArgoCD sync or helm install:

# 1. CNPG operator running
kubectl get deployment -n cnpg-system cnpg-controller-manager

# 2. ask-pg cluster ready
kubectl get cluster -n ask ask-pg | grep "Cluster in healthy"

# 3. VSO secrets rendered
kubectl get secret -n ask core-service-secret zitadel-secret-kv zitadel-psql-secret

# 4. Pull secret present
kubectl get secret -n ask harbor-pull-secret

# 5. Images reachable from cluster (spot-check)
kubectl run img-check --rm -it --restart=Never \
  --image=harbor.cl1.sq4.aegis.internal/dockerhub/postgres:16-alpine \
  --overrides='{"spec":{"imagePullSecrets":[{"name":"harbor-pull-secret"}]}}' \
  -n ask -- psql --version

# 6. Zitadel subchart vendored in helm/charts/
ls helm/charts/zitadel-1.0.1.tgz

# 7. DNS resolves
curl -sk https://core-service.ask.mod.auh1.dev.dir/api/v1/health/live   # will fail until deployed — just checking DNS

➡️ Related: Discovery · Air-Gap Alignment · Config Updates · Deployment Runbook