Phase 8 — CloudNativePG Operator (CNPG)¶

Deploy the CloudNativePG operator as a cluster add-on. CNPG manages PostgreSQL clusters as Kubernetes-native CRDs — it replaces Azure Managed Postgres in this air-gapped environment.

Why CNPG¶

Upstream (Azure)	This cluster
`ai71-uaen-stg-ask71-psql-01.postgres.database.azure.com`	`ask-pg-rw.ask.svc.cluster.local:5432`
Azure PaaS — unreachable in air-gap	CloudNativePG on Ceph RBD
Managed TLS, managed backups	CNPG-managed TLS, backup via Ceph snapshots

CNPG is deployed in two layers:

Layer	What	Where
Operator (this phase)	`cloudnative-pg` Helm chart — CRDs + controller	`cnpg-system` namespace
PostgreSQL Cluster (Phase ASK deploy)	`Cluster` CR — 1 primary + 2 replicas	`ask` namespace

The operator must be running before any Cluster CR is applied.

Architecture¶

cnpg-system namespace
└── cloudnative-pg controller (Deployment)
      ↓ watches
ask namespace
└── Cluster/ask-pg  (CR — 3 pods: 1 primary + 2 replicas)
    ├── ask-pg-rw   (Service → primary, port 5432)   ← app connects here
    ├── ask-pg-ro   (Service → replicas, read-only)
    └── PVCs        (3 × 20Gi csi-rbd-sc)

Two databases on one Postgres server (created by ASK chart PreSync hooks):

Database	Role	Created by
`zitadel`	`zitadel`	`zitadelPostgresInit` Job (wave −1)
`core-service`	`core-service`	`postgresInit` Job (wave −1)

Component Versions¶

Component	Version
CNPG Helm chart	`0.22.1`
CNPG operator image	`cloudnative-pg/cloudnative-pg:1.24.1`
Harbor path	`harbor.cl1.sq4.aegis.internal/ghcr/cloudnative-pg/cloudnative-pg:1.24.1`
Namespace	`cnpg-system`

Prerequisites¶

Item	Required state
ArgoCD	Running
cert-manager	Running with `internal-ca-issuer`
ESO	Running — `harbor-pull-secret` auto-created via label
`cloudnative-pg:1.24.1`	Present in Harbor `dockerhub/` project
`cloudnative-pg` chart	Pushed to GitLab Helm registry (project 14)

Step 1 — Verify image in Harbor¶

HARBOR_USER=$(python3 -c "import json; d=json.load(open('${HOME}/harbor-images/harbor-robot.json')); print(d.get('name', d.get('username')))")
HARBOR_PASS=$(python3 -c "import json; d=json.load(open('${HOME}/harbor-images/harbor-robot.json')); print(d.get('secret', d.get('token', d.get('password'))))")

curl -sk -u "${HARBOR_USER}:${HARBOR_PASS}" \
  "https://harbor.cl1.sq4.aegis.internal/api/v2.0/projects/ghcr/repositories/cloudnative-pg%2Fcloudnative-pg/artifacts?page_size=5" \
  | python3 -m json.tool | grep '"name"'
# Expected: "1.24.1"

Step 2 — Files in mod-ask-devops¶

applications/cluster-addons/cnpg/
└── values-mod-auh1-dev-ask.yaml

environment/auh1/dev/02-argocd-application/cluster-addons/
└── cnpg.yaml

values-mod-auh1-dev-ask.yaml¶

image:
  repository: harbor.cl1.sq4.aegis.internal/ghcr/cloudnative-pg/cloudnative-pg
  tag: "1.24.1"
  pullPolicy: IfNotPresent

imagePullSecrets:
  - name: harbor-pull-secret

resources:
  requests:
    memory: 256Mi
    cpu: 100m
  limits:
    memory: 512Mi
    cpu: 500m

cnpg.yaml (ArgoCD Application)¶

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cnpg
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.io
spec:
  project: mod-auh1-dev-ask.cluster-addons
  sources:
    - repoURL: 'https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/14/packages/helm/stable'
      chart: cloudnative-pg
      targetRevision: "0.24.1"
      helm:
        valueFiles:
          - $values/applications/cluster-addons/cnpg/values-mod-auh1-dev-ask.yaml
    - repoURL: 'https://gitlab.cl1.sq4.aegis.internal/ai71/ask71/mod-ask-devops.git'
      targetRevision: main
      ref: values
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: cnpg-system
  syncPolicy:
    automated:
      prune: true
    syncOptions:
      - CreateNamespace=true
    managedNamespaceMetadata:
      labels:
        pod-security.kubernetes.io/enforce: baseline
        pod-security.kubernetes.io/warn: baseline
        harbor-pull-secret: "true"

harbor-pull-secret: true label

managedNamespaceMetadata includes harbor-pull-secret: "true" — ESO automatically creates harbor-pull-secret in cnpg-system within seconds of namespace creation. No manual copy needed.

Step 3 — Push chart to GitLab Helm registry¶

# From Mac
export GITLAB_TOKEN=$(kubectl exec -n vault vault-0 -- \
  env VAULT_CACERT=/vault/userconfig/vault-tls/ca.crt \
  env VAULT_TOKEN=${MAIN_ROOT} \
  vault kv get -field=token secret/gitlab/migration-token)

cd /path/to/helm-push-scripts
python3 gitlab-push-charts.py -c gitlab-charts.yaml --only cloudnative-pg

Verify:

curl -sk --header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" \
  "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/14/packages?package_type=helm" \
  | python3 -m json.tool | grep '"name"'
# Expected: "cloudnative-pg" in output

Step 4 — Push and sync¶

# Push from Mac
cd /Users/tarun.mittal/Documents/modgpt/mod-ask-devops
git add applications/cluster-addons/cnpg/ \
        environment/auh1/dev/02-argocd-application/cluster-addons/cnpg.yaml
git commit -m "feat: deploy CloudNativePG operator v1.24.1"
git push

# Sync (or wait for automated sync)
argocd app sync cnpg --grpc-web --assumeYes

Step 5 — Verify operator is running¶

kubectl get pods -n cnpg-system
# Expected: cloudnative-pg-<hash>   1/1 Running

kubectl get crds | grep cnpg
# Expected: clusters.postgresql.cnpg.io and others

kubectl get clustersecretstore vault-backend
# Unrelated but confirms ESO is still healthy

Step 6 — Verify harbor-pull-secret was auto-created¶

kubectl get secret harbor-pull-secret -n cnpg-system
# Expected: kubernetes.io/dockerconfigjson — created automatically by ESO

Storage Resize (Production Runbook)¶

CNPG does not auto-grow storage. Manual resize is required.

Trigger: Alert when cnpg_pg_database_size_bytes / cnpg_pg_volume_size_bytes > 0.8 (80% full).

Resize — zero downtime:

kubectl patch cluster <cluster-name> -n <namespace> \
  --type merge -p '{"spec":{"storage":{"size":"<new-size>"}}}'

# e.g. grow ask-pg from 20Gi to 40Gi:
kubectl patch cluster ask-pg -n ask \
  --type merge -p '{"spec":{"storage":{"size":"40Gi"}}}'

CNPG propagates the resize to all instance PVCs automatically. Ceph RBD (csi-rbd-sc) supports online expansion — no pod restart required.

WAL volume (separate PVC, prevents WAL growth from filling data volume):

spec:
  walStorage:
    size: 4Gi
    storageClass: csi-rbd-sc

Set walStorage on the production ask-pg cluster to isolate WAL growth.

Summary Checklist¶

[x] cloudnative-pg:1.28.1 image present in Harbor ghcr/ project
[x] cloudnative-pg chart 0.27.1 pushed to GitLab Helm registry
[x] Values file committed — applications/cluster-addons/cnpg/values-mod-auh1-dev-ask.yaml
[x] ArgoCD Application committed — environment/auh1/dev/02-argocd-application/cluster-addons/cnpg.yaml
[x] ServerSideApply=true in syncOptions (required — CRDs exceed 262 KB annotation limit)
[x] cnpg ArgoCD Application — Synced, Healthy
[x] cloudnative-pg pod Running in cnpg-system — confirmed cnpg-cloudnative-pg 1/1 (2026-06-19)
[x] CRDs installed — clusters.postgresql.cnpg.io present (core-pg / hatchet-pg / knowledge-pg Clusters running)
[x] harbor-pull-secret auto-created in cnpg-system by ESO
[ ] Prometheus alert configured for disk > 80% — observability follow-up (not yet done)

Troubleshooting¶

Symptom	Cause	Fix
`ImagePullBackOff` on cnpg pod	`harbor-pull-secret` not in `cnpg-system`	Verify `harbor-pull-secret: "true"` label on namespace; check ESO CES status
CRDs not installed	Chart sync failed	`argocd app sync cnpg --grpc-web`; check app logs
`Cluster` CR stuck `Setting up primary`	CNPG operator not running	Verify operator pod is `1/1 Running` before applying Cluster CR
Pod `Pending` — PSA violation	PSA `restricted` blocks CNPG init containers	Namespace must be labeled `baseline` (handled by `managedNamespaceMetadata`)