34 — ASK Domain Migration: `*.ask.mod.auh1.dev.dir`¶

Move all services in this deployment — ASK application services and platform infrastructure — from *.mod.auh1.dev.dir to *.ask.mod.auh1.dev.dir. This cluster is dedicated to the ASK platform; the ask subdomain identifies the platform, not just the application tier.

Why a Dedicated ASK Subdomain?¶

Concern	Old pattern	New pattern
Platform identity	Flat `*.mod.auh1.dev.dir` — no grouping	`ask.mod.auh1.dev.dir` — cluster-wide ASK identity
DNS delegation	Flat — no logical zone	`ask.mod.auh1.dev.dir` is a delegatable zone
Wildcard coverage	`*.mod.auh1.dev.dir` one level only	`*.ask.mod.auh1.dev.dir` covers all ASK services
Future environments	Confusing alongside other tenants	`ask.mod.auh1.dev.dir` clearly owned by ASK platform

mod.auh1.dev.dir
└── ask.mod.auh1.dev.dir               ← entire cluster is ASK-dedicated
    ├── argocd.ask.mod.auh1.dev.dir    ← platform infra
    ├── vault.ask.mod.auh1.dev.dir     ← platform infra
    ├── hatchet.ask.mod.auh1.dev.dir   ← ASK services
    ├── core-service.ask.mod.auh1.dev.dir
    └── sso.ask.mod.auh1.dev.dir

Hostname Change Summary¶

Service	Old hostname	New hostname
Core Service API	`core-service.mod.auh1.dev.dir`	`core-service.ask.mod.auh1.dev.dir`
Zitadel SSO	`sso.mod.auh1.dev.dir`	`sso.ask.mod.auh1.dev.dir`
Hatchet UI	`hatchet.mod.auh1.dev.dir`	`hatchet.ask.mod.auh1.dev.dir`
ArgoCD	`argocd.mod.auh1.dev.dir`	`argocd.ask.mod.auh1.dev.dir`
Vault	`vault.mod.auh1.dev.dir`	`vault.ask.mod.auh1.dev.dir`

Note: *.cl1.sq4.aegis.internal hostnames (GitLab, Harbor, ArgoCD, Vault) are unchanged — that is a separate internal domain.

Files Changed (mod-ask-devops)¶

File	What changed
`applications/hatchet/values-mod-auh1-dev-ask.yaml`	4 hostname references
`applications/core-service/values-mod-auh1-dev-ask.yaml`	Ingress hosts + TOML `public_base_url` + Zitadel `ExternalDomain`
`applications/cluster-addons/argocd/values-mod-auh1-dev-ask.yaml`	`extraHosts` + `extraTls` hostname
`applications/cluster-addons/vault/values-mod-auh1-dev-ask.yaml`	Ingress host + TLS hosts
`applications/cluster-addons/vault/manifests/vault-tls-certificate.yaml`	`dnsNames` entry

Step 34.1 — Update Wildcard Certificate¶

The current wildcard cert covers *.mod.auh1.dev.dir (one level). *.ask.mod.auh1.dev.dir is a second level — a wildcard does not recurse — so it must be added explicitly.

Run on bastion (or any node with kubectl cluster-admin):

kubectl apply -f - << 'EOF'
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-tls
  namespace: cert-manager
spec:
  secretName: wildcard-tls-secret
  duration: 8760h
  renewBefore: 720h
  issuerRef:
    name: internal-ca-issuer
    kind: ClusterIssuer
  dnsNames:
    - "*.cl1.sq4.aegis.internal"
    - "cl1.sq4.aegis.internal"
    - "*.dev.dir"
    - "dev.dir"
    - "*.mod.auh1.dev.dir"
    - "mod.auh1.dev.dir"
    - "*.ask.mod.auh1.dev.dir"
    - "ask.mod.auh1.dev.dir"
    - "s3.cl1.sq4.aegis.internal"
EOF

Wait for cert-manager to reissue:

kubectl get certificate wildcard-tls -n cert-manager -w
# Expected: READY=True (reissues within ~10 seconds)

Verify new SAN is present:

kubectl get secret wildcard-tls-secret -n cert-manager \
  -o jsonpath='{.data.tls\.crt}' | base64 -d \
  | openssl x509 -noout -text | grep -A10 "Subject Alternative"
# Must include: DNS:*.ask.mod.auh1.dev.dir

Step 34.2 — HAProxy (Not Required for This Migration)¶

HAProxy on the Ceph MON nodes serves only s3.cl1.sq4.aegis.internal (Ceph RGW S3 endpoint). It has no role in serving *.ask.mod.auh1.dev.dir — that traffic goes through the nginx ingress controller inside the cluster.

The HAProxy cert does not need to be updated for this domain migration.

When HAProxy cert push IS needed

Only required after the annual wildcard cert renewal (renewBefore: 720h), when the cert-manager auto-renewed wildcard-tls-secret must be re-pushed to the Ceph MON nodes to keep S3 TLS working. See Phase 6 — Step 6.6.

Step 34.3 — Commit Updated Values + Sync ArgoCD¶

The values files in mod-ask-devops have been updated. Commit and push:

cd /path/to/mod-ask-devops
git add applications/hatchet/values-mod-auh1-dev-ask.yaml \
        applications/core-service/values-mod-auh1-dev-ask.yaml

git commit -m "feat(ask): migrate ASK services to *.ask.mod.auh1.dev.dir subdomain

- hatchet: serverUrl, grpcBroadcastAddress, ingress hosts
- core-service: ingress host, Zitadel ExternalDomain + ingress, TOML public URLs
- sso: ingress host in core-service chart"

git push origin main

ArgoCD will detect the change on next poll (or force sync):

argocd app sync mod-auh1-dev-ask.ask.hatchet
argocd app sync mod-auh1-dev-ask.ask.core-service

cert-manager auto-issues new per-service TLS certs for the new hostnames within ~30 seconds of ingress creation.

Step 34.4 — Update Mac /etc/hosts DNS¶

Add the new *.ask.mod.auh1.dev.dir entries alongside the existing ones.

The ingress controller VIP or NodePort IP stays the same — only the hostname changes.

First, get the current ingress controller external IP:

kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
# or if NodePort: use a worker node IP

Then on your Mac:

sudo vi /etc/hosts

Add:

# ASK platform services — ask.mod.auh1.dev.dir zone
<INGRESS_IP>  core-service.ask.mod.auh1.dev.dir
<INGRESS_IP>  sso.ask.mod.auh1.dev.dir
<INGRESS_IP>  hatchet.ask.mod.auh1.dev.dir

Remove (or comment out) the old entries:

# OLD — remove after migration verified
# <INGRESS_IP>  core-service.mod.auh1.dev.dir
# <INGRESS_IP>  sso.mod.auh1.dev.dir
# <INGRESS_IP>  hatchet.mod.auh1.dev.dir

Flush DNS cache:

sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder

Step 34.5 — Verify¶

# Wildcard cert has new SAN
kubectl get secret wildcard-tls-secret -n cert-manager \
  -o jsonpath='{.data.tls\.crt}' | base64 -d \
  | openssl x509 -noout -text | grep "ask\.mod\.auh1"
# Expected: DNS:*.ask.mod.auh1.dev.dir

# Per-service certs issued for new hostnames
kubectl get certificate -n ask
# Expected: core-service-tls, zitadel-tls (or zitadel-login-tls), hatchet-tls — all READY=True

# Ingresses using new hostnames
kubectl get ingress -n ask -o wide
kubectl get ingress -n hatchet -o wide

# TLS works end-to-end (Mac)
curl -sv https://core-service.ask.mod.auh1.dev.dir/api/v1/health/live 2>&1 \
  | grep -E "subject|issuer|< HTTP"
# Expected: subject=core-service.ask.mod.auh1.dev.dir, HTTP/2 200

curl -sv https://hatchet.ask.mod.auh1.dev.dir 2>&1 | grep "< HTTP"
# Expected: HTTP/2 200 or 302 (redirect to login)

curl -sv https://sso.ask.mod.auh1.dev.dir/debug/healthz 2>&1 | grep "< HTTP"
# Expected: HTTP/2 200

Rollback¶

If you need to revert before ArgoCD syncs:

# Revert values files
git revert HEAD --no-edit
git push origin main
argocd app sync mod-auh1-dev-ask.ask.hatchet
argocd app sync mod-auh1-dev-ask.ask.core-service

The wildcard cert update is additive (old SANs remain) — no rollback needed for the cert.

Checklist¶

[ ] Wildcard cert updated — *.ask.mod.auh1.dev.dir in SAN — READY=True ✅
[ ] mod-ask-devops committed and pushed — all 5 values files
[ ] ArgoCD synced — hatchet + core-service Applications Healthy
[ ] Per-service certs issued — kubectl get certificate -n ask all READY=True
[ ] Mac /etc/hosts updated — new entries, old removed
[ ] DNS cache flushed — dscacheutil -flushcache
[ ] Browser test — https://core-service.ask.mod.auh1.dev.dir shows padlock
[ ] Browser test — https://hatchet.ask.mod.auh1.dev.dir shows padlock
[ ] Browser test — https://sso.ask.mod.auh1.dev.dir shows padlock

34 — ASK Domain Migration: *.ask.mod.auh1.dev.dir¶