Skip to content

Phase 6 — TLS with cert-manager (Bootstrap CA)

Configure cert-manager to act as its own CA using the self-signed bootstrap approach. No external PKI or Vault dependency for TLS certificate issuance.

Approach Decision

Option Chosen Reason
Enterprise CA (external PKI) ❌ No access to ca.key — enterprise PKI team controls it
Vault PKI ❌ Vault outage = cert renewal blocked → service TLS breaks
cert-manager Bootstrap (self-signed CA) ✅ No external dependency, cert issuance always available

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                        cert-manager (bootstrap)                      │
│                                                                      │
│   ClusterIssuer/selfsigned-bootstrapper  (self-signed — one-time)   │
│                      │                                               │
│                      │ issues CA cert                                │
│                      ▼                                               │
│          Secret/internal-ca-secret  (cert-manager ns)               │
│           tls.crt = CA cert  │  tls.key = CA private key            │
│                      │                                               │
│                      ▼                                               │
│          ClusterIssuer/internal-ca-issuer                            │
│                      │                                               │
│         ┌────────────┴──────────────┐                               │
│         │                           │                               │
│         ▼                           ▼                               │
│  Secret/wildcard-tls-secret   Per-app certs (optional)              │
│  (cert-manager ns)            cert-manager auto-issues on demand    │
└──────────┬──────────────────────────────────────────────────────────┘
           │
           │  One cert — two consumers
           │
    ┌──────┴──────────────────────────────────────┐
    │                                              │
    ▼                                              ▼
┌───────────────────────────┐        ┌────────────────────────────────┐
│  HAProxy (external)       │        │  K8s Nginx Ingress Controller  │
│  Ceph MON nodes           │        │  (in-cluster)                  │
│  100.115.1.10 (VIP)       │        │                                │
│                           │        │  *.mod.auh1.dev.dir            │
│  s3.cl1.sq4.aegis.int     │        │  *.cl1.sq4.aegis.internal      │
│  ──────────────────────   │        │  ┌─────────────────────────┐  │
│  TLS termination for RGW  │        │  │ ask.mod.auh1.dev.dir     │  │
│  S3 object storage        │        │  │ api.mod.auh1.dev.dir     │  │
│                           │        │  │ argocd.mod.auh1.dev.dir  │  │
│  ⚠ Manual push once/year  │        │  │ gitlab.mod.auh1.dev.dir  │  │
│  Step 6.6 — scp haproxy   │        │  │ vault.mod.auh1.dev.dir   │  │
│  .pem to all 3 MON nodes  │        │  │ harbor.mod.auh1.dev.dir  │  │
│                           │        │  │ argocd.cl1.sq4.aegis.int │  │
│                           │        │  │ gitlab.cl1.sq4.aegis.int │  │
│                           │        │  │ vault.cl1.sq4.aegis.int  │  │
│                           │        │  │ harbor.cl1.sq4.aegis.int │  │
│                           │        │  └─────────────────────────┘  │
└───────────────────────────┘        │                                │
                                     │  ✅ Auto-renewed by cert-manager│
                                     │  Zero manual steps             │
                                     └────────────────────────────────┘

Certificate Distribution Summary

Consumer Cert Delivery Renewal Manual Step
HAProxy (RGW S3) scp haproxy.pem → all 3 MON nodes cert-manager auto-renews K8s secret Re-push PEM once/year
Nginx Ingress (K8s apps) secretName: wildcard-tls-secret in Ingress spec ✅ Fully automatic None
Per-app cert (optional) cert-manager.io/cluster-issuer annotation ✅ Fully automatic None

Known Risks & Mitigations

Known Risks — Read Before Proceeding

Risk Severity Mitigation
CA private key in etcd Medium K8s Secret in cert-manager namespace — protected by RBAC. Not internet-facing. Restrict get secret in this namespace to cluster-admins only.
No certificate revocation Low Cannot revoke a compromised leaf cert — must wait for expiry. Set short TTL (90d) for sensitive services if needed.
No audit trail Low No log of which certs were issued. Mitigate with K8s API server audit logging.
CA cert expiry High CA set to 10 years. If it expires, ALL certs become untrusted immediately. Monitor CA expiry — set a calendar reminder.
Wildcard cert on HAProxy — manual renewal Low Cert expires in 1 year. cert-manager auto-renews in K8s but the copy on HAProxy must be re-pushed manually. Set a calendar reminder at 11 months.
Client trust distribution Medium All clients (browsers, nodes, pods) must trust the bootstrap CA cert — distribute modgpt-ca.crt manually.

Why not Vault PKI?

Vault PKI is more secure (CA key never leaves Vault, full revocation, audit trail). However: if Vault is down, cert-manager cannot renew expiring certs — services lose TLS. For this deployment, cert availability > cert auditability. Vault PKI remains the production-correct long-term target once Vault HA is proven stable.

Future migration path

When enterprise CA access is granted or Vault PKI is adopted:

  1. Create new ClusterIssuer pointing to enterprise CA or Vault
  2. cert-manager re-issues all K8s certs on next renewal cycle — automatic
  3. Re-push wildcard cert to HAProxy once
  4. Remove bootstrap CA secret

No downtime required.

Prerequisites

Item Required state
cert-manager Installed, pods Running in cert-manager namespace
kubectl Working from bastion
SSH access bastion → Ceph MON nodes ([email protected]/12/13)
HAProxy Installed on Ceph MON nodes (see Ceph Infrastructure → RGW HA)

Step 6.1 — Create Self-Signed Bootstrap Issuer

One-time bootstrapper — used only to issue the CA cert, nothing else.

kubectl apply -f - << 'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-bootstrapper
spec:
  selfSigned: {}
EOF

kubectl get clusterissuer selfsigned-bootstrapper
# Expected: READY=True

Step 6.2 — Issue the Internal CA Certificate

kubectl apply -f - << 'EOF'
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-ca
  namespace: cert-manager
spec:
  isCA: true
  secretName: internal-ca-secret
  duration: 87600h        # 10 years
  renewBefore: 720h
  subject:
    organizations:
      - "ai71"
  commonName: "modgpt"
  issuerRef:
    name: selfsigned-bootstrapper
    kind: ClusterIssuer
EOF

# Wait for CA cert
kubectl get certificate internal-ca -n cert-manager -w
# Expected: READY=True

# Verify it is a CA cert
kubectl get secret internal-ca-secret -n cert-manager \
  -o jsonpath='{.data.tls\.crt}' | base64 -d \
  | openssl x509 -noout -subject -dates -ext basicConstraints
# Expected:
# subject=CN=modgpt, O=ai71
# CA:TRUE
# notAfter=<10 years from now>

Step 6.3 — Create the CA ClusterIssuer

All applications and services use this issuer to request certificates.

kubectl apply -f - << 'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca-issuer
spec:
  ca:
    secretName: internal-ca-secret
EOF

kubectl get clusterissuer internal-ca-issuer
# Expected: READY=True

Step 6.4 — Extract CA Certificate for Trust Distribution

Clients need the CA cert to verify any cert issued by this issuer.

# Extract CA cert
kubectl get secret internal-ca-secret -n cert-manager \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/modgpt-ca.crt

openssl x509 -noout -subject -dates -in /tmp/modgpt-ca.crt

Push CA to all nodes (K8s + Ceph MON)

for node in 100.115.1.11 100.115.1.12 100.115.1.13 100.115.1.14 100.115.1.15 100.115.1.16 100.115.1.17; do
  scp /tmp/modgpt-ca.crt cloud-user@${node}:/tmp/
  ssh cloud-user@${node} "sudo cp /tmp/modgpt-ca.crt /usr/local/share/ca-certificates/modgpt-ca.crt \
    && sudo update-ca-certificates"
  echo "✅ ${node} trusts modgpt CA"
done

Trust in browser (Mac)

sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain /tmp/modgpt-ca.crt

Verify CA is trusted on every node

Check the CA file is present and valid on all nodes:

# Check file exists
for node in 100.115.1.11 100.115.1.12 100.115.1.13 100.115.1.14 100.115.1.15 100.115.1.16 100.115.1.17; do
  echo -n "Node $node: "
  ssh cloud-user@$node "ls /usr/local/share/ca-certificates/modgpt-ca.crt 2>/dev/null \
    && echo '✅ present' || echo '❌ missing'"
done

# Verify CA cert is valid and trusted
for node in 100.115.1.11 100.115.1.12 100.115.1.13 100.115.1.14 100.115.1.15 100.115.1.16 100.115.1.17; do
  echo -n "Node $node: "
  ssh cloud-user@$node "openssl verify \
    -CAfile /usr/local/share/ca-certificates/modgpt-ca.crt \
    /usr/local/share/ca-certificates/modgpt-ca.crt 2>/dev/null \
    && echo '✅ CA valid' || echo '❌ CA not trusted'"
done

Expected output: 

Node 100.115.1.11: ✅ CA valid
Node 100.115.1.12: ✅ CA valid
Node 100.115.1.13: ✅ CA valid
Node 100.115.1.14: ✅ CA valid
Node 100.115.1.15: ✅ CA valid
Node 100.115.1.16: ✅ CA valid
Node 100.115.1.17: ✅ CA valid

Step 6.4b — Trust on Bastion Node

All CLI tools on the bastion (curl, helm, argocd) call https://gitlab.cl1.sq4.aegis.internal and other internal HTTPS endpoints. Without the CA, every call needs --insecure or --skip-tls-verify.

# On bastion (cloud-user@bastion1)
# Get the CA cert from the K8s secret or copy from where it was extracted
kubectl get secret internal-ca-secret -n cert-manager \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/modgpt-ca.crt

# Add to system trust store
sudo cp /tmp/modgpt-ca.crt /usr/local/share/ca-certificates/modgpt-ca.crt
sudo update-ca-certificates

# Verify
openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt /tmp/modgpt-ca.crt
# Expected: /tmp/modgpt-ca.crt: OK

# Test — curl to GitLab should work WITHOUT --insecure
curl -s https://gitlab.cl1.sq4.aegis.internal/-/readiness | python3 -m json.tool | head -5
# Expected: {"master_check":[{"status":"ok"}],...}

Keep the cert on bastion for reuse: 

mkdir -p ~/certificates
cp /tmp/modgpt-ca.crt ~/certificates/modgpt-ca.crt

Step 6.4c — Trust in ArgoCD Cert Store

ArgoCD repo-server connects to https://gitlab.cl1.sq4.aegis.internal to pull GitOps repos and https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/.../packages/helm/... for Helm charts. Without the CA, every connection fails TLS verification.

# Add modgpt CA to ArgoCD TLS cert store (keyed by hostname)
argocd cert add-tls gitlab.cl1.sq4.aegis.internal \
  --from ~/certificates/modgpt-ca.crt \
  --grpc-web --upsert

# Cover the Helm registry hostname (same cert, same host)
# No separate entry needed — hostname is the same

# Verify
argocd cert list --grpc-web | grep gitlab

Expected: gitlab.cl1.sq4.aegis.internal https <CN=modgpt>

ArgoCD cert store vs --insecure

argocd cert add-tls stores the CA in the argocd-tls-certs-cm ConfigMap. ArgoCD repo-server loads it at startup. After adding, repos connecting to that hostname no longer need --insecure-skip-server-verification.

Check the ConfigMap directly: 

kubectl get configmap argocd-tls-certs-cm -n argocd -o yaml | grep -A3 gitlab

Step 6.4d — Trust in GitLab (Outbound HTTPS)

GitLab makes outbound HTTPS calls when: - Connecting to Vault for secrets - Webhook calls to other internal services - GitLab Runners connecting to internal registries

Add the modgpt CA to GitLab via its Helm values global.certificates.customCACerts:

# Extract the GitLab Helm release values
helm get values gitlab -n gitlab > /tmp/gitlab-current-values.yaml

# Check if customCACerts is already set
grep -A5 customCACerts /tmp/gitlab-current-values.yaml || echo "Not set yet"

Add the CA via a Kubernetes secret and reference it in GitLab:

# Create a secret with the modgpt CA cert in the gitlab namespace
kubectl create secret generic modgpt-ca \
  -n gitlab \
  --from-file=modgpt-ca.crt=~/certificates/modgpt-ca.crt \
  --dry-run=client -o yaml | kubectl apply -f -

In the GitLab Helm values (in GitOps — update values-mod-auh1-dev-ask.yaml or GitLab's values file):

global:
  certificates:
    customCACerts:
      - secret: modgpt-ca

Requires GitLab Helm upgrade to take effect

The customCACerts value causes GitLab to mount the secret and inject the CA into its internal containers (webservice, sidekiq, registry). Apply via helm upgrade or through the ArgoCD Application sync (if GitLab is managed by ArgoCD).

If GitLab is NOT managed by ArgoCD in this deployment, run: 

helm upgrade gitlab gitlab/gitlab -n gitlab \
  --reuse-values \
  --set "global.certificates.customCACerts[0].secret=modgpt-ca"

In practice here: GitLab outbound HTTPS calls to Vault/internal services are not required until Phase 7 (ESO + Vault). Skip this step until then — it's documented here for completeness.

Step 6.5 — Issue Wildcard Certificate

One wildcard cert covers all services — K8s Ingress, HAProxy VIP, and any future services.

kubectl apply -f - << 'EOF'
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-tls
  namespace: cert-manager
spec:
  secretName: wildcard-tls-secret
  duration: 8760h          # 1 year
  renewBefore: 720h        # cert-manager auto-renews in K8s — triggers HAProxy re-push reminder
  issuerRef:
    name: internal-ca-issuer
    kind: ClusterIssuer
  dnsNames:
    - "*.cl1.sq4.aegis.internal"    # covers all subdomains
    - "cl1.sq4.aegis.internal"      # bare domain
    - "*.dev.dir"                   # additional TLD
    - "dev.dir"                     # bare domain
    - "*.mod.auh1.dev.dir"          # ASK platform domain
    - "mod.auh1.dev.dir"            # bare domain
    - "s3.cl1.sq4.aegis.internal"   # RGW S3 endpoint
EOF

kubectl get certificate wildcard-tls -n cert-manager -w
# Expected: READY=True

# Verify SAN
kubectl get secret wildcard-tls-secret -n cert-manager \
  -o jsonpath='{.data.tls\.crt}' | base64 -d \
  | openssl x509 -noout -text | grep -A6 "Subject Alternative"
# Expected:
# DNS:*.cl1.sq4.aegis.internal
# DNS:cl1.sq4.aegis.internal
# DNS:*.dev.dir
# DNS:dev.dir
# DNS:*.mod.auh1.dev.dir
# DNS:mod.auh1.dev.dir
# DNS:s3.cl1.sq4.aegis.internal

Step 6.6 — Push Wildcard Cert to HAProxy (Manual — Once per Year)

Why manual?

HAProxy runs on external Ceph nodes outside the cluster. A wildcard cert with 1-year TTL only needs to be pushed once. cert-manager auto-renews the K8s Secret — when renewal happens, re-run this step. Set a calendar reminder at 11 months.

# Extract cert and key from K8s
kubectl get secret wildcard-tls-secret -n cert-manager \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/tls.crt

kubectl get secret wildcard-tls-secret -n cert-manager \
  -o jsonpath='{.data.tls\.key}' | base64 -d > /tmp/tls.key

# Combine into HAProxy PEM format (cert + key in one file)
cat /tmp/tls.crt /tmp/tls.key > /tmp/haproxy.pem

# Create cert directory and push to all 3 Ceph MON nodes
for node in 100.115.1.11 100.115.1.12 100.115.1.13; do
  ssh cloud-user@${node} "sudo mkdir -p /opt/certs/rgw"
  scp /tmp/haproxy.pem cloud-user@${node}:/tmp/haproxy.pem
  ssh cloud-user@${node} "sudo mv /tmp/haproxy.pem /opt/certs/rgw/haproxy.pem \
    && sudo chmod 600 /opt/certs/rgw/haproxy.pem \
    && sudo systemctl reload haproxy"
  echo "✅ ${node} updated"
done

# Clean up sensitive files from bastion
rm /tmp/tls.crt /tmp/tls.key /tmp/haproxy.pem

echo "⏰ Set calendar reminder: re-push wildcard cert in 11 months"

Verify TLS on VIP: 

curl -sv --cacert /tmp/modgpt-ca.crt https://100.115.1.10:7480 2>&1 \
  | grep -E "subject|issuer|< HTTP"
# Expected: HTTP/1.1 200, subject=*.cl1.sq4.aegis.internal

Step 6.7 — Configure HAProxy for TLS

# Run on all 3 Ceph MON nodes
for node in 100.115.1.11 100.115.1.12 100.115.1.13; do
ssh cloud-user@${node} "sudo tee /etc/haproxy/haproxy.cfg > /dev/null << 'HAEOF'
global
    log /dev/log local0
    maxconn 4096
    user haproxy
    group haproxy
    daemon

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5s
    timeout client  30s
    timeout server  30s

frontend rgw_frontend
    bind *:7480 ssl crt /opt/certs/rgw/haproxy.pem
    default_backend rgw_backends

backend rgw_backends
    balance roundrobin
    option  httpchk GET /
    http-check expect status 200
    server rgw1 100.115.1.11:7481 check inter 2s fall 2 rise 2
    server rgw2 100.115.1.12:7481 check inter 2s fall 2 rise 2
    server rgw3 100.115.1.13:7481 check inter 2s fall 2 rise 2

listen stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    stats auth admin:admin
HAEOF
sudo systemctl reload haproxy && echo '✅ HAProxy reloaded on ${node}'"
done

Step 6.8 — Enable TLS on K8s App Ingresses

For K8s apps, cert-manager issues a dedicated cert per service via the cluster-issuer annotation. cert-manager handles issuance and renewal automatically — no manual steps.

Production Recommendation — Option B

Use Option B (per-service cert via annotation) for all ingresses. Each service gets an isolated cert — smaller blast radius, auto-renewed, no secret copying.

Option A — Not Recommended for Production

Copying the wildcard secret across namespaces creates secret sprawl and requires manual re-copy on every wildcard renewal. Use only as a temporary workaround.

Option A — Reference wildcard secret directly ⚠️ Non-production only

# Copy wildcard secret to target namespace
kubectl get secret wildcard-tls-secret -n cert-manager -o yaml \
  | sed 's/namespace: cert-manager/namespace: gitlab/' \
  | kubectl apply -f -

spec:
  tls:
    - hosts:
        - gitlab.cl1.sq4.aegis.internal
      secretName: wildcard-tls-secret

Option B — cert-manager annotation (per-service cert) ✅ Production

cert-manager automatically issues and renews a dedicated cert for each ingress hostname.

metadata:
  annotations:
    cert-manager.io/cluster-issuer: "internal-ca-issuer"
spec:
  tls:
    - hosts:
        - gitlab.cl1.sq4.aegis.internal
      secretName: gitlab-tls-secret    # cert-manager creates this automatically

Patch Commands — All Ingresses

Apply TLS to every existing ingress using Option B:

ArgoCD

kubectl patch ingress argocd-server -n argocd --type=merge -p '{
  "metadata": {
    "annotations": {
      "cert-manager.io/cluster-issuer": "internal-ca-issuer",
      "nginx.ingress.kubernetes.io/ssl-redirect": "true"
    }
  },
  "spec": {
    "tls": [{
      "hosts": [
        "argocd.cl1.sq4.aegis.internal",
        "argocd.mod.auh1.dev.dir"
      ],
      "secretName": "argocd-tls-secret"
    }]
  }
}'

GitLab

kubectl patch ingress gitlab-webservice-default -n gitlab --type=merge -p '{
  "metadata": {
    "annotations": {
      "cert-manager.io/cluster-issuer": "internal-ca-issuer",
      "nginx.ingress.kubernetes.io/ssl-redirect": "true"
    }
  },
  "spec": {
    "tls": [{
      "hosts": [
        "gitlab.cl1.sq4.aegis.internal",
        "gitlab.mod.auh1.dev.dir"
      ],
      "secretName": "gitlab-tls-secret"
    }]
  }
}'

GitLab Registry

kubectl patch ingress gitlab-registry -n gitlab --type=merge -p '{
  "metadata": {
    "annotations": {
      "cert-manager.io/cluster-issuer": "internal-ca-issuer"
    }
  },
  "spec": {
    "tls": [{
      "hosts": [
        "registry.cl1.sq4.aegis.internal"
      ],
      "secretName": "gitlab-registry-tls-secret"
    }]
  }
}'

GitLab KAS

kubectl patch ingress gitlab-kas -n gitlab --type=merge -p '{
  "metadata": {
    "annotations": {
      "cert-manager.io/cluster-issuer": "internal-ca-issuer"
    }
  },
  "spec": {
    "tls": [{
      "hosts": [
        "kas.cl1.sq4.aegis.internal"
      ],
      "secretName": "gitlab-kas-tls-secret"
    }]
  }
}'

Monitoring (Grafana, Prometheus, Alertmanager)

for ingress in rancher-monitoring-grafana rancher-monitoring-prometheus rancher-monitoring-alertmanager; do
  host=$(kubectl get ingress $ingress -n cattle-monitoring-system -o jsonpath='{.spec.rules[0].host}')
  kubectl patch ingress $ingress -n cattle-monitoring-system --type=merge -p "{
    \"metadata\": {
      \"annotations\": {
        \"cert-manager.io/cluster-issuer\": \"internal-ca-issuer\"
      }
    },
    \"spec\": {
      \"tls\": [{
        \"hosts\": [\"${host}\"],
        \"secretName\": \"${ingress}-tls-secret\"
      }]
    }
  }"
  echo "✅ TLS patched: $ingress"
done

Verify Certs Issued

# Check all certificates
kubectl get certificate -A

# Expected: all READY=True
# NAME                         READY   SECRET                       AGE
# argocd-tls-secret            True    argocd-tls-secret            30s
# gitlab-tls-secret            True    gitlab-tls-secret            30s
# gitlab-registry-tls-secret   True    gitlab-registry-tls-secret   30s
# ...

Test in browser: 

https://argocd.cl1.sq4.aegis.internal   ← should show padlock 🔒
https://argocd.mod.auh1.dev.dir         ← should show padlock 🔒
https://gitlab.cl1.sq4.aegis.internal   ← should show padlock 🔒

Browser shows untrusted cert?

The CA cert (modgpt-ca.crt) must be trusted in your browser/OS keychain. See Step 6.4 — Trust on Mac to add it.

Renewal — What Happens When Cert Expires

cert-manager auto-renews wildcard cert 30 days before expiry
        │
        ▼
K8s Secret (wildcard-tls-secret) updated automatically
K8s Ingress certs — renewed automatically ✅
        │
        ▼
HAProxy cert on Ceph nodes — still has OLD cert ⚠️
        │
        ▼
Re-run Step 6.6 (takes ~5 minutes)  ← only manual step per year

Annual reminder — re-push wildcard cert to HAProxy

cert-manager renews the K8s wildcard cert automatically. HAProxy on the Ceph nodes must be updated manually — re-run Step 6.6. Set a calendar reminder at 11 months from today.

Summary Checklist

cert-manager Bootstrap

  • [ ] ClusterIssuer/selfsigned-bootstrapper — READY=True
  • [ ] Certificate/internal-ca — READY=True, isCA=true, 10yr expiry
  • [ ] ClusterIssuer/internal-ca-issuer — READY=True

Trust Distribution

  • [ ] modgpt-ca.crt added to all K8s nodes — update-ca-certificates done
  • [ ] modgpt-ca.crt added to all Ceph MON nodes — update-ca-certificates done
  • [ ] modgpt-ca.crt added to bastion (/usr/local/share/ca-certificates/) — update-ca-certificates done
  • [ ] curl https://gitlab.cl1.sq4.aegis.internal works on bastion without --insecure
  • [ ] modgpt-ca.crt added to ArgoCD cert store (argocd cert add-tls gitlab.cl1.sq4.aegis.internal)
  • [ ] ArgoCD repo list shows STATUS: Successful for GitLab repo (no TLS errors)
  • [ ] modgpt-ca.crt trusted in Mac browser keychain (Keychain Access → System → Always Trust)
  • [ ] GitLab modgpt-ca secret created in gitlab namespace (for Phase 7 Vault integration)

Wildcard Certificate

  • [ ] Certificate/wildcard-tls — READY=True, SAN includes *.cl1.sq4.aegis.internal, *.dev.dir, *.mod.auh1.dev.dir, s3.cl1.sq4.aegis.internal
  • [ ] haproxy.pem pushed to all 3 Ceph nodes
  • [ ] sudo systemctl reload haproxy done on all 3 Ceph nodes
  • [ ] curl --cacert modgpt-ca.crt https://100.115.1.10:7480 → HTTP 200
  • [ ] ⏰ Calendar reminder set — re-push cert in 11 months

K8s App TLS

  • [ ] GitLab Ingress — TLS enabled
  • [ ] ArgoCD Ingress — TLS enabled
  • [ ] global.ingress.tls.enabled: true in GitDeployed values → helm upgrade

Troubleshooting

Symptom Cause Fix
ClusterIssuer READY=False internal-ca-secret not in cert-manager namespace Check namespace
Certificate READY=False ClusterIssuer not ready kubectl describe certificate → check events
curl: SSL certificate verify failed CA not trusted on client Add modgpt-ca.crt to trust store + update-ca-certificates
HAProxy TLS handshake fails haproxy.pem missing or wrong format Re-run Step 6.6 — ensure cat tls.crt tls.key > haproxy.pem
Wildcard cert not covering a hostname Hostname outside *.cl1.sq4.aegis.internal and *.dev.dir Add explicit SAN to Certificate CR
HAProxy cert expired (browser error) Forgot annual re-push Re-run Step 6.6