Skip to content

Fix S3 (Ceph RGW) Browser Cert Trust

Browser file uploads in KMS and core-service fail with net::ERR_CERT_AUTHORITY_INVALID (Chrome) / CORS request did not succeed… Status code: (null) (Firefox). This page documents the root cause and the implemented fix: distributing the Ceph RGW's CA (Aegis CL1.SQ4 CA Root) to client trust stores.

Resolution — Aegis root CA distributed to clients (implemented)

The browser-facing S3 hostname stays s3.cl1.sq4.aegis.internal (unchanged endpoint_url, no DNS change, no Ceph change). Uploads are fixed by making the client browser trust the Aegis root CA that signs RGW's cert — the same class of fix that makes ask.mod.auh1.dev.dir trusted today (the modgpt root CA is already distributed to clients). See Solution below.

The cert-manager / nginx-ingress alternative (routing s3.mod.auh1.dev.dir through the cluster ingress) was investigated and rejected — it required RGW to recognize the new hostname, a Ceph-side radosgw-admin zonegroup change we couldn't make (Ceph is managed by a separate team, no access). Kept as an appendix for the record.

Root cause (confirmed)

Value
RGW cert issuer O=Aegis CL1.SQ4 CA (infra provider CA — browsers don't trust it by default)
RGW cert chain leaf CN=s3.cl1.sq4.aegis.internal → Aegis…Intermediate CA → Aegis…Root CA (full chain served)
RGW cert SANs s3.cl1.sq4.aegis.internal, IP 100.115.1.130
Frontend cert issuer O=ai71, CN=modgpt (cert-manager internal-ca-issuer — browsers already trust it, because the modgpt root CA is already on client machines)

RGW terminates its own TLS with a cert chained to the Aegis CA. Presigned URLs are fetched directly by the user's browser, which validates that cert against the OS/browser trust store — and fails, because the Aegis root CA isn't installed there. verify_ssl: false only affects the pod→RGW hop (boto3); there is no equivalent switch for the browser→RGW hop (a fetch/XHR to a presigned URL has no click-through and no app-settable "skip verify"). So the client must actually trust the CA.

Solution — distribute the Aegis root CA

The Aegis root CA is a self-signed root (O=Aegis CL1.SQ4 CA, CN=Aegis CL1.SQ4 CA Root CA, CA:TRUE pathlen:1, valid 2026-05-20 → 2036-05-17). RGW serves the full chain (leaf + intermediate), so trusting the root alone is sufficient — the browser builds leaf → intermediate → root and validates.

Obtain / verify the root CA

# Confirm RGW ships the intermediate (so root-only trust is enough)
openssl s_client -connect s3.cl1.sq4.aegis.internal:443 -showcerts </dev/null 2>/dev/null \
  | grep -E "^ *[0-9]+ s:|^ *i:"
# Expect entry 0 (leaf) AND entry 1 (…Intermediate CA, issued by …Root CA)

Save the Aegis root CA PEM to aegis-root-ca.crt. Verify: 

openssl x509 -in aegis-root-ca.crt -noout -subject -issuer -dates
# subject == issuer == O=Aegis CL1.SQ4 CA, CN=Aegis CL1.SQ4 CA Root CA  → self-signed root

Install on a client (macOS — Chrome/Safari)

Chrome and Safari use the macOS System keychain: 

sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain aegis-root-ca.crt
Fully quit (⌘Q) and reopen the browser. Verify trust without -k: 
curl -sw "%{http_code}\n" https://s3.cl1.sq4.aegis.internal -o /dev/null   # HTTP code, no cert error

Firefox uses its own NSS store (ignores the keychain): Settings → Privacy & Security → Certificates → View Certificates → Authorities → Import → select the .crt → tick "Trust this CA to identify websites."

Linux clients

sudo cp aegis-root-ca.crt /usr/local/share/ca-certificates/aegis-root-ca.crt   # Debian/Ubuntu
sudo update-ca-certificates
# RHEL/Fedora: cp to /etc/pki/ca-trust/source/anchors/ && sudo update-ca-trust

Org-wide rollout — this is the part that makes it a real fix, not a one-machine unblock

Trust must land on every current and future client machine, via the same distribution channel that already pushed the modgpt root CA (MDM / GPO / imaging / manual runbook — that's why ask.mod.auh1.dev.dir is trusted today). Add the Aegis root CA to that same bundle. A per-machine security add-trusted-cert unblocks one tester but does not fix other users.

Why this is acceptable here (documented deviation)

Presenting a single cert-manager-issued cert (the appendix approach) would avoid distributing a second CA, but it needs a Ceph-side hostname registration that's owned by another team. Given that constraint, trusting the existing Aegis root CA — which is already the production PKI for the cl1.sq4.aegis.internal zone — is the pragmatic, correct choice. It reuses the infra's own CA rather than overriding infra DNS/hostnames from the app side.

Appendix — rejected alternative: cert-manager ingress

Investigated and NOT deployed — kept for the record

This approach routes s3.mod.auh1.dev.dir through the cluster ingress with a cert-manager cert. It was abandoned because RGW (managed by a separate team, no access) does not recognize s3.mod.auh1.dev.dir as one of its hostnames, so it mis-parses the Host header as a bucket name and returns NoSuchBucket — even for path-style requests. Fixing that needs a Ceph-side radosgw-admin zonegroup modify --hostnames=…,s3.mod.auh1.dev.dir && period update --commit, which we could not perform. The Aegis-CA-trust solution above was implemented instead. The Git resources (manifests/ceph-rgw/, the mod-auh1-dev-ask.ask.s3-ingress Application) were removed.

Route s3.mod.auh1.dev.dir through the cluster ingress (same pattern as the GitLab alias, 10-apphaproxy Step 9.9):

Browser
  │  https://s3.mod.auh1.dev.dir/<bucket>/<key>?X-Amz-...   (presigned, host=s3.mod.auh1.dev.dir)
  ▼
DNS s3.mod.auh1.dev.dir → 100.115.1.51/.52/.53 (masters, same target as ask.mod.auh1.dev.dir)
  ▼
rke2-ingress-nginx (hostPort :443)
  │  • terminates TLS with cert-manager cert for s3.mod.auh1.dev.dir  (issuer O=ai71,CN=modgpt — TRUSTED)
  │  • preserves Host: s3.mod.auh1.dev.dir  (required for AWS SigV4 — see below)
  ▼  re-encrypt (backend-protocol HTTPS, proxy-ssl-verify off — RGW's own cert not trusted on this hop, fine, internal L2)
Service ceph-rgw-external (no selector) → Endpoints 100.115.1.130:443
  ▼
Ceph RGW  (validates SigV4 using received Host header)

Why the Host header must be preserved end-to-end (AWS SigV4)

Presigned URLs are signed with X-Amz-SignedHeaders=content-type;host. boto3 computes the signature over the endpoint host. RGW recomputes it using the Host header it receives — the two must match or RGW returns SignatureDoesNotMatch. Therefore:

  1. storage.s3.endpoint_url must be https://s3.mod.auh1.dev.dir so boto3 signs with that host.
  2. nginx-ingress must forward Host: s3.mod.auh1.dev.dir unchanged to RGW (its default — do not set upstream-vhost).
  3. We use path-style addressing (addressing_style: path), so RGW takes the bucket from the URL path, not the hostname — rgw_dns_name does not need to include s3.mod.auh1.dev.dir.

This is why the endpoint_url revert (back to s3.cl1.sq4.aegis.internal) must be re-applied to s3.mod.auh1.dev.dir as part of this fix — but only after the ingress + DNS are in place.

Ordering (non-disruptive → cutover)

Steps 1–2 add new resources and can be fully validated without affecting anything currently working (pods keep hitting RGW directly until DNS changes). Step 3 (endpoint_url) + Step 4 (DNS) are the cutover. Pods never break: with verify_ssl: false they don't care about the cert on either path.

Step 1 — Create the ingress + backend Service/Endpoints via ArgoCD (GitOps, non-disruptive)

Confirm the ClusterIssuer name first

kubectl get clusterissuer
# expect: internal-ca-issuer  (the O=ai71,CN=modgpt CA, Ready). If the name differs, edit the manifest.

Rather than a manual kubectl apply, this is delivered as an ArgoCD Application so it's reconciled declaratively (production mandate). Two files are added to the mod-ask-devops repo:

File Purpose
manifests/ceph-rgw/s3-rgw-ingress.yaml The raw Service (no selector) + Endpoints → 100.115.1.130:443 + Ingress (cert-manager annotation).
environment/auh1/dev/02-argocd-application/ask/argocd-application-s3-ingress.yaml Leaf ArgoCD Application (mod-auh1-dev-ask.ask.s3-ingress) with a directory source pointing at manifests/ceph-rgw.

The parent app-of-apps mod-auh1-dev-ask.ask recursively syncs every *.yaml in .../02-argocd-application/ask/, so committing the new Application file makes ArgoCD auto-create and sync it — no manual argocd app create.

cd mod-ask-devops
git add manifests/ceph-rgw/s3-rgw-ingress.yaml \
        environment/auh1/dev/02-argocd-application/ask/argocd-application-s3-ingress.yaml
git commit -m "feat(s3): cert-manager-trusted ingress for s3.mod.auh1.dev.dir (Ceph RGW)"
git push origin main

Watch ArgoCD pick it up (the parent reconciles within a minute, or force a refresh):

kubectl get application -n argocd mod-auh1-dev-ask.ask.s3-ingress
kubectl get svc,ep,ingress -n ask -l app.kubernetes.io/part-of=ask | grep -E 'ceph-rgw|s3-alias'

Wait for cert-manager to issue the cert (usually < 30s after the Ingress syncs):

kubectl get certificate s3-mod-tls -n ask -w
# READY must become True
kubectl get secret s3-mod-tls -n ask   # tls.crt / tls.key present

Step 1 verify — before any DNS change

Hit the ingress path directly on a master node, faking DNS with --resolve. This proves the trusted cert is served and RGW answers, without repointing DNS yet:

# Trusted cert now served for s3.mod.auh1.dev.dir via the ingress on a master
curl -v --resolve s3.mod.auh1.dev.dir:443:100.115.1.51 https://s3.mod.auh1.dev.dir 2>&1 \
  | grep -E "issuer:|subject:|SSL certificate verify|< HTTP"
# Expect issuer: O=ai71, CN=modgpt  (NOT Aegis) and an S3 XML/HTTP response from RGW

If the issuer shows O=ai71, CN=modgpt and RGW responds, Step 1 is good. Nothing is live yet — current traffic still uses the direct 100.115.1.130 path.

Step 2 — Re-apply endpoint_url = s3.mod.auh1.dev.dir (3 files)

So boto3 signs presigned URLs with the new host. (This reverts the earlier revert — now it's backed by a real trusted-cert path.)

Repo / file Line
mod-ask-knowledge-management-service/helm/environments/values-mod-auh1-dev-ask.yaml endpoint_url = "https://s3.mod.auh1.dev.dir/"
mod-ask-core-service/helm/environments/values-mod-auh1-dev-ask.yaml endpoint_url = "https://s3.mod.auh1.dev.dir"
mod-ask-devops/applications/core-service/values-mod-auh1-dev-ask.yaml endpoint_url = "https://s3.mod.auh1.dev.dir" (overlay wins)

Keep verify_ssl = false (the pod→ingress→RGW hop still involves RGW's own untrusted backend cert; proxy-ssl-verify: off on the ingress handles the ingress→RGW leg, and the cert is now trusted on the browser leg — but boto3 in-pod still traverses to RGW). Commit, push, ArgoCD sync, then restart:

kubectl rollout restart deploy -n ask -l app.kubernetes.io/instance=knowledge-management-service
kubectl rollout restart deploy -n ask -l app.kubernetes.io/instance=core-service

Pods keep working throughout

Even before Step 4, with endpoint_url=s3.mod.auh1.dev.dir and DNS still →100.115.1.130, pods resolve via BIND to the RGW IP directly and boto3 connects with verify=False — functional. After Step 4 they traverse the ingress instead — also functional (cert now trusted, Host preserved).

Step 3 — (skipped) no CoreDNS change needed

Verified: the CoreDNS hosts block does not contain any s3.* name, and fallthrough sends s3.mod.auh1.dev.dir to BIND. So pod + browser resolution are both governed by the single BIND record changed in Step 4 — no CoreDNS edit required.

Step 4 — Repoint DNS s3.mod.auh1.dev.dir → ingress (cutover)

Change the s3 record in the mod.auh1.dev.dir zone from the direct RGW IP to the same target ask.mod.auh1.dev.dir uses (the masters running nginx-ingress). Confirm that target first:

dig +short ask.mod.auh1.dev.dir @100.115.1.21   # expect 100.115.1.51 / .52 / .53

On primary BIND 100.115.1.21:

ZONE_FILE=/var/named/mod.auh1.dev.dir.zone
sudo cp $ZONE_FILE ${ZONE_FILE}.bak.$(date +%Y%m%d-%H%M)

# Replace the single  s3  IN  A  100.115.1.130  line with 3 A records to the masters
sudo sed -i '/^s3[[:space:]].*100\.115\.1\.130/d' $ZONE_FILE
sudo tee -a $ZONE_FILE > /dev/null <<'EOF'
s3          IN  A  100.115.1.51
s3          IN  A  100.115.1.52
s3          IN  A  100.115.1.53
EOF

# Bump the SOA serial (must increase), then:
sudo named-checkzone mod.auh1.dev.dir $ZONE_FILE
sudo systemctl reload named

Replicate to secondary BIND 100.115.1.22 (retransfer if slave, else copy the zone file — see 05b-dns.md Step 5b.4), then verify both:

dig +short s3.mod.auh1.dev.dir @100.115.1.21   # expect .51 .52 .53
dig +short s3.mod.auh1.dev.dir @100.115.1.22   # expect .51 .52 .53

GitLab's S3 is unaffected

GitLab reaches Ceph via a different endpoint (100.115.1.10:7480, see ceph-infrastructure.md) — not s3.mod.auh1.dev.dir. This change only moves the ASK apps' browser-facing S3 name onto the ingress path.

Step 5 — Validate end-to-end

# 1. Trusted cert now served on the real name (no --resolve trick)
curl -v https://s3.mod.auh1.dev.dir 2>&1 | grep -E "issuer:|SSL certificate verify ok|< HTTP"
#    issuer: O=ai71, CN=modgpt   +   SSL certificate verify ok

# 2. Pod round-trip still green (KMS ingestion worker, boto3)
#    (same PUT/GET/LIST/DELETE script as 40-kms-s3-wiring Step 5B, endpoint now s3.mod.auh1.dev.dir)
  1. Browser: re-run a KMS / core-service file upload. The presigned PUT to https://s3.mod.auh1.dev.dir/... should now succeed — no ERR_CERT_AUTHORITY_INVALID, no Status code: (null). (CORS was already applied to both buckets for origin https://ask.mod.auh1.dev.dir — unchanged by this fix.)

Rollback

Fastest revert is DNS-only — repoint s3.mod.auh1.dev.dir back to 100.115.1.130 on both BIND servers (bump serial, reload). That restores the pre-fix behaviour (browser uploads broken again, pods fine) without touching the app config. Then, if needed, revert the Step 2 endpoint_url commits and delete the ceph-rgw-s3-alias ingress / ceph-rgw-external Service+Endpoints.

GitOps

The ingress + Service + Endpoints (Step 1) should live in the devops repo under manifests/ (or a cluster-addons Application) so they're reconciled by ArgoCD, not left as a manual kubectl apply. Same treatment the GitLab alias should get. Commit before/after applying so the declared state matches the cluster.