Ceph Infrastructure¶

Reference page for the Ceph storage cluster used in the Core42 deployment. This is an external Ceph cluster — not managed by Rook. The Kubernetes cluster connects to it via the Ceph CSI drivers (ceph-csi-rbd and ceph-csi-cephfs namespaces).

Cluster Overview¶

Item	Detail
Cluster ID	`b8085298-5426-11f1-bd37-5000e63565cc`
Management	External (not Rook-managed)
K8s integration	Ceph CSI drivers (RBD + CephFS)
Object storage	Ceph RGW (S3-compatible) via `https://s3.mod.auh1.dev.dir` (HAProxy VIP)
RGW HA	HAProxy + Keepalived — VIP floats across all 3 MON nodes

Monitor Nodes¶

Ceph monitors (MON) handle cluster state and coordination. RGW (object storage gateway) also runs on these nodes, fronted by HAProxy + Keepalived for HA.

Node	IP (Network 1)	MON Port	RGW Port	Role
VIP	`100.115.1.10`	—	`7480`	Keepalived floating IP — use this in all secrets/configs
ceph-mon-1	`100.115.1.11`	`6789`	`7480`	MON + RGW backend
ceph-mon-2	`100.115.1.12`	`6789`	`7480`	MON + RGW backend
ceph-mon-3	`100.115.1.13`	`6789`	`7480`	MON + RGW backend

Always use the DNS endpoint — never hardcode a node IP

All Kubernetes secrets, application configs, and S3 connection strings must reference https://s3.mod.auh1.dev.dir (DNS → VIP 100.115.1.10:7480), not individual node IPs. If a MON node goes down, HAProxy health-checks detect it within seconds and stop routing to it. The VIP floats to a surviving node via Keepalived — zero downtime for consumers.

Storage Services¶

Block Storage (Ceph RBD)¶

Used for RWO (ReadWriteOnce) PVCs — databases, Gitaly, Redis.

Item	Detail
StorageClass	`csi-rbd-sc`
Provisioner	`rbd.csi.ceph.com`
Access Mode	`ReadWriteOnce` (RWO)
Reclaim Policy	`Delete`
Volume Binding	`Immediate`
Volume Expansion	✅ Supported
Default	✅ Yes — used when no `storageClass` specified

# Verify
kubectl get storageclass csi-rbd-sc

Filesystem Storage (CephFS)¶

Used for RWX (ReadWriteMany) PVCs — shared mounts across multiple pods.

Item	Detail
StorageClass	`csi-cephfs-sc`
Provisioner	`cephfs.csi.ceph.com`
Access Mode	`ReadWriteMany` (RWX)
Reclaim Policy	`Delete`
Volume Binding	`Immediate`
Volume Expansion	✅ Supported
Default	❌ No

# Verify
kubectl get storageclass csi-cephfs-sc

Object Storage (Ceph RGW — S3 compatible)¶

Used for GitLab uploads, LFS, artifacts, packages, registry blobs, backups.

Item	Detail
Protocol	S3-compatible (AWS Signature v4)
Endpoint (consumer)	`https://s3.mod.auh1.dev.dir` — use this in all configs
Backend nodes	`100.115.1.11:7481`, `100.115.1.12:7481`, `100.115.1.13:7481` (HAProxy backends)
VIP	`100.115.1.10:7480` (HAProxy + Keepalived)
TLS	✅ HTTPS via wildcard cert — `s3.mod.auh1.dev.dir` + `s3.cl1.sq4.aegis.internal` explicit SANs
Path style	`true` (required — no virtual-hosted style)
Region	`us-east-1` (any value works — Ceph ignores region)

CSI Driver Namespaces¶

# Check CSI driver pods
kubectl get pods -n ceph-csi-rbd
kubectl get pods -n ceph-csi-cephfs

Secrets¶

Namespace	Secret	Purpose
`ceph-csi-rbd`	`csi-rbd-secret`	RBD provisioner credentials
`ceph-csi-cephfs`	`csi-cephfs-secret`	CephFS provisioner credentials

ConfigMaps¶

# View Ceph cluster config (monitors + cluster ID)
kubectl get configmap ceph-csi-config -n ceph-csi-rbd -o jsonpath='{.data}' | python3 -m json.tool

Output:

{
    "cluster-mapping.json": "[]",
    "config.json": "[{
        \"clusterID\": \"b8085298-5426-11f1-bd37-5000e63565cc\",
        \"monitors\": [
            \"100.115.1.11:6789\",
            \"100.115.1.12:6789\",
            \"100.115.1.13:6789\"
        ]
    }]"
}

RGW High Availability — HAProxy + Keepalived¶

Architecture¶

All consumers (K8s, apps, secrets)
        │
        ▼
   VIP: 100.115.1.10:7480   ← Keepalived floating IP
        │
   HAProxy (on each MON node — active/active)
        ├── 100.115.1.11:7480  ← health-checked every 2s
        ├── 100.115.1.12:7480  ← health-checked every 2s
        └── 100.115.1.13:7480  ← health-checked every 2s

Keepalived manages the VIP — elects a MASTER node; VIP floats to BACKUP within ~3s if MASTER fails
HAProxy runs on all 3 nodes — load balances RGW requests, removes dead backends automatically
Failover time: < 5 seconds (Keepalived election + HAProxy health-check interval)

Step — Install HAProxy + Keepalived on all 3 MON nodes¶

Run on: ceph-mon-1, ceph-mon-2, ceph-mon-3

Execute on each node unless stated otherwise.

Step 1 — Install packages¶

sudo apt-get update && sudo apt-get install -y haproxy keepalived

Step 2 — Configure HAProxy¶

Write the same config on all 3 nodes:

sudo tee /etc/haproxy/haproxy.cfg > /dev/null << 'EOF'
global
    log /dev/log local0
    maxconn 4096
    user haproxy
    group haproxy
    daemon

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5s
    timeout client  30s
    timeout server  30s

frontend rgw_frontend
    bind *:7480
    default_backend rgw_backends

backend rgw_backends
    balance roundrobin
    option  httpchk GET /
    http-check expect status 200
    server rgw1 100.115.1.11:7480 check inter 2s fall 2 rise 2
    server rgw2 100.115.1.12:7480 check inter 2s fall 2 rise 2
    server rgw3 100.115.1.13:7480 check inter 2s fall 2 rise 2

listen stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    stats auth admin:admin
EOF

HAProxy listens on 7480

HAProxy takes over port 7480 on each node. The local Ceph RGW daemon must listen on a different port (e.g. 7481) so HAProxy can proxy to it. Update rgw frontends in ceph.conf or via ceph config set:

ceph config set client.rgw.* rgw_frontends "beast port=7481"

Then update HAProxy backend ports to 7481.

Step 3 — Configure Keepalived¶

ceph-mon-1 (MASTER):

vrrp_script    } vrrp_instance

href="#__codelineno-9-1">sudo tee /etc/keepalived/keepalived.conf > /dev/null << 'EOF' chk_haproxy { script "kill -0 $(cat /var/run/haproxy/haproxy.pid)" interval 2 weight -20 span> RGW_VIP { state MASTER interface eth0 # adjust to your NIC name virtual_router_id 51 priority 110 advert_int 1 authentication { auth_type PASS auth_pass ceph-rgw } virtual_ipaddress { 100.115.1.10/24 } track_script { chk_haproxy } } EOF

ceph-mon-2 (BACKUP, priority 100):

# Same as above but: state BACKUP, priority 100
sudo sed 's/state MASTER/state BACKUP/; s/priority 110/priority 100/' \
  /etc/keepalived/keepalived.conf | sudo tee /etc/keepalived/keepalived.conf

ceph-mon-3 (BACKUP, priority 90):

# state BACKUP, priority 90
sudo sed 's/state MASTER/state BACKUP/; s/priority 110/priority 90/' \
  /etc/keepalived/keepalived.conf | sudo tee /etc/keepalived/keepalived.conf

Step 4 — Enable and start services¶

# On all 3 nodes
sudo systemctl enable --now haproxy keepalived
sudo systemctl status haproxy keepalived

Step 5 — Verify VIP is active¶

# On ceph-mon-1 (should be MASTER)
ip addr show | grep 100.115.1.10
# Expected: inet 100.115.1.10/24 scope global secondary eth0

Step 6 — Test VIP connectivity from bastion¶

curl -s --connect-timeout 5 -o /dev/null -w "%{http_code}" http://100.115.1.10:7480
# Expected: 200

Step 7 — Test failover¶

# Simulate ceph-mon-1 failure (stop keepalived on MASTER)
ssh ceph-mon-1 "sudo systemctl stop keepalived"

# VIP should move to ceph-mon-2 within ~3 seconds
sleep 5
curl -s --connect-timeout 5 -o /dev/null -w "%{http_code}" http://100.115.1.10:7480
# Expected: still 200

# Restore
ssh ceph-mon-1 "sudo systemctl start keepalived"

Update Kubernetes Secrets to use VIP¶

After VIP is live, update all secrets to use 100.115.1.10 instead of individual node IPs:

export AWS_ACCESS_KEY_ID="<your-access-key>"
export AWS_SECRET_ACCESS_KEY="<your-secret-key>"
export RGW_ENDPOINT="http://100.115.1.10:7480"   # ← VIP, not a node IP

# Recreate GitLab S3 secret
kubectl delete secret gitlab-s3-connection -n gitlab --ignore-not-found
kubectl create secret generic gitlab-s3-connection \
  -n gitlab \
  --from-literal=connection="$(cat << CONNEOF
provider: AWS
region: us-east-1
aws_access_key_id: ${AWS_ACCESS_KEY_ID}
aws_secret_access_key: ${AWS_SECRET_ACCESS_KEY}
aws_signature_version: 4
host: 100.115.1.10
endpoint: ${RGW_ENDPOINT}
path_style: true
CONNEOF
)"

# Recreate GitLab registry storage secret
kubectl delete secret gitlab-registry-storage -n gitlab --ignore-not-found
kubectl create secret generic gitlab-registry-storage \
  -n gitlab \
  --from-literal=config="$(cat << REGEOF
s3:
  bucket: gitlab-registry
  accesskey: ${AWS_ACCESS_KEY_ID}
  secretkey: ${AWS_SECRET_ACCESS_KEY}
  regionendpoint: ${RGW_ENDPOINT}
  region: us-east-1
  pathstyle: true
  secure: false
REGEOF
)"

kubectl get secret gitlab-s3-connection gitlab-registry-storage -n gitlab
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY RGW_ENDPOINT

S3 / RGW Quick Reference¶

Test connectivity¶

# Anonymous test via VIP — should return S3 XML
curl -s http://100.115.1.10:7480

# Authenticated test (requires access_key + secret_key)
export AWS_ACCESS_KEY_ID="<access-key>"
export AWS_SECRET_ACCESS_KEY="<secret-key>"

aws s3 ls --endpoint-url http://100.115.1.10:7480 --no-verify-ssl

unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY

GitLab S3 buckets¶

Bucket	Used for
`gitlab-artifacts`	CI/CD artifacts
`gitlab-lfs`	Git LFS objects
`gitlab-uploads`	User uploads / attachments
`gitlab-packages`	Package registry (PyPI, Helm, etc.)
`gitlab-registry`	Container registry blobs
`gitlab-backups`	GitLab backups
`gitlab-backups-tmp`	Backup staging

# List all GitLab buckets
export AWS_ACCESS_KEY_ID="<access-key>"
export AWS_SECRET_ACCESS_KEY="<secret-key>"

aws s3 ls --endpoint-url http://100.115.1.11:7480 --no-verify-ssl | grep gitlab

unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY

S3 connection parameters for GitLab¶

Use VIP — not a node IP

Always use 100.115.1.10 (VIP). Never hardcode 100.115.1.11/12/13.

provider: AWS
region: us-east-1
aws_access_key_id: <access-key>
aws_secret_access_key: <secret-key>
aws_signature_version: 4
host: 100.115.1.10
endpoint: http://100.115.1.10:7480
path_style: true

Bucket CORS — browser presigned uploads (ASK)¶

KMS and core-service upload files by handing the browser a presigned S3 URL. The browser (origin https://ask.mod.auh1.dev.dir) then PUTs directly to the RGW endpoint — a cross-origin request. Without a CORS policy on the bucket, the browser blocks the preflight/response and the upload fails. So each ASK bucket needs a CORS policy allowing the frontend origin.

Applied to the ASK buckets

Applied to ask-knowledge-management and ask-core-service (the buckets that receive browser presigned uploads). GitLab buckets don't need CORS (server-to-server only).

Policy (/tmp/cors.json):

{
  "CORSRules": [
    {
      "AllowedOrigins": ["https://ask.mod.auh1.dev.dir"],
      "AllowedMethods": ["GET", "PUT", "POST", "DELETE", "HEAD"],
      "AllowedHeaders": ["*"],
      "ExposeHeaders": ["ETag"],
      "MaxAgeSeconds": 3600
    }
  ]
}

Apply (from bastion — use each bucket's own credentials from Vault, secret/ask/<service>/auh1-dev; the service user has PutBucketCors on its own bucket):

export AWS_ACCESS_KEY_ID=$(vault kv get -field=STORAGE__S3__ACCESS_KEY_ID secret/ask/knowledge-management-service/auh1-dev)
export AWS_SECRET_ACCESS_KEY=$(vault kv get -field=STORAGE__S3__SECRET_ACCESS_KEY secret/ask/knowledge-management-service/auh1-dev)

aws s3api put-bucket-cors \
  --bucket ask-knowledge-management \
  --cors-configuration file:///tmp/cors.json \
  --endpoint-url https://s3.cl1.sq4.aegis.internal --no-verify-ssl

# repeat for ask-core-service with secret/ask/core-service/auh1-dev creds
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY

put-bucket-cors is silent on success (the InsecureRequestWarning from --no-verify-ssl is harmless — the RGW cert is self-signed, see Fix S3 Browser Cert Trust).

Read-back has a permission asymmetry

The service user can PutBucketCors on its own bucket but not GetBucketCors (RGW gates the read on bucket ownership, not the policy grant) — get-bucket-cors returns AccessDenied even though the write succeeded. To verify, use the bucket-owner / Ceph S3 admin credential, or radosgw-admin on a Ceph node. A clean put-bucket-cors response is sufficient confirmation.

CORS is necessary but not sufficient for browser uploads

A CORS policy only matters after the TLS handshake succeeds. If the browser can't trust RGW's cert, the upload fails with ERR_CERT_AUTHORITY_INVALID before CORS is ever evaluated — that's a separate fix (trust the Ceph CA), see Fix S3 Browser Cert Trust.

Network Topology¶

Kubernetes cluster nodes
        │
        │  CSI RBD (block)  →  100.115.1.11/12/13 : 6789 (MON)
        │  CSI CephFS        →  100.115.1.11/12/13 : 6789 (MON)
        │  S3 / RGW          →  100.115.1.11/12/13 : 7480 (RGW)
        │
        ▼
External Ceph Cluster
  ├── ceph-mon-1  100.115.1.11  MON + RGW
  ├── ceph-mon-2  100.115.1.12  MON + RGW
  └── ceph-mon-3  100.115.1.13  MON + RGW

Troubleshooting¶

Symptom	Cause	Fix
PVC stuck `Pending`	CSI driver can't reach monitors	Check connectivity: `nc -zv 100.115.1.11 6789`
S3 `403 Forbidden`	Wrong access/secret key	Verify credentials with `aws s3 ls`
S3 `Connection refused`	RGW not reachable	Test: `curl -s http://100.115.1.11:7480`
`path_style` errors	Virtual-hosted bucket URL not supported	Ensure `path_style: true` in S3 config
GitLab registry crash	`gitlab-registry` bucket missing	Pre-create bucket — see Phase 2 Step 2.4