Phase 2 — Storage Validation¶

Validate Ceph block storage (RBD) and Ceph object storage (RGW/S3) before deploying any applications. Both must be confirmed working before GitLab install.

Environment¶

Full Ceph reference

See Ceph Infrastructure → for full details on monitors, RGW endpoints, CSI config, and troubleshooting.

Component	Detail
Ceph RBD (block)	StorageClass `csi-rbd-sc` — default, used for PVCs
Ceph CephFS (filesystem)	StorageClass `csi-cephfs-sc` — used for RWX
Ceph RGW (S3 object)	`http://100.115.1.11:7480` (also `.12`, `.13` — all same port)
Ceph monitors	`100.115.1.11:6789`, `100.115.1.12:6789`, `100.115.1.13:6789`
Cluster ID	`b8085298-5426-11f1-bd37-5000e63565cc`

Step 2.1 — Verify storage classes¶

kubectl get storageclass

Expected output (verified ✅):

NAME                   PROVISIONER           RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
csi-cephfs-sc          cephfs.csi.ceph.com   Delete          Immediate           true                   13d
csi-rbd-sc (default)   rbd.csi.ceph.com      Delete          Immediate           true                   13d

Storage Class	Type	Access Mode	Use for
`csi-rbd-sc`	Ceph RBD block	RWO	Databases, Gitaly, Redis
`csi-cephfs-sc`	CephFS filesystem	RWX	Shared mounts

Step 2.2 — Verify Ceph RGW endpoints¶

Ceph RGW (S3-compatible object storage) runs on port 7480 on all 3 monitor nodes, fronted by a HAProxy + Keepalived VIP at 100.115.1.10.

Test VIP first (primary — this is what all apps use):

result=$(curl -s --connect-timeout 3 -o /dev/null -w "%{http_code}" https://s3.mod.auh1.dev.dir)
echo "VIP 100.115.1.10 port 7480: HTTP ${result}"
# Expected: HTTP 200

Test individual nodes (for HA validation):

for ip in 100.115.1.11 100.115.1.12 100.115.1.13; do
  echo "=== $ip ==="
  result=$(curl -s --connect-timeout 3 -o /dev/null -w "%{http_code}" http://${ip}:7481)
  echo "  port 7481: HTTP ${result}"
done

Port 7481 on individual nodes

Once HAProxy is deployed, local Ceph RGW daemons listen on 7481 (HAProxy takes 7480). The VIP 100.115.1.10:7480 is the only endpoint consumers should use.

Expected output (verified ✅):

VIP 100.115.1.10 port 7480: HTTP 200
=== 100.115.1.11 ===
  port 7481: HTTP 200
=== 100.115.1.12 ===
  port 7481: HTTP 200
=== 100.115.1.13 ===
  port 7481: HTTP 200

Step 2.3 — Test S3 connectivity with credentials¶

Prerequisites

You need the Ceph RGW access_key and secret_key for your S3 user. These are provided by the Ceph administrator or retrieved from the Ceph cluster.

export AWS_ACCESS_KEY_ID="<your-access-key>"
export AWS_SECRET_ACCESS_KEY="<your-secret-key>"
export RGW_ENDPOINT="https://s3.mod.auh1.dev.dir"   # VIP — HAProxy + Keepalived across all 3 RGW nodes

Step 2.3a — List buckets (confirms auth works)¶

aws s3 ls --endpoint-url ${RGW_ENDPOINT} --no-verify-ssl

Step 2.3b — Create test bucket¶

aws s3 mb s3://test-connectivity \
  --endpoint-url ${RGW_ENDPOINT} --no-verify-ssl
# Expected: make_bucket: test-connectivity

Step 2.3c — Upload a test file¶

echo "rgw test" | aws s3 cp - s3://test-connectivity/test.txt \
  --endpoint-url ${RGW_ENDPOINT} --no-verify-ssl

Step 2.3d — Verify file is stored¶

aws s3 ls s3://test-connectivity \
  --endpoint-url ${RGW_ENDPOINT} --no-verify-ssl
# Expected: 2026-06-03 22:27:49  9  test.txt

Step 2.3e — Cleanup test bucket¶

aws s3 rb s3://test-connectivity --force \
  --endpoint-url ${RGW_ENDPOINT} --no-verify-ssl
# Expected:
#   delete: s3://test-connectivity/test.txt
#   remove_bucket: test-connectivity

Step 2.3f — Unset credentials¶

unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY RGW_ENDPOINT

Full test verified ✅ on bastion1 — bucket create, upload, list, delete all successful.

Step 2.4 — Pre-create GitLab S3 buckets¶

Create all buckets GitLab needs before installing GitLab. GitLab does not auto-create S3 buckets on startup — missing buckets cause CrashLoopBackOff.

export AWS_ACCESS_KEY_ID="<your-access-key>"
export AWS_SECRET_ACCESS_KEY="<your-secret-key>"
export RGW_ENDPOINT="https://s3.mod.auh1.dev.dir"   # VIP — HAProxy + Keepalived across all 3 RGW nodes

for bucket in \
  gitlab-artifacts \
  gitlab-lfs \
  gitlab-uploads \
  gitlab-packages \
  gitlab-registry \
  gitlab-backups \
  gitlab-backups-tmp; do
  aws s3 mb s3://${bucket} \
    --endpoint-url ${RGW_ENDPOINT} --no-verify-ssl \
    2>/dev/null && echo "✅ created: ${bucket}" || echo "ℹ️  exists:  ${bucket}"
done

# Verify all buckets exist
echo ""
echo "=== All GitLab buckets ==="
aws s3 ls --endpoint-url ${RGW_ENDPOINT} --no-verify-ssl | grep gitlab

unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY RGW_ENDPOINT

Expected output:

✅ created: gitlab-artifacts
✅ created: gitlab-lfs
✅ created: gitlab-uploads
✅ created: gitlab-packages
✅ created: gitlab-registry
✅ created: gitlab-backups
✅ created: gitlab-backups-tmp

=== All GitLab buckets ===
2026-06-03  gitlab-artifacts
2026-06-03  gitlab-lfs
2026-06-03  gitlab-uploads
2026-06-03  gitlab-packages
2026-06-03  gitlab-registry
2026-06-03  gitlab-backups
2026-06-03  gitlab-backups-tmp

Step 2.5 — Create GitLab S3 Kubernetes secrets¶

These two secrets are required by the GitLab Helm chart before install.

Run from

bastion1 — any directory

export AWS_ACCESS_KEY_ID="<your-access-key>"
export AWS_SECRET_ACCESS_KEY="<your-secret-key>"
export RGW_ENDPOINT="https://s3.mod.auh1.dev.dir"   # VIP — HAProxy + Keepalived across all 3 RGW nodes

# Secret 1: S3 connection for uploads, LFS, artifacts, packages
kubectl create secret generic gitlab-s3-connection \
  -n gitlab \
  --from-literal=connection="$(cat << CONNEOF
provider: AWS
region: us-east-1
aws_access_key_id: ${AWS_ACCESS_KEY_ID}
aws_secret_access_key: ${AWS_SECRET_ACCESS_KEY}
aws_signature_version: 4
host: 100.115.1.10
endpoint: ${RGW_ENDPOINT}
path_style: true
CONNEOF
)"

# Secret 2: S3 config for GitLab container registry blobs
kubectl create secret generic gitlab-registry-storage \
  -n gitlab \
  --from-literal=config="$(cat << REGEOF
s3:
  bucket: gitlab-registry
  accesskey: ${AWS_ACCESS_KEY_ID}
  secretkey: ${AWS_SECRET_ACCESS_KEY}
  regionendpoint: ${RGW_ENDPOINT}
  region: us-east-1
  pathstyle: true
  secure: false
REGEOF
)"

# Verify
kubectl get secret gitlab-s3-connection gitlab-registry-storage -n gitlab

unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY RGW_ENDPOINT

Summary checklist¶

[x] Storage classes verified — csi-rbd-sc (default) and csi-cephfs-sc
[x] Ceph RGW reachable on all 3 nodes at port 7480
[x] S3 credentials tested — bucket create/upload/delete working
[ ] GitLab S3 buckets pre-created (7 buckets)
[ ] gitlab-s3-connection secret created in gitlab namespace
[ ] gitlab-registry-storage secret created in gitlab namespace

Next: Phase 3 — Registry Auth & imagePullSecret →