Ceph S3 Wiring — Object Storage for core-service

core-service uses S3-compatible object storage for two purposes:

1. User/org uploads — user avatars, organisation logos uploaded via the API
2. Static model icons — SVG logos for the LLM model catalog served to the frontend

In the air-gapped environment that's Ceph RGW (s3.cl1.sq4.aegis.internal), bucket ask-core-service. Credentials belong to RGW user ask-cs, stored in Vault.

How core-service uses S3

• config.storage.s3 drives the storage client: endpoint_url, access_key_id, secret_access_key, verify_ssl, bucket_name.
• access_key_id / secret_access_key are optional in the schema (SecretStr | None) — the app boots without them; uploads fail only when attempted.
• The cdn_base_url config value is prepended to every object key before returning it to the frontend. Model icon URLs are built as: {cdn_base_url}/static/logos/llm_models/{logo_file_name} (model_service._with_logo_url()).
• Keys arrive as STORAGE__S3__ACCESS_KEY_ID / STORAGE__S3__SECRET_ACCESS_KEY env vars (Vault → ExternalSecret → core-service-secret → @format in the TOML).

---

Prerequisites

• [x] core-service deployed; storage.type = "s3", endpoint_url, region, bucket_name set
• [x] Ceph RGW reachable: getent hosts s3.cl1.sq4.aegis.internal → 100.115.1.130
• [x] Access to radosgw-admin (Rook toolbox pod or a Ceph node)
• [x] Vault reachable + unsealed
• [x] SVG logo files from assets/logos/ in the core-service repo

---

Step 1 — Create the RGW S3 user

On the Rook toolbox pod (or a Ceph node):

# User name convention: ask-cs (shared service account for ASK core-service storage)
radosgw-admin user create \
  --uid=ask-cs \
  --display-name="ASK Core Service Storage"

Copy access_key and secret_key from the "keys" array in the JSON output.

Production note

In production use an ObjectBucketClaim so Rook manages the bucket lifecycle via GitOps. For this air-gap environment, radosgw-admin + boto3 is the pragmatic path.

---

Step 2 — Create the bucket

AWS CLI v1 (installed on bastion) has a bug with Ceph RGW — s3 cp / s3 mb fail with argument of type 'NoneType' is not iterable against a non-existent bucket. Use boto3 instead.

Set env vars first:

export ENDPOINT=https://s3.cl1.sq4.aegis.internal
export BUCKET=ask-core-service
export AWS_ACCESS_KEY_ID=<access_key_from_step_1>
export AWS_SECRET_ACCESS_KEY=<secret_key_from_step_1>
# Proxy — S3 endpoint must bypass the corporate proxy
export http_proxy=http://10.96.137.37:3128
export https_proxy=http://10.96.137.37:3128
export no_proxy="s3.cl1.sq4.aegis.internal,localhost,127.0.0.1,.cluster.local"

Create the bucket:

python3 - << 'PYEOF'
import boto3, urllib3, os
urllib3.disable_warnings()
s3 = boto3.client("s3",
    endpoint_url=os.environ["ENDPOINT"],
    aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
    aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
    verify=False)
s3.create_bucket(Bucket=os.environ["BUCKET"])
print(f"created: {os.environ['BUCKET']}")
PYEOF

boto3 not installed?

export http_proxy=http://10.96.137.37:3128
export https_proxy=http://10.96.137.37:3128
pip3 install boto3 --user --quiet

Verify:

aws --endpoint-url $ENDPOINT --no-verify-ssl s3 ls | grep ask-core-service

---

Step 3 — Upload model icon SVGs

The frontend renders model icons from logo_path URLs returned by the /api/v1/models endpoint. The 5 SVGs needed for air-gap served models live in assets/logos/ of the core-service repo.

# Run from the directory containing the logos/ folder
for svg in claude openai qwen falcon llama; do
  aws s3 cp logos/${svg}.svg s3://$BUCKET/static/logos/llm_models/${svg}.svg \
    --endpoint-url $ENDPOINT --no-verify-ssl
done

Once the bucket exists, aws s3 cp works (the earlier NoneType bug only affects non-existent buckets).

Apply a bucket policy so the browser can fetch SVGs without credentials:

python3 - << 'PYEOF'
import boto3, urllib3, os, json
urllib3.disable_warnings()
s3 = boto3.client("s3",
    endpoint_url=os.environ["ENDPOINT"],
    aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
    aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
    verify=False)
policy = json.dumps({
    "Version": "2012-10-17",
    "Statement": [{
        "Sid": "PublicReadLogos",
        "Effect": "Allow",
        "Principal": "*",
        "Action": "s3:GetObject",
        "Resource": f"arn:aws:s3:::{os.environ['BUCKET']}/static/logos/*"
    }]
})
s3.put_bucket_policy(Bucket=os.environ["BUCKET"], Policy=policy)
print("bucket policy applied")
PYEOF

Also apply CORS so the browser can make cross-origin requests:

python3 - << 'PYEOF'
import boto3, urllib3, os
urllib3.disable_warnings()
s3 = boto3.client("s3",
    endpoint_url=os.environ["ENDPOINT"],
    aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
    aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
    verify=False)
s3.put_bucket_cors(Bucket=os.environ["BUCKET"], CORSConfiguration={
    "CORSRules": [{
        "AllowedOrigins": ["https://ask.mod.auh1.dev.dir"],
        "AllowedMethods": ["GET", "HEAD"],
        "AllowedHeaders": ["*"],
        "ExposeHeaders": ["ETag", "Content-Length", "Content-Type"],
        "MaxAgeSeconds": 3600
    }]
})
print("CORS applied")
PYEOF

Verify uploads:

aws --endpoint-url $ENDPOINT --no-verify-ssl s3 ls s3://$BUCKET/static/logos/llm_models/

Expected output — 5 SVGs:

claude.svg   falcon.svg   llama.svg   openai.svg   qwen.svg

SVG	Models covered
qwen.svg	qwen/qwen3-5-122b-a10b, Qwen/Qwen3-4B-Instruct-2507, embedding
openai.svg	openai/gpt-oss-120b
llama.svg	nvidia/llama-nemotron-rerank-1b-v2 (reranker)
falcon.svg	Falcon models
claude.svg	Anthropic models (cloud-only, icon still useful)

Missing icons — not a blocker

minimax.svg, xai.svg, zai.svg are referenced in models.yml but not present in the repo — those are cloud-only models not served in the air-gap Foundry. The frontend renders a fallback placeholder when logo_path returns a 404.

---

Step 4 — Store the credentials in Vault

Store both keys as fields under a single KV v2 path:

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
vault kv put secret/core-service/storage \
  access_key_id="<access_key_from_step_1>" \
  secret_access_key="<secret_key_from_step_1>"

KV v2 path naming

vault kv put secret/core-service/storage stores at the internal path secret/data/core-service/storage. In ExternalSecret use key: core-service/storage (ESO adds the data/ prefix automatically for KV v2).

Verify:

vault kv get secret/core-service/storage

Expected:

========== Secret Path ==========
secret/data/core-service/storage

====== Data ======
Key                Value
---                -----
access_key_id      <redacted>
secret_access_key  <redacted>

---

Step 4a — Update the Vault ESO policy

The eso-read policy must include the new secret/core-service/* path so ESO can read it. The policy lives in mod-ask-devops/applications/cluster-addons/vault/manifests/vault-bootstrap-config.yaml and has been updated in Git. Apply it live immediately (don't wait for ArgoCD):

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal

vault policy write eso-read - << 'POLICY'
path "secret/data/argocd/*"        { capabilities = ["read"] }
path "secret/data/apps/*"          { capabilities = ["read"] }
path "secret/data/harbor/*"        { capabilities = ["read"] }
path "secret/data/foundry"         { capabilities = ["read"] }
path "secret/data/core-service/*"  { capabilities = ["read"] }
path "secret/metadata/*"           { capabilities = ["read", "list"] }
POLICY

Verify:

vault policy read eso-read
# expected: all 6 paths present including secret/data/core-service/*

---

Step 5 — Add the keys to the ExternalSecret

In mod-ask-devops/applications/core-service/core-service-secrets.yaml, add two entries to the core-service-secret ExternalSecret — both the template.data mapping and the data ref:

  target:
    template:
      data:
        # ... existing keys ...
        STORAGE__S3__ACCESS_KEY_ID: "{{ .s3AccessKeyId }}"
        STORAGE__S3__SECRET_ACCESS_KEY: "{{ .s3SecretAccessKey }}"
  data:
    # ... existing refs ...
    - secretKey: s3AccessKeyId
      remoteRef:
        key: core-service/storage   # KV v2: omit the data/ prefix; ESO adds it automatically
        property: access_key_id
    - secretKey: s3SecretAccessKey
      remoteRef:
        key: core-service/storage
        property: secret_access_key

---

Step 6 — Reference the keys in the TOML

In both values files, add the two @format references under [mod-auh1-dev-ask.storage.s3] (the rest is already present):

[mod-auh1-dev-ask.storage.s3]
access_key_id = "@format {env[STORAGE__S3__ACCESS_KEY_ID]}"
secret_access_key = "@format {env[STORAGE__S3__SECRET_ACCESS_KEY]}"
region = "us-east-1"
bucket_name = "ask-core-service"
endpoint_url = "https://s3.cl1.sq4.aegis.internal"
verify_ssl = false

Air-gapped TLS

verify_ssl = false because the RGW endpoint cert is the internal CA. To verify properly, mount the air-gap root CA into the pod's trust store and set verify_ssl = true.

Commit + push both files, then re-sync (config-toml + secret both change → Reloader restarts the pod):

kubectl -n argocd patch application mod-auh1-dev-ask.ask.core-service --type merge \
  -p '{"operation":{"initiatedBy":{"username":"manual"},"sync":{"revision":"HEAD"}}}'

---

Step 7 — Validate

# keys present in the running pod
kubectl exec -n ask deploy/core-service -- printenv STORAGE__S3__ACCESS_KEY_ID | head -c 6; echo '…'
# exercise an upload (avatar / org logo) from the API, then check for S3 errors
kubectl logs -n ask deploy/core-service --tail=40 | grep -iE "s3|upload|avatar|NoSuchBucket|AccessDenied|endpoint"

Done when: an avatar/logo upload succeeds and the object lands in the bucket:

AWS_ACCESS_KEY_ID=<ak> AWS_SECRET_ACCESS_KEY=<sk> \
aws --endpoint-url https://s3.cl1.sq4.aegis.internal --no-verify-ssl s3 ls s3://ask-core-service --recursive | head

---

What's stored where

Value	Source of truth	Delivered via	Consumed as
S3 access key	Vault secret/data/core-service/storage → access_key_id	ExternalSecret (key: core-service/storage) → core-service-secret	STORAGE__S3__ACCESS_KEY_ID → @format → storage.s3.access_key_id
S3 secret key	Vault secret/data/core-service/storage → secret_access_key	ExternalSecret (key: core-service/storage) → core-service-secret	STORAGE__S3__SECRET_ACCESS_KEY → @format → storage.s3.secret_access_key
Bucket / endpoint / region	Git (TOML)	ConfigMap core-service-config-toml	storage.s3.*

Production / air-gap notes

• Scope the RGW user/bucket policy to least privilege (this one bucket only).
• Prefer ObjectBucketClaim (Option B) so bucket lifecycle is GitOps-managed by Rook.
• Rotate the S3 keys on a schedule; on rotation, re-run Step 2 only.