LiteLLM Deployment¶

LiteLLM is an OpenAI-compatible model proxy that routes requests to vLLM backends (Foundry), handles caching via Redis, tracks spend in PostgreSQL, and enforces model aliases / guardrails.

Stack: litellm-helm (in-repo chart) → ArgoCD Application → foundry namespace.

Architecture¶

ASK Services (core-service / assistant-service / KMS)
        │  OpenAI-compatible HTTP
        ▼
  LiteLLM (port 4000) ──► Redis (cache)
        │                  ──► PostgreSQL (spend logs / DB state)
        ▼
  vLLM backends (foundry namespace)
    - Qwen3-4B-Instruct-2507
    - gpt-oss-120b
    - Qwen3-5-122B-A10B (main + kms instance)
    - Qwen3-Embedding-0.6B
    - llama-nemotron-rerank-1b-v2

Production rationale:

LiteLLM decouples ASK services from vLLM pod addresses — services call one stable endpoint, routing and failover happen in LiteLLM.
Redis caching reduces redundant LLM calls for identical prompts.
store_prompts_in_spend_logs: true gives full request/response auditability in the DB.
LITELLM_LOCAL_MODEL_COST_MAP: "True" prevents outbound network calls at startup (air-gap requirement).

Prerequisites¶

Resource	Where
`foundry-secret` VSO secret	`foundry` namespace — contains `PROXY_MASTER_KEY`, `PROXY_SALT_KEY`, `VLLM_API_KEY`, `DATABASE_*`, `REDIS_PASSWORD`
`internal-ca-bundle` secret	`foundry` namespace — internal CA cert at `ca.pem`
`harbor-pull-secret`	`foundry` namespace — injected by ESO ClusterExternalSecret (label `harbor-pull-secret: "true"`)
CNPG PostgreSQL cluster	`foundry` namespace — database `foundry`
vLLM engine services	`foundry` namespace — one ClusterIP svc per model

Step 1 — Verify prerequisites¶

kubectl get secret foundry-secret -n foundry
kubectl get secret internal-ca-bundle -n foundry
kubectl get secret harbor-pull-secret -n foundry
kubectl get cluster -n foundry          # CNPG cluster
kubectl get svc -n foundry | grep vllm  # vLLM engine services

Step 2 — Mirror images and chart¶

Image¶

Image already lives in Harbor under mlops/berriai/litellm-database:v1.83.10-stable.patch.3-prisma.

Verify:

curl -sk https://harbor.cl1.sq4.aegis.internal/api/v2.0/projects/mlops/repositories \
  -H "Authorization: Basic $(cat ~/harbor-images/harbor-robot.json | jq -r '.auth')" \
  | jq '.[].name' | grep litellm

Helm chart¶

The litellm-helm chart is an in-repo chart at mlops-mod-production/helm-charts/litellm-helm/. It must be pushed to GitLab as a package.

Check if already published:

curl -sk "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/<MLOPS_PROJECT_ID>/packages?package_type=helm" \
  -H "PRIVATE-TOKEN: $(vault kv get -field=token secret/gitlab/migration-token)" \
  | jq '.[].name'

If not published, package and push from bastion:

cd /path/to/mlops-mod-production/helm-charts
helm package litellm-helm
curl -sk --request POST \
  --form "[email protected]" \
  -H "PRIVATE-TOKEN: <token>" \
  "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/<ID>/packages/helm/api/stable/charts"

Step 3 — Vault secrets¶

All secrets must exist in Vault KV v2 under secret/foundry/* before VSO can render foundry-secret.

Check what is already seeded:

vault kv get secret/foundry/litellm

Required keys:

Key	Description
`PROXY_MASTER_KEY`	LiteLLM admin master key (≥32 chars)
`PROXY_SALT_KEY`	Salt for key hashing (≥32 chars)
`VLLM_API_KEY`	API key accepted by vLLM engines
`DATABASE_USERNAME`	PostgreSQL user
`DATABASE_PASSWORD`	PostgreSQL password
`DATABASE_HOST`	PostgreSQL host (e.g. `foundry-pg-rw.foundry.svc.cluster.local`)
`REDIS_PASSWORD`	Redis password

Seed if missing:

vault kv put secret/foundry/litellm \
  PROXY_MASTER_KEY="$(openssl rand -hex 32)" \
  PROXY_SALT_KEY="$(openssl rand -hex 32)" \
  VLLM_API_KEY="<existing-key>" \
  DATABASE_USERNAME="litellm" \
  DATABASE_PASSWORD="$(openssl rand -hex 16)" \
  DATABASE_HOST="foundry-pg-rw.foundry.svc.cluster.local" \
  REDIS_PASSWORD="$(openssl rand -hex 16)"

Step 4 — Values overlay¶

File: applications/litellm/values-mod-auh1-dev-ask.yaml

Key differences from PP prod values: - replicaCount: 1 (dev; PP uses 2) - Resources reduced - Same model list as PP — vLLM services must exist in foundry namespace

Step 5 — ArgoCD Application¶

File: environment/auh1/dev/02-argocd-application/ask/argocd-application-litellm.yaml

Namespace: foundry (PSA restricted — securityContext required; already in chart).

Step 6 — Sync and verify¶

argocd app sync mod-auh1-dev-ask.ask.litellm --grpc-web
kubectl get pods -n foundry -l app.kubernetes.io/name=litellm-helm
kubectl logs -n foundry -l app.kubernetes.io/name=litellm-helm --tail=30

Health check:

kubectl exec -n foundry deploy/litellm-helm -- \
  curl -s http://localhost:4000/health/liveliness

Model list:

kubectl exec -n foundry deploy/litellm-helm -- \
  curl -s http://localhost:4000/models \
  -H "Authorization: Bearer <PROXY_MASTER_KEY>" | jq '.data[].id'

Troubleshooting¶

Symptom	Cause	Fix
`SSL certificate verify failed`	CA not trusted	Verify `internal-ca-bundle` secret mounted at `/etc/ssl/internal/ca.pem`
`DISABLE_SCHEMA_UPDATE=true` migration skip	Migration job must run first with schema update enabled	Set `DISABLE_SCHEMA_UPDATE=false` on first deploy only
Redis connection refused	Redis not ready	Check `foundry-litellm-redis-master-0` pod
`model not found`	vLLM service not reachable	`kubectl get svc -n foundry \| grep vllm-<model-name>`
OOMKilled	8Gi request too high for dev	Reduce `resources.requests.memory` to `2Gi` for dev