Foundry / LLM Endpoint (Air-Gapped) — for assistant-service¶

Foundry is the OpenAI-compatible LLM gateway (LiteLLM in front of vLLM) that assistant-service's intelligence and tooling layers call for every chat completion, embedding, and rerank. It is an independent service referenced by in-cluster URL — not a subchart.

CONFIRMED & WIRED (2026-06-17)

The endpoint and model set are now confirmed. assistant-service is wired to the real in-cluster LiteLLM gateway with OSS models. This page is the live configuration of record — the earlier "deferred / placeholder" approach is superseded.

Confirmed endpoint¶

http://foundry-litellm.foundry.svc.cluster.local:4000/v1

In-cluster service DNS — no internet, no ingress TLS. The reranker is the same gateway with the /v1/rerank path. Auth is a LiteLLM virtual key passed as a bearer token (AI71_FOUNDRY_KEY).

Models served by Foundry¶

How assistant-service references these models

There are two layers — the internal [foundry]/[tooling] TOML fields (set in the env values), and the user-selectable catalog models.yaml (baked in the image, needs an air-gap override). Full detail: Model References (two-layer).

Model id	Role	Used by
`qwen/qwen3-5-122b-a10b`	Flagship chat / reasoning / data-viz	assistant-service (main + intelligence)
`Qwen/Qwen3-4B-Instruct-2507`	Small / lightweight tasks, summaries	assistant-service (alias `qwen3-4b-instruct-2507`)
`qwen/qwen3-5-122b-a10b`	Code generation, DOCX context, data-analysis tool	assistant-service (tooling)
`qwen/qwen3-embedding-0-6b`	Embeddings (tool search)	assistant-service
`nvidia/llama-nemotron-rerank-1b-v2`	RAG reranking	assistant-service (intelligence)
`qwen/qwen3-5-122b-a10b-kms`	KMS-only — separate instance	KMS (do not use here)

qwen3-5-122b-a10b-kms is reserved

The -kms suffix is a separate Foundry instance dedicated to the Knowledge Management Service so its ingestion load does not contend with interactive assistant traffic. Never point assistant-service at it — use the plain qwen/qwen3-5-122b-a10b.

Why every model name had to change¶

The chart's baked config/default.toml ships cloud model names that the air-gapped Foundry does not serve. Leaving any of them unmapped means that call returns 404 model not found:

`default.toml` field	Cloud default (absent here)	Air-gap override
`llm_model`	`azure/gpt-5-chat` ❌	`qwen/qwen3-5-122b-a10b`
`small_llm_model`	`azure/gpt-5-chat` ❌	`Qwen/Qwen3-4B-Instruct-2507`
`data_visualization_llm`	`anthropic/claude-3-7-sonnet` ❌	`qwen/qwen3-5-122b-a10b`
`tool_embedding_model`	`BAAI/bge-large-en-v1.5` ❌	`qwen/qwen3-embedding-0-6b`
`reranker.model`	`BAAI/bge-reranker-v2-m3` ❌	`nvidia/llama-nemotron-rerank-1b-v2`
`code_generator_model`	`qwen/qwen3-5-122b-a10b` ✅	unchanged (served)
`docx_prepare_context_model`	`qwen/qwen3-5-122b-a10b` ✅	unchanged (served)

FoundryConfig (pydantic) makes all of these required fields — they have no usable runtime default, so the override must be explicit in the env TOML.

Configuration of record (env values TOML)¶

In helm/environments/values-mod-auh1-dev-ask.yaml under envConf.configToml:

[mod-auh1-dev-ask.foundry]
endpoint = "http://foundry-litellm.foundry.svc.cluster.local:4000/v1"
llm_model = "qwen/qwen3-5-122b-a10b"          # main chat/reasoning (flagship)
small_llm_model = "Qwen/Qwen3-4B-Instruct-2507"  # lightweight tasks, summaries
code_generator_model = "qwen/qwen3-5-122b-a10b"  # code gen
docx_prepare_context_model = "qwen/qwen3-5-122b-a10b"
data_visualization_llm = "qwen/qwen3-5-122b-a10b"
tool_embedding_model = "qwen/qwen3-embedding-0-6b"

[mod-auh1-dev-ask.foundry.reranker]
endpoint = "http://foundry-litellm.foundry.svc.cluster.local:4000/v1/rerank"
model = "nvidia/llama-nemotron-rerank-1b-v2"

The tooling service reads ENV, not this TOML

The tooling workload has its own ToolsSettings (pydantic-settings) that reads AI71_FOUNDRY_ENDPOINT directly from the environment — the [foundry] TOML block does not reach it. That env var is set separately in envConf.extraEnv:

- name: AI71_FOUNDRY_ENDPOINT
  value: "http://foundry-litellm.foundry.svc.cluster.local:4000/v1"

Keep the two endpoints in sync.

The gateway's own secret — `foundry-secret` (foundry namespace)¶

The LiteLLM gateway authenticates clients and reaches its own backends/state stores using an 11-key Opaque Secret foundry-secret in the foundry namespace. assistant-service consumes exactly one of these (PROXY_MASTER_KEY); the rest are internal to the gateway and listed here so the boundary is clear.

kubectl get secret -n foundry foundry-secret -o jsonpath='{.data}' \
  | python3 -c "import sys,json; [print(k) for k in json.load(sys.stdin)]"

Key	Purpose	Consumed by assistant-service?
`PROXY_MASTER_KEY`	Client-facing LiteLLM admin/master key (`sk-…`). What callers put in `Authorization: Bearer`. This is the value copied into Vault as `AI71_FOUNDRY_KEY`.	✅ yes → `AI71_FOUNDRY_KEY`
`PROXY_SALT_KEY`	LiteLLM `LITELLM_SALT_KEY` — encrypts/decrypts the upstream model credentials stored in its Postgres DB. Rotating it invalidates stored keys.	❌ gateway-internal
`VLLM_API_KEY`	Key LiteLLM presents to the upstream vLLM model servers (the GPU pods serving the qwen / gpt-oss chat + LLM models).	❌ gateway→backend
`INFINITY_API_KEY`	Key LiteLLM presents to the Infinity inference backend serving embeddings (`qwen3-embedding`) and the reranker (`llama-nemotron-rerank`).	❌ gateway→backend
`DATABASE_HOST`	Postgres host for LiteLLM's state DB (virtual keys, spend/budgets, model config).	❌ gateway-internal
`DATABASE_NAME`	LiteLLM Postgres database name.	❌ gateway-internal
`DATABASE_USERNAME`	LiteLLM Postgres role.	❌ gateway-internal
`DATABASE_PASSWORD`	LiteLLM Postgres role password.	❌ gateway-internal
`REDIS_PASSWORD`	Redis auth — LiteLLM uses Redis for response caching, rate-limit counters, and multi-replica router state.	❌ gateway-internal
`AWS_ACCESS_KEY_ID`	S3 (Ceph RGW) access key — model-weight storage (`foundry-models` bucket) / LiteLLM logging.	❌ gateway-internal
`AWS_SECRET_ACCESS_KEY`	S3 (Ceph RGW) secret key, paired with the above.	❌ gateway-internal

Boundary: assistant-service only needs the master key

The chain is assistant-service → (PROXY_MASTER_KEY) → LiteLLM → (VLLM_API_KEY / INFINITY_API_KEY) → vLLM / Infinity. assistant-service never sees the backend keys, the DB, or Redis — those are the gateway's private plumbing. Only PROXY_MASTER_KEY crosses the namespace boundary, copied into Vault secret/ask/assistant-service/auh1-dev as AI71_FOUNDRY_KEY.

Least-privilege follow-up

Using PROXY_MASTER_KEY (admin-scoped) proves the wiring but is over-privileged for a single service. The production-correct approach is to mint a virtual key in LiteLLM scoped to the models assistant-service needs, and patch that into AI71_FOUNDRY_KEY instead of the master key. Rotating the master key would otherwise break every consumer at once.

Patch the Foundry key (Vault)¶

The key name must be AI71_FOUNDRY_KEY — default.toml reads api_key = "@format {env[AI71_FOUNDRY_KEY]}". The ESO ExternalSecret uses dataFrom: extract, so every key in the Vault path becomes a key in the assistant-service-secret k8s Secret.

A non-empty key can still be a placeholder → 401

Bring-up seeds AI71_FOUNDRY_KEY with a random 32-byte hex placeholder (openssl rand -hex 32 → 64 chars) so config validates. That value is present but invalid, so /v1/models returns 401 Authentication Error. A real LiteLLM virtual key looks like sk-.... Length 64 = placeholder, not patched yet. Verify:

kubectl get secret -n ask assistant-service-secret \
  -o jsonpath='{.data.AI71_FOUNDRY_KEY}' | base64 -d | wc -c   # 64 = placeholder

1. Patch Vault (on bastion)¶

export VAULT_ADDR=https://vault.cl1.sq4.aegis.internal
export VAULT_SKIP_VERIFY=true

vault kv patch secret/ask/assistant-service/auh1-dev \
  AI71_FOUNDRY_KEY="<litellm-virtual-key>"

patch (not put) preserves the existing keys — HATCHET__CLIENT_TOKEN, DB creds, etc.

2. Verify in Vault¶

vault kv get -field=AI71_FOUNDRY_KEY secret/ask/assistant-service/auh1-dev

3. Force ESO re-sync (refreshInterval is 1m; this is immediate)¶

kubectl annotate externalsecret -n ask assistant-service-external-secret \
  force-sync=$(date +%s) --overwrite

4. Confirm the key reached the k8s Secret¶

kubectl get secret -n ask assistant-service-secret \
  -o jsonpath='{.data.AI71_FOUNDRY_KEY}' | base64 -d; echo

5. Roll the pods (envFrom is read once at process start)¶

kubectl rollout restart deploy/assistant-service \
  deploy/assistant-service-intelligence deploy/assistant-service-tooling -n ask

Ordering vs ArgoCD

The model-name overrides live in the chart TOML — they apply only after git push → ArgoCD syncs the new ConfigMap (which itself rolls the pods). If you patch the key before that sync settles, the ArgoCD-triggered restart already picks up the new Secret and step 5 is redundant. Patch after a settled sync → run step 5.

Validate from assistant-service¶

Embedding smoke test (intelligence pod) — confirms key + endpoint + embedding model resolve:

FOUNDRY_KEY=$(kubectl get secret -n ask assistant-service-secret \
  -o jsonpath='{.data.AI71_FOUNDRY_KEY}' | base64 -d)
kubectl exec -n ask deploy/assistant-service-intelligence -- python3 - <<EOF
import httpx, os
key = "${FOUNDRY_KEY}"
r = httpx.post(
    "http://foundry-litellm.foundry.svc.cluster.local:4000/v1/embeddings",
    headers={"Authorization": f"Bearer {key}"},
    json={"model": "qwen/qwen3-embedding-0-6b", "input": "hello world"},
    timeout=30,
)
print("status:", r.status_code)
d = r.json()
print("dims:", len(d["data"][0]["embedding"]) if "data" in d else d)
EOF

Chat smoke test:

kubectl exec -n ask deploy/assistant-service-intelligence -- python3 - <<EOF
import httpx
key = "$(kubectl get secret -n ask assistant-service-secret -o jsonpath='{.data.AI71_FOUNDRY_KEY}' | base64 -d)"
r = httpx.post(
    "http://foundry-litellm.foundry.svc.cluster.local:4000/v1/chat/completions",
    headers={"Authorization": f"Bearer {key}"},
    json={"model": "qwen/qwen3-5-122b-a10b",
          "messages": [{"role": "user", "content": "ping"}], "max_tokens": 5},
    timeout=60,
)
print(r.status_code, r.json().get("choices", r.text[:200]))
EOF

Done when: the embedding returns a vector and the chat returns a completion — no model not found (a model name is wrong) and no 401 (the AI71_FOUNDRY_KEY is wrong or stale in the Secret).

Watch the logs for any residual mismatch:

kubectl logs -n ask deploy/assistant-service-intelligence --tail=40 \
  | grep -iE "foundry|model|401|404|embed|rerank"

Independent of Foundry, the intelligence/main pods stall ~100s at startup while tiktoken tries to download o200k_base from openaipublic.blob.core.windows.net (no route, OS SYN timeout). This is a harmless warning functionally (it falls back to approximate token counts) but the latency raced the liveness probe and caused restarts. Mitigated by raising initialDelaySeconds to 180 on main + intelligence; the production-correct fix is to bake the tiktoken vocab into the image. See Integration Tests → common failure patterns and Troubleshooting.

Production / air-gap notes¶

vLLM serves the models on the GPU node pool (tainted nvidia.com/gpu), separate from the ask71 worker pool that runs assistant-service.
Model weights live in the foundry-models Ceph S3 bucket; a download job hydrates the vLLM PVC.
Track the cloud→OSS model mapping explicitly (the table above) so a chart bump that resets default.toml does not silently reintroduce azure/* or anthropic/* names with no backend.
The reranker uses api_key = "@null" from default.toml — it relies on the same LiteLLM gateway auth path. If rerank calls 401 while chat/embeddings succeed, set the reranker key explicitly.