Service Model — Foundry Deployment¶
All AI/ML model calls made by the ASK platform, mapped to their Foundry deployment status.
Services covered: Assistant-Service · KMS · Core-Service · Frontend
| Feature | Service | Model (Config) | Model (Foundry alias) | Foundry Served | Status |
|---|---|---|---|---|---|
| Orchestrator (main chat / reasoning) | Assistant | qwen/qwen3-5-122b-a10b — L1 [foundry] llm_model |
qwen/qwen3-5-122b-a10b |
✅ yes | ✅ Active |
| Lightweight tasks (history summary, title gen) | Assistant | Qwen/Qwen3-4B-Instruct-2507 — L1 [foundry] small_llm_model |
Qwen/Qwen3-4B-Instruct-2507 |
✅ yes | ✅ Active |
| Code generation | Assistant | qwen/qwen3-5-122b-a10b — L1 [foundry] code_generator_model |
qwen/qwen3-5-122b-a10b |
✅ yes | ✅ Active |
| DOCX context prep | Assistant | qwen/qwen3-5-122b-a10b — L1 [foundry] docx_prepare_context_model |
qwen/qwen3-5-122b-a10b |
✅ yes | ✅ Active |
| Data visualisation | Assistant | qwen/qwen3-5-122b-a10b — L1 [foundry] data_visualization_llm |
qwen/qwen3-5-122b-a10b |
✅ yes | ✅ Active |
| Tool search embeddings | Assistant | qwen/qwen3-embedding-0-6b — L1 [foundry] tool_embedding_model |
qwen/qwen3-embedding-0-6b |
✅ yes | ✅ Active |
| Data analysis (profiler / validator / mapper) | Assistant | qwen/qwen3-5-122b-a10b — L1 [tooling.data_analysis] *_model ×4 |
qwen/qwen3-5-122b-a10b |
✅ yes | ✅ Active |
| User-selectable chat models (UI catalog) | Assistant | qwen/qwen3-5-122b-a10b, Qwen/Qwen3-4B-Instruct-2507, qwen/qwen3-5-122b-a10b — L2 models.yaml ConfigMap (api_model_name) |
direct model ids | ✅ yes | ✅ Active (ConfigMap override required — baked api_model_name values differ) |
| User-selectable embedding model (UI catalog) | Core | AI71ai/agrillm-embedding-bge-m3-v1 — L2 models.yml catalog id |
AI71ai/agrillm-embedding-bge-m3-v1 → qwen/qwen3-embedding-0-6b |
✅ yes (alias) | ✅ Active |
| Post-answer citations | Assistant | azure/gpt-5-chat — L3 hardcoded post_answer_citations.py:37 |
no alias defined | ❌ no | ❌ 404 (citations silently skipped) |
| Scratchpad reasoning | Assistant | anthropic/claude-haiku-4-5 — L3 hardcoded scratchpad_node.py:247 (default arg) |
no alias defined | ❌ no | ❌ 404 if scratchpad path runs |
| Tooling settings defaults | Assistant | azure/gpt-5-chat, anthropic/claude-3-7-sonnet — L3 hardcoded settings.py:168/172/176/462 (Pydantic defaults) |
no alias defined | ❌ no | ❌ 404 unless overridden by ENV |
| Custom policies guardrail | Assistant | [guardrails] custom_policy_model = qwen3-5-122b-a10b (commit 32aaae8) |
bare name | ⚠️ 400 on Foundry — verify resolution | ⚠️ Under verification |
| Prompt injection / safety guardrail | Assistant | [guardrails] safety_filters_model = qwen-3-4b-it-2507 (commit 32aaae8) |
bare name | ⚠️ 400 on Foundry — verify resolution | ⚠️ Under verification |
| OpenAI moderation filter | Assistant | omni-moderation-latest — L3 hardcoded openai_moderation_guardrail.py:57 |
no alias — alias won't fix this | ❌ no | ❌ Calls /v1/moderations API (not chat completions); Qwen doesn't serve that format. Needs source patch: replace client.moderations.create() with a chat completion using Qwen/Qwen3-4B-Instruct-2507 (already served). Plan: use the served Qwen/Qwen3-4B-Instruct-2507 (gpt-oss-safeguard-120b is decommissioned). |
| Reranker (via KMS — not called directly) | Assistant | nvidia/llama-nemotron-rerank-1b-v2 — L1 [foundry.reranker] model (unused) |
nvidia/llama-nemotron-rerank-1b-v2 |
✅ yes | 🔵 Reranking happens inside KMS — KMS calls Foundry reranker and returns pre-ranked chunks; assistant-service stores them in state["reranked"] but makes no direct /v1/rerank call itself. config.foundry.reranker block is dead. |
| Document embedding (ingest) | KMS | qwen/qwen3-embedding-0-6b — L1 [organization] embedding_model_id |
qwen/qwen3-embedding-0-6b |
✅ yes | ✅ Active |
| Query embedding (search) | KMS | qwen/qwen3-embedding-0-6b — L1 [organization] embedding_model_id |
qwen/qwen3-embedding-0-6b |
✅ yes | ✅ Active |
| VLM / OCR (visual doc parsing) | KMS | qwen/qwen3-5-122b-a10b-kms — L1 [organization] vlm_model_name |
qwen/qwen3-5-122b-a10b-kms (KMS-dedicated vLLM) |
✅ yes | ✅ Active |
| Retrieval reranker | KMS | nvidia/llama-nemotron-rerank-1b-v2 — L1 [retriever.reranker] model |
nvidia/llama-nemotron-rerank-1b-v2 |
✅ yes | ✅ Active — code, config, and model all present; E2E not yet tested (upload a doc → search → confirm rerank fires in worker logs) |
| Document summarization | KMS | not configured — L1 [document_understanding] summarization = @none |
N/A | — | 🚫 Disabled |
| Keyword / NER extraction | KMS | not configured — L1 [document_understanding] keyword_extraction = @none |
External NER endpoint (not Foundry) | — | 🚫 Disabled (no NER backend deployed) |
| Model catalog (API only — no LLM calls) | Core | models.yml catalog — L2 YAML listing served model ids |
N/A | — | Core-service does not make LLM calls; it serves the catalog and issues Foundry API keys |
| Frontend | Frontend | No model calls — domain URL baked at build time | N/A | — | No Foundry dependency |
Foundry backends (live as of 2026-06-28)¶
Six vLLM instances behind the LiteLLM proxy:
| Backend model ID | Kind | Used by |
|---|---|---|
Qwen/Qwen3-4B-Instruct-2507 |
Chat / instruct | Assistant small_llm_model, user catalog |
qwen/qwen3-5-122b-a10b |
Chat / reasoning | Assistant code_generator_model, docx, data-analysis; guardrails (pending) |
qwen/qwen3-5-122b-a10b |
Chat / VLM | Assistant llm_model, data_visualization_llm; user catalog |
qwen/qwen3-5-122b-a10b-kms |
Chat / VLM | KMS vlm_model_name — dedicated instance, isolated from assistant traffic |
qwen/qwen3-embedding-0-6b |
Embedding (1024-dim) | Assistant tool_embedding_model; KMS embed+search; UI catalog via alias |
nvidia/llama-nemotron-rerank-1b-v2 |
Reranker | KMS retrieval reranker (active); Assistant reranker (wired but not called) |
Open issues summary¶
| Priority | Service | Feature | Fix |
|---|---|---|---|
| 🔴 High | Assistant | Post-answer citations (azure/gpt-5-chat L3) |
Add Foundry alias azure/gpt-5-chat → qwen/qwen3-5-122b-a10b, or patch source |
| 🔴 High | Assistant | OpenAI moderation filter (omni-moderation-latest) |
Source patch required — guardrail calls /v1/moderations API (not chat completions); alias won't work. Replace client.moderations.create() with chat completion call using Qwen/Qwen3-4B-Instruct-2507 (already served; no new deployment needed). |
| 🟡 Medium | Assistant | Scratchpad (anthropic/claude-haiku-4-5 L3) |
Source patch or alias if scratchpad path is exercised |
| 🟡 Medium | Assistant | Tooling defaults (azure/gpt-5-chat L3 Pydantic) |
Set ENV override or source patch |
| ⚠️ Verify | Assistant | Guardrail models moved to qwen-3-4b-it-2507 / qwen3-5-122b-a10b (commit 32aaae8) |
Bare names 400 on direct Foundry call — confirm alias resolution or use prefixed served names |
| ℹ️ By design | Assistant | Reranker (nvidia/llama-nemotron-rerank-1b-v2) |
Reranking is delegated to KMS — KMS calls Foundry reranker and returns pre-ranked chunks back to assistant-service. Assistant-service's own [foundry.reranker] config block is intentionally unused. |
| ⚠️ E2E test needed | KMS | Retrieval reranker (nvidia/llama-nemotron-rerank-1b-v2) |
Code path exists, config enabled, model served — just needs a live retrieval query in the cluster to confirm it fires end-to-end |