E2E Testing Without the Frontend¶

An ordered sequence to validate the whole ASK platform — every model and every feature — using only kubectl/API calls, while the frontend is blocked on its air-gap rebuild. Run the stages in order; each builds on the previous, so a failure localises the break (model → service → feature).

How this stays accurate

Deep request bodies are read from each service's live /openapi.json (Stage 3) rather than hand-written — that is the authoritative contract for the running image. Routes below are confirmed from source; exact field names for POST bodies should be taken from /openapi.json.

Stage 0 — Setup (keys + endpoints)¶

# Foundry virtual key (model calls) and admin key (alias inspection)
KEY=$(kubectl get secret -n ask assistant-service-secret -o jsonpath='{.data.AI71_FOUNDRY_KEY}' | base64 -d)
ADMIN=$(kubectl get secret -n ask core-service-secret -o jsonpath='{.data.FOUNDRY_ADMIN_KEY}' | base64 -d)
# core-service machine token (for auth-gated core APIs), if seeded:
PAT=$(kubectl get secret -n ask core-service-secret -o jsonpath='{.data.APP__CORE_SERVICES_MACHINE_TOKEN}' | base64 -d 2>/dev/null)

In-cluster service base URLs (no TLS, no ingress):

Service	In-cluster base
core-service	`http://core-service.ask.svc.cluster.local`
assistant — intelligence	`http://assistant-service-intelligence:8003`
assistant — tooling	`http://assistant-service-tooling:8001`
KMS	`http://knowledge-management-service.ask.svc.cluster.local`
Foundry (LiteLLM)	`http://foundry-litellm.foundry.svc.cluster.local:4000`

Run probes from a pod with python3 (app images lack curl)

All snippets kubectl exec -i … deploy/assistant-service-intelligence -- python3 — that pod has httpx. Heredocs must use -i.

Stage 1 — All models (the foundation)¶

Every feature ultimately calls one of these. Green here first.

kubectl exec -i -n ask deploy/assistant-service-intelligence -- env KEY="$KEY" python3 - <<'EOF'
import os, httpx
B="http://foundry-litellm.foundry.svc.cluster.local:4000"; H={"Authorization":"Bearer "+os.environ["KEY"]}
def chat(m):
    r=httpx.post(B+"/v1/chat/completions",headers=H,
      json={"model":m,"messages":[{"role":"user","content":"Reply OK"}],"max_tokens":5},timeout=90)
    return r.status_code
print("chat  qwen3-5-122b-a10b     ", chat("qwen/qwen3-5-122b-a10b"))
print("chat  qwen3-4b-instruct      ", chat("Qwen/Qwen3-4B-Instruct-2507"))
print("chat  gpt-oss-120b          ", chat("qwen/qwen3-5-122b-a10b"))
print("chat  qwen3-5-122b-a10b-kms ", chat("qwen/qwen3-5-122b-a10b-kms"))
r=httpx.post(B+"/v1/embeddings",headers=H,json={"model":"qwen/qwen3-embedding-0-6b","input":"hi"},timeout=60)
print("embed qwen3-embedding-0-6b  ", r.status_code, "dim="+str(len(r.json()["data"][0]["embedding"])) if r.status_code==200 else "")
r=httpx.post(B+"/v1/rerank",headers=H,json={"model":"nvidia/llama-nemotron-rerank-1b-v2","query":"capital of UAE","documents":["Abu Dhabi is the UAE capital","Bananas are yellow"]},timeout=60)
print("rerank llama-nemotron       ", r.status_code)
EOF

Pass: all chat 200, embed 200 dim=1024, rerank 200.

Stage 2 — Service health (each process up)¶

kubectl exec -i -n ask deploy/assistant-service-intelligence -- python3 - <<'EOF'
import httpx
checks = {
  "core-service":      "http://core-service.ask.svc.cluster.local/api/v1/health/ready",
  "kms":               "http://knowledge-management-service.ask.svc.cluster.local/health/",
  "assistant-intel":   "http://assistant-service-intelligence:8003/health",
  "assistant-tooling": "http://assistant-service-tooling:8001/health/mcp",
}
for name,url in checks.items():
    try: print(f"{name:18s}", httpx.get(url,timeout=8).status_code)
    except Exception as e: print(f"{name:18s} ERR {e}")
EOF

Verified output (2026-06-22):

core-service       200
kms                200
assistant-intel    200
assistant-tooling  200

Each service has a DIFFERENT health path — a uniform path lies

Confirmed from /openapi.json (Stage 3):

Service	Health path	Notes
core-service	`/api/v1/health/ready` (+ `/live`)
KMS	`/health/`	other paths need auth → `401`, not down
assistant-intel	`/health` (+ `/health/liveness`, `/health/readiness`)
assistant-tooling	`/health/mcp`	no plain `/health` — only the MCP health route exists, so `/health` 404s by design

A loop that hits /api/v1/health/ready on all four returns 200/404/401/404 even when all are healthy — only core uses that path.

Stage 3 — Enumerate exact endpoints (authoritative)¶

Pull each service's OpenAPI to get the exact paths + request schemas for the deeper stages (no guessing):

for svc in \
  "core-service|http://core-service.ask.svc.cluster.local" \
  "kms|http://knowledge-management-service.ask.svc.cluster.local" \
  "assistant-intel|http://assistant-service-intelligence:8003" \
  "assistant-tooling|http://assistant-service-tooling:8001"; do
  name=${svc%%|*}; base=${svc##*|}
  echo "== $name =="
  kubectl exec -i -n ask deploy/assistant-service-intelligence -- env B="$base" python3 - <<'EOF'
import os, httpx
try:
    spec=httpx.get(os.environ["B"]+"/openapi.json",timeout=10).json()
    for p,methods in sorted(spec.get("paths",{}).items()):
        print("  ",",".join(m.upper() for m in methods), p)
except Exception as e: print("  (no /openapi.json:",e,")")
EOF
done

Use the printed paths/bodies as the source of truth for Stages 4–6.

Stage 4 — core-service (IAM / org / model catalog)¶

Real routes (from openapi): discovery GET /api/v1/organizations/discovery, login POST /api/v1/auth/login, model catalog GET /api/v1/models/ (+ /enabled, /providers, /disabled), orgs GET /api/v1/organizations.

Test user (verified 2026-06-23): [email protected] / org 378447542100166005.

# 4a. org discovery — no auth
kubectl exec -i -n ask deploy/assistant-service-intelligence -- python3 - <<'EOF'
import httpx
r=httpx.get("http://core-service.ask.svc.cluster.local/api/v1/organizations/discovery",
            params={"email":"[email protected]"},timeout=10)
print("discovery", r.status_code, r.text[:200])
EOF

Pass: 200 + the ask-default org.

The catalog/org endpoints are auth-gated — get a session via login (verified schema LoginDTO = org_id + email + password, a simple password grant). This login also bypasses the broken frontend and proves the login backend itself.

The login username MUST be email#<org-id> in Zitadel — or login 404s

auth_service.login() builds the Zitadel login name as email + "#" + org_id (utils/auth_utils.py:fetch_username_from_email). If the user's Zitadel preferred login name uses a different suffix (e.g. the user-id), Zitadel returns 404 "User could not be found" even though the DB row is fine. The two error codes localise the failure exactly:

Login result	Failing check	Fix
`401 "User not found in organization"`	DB lookup `get_by_email_in_organization`	seed/insert the `core_service.users` row
`404 "User could not be found"`	Zitadel `create_session` by loginName	rename the Zitadel user to `email#<org-id>`
`200` + `sessionToken`	both pass ✓	—

Full detail: Troubleshooting → login 404.

# 4b. LOGIN → carry the session → model catalog (L2), in one run
export PW='<testadmin password from Vault>'
kubectl exec -i -n ask deploy/assistant-service-intelligence -- env PW="$PW" python3 - <<'EOF'
import os, httpx
CS="http://core-service.ask.svc.cluster.local"
with httpx.Client(base_url=CS, timeout=20) as c:
    r=c.post("/api/v1/auth/login", json={
        "org_id":"378447542100166005",
        "email":"[email protected]",
        "password":os.environ["PW"]})
    print("login", r.status_code, r.text[:160])
    tok=None
    try: tok=r.json().get("sessionToken") or r.json().get("access_token")
    except Exception: pass
    h={"Authorization":"Bearer "+tok} if tok else {}
    m=c.get("/api/v1/models/enabled", headers=h)
    print("models/enabled", m.status_code, m.text[:400])
EOF

Verified (2026-06-23): login → 200 with {"sessionId","sessionToken","userId","organizationId",...}. The sessionToken is the Bearer token for all downstream auth-gated calls.

iam_client auth proof (2026-06-23)

After the iam_client 401 fix (gateway ingress + SSL_CERT_FILE + internal CA ConfigMap), passing the sessionToken to KMS GET /collections/ and core-service GET /api/v1/models/enabled both return 403 Insufficient permissions (not 401) — confirming the full chain: token → iam_client → core-service /api/v1/users/me → 200 → service applies authz → 403. A 401 here means iam_client is still broken; 403 means auth passed.

models/enabled requires a superadmin or model-admin role — testadmin (ask71_admin role) gets 403. The machine PAT (APP__CORE_SERVICES_MACHINE_TOKEN) also returns 401 "Token Validation Failed" here — it is not a user session token and cannot be resolved by users/me. Use superadmin credentials to pull the full catalog, or confirm model availability via Foundry /v1/models (Stage 1) which uses the Foundry key.

Superadmin verified (2026-06-23): [email protected] login 200 → models/enabled 200 → catalog is populated (38 inference / 2 embedding across all orgs) but all disabled by default for new orgs.

4c. Enable models for the org (superadmin, one-time)¶

kubectl exec -i -n ask deploy/assistant-service-intelligence -- env PW="$SUPERPW" python3 - <<'EOF'
import os, httpx, json
CS="http://core-service.ask.svc.cluster.local"
with httpx.Client(base_url=CS, timeout=20) as c:
    r=c.post("/api/v1/auth/login", json={
        "org_id":"378447542100166005",
        "email":"[email protected]",
        "password":os.environ["PW"]})
    tok=r.json().get("sessionToken")
    h={"Authorization":"Bearer "+tok}
    r=c.post("/api/v1/models/enable", headers=h, json={
        "model_id": ["qwen3-5-122b-a10b", "gpt-oss-120b"],
        "default_model_id": "qwen3-5-122b-a10b",
        "org_id": "378447542100166005"
    })
    print("enable", r.status_code, r.text[:200])
    m=c.get("/api/v1/models/enabled", headers=h)
    print("enabled after:", m.status_code, m.text[:400])
EOF

Verified (2026-06-23):

enable 200  → {"message":"Models enabled successfully.","enabled_models_count":2}

enabled after: 200
{"inference":[
  {"id":"gpt-oss-120b","provider":"OpenAI","name":"GPT-OSS 120B","version":"oss-120b",
   "description":"Open-source GPT model with 120B parameters for advanced tasks.",
   "type":"inference","supported_languages":["English"],"logo_file_name":"openai.svg",
   "logo_path":"https://cdn.ask.mod.auh1.dev.dir/static/logos/llm_models/openai.svg",
   "is_default":false,"is_recommended":true},
  {"id":"qwen3-5-122b-a10b", ... (truncated — 400-char limit hit)}
 ],
 "embedding":[]}

Enabled catalog → Foundry model mapping for this org:

Catalog ID	Name	Foundry model	Role
`qwen3-5-122b-a10b`	QWEN-3-5-122B	`qwen/qwen3-5-122b-a10b`	default chat
`gpt-oss-120b`	GPT-OSS 120B	`qwen/qwen3-5-122b-a10b`	code / data analysis

Embedding (BAAI/bge-m3) skipped — embedding is wired server-side via the TOML, not user-selectable.

cdn_base_url is required for /api/v1/models/ (not /models/enabled)

GET /models/ and /models/disabled build logo URLs via get_public_cdn_url() which raises ValueError: Storage CDN base URL is not configured. if cdn_base_url is absent from the [storage] TOML block. Add it to the devops overlay (mod-ask-devops/applications/core-service/values-mod-auh1-dev-ask.yaml), NOT the chart-repo env file (that's shadowed). Value: "https://cdn.ask.mod.auh1.dev.dir" (no /static suffix — the code appends the path). /models/enabled still returns 200 even without it because the enabled list is cached differently.

Console-created users need a DB row

A user made in the Zitadel Console (Email-Verified checkbox) is not auto-synced to the core-service DB → login 401 until you insert it:

INSERT INTO core_service.users
  (external_user_id, organization_id, email, display_name, first_name, last_name, state, created_at, updated_at)
VALUES ('<ZITADEL-USER-ID>','378447542100166005','<email>','<name>','<first>','<last>','active',NOW(),NOW());
INSERT INTO core_service.user_roles (created_at, role_id, user_id)
SELECT NOW(), 1, id FROM core_service.users WHERE email='<email>';   -- role_id 1 = ask71_admin

Stage 5 — KMS (ingest → retrieve → rerank)¶

Full pipeline: create collection → upload document → Hatchet processes (embed+VLM) → retrieve with rerank. All non-health KMS routes are auth-gated (org context via sessionToken).

5a. Health + route inventory¶

kubectl exec -i -n ask deploy/assistant-service-intelligence -- python3 - <<'EOF'
import httpx, json
KMS="http://knowledge-management-service.ask.svc.cluster.local"
h=httpx.get(KMS+"/health/",timeout=8)
print("health", h.status_code, h.text[:200])
s=httpx.get(KMS+"/openapi.json",timeout=10).json()
paths=list(s.get("paths",{}).keys())
print("\nROUTES:")
for p in sorted(paths): print(" ", p)
EOF

Verified (2026-06-23):

 ROUTES:   /admin/collection/{collection_id}   /chunks/document/{document_id}   /chunks/search   /collections/   /collections/search   /collections/{collection_id}/actors   /collections/{collection_id}/share   /databases/types   /databases/{connection_id}/actors   /documents/   /documents/collection/{   /documents/download-urls   /documents/import   /documents/queued/count   /documents/search   /documents/update/initiate   /documents/upload/{docu   /documents/{document_id}   /documents/{document_id}/restore   /documents/{document_id}/summary   /folders/children   /folders/{folder_id}   /folders/{folder_id}/move   /health/   /organizations/onboard   /tables/document/{document_id}

delineno-11-1">health 200 {"status":"HEALTHY","database":{"status":"UP","error":null}} /admin/document/{document_id} /chunks/neighbours /chunks/{chunk_id} /collections/is_shared /collections/{collection_id} /collections/{collection_id}/restore /databases/ /databases/{connection_id} /databases/{connection_id}/share /documents/cancel-processing collection_id}/documents/content /documents/file_formats /documents/move /documents/retry-processing /documents/sync /documents/upload/initiate ment_id}/update-status /documents/{document_id}/actors /documents/{document_id}/share /folders/ /folders/tree /folders/{folder_id}/actors /folders/{folder_id}/share /organizations/me /retrieve/ /tables/{table_id}

Stage 6 — assistant-service (chat + tools)¶

Real routes (openapi):

Concern	Pod	Route
Chat (sync)	intelligence `:8003`	`POST /orchestration/invoke`
Chat (stream)	intelligence `:8003`	`POST /orchestration/stream`
Orchestrator	intelligence `:8003`	`POST /orchestrator/call`, `POST /orchestrator/process`
Tool invoke	tooling `:8001`	`POST /toolbus/call` (+ `POST /toolbus/prepare`)
Tool catalog	tooling `:8001`	`GET /catalog/my-tools`, `GET /catalog/connectors`
KMS proxy	tooling `:8001`	`GET /kms/collections`, `GET /kms/documents`

Chat (POST /orchestration/invoke) is the main flow — exercises [foundry] llm_model (orchestrator) and, by message type, the small LLM / code-gen / data-viz models. Needs auth + assistant/conversation context; pull the exact body from the schema:

kubectl exec -i -n ask deploy/assistant-service-intelligence -- python3 - <<'EOF'
import httpx, json
s=httpx.get("http://assistant-service-intelligence:8003/openapi.json",timeout=10).json()
print(json.dumps(s["paths"]["/orchestration/invoke"]["post"].get("requestBody",{}),indent=2)[:900])
EOF

Tools (POST /toolbus/call on tooling :8001) drives a specific tool:
data-analysis → [tooling.data_analysis] * (gpt-oss-120b)
data visualization → [foundry] data_visualization_llm (qwen3-5-122b-a10b) — see the discrepancy note in Model Aliases §4
KMS proxy (GET /kms/collections) → assistant reaching KMS

Deep chat flows are async (Hatchet)

A real chat run is queued through Hatchet and streamed (/orchestration/stream). For a scripted check, drive /orchestration/invoke with the schema body and read the response. The repos' tests/intelligence/tests_integration suites are the canonical E2E for these flows.

Stage 7 — Feature → model → test matrix¶

Feature (tracker)	Model	Stage that proves it
Orchestrator	`qwen3-5-122b-a10b`	1 (model) + 6 (chat)
Guardrails (policy + safety)	`Qwen/Qwen3-4B-Instruct-2507` (alias `qwen3-4b-instruct-2507`)	1 (model) + 6 (send a tripping message → blocked)
Data visualization	`qwen3-5-122b-a10b` *	1 + 6 (data-viz tool)
Data-analysis tools	`gpt-oss-120b`	1 + 6 (tooling)
KMS embedding	`qwen3-embedding-0-6b`	1 (dim=1024) + 5 (search)
KMS OCR / VLM	`qwen3-5-122b-a10b-kms`	1 (model) + 5 (ingest an image)
KMS reranker	`llama-nemotron-rerank-1b-v2`	1 (model) + 5 (search ranked)
Mem0 / Voice / Meta-Agent	NA (disabled)	confirm off, not erroring

* tracker sheet lists gpt-oss-120b for data-viz — config says qwen3-5-122b-a10b; confirm which the tool actually calls.

Recommended run order¶

Stage 1 (models) — must be green first.
Stage 2 (health) — every service up.
Stage 3 (openapi) — capture exact bodies once.
Stage 4 (core discovery) — IAM/DB wiring.
Stage 5 (KMS search) — embed + rerank in context.
Stage 6 (assistant chat + one tool) — orchestrator + tool models.

If a stage fails, re-run Stage 1 for that feature's model to isolate model vs service. Per-service request detail: core-service Onboarding §6, assistant Integration Tests, 37 KMS Deploy.

Model Aliases & Testing — alias inventory, per-model probes (Stage 1 detail)
LLM → Service Config Mappings — feature → model reference