Hatchet Test Cases — IAM Async Flows¶

These test cases verify the Hatchet ↔ core-service ↔ Zitadel (IAM) integration end-to-end. The IAM-relevant async flows are the bulk user operations: each API call dispatches a Hatchet workflow whose tasks call the IAM service (Zitadel) to create/delete users or change RBAC, tracked by a job_id.

flowchart LR
    A["API call<br/>POST /users/bulk-*"] --> B["core-service<br/>fire_bulk_*_workflow"]
    B -->|"gRPC :7070 + token"| C["hatchet-engine"]
    C --> D["bulk-operations-worker<br/>(invite/delete/role tasks)"]
    D -->|"IAM client (PAT)"| E["Zitadel<br/>create/delete user, roles"]
    D --> F["core-pg<br/>bulk_job status + local user rows"]
    A -.->|"poll job_id"| G["GET /users/bulk-operations/{job_id}/status"]

    style A fill:#1e3a5f,stroke:#4a90d9,color:#fff
    style C fill:#1e5f3a,stroke:#4ad98a,color:#fff
    style D fill:#5f4a1e,stroke:#d9a94a,color:#fff
    style E fill:#5f1e3a,stroke:#d94a8a,color:#fff
    style F fill:#3a1e5f,stroke:#a94ad9,color:#fff

Three layers to assert on

A passing test must show the result on all three layers, not just one:

core-service — job_id reaches COMPLETED (or per-row success counts) via the status API
Hatchet — the workflow run shows SUCCEEDED (engine logs / Hatchet dashboard)
IAM (Zitadel) + DB — the user actually exists/is removed in Zitadel and the local core-service DB. This is the "in regards to IAM service" assertion.

Preconditions¶

[x] Hatchet token wired (HATCHET__CLIENT_TOKEN) — see Hatchet Wiring
[x] bulkOperationsWorker.enabled: true and the worker pod is Running
[x] core-service Running/Ready; Zitadel reachable; PAT valid (Zitadel onboarding done)
[x] An admin session token for the API (see "Getting the token" below — the bulk endpoints require an authenticated org admin, not the service PAT)

# handles used throughout
NS=ask
API=https://core-service.ask.mod.auh1.dev.dir       # add to /etc/hosts → ingress IP
ORG_PAT='<core-service-service-user-PAT>'            # to verify the Zitadel side
ORG_ID='377526562817377089'

Getting the auth token (curl)¶

The bulk endpoints need a session token (sessionToken from POST /api/v1/auth/login).

Air-gapped bootstrap constraint

Public signup won't work here — POST /auth/signup sends an activation email, and there is no SMTP relay in the air-gapped env, so the user can never set a password that way.
/auth/login requires the user to already exist in core-service's DB and in Zitadel with a password (login raises 401 User not found in organization otherwise).

So the first admin is a one-time bootstrap (below). After that, login is a normal curl.

One-time: bootstrap an admin user (no SMTP needed):

In the Zitadel console (default org) → Users → Human → New → create [email protected] with a password; mark email verified, password-change not required. Copy the user's Resource Id (its Zitadel user id).

Seed the matching org + admin row in core-service's DB (links via external_user_id). Either set postgresInit.seedData in the values and re-sync, or insert directly for testing:

kubectl exec -n ask core-pg-1 -- psql -U postgres -d core-service -v ON_ERROR_STOP=1 <<'SQL'
-- minimal: an org + an admin user mapped to the Zitadel user id (roleId 1 = admin)
INSERT INTO core_service.organizations (external_organization_id, name, primary_domain)
  VALUES ('377526562817377089','ask','ask.local') ON CONFLICT DO NOTHING;
INSERT INTO core_service.users (email, external_user_id, organization_id, role_id, status)
  SELECT '[email protected]','<ZITADEL_USER_ID>', id, 1, 'active'
  FROM core_service.organizations WHERE name='ask' ON CONFLICT DO NOTHING;
SQL

(column names per your migration; adjust to the actual schema.)

Then log in (curl) → grab the session token:

TOKEN=$(curl -sk -X POST "$API/api/v1/auth/login" \
  -H "Content-Type: application/json" \
  -d "{\"org_id\":\"$ORG_ID\",\"email\":\"[email protected]\",\"password\":\"<password>\"}" \
  | jq -r '.sessionToken')
echo "token: ${TOKEN:0:12}…"
# sanity: who am I
curl -sk "$API/api/v1/users/me" -H "Authorization: Bearer $TOKEN" | jq '{email, role: .role_id}'

Expect a non-empty sessionToken and your admin user from /users/me. A 401 User not found in organization means the DB seed (step 2) didn't match the Zitadel user / org.

TC-0 — Connectivity & workflow registration (smoke)¶

Objective: core-service can reach the engine and the worker has registered its workflows.

Step	Command / action	Expected
1	`kubectl logs -n $NS deploy/core-service \\| grep -i "Hatchet client initialized"`	client built, no `HATCHET_CLIENT_TOKEN not configured`
2	`kubectl logs -n $NS deploy/core-service-bulk-operations-worker --tail=50 \\| grep -iE "listening\\|registered\\|workflow"`	worker connected, registers `core_service_bulk_invitation`, `core_service_bulk_delete`, `core_service_bulk_role_assign`
3	`kubectl logs -n $NS deploy/hatchet-engine --tail=50 \\| grep -iE "worker\\|register"`	engine shows the worker registered

Pass: client initialized + worker registered the bulk workflows. Fail signals: 401/permission (bad token), connection refused (engine unreachable), tls errors (TLS strategy mismatch).

TC-1 — Bulk invite (CSV) creates users in Zitadel (core IAM path)¶

Objective: an async bulk invite provisions real users in Zitadel + the local DB.

Steps:

# 1. trigger — upload a small CSV (2 rows)
printf 'email,first_name,last_name,role\n [email protected],Test,One,member\n [email protected],Test,Two,member\n' > /tmp/invite.csv
JOB=$(curl -sk -X POST "$API/api/v1/users/bulk-invite/csv" \
  -H "Authorization: Bearer $TOKEN" -F "file=@/tmp/invite.csv" | jq -r '.job_id')
echo "job_id=$JOB"

# 2. poll status until terminal
watch -n3 "curl -sk \"$API/api/v1/users/bulk-operations/$JOB/status\" -H \"Authorization: Bearer $TOKEN\" | jq '{status, processed, succeeded, failed}'"

Expected:

Layer	Assertion
core-service	status → `COMPLETED`, `succeeded: 2`, `failed: 0`
Hatchet	`kubectl logs -n $NS deploy/hatchet-engine \\| grep core_service_bulk_invitation` → run `SUCCEEDED`
Worker	`kubectl logs -n $NS deploy/core-service-bulk-operations-worker \\| grep -iE "invite\\|[email protected]"` → per-row task ran, no IAM error
IAM (Zitadel)	the users exist in the org:

# verify the user was created in Zitadel (management API, org-scoped)
curl -sk -X POST "https://sso.ask.mod.auh1.dev.dir/management/v1/users/_search" \
  -H "Authorization: Bearer $ORG_PAT" -H "x-zitadel-orgid: $ORG_ID" \
  -H "Content-Type: application/json" \
  -d '{"queries":[{"emailQuery":{"emailAddress":"[email protected]"}}]}' | jq '.result[].userName'
# verify the local DB row
kubectl exec -n $NS core-pg-1 -- psql -U postgres -d core-service -tA \
  -c "select email from core_service.users where email='[email protected]';"

Pass: both Zitadel search and the DB query return the invited user. Fail (IAM-specific): job COMPLETED but Zitadel search empty → the worker's IAM call failed silently (check worker logs for 401/403 from Zitadel = PAT/role problem, not a Hatchet problem).

TC-2 — Bulk role assignment updates RBAC (IAM/RBAC path)¶

Objective: async role assignment changes the users' roles in core-service (and IAM grants).

JOB=$(curl -sk -X POST "$API/api/v1/users/bulk-role-assign" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"assignments":[{"email":"[email protected]","role":"group_admin"}]}' | jq -r '.job_id')
curl -sk "$API/api/v1/users/bulk-operations/$JOB/status" -H "Authorization: Bearer $TOKEN" | jq '{status, succeeded, failed}'

Expected: status COMPLETED, succeeded: 1; the DB shows the new role:

kubectl exec -n $NS core-pg-1 -- psql -U postgres -d core-service -tA \
  -c "select u.email, r.name from core_service.users u
      join core_service.user_roles ur on ur.user_id=u.id
      join core_service.roles r on r.id=ur.role_id
      where u.email='[email protected]';"

Pass: the role row reflects group_admin.

TC-3 — Bulk delete removes users from Zitadel (IAM teardown path)¶

JOB=$(curl -sk -X POST "$API/api/v1/users/bulk-delete" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"emails":["[email protected]"]}' | jq -r '.job_id')
curl -sk "$API/api/v1/users/bulk-operations/$JOB/status" -H "Authorization: Bearer $TOKEN" | jq '{status, succeeded, failed}'

Expected: status COMPLETED; then both sides show the user gone:

# Zitadel: search returns empty
curl -sk -X POST "https://sso.ask.mod.auh1.dev.dir/management/v1/users/_search" \
  -H "Authorization: Bearer $ORG_PAT" -H "x-zitadel-orgid: $ORG_ID" \
  -H "Content-Type: application/json" \
  -d '{"queries":[{"emailQuery":{"emailAddress":"[email protected]"}}]}' | jq '.result | length'   # → 0
# DB: row removed (or soft-deleted per app semantics)
kubectl exec -n $NS core-pg-1 -- psql -U postgres -d core-service -tA \
  -c "select count(*) from core_service.users where email='[email protected]';"

TC-4 — Job status tracking & idempotency¶

Objective: the status API accurately reflects progress and re-runs don't double-create.

Step	Expected
`GET /bulk-operations/{job_id}` (details)	returns job metadata: type, totals, per-row results
`GET /bulk-operations/{job_id}/status` during run	transitions `PENDING → RUNNING → COMPLETED`
Re-submit TC-1 with the same CSV	already-existing users counted as `skipped`/`failed` (duplicate), not created twice in Zitadel

TC-5 — Failure & retry handling (negative)¶

Objective: bad input fails gracefully and retryable IAM errors are retried per config.

# malformed row (invalid email) + one valid
printf 'email,first_name,last_name,role\n not-an-email,Bad,Row,member\n [email protected],Test,Three,member\n' > /tmp/invite-bad.csv
JOB=$(curl -sk -X POST "$API/api/v1/users/bulk-invite/csv" -H "Authorization: Bearer $TOKEN" -F "file=@/tmp/invite-bad.csv" | jq -r '.job_id')
curl -sk "$API/api/v1/users/bulk-operations/$JOB/status" -H "Authorization: Bearer $TOKEN" | jq '{status, succeeded, failed, errors}'

Expected: the valid row succeeds (succeeded: 1), the bad row is reported in failed with a reason — the whole job is not aborted. Worker logs show the bad row's validation error, not a crash. Retryable Zitadel errors (5xx / timeout, per constants.py retry config) show retry attempts in the worker logs rather than immediate failure.

Result matrix¶

TC	Flow	Hatchet asserted	IAM (Zitadel) asserted	DB asserted
TC-0	connectivity / registration	✅	—	—
TC-1	bulk invite (create)	✅	✅ user exists	✅ row exists
TC-2	bulk role assign	✅	✅ grant	✅ role row
TC-3	bulk delete	✅	✅ user gone	✅ row gone
TC-4	status / idempotency	✅	✅ no dup	✅
TC-5	failure / retry	✅	✅ partial	✅ partial

Interpreting failures

Job stuck PENDING/RUNNING forever → Hatchet problem: worker not registered, token wrong tenant, or engine unreachable (TC-0).
Job COMPLETED but Zitadel unchanged → IAM problem, not Hatchet: the worker's IAM call got 401/403 (PAT invalid or Iam Org Manager missing a grant) — see Zitadel onboarding Step 4/8.
Job FAILED immediately → input validation or DB error before dispatch — check the API pod logs, not the worker.