Hatchet — Production Deployment Guide

Hatchet is an async workflow engine used by the ASK platform to run background jobs — things like document ingestion, AI processing pipelines, and scheduled tasks that cannot run inline in a web request.

This guide is written for a junior engineer taking over the deployment. Read it top to bottom before touching anything.

---

What is Hatchet?

When a user uploads a document or asks a question that triggers a long-running operation, ASK services (core-service, assistant-service, KMS) submit a workflow to Hatchet instead of processing it inline. Hatchet queues it, assigns it to a worker, tracks its progress step-by-step, retries on failure, and stores the result.

Think of Hatchet as a job queue + task scheduler + workflow tracker — similar to Celery (Python) or BullMQ (Node), but with a UI, gRPC API, and first-class step-dependency support.

---

Component Overview

┌─────────────────────────────────────────────────────────────────┐
│                        ASK Services                             │
│         core-service  /  assistant-service  /  KMS              │
│                 (submit workflows via gRPC)                      │
└────────────────────────┬────────────────────────────────────────┘
                         │ gRPC :7070
                         ▼
┌─────────────────────────────────────────────────────────────────┐
│                     Hatchet Engine (x2)                         │
│  • Receives workflow submissions                                 │
│  • Schedules steps, manages retries, tracks state               │
│  • Leader-elected via PostgreSQL — only 1 active scheduler      │
│  • Reads/writes workflow state to PostgreSQL                     │
│  • Dispatches step tasks to workers via RabbitMQ                │
└──────────┬────────────────────────────┬────────────────────────┘
           │                            │
           │ AMQP                       │ SQL
           ▼                            ▼
┌──────────────────────┐    ┌───────────────────────┐
│  RabbitMQ (x3)       │    │  PostgreSQL / CNPG     │
│  Message broker      │    │  (1 primary + 2        │
│  Quorum queues       │    │   read replicas)       │
│  Durable messages    │    │  Workflow state,        │
│  Erlang cluster      │    │  run history, crons    │
└──────────┬───────────┘    └───────────────────────┘
           │ AMQP
           ▼
┌─────────────────────────────────────────────────────────────────┐
│                     ASK Workers                                  │
│  (run inside core-service / assistant-service / KMS pods)       │
│  • Pull tasks from RabbitMQ                                      │
│  • Execute the actual business logic (AI calls, DB writes)      │
│  • Report step results back to Engine via gRPC                  │
└─────────────────────────────────────────────────────────────────┘
           ▲
           │ HTTPS :443
┌──────────────────────┐    ┌───────────────────────┐
│  Hatchet API (x2)    │    │  Hatchet Frontend (x2) │
│  REST + gRPC API     │    │  Web UI (static JS)    │
│  Auth, tenant mgmt   │◄───│  Dashboard, workflow   │
│  Workflow history    │    │  viewer, worker status │
└──────────────────────┘    └───────────────────────┘

Component Roles

Component	What it does	Why it matters
Engine	Scheduler and orchestrator. Decides which step runs next, handles retries, holds the global cron lock	Brain of Hatchet — if it goes down, no new steps are dispatched
API	REST + gRPC server. ASK services connect here to submit workflows and query run status	Gateway for all workflow submissions
Frontend	React dashboard served as static files. Shows workflow runs, worker status, cron schedules	Ops visibility — not in the critical path for workflow execution
RabbitMQ	Message broker. Engine publishes step tasks; workers consume them	Decouples engine from workers; durable queues survive pod restarts
PostgreSQL (CNPG)	Stores all workflow state, run history, event data, cron schedules	Source of truth — losing this means losing all workflow history
Workers	Not a Hatchet pod — they run inside your app pods (core-service, KMS, etc.) and register with the Engine via gRPC	Execute the actual business logic

---

Current Deployment State

Chart: hatchet-stack v0.10.0 / App v0.84.0 Namespace: ask ArgoCD App: mod-auh1-dev-ask.ask.hatchet UI: https://hatchet.ask.mod.auh1.dev.dir Login: [email protected] / ModGPT@2026! (stored in Vault: secret/ask/hatchet)

Component	Replicas	CPU Request	CPU Limit	Memory Request	Memory Limit	Anti-Affinity	PDB
hatchet-api	2	100m	500m	128Mi	512Mi	✅ preferred	✅ minAvailable: 1
hatchet-engine	2	250m	1	256Mi	2Gi	✅ preferred	✅ minAvailable: 1
hatchet-frontend	2	25m	100m	32Mi	64Mi	—	✅ minAvailable: 1
hatchet-rabbitmq	3	250m	500m	512Mi	1Gi	✅ preferred	—
hatchet-pg (CNPG)	3 (1P+2R)	—	—	—	—	✅ CNPG managed	✅ CNPG managed

---

Traffic Flow — Step by Step

1. User uploads a document via ASK UI
2. core-service submits a DocumentIngest workflow to Hatchet API (grpc :7070)
3. Engine receives the workflow, creates run records in PostgreSQL, schedules Step 1
4. Engine publishes a step task to RabbitMQ (amqp :5672)
5. A Worker (running inside core-service) consumes the task from RabbitMQ
6. Worker executes the step (e.g., parse PDF, call LiteLLM, write to Weaviate)
7. Worker reports the result back to Engine via gRPC
8. Engine marks Step 1 complete, schedules Step 2 — repeat until workflow done
9. Frontend shows the run as COMPLETED in the dashboard

---

Key Design Decisions

Why RabbitMQ and not a simpler queue?

Hatchet uses RabbitMQ quorum queues which guarantee message durability — a message written to the queue survives RabbitMQ pod restarts and node failures (as long as quorum is maintained). This means a workflow submitted during a maintenance window won't be lost.

Why is Engine leader-elected?

Running 2 engine replicas is safe because Hatchet uses PostgreSQL row-locking for leader election. Only one engine holds the scheduler lease at any time. If that pod crashes, the other acquires the lease within seconds. This gives zero-downtime during engine restarts.

Why is the Erlang cookie pinned in Vault?

RabbitMQ nodes form a cluster using a shared Erlang cookie (a shared secret). If ArgoCD syncs and the Bitnami chart regenerates the cookie, the new pods have a different cookie and cannot rejoin the existing cluster — messages are lost. The cookie is stored in Vault and injected via ESO to prevent this.

---

Production Status Checklist

#	Item	Status	Notes
1	replicaCount ≥ 2 for all stateless components	✅ Done	api=2, engine=2, frontend=2
2	Resource requests + limits on all pods	✅ Done	See table above
3	Pod anti-affinity (spread across nodes)	✅ Done	api, engine, rabbitmq
4	PodDisruptionBudgets	✅ Done	api, engine, frontend
5	RabbitMQ quorum (3 replicas) + Erlang cookie pinned	✅ Done
6	OTel traces + logs → SigNoz	✅ Done	api + engine
7	CNPG PostgreSQL HA (3 instances)	✅ Done
8	CNPG S3 backup (WAL + daily base backup)	❌ Pending	Ceph S3 target
9	Change default admin password	✅ Done	Changed to ModGPT@2026! via DB update; stored in Vault at secret/ask/hatchet
10	Dedicated namespace (remove privileged PSA from ask)	❌ Pending	Medium priority

---

Pending Items

1 — Change Default Admin Password

The default credentials ([email protected] / Admin123!!) must be changed before this is considered production.

Hatchet v0.84 has no password change option in the UI. The password is stored as a bcrypt hash in the UserPassword table in PostgreSQL. Change it via direct DB update.

Step 1 — Generate a bcrypt hash for the new password:

python3 -c "import bcrypt; print(bcrypt.hashpw(b'<new-password>', bcrypt.gensalt(10)).decode())"

Step 2 — Write the SQL file:

cat > /tmp/update-pw.sql << 'EOF'
UPDATE "UserPassword" SET hash = '<bcrypt-hash-from-step-1>'
WHERE "userId" = (SELECT id FROM "User" WHERE email = '[email protected]');
EOF

Step 3 — Apply to PostgreSQL:

# Find the primary CNPG pod
kubectl get pods -n ask | grep hatchet-pg

# Apply (use the lowest-numbered pod e.g. hatchet-pg-2)
kubectl exec -n ask hatchet-pg-2 -- psql -U postgres -d hatchet -f - < /tmp/update-pw.sql

Expected output: UPDATE 1

Step 4 — Verify by logging in to https://hatchet.ask.mod.auh1.dev.dir with the new password.

Step 5 — Store in Vault:

vault kv put secret/ask/hatchet \
  admin-email="[email protected]" \
  admin-password="<new-password>"

Why direct DB update?

Hatchet stores passwords as bcrypt hashes in a separate UserPassword table (not in the User table). The User table has no password column. There is no admin CLI or UI option to change the password in v0.84 — direct DB update is the only method.

Change admin email (same approach — no UI option):

kubectl exec -n ask hatchet-pg-2 -- psql -U postgres -d hatchet -c \
  "UPDATE \"User\" SET email = '[email protected]' WHERE email = '[email protected]';"

Expected output: UPDATE 1

After this, login with [email protected] / ModGPT@2026!

Update Vault:

vault kv put secret/ask/hatchet \
  admin-email="[email protected]" \
  admin-password="ModGPT@2026!"

2 — CNPG S3 Backup

Add to applications/ask-db/hatchet/hatchet-pg-cluster.yaml:

backup:
  retentionPolicy: "7d"
  barmanObjectStore:
    destinationPath: s3://hatchet-pg-backups/
    endpointURL: https://s3.cl1.sq4.aegis.internal
    s3Credentials:
      accessKeyId:
        name: hatchet-pg-s3-creds
        key: ACCESS_KEY_ID
      secretAccessKey:
        name: hatchet-pg-s3-creds
        key: SECRET_ACCESS_KEY
    wal:
      compression: gzip
      maxParallel: 2
    data:
      compression: gzip

scheduledBackup:
  - name: hatchet-pg-daily
    spec:
      schedule: "0 2 * * *"
      backupOwnerReference: self
      cluster:
        name: hatchet-pg

3 — Dedicated Namespace

Currently Hatchet runs in the ask namespace which has privileged PSA (required by Hatchet's setup jobs). This relaxes security for all other workloads in ask. The production-correct approach is a dedicated hatchet namespace.

Migration steps: 1. Delete existing hatchet resources from ask namespace 2. Update ArgoCD Application destination.namespace: hatchet 3. Update all CNPG, ESO, secret resources to namespace: hatchet 4. Update DATABASE_URL in hatchet-secret — FQDN changes from hatchet-pg-rw.ask.svc.cluster.local → hatchet-pg-rw.hatchet.svc.cluster.local 5. Re-sync

---

Day-2 Operations

Check workflow run status

# All hatchet pods
kubectl get pods -n ask -l app.kubernetes.io/instance=hatchet

# Resource usage
kubectl top pods -n ask | grep hatchet

# Engine logs (scheduler activity)
kubectl logs -n ask -l app.kubernetes.io/name=engine --tail=50

# RabbitMQ queue depth
kubectl exec -n ask hatchet-rabbitmq-0 -- rabbitmqctl list_queues name messages

Check database size

kubectl exec -n ask hatchet-pg-1 -- psql -U postgres -d hatchet \
  -c "SELECT pg_size_pretty(pg_database_size('hatchet'));"

# Top tables by size
kubectl exec -n ask hatchet-pg-1 -- psql -U postgres -d hatchet \
  -c "SELECT tablename, pg_size_pretty(pg_total_relation_size('public.'||tablename)) AS size
      FROM pg_tables WHERE schemaname='public'
      ORDER BY pg_total_relation_size('public.'||tablename) DESC LIMIT 10;"

Expand PostgreSQL storage (no downtime)

kubectl patch cluster hatchet-pg -n ask \
  --type merge -p '{"spec":{"storage":{"size":"50Gi"}}}'
kubectl get pvc -n ask -l cnpg.io/cluster=hatchet-pg -w

RabbitMQ cluster health

kubectl exec -n ask hatchet-rabbitmq-0 -- rabbitmqctl cluster_status
kubectl exec -n ask hatchet-rabbitmq-0 -- rabbitmqctl status | grep -E "memory|vm_memory"