Hatchet Upgrade — v0.71.0 → v0.84.0¶

Upgrade date: 2026-06-30
Chart: hatchet-stack 0.10.0 → 0.11.0
Engine: v0.71.0 → v0.84.0

Why We Upgraded — Scheduler Lock Starvation¶

Symptom¶

Starting 2026-06-29, the Hatchet engine pods were restarting ~5 times per day (31 restarts over 6 days). Each restart caused a burst of Connection reset by peer errors on hatchet-pg and caused the setup/worker-token init Jobs to stall until a manual ArgoCD resync.

hatchet-engine   31 restarts  (Exit Code: 0 — clean shutdown, not OOM)
hatchet-engine   30 restarts
hatchet-api       2 restarts

Root Cause¶

The engine hit a scheduler lock starvation bug under concurrent KMS queue load. Observed in kubectl logs ... --previous:

long lock /hatchet/pkg/scheduling/v1/scheduler.go:815   duration=31684ms
long transaction start acquired_connections=9            duration=31700-31900ms
queue refill_time=31695ms                               (expected: <100ms)
context canceled → engine graceful shutdown (exit 0)

What happened step by step:

KMS has 8+ concurrent queues (kms_document_ingestion, kms_document_parsing, kms_document_chunking, kms_document_summarizing, kms_document_embedding, kms_keyword_extraction, kms_document_status_sync, kms_document_cleanup, kms_metrics).
All queues competed for the same internal scheduler lock in scheduler.go:815.
DB transactions held locks for ~32 seconds (normal: <100ms).
Context deadline exceeded → engine shut down cleanly (Exit Code: 0).
All 10-11 open DB connections dropped simultaneously → burst of 08006 Connection reset by peer on hatchet-pg.
The setup job check-db-connection init container was mid-retry during the pod cycling window → jobs appeared stuck until pods stabilised.

Not OOM — not DB exhaustion

Memory usage at time of restarts: 89–104 Mi (limit: 2 Gi). Current connections: 71 idle / 200 max (35%). No failover or switchover events. Pure scheduler lock contention.

What Changed in v0.74.9 (the key fix)¶

v0.74.9 was the first release to remove the majority of the V0 scheduler code. From the release notes:

The removal of v0 paths on the Hatchet engines have shown up to a 30% latency improvement in internal benchmarking.

A number of deadlocks which would appear under high load have been fixed.

ANALYZE is now run on partitioned tables every few hours to recompute statistics, which greatly improves query performance.

Fix statement timeout — directly addresses long-running transactions.

Actually reconnect to postgres if conn fails — improves recovery after connection drops.

This is the release that addresses the scheduler lock starvation we observed.

What's New — v0.71.0 → v0.84.0 Highlights¶

Performance & Reliability (most relevant to us)¶

Version	Change
`v0.74.9`	30% scheduler latency improvement — V0 queue code removed; deadlock fixes; statement timeout fix; postgres reconnect fix
`v0.74.14`	Fix statement timeout regression; fix worker ID; fix postgres reconnect
`v0.82.3`	Native OpenTelemetry support — `SERVER_OBSERVABILITY_ENABLED=true` adds spans/traces per workflow run in UI
`v0.83.0`	Full-text log search — tenant-level log aggregation, new `Logs` tab in UI, per-DAG log aggregation
`v0.84.0`	Performance optimisation on event queries and cron schedule polling; cached CEL programs (reduces allocations during event matching); fixed OTel trace lookup deadlocks

New Features (available after upgrade)¶

Feature	How to use
OpenTelemetry native tracing	Set `SERVER_OBSERVABILITY_ENABLED=true` on engine → view spans in workflow run UI. Pairs with SigNoz.
Full-text log search	Built-in — top-level `Logs` tab appears in Hatchet UI after upgrade
Webhooks (GA)	Register incoming webhooks (GitHub, Stripe, Slack, Generic) as Hatchet event triggers
Durable execution	Long-running tasks that survive engine restarts (Go, Python, Ruby SDKs)
`runMany` / `runManyNoWait`	Bulk workflow dispatch with per-item options
Wait-for-event lookback windows	Resume durable tasks from past events

Dashboard UX¶

Workflow output tabs
Log filtering by workflow
DAG view improvements
Trace view fixes

What We Changed¶

Files updated¶

File	Change
`mirror-config.yaml`	Hatchet images `v0.71.0` → `v0.84.0`
`helm-charts/mirror-charts.yaml`	Chart `0.10.0` → `0.11.0`
`applications/hatchet/values-mod-auh1-dev-ask.yaml`	All image tags `v0.71.0` → `v0.84.0`
`environment/.../argocd-application-hatchet.yaml`	`targetRevision: 0.10.0` → `0.11.0`

New chart defaults (`0.11.0` vs `0.10.0`)¶

Setting	Old	New	Impact
`sharedConfig.image.tag`	(not set)	`v0.84.0`	Global tag override — cleaner than per-component
`retainFailedHooks`	(not set)	`true`	Failed migration job pods kept for log inspection
`migrationJob.backoffLimit`	(not set)	`1`	Migration gets 1 retry before marking failed
`migrationJob.activeDeadlineSeconds`	(not set)	`900`	15-min hard timeout for migration job
`engine.migrationJob.enabled`	`true`	`false`	Engine no longer runs its own migration — consolidated into api migrationJob

Upgrade Procedure¶

Step 1 — Mirror images to Harbor¶

Run from bastion:

cd /path/to/harbor-mirror
python3 mirror-images.py --config mirror-config.yaml --filter hatchet

Verify in Harbor UI: harbor.cl1.sq4.aegis.internal → project ghcr → hatchet-dev/hatchet/*:v0.84.0

Step 2 — Mirror chart to GitLab¶

cd helm-charts
python3 core42-mirror-charts.py --config mirror-charts.yaml --filter hatchet

Verify: chart hatchet-stack-0.11.0 appears in GitLab HTTP package registry.

Step 3 — Push devops repo¶

cd mod-ask-devops
git add applications/hatchet/values-mod-auh1-dev-ask.yaml \
        environment/auh1/dev/02-argocd-application/ask/argocd-application-hatchet.yaml
git commit -m "feat(hatchet): upgrade to chart 0.11.0 / v0.84.0 — scheduler fix"
git push

Step 4 — ArgoCD sync¶

argocd app sync mod-auh1-dev-ask.ask.hatchet --grpc-web
argocd app wait mod-auh1-dev-ask.ask.hatchet --health --grpc-web

Step 5 — Verify¶

# Engine running new version
kubectl get pods -n ask | grep hatchet-engine

# Check engine logs — should not see "long lock" warnings
kubectl logs -n ask -l app.kubernetes.io/component=engine --tail=100 | grep -E "long lock|long transaction|context canceled"

# Restart count should reset to 0
kubectl get pods -n ask | grep hatchet-engine

Enabling OpenTelemetry (post-upgrade)¶

Once upgraded, you can enable native Hatchet OTel tracing by adding to values-mod-auh1-dev-ask.yaml:

engine:
  env:
    - name: SERVER_OBSERVABILITY_ENABLED
      value: "true"
    - name: SERVER_OTEL_EXPORTER_OTLP_ENDPOINT
      value: "http://otel-collector.signoz.svc.cluster.local:4317"
    - name: SERVER_OTEL_SERVICE_NAME
      value: "hatchet-engine"

This wires Hatchet workflow spans directly into SigNoz — each workflow run becomes a distributed trace visible in the SigNoz UI.

Troubleshooting¶

Symptom	Cause	Fix
Migration job fails with `backoffLimit exceeded`	Schema change failing	`kubectl logs -n ask -l job-name=<migration-job> -c migration-job` — check for SQL errors
Engine pods restart after upgrade	Incompatible env var from old chart	Check `SERVER_*` env vars against new chart defaults
`long lock` warnings still appearing	Old engine pods not terminated	`kubectl rollout restart deployment hatchet-engine -n ask`
Worker-token job stuck	Engine not ready when job ran	`argocd app sync mod-auh1-dev-ask.ask.hatchet --grpc-web`