SigNoz Deployment (Air-Gap)¶

Chart: signoz 0.130.1 / App: v0.130.1
Namespace: signoz
Ingress: https://signoz.ask.mod.auh1.dev.dir

Architecture¶

Services (ASK core/assistant/KMS/Hatchet)
        │  OTLP gRPC :4317 / HTTP :4318
        ▼
OTel Collector gateway  (2 replicas — HA)
        │
        ▼
ClickHouse  (200 Gi Ceph RBD)  ←  ZooKeeper (10 Gi Ceph RBD)
        │
        ▼
SigNoz query-service  →  Ingress (nginx + TLS)  →  Browser

Component	Replicas	Storage
`signoz` (query-service + alertmanager)	1	—
`otelCollector` (gateway)	2	—
`clickhouse`	1	200 Gi `csi-rbd-sc`
`zookeeper`	1	10 Gi `csi-rbd-sc`

GitOps Structure¶

mod-ask-devops/
├── environment/auh1/dev/02-argocd-application/
│   ├── _root/
│   │   ├── argocd-project-observability.yaml      ← AppProject
│   │   └── argocd-applications-observability.yaml ← App-of-Apps for observability/
│   └── observability/
│       └── argocd-application-signoz.yaml         ← SigNoz Application (3-source)
└── applications/signoz/
    ├── values-mod-auh1-dev-ask.yaml               ← Helm values (air-gap)
    └── signoz-gitlab-token.yaml                   ← ExternalSecret for GitLab deploy token

The ArgoCD Application uses 3 sources: 1. Helm chart from GitLab package registry (project 14) 2. Values file from mod-ask-devops repo (ref: values) 3. Raw manifests from applications/signoz/ (ExternalSecret)

Pre-requisites¶

Step 1 — Mirror images to Harbor¶

Images required (all docker.io → Harbor dockerhub project):

Image	Tag
`signoz/signoz`	`v0.130.1`
`signoz/signoz-otel-collector`	`v0.144.5`
`clickhouse/clickhouse-server`	`25.5.6`
`altinity/clickhouse-operator`	`0.21.2`
`altinity/metrics-exporter`	`0.21.2`
`signoz/zookeeper`	`3.7.1`
`alpine`	`3.18.2`
`busybox`	`1.35`

On bastion:

cd /path/to/harbor-mirror
python3 mirror-images.py --filter signoz

Step 2 — Mirror chart to Harbor + GitLab¶

cd /path/to/harbor-mirror/helm-charts
python3 core42-mirror-charts.py --config mirror-charts.yaml --filter signoz
python3 gitlab-push-charts.py -c gitlab-charts.yaml --only signoz

Verify:

curl -sk --header "PRIVATE-TOKEN: $TOKEN" \
  "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/14/packages?package_name=signoz" \
  | jq '.[].version'

Step 3 — Upload histogramQuantile binary to GitLab (one-time)¶

The ClickHouse UDF histogramQuantile is required for p50/p95/p99 latency percentiles. The binary is fetched at pod start from GitLab Generic Package Registry (GitHub is blocked in air-gap).

What it is: A custom ClickHouse executable UDF that implements histogram quantile calculation (p50/p95/p99). SigNoz uses it to display latency percentiles in the UI. Without it, ClickHouse starts but latency percentile queries fail.

One-time upload (do this once per environment):

# On Mac — download the binary
curl -L -o /tmp/histogram-quantile \
  https://github.com/SigNoz/signoz/releases/download/v0.130.1/histogram-quantile_linux_amd64

# Upload to GitLab Generic Package Registry (project 14)
curl -sk --request PUT \
  --header "PRIVATE-TOKEN: $TOKEN" \
  --upload-file /tmp/histogram-quantile \
  "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/14/packages/generic/histogram-quantile/v0.0.1/histogram-quantile_linux_amd64"

Create a read-only deploy token (one-time):

curl -sk --request POST \
  --header "PRIVATE-TOKEN: $TOKEN" \
  --header "Content-Type: application/json" \
  --data '{"name":"histogram-quantile-read","scopes":["read_package_registry"]}' \
  "https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/14/deploy_tokens" | jq .

Store in Vault:

vault kv put secret/gitlab/histogram-quantile-deploy-token \
  username="gitlab+deploy-token-1" \
  token="<token-from-above>"

The deploy token is embedded directly in the wget URL in values-mod-auh1-dev-ask.yaml. The Altinity ClickHouse operator renders its own pod spec from the ClickHouseInstallation CR and does not pass through env vars from Helm values for init containers — so env var injection does not work; credentials must be in the URL.

Why this runs on every pod start: The ClickHouse init container runs on every pod start, but the binary persists on the PVC at /var/lib/clickhouse/user_scripts/histogramQuantile. On subsequent starts it overwrites with the same binary — idempotent and fast.

Step 4 — Vault ESO policy¶

The eso-read policy must include secret/data/gitlab/*. This is managed via GitOps in applications/cluster-addons/vault/manifests/vault-bootstrap-config.yaml. Verify it is present:

vault policy read eso-read | grep gitlab
# Expected: path "secret/data/gitlab/*" { capabilities = ["read"] }

Deployment¶

Step 5 — Push devops repo¶

cd mod-ask-devops
git add environment/auh1/dev/02-argocd-application/_root/ \
        environment/auh1/dev/02-argocd-application/observability/ \
        applications/signoz/
git commit -m "feat(observability): add SigNoz deployment (chart 0.130.1)"
git push

Step 6 — ArgoCD sync¶

# Sync root app to pick up new AppProject + App-of-Apps
argocd app sync mod-auh1-dev-ask --grpc-web

# Sync SigNoz
argocd app sync mod-auh1-dev-ask.observability.signoz --grpc-web
kubectl get pods -n signoz -w

If sync is stuck:

argocd app terminate-op mod-auh1-dev-ask.observability.signoz --grpc-web
argocd app sync mod-auh1-dev-ask.observability.signoz --grpc-web

Verification¶

# All pods running
kubectl get pods -n signoz

# histogramQuantile installed
kubectl logs -n signoz chi-signoz-clickhouse-cluster-0-0-0 -c signoz-clickhouse-udf-init
# Expected: "histogramQuantile installed successfully"

# Ingress
curl -sk https://signoz.ask.mod.auh1.dev.dir | grep -i signoz

Expected pods (all Running):

chi-signoz-clickhouse-cluster-0-0-0   1/1   Running
signoz-clickhouse-operator-*          2/2   Running
signoz-zookeeper-0                    1/1   Running
signoz-otel-collector-* (x2)          1/1   Running
signoz-0                              1/1   Running

Default admin credentials: - Email: [email protected] - Password: set on first login

Post-Deploy¶

Set retention (25 days)¶

SigNoz UI → Settings → Retention Period → set Traces/Metrics/Logs to 25 days

Add DNS record¶

Add signoz.ask.mod.auh1.dev.dir to the internal DNS / /etc/hosts on bastion.

Troubleshooting¶

Symptom	Cause	Fix
ClickHouse `Pending`	PVC not bound — wrong StorageClass	Verify `csi-rbd-sc` exists; delete StatefulSet + PVC if VCT is immutable
`ImagePullBackOff`	Image not in Harbor or `harbor-pull-secret` missing	Label namespace: `kubectl label ns signoz harbor-pull-secret=true --overwrite`
OTel `check` init: `connection refused`	ClickHouse still starting	Normal — retries automatically; wait for ClickHouse pod Running
OTel `check` init: `Authentication failed`	ClickHouse PVC has old password from previous install	Delete CHI + PVC: `kubectl delete chi signoz-clickhouse -n signoz && kubectl delete pvc -n signoz <clickhouse-pvc>` then resync
`histogramQuantile` init: `401 Unauthorized`	Deploy token not in wget URL / operator ignores env vars	Verify token is embedded in URL in `values-mod-auh1-dev-ask.yaml` under `clickhouse.initContainers.udf.command`
`histogramQuantile` init: `connection refused`	GitHub URL in command (blocked in air-gap)	Use GitLab Generic Package Registry URL
ZooKeeper PVC on wrong StorageClass	VCT immutable on StatefulSet	Delete StatefulSet + PVC; add `clickhouse.zookeeper.persistence.storageClass` AND `clickhouse.zookeeper.global.storageClass` to values
CSI `Operation Already Exists`	CSI provisioner in-memory cache stale	`kubectl rollout restart deploy/ceph-csi-rbd-provisioner -n ceph-csi-rbd`
ArgoCD sync stuck	Another operation in progress	`argocd app terminate-op <app> --grpc-web` then retry sync
All pods 403 / auth failed from remote	Pod CIDR `172.32.0.0/14` not in default `allowedNetworkIps` (covers only to `172.31.x.x`)	Set `clickhouse.allowedNetworkIps: ["0.0.0.0/0"]` in values
Ingress `CLASS: <none>` / not routing	Chart key is `signoz.ingress.className` not `ingressClassName`	Use `className: nginx` under `signoz.ingress`
Ingress sync error: port required	Chart requires `port:` in each path spec	Add `port: 8080` to each path under `signoz.ingress.hosts[].paths[]`
StatefulSet forbidden update	Immutable VCT field changed	`kubectl delete statefulset signoz -n signoz --cascade=orphan` then resync
`cannot create agent without orgId`	Normal on first boot — no org created yet	Complete signup at the SigNoz UI to create admin account