SigNoz — Infrastructure & ASK Service Metrics¶

Covers two instrumentation layers:

k8s-infra — DaemonSet + Deployment that collects RKE2 node/cluster metrics and forwards to SigNoz
ASK service OTel — env vars added to each ASK service to emit traces, metrics, and logs

Architecture¶

RKE2 Nodes
  └─ k8s-infra DaemonSet (hostmetrics: CPU/mem/disk/net per node)
  └─ k8s-infra Deployment (k8s cluster metrics: pod/node/deployment state)
        │  OTLP gRPC :4317
        ▼
SigNoz OTel Collector gateway (signoz-otel-collector.signoz.svc.cluster.local:4317)
        │
        ▼
ClickHouse → SigNoz UI

ASK Services (core-service, assistant-service, KMS, Hatchet)
  └─ OTel SDK env vars → OTLP gRPC → SigNoz OTel Collector

Part 1 — k8s-infra (RKE2 Infrastructure Metrics)¶

Step 1 — Mirror k8s-infra chart¶

Add to mirror-charts.yaml and gitlab-charts.yaml:

# mirror-charts.yaml
- name: k8s-infra
  repo: https://charts.signoz.io
  version: "0.13.3"
  harbor_project: signoz

# gitlab-charts.yaml
- name: k8s-infra
  version: "0.13.3"

Mirror images required (add to mirror-config.yaml):

Image	Tag
`signoz/signoz-otel-collector`	`v0.144.5` (already mirrored)
`otel/opentelemetry-collector-contrib`	`0.116.0`

Run on bastion:

python3 mirror-images.py --filter k8s-infra
python3 core42-mirror-charts.py --config mirror-charts.yaml --filter k8s-infra
python3 gitlab-push-charts.py -c gitlab-charts.yaml --only k8s-infra

Step 2 — ArgoCD Application¶

Create environment/auh1/dev/02-argocd-application/observability/argocd-application-k8s-infra.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: mod-auh1-dev-ask.observability.k8s-infra
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.io
spec:
  project: mod-auh1-dev-ask.observability
  sources:
    - repoURL: 'https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/14/packages/helm/stable'
      chart: k8s-infra
      targetRevision: "0.13.3"
      helm:
        releaseName: k8s-infra
        valueFiles:
          - $values/applications/k8s-infra/values-mod-auh1-dev-ask.yaml
    - repoURL: 'https://gitlab.cl1.sq4.aegis.internal/ai71/ask71/mod-ask-devops.git'
      targetRevision: main
      ref: values
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: k8s-infra
  syncPolicy:
    automated:
      prune: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
    managedNamespaceMetadata:
      labels:
        pod-security.kubernetes.io/enforce: privileged
        harbor-pull-secret: "true"

Privileged PSA

The k8s-infra DaemonSet needs host network + pid access to collect node metrics. Namespace must be privileged PSA.

Step 3 — Values file¶

applications/k8s-infra/values-mod-auh1-dev-ask.yaml:

global:
  deploymentEnvironment: mod-auh1-dev-ask
  imagePullSecrets:
    - name: harbor-pull-secret

otelCollectorEndpoint: "http://signoz-otel-collector.signoz.svc.cluster.local:4317"

otelCollector:
  image:
    repository: harbor.cl1.sq4.aegis.internal/dockerhub/signoz/signoz-otel-collector
    tag: v0.144.5
    pullPolicy: IfNotPresent

otelAgent:
  image:
    repository: harbor.cl1.sq4.aegis.internal/dockerhub/signoz/signoz-otel-collector
    tag: v0.144.5
    pullPolicy: IfNotPresent

Part 2 — ASK Service OTel Instrumentation¶

Each ASK service (core-service, assistant-service, KMS, Hatchet) needs OTel env vars added to its Helm values.

OTel env vars (add to each service's values)¶

extraEnvVars:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://signoz-otel-collector.signoz.svc.cluster.local:4317"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "grpc"
  - name: OTEL_SERVICE_NAME
    value: "<service-name>"   # core-service, assistant-service, kms, hatchet
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: "deployment.environment=mod-auh1-dev-ask"
  - name: OTEL_TRACES_EXPORTER
    value: "otlp"
  - name: OTEL_METRICS_EXPORTER
    value: "otlp"
  - name: OTEL_LOGS_EXPORTER
    value: "otlp"

Service name mapping¶

Service	`OTEL_SERVICE_NAME`
core-service	`core-service`
assistant-service	`assistant-service`
KMS	`knowledge-management-service`
Hatchet	`hatchet`

SDK required

OTel env vars only work if the service has the OTel SDK built in. Verify with the service team that each app initialises the SDK on startup. Without the SDK, the env vars are ignored.

Verification¶

After k8s-infra is deployed:

# DaemonSet running on all nodes
kubectl get daemonset -n k8s-infra

# Check collector sending metrics
kubectl logs -n k8s-infra -l app=k8s-infra-otel-agent --tail=20

In SigNoz UI: - Infrastructure → should show RKE2 nodes with CPU/memory/disk - Services → should show ASK services after OTel SDK instrumentation

Troubleshooting¶

Symptom	Cause	Fix
DaemonSet pods `Pending`	PSA not `privileged`	Label namespace: `pod-security.kubernetes.io/enforce: privileged`
No node metrics in UI	Wrong OTel collector endpoint	Verify `otelCollectorEndpoint` points to `signoz-otel-collector.signoz.svc.cluster.local:4317`
Services not appearing	OTel SDK not initialised in app	Check app startup logs for OTel initialisation
`ImagePullBackOff`	Image not in Harbor	Mirror `otel/opentelemetry-collector-contrib`