Core Service — TOML Config Flow (Repo → ConfigMap → Container)¶

This page explains how the core-service application gets its configuration in our air-gapped environment deployment — from the Helm values file in Git, through an ArgoCD-rendered ConfigMap, into a file mounted in the running pod, and finally merged by Dynaconf at startup.

If you have ever asked "the image is pre-baked, so how does [mod-auh1-dev-ask.database] override anything at runtime?" — this page is the answer.

1. The mental model: base image + runtime patch¶

The single most important idea: one image, many environments. The container image is environment-agnostic — it ships only universal defaults. Everything specific to this cluster is injected at runtime.

flowchart LR
    subgraph IMG["🐳 Container image (baked once)"]
        D["/app/config/default.toml<br/><b>[default.*]</b> only<br/>host = localhost"]
    end
    subgraph RUN["☸️ Runtime injection (per environment)"]
        O["/app/config/override.toml<br/><b>[mod-auh1-dev-ask.*]</b><br/>host = core-pg-rw.ask…"]
    end
    IMG --> M["Dynaconf merges<br/>default + override"]
    RUN --> M
    M --> C["Final config<br/>host = core-pg-rw.ask…"]

    style IMG fill:#1e3a5f,stroke:#4a90d9,color:#fff
    style RUN fill:#1e5f3a,stroke:#4ad98a,color:#fff
    style M fill:#5f4a1e,stroke:#d9a94a,color:#fff
    style C fill:#3a1e5f,stroke:#a94ad9,color:#fff

The one-line summary

The image is the base. The ConfigMap is the patch. They live in the same directory inside the pod, and Dynaconf merges them at process startup. The image never contains environment-specific config — that is the whole point.

2. Two repos, two responsibilities¶

Our deployment splits cleanly across two Git repositories. Understanding this split removes most of the confusion.

flowchart TB
    subgraph SRC["📦 ask-core-service (FULL — app source)"]
        direction TB
        S1["src/ — Python code"]
        S2["config/default.toml — baked into image"]
        S3["Dockerfile"]
        S4["helm/ — the chart"]
    end

    subgraph SAN["📦 ask-core-service (SANITIZED — what lab GitLab holds)"]
        direction TB
        H1["helm/ ONLY"]
        H2["helm/environments/values-mod-auh1-dev-ask.yaml"]
        H3["helm/charts/zitadel-1.2.0.tgz"]
    end

    SRC -->|"docker build + push"| HARBOR["🐳 Harbor<br/>harbor.cl1.sq4.aegis.internal/ask/core-service:staging"]
    SRC -->|"sanitize: strip src/, config/, Dockerfile"| SAN
    SAN -->|"git push"| GL["🦊 lab GitLab<br/>gitlab.cl1.sq4.aegis.internal"]

    style SRC fill:#1e3a5f,stroke:#4a90d9,color:#fff
    style SAN fill:#1e5f3a,stroke:#4ad98a,color:#fff
    style HARBOR fill:#5f1e3a,stroke:#d94a8a,color:#fff
    style GL fill:#5f4a1e,stroke:#d9a94a,color:#fff

Repo	Contains	Who reads it	Why
ask-core-service (full, original)	`src/`, `config/default.toml`, `Dockerfile`, `helm/`	Docker build	Produces the image (with `default.toml` baked in)
ask-core-service (sanitized → lab GitLab)	`helm/` only	ArgoCD	Renders Kubernetes manifests

Why config/default.toml is NOT in the sanitized repo

It is application code, not infrastructure config. It already lives inside the image that was built from the original repo and pushed to Harbor. Duplicating it into the Helm chart would create two copies that can drift. The sanitized repo's only job is to give ArgoCD a chart to render.

3. The layered config model (Dynaconf)¶

The app uses Dynaconf with environment layering. Later sources override earlier ones; anything not overridden falls back to the layer beneath.

flowchart TB
    L1["<b>Layer 1 — default.toml</b> (baked in image)<br/>[default.*] — universal baseline"]
    L2["<b>Layer 2 — override.toml</b> (ConfigMap, runtime)<br/>[mod-auh1-dev-ask.*] — env-specific values"]
    L3["<b>Layer 3 — OS env vars</b> (Secret → env)<br/>SECTION__KEY + @format {env[…]} — secrets"]

    L1 --> MERGE["🔀 Dynaconf merge_enabled=True<br/>env_switcher = APP__ENV"]
    L2 --> MERGE
    L3 --> MERGE
    MERGE --> FINAL["✅ config.database.host<br/>config.iam_service.pat<br/>…"]

    style L1 fill:#1e3a5f,stroke:#4a90d9,color:#fff
    style L2 fill:#1e5f3a,stroke:#4ad98a,color:#fff
    style L3 fill:#5f1e3a,stroke:#d94a8a,color:#fff
    style MERGE fill:#5f4a1e,stroke:#d9a94a,color:#fff
    style FINAL fill:#3a1e5f,stroke:#a94ad9,color:#fff

Layer	Source	Set via	Use for
1	`default.toml` (baked)	Image build	Universal defaults only — never env-specific
2	`override.toml` (ConfigMap file)	`envConf.configToml` in Helm values	Non-secret env config: DB host, IAM URL, Redis host, feature flags
3	OS env vars (`SECTION__KEY`)	`extraEnv` / Secret → env	Secrets + one-off scalar overrides

The selector that ties it together: APP__ENV=mod-auh1-dev-ask (set in extraEnv) tells Dynaconf which […] section is the active environment. It must exactly match the TOML section headers ([mod-auh1-dev-ask.*]).

4. The end-to-end flow: Git string → file in the pod¶

This is the chain that turns a YAML string in a Git repo into a file the application reads.

flowchart TB
    A["<b>helm/environments/values-mod-auh1-dev-ask.yaml</b><br/>envConf.configToml: |<br/>&nbsp;&nbsp;[mod-auh1-dev-ask.database]<br/>&nbsp;&nbsp;host = core-pg-rw.ask…"]
    A -->|"ArgoCD: helm template"| B["<b>templates/configmap-config-toml.yaml</b><br/>{{- if .Values.envConf.configToml }}"]
    B --> C["<b>ConfigMap: core-service-config-toml</b><br/>data:<br/>&nbsp;&nbsp;override.toml: |<br/>&nbsp;&nbsp;&nbsp;&nbsp;[mod-auh1-dev-ask.database]…"]
    C -->|"kubelet volume mount (subPath)"| D["<b>Pod filesystem</b><br/>/app/config/override.toml"]
    D -->|"Dynaconf glob config/*.toml"| E["<b>Merged config</b> at process startup"]

    style A fill:#1e5f3a,stroke:#4ad98a,color:#fff
    style B fill:#5f4a1e,stroke:#d9a94a,color:#fff
    style C fill:#1e3a5f,stroke:#4a90d9,color:#fff
    style D fill:#5f1e3a,stroke:#d94a8a,color:#fff
    style E fill:#3a1e5f,stroke:#a94ad9,color:#fff

The Helm template (the guard that matters)¶

templates/configmap-config-toml.yaml

{{- if .Values.envConf.configToml }}     # ← empty string = NO ConfigMap created
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Values.envConf.configTomlConfigMapName }}   # core-service-config-toml
data:
  {{ base .Values.envConf.configTomlFileMountPath }}: | # override.toml
    {{- .Values.envConf.configToml | nindent 4 }}
{{- end }}

`envConf.configToml` value	ConfigMap created?	Volume mount added?
`""` (base `values.yaml` default)	❌ No	❌ No
Set in env values file	✅ Yes	✅ Yes

Both the ConfigMap and the Deployment's volume mount are wrapped in the same {{- if .Values.envConf.configToml }} guard — so they are always created together or not at all.

5. The critical detail: `subPath` mount¶

This is the part that confuses everyone. The override does not replace the baked-in file — it sits next to it in the same directory.

templates/deployment.yaml (lines 98-118)

volumeMounts:
  - name: config-toml
    mountPath: /app/config/override.toml   # full path to a single file
    subPath: override.toml                 # ← THE CRITICAL LINE
    readOnly: true
volumes:
  - name: config-toml
    configMap:
      name: core-service-config-toml

flowchart LR
    subgraph WITHOUT["❌ WITHOUT subPath — mount at /app/config/"]
        W1["/app/config/<br/>└── override.toml ONLY<br/>(default.toml HIDDEN)"]
        W2["💥 app breaks —<br/>defaults gone"]
        W1 --> W2
    end
    subgraph WITH["✅ WITH subPath: override.toml"]
        V1["/app/config/<br/>├── default.toml (image)<br/>└── override.toml (ConfigMap)"]
        V2["✔ both present —<br/>Dynaconf merges"]
        V1 --> V2
    end

    style WITHOUT fill:#5f1e1e,stroke:#d94a4a,color:#fff
    style WITH fill:#1e5f3a,stroke:#4ad98a,color:#fff
    style W2 fill:#5f1e1e,stroke:#d94a4a,color:#fff
    style V2 fill:#1e5f3a,stroke:#4ad98a,color:#fff

Why subPath is mandatory here

A ConfigMap volume mounted at a directory path (/app/config/) shadows the entire directory — the baked-in default.toml would vanish and the app would lose all its defaults. subPath injects only the single file override.toml, leaving default.toml untouched. This is the mechanism that lets the runtime patch coexist with the baked base.

Resulting filesystem inside the running container:

/app/config/
├── default.toml    ← from image  (baked,    [default.*])
└── override.toml   ← from ConfigMap (runtime, [mod-auh1-dev-ask.*])

6. How Dynaconf merges at startup¶

src/config/__init__.py (baked in the image)

CONFIG_DIR = Path.cwd() / "config"            # = /app/config  (WORKDIR is /app)
default_config = CONFIG_DIR / "default.toml"
_all_configs = sorted(CONFIG_DIR.glob("*.toml") ...)   # ← globs BOTH files

_raw = Dynaconf(
    envvar_prefix=False,
    load_dotenv=True,
    settings_files=[default_config, *env_configs, ...],  # default FIRST, override AFTER
    environments=True,
    merge_enabled=True,            # later files merge ON TOP of earlier
    env_switcher="APP__ENV",       # APP__ENV picks the active [...] section
).as_dict()

sequenceDiagram
    participant K as kubelet
    participant P as Pod (core-service)
    participant DC as Dynaconf
    participant APP as App config object

    K->>P: mount override.toml (subPath) into /app/config/
    K->>P: inject env vars (APP__ENV, secrets) from Secret
    P->>DC: import config → bootstrap
    DC->>DC: glob /app/config/*.toml → [default.toml, override.toml]
    DC->>DC: load default.toml → [default.*] baseline
    DC->>DC: load override.toml → [mod-auh1-dev-ask.*]
    DC->>DC: APP__ENV=mod-auh1-dev-ask → activate that section
    DC->>DC: merge_enabled → override layered over default
    DC->>DC: resolve @format {env[KEY]} → read Secret env vars
    DC->>APP: config.database.host = core-pg-rw.ask.svc.cluster.local

Worked example¶

# default.toml (baked)              # override.toml (ConfigMap)
[default.database]                  [mod-auh1-dev-ask.database]
  host = "localhost"                  host = "core-pg-rw.ask.svc.cluster.local"
  pool_size = 12                      pool_size = 5
  connection_prefix = "postgresql…"   # (not specified)

With APP__ENV=mod-auh1-dev-ask, the final resolved config is:

Key	Value	Came from
`database.host`	`core-pg-rw.ask.svc.cluster.local`	override (Layer 2)
`database.pool_size`	`5`	override (Layer 2)
`database.connection_prefix`	`postgresql+psycopg2`	default (Layer 1 fallback)
`database.password`	(from Secret)	env var `DATABASE__PASSWORD` (Layer 3)

This is why the env file only specifies what differs — everything else falls through to defaults.

7. How this works in OUR current deployment (ArgoCD)¶

Now the full picture, including GitOps. ArgoCD uses a multi-source Application: the chart comes from one repo, extra values from another, and the image from Harbor.

flowchart TB
    subgraph GIT["🦊 lab GitLab (gitlab.cl1.sq4.aegis.internal)"]
        R1["ask-core-service.git<br/>helm/ + environments/values-mod-auh1-dev-ask.yaml"]
        R2["mod-ask-devops.git<br/>applications/core-service/values-mod-auh1-dev-ask.yaml<br/>+ ExternalSecrets"]
    end
    subgraph HARBOR["🐳 Harbor"]
        IMG["ask/core-service:staging<br/>(default.toml baked in)"]
    end
    subgraph VAULT["🔐 Vault + ESO"]
        SEC["core-service-secret<br/>(APP__CORE_SERVICES_MACHINE_TOKEN, IAM_SERVICE__PAT, …)"]
    end

    ARGO["🚢 ArgoCD Application<br/>mod-auh1-dev-ask.ask.core-service<br/>(multi-source)"]
    R1 -->|"Source 1: chart + env values"| ARGO
    R2 -->|"Source 2: $values overrides"| ARGO
    ARGO -->|"helm template → apply"| K8S

    subgraph K8S["☸️ ask namespace"]
        CM["ConfigMap: core-service-config-toml"]
        DEP["Deployment: core-service"]
        CM -.->|"subPath mount"| DEP
        SEC -.->|"secretRef → env"| DEP
        IMG -.->|"image pull"| DEP
    end

    style GIT fill:#5f4a1e,stroke:#d9a94a,color:#fff
    style HARBOR fill:#5f1e3a,stroke:#d94a8a,color:#fff
    style VAULT fill:#1e3a5f,stroke:#4a90d9,color:#fff
    style ARGO fill:#1e5f3a,stroke:#4ad98a,color:#fff
    style K8S fill:#3a1e5f,stroke:#a94ad9,color:#fff

The ArgoCD multi-source spec¶

mod-ask-devops/…/argocd-application-core-service.yaml

spec:
  sources:
    # Source 1: the Helm chart + chart-internal env values
    - repoURL: 'https://gitlab.cl1.sq4.aegis.internal/ai71/ask71/ask-core-service.git'
      targetRevision: main
      path: helm
      helm:
        releaseName: core-service
        valueFiles:
          - environments/values-mod-auh1-dev-ask.yaml            # ← contains configToml
          - $values/applications/core-service/values-mod-auh1-dev-ask.yaml
    # Source 2: provides the $values ref + ExternalSecrets
    - repoURL: 'https://gitlab.cl1.sq4.aegis.internal/ai71/ask71/mod-ask-devops.git'
      targetRevision: main
      ref: values

Two values files, stacked

environments/values-mod-auh1-dev-ask.yaml (in the chart repo) — the primary env config including the full configToml block. Matches the production pattern where every environment has its own file inside the chart.
$values/applications/core-service/values-mod-auh1-dev-ask.yaml (in mod-ask-devops) — stacked on top for any devops-level overrides. Helm applies value files left-to-right, so the second wins on conflicts.

Where secrets come from (never in the TOML file)¶

flowchart LR
    V["🔐 Vault KV<br/>secret/…/core-service"] -->|"ExternalSecret"| ESO["ESO"]
    ESO -->|"creates"| KS["K8s Secret<br/>core-service-secret"]
    KS -->|"envConf.secretRef → envFrom"| POD["Pod env vars<br/>IAM_SERVICE__PAT=…"]
    POD -->|"@format {env[IAM_SERVICE__PAT]}"| TOML["resolved in override.toml<br/>at Dynaconf parse time"]

    style V fill:#1e3a5f,stroke:#4a90d9,color:#fff
    style ESO fill:#1e5f3a,stroke:#4ad98a,color:#fff
    style KS fill:#5f4a1e,stroke:#d9a94a,color:#fff
    style POD fill:#5f1e3a,stroke:#d94a8a,color:#fff
    style TOML fill:#3a1e5f,stroke:#a94ad9,color:#fff

The TOML carries only placeholders:

[mod-auh1-dev-ask.iam_service]
pat = "@format {env[IAM_SERVICE__PAT]}"   # filled from env var at parse time, NOT stored in Git

Secrets rule

Never put a real secret value inside configToml. The file becomes a ConfigMap (plaintext, in Git). Secrets flow only through Vault → ESO → core-service-secret → env var, referenced by the @format {env[KEY]} token.

8. The three runtime injection paths — when to use which¶

flowchart TB
    Q{"What are you<br/>injecting?"}
    Q -->|"A whole env section<br/>(many keys)"| P2["✅ Path 2: configToml<br/>→ ConfigMap file"]
    Q -->|"A few scalar overrides"| P3["Path 3: extraEnv<br/>SECTION__KEY env vars"]
    Q -->|"Secrets / tokens"| P3S["Path 3: Secret → env<br/>+ @format {env[KEY]} in TOML"]
    Q -->|"Universal default<br/>(all envs)"| P1["Path 1: default.toml<br/>(rebuild image)"]

    style P2 fill:#1e5f3a,stroke:#4ad98a,color:#fff
    style P3 fill:#5f4a1e,stroke:#d9a94a,color:#fff
    style P3S fill:#5f1e3a,stroke:#d94a8a,color:#fff
    style P1 fill:#1e3a5f,stroke:#4a90d9,color:#fff

Scenario	Path	Mechanism
Whole `[mod-auh1-dev-ask.*]` block, many keys	2 — `configToml` file ✅ (what we use)	ConfigMap → `override.toml` subPath mount
A handful of scalar overrides	3 — `extraEnv` env vars	`DATABASE__HOST=…` (double-underscore = section/key)
Secrets (tokens, passwords)	3 — Secret → env	`@format {env[KEY]}` placeholder in TOML
A new default for every environment	1 — `default.toml`	Edit original repo, rebuild + re-mirror image

9. Troubleshooting: `configmap "core-service-config-toml" not found`¶

Symptom:

MountVolume.SetUp failed for volume "config-toml": configmap "core-service-config-toml" not found

Cause: ArgoCD rendered the Deployment's volume mount but the ConfigMap was not created. Because both are guarded by {{- if .Values.envConf.configToml }}, this only happens when ArgoCD renders with an empty configToml — almost always because the env values file that sets it is missing from the chart repo (committed locally but not pushed), so ArgoCD's helm template either fails or falls back to the base configToml: "".

flowchart TB
    ERR["❌ ConfigMap not found"]
    ERR --> Q1{"Is environments/<br/>values-mod-auh1-dev-ask.yaml<br/>pushed to GitLab?"}
    Q1 -->|No| F1["git push origin main<br/>→ hard-refresh ArgoCD"]
    Q1 -->|Yes| Q2{"Does helm template<br/>render the ConfigMap<br/>locally?"}
    Q2 -->|No| F2["Check configToml is<br/>non-empty in the values file"]
    Q2 -->|Yes| F3["ArgoCD hard-refresh<br/>(clear manifest cache)"]

    style ERR fill:#5f1e1e,stroke:#d94a4a,color:#fff
    style F1 fill:#1e5f3a,stroke:#4ad98a,color:#fff
    style F2 fill:#1e5f3a,stroke:#4ad98a,color:#fff
    style F3 fill:#1e5f3a,stroke:#4ad98a,color:#fff

Verify locally before pushing:

cd helm
helm template core-service . \
  -f environments/values-mod-auh1-dev-ask.yaml \
  --namespace ask | grep -A2 "name: core-service-config-toml"

A non-empty result means the ConfigMap will render. If it does, push the file and hard-refresh ArgoCD. See also the Troubleshooting page for the "no such file or directory" manifest-generation error (same root cause: unpushed env file).

10. Summary checklist for deploying this service¶

[ ] Image built from original repo, pushed to Harbor (ask/core-service:staging) — default.toml baked in
[ ] Sanitized helm/ pushed to lab GitLab — including helm/environments/values-mod-auh1-dev-ask.yaml
[ ] envConf.configToml set with the full [mod-auh1-dev-ask.*] block (non-secret config)
[ ] extraEnv sets APP__ENV: mod-auh1-dev-ask (matches TOML section headers)
[ ] Secrets in Vault → ExternalSecret → core-service-secret; TOML uses @format {env[KEY]} only
[ ] ArgoCD multi-source Application points Source 1 at environments/values-mod-auh1-dev-ask.yaml
[ ] helm template renders core-service-config-toml ConfigMap locally before push
[ ] ArgoCD synced; ConfigMap present in ask namespace; pod mounts /app/config/override.toml