26 — Deploying a Multi-Microservice Application (Pattern)

The reusable pattern for onboarding a whole application made of many microservices onto the cluster using the 3-layer App of Apps hierarchy — one scoped AppProject per group, one Application per service, traceable names modgpt.<app>.<service>.

Worked example → ASK

For a concrete, fill-in-the-blanks onboarding of a real app, see 27 — ASK Application Onboarding. The creation runbook is in How-To → Create a New App Group.

---

The Pattern at a Glance

LAYER 1 — modgpt (root, shared across all apps)
  AppProject: modgpt
  Application: modgpt  ──→  cluster-addons/root-bootstrap/

LAYER 2 — one entry per application group
  AppProject: modgpt.<app>           ← scopes this app's repos + namespaces
  Application: modgpt.<app>  ──→  <app>/bootstrap/

LAYER 3 — one entry per microservice
  Application: modgpt.<app>.postgres   ← Helm chart, wave 0
  Application: modgpt.<app>.api        ← Helm chart, wave 1
  Application: modgpt.<app>.worker     ← Helm chart, wave 1
  Application: modgpt.<app>.frontend   ← Helm chart, wave 2

Each Application is a plain YAML file in bootstrap/ of its app's git repo. No ApplicationSet, no generators, no templates. Every deployed service is explicit, auditable, and independently rollback-able.

---

Key Design Decisions

Question	Answer	Rationale
One Application per microservice or umbrella chart?	One per microservice	Independent sync/health/rollback — umbrella charts hide which service is broken
ApplicationSet or App of Apps?	Pure App of Apps	Explicit YAML per service — no auto-discovery surprises, full audit trail
Per-service AppProject?	No — reuse Layer 2 modgpt.<app>	Layer 2 AppProject already scopes sources + destinations; per-service AppProjects add overhead with no RBAC benefit for a single operator
New AppProject per app group?	Yes — one modgpt.<app> AppProject	Each application group gets its own scoped boundary
Naming convention?	modgpt.<app>.<service>	Traceable from L1 → L3 in the ArgoCD portal

---

Repository Layout (per application)

Each application has its own GitLab repo (gitlab.cl1.sq4.aegis.internal/modgpt/<app>):

<app>/
├── bootstrap/                        ← Layer 2 app points here
│   ├── application-postgres.yaml     ← Layer 3: data store (wave 0)
│   ├── application-api.yaml          ← Layer 3: API server (wave 1)
│   ├── application-worker.yaml       ← Layer 3: background worker (wave 1)
│   └── application-frontend.yaml     ← Layer 3: frontend (wave 2)
│
├── charts/                           ← in-house Helm chart (if no public chart)
│   └── <app>-service/
│       ├── Chart.yaml
│       └── templates/
│
└── values/                           ← Helm values per service
    ├── postgres/<env>.yaml
    ├── api/<env>.yaml
    ├── worker/<env>.yaml
    └── frontend/<env>.yaml

The cluster-addons repo (root-bootstrap/) holds the Layer 2 pair that points to this repo:

cluster-addons/root-bootstrap/
  appproject-<app>.yaml     ← Layer 2 AppProject (sync-wave: -1)
  application-<app>.yaml    ← Layer 2 Application (sync-wave:  0)

---

Layer 3 Application YAML (template)

Each service is one Application file. The structure is identical for every service — only name, namespace, chart, and values differ:

# bootstrap/application-api.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: modgpt.<app>.api
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"       # after data stores (wave 0)
spec:
  project: modgpt.<app>                      # Layer 2 AppProject
  destination:
    server: https://kubernetes.default.svc
    namespace: <app>
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [CreateNamespace=true]
  sources:
    - repoURL: https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/<PID>/packages/helm/stable
      chart: <app>-service
      targetRevision: "1.0.0"
      helm:
        releaseName: api                      # short name — no dots
        valueFiles: ["$values/values/api/api-stg.yaml"]
    - repoURL: https://gitlab.cl1.sq4.aegis.internal/modgpt/<app>.git
      targetRevision: HEAD
      ref: values

Sync wave ordering

Wave	Services	Examples
0	Data stores	PostgreSQL, Redis, Kafka
1	Orchestrators + backend	Hatchet, core-service, worker
2	Consumers + frontend	Web UI, ingress

---

Cross-Cutting Concerns

Concern	Implementation
Secrets	Vault KV v2 (secret/<app>/*) → VSO VaultStaticSecret renders into app namespace. Never in values files.
Images	Harbor container registry (harbor.cl1.sq4.aegis.internal/ask/<image>). Pull secret in every app namespace.
Ingress + TLS	cert-manager issues the <app>.mod.auh1.dev.dir cert from the internal CA. One ingress on the frontend/gateway service.
Resource limits	Every service sets requests + limits. No burstable/BestEffort pods in production.
Health checks	Liveness + readiness probes on every Deployment so ArgoCD health is meaningful.
Stateful services	HA + PDB + OnDelete update strategy. Roll one pod at a time. See KB-01 — Raft / Quorum Clusters.
Rollback	Per-service: argocd app rollback modgpt.<app>.<svc> — rolls back only the broken microservice.

---

Adding a New Microservice to an Existing App

No changes to cluster-addons. Only the app's own repo:

cd ~/<app>
mkdir -p values/<newsvc>
vi values/<newsvc>/<newsvc>-stg.yaml

cat > bootstrap/application-<newsvc>.yaml <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: modgpt.<app>.<newsvc>
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  project: modgpt.<app>
  destination:
    server: https://kubernetes.default.svc
    namespace: <app>
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [CreateNamespace=true]
  sources:
    - repoURL: https://gitlab.cl1.sq4.aegis.internal/api/v4/projects/<PID>/packages/helm/stable
      chart: <newsvc>
      targetRevision: "1.0.0"
      helm:
        releaseName: <newsvc>
        valueFiles: ["$values/values/<newsvc>/<newsvc>-stg.yaml"]
    - repoURL: https://gitlab.cl1.sq4.aegis.internal/modgpt/<app>.git
      targetRevision: HEAD
      ref: values
EOF

git add bootstrap/ values/
git commit -m "feat: add <newsvc> microservice"
git push origin main
# modgpt.<app> creates modgpt.<app>.<newsvc> automatically

---

Full Platform View

ArgoCD portal after onboarding two apps:

modgpt                               L1 — root
  modgpt.cluster-addons              L2 — platform add-ons
    modgpt.cluster-addons.vault      L3
    modgpt.cluster-addons.vso        L3
    modgpt.cluster-addons.reloader   L3
    modgpt.cluster-addons.pdb        L3
  modgpt.ask                         L2 — ASK application
    modgpt.ask.postgres              L3 — wave 0
    modgpt.ask.core-service          L3 — wave 1
    modgpt.ask.hatchet               L3 — wave 1
    modgpt.ask.frontend              L3 — wave 2

---

➡️ Next: apply this pattern to a real app — 27 — ASK Application Onboarding