Phase 9 — AppHAProxy + Connectivity Checks¶

Deploy HA Proxy (Active/Passive via Keepalived) for the mod.auh1.dev.dir zone services, then verify Harbor and S3 (Ceph RGW) are reachable from the cluster.

Architecture¶

DNS: *.mod.auh1.dev.dir → 100.115.2.210 (VIP)
                                │
              ┌─────────────────┴──────────────────┐
       apphaproxy1                          apphaproxy2
       100.115.1.201 (ACTIVE)              100.115.1.202 (STANDBY)
              │                                    │
              └──────── Keepalived VIP ────────────┘
                        100.115.2.210

              HAProxy mode=tcp (SNI passthrough)
                         │
         ┌───────────────┼───────────────┐
   k8s-worker1..7  (100.115.1.54–60)
   rke2-ingress-nginx DaemonSet (kube-system)
   :443 / :80

Cert-manager terminates TLS inside the cluster.
HAProxy passes TLS through — no certs on HAProxy.

Masters (100.115.1.51–53) run the ingress DaemonSet too but are excluded
from HAProxy backends — application traffic should not flow through control-plane.

Item	Value
apphaproxy1	`100.115.1.201`
apphaproxy2	`100.115.1.202`
VIP	`100.115.2.210`
HAProxy mode	TCP passthrough (SNI)
Keepalived	VRRP — active/passive
DNS A record	`*.mod.auh1.dev.dir → 100.115.2.210`

Why TCP passthrough, not HTTP mode? cert-manager issues TLS certificates inside the cluster and Nginx ingress terminates them. If HAProxy also terminated TLS it would need copies of all certs and a rotation process. TCP passthrough lets HAProxy be cert-free and decoupled from cert rotation entirely.

Step 9.1 — Kubernetes Ingress Node IPs (Confirmed)¶

RKE2 ships a bundled ingress controller (rke2-ingress-nginx) as a DaemonSet in kube-system — there is no separate ingress-nginx namespace.

Confirmed worker nodes (HAProxy backends):

Node	IP
k8s-worker1	`100.115.1.54`
k8s-worker2	`100.115.1.55`
k8s-worker3	`100.115.1.56`
k8s-worker4	`100.115.1.57`
k8s-worker5	`100.115.1.58`
k8s-worker6	`100.115.1.59`
k8s-worker7	`100.115.1.60`

Masters (100.115.1.51–53) also run the DaemonSet but are excluded from HAProxy backends — control-plane nodes should not carry application traffic in production.

# Verify ingress pods are running
kubectl get pods -n kube-system | grep rke2-ingress-nginx-controller
# Expected: 17 pods Running (one per node including masters and workers; GPU nodes NotReady)

Step 9.2 — Install HAProxy + Keepalived on Both Nodes¶

Run on both apphaproxy1 and apphaproxy2:

sudo apt-get update
sudo apt-get install -y haproxy keepalived

Verify versions:

haproxy -v
keepalived --version

Step 9.3 — HAProxy Configuration¶

Why TCP Passthrough (not SSL Termination)¶

The existing HAProxy config used SSL termination (bind *:443 ssl crt /etc/haproxy/certs/svc-wildcard.pem). This means HAProxy decrypted every HTTPS connection using its own wildcard cert (app.cl1.sq4.aegis.internal), then re-encrypted to the backend. Consequences:

Every browser sees app.cl1.sq4.aegis.internal regardless of the hostname they connected to
cert-manager certs (per-service, properly scoped) are never presented to clients
cert rotation requires updating HAProxy's cert file — not automated
HAProxy becomes a TLS bottleneck and a secret store

Production-correct approach: TCP passthrough. HAProxy inspects the SNI field in the TLS ClientHello (without decrypting it) and forwards the raw TCP stream to the backend. nginx ingress terminates TLS using the cert-manager-issued cert for that specific hostname. cert-manager handles rotation automatically.

Ansible-managed config

The existing haproxy.cfg has a # managed by Ansible header. Applying this config manually will be overwritten on the next Ansible run. In production, this change should be made in the Ansible playbook/role that manages HAProxy.

The same haproxy.cfg goes on both nodes (backends are the same — only the VIP owner differs).

Write /etc/haproxy/haproxy.cfg:

#---------------------------------------------------------------------
# HAProxy configuration — TCP passthrough for Kubernetes ingress
#---------------------------------------------------------------------

global
    log         /dev/log local0
    log         /dev/log local1 notice
    chroot      /var/lib/haproxy
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    ssl-default-bind-ciphers PROFILE=SYSTEM
    ssl-default-server-ciphers PROFILE=SYSTEM

defaults
    log                     global
    option                  dontlognull
    option                  redispatch
    retries                 3
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout tunnel          1h
    maxconn                 3000

#---------------------------------------------------------------------
# Stats
#---------------------------------------------------------------------
frontend stats
    bind *:8404
    mode http
    stats enable
    stats uri /stats
    stats refresh 10s
    stats auth admin:changeme
    stats admin if TRUE

#---------------------------------------------------------------------
# HTTP — redirect all port 80 to HTTPS
#---------------------------------------------------------------------
frontend ingress_http_front
    bind 100.115.2.210:80
    mode http
    option httplog
    redirect scheme https code 301

#---------------------------------------------------------------------
# HTTPS — TCP passthrough (SNI inspection, no TLS termination)
#---------------------------------------------------------------------
frontend ingress_https_front
    bind 100.115.2.210:443
    mode tcp
    option tcplog
    tcp-request inspect-delay 5s
    tcp-request content accept if { req_ssl_hello_type 1 }
    default_backend ingress_back

#---------------------------------------------------------------------
# Backend — all 7 worker nodes running rke2-ingress-nginx
# Masters (100.115.1.51-53) excluded — no application traffic on control-plane
#---------------------------------------------------------------------
backend ingress_back
    mode tcp
    balance roundrobin
    option ssl-hello-chk
    server k8s-worker1 100.115.1.54:443 check inter 5s rise 2 fall 3
    server k8s-worker2 100.115.1.55:443 check inter 5s rise 2 fall 3
    server k8s-worker3 100.115.1.56:443 check inter 5s rise 2 fall 3
    server k8s-worker4 100.115.1.57:443 check inter 5s rise 2 fall 3
    server k8s-worker5 100.115.1.58:443 check inter 5s rise 2 fall 3
    server k8s-worker6 100.115.1.59:443 check inter 5s rise 2 fall 3
    server k8s-worker7 100.115.1.60:443 check inter 5s rise 2 fall 3

Apply on both nodes:

# Validate first
sudo haproxy -c -f /etc/haproxy/haproxy.cfg

# Apply
sudo systemctl restart haproxy
sudo systemctl status haproxy

Step 9.4 — Keepalived Configuration (VIP Failover)¶

apphaproxy1 — MASTER (`/etc/keepalived/keepalived.conf`)¶

! Keepalived config — apphaproxy1 MASTER
global_defs {
    router_id apphaproxy1
}

vrrp_script chk_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight -10
}

vrrp_instance VI_1 {
    state MASTER
    interface ens3              # ← replace with actual interface (ip a)
    virtual_router_id 51
    priority 110
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass changeme123   # ← replace with a real shared password
    }

    virtual_ipaddress {
        100.115.2.210/24
    }

    track_script {
        chk_haproxy
    }
}

apphaproxy2 — BACKUP (`/etc/keepalived/keepalived.conf`)¶

! Keepalived config — apphaproxy2 BACKUP
global_defs {
    router_id apphaproxy2
}

vrrp_script chk_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight -10
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens3              # ← replace with actual interface (ip a)
    virtual_router_id 51
    priority 100
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass changeme123   # ← same as MASTER
    }

    virtual_ipaddress {
        100.115.2.210/24
    }

    track_script {
        chk_haproxy
    }
}

Apply on both nodes:

sudo systemctl restart keepalived
sudo systemctl enable keepalived
sudo systemctl status keepalived

Check VIP is owned by apphaproxy1:

# On apphaproxy1
ip addr show | grep 100.115.2.210
# Expected: inet 100.115.2.210/24 scope global secondary ens3

# On apphaproxy2
ip addr show | grep 100.115.2.210
# Expected: (empty — not active)

Verify VIP is pingable from bastion:

ping -c 3 100.115.2.210

Step 9.5 — Update DNS Records¶

Point all mod.auh1.dev.dir zone services at the VIP. Harbor and S3 point directly at their dedicated IPs (no HAProxy needed).

DNS Record	Type	Value	Via
`vault.mod.auh1.dev.dir`	A	`100.115.2.210`	HAProxy → K8s ingress
`argocd.mod.auh1.dev.dir`	A	`100.115.2.210`	HAProxy → K8s ingress
`gitlab.mod.auh1.dev.dir`	A	`100.115.2.210`	HAProxy → K8s ingress
`ask.mod.auh1.dev.dir`	A	`100.115.2.210`	HAProxy → K8s ingress
`api.mod.auh1.dev.dir`	A	`100.115.2.210`	HAProxy → K8s ingress
`hatchet.mod.auh1.dev.dir`	A	`100.115.2.210`	HAProxy → K8s ingress (wildcard covers)
`harbor.mod.auh1.dev.dir`	A	`100.115.1.101`	Direct
`s3.mod.auh1.dev.dir`	A	`100.115.1.130`	Direct

Harbor and S3 bypass HAProxy

Harbor has its own load balancer at 100.115.1.101 and Ceph RGW at 100.115.1.130. They do not route through HAProxy — DNS points directly at their IPs.

Step 9.6 — Connectivity Checks¶

9.6a — Harbor Registry¶

# From bastion — verify Harbor API responds
curl -sk https://harbor.cl1.sq4.aegis.internal/api/v2.0/systeminfo \
  | python3 -m json.tool | grep -E '"harbor_version"|"with_chartmuseum"'

# Test image push/pull capability — login
docker login harbor.cl1.sq4.aegis.internal \
  -u "robot\$ask" -p "<robot-password>"

# List all Harbor projects
curl -sk -u "robot\$ask:<robot-password>" \
  "https://harbor.cl1.sq4.aegis.internal/api/v2.0/projects" \
  | python3 -m json.tool | grep '"name"'

# Verify reloader image is present
curl -sk -u "robot\$ask:<robot-password>" \
  "https://harbor.cl1.sq4.aegis.internal/api/v2.0/projects/ghcr/repositories" \
  | python3 -m json.tool | grep '"name"'
# Expected: "ghcr/stakater/reloader"

Expected output:

"harbor_version": "v2.x.x"

9.6b — S3 / Ceph RGW¶

S3 endpoint is HTTPS only

Ceph RGW at 100.115.1.130 listens on HTTPS port 443 only. HTTP port 80 and port 7480 (Ceph default) are not exposed. Always use https://s3.cl1.sq4.aegis.internal as the endpoint URL.

# Verify RGW is up (HTTPS)
curl -sk https://s3.cl1.sq4.aegis.internal/
# Expected: XML response — <?xml version="1.0" ?> or ListAllMyBucketsResult

# Test with AWS CLI
export AWS_ACCESS_KEY_ID=<ceph-s3-access-key>
export AWS_SECRET_ACCESS_KEY=<ceph-s3-secret-key>
export AWS_DEFAULT_REGION=us-east-1

# List buckets
aws s3 ls --endpoint-url https://s3.cl1.sq4.aegis.internal --no-verify-ssl

# Create a test bucket
aws s3 mb s3://connectivity-test --endpoint-url https://s3.cl1.sq4.aegis.internal --no-verify-ssl

# Upload a test object
echo "test" | aws s3 cp - s3://connectivity-test/test.txt \
  --endpoint-url https://s3.cl1.sq4.aegis.internal --no-verify-ssl

# Read it back
aws s3 cp s3://connectivity-test/test.txt - \
  --endpoint-url https://s3.cl1.sq4.aegis.internal --no-verify-ssl

# Cleanup
aws s3 rm s3://connectivity-test/test.txt \
  --endpoint-url https://s3.cl1.sq4.aegis.internal --no-verify-ssl
aws s3 rb s3://connectivity-test \
  --endpoint-url https://s3.cl1.sq4.aegis.internal --no-verify-ssl

9.6c — HAProxy Stats Page¶

Once HAProxy is running:

curl -sk http://100.115.2.210:8404/stats -u admin:changeme | grep -E "k8s-ingress|UP|DOWN"

Expected:

k8s-ingress/ingress-1  ... UP
k8s-ingress/ingress-2  ... UP

Step 9.7 — End-to-End Test via VIP¶

Test that the VIP correctly routes to ArgoCD (already deployed):

# ArgoCD must be accessible via the new hostname
curl -sk https://argocd.mod.auh1.dev.dir/healthz
# Expected: ok

# Or open in browser:
# https://argocd.mod.auh1.dev.dir

Step 9.8 — Failover Test¶

Verify VIP fails over correctly:

# Simulate apphaproxy1 failure — stop keepalived on MASTER
ssh [email protected] "sudo systemctl stop keepalived"

# Check VIP moved to apphaproxy2
ssh [email protected] "ip addr show | grep 100.115.2.210"
# Expected: inet 100.115.2.210/24

# Verify service still reachable
curl -sk https://argocd.mod.auh1.dev.dir/healthz
# Expected: ok

# Restore apphaproxy1
ssh [email protected] "sudo systemctl start keepalived"

Step 9.9 — Add `gitlab.mod.auh1.dev.dir` Ingress Alias¶

Design Decisions¶

Decision 1: Separate Ingress resource, not `helm upgrade`¶

GitLab is deployed via Helm outside ArgoCD — there is no Application managing it. Running helm upgrade --reuse-values to add an extra hostname carries real risk:

--reuse-values silently drops any values set via --set flags that were not captured in the saved release values
A misapplied upgrade can restart GitLab pods (Sidekiq, Webservice, Gitaly) causing downtime
The alias hostname (mod.auh1.dev.dir) is a lab/external access concern unrelated to GitLab's own configuration

Production-correct approach: create a standalone Ingress resource that points to the same backend service. This is isolated from the Helm release, survives helm upgrade, and can be committed to Git independently.

Decision 2: New TLS cert via cert-manager, not reusing `gitlab-tls-secret`¶

The existing gitlab-tls-secret has SANs for gitlab.cl1.sq4.aegis.internal only (issued by the GitLab Helm chart or manually). Adding gitlab.mod.auh1.dev.dir to that cert would require:

Patching the original GitLab ingress
Reissuing the certificate with an extra SAN
Coordinating cert rotation with the Helm-managed resource

Instead: annotate the new Ingress with cert-manager.io/cluster-issuer: internal-ca-issuer. cert-manager auto-issues a new certificate scoped to gitlab.mod.auh1.dev.dir and stores it in a new secret gitlab-mod-tls-secret. Clean separation of concerns.

Decision 3: Workers-only HAProxy backends (not masters)¶

The rke2-ingress-nginx DaemonSet runs on all 17 nodes including control-plane masters (100.115.1.51–53). They are excluded from HAProxy backends because:

Control-plane nodes run etcd, kube-apiserver, and kube-scheduler — resource contention with application traffic degrades API server response time and can cascade into node NotReady flaps
In production, control-plane nodes are explicitly tainted to prevent workload scheduling; routing ingress traffic through them contradicts that boundary
7 dedicated workers (100.115.1.54–60) provide sufficient capacity and can be scaled independently

Decision 4: GitLab ingress TLS vs. Helm `global.ingress.tls.enabled: false`¶

The GitLab Helm release has global.ingress.tls.enabled: false and global.hosts.https: false. This means:

GitLab generates internal http:// URLs (clone URLs, webhooks, redirect URIs)
The Helm chart does not inject TLS config into the generated Ingress resources

However, the actual ingress gitlab-webservice-default has a tls: stanza with gitlab-tls-secret — this was added manually after deployment. HTTPS termination works at the nginx ingress level. The new alias ingress follows the same pattern: cert-manager issues a cert, nginx terminates TLS.

Manifest¶

Store this in manifests/gitlab/ in the devops repo and apply via kubectl apply -f:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gitlab-webservice-mod-alias
  namespace: gitlab
  annotations:
    cert-manager.io/cluster-issuer: internal-ca-issuer
    kubernetes.io/ingress.provider: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "15"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/service-upstream: "true"
spec:
  ingressClassName: nginx
  rules:
  - host: gitlab.mod.auh1.dev.dir
    http:
      paths:
      - backend:
          service:
            name: gitlab-webservice-default
            port:
              number: 8181
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - gitlab.mod.auh1.dev.dir
    secretName: gitlab-mod-tls-secret

Annotation rationale

All annotations are copied verbatim from gitlab-webservice-default (confirmed via kubectl get ingress -o jsonpath).

proxy-body-size: "0" — unlimited; the main GitLab ingress uses 512m (set by Helm) but this alias uses "0" to ensure no size limit is imposed on Git push/clone operations routed through the external hostname
proxy-connect-timeout: 15 — fail fast if the backend pod is unresponsive
proxy-read-timeout: 600 — allow up to 10 min for large Git operations
service-upstream: "true" — nginx proxies to the Service ClusterIP instead of individual pod IPs; avoids stale endpoints when pods restart mid-request
cert-manager.io/cluster-issuer: internal-ca-issuer — cert-manager auto-issues and rotates the TLS cert for gitlab.mod.auh1.dev.dir

Apply:

kubectl apply -f manifests/gitlab/gitlab-webservice-mod-alias.yaml

# Watch cert-manager issue the cert (usually < 30s)
kubectl get certificate -n gitlab -w

# Verify ingress is active
kubectl get ingress -n gitlab
curl -sk https://gitlab.mod.auh1.dev.dir -o /dev/null -w "HTTP %{http_code}\n"
# Expected: HTTP 302 (GitLab redirects / → /users/sign_in)

Step 9.10 — Bastion `/etc/hosts` Fix¶

The bastion's /etc/hosts had stale entries pointing all cluster services to 100.115.1.14 (a GPU node with no ingress controller). This caused HTTP 000 from bastion for all cluster hostnames.

mod.auh1.dev.dir hostnames resolve correctly via DNS (external zone → VIP 100.115.2.210). cl1.sq4.aegis.internal hostnames are NOT in the external DNS — bastion needs /etc/hosts.

Correct entries (point directly to a worker node — same network, no HAProxy needed):

sudo tee /etc/hosts << 'EOF'
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

# Cluster services — direct to k8s-worker1 (bypasses HAProxy, same L2 network)
100.115.1.54  gitlab.cl1.sq4.aegis.internal registry.cl1.sq4.aegis.internal kas.cl1.sq4.aegis.internal
100.115.1.54  argocd.cl1.sq4.aegis.internal
100.115.1.54  vault.cl1.sq4.aegis.internal
EOF

Apply the same fix on bastion2 (100.115.1.24):

ssh -i Tarun-core42-Key [email protected] "sudo tee /etc/hosts" << 'EOF'
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

100.115.1.54  gitlab.cl1.sq4.aegis.internal registry.cl1.sq4.aegis.internal kas.cl1.sq4.aegis.internal
100.115.1.54  argocd.cl1.sq4.aegis.internal
100.115.1.54  vault.cl1.sq4.aegis.internal
EOF

Why not use the VIP?

Bastion is on the 100.115.1.x network. Routing through HAProxy VIP (100.115.2.x) requires a routed path that may not exist. Going directly to a worker node on the same network is simpler, more reliable, and doesn't add a proxy hop for internal traffic.

Summary Checklist¶

Infrastructure¶

[x] HAProxy running on both nodes (apphaproxy1 100.115.1.201, apphaproxy2 100.115.1.202)
[x] HAProxy switched from SSL termination → TCP passthrough (cert-manager certs served directly)
[x] HAProxy backends: all 7 workers 100.115.1.54–60 (masters excluded)
[x] Keepalived VIP 100.115.2.210 active
[x] Config backed up: /etc/haproxy/haproxy.cfg.ai71_config.<timestamp>

DNS¶

[x] mod.auh1.dev.dir zone resolves to VIP 100.115.2.210 via external DNS
[x] Bastion /etc/hosts corrected — cl1.sq4.aegis.internal → 100.115.1.54 (worker node)

Services¶

[x] GitLab https://gitlab.cl1.sq4.aegis.internal — HTTP 302, TLS via internal-ca-issuer
[x] GitLab alias https://gitlab.mod.auh1.dev.dir — HTTP 302, cert O=ai71 CN=modgpt ✅
[x] ArgoCD https://argocd.cl1.sq4.aegis.internal — HTTP 200 ✅
[x] ArgoCD alias https://argocd.mod.auh1.dev.dir — HTTP 200 ✅
[x] Harbor https://harbor.cl1.sq4.aegis.internal — own load balancer, direct IP 100.115.1.101
[ ] Vault https://vault.cl1.sq4.aegis.internal — not yet deployed
[ ] VIP failover test

Acceptance Test Commands¶

Run from bastion to verify all services:

echo "=== cl1.sq4 zone (via /etc/hosts → worker direct) ==="
curl -sk https://gitlab.cl1.sq4.aegis.internal -o /dev/null -w "gitlab.cl1:    %{http_code}\n"
curl -sk https://argocd.cl1.sq4.aegis.internal/healthz -w "argocd.cl1:   %{http_code}\n"
curl -sk https://harbor.cl1.sq4.aegis.internal/api/v2.0/systeminfo -o /dev/null -w "harbor.cl1:   %{http_code}\n"
curl -sk https://s3.cl1.sq4.aegis.internal -o /dev/null -w "s3.cl1:       %{http_code}\n"

echo ""
echo "=== mod.auh1.dev.dir zone (via DNS → VIP → HAProxy → ingress) ==="
curl -sk https://gitlab.mod.auh1.dev.dir -o /dev/null -w "gitlab.mod:    %{http_code}\n"
curl -sk https://argocd.mod.auh1.dev.dir/healthz -w "argocd.mod:   %s\n"