Kubernetes deployment

The Helm chart in deploy/helm/hango runs the same stack as compose.yaml on Kubernetes: hango, the OpenTelemetry Collector, Postgres and Grafana LGTM, with self-hosted Langfuse as an optional extra. This page explains each Kubernetes object the chart creates, then walks through installing it, using real output from a test on the local Rancher Desktop k3s cluster.

Kubernetes in five minutes

Helm is a package manager for these objects. A chart is a folder of YAML templates plus values.yaml. helm install renders the templates with your values and applies the result. A release is one named installation of a chart.

What the chart deploys

hango sends all telemetry to the collector over OTLP/HTTP. The collector fans out to LGTM and, when keys are set, to Langfuse Cloud and Grafana Cloud, exactly like compose.

The collector config is not copied into the chart. deploy/helm/hango/files/otelcol-config.yaml is a symlink to deploy/otelcol/config.yaml, so compose and Kubernetes always run the same file. Helm follows the symlink in helm template, helm install and helm package.

The collector exports to the host name lgtm, so the Services otelcol, lgtm and postgres have fixed names. The consequence is one release per namespace. hango itself is named after the release. When lgtm.enabled=false the chart sets LGTM_ENABLED=false on the collector, which makes the lgtm leg drop its data instead of retrying against a host that does not exist.

Prerequisites

Rancher Desktop with Kubernetes enabled (this gives k3s, Traefik and a default storage class), and helm and kubectl on your PATH. If your kubeconfig has several clusters, always name the cluster explicitly so a typo cannot touch the wrong one:

export PATH=$HOME/.rd/bin:$PATH
kubectl --context rancher-desktop get nodes
helm list --kube-context rancher-desktop -A

Install

Fetch the Langfuse subchart (always, even with Langfuse off)

The chart declares the Langfuse chart as a dependency. git ignores the downloaded charts/langfuse-*.tgz, and Helm refuses to render a chart whose declared dependency is missing, even when langfuse.enabled=false. On a fresh clone, run this once before any helm template, helm install or helm lint:

helm dependency update deploy/helm/hango

It reads the repository URL from Chart.yaml, so no helm repo add is needed, and it reproduces the committed Chart.lock (pinned to 2.1.5).

The hango image is not published yet

ghcr.io/phihdn/hango (the default image.repository) does not exist yet, so a default install ends in ImagePullBackOff. Build the image yourself and point the chart at it. For a registry, tag and push with a unique tag per build, then pass the repository and tag:

docker build -t registry.example.com/me/hango:0.1.0 .
docker push registry.example.com/me/hango:0.1.0
helm install hango deploy/helm/hango -n hango --create-namespace \
  --set image.repository=registry.example.com/me/hango --set image.tag=0.1.0

For a private registry see Registry and image.

Local install on Rancher Desktop

An image built with the docker engine (dockerd/moby) is visible to k3s, so no registry is needed. Use pullPolicy=Never so Kubernetes does not try to pull hango:dev from the internet. If Rancher Desktop uses the containerd engine instead, docker build is not visible to k3s: build with nerdctl --namespace k8s.io build -t hango:dev . (the k8s.io namespace is the one k3s reads), or push to a registry.

docker build -t hango:dev .
helm install hango-test deploy/helm/hango \
  --kube-context rancher-desktop -n hango-test --create-namespace \
  --set image.repository=hango --set image.tag=dev --set image.pullPolicy=Never
kubectl --context rancher-desktop -n hango-test wait --for=condition=Ready pod --all --timeout=300s
kubectl --context rancher-desktop -n hango-test get pods,svc,pvc

Output from the test run:

NAME                            READY   STATUS    RESTARTS   AGE
pod/hango-test-f57598c6-vm5pd   1/1     Running   0          28s
pod/lgtm-0                      1/1     Running   0          28s
pod/otelcol-66b4556f89-5t5lv    1/1     Running   0          28s
pod/postgres-0                  1/1     Running   0          28s

NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                        AGE
service/hango-test          ClusterIP   10.43.6.168     <none>        8080/TCP                                       28s
service/lgtm                ClusterIP   10.43.102.129   <none>        3000/TCP,3200/TCP,4317/TCP,4318/TCP,9090/TCP   28s
service/lgtm-headless       ClusterIP   None            <none>        3000/TCP                                       28s
service/otelcol             ClusterIP   10.43.99.173    <none>        4317/TCP,4318/TCP,13133/TCP                    28s
service/postgres            ClusterIP   10.43.50.41     <none>        5432/TCP                                       28s
service/postgres-headless   ClusterIP   None            <none>        5432/TCP                                       28s

NAME                                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
persistentvolumeclaim/data-lgtm-0       Bound    pvc-3a7e8a64-8b94-4bef-802e-4b2379a86dd2   5Gi        RWO            local-path
persistentvolumeclaim/data-postgres-0   Bound    pvc-df9baf75-bf23-431a-86e3-d74b4a0466d4   2Gi        RWO            local-path

The *-headless Services have no virtual IP. A StatefulSet needs one to give each pod its own DNS name (postgres-0.postgres-headless); use the plain postgres and lgtm Services for normal traffic.

Reach it with port-forward

kubectl port-forward tunnels a local port to a Service inside the cluster. It needs no Ingress and is the quickest way to look around. Each command keeps running, so use one terminal per forward or add &, and stop it with Ctrl-C or kill.

kubectl --context rancher-desktop -n hango-test port-forward svc/hango-test 18080:8080
kubectl --context rancher-desktop -n hango-test port-forward svc/lgtm 13000:3000 13200:3200

$ curl -si localhost:18080/healthz
HTTP/1.1 200 OK
Content-Type: application/json

{"status":"ok"}
$ curl -si localhost:18080/readyz
HTTP/1.1 200 OK
X-Trace-Id: 48074052fe5cda84ebc02eb1494da362

{"checks":{},"ready":true}

Grafana is then at localhost:13000 (admin/admin). Tempo’s API confirms the trace travelled hango, collector, LGTM:

$ curl -s "localhost:13200/api/search?limit=5"
{"traces":[{"traceID":"2dc6470c313f2f4859f20616a256f4c","rootServiceName":"hango","rootTraceName":"GET /readyz", ...}]}
$ curl -s localhost:13200/api/traces/48074052fe5cda84ebc02eb1494da362   # the X-Trace-Id above
{"batches":[{"resource":{"attributes":[{"key":"deployment.environment.name","value":{"stringValue":"prod"}}, ...

The collector logs a few Exporting failed ... connection refused lines for lgtm:4317 if it starts before LGTM is up. It retries, and the data arrives once LGTM is ready.

Ingress

An Ingress needs a controller. k3s ships Traefik, and deploy/helm/hango/examples/values-k3s-traefik.yaml sets ingress.enabled, className: traefik and host: hango.localhost:

helm upgrade hango-test deploy/helm/hango --kube-context rancher-desktop -n hango-test \
  --reset-then-reuse-values -f deploy/helm/hango/examples/values-k3s-traefik.yaml
kubectl --context rancher-desktop -n hango-test get ingress
curl -s http://hango.localhost/healthz

NAME         CLASS     HOSTS             ADDRESS        PORTS   AGE
hango-test   traefik   hango.localhost   192.168.64.4   80      0s
{"status":"ok"}

Traefik needs a few seconds to learn a new rule, so the first request can return 404 page not found. By default ingress.enabled is false and ingress.className is empty, because no controller is chosen yet. For TLS set ingress.tls.enabled=true and ingress.tls.secretName to a Secret holding tls.crt and tls.key. Do not expose the Grafana in the LGTM pod: it keeps the admin/admin login.

Upgrade, uninstall, turn parts on and off

# Change a value or roll out a new image tag
helm upgrade hango-test deploy/helm/hango --kube-context rancher-desktop -n hango-test --reset-then-reuse-values --set image.tag=v2

# Turn a component off (or on again)
helm upgrade hango-test deploy/helm/hango --kube-context rancher-desktop -n hango-test --reset-then-reuse-values --set lgtm.enabled=false

# Remove everything (the postgres Secret is kept, see below)
helm uninstall hango-test --kube-context rancher-desktop -n hango-test
kubectl --context rancher-desktop -n hango-test delete secret postgres
kubectl --context rancher-desktop delete namespace hango-test

Use --reset-then-reuse-values rather than --reuse-values for upgrades: it takes the new chart defaults first and then re-applies the values you set earlier, so a value added to values.yaml by a newer chart version is not missing. (Keeping your own values file and passing -f every time works too.)

The image tag dev is mutable. When image.pullPolicy is empty (the default) the chart uses Always for the tag dev and IfNotPresent for any other tag, so use a unique tag per build (as with v2 above). Set image.pullPolicy=Never for an image built locally.

Every component has an enabled flag, see deploy/helm/hango/values.yaml. With otelcol.enabled=false and no hango.config.otlpEndpoint, hango starts with telemetry export off (HANGO_OTEL_ENABLED=false). Set hango.config.otlpEndpoint to use a collector elsewhere. PVCs created by a StatefulSet are not deleted by helm uninstall, so deleting the namespace is what frees the disks.

The generated Postgres password Secret (postgres) carries helm.sh/resource-policy: keep, so helm uninstall leaves it in place together with the PVC. Postgres reads POSTGRES_PASSWORD only when it first initialises an empty volume, so a new random password after a reinstall would not match the data already on the disk. A later helm install of the same release in the same namespace adopts the kept Secret and its password (the test reinstall printed identical password hashes). Consequences: setting postgres.password after the first install changes nothing in the database, and deleting the namespace is what removes the Secret. The password is generated with Helm’s lookup, which returns nothing under helm template, so Argo CD and Flux (which render with helm template) would write a new random password on every sync. With those tools create the Secret yourself and set postgres.existingSecret (key POSTGRES_PASSWORD).

Registry and image

The build, push and install commands are in Install. image.tag defaults to the chart appVersion (dev). For a private registry create a pull Secret and reference it:

kubectl create secret docker-registry my-registry -n hango \
  --docker-server=registry.example.com --docker-username=me --docker-password='...'
helm upgrade --install hango deploy/helm/hango -n hango --set 'imagePullSecrets[0].name=my-registry' ...

Langfuse and Grafana Cloud keys

Never put keys in a values file that is committed. Create a Secret and point the chart at it. The Secret keys are the same variable names as .env.example. An unset key keeps that leg off, so only add the keys you have, and never an empty value.

kubectl --context rancher-desktop -n hango-test create secret generic otelcol-cloud \
  --from-literal=LANGFUSE_PUBLIC_KEY='pk-lf-...' \
  --from-literal=LANGFUSE_SECRET_KEY='sk-lf-...' \
  --from-literal=GRAFANA_CLOUD_INSTANCE_ID='123456' \
  --from-literal=GRAFANA_CLOUD_API_TOKEN='glc_...' \
  --from-literal=GRAFANA_CLOUD_OTLP_ENDPOINT='https://otlp-gateway-prod-us-east-0.grafana.net/otlp'
helm upgrade hango-test deploy/helm/hango --kube-context rancher-desktop -n hango-test \
  --reset-then-reuse-values --set otelcol.existingSecret=otelcol-cloud

The collector reads the Secret only at start. The first upgrade that sets otelcol.existingSecret rolls the pod because the Secret name changes. After that, Helm cannot see edits to a Secret you manage yourself, so helm upgrade with unchanged values restarts nothing: after changing the Secret run kubectl --context rancher-desktop -n hango-test rollout restart deployment/otelcol. The same holds for hango.existingSecret and kubectl rollout restart deployment/hango-test. For chart-managed Secrets (inline otelcol.cloud.* values, hango.secret.*) the pod template carries a hash of the values, so helm upgrade rolls the pods by itself. Where to get the keys is in observability-cloud-setup.md. Inline values (otelcol.cloud.langfuse.publicKey and friends) also work for a quick local try with --set, but they land in the Helm release Secret in the cluster. hango has the same pattern: hango.existingSecret or hango.secret.NAME=value, loaded as env vars.

Self-hosted Langfuse

langfuse.enabled=true installs the official Langfuse chart (pinned to 2.1.5 in Chart.yaml) as a subchart. It is heavy: it brings its own Postgres, ClickHouse, Redis and an S3-compatible store, and it requires the ClickHouse operator and cert-manager to be installed in the cluster first (see that chart’s README). Without them, helm template and helm install stop with ClickHouse operator CRDs not found. This chart’s tests only rendered it (with --set langfuse.clickhouse.crdCheck=false) and did not install it. Everything under the langfuse: key other than enabled is passed to that chart. Because the dependency is itself called langfuse and its own values also start with langfuse, its settings nest twice, for example langfuse.langfuse.logging.level (first langfuse selects the subchart, second is the key inside the subchart):

langfuse:
  enabled: true
  langfuse:
    logging:
      level: debug

After the install, create a project in the Langfuse UI and put its keys, and LANGFUSE_BASE_URL pointing at the Langfuse web Service, into the Secret above.

The subchart is downloaded by helm dependency update deploy/helm/hango, see Install.

Graceful shutdown

When Kubernetes stops a pod it does two things at once: it removes the pod from the Service endpoints because the pod is terminating (a failing /readyz plays no part in that), and it starts the stop sequence. Kube-proxy and ingress controllers such as Traefik need a few seconds to notice the endpoint change, so hango first sleeps in a preStop hook (hango.preStopSeconds, 5s, the native sleep action because the distroless image has no shell). Then the container gets SIGTERM, hango marks /readyz failing, drains for the HANGO_SHUTDOWN_TIMEOUT budget (15s) and flushes telemetry. If the container is still running after terminationGracePeriodSeconds (30s, counted from the start of preStop) Kubernetes sends SIGKILL, so the grace period must exceed preStop plus the budget (5 + 15 = 20 < 30). The chart refuses to render when it does not. Deleting the pod in the test (the last readyz log line is at :09.017, SIGTERM lands about 5s after the delete):

$ kubectl --context rancher-desktop -n hango-test delete pod hango-test-67bd5dc7c6-gdxkk
pod "hango-test-67bd5dc7c6-gdxkk" deleted from hango-test namespace      # 7 seconds
{"time":"...02:45:09.017...","level":"INFO","msg":"request","route":"GET /readyz","status":200, ...}
{"time":"...02:45:09.623...","level":"INFO","msg":"draining: readiness now failing","delay":1000000000,"budget":13000000000}
{"time":"...02:45:10.624...","level":"INFO","msg":"http server stopped"}
{"time":"...02:45:10.624...","level":"INFO","msg":"hango stopped"}

The container exited by itself, well inside the grace period, and the Deployment started a replacement pod.

Troubleshooting

• ErrImageNeverPull: the image is not on the node. Build it with the docker engine (or nerdctl --namespace k8s.io on containerd), or push to a registry and drop pullPolicy=Never.
• ImagePullBackOff on ghcr.io/phihdn/hango: that image is not published, build and push your own (see Install).
• CreateContainerConfigError on the collector or hango: a Secret named by otelcol.existingSecret or hango.existingSecret does not exist.
• Pod stuck Pending: kubectl describe pod NAME shows why, often a PVC waiting for a storage class.
• Check logs with kubectl --context rancher-desktop -n hango-test logs deploy/hango-test.
• helm template deploy/helm/hango renders the YAML without installing, the best way to see what a value changes.