Foundation and Observability Spine

Phase 1 builds a server that does almost nothing yet: it answers two health checks and shuts down cleanly. What matters is that from the first request you can see everything the server does. Every request becomes a trace in Grafana, every log line carries the trace id, and Go runtime metrics are graphed. Every later phase (auth, runs, the agent loop, tools) adds its spans into this spine instead of inventing its own.

What you will learn

• What a composition root is and why hango has exactly one.
• The three telemetry signals (traces, logs, metrics) and how one id ties them together.
• Why hango sends telemetry to an OpenTelemetry Collector instead of straight to Grafana and Langfuse.
• How a request moves through middleware, and why a small router on the standard library is enough.
• How to find one request in Grafana, starting from a response header.
• How to debug this layer when something does not show up.

Words used in this lesson

Why observability comes first

An agent harness fails in ways a plain web server does not. A run can loop, a tool can hang, a model call can take 40 seconds, and one tenant can slow down another. When that happens you need to answer “what happened during this request?” with data, not guesses. If tracing is added at phase 10, phases 2 to 9 are built blind and then retrofitted. Building it first means each later phase only adds spans to a pipeline that already works.

The smallest version

Before the real code, here is the whole idea in one file. It starts a tracer, wraps a handler with otelhttp, and prints the trace id.

package main

import (
	"context"
	"log"
	"net/http"

	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	"go.opentelemetry.io/otel/trace"
)

func main() {
	exp, err := otlptracehttp.New(context.Background()) // sends to localhost:4318
	if err != nil {
		log.Fatal(err)
	}
	tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exp))
	defer tp.Shutdown(context.Background())

	mux := http.NewServeMux()
	mux.HandleFunc("GET /hello", func(w http.ResponseWriter, r *http.Request) {
		id := trace.SpanFromContext(r.Context()).SpanContext().TraceID()
		w.Write([]byte("trace " + id.String() + "\n"))
	})
	handler := otelhttp.NewHandler(mux, "hello", otelhttp.WithTracerProvider(tp))
	log.Fatal(http.ListenAndServe(":8080", handler))
}

Everything else in phase 1 grows from this: logs that carry the same id, metrics next to traces, a clean shutdown that flushes the batch, and a collector between the app and the backends.

How hango is wired

The composition root

cmd/hango/main.go only parses flags, turns SIGINT and SIGTERM into a cancelled context with signal.NotifyContext, and calls app.Run. internal/app/app.go builds everything in order:

config.Load -> telemetry.Init -> server.New -> errgroup runs server.Start -> wait for ctx

Shutdown runs in reverse. The server drains first, then telemetry flushes, so spans and logs from the last requests still leave the process. No other package constructs its own dependencies. When a later phase adds a database pool or a run worker, it is created in app.go and passed in. Reading one file tells you what the program is made of.

Configuration

internal/config reads environment variables only, with no config file and no hot reload. It collects every bad value before failing, so you fix them all in one go. This is real output:

$ HANGO_ENV=staging HANGO_SHUTDOWN_TIMEOUT=soon OTEL_EXPORTER_OTLP_PROTOCOL=grpc go run ./cmd/hango
hango: invalid configuration:
  HANGO_ENV="staging": must be one of dev, prod
  HANGO_SHUTDOWN_TIMEOUT="soon": must be a positive duration such as 15s
  OTEL_EXPORTER_OTLP_PROTOCOL="grpc": must be one of http/protobuf

Telemetry

telemetry.Init creates one of each provider and installs them globally:

• A TracerProvider with a batch exporter. The sampler is parent-based: if an incoming request already carries a sampled trace, hango continues it, and otherwise keeps HANGO_OTEL_SAMPLE_RATIO of new traces.
• A MeterProvider with a periodic reader. Go runtime metrics start here.
• A LoggerProvider behind the otelslog bridge.

The default slog logger writes to two places: stdout, and the otelslog bridge that ships logs to the collector. Stdout means you can still read logs when the collector is down. Both outputs carry trace_id and span_id, so a line on your terminal leads to its trace in Grafana.

With HANGO_OTEL_ENABLED=false (or the standard OTEL_SDK_DISABLED=true), Init returns no-op providers. The code that creates spans does not change, and nothing is exported.

Owned names

Three files hold names as constants instead of string literals scattered across the code:

Dashboards and redaction both break without an error when a name drifts. Owned lists plus tests turn that silent drift into a failing build.

Why a collector

hango sends all telemetry to one place, the collector at OTEL_EXPORTER_OTLP_ENDPOINT, and the collector fans out:

hango --OTLP/HTTP--> otelcol --+-- traces  --> Langfuse
                               +-- traces  --> Tempo       (content stripped)
                               +-- metrics --> Prometheus
                               +-- logs    --> Loki        (content stripped)

Exporting straight to both backends from the app was rejected for three reasons:

1. Langfuse accepts OTLP over HTTP only, not gRPC. With two in-process exporters the app would need two protocol configs and two batch queues.
2. Deleting prompt bodies is a rule in deploy/otelcol/config.yaml. Changing it means restarting the collector, not redeploying hango.
3. The collector queues and retries. A Langfuse outage does not fill hango’s memory.

The router

hango uses the standard library http.ServeMux plus about 200 lines of its own (internal/server/router.go). Since Go 1.22 the mux matches methods and path parameters ("GET /v1/things/{id}"), which removes the main reason to use a framework. What frameworks such as Echo, Gin, chi and Fiber add, and how hango covers it:

Fiber is not an option at all. It is built on fasthttp, not net/http, so otelhttp, request-context cancellation and server-sent events (phase 6) would all need replacements. goclaw, a production Go harness used as a reference throughout this course, serves its whole API from http.ServeMux the same way (internal/http).

The middleware chain

Outermost first:

A bug found while testing shows why the order and details matter. The first test run printed this:

--- FAIL: TestOneServerSpanPerRequestNamedByRoute (0.00s)
    server_test.go:91: span "GET" kind server
--- FAIL: TestRequestDurationMetricCarriesRoute (0.00s)
    server_test.go:119: http.server.request.duration has no http.route=/v1/things/{id} data point

otelhttp reads r.Pattern after the handler returns. But requestID calls r.WithContext(ctx), which returns a copy of the request. The mux set Pattern on that copy, and otelhttp still held the original with an empty Pattern. The fix is three lines in requestID: after next.ServeHTTP, copy inner.Pattern back to r.Pattern. Any middleware that replaces the request has this problem, so keep it in mind when you write one.

Liveness, readiness and shutdown

GET /healthz always returns 200 while the process runs. Orchestrators call it every few seconds, so it is not traced or logged. GET /readyz runs every registered check concurrently with a 2-second timeout each, and returns 503 if any fails. The response names the failing check with "fail", and the real error goes to the log, because errors such as dial tcp 10.0.0.5:5432 describe the internal network. A check that ignores its context is abandoned at the timeout. Phase 2 registers the database check.

On SIGTERM, the server:

1. Marks itself draining, so /readyz returns 503 and load balancers stop sending traffic.
2. Keeps serving for 1 second so they notice.
3. Calls http.Server.Shutdown, which waits for in-flight requests until the shutdown deadline minus a short reserve for telemetry (2s, or a quarter of a short budget).
4. Returns, and app.Run flushes telemetry until the shutdown deadline.

HANGO_SHUTDOWN_TIMEOUT is the total budget. app.Run takes one deadline at the moment the signal arrives, and the drain delay, the server drain and the telemetry flush all measure against it, so the process exits within one timeout.

Real output from docker compose stop hango, which exited 0 after about 1 second:

level=INFO msg="draining: readiness now failing" delay=1s budget=13s
level=INFO msg="http server stopped"
level=INFO msg="hango stopped"

Telemetry gets two thirds of its budget for traces and metrics and the rest for logs, so a stalled trace exporter cannot stop the final log lines from leaving. Compose sets stop_grace_period: 20s, the 15s budget plus margin, because Docker’s default of 10s would kill the process mid-flush. A second Ctrl-C quits immediately.

WriteTimeout stays 0 on purpose. A global write timeout would cut every server-sent event stream in phase 6 at a fixed age.

How to see what the server is doing

1. Start the stack and send a request:

   make compose-up
   curl -i localhost:8080/readyz

   Real response:

   HTTP/1.1 200 OK
   Content-Type: application/json
   X-Request-Id: 01a13cf0-9c8e-726a-aa04-4f57a4175f2e
   X-Trace-Id: ec55eb5d66c12d5222b23dbb1154302b
   
   {"checks":{},"ready":true}

2. Copy the X-Trace-Id value.

3. Open Grafana at localhost:3000/explore and log in with admin/admin. Pick the Tempo data source, choose the TraceQL tab, paste the trace id and run the query. You see one span, GET /readyz, of kind server. Its attributes include http.response.status_code, url.path and hango.request.id.

4. Switch the data source to Loki and run:

   {service_name="hango"} | trace_id="ec55eb5d66c12d5222b23dbb1154302b"

   You get the request log line for that request, with hango_request_id matching the X-Request-Id header.

5. Switch to Prometheus and run go_memory_used_bytes or go_goroutine_count for runtime metrics. For p95 latency per route:

   histogram_quantile(0.95, sum by (le, http_route) (rate(http_server_request_duration_seconds_bucket[5m])))

Metrics are exported every 10 seconds in compose (OTEL_METRIC_EXPORT_INTERVAL=10000), so wait that long before expecting new points.

How to debug this layer

The server will not start

The config error lists every bad variable, as shown above. Fix them all and start again. If the error is address already in use, another process holds port 8080: find it with lsof -i :8080.

The server runs but nothing appears in Grafana

1. Check the startup line. It prints otel_enabled and otlp_endpoint. Inside compose the endpoint must be http://otelcol:4318. On the host it is http://localhost:4318.

2. Check the collector: docker compose logs otelcol. It must print Everything is ready. Connection errors to lgtm:4317 right after startup mean lgtm was still booting. Compose now waits for lgtm to be healthy before starting the collector.

3. Collector warnings are worth reading. Collector 0.161 printed this for the first version of our config:

   warn  "otlp" alias is deprecated; use "otlp_grpc" instead
   warn  "otlphttp" alias is deprecated; use "otlp_http" instead

   The exporters are now named otlp_grpc/lgtm and otlp_http/langfuse.

4. Metrics take up to the export interval to arrive. Traces and logs are batched and arrive within a few seconds.

Traces appear in Tempo but not in Langfuse

First check that LANGFUSE_PUBLIC_KEY is set in .env. When it is empty, the Langfuse leg is off: filter/langfuse_disabled in the collector config drops every span on that leg, so nothing leaves for Langfuse and no error is logged. This keeps traces, and later prompt text, from going to a third party you have not set up.

With a key that Langfuse rejects, the collector logs this, and the Tempo, Loki and Prometheus legs keep working:

error  Exporting failed. Dropping data.  "otelcol.component.id": "otlp_http/langfuse"
"error": "not retryable error: Permanent error: rpc error: code = Unauthenticated desc = error exporting items,
request to https://cloud.langfuse.com/api/public/otel/v1/traces responded with HTTP Status Code 401"

A 401 means the public or secret key is wrong. Put LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY in .env and run docker compose up -d --force-recreate otelcol. A plain restart does not reread .env. If traces arrive but appear minutes late, check that the x-langfuse-ingestion-version: "4" header is still in the exporter config.

Trade-offs

How goclaw and deepagents do this

goclaw puts startup in a cmd/ folder of more than 100 files, many of them gateway_* wiring files. It has a single /health endpoint, reads JSON5 config and hot-reloads it with fsnotify. Its tracing is home-grown: spans are buffered in memory and flushed to traces and spans tables in its own database, with OTLP export available behind a build tag.

langchain deepagents is not a server. Its composition happens in create_deep_agent, which assembles an ordered list of middleware and hands it to the LangChain agent loop. It is the same idea as hango’s composition root, one constructor that owns the order, at a different layer. Tracing comes from LangSmith through LangChain callbacks rather than OpenTelemetry.

hango diverges on purpose:

• One wiring file instead of many, so the whole program can be read at once.
• Two health endpoints, because “alive” and “ready for traffic” are different questions during shutdown and when the database is down.
• No hot reload. Config changes mean a restart, which the drain sequence makes safe.
• Standard OpenTelemetry instead of a custom span store, so any OTel backend works and nothing in hango stores or queries spans.

Exercises

1. Add a config value end to end. Add HANGO_DRAIN_DELAY (a duration, default 1s) to internal/config, test that a bad value is reported, and use it in place of the drainDelay constant in internal/server/server.go.
2. Make /readyz fail and watch the drain. In app.go, register a check with srv.Health().Register("demo", func(context.Context) error { return errors.New("not yet") }). Curl /readyz and read the 503 body. Find the trace in Tempo, then send SIGTERM and read the drain log lines.
3. Turn telemetry off. Run HANGO_OTEL_ENABLED=false go run ./cmd/hango with no collector running. Confirm that the server starts, answers /readyz with both headers, and shuts down cleanly. Note that X-Trace-Id is missing: work out why from requestID.
4. Break the content guard. Add a key to ContentAttrs in internal/telemetry/content_attrs.go, run go test ./internal/telemetry, read the failure, and then fix it with the -update flag the message suggests.