Foundation, server skeleton, and observability spine

Overview

Create the Go module, the single binary, the composition root, env config, a standard-library net/http server with health endpoints, signal-driven graceful shutdown, and the local toolchain. Critically, observability is foundation, not a later phase: this phase builds internal/telemetry (one TracerProvider, one MeterProvider, one LoggerProvider with the otelslog bridge installed as the default slog handler), Go runtime metrics, otelhttp-wrapped HTTP handling, hango attribute constants, locally-defined GenAI semconv constants, and a compose.yaml that fans out through an OpenTelemetry Collector to Grafana LGTM and Langfuse. From phase 1 onward, every log line carries trace_id/span_id, and every later phase instruments itself.

Requirements

Functional

• cmd/hango builds a binary that starts an HTTP server on a configured address.
• GET /healthz returns 200 unconditionally; GET /readyz runs registered checks and reports draining once shutdown begins.
• Config is read from environment variables with documented defaults; missing or invalid required values fail at startup listing every bad variable.
• SIGINT/SIGTERM trigger graceful shutdown within a bounded timeout, including telemetry provider flush.
• Every request gets a request id; the response carries X-Request-Id and X-Trace-Id so a user can jump from a response straight to the trace.
• internal/telemetry.Init(ctx, cfg) returns a Shutdown(ctx) error that flushes traces, metrics and logs.
• docker compose up starts hango, postgres, otelcol and lgtm; Grafana at localhost:3000 shows the server’s traces, logs and runtime metrics.

Non-functional

• No third-party config, logging or DI libraries: stdlib env helpers, log/slog with the OTel bridge.
• One OTLP egress path (http/protobuf to the collector) for all three signals; the app never talks to Langfuse directly.
• Prompt and completion bodies never reach Tempo or Loki: stripping happens in the collector, not the app.
• GenAI semantic-convention attribute names are declared as hango-local constants; no dependency on a generated semconv package (GenAI semconv is Development status and has already moved repositories).
• Every metric name emitted by any phase is a constant in `internal/telemetry/metrics_names.go`, introduced here. A phase that emits a metric without adding its constant is a defect; phase 14 consumes this list rather than retrofitting it.
• The set of attribute keys that may carry prompt, completion, tool-input or tool-output content is a hango-owned constant list in `internal/telemetry/content_attrs.go`. The collector strips exactly that list; the app never sets a content attribute whose key is absent from it.
• Final image is gcr.io/distroless/static-debian12:nonroot, CGO_ENABLED=0, -trimpath, version stamped via -ldflags -X.
• golangci-lint run clean with the v2 baseline linter set.

Architecture

Composition root: app.Run(ctx) builds config → telemetry → logger → readiness registry → server, then blocks on errgroup. Nothing else constructs dependencies.

// internal/telemetry/telemetry.go
type Config struct {
	ServiceName    string // hango
	ServiceVersion string // build stamp
	Environment    string // deployment.environment.name
	OTLPEndpoint   string // http://otelcol:4318
	Enabled        bool   // false -> no-op providers, everything still compiles and runs
	SampleRatio    float64
}

type Telemetry struct {
	Tracer trace.Tracer
	Meter  metric.Meter
	Logger *slog.Logger // otelslog bridge; also installed via slog.SetDefault
}

func Init(ctx context.Context, cfg Config) (*Telemetry, func(context.Context) error, error)

// internal/telemetry/attrs.go — hango-owned attribute keys
const (
	AttrTenantID  = "hango.tenant.id"
	AttrAgentID   = "hango.agent.id"
	AttrSessionID = "hango.session.id"
	AttrRunID     = "hango.run.id"
	AttrTurn      = "hango.turn"
	AttrToolName  = "hango.tool.name"
	AttrLane      = "hango.lane"
	// Red team: F13 — per-tenant LLM-content opt-in; set from tenants.trace_llm_content (phase 2)
	AttrTenantTraceLLMContent = "hango.tenant.trace_llm_content"
)

// internal/telemetry/genai.go — locally declared, NOT imported from a generated package
const (
	GenAIOperationName = "gen_ai.operation.name" // chat | invoke_agent | execute_tool
	GenAIProviderName  = "gen_ai.provider.name"
	GenAISystem        = "gen_ai.system"         // older name Langfuse still documents; emit both
	GenAIRequestModel  = "gen_ai.request.model"
	GenAIResponseModel = "gen_ai.response.model"
	GenAIFinishReasons = "gen_ai.response.finish_reasons"
	GenAIUsageInput    = "gen_ai.usage.input_tokens"
	GenAIUsageOutput   = "gen_ai.usage.output_tokens"
	GenAIUsageCacheRead  = "gen_ai.usage.cache_read.input_tokens"
	GenAIUsageCacheWrite = "gen_ai.usage.cache_write.input_tokens"
	GenAITimeToFirstChunk = "gen_ai.response.time_to_first_chunk"
	GenAIConversationID   = "gen_ai.conversation.id"
	GenAIAgentName        = "gen_ai.agent.name"
	GenAIToolName         = "gen_ai.tool.name"
	GenAIToolCallID       = "gen_ai.tool.call.id"
	// Red team: F13 — used by phase 3 onward; previously referenced without being declared
	GenAIRequestMaxTokens   = "gen_ai.request.max_tokens"
	GenAIRequestTemperature = "gen_ai.request.temperature"
	GenAIRequestStream      = "gen_ai.request.stream"
	GenAIResponseID         = "gen_ai.response.id"
	GenAIUsageReasoning     = "gen_ai.usage.reasoning.output_tokens"
)

// internal/telemetry/langfuse.go — langfuse.* always wins over generic conventions
const (
	LFTraceName      = "langfuse.trace.name"
	LFTraceTags      = "langfuse.trace.tags"
	LFTraceMetaPfx   = "langfuse.trace.metadata."
	LFSessionID      = "langfuse.session.id"
	LFUserID         = "langfuse.user.id"
	LFEnvironment    = "langfuse.environment"
	LFObsType        = "langfuse.observation.type" // span|generation|event|agent|tool|chain|...
	LFObsInput       = "langfuse.observation.input"
	LFObsOutput      = "langfuse.observation.output"
	LFObsMetaPfx     = "langfuse.observation.metadata."
	LFObsUsageDetail = "langfuse.observation.usage_details"
	LFObsCostDetail  = "langfuse.observation.cost_details"
	LFObsModelName   = "langfuse.observation.model.name"
	LFObsLevel       = "langfuse.observation.level"
	LFObsStatusMsg   = "langfuse.observation.status_message"
	LFObsCompletionStart = "langfuse.observation.completion_start_time"
)

// internal/config/config.go
type Config struct {
	Env             string // dev|prod
	HTTPAddr        string
	LogLevel        slog.Level
	ShutdownTimeout time.Duration
	Version         string
	Telemetry       telemetry.Config
}

func Load() (Config, error)

// internal/server/server.go
type Server struct{ /* mux, cfg, log, health */ }

func New(cfg config.Config, log *slog.Logger, tel *telemetry.Telemetry) *Server
func (s *Server) Start(ctx context.Context) error
func (s *Server) Health() *Health

// internal/server/router.go
// A handler returns an error; one place turns errors into status codes and JSON bodies.
type HandlerFunc func(w http.ResponseWriter, r *http.Request) error
type Middleware func(http.Handler) http.Handler

type Router struct{ /* *http.ServeMux, middleware for this group */ }

func (rt *Router) Handle(pattern string, h HandlerFunc) // pattern like "POST /v1/runs/{id}/cancel"
func (rt *Router) Group(mw ...Middleware) *Router     // same mux, extra middleware
func DecodeJSON(w http.ResponseWriter, r *http.Request, v any) error // MaxBytesReader, DisallowUnknownFields

// internal/server/health.go
type Check func(ctx context.Context) error

func (h *Health) Register(name string, c Check)
func (h *Health) MarkDraining()
func (h *Health) Ready(ctx context.Context) (ok bool, details map[string]string)

Instrumentation

• Spans: one SERVER span per HTTP request from otelhttp.NewHandler wrapping the mux, with the span name set to {method} {route pattern} from r.Pattern by the router so cardinality stays bounded.
• Attributes on every request span and log record: hango.request.id, plus hango.tenant.id from phase 2 onward.
• Metrics: contrib/instrumentation/runtime (runtime.Start) for Go runtime series; http.server.request.duration from otelhttp. otelhttp wraps the mux and runs before routing, so it cannot see the route. The router adds http.route (the path part of r.Pattern) to the metric labeler via otelhttp.LabelerFromContext(r.Context()).Add(...) after routing; without this the phase-14 per-route API dashboard cannot be built. All metric names come from internal/telemetry/metrics_names.go.
• Logs: otelslog bridge set as the default slog handler, so slog.InfoContext(ctx, ...) records carry the active trace and span ids into Loki automatically. Health-check requests are excluded from tracing by an otelhttp.WithFilter.
• Collector fan-out: traces → Langfuse (otlphttp with basic auth + x-langfuse-ingestion-version: 4) and → lgtm; metrics and logs → lgtm. Content stripping is driven by the hango-owned constant list in internal/telemetry/content_attrs.go (prompt, completion, input/output messages, system instructions, langfuse.observation.input|output, and the tool input/output attribute keys), applied unconditionally on the lgtm pipelines and conditionally on the Langfuse pipeline for spans carrying hango.tenant.trace_llm_content=false. The list is generated into the collector config, so adding a content attribute without adding it to the list is caught by the phase-3 test.

No tables in this phase.

Related Code Files

Create

• go.mod, go.sum — module github.com/phihdn/hango, Go 1.27.
• cmd/hango/main.go — flags (-version), signal.NotifyContext, calls app.Run.
• internal/app/app.go — the only wiring file.
• internal/config/config.go, internal/config/config_test.go.
• internal/telemetry/telemetry.go, internal/telemetry/attrs.go, internal/telemetry/genai.go, internal/telemetry/langfuse.go, internal/telemetry/noop.go, internal/telemetry/telemetry_test.go.
• `internal/telemetry/metrics_names.go` (every hango metric name as a constant; phases 2-16 add theirs here first) and `internal/telemetry/content_attrs.go` (the content attribute key list plus a `IsContentAttr(key) bool` helper), each with a test.
• internal/server/server.go, internal/server/health.go, internal/server/router.go (Handle, Group, the error adapter, DecodeJSON, the status-recording response writer), internal/server/middleware.go (request id, recover), internal/server/router_test.go, internal/server/server_test.go, internal/server/health_test.go.
• Makefile, .golangci.yml, Dockerfile, compose.yaml, deploy/otelcol/config.yaml, deploy/langfuse/compose.langfuse.yaml (documented, profile-gated, not started by default).
• .env.example, .dockerignore, .gitignore.
• README.md, docs/system-architecture.md (skeleton: Overview, Packages, Data model, Request flows, Run lifecycle, Tenancy, Observability, Deployment), docs/learning/README.md, docs/learning/01-foundation-and-observability-spine.md.

Modify

• None (greenfield).

Implementation Steps

1. go mod init github.com/phihdn/hango; go 1.27, toolchain go1.27.1. Add go.opentelemetry.io/otel + /sdk + /sdk/metric v1.46.0, go.opentelemetry.io/otel/sdk/log v0.22.0, otlptracehttp/otlpmetrichttp/otlploghttp, contrib/bridges/otelslog v0.20.1, contrib/instrumentation/runtime v0.71.0, contrib/instrumentation/net/http/otelhttp v0.71.0, golang.org/x/sync.
2. Write internal/config: HANGO_ENV, HANGO_HTTP_ADDR, HANGO_LOG_LEVEL, HANGO_SHUTDOWN_TIMEOUT, HANGO_OTEL_ENABLED, HANGO_OTEL_SAMPLE_RATIO. The OTLP endpoint, protocol, service name and resource attributes are read from the standard OTEL_* variables honoured by the Go exporters; Load only validates they are parseable.
3. Write internal/telemetry/telemetry.go: build one resource.Resource (service.name, service.version, deployment.environment.name), one sdktrace.TracerProvider with WithBatcher(otlptracehttp.New(...)) and a parent-based ratio sampler, one sdkmetric.MeterProvider with a periodic reader over otlpmetrichttp, one sdklog.LoggerProvider over otlploghttp. Set the global providers and otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(tracecontext, baggage)). Build the slog.Logger as a multi-handler: otelslog.NewHandler for OTLP plus a local slog.JSONHandler/TextHandler on stdout (local output stays readable and survives a collector outage); slog.SetDefault it. Return a Shutdown that flushes all three providers with a bounded timeout.
4. Write internal/telemetry/noop.go: when Enabled is false, return no-op providers and a plain stdout slog handler so tests and the -version path never require a collector.
5. Write attrs.go, genai.go, langfuse.go with the constants above and a short comment on each group stating why hango declares them locally. Write metrics_names.go (phase-1 metric names only, to be extended by every later phase) and content_attrs.go with the content attribute key list and IsContentAttr.
6. Call runtime.Start(runtime.WithMeterProvider(mp)) in Init.
7. Write internal/server/health.go: mutex-guarded named checks with per-check timeout, a draining flag, and a JSON detail map.
8. Write internal/server/router.go and internal/server/middleware.go. The router wraps the Go 1.22+ http.ServeMux:
   1. Handle(pattern, h) registers a HandlerFunc with the group’s middleware. After routing it renames the span to {method} {route pattern} from r.Pattern and adds http.route to the otelhttp metric labeler, so HTTP metrics carry the route . Check first whether otelhttp v0.71.0 already reads r.Pattern, and drop whatever it already does.
   2. The error adapter turns a returned error into one JSON shape ({"error": {"code", "message"}}): typed errors map to 400, 401, 403, 404 and 409, and anything else is a 500 whose message is never the internal error text.
   3. DecodeJSON limits the body with http.MaxBytesReader (1 MiB default), rejects unknown fields, and returns a 400 error.
   4. A status-recording ResponseWriter keeps the status for the audit middleware (phase 2) and implements Unwrap(), so http.ResponseController can still flush SSE in phase 6.
   5. Middleware: generate or accept X-Request-Id; set X-Request-Id and X-Trace-Id (from trace.SpanContextFromContext) on the response; a recover middleware that logs the stack with slog.ErrorContext and records the panic on the span.
9. Write internal/server/server.go: build the http.ServeMux and the root Router, register health routes, then wrap the mux with otelhttp.NewHandler(mux, "hango", otelhttp.WithFilter(skipHealthz)) and hand that to http.Server. Set ReadHeaderTimeout and IdleTimeout; leave WriteTimeout zero (SSE arrives in phase 6 and a global write timeout silently truncates streams). Start runs the server, and on ctx.Done() marks draining, waits 1s so load balancers see /readyz fail, then shuts down.
10. Write internal/app/app.go: config.Load → telemetry.Init → logger → server → errgroup; log one startup line with version and config summary and never a secret; defer telemetryShutdown after server shutdown.
11. Write cmd/hango/main.go with signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM).
12. Write deploy/otelcol/config.yaml exactly as in the observability report section 3: basicauth/langfuse extension from LANGFUSE_PUBLIC_KEY/LANGFUSE_SECRET_KEY, otlp receiver on 4317/4318, attributes/strip_content deleting exactly the keys listed in internal/telemetry/content_attrs.go (never a pattern) on both the lgtm pipelines and — via attributes/strip_content_langfuse guarded by hango.tenant.trace_llm_content=false — on the Langfuse pipeline, so a tenant that opted out never has content leave the collector on any leg , exporters otlphttp/langfuse (${LANGFUSE_HOST}/api/public/otel, header x-langfuse-ingestion-version: "4") and otlp/lgtm (lgtm:4317, insecure), and the four pipelines (traces/langfuse, traces/lgtm, metrics, logs).
13. Write compose.yaml with four services: hango (env OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4318, OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf, OTEL_SERVICE_NAME=hango, OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=dev,service.version=${GIT_SHA:-dev}), postgres:17-alpine (used from phase 2), otelcol (otel/opentelemetry-collector-contrib:0.161.0, config mounted, ports 4317/4318, Langfuse env), lgtm (grafana/otel-lgtm:0.33.1, ports 3000/4317/4318/3200/9090, volume lgtm-data:/data). Add deploy/langfuse/compose.langfuse.yaml behind profile langfuse-local documenting the 6-service self-host stack (langfuse-web, langfuse-worker, clickhouse, redis, postgres, minio) with its ≥4 core / 16 GiB note — documented only, not wired this phase.
14. Write Makefile (build run test vet lint docker compose-up compose-down obs-up), .golangci.yml (errcheck, govet, staticcheck, ineffassign, unused, gosec, bodyclose, sqlclosecheck, revive), Dockerfile (multi-stage, distroless static nonroot, -trimpath -ldflags "-s -w -X main.version=$(VERSION)"), .env.example (including LANGFUSE_HOST=https://cloud.langfuse.com, placeholder keys).
15. Write tests: config table test; health pass/fail/draining; server test asserting 200s plus the presence of X-Request-Id and X-Trace-Id; a telemetry test using an in-memory span recorder asserting one SERVER span per request with the route-pattern name and that /healthz is filtered out.
16. Write README.md, the docs/system-architecture.md skeleton with the Observability section filled, docs/learning/README.md, and the phase learning doc.
17. Run go build ./... && go vet ./... && go test ./... && golangci-lint run, then docker compose up --build, curl the health endpoints, and confirm in Grafana that the request trace, its log lines and Go runtime metrics are present.

Learning Doc Outline

docs/learning/01-foundation-and-observability-spine.md must contain:

• Concept: composition root, and why observability is foundation — you cannot debug a harness you cannot see.
• Design choices: the standard-library http.ServeMux with a small router of our own instead of a framework, env-only config, liveness vs readiness, signal.NotifyContext, one TracerProvider/MeterProvider/LoggerProvider, otelslog as the default slog handler, one OTLP egress to a collector, hango-local GenAI and Langfuse attribute constants.
• Why the collector and not two in-process exporters: Langfuse rejects gRPC, prompt-body redaction must be configurable without redeploying, and the collector’s queue survives a Langfuse outage.
• Why no web framework: what Echo, Gin, chi and Fiber add (binding, central errors, groups, ready-made middleware), how each fits in about 100 lines here, and why Fiber’s fasthttp would break SSE, otelhttp and cancellation. goclaw serves its whole API from http.ServeMux the same way (internal/http).
• Trade-offs: Grafana LGTM one-container (dev/demo only) vs a split collector/Tempo/Loki/Prometheus stack; Langfuse Cloud Hobby vs the 6-service self-host compose; sampling ratio vs full fidelity.
• “How to see what the server is doing”: a step-by-step walkthrough — send a request, read X-Trace-Id from the response, open Grafana Explore → Tempo → paste the trace id → inspect the span → jump to Loki logs filtered by trace_id → open Prometheus for runtime metrics. Include the exact Grafana URLs and queries.
• How to debug this layer: server will not start (config error listing), server starts but nothing appears in Grafana (collector logs, OTEL_EXPORTER_OTLP_ENDPOINT, HANGO_OTEL_ENABLED), traces appear in Tempo but not Langfuse (basic auth, ingestion-version header, collector exporter queue).
• goclaw comparison: 100+ file cmd/, single /health, JSON5 config with fsnotify hot-reload, and its home-grown tracing that buffers spans in memory and flushes to a traces/spans table with OTLP behind a build tag.
• deepagents comparison: not a server; composition is create_deep_agent assembling a middleware list — the same “one constructor owns order” idea at a different layer; tracing comes from LangSmith/LangChain callbacks rather than OTel.
• Why hango diverges: one wiring file, two health endpoints, no hot-reload, standard OTel rather than a bespoke span store.
• Why metric names and content attribute keys are owned lists in code rather than conventions: dashboards and redaction both break silently when a name drifts.
• Exercises: add a config value end to end; make /readyz fail and watch the drain sequence; set HANGO_OTEL_ENABLED=false and confirm the binary still works.
• Time budget: 15h implementation + 2h learning doc.

Success Criteria

• go build ./... && go vet ./... && go test ./... all pass.
• golangci-lint run reports no findings.
• curl -i localhost:8080/readyz returns 200 with both X-Request-Id and X-Trace-Id headers set.
• docker compose up --build starts hango, postgres, otelcol, lgtm; all healthy.
• In Grafana (localhost:3000, admin/admin): the request trace is findable in Tempo by the X-Trace-Id value; its log lines are findable in Loki by trace_id; go_ runtime metric series exist in Prometheus.
• The same trace appears in Langfuse when LANGFUSE_PUBLIC_KEY/SECRET_KEY are set (manual check, screenshot or note in the learning doc); with the keys unset, the collector logs an export failure and the lgtm pipeline is unaffected.
• /healthz requests produce no spans (filter assertion in the telemetry test).
• http.server.request.duration series carry an http.route label equal to the route pattern (test asserts the labeler is applied).
• A handler error becomes the JSON error shape with the right status, a 500 never contains the internal error text, an oversize or unknown-field body is a 400, and the status-recording writer still flushes through http.ResponseController (tests).
• metrics_names.go exists and every metric registered in this phase is registered through one of its constants (test asserts no literal metric-name string outside the file).
• content_attrs.go lists every content attribute key, and the collector config’s strip list is byte-identical to it (test compares the generated list with deploy/otelcol/config.yaml).
• Sending SIGTERM logs a drain line, flushes telemetry, and exits 0 within HANGO_SHUTDOWN_TIMEOUT.
• README.md, docs/system-architecture.md, docs/learning/README.md, docs/learning/01-foundation-and-observability-spine.md exist and are accurate.

Risk Assessment

Rollback

Phase 1 is greenfield; rollback is deleting the repository contents. Within the phase, HANGO_OTEL_ENABLED=false degrades to no-op providers and stdout logging, so a broken collector or Langfuse account never blocks development.