hango Roadmap: Anatomy of an Agent Harness and the Order to Build It

This is the master document for hango. It explains what an agent harness is made of, why each part exists, in which order to build the parts, and how each step opens the seam for the next. The detailed contract for each step lives in plans/260922-0123-hango-agent-harness-roadmap/phase-NN-*.md and is refined right before that step is implemented. Every step ends with a working, tested binary and a learning doc in docs/learning/.

Sources behind the design: the goclaw reference (~/ws/personal/goclaw, production multi-tenant Go gateway), langchain deepagents (middleware-composed harness core), and the research reports in plans/reports/. Locked stack decisions are in docs/tech-stack.md.

1. What a harness contains

An agent harness is the machinery around a language model that turns “call the model” into “run an agent safely, repeatedly, for many users, and know what happened”. The model is the smallest part. The harness is these layers:

The observability layer is listed last but built first. Every other layer is designed to be visible through it.

    flowchart TB
  C[Client] -->|POST run, GET SSE| S[server: net/http + auth + tenancy]
  S --> R[run: durable runs, lanes, broadcaster]
  R --> L[agent loop + middleware chain]
  L --> P[provider: Anthropic, OpenAI adapters]
  L --> T[tool registry + builtins]
  L --> F[fs Backend: files, offload, memory, skills]
  L --> U[usage: tokens, cost, quotas]
  S --> DB[(Postgres: tenant_id on every table)]
  R --> DB
  F --> DB
  U --> DB
  O[telemetry: traces, metrics, logs] -.-> S & R & L & P & T & F & U
  O -->|OTLP| COL[OTel Collector] --> G[Grafana LGTM] & LF[Langfuse]
  

2. Design principles that hold across every step

• Observability is foundation. One OpenTelemetry tracer, meter, and logger provider from phase 1. Logs go through the OTel slog bridge so every log line carries the trace and span IDs. One OTLP egress to a collector, which fans out to Grafana LGTM (Tempo traces, Loki logs, Prometheus metrics) and Langfuse (LLM-level traces, generations, tokens, cost). Every later phase declares the spans, attributes, metrics, and log fields it adds, and its learning doc has a “how to debug this layer” section.
• Tenant scoping by construction. Every tenant-owned table has tenant_id uuid NOT NULL with a tenant-leading index. Queries go through a scoped store built from the request context that fails closed when the tenant is missing. Each table gets a cross-tenant read test. Inside a tenant, members see and change only what they created; owners and admins see everything, enforced per route. Row-level security arrives late as a backstop, never as the primary mechanism.
• Durable runs, not request-bound loops. A run is a row with an owner and a lease. A worker claims it with a conditional update, heartbeats, and executes it. SSE subscribes to its persisted event log and can resume from the last event ID. On restart, expired leases are reclaimed and runs resume at the last completed turn. This is what makes cancel, drain, crash recovery, and human-in-the-loop possible without redesign.
• One loop, extended by middleware. After the loop exists, no feature modifies it directly. Planning, filesystem, subagents, summarization, memory, skills, and approval are all middlewares in one ordered slice. goclaw’s dual loop implementations are the cautionary example.
• Everything is a filesystem. Large tool results, evicted history, memory, and skills all ride on one Backend interface. Building that interface once unlocks four features.
• Audit everything. Every mutating API request and every approval decision writes an append-only audit row carrying the actor, action, target, request ID, and trace ID. Audit rows, run events, spans, and logs all join on the trace ID, so “who did what, and what happened next” is one query.
• Errors are data. A failing tool returns a tool result flagged as error. The model recovers. The loop never panics and never hides the failure from the trace.
• Policy lives in tool descriptions. No long authored system prompt. The prompt stays short and dynamic; each tool carries its own usage policy.
• Minimal dependencies, explicit SQL, no framework. Standard-library net/http, pgx, goose, the official Anthropic SDK behind our own interface, OpenTelemetry. Nothing that hides the loop, the queries, or the tenancy.

3. Build order and why

Each entry says what the step delivers, why it comes now, and the seam it opens for later steps. Effort estimates are in the plan.

The entries are numbered by phase, but the build order is 1 → 3 → 2 → 4 → 5 → … → 13 → 18 → 19 → 14 → 15 → 16 → 20 → 17. Phase 3 depends only on phase 1, so a real model streams into Tempo and Langfuse before the database work starts. Phase 5 stops twice for review: 5a for tools and the loop against the fake provider, 5b for durability. Phases 18 and 19 come right after phase 13, because the first users are the owner’s family and they need a chat client before the dashboard and hardening work. Phase 20 comes after phase 16, so its tables start with row-level security.

1. Foundation and observability spine. Module, composition root, env config, net/http server, health and readiness, graceful shutdown, Makefile, lint, distroless Dockerfile, compose with Postgres, OTel Collector, and Grafana LGTM, Langfuse wired through the collector. Why now: every later step must be observable from its first test run. Seam: internal/telemetry constants for tenant, agent, session, run, and turn attributes that all later spans use.
2. Persistence and tenancy core. pgx pool with tracing, embedded migrations, tenants, users, API keys, audit events, scoped store, auth middleware with per-route authorization, one-shot bootstrap admin key, admin API, audit middleware, a per-package testcontainers database for integration tests. Why now: tenancy retrofitted later leaves permanent residue, as goclaw shows. Seam: the scoped store pattern every later table follows, and the cross-tenant test template.
3. Provider abstraction and Anthropic adapter. Normalized message, block, and stream event model; Provider interface; Anthropic adapter with stream accumulation, thinking passback, stop reasons, retries. Chat spans with GenAI attributes and token metrics. Why now: the loop needs a provider, and the normalized model must be designed so a second adapter can synthesize block framing. Seam: the Event model the SSE layer later passes through unchanged.
4. Agents, sessions, messages. Tenant-scoped agents created by users, sessions, append-only messages with sequence numbers. Why now: the loop needs somewhere to read and append history. Seam: per-message addressing that compaction and subagents rely on.
5. Tools, agent loop, durable runs. Tool interface with result split, registry, two safe builtins, think-act-observe loop with iteration cap and dangling tool-call repair, runs table, worker pool with per-session serialization, create run and poll endpoints. Root invoke_agent span, execute_tool spans, run metrics, trace ID stored on the run. Why now: the loop is the heart; running it as a durable job from day one avoids a later rewrite. Seam: the run event model that streaming subscribes to next.
6. Streaming, cancellation, drain. Persisted run events, in-memory broadcaster, SSE endpoint with resume, keepalives, cancel endpoint, scheduler lanes, draining on shutdown, panic-recovered goroutines, goroutine-leak test. Why now: the harness becomes usable interactively, and shutdown semantics are settled before features multiply. Seam: lanes that subagents reuse.
7. Middleware chain and planning tool. Hook interfaces, ordered chain, loop refactored onto it, write_todos middleware, repeat-call detection as middleware. Why now: every following feature is a middleware; the chain must exist before the first one. Seam: the hook set and the prompt-contributor mechanism.
8. Filesystem backend and large-result offload. Backend interface, Postgres-backed store namespaced by tenant and agent, file tools, path validation, offload of oversized tool results to files. Why now: the first context-defense layer and the substrate for memory, skills, and history offload. Seam: the reserved prefixes that memory uses.
9. Subagents. task tool, subagent specs on the agent, general-purpose default, isolated child history, compact result return, depth cap, subagent lane, nested agent spans. Why now: needs loop, tools, filesystem, and lanes. Seam: the child-run pattern reused by approval resume.
10. Context management. Token estimation, tool-argument truncation, non-mutating summarization, history offload, overflow retry, prompt-cache breakpoints. Why now: real sessions now exceed the window; needs message addressing and the backend. Seam: the compaction event and cache breakpoint that memory placement depends on.
11. Long-term memory and skills. /memories/ and /skills/ as reserved prefixes on the one backend, AGENTS.md injection after the cache breakpoint, SKILL.md progressive disclosure. Why now: trivial once the backend and compaction exist. Seam: the memory convention the user can extend.
12. OpenAI-compatible provider and provider registry. Adapter that synthesizes block framing and accumulates tool arguments by index, a provider registry with an injected credential lookup resolving from server config, per-agent provider and model selection. Why now: proves the phase 3 abstraction while it is still cheap to fix. Seam: the credential lookup that a later bring-your-own-key feature plugs into, and the registry that usage pricing keys on.
13. Usage, quotas, rate limits. Usage records per turn with cache tokens, pricing table, tenant quotas, per-key rate limiting, cost attributes into Langfuse. Why now: needs both providers to be meaningful. Seam: the data dashboards read.
14. Dashboards, alerts, debugging runbook. Provisioned Grafana dashboards, trace-to-log correlation, alert rules, Langfuse conventions, and an end-to-end runbook from a failed run ID to the exact span, log line, and tool call. Why now: all signals exist; this step turns them into answers. Seam: the operating model for production.
15. Human in the loop. Per-tool interrupt configuration, awaiting_approval run state, persisted pending tool calls, approval endpoint, resume with span links. Why now: it forces the loop to be a resumable step function, which everything before has prepared. Seam: the approval pattern for future permission systems.
16. Production hardening and RLS backstop. Row-level security with a non-owner role, security review, load and soak tests, goroutine-leak profiling, CI with Postgres, stamped release build, deployment guide, final architecture doc. Why now: hardening a finished shape.
17. Evaluation loop. A fixed task set in evals/, a hango eval command that drives every task through real runs and scores it with deterministic checks, scores and dataset runs in Langfuse on each run’s trace, and hango eval compare between two runs of the set. Why last: it needs every feature it measures, and it is the tool for every change after v1. Seam: the regression suite for agent behaviour.
18. Telegram channel. A Telegram bot with long polling, private chats only, a static map from Telegram user to hango user and agent, replies streamed by editing one message and finished as Telegram HTML converted from Markdown, /new, /cancel, /whoami and /help registered as the bot menu, rate-limit and conflict handling, and run creation shared with the HTTP handler. Why now: the first users are family members who will not call an HTTP API; it needs streaming (6) and quotas (13) so a chat cannot run up the API bill. Seam: run creation as a function any channel calls, which web chat reuses later.
19. Claude Code CLI runtime for the owner. Agents with provider = "claude_cli" run the real claude -p as a child process on the owner’s subscription, with hango’s tools exposed over a loopback MCP bridge, owner-only checks at agent save, run creation and claim, and usage kept out of tenant quotas. Why now: it reuses lanes (6), tool offload (8) and usage accounting (13), and it is only useful once a channel exists. Seam: a second runtime beside hango’s own loop, which shows what observability a harness loses when it hands the loop to another program.
20. Telegram media, pairing and approvals. Photos and albums as image blocks, documents saved to the filesystem, voice notes transcribed (Vietnamese included), files sent back with attach_file, pairing codes instead of config edits, approval buttons signed with an HMAC, and status reactions. Why now: approval buttons need phase 15, and the new tables should start with the row-level security from phase 16. Seam: the provider image block and the attachment run event, which the web chat reuses.

4. Where to look when something goes wrong

The span hierarchy is the map. A run produces one trace: invoke_agent {agent} at the root, a chain span per turn, a chat {model} client span per provider call, an execute_tool {tool} span per tool call, and a nested invoke_agent per subagent. Every span carries the tenant, agent, session, run, and turn attributes. Every log line carries the trace and span IDs. The run row stores the trace ID and returns it from the run endpoint.

Phase 14 writes this table out as a full runbook with worked failure signatures.

5. How each step is planned and delivered

• Before a step starts, its phase file is reviewed and refined against the code that now exists. The phase file is the contract: requirements, architecture with interface and DDL sketches, files to create, numbered steps, success criteria with exact commands, and risks with pre-decided responses.
• Delivery of a step: implementation, tests including at least one against real Postgres when the store is touched, build and vet and lint green, spans and metrics visible in Grafana and Langfuse where relevant, learning doc written, architecture doc updated, success criteria ticked in the phase file, plan.html regenerated with go run build_plan_html.go from the plan directory, commit.
• The learning doc for each step covers the concept, hango’s design and why, how goclaw does it, how deepagents does it, where hango diverges, and how to debug the layer.
• Review gate: the user reviews the binary, the tests, and the learning doc before the next step begins.

6. Out of scope for this roadmap

MCP bridge (MCP servers as tool implementations), bring-your-own provider credentials per tenant (needs a base URL allowlist and encrypted storage), Anthropic server tools and the opaque block passthrough they require, multi-replica deployment (shared rate limiter, distributed run registry, event fan-out), sandboxed execution tool, chat channels other than Telegram, Telegram group chats (goclaw design kept in phase 20), Telegram webhook mode, web chat UI (planned after Telegram) and an admin web page, both in Svelte 5, an LLM gateway with per-user quotas, desktop app, i18n, provider breadth beyond Anthropic and OpenAI-compatible. Each is a natural extension of a seam above and is listed as a follow-up in the plan.