Context inspector

Not a v1 phase. This follow-up starts after the admin web page exists, because its main surface is a checkbox panel on that page. It needs every v1 phase it builds on to be done: the phase 7 chain, phase 10 compaction and estimator, phase 11 memory and skills, and phase 13 usage records.

Overview

Show exactly what hango sends to the model for a session, part by part, and let the user leave parts out. A model request is assembled from many sources: the agent’s system prompt, the blocks each middleware contributes (planning, filesystem, subagents, memory, skills index), the compaction summary, every message in the history and every tool definition. After phase 11 nobody can see that assembly without reading Langfuse, and nobody can change it without editing code. This follow-up adds four things on top of the phase 7 chain: an inspect endpoint that returns the next model request of a session as a list of tagged parts with token estimates, saved overrides that exclude or replace parts on future model calls, and a try endpoint that sends one model call with ad-hoc exclusions and returns the answer without touching the session, and an admin web page panel that shows the parts with a checkbox each.

The idea comes from AICoworker’s “Optimize context” panel and was listed as A4 in plans/reports/research-261014-1634-aicoworker-goclaw-hango-comparison.md. AICoworker is a gateway: it captures requests on their way to the provider and groups them by API key and conversation. hango is the harness, so it does not need to capture anything. It builds the request itself, so the chain only has to remember where each part came from.

Requirements

Functional

• Every part of an assembled model request has a stable part id that stays the same across turns and runs:

  • system:<contributor> for the blocks one contributor adds, for example system:agent, system:todo, system:filesystem, system:skills, and system:memory:/memories/AGENTS.md for one memory source.
  • tool:<name> for one tool definition.
  • message:<seq> for one persisted message, and summary for the phase 10 compaction summary.

• GET /v1/sessions/{id}/context returns what the next model call of the session would send. It builds a fresh run state the way a new run would, runs BeforeRun, assembles system blocks, tools and messages through the real chain, and stops at a capture handler instead of the provider. Nothing is written and no model is called. The response groups parts into system, tools and messages. Each part has its id, kind, label, a 200-character preview, a token estimate, included (false when an override excludes it), and override (exclude, replace or none). ?full=<part id> returns the full content of one part. Totals are given before and after overrides.

• Token estimates per part use the phase 10 Estimator. The request total uses the provider’s TokenCounter when the provider has one, and the estimate otherwise; the response says which one it used.

• When the next call would trigger summarization, the inspect response says so with would_compact: true and shows the messages as they are now. The summarizer is never called by an inspect request.

• PUT /v1/agents/{id}/context-overrides replaces the agent’s overrides for system:* and tool:* parts. They apply to every session of the agent. PUT /v1/sessions/{id}/context-overrides replaces the session’s overrides for message:* and summary parts. Each override is {part, action, replacement} with action exclude or replace. GET on both paths returns the current set. An empty list clears it.

• replace is allowed for system and message parts and refused for tool parts with 422, because a changed tool schema would no longer match the tool’s argument validation.

• Overrides are loaded once per run in BeforeRun, so a change applies from the next run. A run never sees its context change between turns.

• An override set must leave a valid request. Excluding a message that holds a tool_use also excludes the message with its tool_result, and the other way round. The PUT response lists every part that was added this way. A set that would leave no user message, or would start the history with an assistant message, is refused with 422 and the reason. The check reuses the phase 10 post-compaction validator.

• An override for a part that no longer exists (a deleted memory file, a tool removed from the agent) is kept and reported as stale: true by the inspect endpoint. It does nothing until the part comes back.

• POST /v1/sessions/{id}/context/try takes {exclude: [...], replace: {...}, message: "optional extra user text"}, applies the saved overrides plus the ones in the body, sends one model call and returns the assistant message, the stop reason and the usage. Tool calls in the answer are returned, not executed. Nothing is appended to the session and no run is created. The call has a timeout (HANGO_CONTEXT_TRY_TIMEOUT, default 120s).

• hango context show --session <id> prints the inspect response as a grouped table: a checkbox column, part id, label, preview and tokens, with group totals. hango context exclude --session <id> <part>..., hango context include ... and hango context try --session <id> [--exclude <part>...] [--message <text>] wrap the other endpoints. Like hango eval, the command is an HTTP client of hango’s public API and reads HANGO_API_URL and HANGO_API_KEY.

• The admin web page gets a “Context” panel for a session. It shows three groups, system, tools and messages, with a checkbox, label, preview and token count per part, group totals, and the totals before and after overrides. Clicking a part opens its full content, and a system or message part can be edited, which saves a replace override. Unchecking or checking parts and pressing Save sends the PUT for the agent or the session. A “Try” button sends the current selection to the try endpoint and shows the answer next to the panel. Parts added as pair partners, stale overrides, cached_prefix and would_compact are shown as labels. The panel uses only the HTTP API above, like every other admin page view.

Authorization

The rows follow GET /v1/sessions/{id}/messages and PATCH /v1/agents/{id} from phase 4: whoever may read a session’s messages may see its context, and whoever may change an agent may change its overrides. A target in another tenant or another member’s target is 404.

Non-functional

• Overrides never change persisted history. Like compaction in phase 10, they act only on the outgoing request.
• The inspect and try endpoints return message and memory content to the API caller, the same content GET /v1/sessions/{id}/messages already returns. They never put content on spans or logs. Content on telemetry stays controlled by tenants.trace_llm_content as before.
• Excluding or replacing a system part changes the prompt prefix, so the next call misses the prompt cache. The inspect response marks parts before the cache breakpoint with cached_prefix: true so the user can see the cost.
• A try call is a real model call. It is recorded in usage_records with purpose context_try and no run, and counts against the tenant quota and the API key rate limit like any other call. The migration of this follow-up widens the purpose check with context_try and makes usage_records.run_id nullable.
• An inspect request costs one chain assembly and no model call. Its latency is bounded by BeforeRun, which phase 11 already caps.
• No new Go package. The overrides middleware and the part ids live in internal/agent, the handlers in internal/server, the store in internal/store/pg, and the CLI in cmd/hango. The panel lives in the admin web page’s source tree.

Architecture

// internal/agent/parts.go
type PartID string // "system:memory:/memories/AGENTS.md", "tool:read_file", "message:42", "summary"

type Part struct {
	ID     PartID
	Kind   string // "system", "tool", "message", "summary"
	Label  string
	Tokens int
}

// SourcedBlock keeps provenance next to the block. provider.Block stays unchanged,
// so nothing hango-specific is ever serialized to the provider.
type SourcedBlock struct {
	Part  PartID
	Block provider.Block
}

// internal/agent/chain.go — SystemBlocks keeps its role but returns provenance.
func (c *Chain) SystemBlocks(def store.Agent, s *State) []SourcedBlock

// internal/agent/mwoverrides.go
type Override struct {
	Part        PartID
	Action      string // "exclude" or "replace"
	Replacement string
}

type ContextOverridesMiddleware struct {
	Store store.ContextOverrides
} // RunHook (load agent and session overrides once) + ModelWrapper (messages and summary)

// internal/agent/inspect.go
type Snapshot struct {
	System, Tools, Messages []Part
	WouldCompact            bool
	TotalBefore, TotalAfter int
	Counted                 string // "provider" or "estimate"
}

// Inspect assembles the next request of a session through the real chain and a
// capture handler. It never calls the provider or the summarizer.
func Inspect(ctx context.Context, deps Deps, sessionID uuid.UUID, extra []Override) (*Snapshot, *provider.Request, error)

Where each override applies:

1. System parts are dropped or replaced inside Chain.SystemBlocks, before PromptCacheMiddleware picks its breakpoint index, so the breakpoint lands on a block that is really sent.
2. Tool parts are dropped in the ToolAllowlist slot, which is last in the stack, so no later middleware can add an excluded tool back.
3. Message and summary parts are dropped or replaced by ContextOverridesMiddleware, a new slot right before Summarization, so compaction and the token budget see the filtered history.

-- migrations/NNNNN_context_overrides.sql (the next free migration number)
create table context_overrides (
  id uuid primary key,
  tenant_id uuid not null references tenants(id) on delete cascade,
  agent_id uuid references agents(id) on delete cascade,
  session_id uuid references sessions(id) on delete cascade,
  part text not null,
  action text not null check (action in ('exclude','replace')),
  replacement text,
  created_by uuid not null references users(id),
  created_at timestamptz not null default now(),
  check (num_nonnulls(agent_id, session_id) = 1),
  check ((action = 'replace') = (replacement is not null)),
  check (agent_id is null or part like 'system:%' or part like 'tool:%'),
  check (session_id is null or part like 'message:%' or part = 'summary')
);
-- the same migration widens usage_records for try calls
alter table usage_records alter column run_id drop not null;
-- and replaces the purpose check so it also allows 'context_try'

create unique index context_overrides_agent_part_idx on context_overrides (tenant_id, agent_id, part) where agent_id is not null;
create unique index context_overrides_session_part_idx on context_overrides (tenant_id, session_id, part) where session_id is not null;

A PUT deletes the target’s rows and inserts the new set in one transaction. Queries live in internal/store/pg/queries/context_overrides.sql and go through sqlc like every other table since phase 2.1.

Instrumentation

• Span agent.context_inspect for an inspect request, with hango.context.parts.system, hango.context.parts.tools, hango.context.parts.messages, hango.context.tokens.before, hango.context.tokens.after, hango.context.would_compact.
• Attributes on every chat span: hango.context.overrides (number of active overrides) and hango.context.excluded_tokens. A run whose context was changed by hand is then visible in Tempo and Langfuse, which answers “why did the agent forget X” before anyone suspects the model.
• Span event agent.context_override_stale per override whose part no longer exists, and agent.context_override_paired when a tool_use/tool_result partner was excluded with its pair.
• A try call is an ordinary chat span with hango.context.try = true under a context_try SERVER span, so it shows up in Langfuse beside normal generations but can be filtered out.
• Metrics: hango.context.inspect.duration histogram, hango.context.try counter (attribute outcome), hango.context.overrides.active gauge per tenant.
• Audit actions, added to the closed set from phase 2: context.override for both PUT routes (target kind agent or session, detail holds the part ids and actions, never replacement text) and context.try (detail holds the excluded part ids and the model).

Related Code Files

Create

• migrations/NNNNN_context_overrides.sql, using the next free migration number when this follow-up starts.
• internal/store/pg/queries/context_overrides.sql, internal/store/pg/context_overrides.go, and the isolation cases in internal/store/pg/tenancy_test.go.
• internal/agent/parts.go, internal/agent/parts_test.go — part ids, labels, previews.
• internal/agent/mwoverrides.go, internal/agent/mwoverrides_test.go.
• internal/agent/inspect.go, internal/agent/inspect_test.go.
• internal/server/handlers_context.go, internal/server/handlers_context_test.go.
• cmd/hango/context.go, cmd/hango/context_test.go.
• docs/learning/context-inspector.md.
• The Context panel component and its API client in the admin web page.

Modify

• internal/store/store.go — the ContextOverrides sub-store on Scoped.
• internal/agent/chain.go — SystemBlocks returns []SourcedBlock, applies system overrides, and the base handler strips provenance before building provider.Request.System.
• internal/agent/mwstack.go, internal/agent/mwstack_test.go — the ContextOverrides slot before Summarization; the order test asserts it.
• internal/agent/mwallowlist.go — drop excluded tools.
• internal/agent/mwcache.go — pick the breakpoint index after system overrides are applied.
• internal/agent/mwsummarize.go — in a capture-only call, report would_compact without calling the summarizer.
• Every PromptContributor (mwtodo.go, mwfilesystem.go, mwsubagent.go, mwmemory.go, mwskills.go) — tag its blocks with its part id.
• internal/server/authz.go, internal/server/authz_test.go — the five route rows above.
• internal/store/audit.go — context.override and context.try in the action set.
• internal/config/config.go — HANGO_CONTEXT_TRY_TIMEOUT.
• internal/usage and its store — the context_try purpose and records without a run.
• internal/telemetry/attrs.go, internal/telemetry/metrics_names.go — the names above.
• docs/system-architecture.md, docs/learning/README.md, docs/roadmap.md.

Implementation Steps

1. Write the migration (the table plus the usage_records change), the query file and the scoped store with List(target) and Replace(target, []Override) in one transaction. Add cross-tenant and intra-tenant cases for context_overrides to the isolation table.
2. Write parts.go: part id constructors and parsing, the label for each kind (the tool name, #seq role, the memory path), and a preview that cuts at 200 characters on a rune boundary.
3. Change Chain.SystemBlocks to return []SourcedBlock and tag the blocks of every existing contributor. Make the base handler strip provenance. Every existing chain and order test must pass unchanged apart from the type.
4. Write mwoverrides.go: BeforeRun loads the agent’s and the session’s overrides into typed private state. WrapModelCall drops and replaces message and summary parts, then runs the phase 10 pair validator on the result. A validation failure here means the stored set went bad after it was saved (a later message changed the pairing); the middleware then sends the request without the message overrides, logs at error and records a span event. It never sends an invalid request.
5. Apply system overrides in SystemBlocks before the cache breakpoint is chosen, and tool overrides in the allowlist slot. Add the ContextOverrides slot to DefaultStack and the order test.
6. Write inspect.go: build the chain for the session’s agent, run BeforeRun on a fresh state built from the persisted messages, call ModelCall with a capture handler, and turn the captured request into a Snapshot. Mark the summarizer call as capture-only so it reports would_compact instead of calling the model.
7. Write handlers_context.go: the inspect, overrides and try routes with their authorization rows and audit actions. The overrides PUT validates the set against the current history, adds pair partners, and returns the stored set plus the added parts. The try route runs Inspect with the extra overrides, sends the captured request through the provider with the timeout, and returns the answer.
8. Write cmd/hango/context.go with show, exclude, include and try.
9. Build the Context panel in the admin web page on top of the same routes.
10. Add the instrumentation and the runbook step, then write the tests and the learning doc.

Learning Doc Outline

docs/learning/context-inspector.md must contain:

• Concept: a model request is assembled, not written. Where each part comes from, how much each costs, and why “what did the model actually see?” is the first debugging question for a bad answer.
• Design choices: provenance on blocks instead of parsing the prompt afterwards; preview by dry assembly through the real chain instead of a second assembly path that can drift; overrides loaded once per run; agent-level overrides for system and tools, session-level for messages; replace refused for tools.
• Why overrides must keep the request valid: the tool_use/tool_result pairing, a history that starts with a user message, and what the provider does when either rule is broken.
• The cost of editing the prefix: the prompt cache, and why the response marks cached_prefix.
• Trade-offs: preview of the next call vs a stored copy of every past call (storage, privacy of family messages, Langfuse already keeps the past when content tracing is on); saved overrides vs one-off tries; the admin web page panel vs the CLI table.
• AICoworker comparison: a gateway that captures each request, groups it by API key and conversation, applies unchecked system parts and tools to every conversation of the key and unchecked messages to one conversation, and lets the user view and edit each part. hango keeps the same grouping with agent in place of API key, and adds the pairing check and the cache marker.
• goclaw comparison: GET /v1/agents/{id}/system-prompt-preview in internal/http/agents_prompt_preview.go renders the system prompt for a prompt mode with named sections, a token count and the tool list. It is read-only and covers the system prompt only.
• deepagents comparison: no inspector in the library; LangSmith shows the input of each model call after the fact.
• How to debug this layer: a part missing from the inspect response means its contributor did not tag it (check the contributor’s part id); a run that behaves differently from the inspect preview means overrides changed after the run started (overrides load at BeforeRun); an unexpected cache miss after an override shows as a drop in hango.cache.read_ratio together with hango.context.overrides on the chat span; a stale override shows as agent.context_override_stale.
• Exercises: exclude the skills index and ask the agent about a skill; exclude an old tool result and watch the pair partner go with it; use try to compare an answer with and without memory.
• Time budget: 9h API, store and CLI + 4h admin web page panel + 2h learning doc.

Success Criteria

• go build ./... && go vet ./... && go test -race ./... pass with Docker running, and golangci-lint run reports 0 issues.
• The inspect response for a session lists every system contributor, every tool and every message of the next request, and the sum of part estimates equals Estimator.EstimateRequest on the captured request (test).
• An inspect request calls neither the provider nor the summarizer, and writes no row (test with a fake provider that fails on any call).
• With message:5 excluded on a session, the next model call of a new run in that session does not contain message 5, and GET /v1/sessions/{id}/messages still returns it (integration test).
• Excluding the message with a tool_use also excludes its tool_result message, and the PUT response lists the added part (test).
• An override set that leaves no user message is refused with 422 (test).
• replace on a tool:* part is refused with 422 (test).
• An excluded tool is absent from the request’s tool list even when a skill’s allowed_tools names it (test).
• The cache breakpoint index lands on a block that is sent when a system part before it is excluded (test).
• A try call returns an answer, appends no message and creates no run, and its chat span carries hango.context.try = true (test).
• Every route row in the authorization table is covered, including a member reading another member’s session context (404) and a member changing another member’s agent overrides (404).
• Cross-tenant and intra-tenant isolation cases for context_overrides pass.
• Both PUT routes write a context.override audit row and the try route writes context.try; no replacement text appears in detail (test).
• hango context show --session <id> prints the grouped table against a running server.
• A try call writes one usage_records row with purpose context_try and no run, and is refused with 402 when the tenant quota is used up (test).
• In the admin web page, unchecking a message and saving excludes it from the next run, and the panel totals change to match the inspect response.
• In Tempo, a run with an active override shows hango.context.overrides on its chat spans.

Risk Assessment

Rollback

goose down one step drops context_overrides and restores the usage_records constraints (after deleting context_try rows). Removing the ContextOverrides slot from DefaultStack and the tool filter from the allowlist slot restores the behaviour before this follow-up. Hiding the panel in the admin web page removes the UI. The provenance on system blocks can stay, because it is never sent.