Context management and summarization

Overview

Keep long sessions inside the model’s context window without destroying the transcript. Add a token-estimation interface (heuristic now, tiktoken later), pre-summarization tool-argument truncation, non-mutating summarization triggered at 0.85 of the window keeping the last 0.10, history offload to /conversation_history/{session_id}.md, retry-on-overflow, and Anthropic prompt-cache breakpoints. Nothing here mutates the persisted message log: compaction is applied inside WrapModelCall only, so the raw history stays replayable.

Requirements

Functional

• Budgeting is computed over the **whole request**: `Estimator.EstimateRequest(req)` counts system blocks (including the memory and skills fragments contributed in phase 11), every message block, and the serialized tool definitions. Estimating only `messages` under-counts by the largest fixed cost in the request and is why a "0.85 trigger" would still overflow.
• Optional `TokenCounter` capability: a provider may implement `CountTokens(ctx, req) (int, error)`; the Anthropic adapter implements it over `Messages.CountTokens`. When present it is used for the trigger decision (with the heuristic as the fallback on error or timeout), so the threshold is based on real numbers rather than `bytes/4`.
• A model-profile table in config supplies max_input_tokens per model with a documented default.
• Tool-argument truncation runs first: old write_file/edit_file/large tool arguments in messages older than the keep window are clipped to a preview, which often avoids summarization entirely.
• Summarization triggers when estimated input tokens exceed TriggerFrac (0.85) of the window; it keeps the last KeepFrac (0.10) of messages verbatim, summarizes the rest with a dedicated summarizer call, and prepends the summary as a single user message.
• **Boundary rule:** the computed keep boundary is snapped *backwards* to the nearest assistant message that has no `tool_use` block left unmatched by a `tool_result` in the kept slice. A boundary that lands inside a tool batch would send a `tool_result` whose `tool_use` was evicted (or vice versa) and the provider 400s permanently for that session.
• A post-compaction validator runs on the compacted slice before it is sent: every `tool_result` must have its `tool_use` earlier in the slice and every `tool_use` must have its `tool_result`. A validation failure aborts the compaction, logs at error, and falls back to mechanical truncation — it never sends an invalid request.
• Evicted messages are appended to /conversation_history/{session_id}.md through the filesystem backend, and the summary embeds that path with a read_file instruction.
• The summarization prompt must include an identifier-preservation clause (ids, paths, file names, tool-call outcomes, open questions, current plan).
• provider.ErrContextOverflow (or stop reason model_context_window_exceeded) triggers summarization and one retry of the same turn. A second overflow does not fail the run: it degrades to mechanical truncation (drop whole oldest turns, preserving tool-batch integrity, until the request fits) and retries once more. Failing the run is the last resort after truncation cannot fit even the keep window.
• Prompt-cache breakpoints are placed by `PromptCacheMiddleware` **during chain-time system-block assembly**: it records the index of the last stable system block and sets `CacheControl` there, and sets a second breakpoint on the summary message that opens the compacted history. Because memory (phase 11) contributes blocks later in the chain, its blocks land at a higher index than the breakpoint by construction, not by convention. The middleware skips breakpoint placement entirely when the resolved provider reports `Capabilities.PromptCaching == false`.
• A compact_conversation tool lets the model compact on demand.

Non-functional

• Compaction is recorded in State.Private and applied only to the outgoing ModelRequest.Messages; State.Messages and the messages table are never rewritten.
• Summarizer calls use the same provider but a configurable (cheaper) model, and are attributed separately in usage.
• Summarization is idempotent per turn: a second WrapModelCall in the same turn reuses the existing compaction event.
• Deterministic tests: the estimator is injectable, and the fake provider scripts both the summarizer response and the overflow error.

Architecture

// internal/agent/tokens.go (extended)
type Estimator interface {
	Estimate(text string) int
	EstimateRequest(req *provider.Request) int // system blocks + messages + tool defs. Red team: F9
}

// Optional provider capability; used for the trigger decision when available. Red team: F9
type TokenCounter interface {
	CountTokens(ctx context.Context, req provider.Request) (int, error)
}

type ModelProfile struct {
	MaxInputTokens int
	TriggerFrac    float64 // 0.85
	KeepFrac       float64 // 0.10
	SummarizerModel string
}

// internal/agent/mwsummarize.go
type SummarizationMiddleware struct {
	Est      Estimator
	Profile  ModelProfile
	Backend  fs.Backend
	Provider provider.Provider
} // ModelWrapper + ToolProvider (compact_conversation)

type compaction struct {
	AtTurn      int
	Summary     string
	KeptFrom    int    // index into State.Messages
	OffloadPath string
	Tokens      struct{ Before, After int }
}

// internal/agent/mwtruncate.go
type ToolArgTruncationMiddleware struct {
	Est      Estimator
	MaxArgTokens int
	KeepRecent   int // messages exempt from truncation
} // ModelWrapper, runs inside Summarization so summarization sees truncated args

// internal/agent/mwcache.go
type PromptCacheMiddleware struct{} // ModelWrapper: sets CacheControl breakpoints

No new tables; sessions gains counters.

-- migrations/00008_session_context_counters.sql
alter table sessions add column last_input_tokens int not null default 0;
alter table sessions add column compactions int not null default 0;
-- Red team: F9 — no `summary` column: nothing reads it. The authoritative summary is the
-- compaction record in runs.state_json plus /conversation_history/{session_id}.md.

Instrumentation

• Span event gen_ai.conversation.compacted on the turn span with hango.tokens.before, hango.tokens.after, hango.messages.evicted, hango.compaction.reason (threshold|overflow|tool), hango.fs.path of the history offload; the attribute gen_ai.conversation.compacted=true is also set on the following chat span so a compacted turn is filterable.
• The summarizer’s own provider call is a separate chat {summarizer model} CLIENT span with hango.call.purpose=summarize, so summarizer cost never hides inside the main generation.
• Span event provider.context_overflow_retry when an overflow error triggers compaction and retry.
• Metrics: hango.context.input_tokens histogram (attribute gen_ai.request.model), hango.context.compactions counter (attribute reason), hango.context.evicted_messages histogram, hango.context.tool_args_truncated counter, gen_ai.client.token.usage gains hango.call.purpose so summarizer tokens are separable, hango.cache.read_ratio histogram derived from cache-read vs input tokens.
• Log fields: one info line per compaction with session id, before/after tokens, evicted count and the offload path.

Related Code Files

Create

• internal/agent/mwsummarize.go, internal/agent/mwsummarize_test.go.
• internal/agent/mwtruncate.go, internal/agent/mwtruncate_test.go.
• internal/agent/mwcache.go, internal/agent/mwcache_test.go.
• internal/agent/summarize_prompt.go — the summarization prompt with the preservation clause.
• internal/agent/profile.go — model profile table loaded from config with a documented fallback (170k trigger / keep 6 messages when a model is unknown).
• migrations/00008_session_context_counters.sql.
• docs/learning/10-context-management.md.

Modify

• internal/agent/tokens.go — add EstimateRequest, the optional TokenCounter capability, and the Estimator interface docs.
• internal/agent/mwstack.go — fill the Summarization slot (after Filesystem/offload, after Subagent) with truncation nested inside it, and the PromptCache slot before the Memory slot.
• internal/agent/loop.go — on ErrContextOverflow, mark the compaction reason and retry the turn once.
• internal/store/pg/session.go — update last_input_tokens and compactions at the turn boundary.
• internal/config/config.go — HANGO_MODEL_PROFILES (JSON), HANGO_SUMMARIZER_MODEL, HANGO_CONTEXT_TRIGGER_FRAC, HANGO_CONTEXT_KEEP_FRAC, HANGO_TOOL_ARG_MAX_TOKENS.
• docs/system-architecture.md, docs/learning/README.md.

Implementation Steps

1. Extend internal/agent/tokens.go with EstimateRequest (system blocks + all message blocks + serialized tool definitions, plus fixed per-message and per-tool-block overheads) and add the optional TokenCounter capability with the Anthropic Messages.CountTokens implementation and a bounded timeout plus heuristic fallback.
2. Write internal/agent/profile.go: parse HANGO_MODEL_PROFILES as {"model": {"max_input_tokens": n, ...}}, fall back to 170000 trigger and a 6-message keep for unknown models, and log once per unknown model.
3. Write internal/agent/mwtruncate.go: a ModelWrapper that copies the request’s message slice and clips tool-use argument JSON and tool-result text in messages older than KeepRecent to a preview with a size note. Never touch State.Messages.
4. Write internal/agent/summarize_prompt.go: instruct the summarizer to preserve identifiers, file paths, tool outcomes, decisions, the current plan and open questions verbatim, and to write the summary as a briefing for a fresh agent.
5. Write internal/agent/mwsummarize.go:
   • In WrapModelCall, estimate the whole outgoing request (EstimateRequest, or CountTokens when the provider supports it). If under the trigger, call next.
   • Otherwise compute the keep boundary, snap it backwards to an assistant message with no unmatched tool_use, append the evicted messages as markdown to /conversation_history/{session_id}.md through the backend, call the summarizer model with the evicted slice, build []provider.Message{summaryUserMessage} + kept, run the tool_use/tool_result validator, store the compaction in typed private state, set the request messages to the compacted slice, then call next.
   • Record the compaction span event and metrics; the session counters are written by the loop at the turn boundary, not here.
   • Expose compact_conversation as a ToolProvider that forces a compaction on the next model call with reason tool.
6. Handle overflow: in loop.go, when a model call returns ErrContextOverflow or the stop reason is ContextExceeded, set a forced-compaction flag with reason overflow and retry the turn once. On a second overflow, degrade to mechanical truncation (drop whole oldest turns, never splitting a tool batch) with reason truncate and retry once more; only a third failure fails the run.
7. Write internal/agent/mwcache.go: during chain-time system-block assembly, record the index of the last stable system block and attach CacheControl there, plus a breakpoint on the summary message that opens the compacted history. Skip entirely when Capabilities.PromptCaching is false. Comment states that anything contributing prompt blocks later in the chain (memory in phase 11) lands after the breakpoint index by construction.
8. Insert truncation, summarization and prompt cache into DefaultStack in the documented slots; update the pinned order test.
9. Tests: estimator monotonicity and that EstimateRequest grows when tool definitions or system blocks grow; truncation clips only old messages and leaves recent ones intact; summarization triggers at the threshold, produces a compacted request, leaves State.Messages and the DB untouched (asserted by reloading history), and writes the offload file; the summary contains the offload path; a keep boundary computed inside a tool batch is snapped back so the compacted slice has no orphaned tool_use or tool_result, asserted by the validator over a fixture whose 0.10 boundary falls between a tool_use and its tool_result ; validator failure falls back to truncation instead of sending; overflow → compaction → second overflow → mechanical truncation → success (fake provider); compact_conversation forces compaction; cache-control placement test asserting the breakpoint index in the assembled system blocks and on the summary message, plus a no-op assertion when PromptCaching is false.
10. Update docs and write the learning doc.

Learning Doc Outline

docs/learning/10-context-management.md must contain:

• Concept: the context window as a budget, and the three independent mechanisms (per-result offload, argument truncation, summarization) with the reason one is never enough.
• Design choices: non-mutating compaction inside WrapModelCall, history offload plus a path in the summary so the agent can recover detail, identifier-preservation clause, separate summarizer model, overflow-triggered retry, cache breakpoints before the memory slot.
• Trade-offs: 0.85/0.10 defaults; summarizing vs sliding window vs retrieval; heuristic estimation vs tiktoken (and why the interface exists now); one retry vs repeated compaction.
• Prompt caching mechanics: what invalidates a prefix, why ordering in the middleware stack is load-bearing, and how to read cache-read vs cache-write tokens.
• goclaw comparison: loop_compact.go summarizes the first ~70% and keeps ~30%/KeepLastMessages with a hardcoded “MUST PRESERVE” prompt, mutating the stored session blob; TokenCounter interface with a tiktoken implementation plus a pure-Go fallback.
• deepagents comparison: create_summarization_middleware with profile-driven 0.85 trigger / 0.10 keep, pre-summarization tool-arg truncation, eviction appended to /conversation_history/{session_id}.md, retry on ContextOverflowError, compaction recorded in a private _summarization_event and applied inside wrap_model_call so state["messages"] stays replayable, plus the model-callable compact_conversation tool.
• Why hango copies deepagents here almost exactly (non-mutating is the correct call for a server that must replay runs) and diverges from goclaw’s mutating compaction.
• How to debug this layer: filter Tempo for spans with gen_ai.conversation.compacted=true to see which turns compacted and the before/after token counts; hango.context.input_tokens and hango.context.compactions by reason show whether a session is thrashing; the summarizer’s own chat span (with hango.call.purpose=summarize) exposes summarizer cost; hango.cache.read_ratio collapsing to zero means a cache breakpoint was invalidated by a later middleware; /conversation_history/{session_id}.md is the ground truth of what was evicted.
• Tool-batch integrity as a hard invariant of any context surgery, and why a validator is cheaper than debugging a permanently 400-ing session.
• Exercises: drive a session past the trigger and read the compaction event; construct a history whose keep boundary lands mid tool batch and watch the snap-back; add a middleware after the cache breakpoint and watch the cache-read ratio drop.
• Time budget: 11h implementation + 2h learning doc.

Success Criteria

• go build ./... && go vet ./... && go test ./... pass.
• Summarization test: a session over the trigger produces a compacted outgoing request while MessageStore.History still returns the full untouched transcript (integration test).
• The evicted transcript exists at /conversation_history/{session_id}.md and its path appears in the summary text (test).
• Tool-arg truncation alone keeps a borderline session under the trigger without summarizing (test).
• Overflow triggers compaction and one retry; a second overflow degrades to mechanical truncation and succeeds; only a third failure fails the run (test).
• A compaction whose raw keep boundary falls inside a tool batch snaps back, and the post-compaction validator confirms every tool_result has its tool_use (test over the adversarial fixture).
• Cache-control breakpoints land at the expected indices of the assembled system blocks and on the summary message; with Capabilities.PromptCaching == false no breakpoint is set.
• EstimateRequest accounts for system blocks and tool definitions (test shows the estimate rising when only the tool list grows); with a TokenCounter-capable provider the trigger uses the counted value and falls back to the heuristic when the count errors.
• Session counters are written exactly once per turn (test counts the UPDATE sessions statements across a two-turn run with one retried model call).
• compact_conversation forces compaction on the next call (test).
• In Grafana, a compacted turn shows the gen_ai.conversation.compacted span event with before/after tokens, and hango.context.compactions increments with the right reason; the summarizer call appears as its own chat span.

Risk Assessment

Rollback

goose down one step drops the session counters. Removing the Summarization, truncation and PromptCache slots from DefaultStack restores phase 9 behaviour; because compaction never mutated stored history, no data repair is needed.