Filesystem backend and large-result offload

Overview

Give agents a persistent virtual filesystem behind a Backend interface, implemented over Postgres and namespaced by {tenant_id, agent_id}. Expose it as a middleware providing ls, read_file, write_file, edit_file, glob, grep with strict path validation and line-numbered reads. Then use it for the highest-value context win available: any tool result above a token limit is written to /large_tool_results/{tool_call_id} and replaced with a head/tail preview plus instructions to read the file. This backend is the prerequisite for subagent sharing, history offload, memory and skills.

Requirements

Functional

• Backend interface: Ls, Read (offset/limit), Write, Edit (old/new, replaceAll), Glob, Grep, Delete, Stat.
• pgstore backend: files as rows keyed (tenant_id, agent_id, path) with content, size, mtime; namespace derived from context, never from tool arguments.
• Path validation: absolute paths only, normalized, no .. traversal, no NUL bytes, max depth and max path length, max file size (HANGO_FS_MAX_FILE_BYTES, default 1MiB).
• read_file returns line-numbered content, clips lines at 5000 characters, supports offset/limit; grep caps matches at 1000; glob supports ** and *.
• edit_file replaces an exact string; a non-unique match without replace_all is an error tool result naming the match count.
• Mutation of the same path twice within one tool-call batch is rejected as an error tool result. This is a within-batch guard only: it says nothing about a parent and a child agent writing concurrently (phase 9), which is why the version column below exists.
• `agent_files` carries `version int not null default 1`. Every `Write`/`Edit` performs `UPDATE ... SET content=..., version=version+1 WHERE ... AND version=$expected`; a zero-row result is surfaced to the model as an error tool result ("the file changed since you read it; re-read and retry"), never as a silent overwrite. This is the only mechanism that makes shared parent/child filesystem access safe.
• Large-result offload: any tool result whose estimated tokens exceed HANGO_TOOL_RESULT_TOKEN_LIMIT (default 20000) is written to /large_tool_results/{session_id}/{tool_call_id} and the model sees a 5-line head, a 5-line tail, the total size and a read_file instruction.
• **Per-session namespacing of machine-written paths.** Offloaded results live under `/large_tool_results/{session_id}/…` and the phase-10 history offload under `/conversation_history/{session_id}.md`. `glob` and `grep` exclude those two prefixes for every session other than the current one, and the files API applies the same exclusion, so one session's transcript and tool payloads are not readable by every other agent run in the tenant. Reads of an explicit path outside the current session under those prefixes return a not-found error result.
• `/memories/` and `/skills/` are reserved prefixes, enforced in `internal/fs/path.go` — the same audited function that rejects traversal. There is no routing layer: they are ordinary paths in the same backend with a naming rule and, from phase 15, an interrupt rule.
• Writes to `/memories/` and `/skills/` are privileged, because both are re-read into future prompts and are therefore a persistent prompt-injection surface. Until phase 15 exists they are writable only through the files API by an `owner`/`admin`, and the agent tools reject them with an explanatory error result. From phase 15 they default to a HITL interrupt rule that the agent's config may relax but not silently remove.
• GET /v1/agents/{id}/files and GET /v1/agents/{id}/files/* expose the filesystem read-only over the API for inspection; PUT /v1/agents/{id}/files/* writes, restricted to owner/admin and to the reserved prefixes.
• The files API validates that the agent belongs to the caller's tenant **and** that the caller may see that agent (creator-scoped for `member`), before any path is resolved. The agent id used to build the namespace comes from the request-scoped agent lookup, never from a path segment the client controls.

Non-functional

• Every backend call is tenant-scoped by construction: the namespace factory reads the scope from context and fails closed.
• Cross-tenant and cross-agent isolation tests for the agent_files table.
• Content is stored as text with a byte-size check; binary uploads are out of scope.
• grep runs in Postgres (position/~ with a bounded scan), not by loading every file into Go.

Architecture

// internal/fs/backend.go
type Backend interface {
	Ls(ctx context.Context, path string) ([]DirEntry, error)
	Read(ctx context.Context, path string, offset, limit int) (File, error)
	Write(ctx context.Context, path, content string) (WriteResult, error)
	Edit(ctx context.Context, path, old, new string, replaceAll bool) (EditResult, error)
	Glob(ctx context.Context, pattern, root string) ([]string, error)
	Grep(ctx context.Context, pattern string, opt GrepOptions) ([]Match, error)
	Delete(ctx context.Context, path string) error
	Stat(ctx context.Context, path string) (DirEntry, error)
}

type GrepOptions struct {
	Path     string
	Include  string // glob filter
	MaxCount int    // default 1000
}

// internal/fs/namespace.go
type Namespace []string // {tenantID, agentID, "files"}

type NamespaceFactory func(ctx context.Context) (Namespace, error) // fail-closed

// Red team: F12 — internal/fs owns the agent-id context key; no other package defines or
// re-defines it, and the worker sets it through this constructor.
type agentKey struct{}

func WithAgentID(ctx context.Context, id uuid.UUID) context.Context
func AgentIDFromContext(ctx context.Context) (uuid.UUID, error) // fail-closed

// internal/fs/pgbackend.go
func NewPG(st store.FileStore, ns NamespaceFactory) Backend // Red team: F15 — was FileStoreFactory

// internal/fs/path.go
func Validate(p string) (string, error) // absolute, cleaned, depth/length capped, no traversal
// Red team: F14, F12 — reserved prefixes and session-scoped machine paths live here, so
// every caller inherits the same rules from one audited function.
const (
	PrefixMemories    = "/memories/"
	PrefixSkills      = "/skills/"
	PrefixLargeResult = "/large_tool_results/"
	PrefixHistory     = "/conversation_history/"
)

func IsReserved(p string) bool
func SessionScoped(p string) (sessionID string, ok bool) // for the glob/grep/API exclusion

// internal/agent/mwfilesystem.go
type FilesystemMiddleware struct {
	Backend fs.Backend
	Limits  fs.Limits
} // ToolProvider (6 tools) + PromptContributor (path conventions only) + ToolWrapper (offload)

-- migrations/00006_agent_files.sql
create table agent_files (
  id uuid primary key,
  tenant_id uuid not null references tenants(id) on delete cascade,
  agent_id uuid not null references agents(id) on delete cascade,
  path text not null,
  content text not null,
  size_bytes int not null,
  version int not null default 1, -- Red team: F12 — optimistic concurrency for shared access
  created_at timestamptz not null default now(),
  updated_at timestamptz not null default now(),
  unique (tenant_id, agent_id, path)
);
create index agent_files_tenant_agent_path_idx on agent_files (tenant_id, agent_id, path);
create index agent_files_tenant_agent_updated_idx on agent_files (tenant_id, agent_id, updated_at desc);

Instrumentation

• Spans: fs.{op} INTERNAL spans (fs.read, fs.write, fs.edit, fs.glob, fs.grep, fs.ls) with hango.fs.path, hango.fs.bytes, hango.fs.matches, hango.agent.id; paths are recorded, content is not.
• The offload wrapper records a span event tool.result_offloaded on the execute_tool span with gen_ai.tool.name, hango.fs.path, hango.tokens.before, hango.tokens.after.
• Metrics: hango.fs.operations counter (attributes op, error.type), hango.fs.bytes_written counter, hango.fs.file_count gauge per tenant, hango.tool.result_offloads counter, hango.tool.result_tokens histogram.
• Log fields: every write and edit logs at info with hango.agent.id, path and byte delta, so “who changed this file” is answerable from Loki alone.

Related Code Files

Create

• internal/fs/backend.go, internal/fs/namespace.go, internal/fs/path.go, internal/fs/path_test.go, internal/fs/pgbackend.go, internal/fs/pgbackend_test.go (integration), internal/fs/trace.go.
• internal/store/pg/file.go — the SQL behind the backend (scoped).
• internal/agent/mwfilesystem.go, internal/agent/mwfilesystem_test.go.
• internal/agent/mwoffload.go, internal/agent/mwoffload_test.go — the ToolWrapper performing large-result offload.
• internal/agent/tokens.go — the Estimator interface with a heuristic implementation. Its primary method is EstimateRequest(req *provider.Request) int, which counts system blocks, message blocks, tool definitions and any injected fragments — not just a string — because every later budget decision is about the whole request. Estimate(string) int remains as a helper for the offload check.
• internal/server/handlers_files.go.
• migrations/00006_agent_files.sql.
• docs/learning/08-virtual-filesystem-and-offload.md.

Modify

• internal/agent/mwstack.go — fill the Filesystem slot (before Subagent and Summarization) and place the offload wrapper immediately inside it so summarization later sees already-offloaded results.
• internal/store/store.go, internal/store/pg/scoped.go, internal/store/pg/tenancy_test.go (add agent_files).
• internal/config/config.go — HANGO_FS_MAX_FILE_BYTES, HANGO_FS_MAX_PATH_DEPTH, HANGO_TOOL_RESULT_TOKEN_LIMIT.
• internal/run/worker.go — set the agent id into the context through fs.WithAgentID.
• internal/app/app.go — build the backend and pass it to the stack factory.
• docs/system-architecture.md, docs/learning/README.md.

Implementation Steps

1. Write internal/fs/path.go first, with an exhaustive table test: /a/b, a/b (reject), /a/../../etc (reject), /a/./b (normalize), embedded NUL (reject), depth and length caps, trailing slashes, plus the reserved-prefix and session-scoped-path helpers.
2. Write migrations/00006_agent_files.sql and the scoped FileStore with Get, Upsert (version-checked), ListPrefix, Delete, Grep (WHERE content ~ $pattern with LIMIT), Glob (translate the glob to a LIKE/regex prefilter then match precisely in Go on the returned paths). Every listing query excludes other sessions’ /large_tool_results/ and /conversation_history/ paths.
3. Write internal/fs/namespace.go: the factory reads store.ScopeFromContext plus the agent id from fs.AgentIDFromContext (the key is owned by this package and set by the worker); it returns an error when either is missing. Validate every component is a non-empty UUID string.
4. Write internal/fs/pgbackend.go implementing Backend over the file store. Read applies offset/limit by line, numbers lines %6d\t, clips at 5000 characters per line and reports truncation, and returns the row’s version. Edit counts occurrences and errors when ambiguous. Write/Edit pass the expected version and surface a conflict as an error result. Write enforces the byte cap.
5. Write internal/agent/tokens.go: Estimator with EstimateRequest(*provider.Request) int and Estimate(string) int, plus a HeuristicEstimator (bytes/4 with fixed per-message and per-tool-definition overheads). Phase 10 adds the TokenCounter capability behind the same interface.
6. Write internal/agent/mwfilesystem.go: six tools, each with its policy text in its own description (no system-prompt prose), delegating to the backend and mapping errors to tool.Errorf. The prompt contribution is minimal: one block naming the root paths in use and the reserved prefixes. Writes to /memories/ and /skills/ are refused with an explanatory error result until phase 15 installs the interrupt rule.
7. Add the within-batch mutation guard: the middleware tracks paths mutated within the current tool-call batch in its typed private state and rejects a second mutation of the same path in that batch; the loop clears the set at batch end. Document explicitly that this guard does not cover parent/child concurrency — the version column does.
8. Write internal/agent/mwoffload.go as a ToolWrapper: call next, estimate tokens on Result.ForLLM, and when over the limit write the full text to /large_tool_results/{session_id}/{tool_call_id} and replace ForLLM with head/tail plus size and read instructions. ForUser keeps the full text reference. Record the span event and metrics (the path is recorded on the span, not in tool.Result, which no longer has a Meta field).
9. Insert both into DefaultStack in the Filesystem slot and update mwstack_test.go’s pinned order.
10. Write internal/server/handlers_files.go: read (ls, read) for any caller who may see the agent, plus PUT restricted to owner/admin and to the reserved prefixes. Both resolve the agent through the scoped store first (tenant + creator visibility), then build the namespace from the resolved agent id. Other sessions’ machine-written paths are excluded from listings. The mutating routes call SetAuditAction with file.write or file.delete, recording the agent id and the path in detail and never the file content.
11. Tests: path validation table including reserved prefixes and session-scoped paths; backend integration tests per operation including grep limits and glob patterns; cross-tenant and cross-agent isolation (agent A cannot read agent B’s file in the same tenant); session isolation — session B’s glob, grep and files-API listing do not see session A’s /large_tool_results/ or /conversation_history/ entries, and an explicit read of them fails; version conflict — two writers with the same expected version produce one success and one conflict error result; agent-tool writes to /memories/ are refused while an owner write through the API succeeds; offload test asserting the preview shape and that the file is readable back through read_file; a loop test where a tool returns 100k characters and the next model request contains the preview, not the payload.
12. Update docs and write the learning doc.

Learning Doc Outline

docs/learning/08-virtual-filesystem-and-offload.md must contain:

• Concept: why an agent harness needs a filesystem abstraction at all, and why context bloat is the central problem of agent engineering.
• Design choices: Backend as the seam, Postgres-backed store namespaced by {tenant, agent}, path validation as a separate audited function, line-numbered reads, grep in SQL, offload as a ToolWrapper rather than loop code.
• Trade-offs: state backend (ephemeral) vs store backend (persistent) vs real disk; per-tenant vs per-agent vs per-user namespace and why hango chose {tenant, agent}; token estimation heuristic vs tiktoken.
• The three-mechanism context strategy and why offload must exist before summarization.
• goclaw comparison: workspace tools with a path jail on real disk; no Backend seam, so there is no place to put memory or skills later.
• deepagents comparison: BackendProtocol with StateBackend/FilesystemBackend/StoreBackend/CompositeBackend, FilesystemMiddleware enforcing FilesystemPermission at the tool layer, _message_eviction.py offloading at 20000 tokens to {artifacts_root}/large_tool_results/{tool_call_id} with a 5-line head/tail preview, MAX_LINE_LENGTH=5000, grep_max_count=1000, and the parallel-file-mutation rejection.
• Why hango diverges: persistent store backend as the default (state backends vanish with the thread), namespace fixed by tenancy rather than caller-supplied, permissions deferred to the phase-7 allowlist plus phase 15 interrupts.
• How to debug this layer: hango.fs.operations by op and error.type shows a misbehaving tool; fs.* spans under a tool span show exactly which path was read or written and how big it was; the tool.result_offloaded span event explains a sudden drop in prompt size; GET /v1/agents/{id}/files lets you inspect what the agent actually wrote.
• Namespacing as a security property: why `{tenant, agent}` is not fine-grained enough for machine-written paths, and why optimistic versioning beats a within-batch guard for shared access.
• Exercises: write a file through the agent and read it back over the API; force an offload with a large http_fetch; try a traversal path and read the rejection; make two writers race and read the conflict result.
• Time budget: 13h implementation + 2h learning doc.

Success Criteria

• go build ./... && go vet ./... && go test ./... pass.
• Path validation table test covers traversal, relative paths, NUL bytes, depth and length caps.
• Backend integration tests pass for all eight operations against real Postgres.
• Agent A cannot read agent B’s file within the same tenant, and tenant B cannot read tenant A’s file at all (two tests); agent_files is in the cross-tenant suite.
• A 100k-character tool result produces a preview under 1000 characters in the next model request and a readable /large_tool_results/{session_id}/{tool_call_id} file (test).
• edit_file with an ambiguous match returns an error result naming the count, and the file is unchanged.
• Two writes to the same path in one tool batch produce an error result for the second.
• A concurrent write from two contexts against the same expected version produces exactly one success and one conflict error result; the file content matches the winner.
• Session B cannot list, glob, grep or read session A’s /large_tool_results/ or /conversation_history/ paths, through the tools or the files API.
• An agent tool write to /memories/AGENTS.md is refused with an explanatory error result; the same write through the files API as owner succeeds and as member returns 403. Both outcomes write an audit_events row with action file.write and outcome ok or denied.
• The files API returns 404 for an agent the caller may not see, before any path handling runs.
• In Grafana, fs.* spans appear under execute_tool spans, tool.result_offloaded events are visible, and hango.fs.operations/hango.tool.result_offloads series exist.

Risk Assessment

Rollback

goose down one step drops agent_files. Removing the Filesystem and offload slots from DefaultStack returns the loop to phase 7 behaviour; no other package depends on internal/fs yet.