Subagents

Overview

Add the task(description, subagent_type) tool: a middleware that spawns a child agent loop with its own isolated message history, its own middleware stack, and the same filesystem backend. Subagent specs live in the parent agent’s config, with a general-purpose default auto-inserted. Results cross back as the last non-empty assistant text. Recursion is refused at call time with a message rather than by hiding the tool, and a depth cap applies. Subagent runs execute on the subagent scheduler lane so they cannot starve top-level runs.

Requirements

Functional

• One tool, task(description, subagent_type), whose description embeds the available subagent list so discovery happens through the schema, not the system prompt.
• Subagent specs come from agents.config->'subagents': name, description, system_prompt, tools (nil inherits parent), model (empty inherits), interrupt_on (phase 15; nil inherits).
• A general-purpose subagent is auto-inserted unless the agent defines one with that name.
• The child loop starts with an empty message history containing only the description as the first user message; the parent’s transcript is never inherited.
• The result returned to the parent is the last non-empty assistant text block; if none exists, an explicit error result naming the problem.
• Child file writes are visible to the parent because both share the {tenant, agent} backend namespace.
• Depth cap HANGO_SUBAGENT_MAX_DEPTH (default 2): at the cap, task returns a refusal string explaining the limit; the tool stays visible.
• Child execution is charged to the same run: it runs synchronously inside the parent's tool call, and the `subagent` lane is acquired **non-blockingly**. At depth > 0 the gate is `TryAcquire`: if no slot is free, `task` returns a non-error tool result saying the delegation capacity is busy and to retry later or do the work directly. A depth-0 caller may wait, bounded by the run deadline. A blocking acquire at depth > 0 would self-deadlock — the parent holds a slot while waiting for one — so re-entrant blocking is forbidden by construction, not by convention.
• Tool calls within one batch execute **sequentially** in v1, including multiple `task` calls. Parallel tool execution is not a v1 feature, so there is no "parallel task calls bounded by the lane" behaviour to test; the lane bounds concurrency across runs, not within a batch.
• Child events are emitted to the parent run’s event stream with a depth and subagent label so an SSE client sees subagent progress.
• Todos, memory state and skills metadata are not propagated back to the parent state.

Non-functional

• Child loops reuse the same agent.Agent, chain builder and store; no duplicated loop implementation. The child is invoked through the phase-5 Agent.Run(ctx, sessionID, input, sink, RunOptions{Depth: d+1, ParentSink: sink}) — the signature does not change in this phase.
• `internal/agent` still imports nothing from `internal/run`. The lane is injected as `agent.Gate func(ctx context.Context) (release func(), error)` (a `TryAcquire` closure) built by `internal/app` from the scheduler, and the child's events go through the `agent.EventSink` the parent already holds.
• A cancelled parent run cancels its children promptly (shared context).

Architecture

// internal/agent/subagent.go
type Subagent struct {
	Name        string   `json:"name"`
	Description string   `json:"description"`
	SystemPrompt string  `json:"system_prompt"`
	Tools       []string `json:"tools,omitempty"`  // nil = inherit parent
	Model       string   `json:"model,omitempty"`  // "" = inherit
	InterruptOn map[string]InterruptConfig `json:"interrupt_on,omitempty"`
}

// Red team: F15 — InterruptConfig is defined here, in the phase that first references it.
// Phase 15 extends it with the When clause and consumes it in HITLMiddleware.
type InterruptConfig struct {
	AllowedDecisions []string `json:"allowed_decisions,omitempty"` // default: all four
}

// internal/agent/mwsubagent.go
// Red team: F5 — Gate is a plain func injected by the composition root, so internal/agent
// has no import of internal/run. Red team: F4 — it never blocks at depth > 0.
type Gate func(ctx context.Context) (release func(), error) // returns ErrNoCapacity when full

type SubagentMiddleware struct {
	Specs    []Subagent
	MaxDepth int
	Gate     Gate
	Build    ChildBuilder // constructs a child Agent + chain from a spec
} // ToolProvider

const refusalNoCapacity = "delegation capacity is busy right now; retry this task later or complete it directly"

type ChildBuilder func(ctx context.Context, parent *Agent, spec Subagent, depth int) (*Agent, error)

const refusalAtDepth = "subagent depth limit reached; complete this work directly instead of delegating"

-- migrations/00007_subagent_calls.sql
create table subagent_calls (
  id uuid primary key,
  tenant_id uuid not null references tenants(id) on delete cascade,
  run_id uuid not null references runs(id) on delete cascade,
  parent_tool_call_id text not null,
  subagent_name text not null,
  depth int not null,
  description text not null,
  state text not null check (state in ('running','succeeded','failed','cancelled')),
  result text not null default '',
  iterations int not null default 0,
  usage jsonb,
  trace_id text not null default '',
  started_at timestamptz not null default now(),
  finished_at timestamptz
);
create index subagent_calls_tenant_run_idx on subagent_calls (tenant_id, run_id, started_at);

Instrumentation

• Nested invoke_agent {subagent name} span per child, parented by the execute_tool task span, carrying gen_ai.operation.name=invoke_agent, gen_ai.agent.name=<subagent>, hango.subagent.depth, hango.subagent.parent_tool_call_id, and langfuse.observation.type=agent so Langfuse renders a nested agent observation inside the parent trace.
• The child’s chat and execute_tool spans hang off the child invoke_agent span, so the Langfuse trace tree mirrors the delegation tree exactly.
• Span events on the parent: subagent.spawned, subagent.refused_depth, subagent.empty_result.
• Metrics: gen_ai.invoke_agent.duration and ...inference_calls gain a hango.subagent.depth attribute; hango.subagent.calls counter (attributes subagent_name, state); hango.subagent.lane_wait histogram.
• Log fields: child logs carry hango.subagent.name and hango.subagent.depth in addition to the parent run fields, so Loki filtered by hango.run.id shows parent and child interleaved but distinguishable.

Related Code Files

Create

• internal/agent/subagent.go, internal/agent/mwsubagent.go, internal/agent/mwsubagent_test.go.
• internal/agent/childbuilder.go — builds the child stack (Filesystem → offload → Summarization slot → RepairToolCalls → RepeatCall → allowlist) without Subagent unless depth allows.
• internal/run/lane.go — the lane semaphore plus TryAcquire, exposed to internal/app as an agent.Gate closure; internal/agent never imports it.
• internal/store/pg/subagent_call.go, internal/store/pg/subagent_call_test.go.
• migrations/00007_subagent_calls.sql.
• docs/learning/09-subagents.md.

Modify

• internal/agent/mwstack.go — fill the Subagent slot (after Filesystem, before Summarization); pass the depth so the child stack omits Subagent at the cap.
• internal/agent/loop.go — populate RunOptions.Depth and wrap the sink with the depth/subagent decorator; the signature is unchanged from phase 5.
• internal/run/scheduler.go — register the subagent lane and expose Acquire/TryAcquire; internal/app adapts them into an agent.Gate.
• internal/server/handlers_runs.go — GET /v1/runs/{id} includes the subagent call summary.
• internal/config/config.go — HANGO_SUBAGENT_MAX_DEPTH, HANGO_LANE_SUBAGENT_CONCURRENCY.
• internal/store/pg/tenancy_test.go — add subagent_calls.
• docs/system-architecture.md, docs/learning/README.md.

Implementation Steps

1. Write internal/agent/subagent.go with the spec type and ParseSubagents(cfg json.RawMessage) ([]Subagent, error), validating names (unique, non-empty, no whitespace) and injecting general-purpose when absent.
2. Write internal/run/lane.go: a buffered-channel semaphore with Acquire(ctx) (blocking, used only at depth 0) and TryAcquire() (non-blocking, used at depth > 0, returning ErrNoCapacity), recording hango.subagent.lane_wait. internal/app wraps the right one per depth into the agent.Gate closure.
3. Write internal/agent/childbuilder.go: given the parent Agent, a spec and a depth, resolve model and tool allowlist (inherit when unset), build the child stack via DefaultStack with Depth: depth+1 and the Subagent slot omitted at the cap, and return a child Agent sharing the store, provider, backend and emitter.
4. Write internal/agent/mwsubagent.go: a ToolProvider exposing task. Its description enumerates available subagents with their descriptions plus the policy text (when to delegate, that the child sees only the description, that the result is the child’s final message). Execute validates subagent_type, checks the depth (returning refusalAtDepth as a non-error result so the model can recover), acquires a lane slot through Gate (returning refusalNoCapacity as a non-error result when the gate reports no capacity, never blocking at depth > 0) , persists a subagent_calls row, runs the child loop with the description as the only user message, extracts the last non-empty assistant text, persists the terminal state, and returns it.
5. Child message persistence: child messages are written to a child session (sessions.key = parent session key + ":sub:" + subagent_call_id) so the transcript is inspectable without polluting the parent session. The parent’s history gains only the tool result.
6. Wire event propagation: wrap the parent’s EventSink so child events carry depth and subagent_name, and emit subagent.started/subagent.finished run events.
7. Add the nested span and metrics per the Instrumentation section.
8. Insert into DefaultStack and update the pinned order test.
9. Tests: general-purpose auto-insertion; child sees only the description (assert the child’s first message list); parent receives the last non-empty text; empty-result path returns a named error; depth cap returns the refusal string and the tool remains listed; a full lane at depth > 0 returns refusalNoCapacity immediately and the parent run completes (the self-deadlock regression test) ; child file write is visible to the parent via read_file, and a concurrent parent/child write to the same path produces a version conflict result rather than a lost update ; parent cancellation cancels the child (assert both terminal states); subagent_calls cross-tenant and intra-tenant cases.
10. Update docs and write the learning doc.

Learning Doc Outline

docs/learning/09-subagents.md must contain:

• Concept: delegation as context isolation, not parallelism — the point is that the child’s transcript never enters the parent’s window.
• Design choices: one task tool with the roster in its description, specs as agent config data, shared backend and separate history, last-non-empty-text result extraction, refusal-at-call-time for recursion, depth cap, a dedicated scheduler lane, child sessions persisted for inspection.
• Trade-offs: synchronous in-tool execution vs enqueuing a separate run (chose synchronous: the parent’s tool call needs the result, and the lane still bounds concurrency); lossy text result vs structured result; sharing the filesystem vs isolating it.
• goclaw comparison: subagent spawn lives in tools plus a subagent_tasks table, with internal/orchestration providing a generic BatchQueue[T] and ChildResult, plus teams/delegation/cron features hango skips.
• deepagents comparison: SubAgentMiddleware adds one task tool with {available_agents} embedded in the description; each declarative SubAgent compiles its own stack (Filesystem → Summarization → PatchToolCalls); result is the last non-empty AI text; _EXCLUDED_STATE_KEYS blocks messages/todos/structured_response/skills_metadata/private from propagating; _FORK_RECURSION_REFUSAL refuses at call time; recursion_limit is effectively unbounded; experimental mode="fork" inherits the parent transcript.
• Why hango diverges: a real depth cap, a bounded lane, persisted child sessions and subagent_calls rows for auditability, no fork mode.
• The known lossiness pitfall: a child that does all its work in tool calls and ends with an empty message, and the walk-back workaround.
• How to debug this layer: the Langfuse trace shows the nested agent observation tree, so “which subagent burned the tokens” is one click; hango.subagent.calls by name and state shows failure concentration; subagent.empty_result span events identify the lossy-result pitfall in production; Loki filtered by hango.run.id plus hango.subagent.depth separates parent from child narration.
• Why a re-entrant blocking semaphore is a deadlock and a non-blocking gate with a "retry later" tool result is not, and why the gate is an injected function rather than an imported type.
• Exercises: define a second subagent in agent config and delegate to it; nest to the depth cap and read the refusal; fill the lane and read the capacity refusal; make a child end with an empty message and observe the error path.
• Time budget: 10h implementation + 2h learning doc.

Success Criteria

• go build ./... && go vet ./... && go test ./... pass.
• general-purpose exists without configuration; a user-defined subagent with that name overrides it (test).
• The child’s initial message list contains exactly the description (test).
• A child file write is readable by the parent through read_file (integration test).
• At the depth cap, task returns the refusal string as a non-error result and the tool is still advertised (test).
• With the subagent lane full, a task call at depth > 0 returns the capacity refusal immediately (no blocking) and the parent run still reaches succeeded; a depth-2 delegation chain never deadlocks.
• Parent cancellation drives the child to cancelled within the lane’s cancellation path (test).
• subagent_calls rows record name, depth, state, iterations, usage and trace id; the table is in the cross-tenant suite.
• In Langfuse, a delegating run renders as a trace with a nested agent observation containing the child’s generations and tool calls; in Tempo the same nesting is visible under the execute_tool task span.
• hango.subagent.calls and hango.subagent.lane_wait series exist in Prometheus.

Risk Assessment

Rollback

goose down one step drops subagent_calls. Removing the Subagent slot from DefaultStack removes the task tool; child sessions already written remain readable through the session API.