Persistence and Tenancy

Phase 2 gives hango a memory and a front door. It stores tenants, users and API keys in Postgres, every request has to present a key, and every query the request makes is locked to the caller’s tenant. It also starts an audit trail: one row for every change anyone makes, with the trace id of the request that made it.

What you will learn

• What multi-tenancy is, and why hango puts the tenant on the store handle instead of passing it to every method.
• How an API key is made, stored and checked without ever storing the key itself.
• Why authorization has two axes, and why they live in two different places.
• How the audit trail, the trace and the log answer three different questions, and how one trace_id joins them.
• How to test isolation against a real Postgres, and the rule every later table follows.

Words used in this lesson

Why tenancy comes this early

Every later phase stores something: agents, sessions, runs, files, memories. If the tenant boundary arrives after those tables exist, every one of them has to be revisited, and one forgotten WHERE tenant_id = ... is a data leak between families. Building the boundary first means each new table is born inside it, and a test fails the day someone forgets.

The smallest version

Here is the whole idea in a few lines. The tenant is not an argument to the query methods. It is baked into the handle when the handle is made, and making the handle fails when there is no tenant.

type Scope struct{ TenantID uuid.UUID }

type ctxKey struct{}

func WithScope(ctx context.Context, s Scope) context.Context {
	return context.WithValue(ctx, ctxKey{}, s)
}

type Users struct {
	db     *pgxpool.Pool
	tenant uuid.UUID
}

// For is the only way to get a Users handle, and it needs a tenant.
func For(ctx context.Context, db *pgxpool.Pool) (*Users, error) {
	s, ok := ctx.Value(ctxKey{}).(Scope)
	if !ok || s.TenantID == uuid.Nil() {
		return nil, errors.New("no tenant") // fail closed
	}
	return &Users{db: db, tenant: s.TenantID}, nil
}

// Get cannot forget the tenant: there is no way to call it without one.
func (u *Users) Get(ctx context.Context, id uuid.UUID) (string, error) {
	var email string
	err := u.db.QueryRow(ctx,
		`select email from users where tenant_id = $1 and id = $2`, u.tenant, id).Scan(&email)
	return email, err
}

A caller cannot pass the wrong tenant, because it never passes one. It cannot forget the tenant, because without one it never gets a handle. Everything else in phase 2 grows from this.

How hango is wired

Startup

internal/app now does more before the server starts:

config.Load -> telemetry.Init -> server.New -> pg.Open -> migrate -> bootstrap -> MountAPI -> serve

pg.Open builds a pgx connection pool with tracing installed and pings the database. migrate applies any SQL files in migrations/ that have not run yet. bootstrap creates the platform operator on the very first start. MountAPI registers the database readiness check and the API routes. When HANGO_DATABASE_URL is not set, hango skips all four and serves only /healthz and /readyz, which keeps phase 1’s “it runs anywhere” property.

The database URL holds a password, so it is never logged and never put in an error message. The startup log says only database_enabled=true.

Migrations

The schema lives in migrations/00001_identity.sql, embedded in the binary with embed.FS and applied by goose. Because the files are inside the binary, the image needs no extra files and the schema always matches the code. hango hands goose a Postgres advisory lock (goose.WithSessionLocker), so two replicas starting at the same time do not both run the same file. goose does not lock unless you ask: an early version of this phase claimed the lock in a comment without passing the locker, and code review caught it. Three hango migrate runs started together on an empty database now give one applied=1 and two applied=0. hango migrate applies migrations and exits, for when you want to migrate before a deploy rather than at startup.

Four tables:

Every tenant-owned table has tenant_id NOT NULL, and every index used for listing starts with tenant_id. The unique api_keys.lookup index does not, because it must work before the tenant is known. The first rule means a row without an owner cannot exist. The second means “this tenant’s rows” is always a fast index range, never a full scan, so one big tenant does not slow another down.

The scoped store

internal/store holds only interfaces and types. internal/store/pg is the Postgres implementation, and it is the only package allowed to import pgx. A test (storeboundary_test.go) fails if any other package does, so every tenant-owned query in the codebase is in one directory, where the isolation test covers it.

The root Store has three doors:

• Tenants() manages the tenants table, which belongs to no tenant. Only the platform uses it.
• Auth().ResolveKey(lookup) finds a key before the tenant is known, because that is how the tenant becomes known.
• For(ctx) returns a Scoped handle bound to the scope on the context. Users, keys and audit rows are reachable only through it.

ScopeFromContext returns ErrNoTenant when the scope is missing, the tenant id is nil or the role is not one of the three. A hand-built scope that forgot a field gets nothing.

Inside a tenant, a member sees less than an owner. The store narrows List and Revoke to the member’s own rows. Get stays tenant-wide on purpose: the authorization layer needs to load the row to tell a member “forbidden” instead of “not found”.

API keys

A key looks like hango_family_7Q2... (44 base58 characters after the tenant tag). It is made from 32 random bytes, so it has 256 bits of entropy. hango stores two things:

• lookup: the first 8 characters of the random part, with a unique index. Resolving a key is one index lookup, however many keys exist.
• key_hash: the SHA-256 of the whole key, compared in constant time.

Why SHA-256 and not bcrypt? bcrypt is slow on purpose, to protect human passwords that are short and guessable. A 256-bit random key cannot be guessed, so the slowness buys nothing and would cost every request tens of milliseconds. The plaintext key appears once, in the response that created it, and nowhere else: not in the database, not in logs, not in spans.

The authentication middleware

For every API request, authenticator runs these steps in order:

1. The pre-auth limiter. A token bucket per client IP. Only failures spend tokens. When the bucket is empty the request gets 429 before any database query, so a flood of bad keys cannot load the database, however many different keys it tries. A key that authenticated in the last ten minutes passes an empty bucket, so a bad client behind the same address (a home router, an ingress) cannot lock out a phone that is working. The limiter holds at most 65,536 buckets, so a flood from many addresses cannot grow its memory either.
2. Look up the key by its lookup.
3. Compare hashes, then check revoked, expired, and that the user and tenant are active.
4. On success, put the Principal and the store scope on the context, and the tenant and user ids on the span, the logs and every child span.
5. Write last_used_at in the background, at most once per key per minute.

Every failure returns the same 401 missing or invalid API key. The reason (unknown_key, revoked, expired, …) goes to the hango.auth.failures metric and the log, never to the client, so an attacker cannot learn which keys exist.

Two axes of authorization

“May this caller do this?” has two separate questions inside it:

• Which tenant can it reach? The store answers this. A scoped handle only sees its own tenant.
• What may it do inside that tenant? internal/server/authz.go answers this, from one table with a row per route and a column per role.

The Platform column applies to customer tenants. Inside its own platform tenant, the platform user is that tenant’s owner and gets the Owner column, so it can rotate and revoke its own keys and read its own audit trail.

A target in another tenant is always 404, never 403. A 403 would confirm that the id exists somewhere. Mixing the two axes into one check is how leaks happen: a role check that also tries to be the tenant check eventually misses a case nobody thought of. Keeping them apart means each one can be tested on its own, and a bug in one is still caught by the other.

The platform operator and the bootstrap key

Someone has to create the first tenant. That is the platform operator, the owner user of the one tenant whose kind is platform. A unique partial index allows at most one.

On the first start against an empty database, hango reads HANGO_BOOTSTRAP_ADMIN_KEY and, in one transaction, creates the platform tenant, the user [email protected] and a key row holding the hash of that key. After that the variable is ignored while the platform has a working key, so revoking or rotating the platform key through the API sticks across restarts.

If the platform has no working key at all (every key revoked or expired), hango cannot be administered. To recover, set a new HANGO_BOOTSTRAP_ADMIN_KEY and restart: at startup hango stores the new key for the platform owner. It never revives a key that was revoked, so restarting with the old value logs a warning and changes nothing.

Why a row and not a simple “if the header equals the env var” check? Because a row can be revoked, expires, has last_used_at, appears in the audit trail and goes through exactly the same code path as every other key. An env comparison would be a second, special authentication path that bypasses all of that, and revoking it would need a redeploy.

The platform’s power inside a customer tenant is kept to the minimum that onboarding needs: the tenant’s first user, as owner, and a key for an owner while the tenant has no working owner key. The second rule covers onboarding and recovery: if the owners lose or revoke every key, the platform can give one of them a new key. While an owner key works, the tenant manages itself, and the platform gets 403 if it tries to mint a key for a user the owner added. Each check is “is there one yet?” followed by a write, and two requests at once could both see “no”. So the handler runs the check and the write in one transaction that first locks the tenant row (select ... for no key update); the second request waits, then sees the first one’s row. for no key update is the weakest lock that still makes the second request wait, so it does not block inserts that only reference the tenant through a foreign key.

When the platform acts inside a customer tenant, the request span and log line carry hango.acting_tenant.id next to hango.tenant.id. hango.tenant.id stays the platform’s own tenant, the caller’s identity, and the acting attribute tells you which tenant it touched.

The audit trail

Every POST, PUT, PATCH and DELETE on the API writes one audit_events row after its handler returns. The handler says what it did with SetAuditAction(r, "api_key.revoke", "api_key", id), and calls it before its authorization check, so a refused attempt is recorded with its action too. The middleware decides the outcome from the response status: 2xx is ok, 401 or 403 is denied, anything else is error. GET requests write nothing.

Two rules keep the trail trustworthy. It is append-only: the store has no UPDATE or DELETE statement for the table, and no route changes it; tests check both. And a failed audit write never fails the user’s request: it is logged and counted on hango.audit.write_failures, which you can alert on.

How to see what the server is doing

Start the stack with a bootstrap key:

echo "HANGO_BOOTSTRAP_ADMIN_KEY=hango_platform_$(openssl rand -base64 48 | tr -dc '1-9A-HJ-NP-Za-km-z' | head -c 44)" >> .env
make compose-up
set -a; . ./.env; set +a
K=$HANGO_BOOTSTRAP_ADMIN_KEY

Create a tenant, its owner and the owner’s key:

TID=$(curl -s -H "Authorization: Bearer $K" -d '{"slug":"family","name":"Family"}' localhost:8080/v1/admin/tenants | jq -r .id)
UID_=$(curl -s -H "Authorization: Bearer $K" -d '{"email":"[email protected]","role":"owner"}' localhost:8080/v1/admin/tenants/$TID/users | jq -r .id)
OWNER=$(curl -s -H "Authorization: Bearer $K" -d '{"name":"phone"}' localhost:8080/v1/users/$UID_/api-keys | jq -r .key)
curl -i -H "Authorization: Bearer $OWNER" localhost:8080/v1/me
curl -s -H "Authorization: Bearer $OWNER" localhost:8080/v1/audit-events | jq '.events[] | {action, outcome, trace_id}'

Copy the X-Trace-Id of the /v1/me response into Grafana → Explore → Tempo. Under the GET /v1/me server span you see one CLIENT span per query and pool acquire, each with its SQL statement. Every span after authentication carries hango.tenant.id; the key lookup itself does not, because it runs before the tenant is known.

Audit, trace and log: three questions, one key

Each one alone is a partial answer. The audit row carries the trace_id of the request that produced it, and every log line carries the same id, so one id turns the three into one story: read the row, paste its trace_id into Tempo to see the queries, then filter Loki for trace_id="..." to read what the code logged.

How to debug this layer

A request returns 401

The client always sees the same message, so look at the server side. In Grafana, the hango_auth_failures_total metric has a reason label. In Loki, filter for the request’s request_id and the line authentication failed shows the reason: unknown_key (wrong key or typo), bad_hash (right prefix, wrong rest), revoked, expired or inactive. A 429 means the pre-auth limiter: the same address sent too many bad keys; by default it allows a burst of 10 failures and refills 20 per minute (HANGO_PREAUTH_BURST, HANGO_PREAUTH_RATE_PER_MINUTE).

A request is slow or fails with a database error

Open the request trace in Tempo. The pgx child spans show each statement, how long it took and its error. A long pool.acquire span means every connection was busy: raise HANGO_DB_MAX_CONNS or look for slow queries holding connections.

hango will not start

A bad HANGO_DATABASE_URL stops startup with parse database url: invalid connection string (the URL itself is never printed) or connect to database: .... A failed migration prints the goose error and the file. /readyz returns 503 with "database":"fail" when Postgres goes away after startup.

You suspect an isolation bug

Reproduce it as a case in internal/store/pg/tenancy_test.go, then confirm in Tempo that the query span for the request carries the hango.tenant.id you expect.

Testing against a real database

The isolation guarantees are about SQL, so they are tested against a real Postgres, not a fake. internal/store/pg/pgtest starts one postgres:17-alpine container with testcontainers and reuses it across packages and runs. Each test package gets its own database (hango_test_pg, hango_test_server, hango_test_app), dropped and recreated at the start, so packages running in parallel never see each other’s rows. go test -short ./... skips every database test, and setting HANGO_TEST_DB_DSN points the tests at a server you already run.

Go has no repo-wide TestMain: each package that needs the database has a one-line TestMain that calls pgtest.Main. The helper is a normal package, not a _test.go file, so it can be shared, and only test files import it. go list -deps ./cmd/hango shows no testcontainers package, so the container library never reaches the binary.

On Rancher Desktop, testcontainers needs DOCKER_HOST=unix://$HOME/.rd/docker.sock and TESTCONTAINERS_RYUK_DISABLED=true.

The isolation table

tenancy_test.go holds one entry per tenant-owned table. Each entry knows how to create a row, read it by id, list rows and change one. For every entry the test checks:

• Cross-tenant: under any role of tenant B, tenant A’s row is not listed, get-by-id is ErrNotFound, and a change is ErrNotFound.
• Cross-user: tenant A’s member does not list or change a row that belongs to A’s owner; A’s owner and admin do.

The rule for every later phase: a new tenant-owned table adds an entry here, in the same pull request that creates the table.

Trade-offs

• Scoped store vs Postgres row-level security (RLS) vs one schema or database per tenant. RLS enforces the boundary inside Postgres, which is stronger, but it needs a session variable set on every connection checkout and is easy to bypass with a superuser connection by accident. Per-tenant schemas multiply migrations by the number of tenants. A scoped store with a test table is simple, visible in code review and fast enough for families. RLS can be added later as a second wall without changing the store interface.
• Hand-written SQL vs sqlc. sqlc generates typed code from SQL, but every query here needs the tenant predicate from the handle and a few are built from filters. Hand-written SQL in one package keeps that visible. Revisit when there are dozens of queries. Phase 2.1 revisited it early, to learn sqlc while the store is small: see Moving the store to sqlc.
• Testcontainers vs SQLite in tests. SQLite would be faster, but the bugs this phase guards against (a missing predicate, a unique index, a partial index, on delete behaviour) are Postgres behaviour. A test that passes on another database proves little.
• A shared pgtest package vs copying the helper into each package. The plan expected a copy per package. A small shared package under internal/store/pg keeps one copy, and the binary check above proves it stays out of production.
• Synchronous audit writes. Writing the row after the handler adds a few milliseconds to each change. In return the row exists before the next request, which makes the trail easy to test. Changes are rare compared to reads, and reads write nothing.

How goclaw and deepagents do this

goclaw (~/ws/personal/goclaw) is multi-tenant too, but the tenant travels differently. A QueryScope (internal/store/scope.go) produces a WhereClause that each query appends by hand, and the tenant is read from the context inside each method. A default MasterTenantID fills in when no tenant is set, which fails open. User ids are free-form text with no foreign key to a users table. API keys are cached for five minutes (newAPIKeyCache(ms, 5*time.Minute)), so a revoked key keeps working for up to five minutes, and there are three separate authentication paths. Because every query adds its own filter, goclaw later needed a separate visibility filter layer to keep them consistent.

langchain deepagents has no tenancy at all. The closest idea is the StoreBackend namespace factory, which prefixes stored files with (user_identity, "filesystem").

hango differs on purpose: the scope lives on the handle so no query can skip it, a missing tenant fails closed instead of falling back to a default, hango owns its users with real foreign keys, there is one authentication path, and a key is checked against the database on every request, so revoking is immediate.

Exercises

1. Break isolation on purpose. In internal/store/pg/queries/api_keys.sql (since phase 2.1; apikey.go in phase 2), change GetAPIKey’s where tenant_id = @tenant_id and id = @id to where @tenant_id::uuid is not null and id = @id, run make generate, and run go test ./internal/store/pg/ -run TestCrossTenant -v. Watch the api_keys case fail, then restore it.
2. Follow one change across all three signals. Revoke a key with curl -X DELETE -H "Authorization: Bearer $OWNER" localhost:8080/v1/api-keys/<id>, find the api_key.revoke row in /v1/audit-events, paste its trace_id into Tempo, then find the same request in Loki.
3. Trip the limiter. Send 11 requests, each with a different made-up key, and watch the 11th return 429. Then call /v1/me with $OWNER, which authenticated a moment ago: it still works. Then check hango_auth_prelimit_rejections_total in Grafana.
4. Add a table. Sketch a notes table with tenant_id, write its migration, and add its entry to tenantOwnedTables. Which store methods did the test force you to write?