Butler

An agent with a short leash

Sole builder and operator

Personal system, running on my own server since August 2026 I am the only user. Shown as architecture only.

Before and after

Before

  1. Garrett Reads each message and document by hand , done by hand
  2. Garrett Decides what needs a task , done by hand
  3. Notes and calendar Keeps the rest in a notes app , automated

After

  1. Garrett Forwards a document or message in , done by hand
  2. Model Classifies it, holding zero tools , done by the model
  3. System Stages a proposed task or fact , automated
  4. Garrett Taps Save, Edit, or Discard , reviewed by a person
  5. System Writes the row only after that tap , automated
  • Model
  • Person reviews
  • Automated
  • By hand
Handling incoming mail and documents, before and after

The work before

Before Butler, I handled my own admin by hand: reading mail and paperwork, deciding what needed a task or a date, and keeping the rest in notes.

The parts worth automating were narrow: sorting incoming mail and documents, tracking dates and tasks, and drafting routine phrasing. The parts worth keeping manual were anything that commits me to something, or costs money, or leaves the house.

The call

The design question was never "can a model do this," it was which of these 20 tools a given task actually needs, and what a model with those tools could do wrong.

Automated

  • Bulk classification of incoming mail and scanned documents
  • Routine task tracking, date math, and anchor-based deadlines
  • Drafting brief and nudge phrasing from a pre-computed template
  • Fetching a fixed, config-defined list of RSS feeds

Kept with a person

  • Sending anything: no tool exists that can send a message
  • Writing a proposal from a document without a human tap to confirm it
  • Any financial write; the finance context is read-only and has no web access
  • Letting a document, email, or calendar entry act as an instruction

Where AI sits

Three model tiers, chosen by call site. Haiku classifies incoming mail and reads scanned documents and photos first-pass, as a one-shot call holding zero tools: its output is text a human or deterministic code checks afterward.

Sonnet runs the interactive chat, briefs, and nudge phrasing, and is the only context with write tools and web access. Opus handles the vision adjudicator's escalation path and the monthly finance review, both low-frequency and high-judgment.

ModelJobWhy
Claude HaikuClassifies incoming mail; first-pass read of scanned documents and photosHigh volume, low judgment per call, and it runs with no tools at all, so a wrong read never leaves the context of its own output.
Claude SonnetInteractive replies, morning briefs, nudge phrasingThe default tier, and the only one with write tools and web access, so it is the one the untrusted-text fence exists for.
Claude OpusVision escalation for ambiguous document reads; the monthly finance reviewReserved for the passes that are rare and high-judgment rather than routine.

Guardrails and evals

Every context (interactive, ceremony, finance, ingest) carries its own tool allowlist, not one shared list. Ingest, which reads whatever a document or email says, gets no tools at all. Text from outside the owner's own typed input is wrapped in a labelled fence with an explicit instruction that it is data, never a directive, before any model sees it.

A prompt or tool-allowlist edit does not ship on green CI alone. Every prompt file and every context file is hashed; a test fails until a live run against the real service has signed a receipt for that exact hash, and CI itself only ever runs the deterministic half of the test set, on purpose, so the model subscription never sits behind an automatic push.

Rules

  • No send tool exists; outbound messages live in transport code the model cannot reach
  • Ingest classification and document/vision reads run with allowedTools: [] and no MCP server
  • db_read accepts only a single SELECT statement, rejected in code if it contains a write verb
  • A document- or mail-derived proposal is staged; only a human tap on "Save" writes the row
  • A kill switch is checked before every model call and stops all of them at once

Evals

  • 59 golden cases across canaries, documents, email, factual answers, refusals, and RSS
  • 53 run deterministically in CI on every push; 6 need real credentials and run on the box
  • A dedicated canary asserts untrusted text is fenced, with an instruction attached, everywhere it reaches a model
  • Editing any prompt or context file changes the hash CI checks, so an untested change cannot pass silently

Rollout and adoption

There is no rollout to describe: I am the only user of this system, running on a box I administer myself. There is no adoption story here beyond my own use, and the site does not claim one.

How it was built

PLAN.md is the decision record every change is checked against. AI coding agents write the code; a local gate (typecheck, tests, the eval set, a web build) must pass before deploy, and deploy itself refuses a dirty tree or unmerged work. A prompt or tool-allowlist change carries one more gate: a live run against the real service, on the box, before CI accepts it.

Tools: Bun, TypeScript, SQLite, SvelteKit, Claude Agent SDK (Claude Haiku, Sonnet, and Opus)

Questions about this one: email me.