Self-hosted agent runtime · Claude Agent SDK

A self-hosted multi-agent runtime built around the way I actually work.

What would an agent system look like if I controlled its memory, tools, interfaces, and failure modes?

Role
Agent architecture, systems integration, and product engineering
Format
Open-source personal system · Ongoing
Tools
Python · TypeScript · Claude Agent SDK · Convex · React · LiveKit
  • 4Chat and voice interfaces
  • 2Persistent memory layers
  • 3-passAdversarial consolidation
  • 1Shared agent runtime

The problem

Building an agent I could inspect and recover.

I was intrigued by agent systems such as OpenClaw and Hermes, but I wanted something shaped around my own routines and inspectable when it behaved unexpectedly. I started with the Claude Agent SDK and built outward, one useful layer at a time.

Trust was not a single feature. I needed to understand what the system could do, what it remembered, which channel a request came from, and how to recover when a long-running task or service failed.

The approach · 6 steps

One shared runtime, with capability added in layers.

  1. Centralized every model turn behind one Agent SDK wrapper so streaming, sessions, tools, and usage logging followed the same path.
  2. Connected Telegram, Discord, iMessage, and browser voice without creating a separate agent runtime for each interface.
  3. Combined a small always-on user context with vector recall for longer-term memories that only surface when relevant.
  4. Added tools, reusable skills, automations, and task-scoped specialist agents for content, development, research, and planning.
  5. Built queues, warm sessions, rate-limit handling, and in-flight recovery so the system could survive real usage instead of only clean demos.
  6. Added a live dashboard and checkpointed operations for inspecting memory, activity, schedules, failures, and system changes.

The result

Trust came from visibility, recovery, and clear boundaries.

  • 4Interfaces on one runtime
  • WarmSession reuse across turns
  • DailyMemory consolidation cycle

The most valuable work was making memory, tools, and operations visible and reversible enough for regular use. Greater autonomy only helped when the boundaries stayed understandable.

What I would do next

  • Add end-to-end integration tests against a disposable Convex deployment.
  • Measure memory retrieval quality and task outcomes over time.
  • Turn the documented setup path into a repeatable deployment workflow.