Sumeet Singh Aulakh

Building a Personal Agentic App in Three Hours with Claude Code

I built Personal Agentic App — a local-first, voice-capable personal agent — almost entirely through Claude Code, in about three hours from first commit to final merge. It runs against local models via Ollama or hosted models via API key, your choice per role, and it plans, uses tools, executes, observes itself, and repeats through a custom loop with no external agent framework underneath it.

The repo has a RETROSPECTIVE.md I wrote afterward. This post pulls out the parts worth remembering.

The numbers

  • Timeline: first commit to final merge in ~3 hours, same day
  • Code: +5,058/-371 lines across 15 pull requests
  • Backend: ~3,200 lines of Python (FastAPI)
  • Frontend: ~900 lines of TypeScript/TSX (Next.js)
  • Testing: 76 backend tests, all passing, plus live browser verification

The build broke into 16 phases (P0 through P15): provider abstraction, then tools, then the loop itself, then MCP, then the API layer, then the UI, then observability, then voice, then approvals, then persistence, then hardening, then docs. Two more followed after the "core" build: deleting old conversations and switching the web search provider (P13, P14), then a cancel-in-progress-run button plus a clarification about approval scope (P15).

What actually took the time

Genuinely easier than expected for the boilerplate-shaped 80% of the work — provider adapters, CRUD endpoints, a WebSocket event stream, a Tailwind UI, a tool registry.

That's not the interesting part, though.

It was not uniformly easy. The 20% that involved actual design judgment or subtle runtime behavior — concurrency, race conditions, what a live LLM actually does versus what the code assumes it does — took real iteration.

The plan/act/observe/reflect loop itself is a few hundred lines of plain Python driven entirely through tool calls — not much code, but the part where "not much code" and "not much thinking" diverge sharply.

The bugs that only showed up when it ran for real

A green test suite didn't catch these. They only surfaced by actually running the app against a real model, or clicking through it in a browser:

  • The model's final answer read as confident, but the underlying tool_result in the trace showed an empty output — a grounding problem test doubles never would have caught.
  • The agent would occasionally spin without making progress; the loop needed an explicit nudge mechanism.
  • A silent mic-permission-denied failure that only surfaced through browser testing.
  • An unhandled exception inside a background task left a run's status stuck.
  • finish_run() only ever wrote the answer column, relying on a separate event-recording call — a race condition waiting to happen.
  • A test mock where the replacement AsyncClient closure accidentally called itself.
  • Adding task.cancel() for an in-progress run wasn't sufficient by itself — cancellation needed cleanup across multiple resource types.
  • The frontend's terminal event types went untouched by a backend cleanup pass, leaving UI buttons stuck disabled.

Test suites and linters are necessary but not sufficient; several of the worst bugs here shipped past a fully green CI run and were only found by actually using the feature.

What I'd tell myself next time

A few things from the recommendations section that generalize past this one project:

  • Keep phases small enough to fully verify — tests plus live use — before moving to the next one. This is, in the author's own words, "the single biggest lever for keeping an AI-built codebase trustworthy at scale."
  • "The tests pass" and "it works" are different claims. Treat them that way.
  • Watch for duplicated constants and logic across a language boundary — Python and TypeScript drifting out of sync was a repeat source of bugs here.
  • Async cancellation needs to be reasoned through explicitly, not bolted on.
  • When an agent finds and self-corrects a bug in its own draft, that's it working as intended — not a red flag.
  • State explicit, honest limitations rather than implying more confidence than the code has earned.
  • A human decision explicitly declined is not a bug to "fix" later.

What's honestly not there

The tool deliberately ships without authentication, command filtering, rate limiting, or RAG, and it's never been run against actual Docker or a physical microphone in production conditions. All of that's documented directly in the repo rather than glossed over — matching the "honest limitations" point above.

The actual takeaway

Three hours of directing, reviewing, and correcting — reading diffs, answering design questions, testing the running app, and pushing back when something wasn't right. The leverage is real, but it isn't "free."

That's the whole thing, really. Claude Code moved fast through the boilerplate-shaped work and needed a firm hand through the concurrency-shaped work — and the difference between those two was never obvious in advance.

Code is on GitHub: full-personal-agentic-app.