Engineering · · 8 min read

Building a Local-First Agent Platform Where the Vendor Is Never in the Middle

How TheYgent keeps inference on hardware you control: two planes and one seam, logical model IDs, an engine manager instead of a model server, agents as content-addressed documents, and durable runs.

By Arman Bahlakeh

Every visual AI builder we tried had the same footnote: local models supported — bring your own server. That footnote is where most of the work lives. Someone has to pick an engine, download the right quantization, size the context to the machine, keep the server running, and wire its URL into the builder. And the builder itself often reports usage home by default.

TheYgent started from the other end. The inference engine is part of the product, and the product is never a middleman you didn't choose between you and your models. This post walks through the architecture that follows from that: what the pieces are, where the boundaries sit, and why they are shaped the way they are.

Two planes, one seam

The most important boundary in TheYgent splits the system into two planes.

The inference plane runs models. It holds everything heavy — engine processes, model weights, the model registry, API credentials — and it runs wherever you point it: the same laptop, a GPU box across the network, or both.

The control plane orchestrates agents: the graph runtime, the agent registry and versions, sessions, triggers, MCP servers, retrieval sources, secrets, and run traces.

One rule keeps the split honest: the control plane never imports an engine. There is no MLX, vLLM, or llama.cpp binding anywhere in it. It reaches inference only through the inference plane's OpenAI-compatible HTTP API, through one shared client. Two things follow.

First, the payload boundary is structural, not a policy. When you chat with a model directly, the browser sends prompts, audio, and images straight to the inference plane. When an agent runs, the control plane — which you host too — sends each model call to the inference plane by logical ID. Nothing routes through a service you don't run, and when data crosses a wire to a hosted API, it does so because you registered that API.

Second, deployment stays boring. Inference is the one component that earns its own service: it pins a GPU, holds multi-gigabyte weights, and is already an HTTP server. Everything else is one FastAPI application, internally modular. The same four process types — interface, control plane, inference plane, and an optional durable worker — run on bare metal with make up, in Docker Compose, and on Kubernetes. Only the wiring changes.

Logical model IDs: the agent never names an engine

Every model reference in an agent, a chat, or a benchmark is a logical ID such as triage-fast. Engine names never appear on the data plane; if a graph tries to call mlx or llamacpp as if it were a model, the run is rejected up front with engine_name_not_allowed.

Binding an ID to something that serves it happens once, on the inference plane's management API:

curl -X PUT http://localhost:8081/admin/models/triage-fast \
  -H 'Content-Type: application/json' \
  -d '{
    "binding": "llamacpp",
    "source": "local-path",
    "model": "/path/to/model-Q4_K_M.gguf",
    "modality": "chat"
  }'

The binding is one of exactly four values: mlx, vllm, llamacpp, or openai-compatible. The first three are engines whose lifecycle TheYgent manages; everything else in the world — a hosted API, a model server you already run — is openai-compatible plus a URL. Two more axes sit beside it and never fold into it: source says where weights come from (hf, local-path, url), and modality says what the server does (chat, vision, embeddings, speech-to-text, text-to-speech, image generation).

The payoff is that moving a model from MLX on a Mac to llama.cpp on a Linux box — or from a local model to a hosted API — is one registration change. No agent is edited and none is republished. And because engine choice is per node, one graph can run a local model and a hosted one side by side: a small local model triages every request and a router sends only the hard ones to an API.

An engine manager, not a model server

The inference plane doesn't ask you to start servers. It starts them.

  • Lazy start. An engine spawns on the first call to one of its logical IDs, on a free local port, and the request waits until it is healthy. That's why the first call to a model is the slow one.
  • A resident ceiling. Only a few engines stay loaded at once — two by default. When a new one is needed, the least recently used, lowest-priority engine is evicted. An engine with a request in flight is never evicted; if nothing can be freed, the call fails with no_capacity rather than thrashing.
  • Idle reaping. An engine idle for fifteen minutes is torn down; mark a model keep warm to exempt it.
  • Abandoned calls are cancelled. If a client hangs up before a non-streaming answer arrives, the request to the engine is closed at once — which matters most for image generation, where a render can take minutes.
  • One launcher per engine and modality. Chat, vision, embeddings, transcription, speech, and images each map to the right server program, and a combination with no server is refused with a clear error instead of being served wrong.

Models arrive through an in-app Hugging Face catalog that shows, for every quantization, whether it fits the machine's memory before you download it. Installed weights register as local files, so loading them never touches the network.

The graph is a document

On the canvas you drag nodes and draw edges. Underneath, an agent is a typed document — an intermediate representation defined once in Pydantic, with the TypeScript types the editor uses generated from it, and a CI check that fails if they drift.

The document carries a content hash, computed server-side over canonical JSON: defaults filled in, keys sorted, and the layout block stripped. Moving a node on the canvas never changes an agent's identity; changing what it does always does. Publishing mints an immutable, content-addressed version you can invoke by API, put behind a trigger, or roll back to.

Secrets never live in the document. Connections carry references to an encrypted store, so rotating an API key keeps the reference — and the agent's hash — unchanged.

Durable when it matters

Agents run on two runtimes. The interactive one is an in-process interpreter that streams tokens live; if the process dies mid-run, the run is honestly marked failed. That's fine when a person is watching. Unattended work — a webhook at 3 a.m., a weekly schedule, an approval that waits for days — needs more: a run that survives a crash and resumes from the last completed step without redoing work.

The durable runtime provides that. It is built on DBOS, an embedded Python library that journals workflow state into the Postgres the control plane already uses — no new stateful service to deploy. A few rules make it correct:

  • One registered workflow. Recovery looks workflows up by name, so instead of generating one per agent there is exactly one, which loads the pinned, immutable agent document and walks it.
  • An honest guarantee. Completed steps are exactly-once: on resume they replay from the journal and are not re-executed, so a POST is not re-sent. The step in flight at the crash is at-least-once, so HTTP tools support an idempotency key.
  • Orchestration does no I/O. Routers, transforms, rule guardrails, and loop control run inside the workflow body and re-derive identically from journaled results — that determinism is what makes resume work. All I/O lives in journaled steps.
  • Tokens are a side effect. Streams go to subscribers as they arrive; only the assembled answer is journaled, so a replayed step never re-streams.

Human approval, subgraphs, loops, and fan-out maps are durable-only node types. A human-approval node can wait for days and still pick up after a restart.

Observability without extra infrastructure

Every run produces spans and per-node inputs and outputs in your own Postgres, and the interface draws them as a zoomable waterfall: which node ran, for how long, on which model, with how many tokens. What gets captured — and for how long — is set per agent, with retention and redaction rules. Exporting to an OpenTelemetry collector is a second, opt-in sink, never a requirement.

Being checkable about privacy

"Your data stays private" is easy to write and hard to verify. So rather than promise, TheYgent publishes the complete list of every connection it makes and when it makes it: Hugging Face when you browse or install models, the MCP registries when you open the hub browser, and whatever you configure yourself — a hosted API, a remote MCP server, a site you crawl.

The list includes the one request we could have left out: at startup, the inference plane's model client library fetches a public model price list from GitHub. It carries none of your data and falls back to a bundled copy offline — but it is a request, so it's on the list. There's no usage analytics, crash reporting, update check, or license server, and once installed, TheYgent runs fully offline.

The promise is precise on purpose: you own every hop. You can register a hosted API whenever you want one. TheYgent just refuses to be a middleman you didn't choose.

Loud failures, deliberate contracts

Two habits run through the codebase. Failures are loud: an unknown input reference, an invalid graph, an engine name where a logical ID belongs, a port nothing can fill — each fails fast with a named error code, never a silent fallback. And contracts are deliberate: the data-plane shape, the agent document envelope, and the four-value binding enum stay frozen until a change is consciously made, named, and tested — even when it's additive.

Try it

TheYgent is source-available under the Sustainable Use License, and it self-hosts on macOS and Linux — bare metal, Docker Compose, or Kubernetes.