Apple Silicon · MLX · llama.cpp · whisper.cpp

Local AI Agents on Apple Silicon, Powered by MLX.

TheYgent runs models natively on M-series Macs with Apple’s MLX — chat, vision, and text-to-speech — alongside llama.cpp for embeddings and whisper.cpp for speech-to-text. One command installs the engines; then you build agents on a no-code canvas and keep every model, prompt, and run on your Mac.

Why MLX on a Mac

MLX is Apple’s array framework for Apple Silicon. Because the CPU and GPU share one pool of unified memory, a model doesn’t have to be copied between them, so MLX runs fast and memory-efficiently on M-series chips. Many model repositories ship both a GGUF build for llama.cpp and an MLX build; on a Mac, TheYgent’s catalog shows whichever your installed engines can run, and MLX is usually the best choice for chat and vision.

You don’t start or babysit the servers. TheYgent spawns an MLX or llama.cpp server on the first call to a model, waits for it to be healthy, keeps it warm while you use it, and tears it down after 15 idle minutes. Two engines stay resident by default (THEYGENT_MAX_RESIDENT), so a laptop never tries to hold every model in memory at once.

What Runs Where on a Mac

Modality	Engine	Server
Chat	MLX (or llama.cpp)	`mlx_lm.server` · `llama-server`
Vision	MLX (or llama.cpp)	`mlx_vlm.server` · `llama-server` with a projector
Text-to-speech	MLX	`mlx_audio.server`
Embeddings	llama.cpp	`llama-server`
Speech-to-text	whisper.cpp	`whisper-server`
Image generation	MLX (or stable-diffusion.cpp)	`mflux` · `sd-cli` — installed separately; the newest path

A voice agent — speech-to-text, a language model, text-to-speech — runs end to end on one Mac with these engines. Watch the five-minute walkthrough.

Set Up in Four Commands

git clone https://github.com/al2m4n/TheYgent.git && cd TheYgent
cp .env.example .env   # the defaults work for a local install
make engines           # llama.cpp, whisper.cpp, ffmpeg via Homebrew; MLX servers via uv
make up                # interface on http://localhost:5174

You need uv, pnpm, and a Postgres to point at — make up starts one in Docker if you don’t have your own.

On an Apple Silicon Mac, make engines installs the whole local set: llama.cpp, whisper.cpp, and ffmpeg through Homebrew, and the MLX chat, vision, and text-to-speech servers through uv tool install. Open the interface, create the local admin account in the first-run wizard, then browse the model catalog: every variant carries a fit badge against your Mac’s memory, so you can see what will run before you download it.

One Agent, Any Machine

Agents reference models by a logical ID, not by engine. Build an agent on your Mac with triage-fast bound to MLX, then run the same agent on a Linux server where triage-fast is bound to llama.cpp or vLLM (experimental) — the graph doesn’t change. One graph can also mix a local MLX model with a hosted API on another node, when you choose to.

Docker on a Mac

Containers on macOS can’t reach the Apple GPU, so MLX runs on bare metal only. To use Docker Compose and keep MLX, run the inference plane on the host and overlay docker-compose.host-inference.yml; everything else runs in containers.

Questions

Asked and Answered.

Can I build AI agents that run locally on a Mac?

Yes. TheYgent is a no-code AI agent builder that runs models on Apple Silicon with MLX for chat, vision, and text-to-speech, llama.cpp for embeddings, and whisper.cpp for speech-to-text. make engines installs the engines, and models install from an in-app Hugging Face catalog with a memory fit check.

Which MLX models can TheYgent run?

MLX chat models through mlx_lm, vision models through mlx_vlm, and text-to-speech models through mlx_audio, plus image generation through mflux once you install it. MLX does not serve embeddings or speech-to-text in TheYgent; llama.cpp and whisper.cpp cover those on the same Mac.

Does TheYgent need Ollama or LM Studio on a Mac?

No. TheYgent runs MLX and llama.cpp itself: it starts each engine on the first call, keeps it warm, and unloads it when idle. There is no separate model server to install or keep running.

Does TheYgent run on Linux too?

Yes. On Linux, llama.cpp and whisper.cpp provide chat, vision, embeddings, and speech-to-text, and vLLM is available as an experimental engine for NVIDIA GPUs. The MLX engines are Apple Silicon only.

Keep Reading

privacy