Running Mixture-of-Experts Models Bigger Than Your Mac’s Memory

Most of the strongest open models released this year are mixtures of experts. Qwen3.6-35B-A3B, gpt-oss, DeepSeek V4, the newest Mixtral and GLM releases — they all compute like small models and need memory like big ones. On a Mac, where the GPU shares one pool of memory with everything else, that second part decides what you can run.

TheYgent's MLX engine can now run a mixture-of-experts model whose weights don't fit in memory, by leaving most of it on the SSD and reading only what each token needs. The answers are the same ones the fully loaded model gives — token for token — only slower. This post explains how it works, what it costs, and what we learned measuring it.

What a mixture of experts is

In a dense model, every token passes through every weight. Most of those weights sit in each layer's feed-forward block. A mixture-of-experts model replaces that one big block with many small ones — the experts — and adds a small router to every layer. For each token, the router scores all the experts and sends the token through only the best few.

Qwen3.6-35B-A3B has 40 such layers with 256 experts each, and the router picks 8. The name says the rest: 35 billion parameters in total, about 3 billion active per token — the "A3B". You get the knowledge of a large model at the compute cost of a small one, which is why these models are fast.

The catch is that different tokens pick different experts. A model never knows in advance which ones the next token will want, so normally all of them have to be in memory. A 35B mixture of experts needs the memory of a 35B model, even though each token only touches about a tenth of it.

Leaving the experts on disk

That tenth is the opening. In the 4-bit Qwen3.6-35B-A3B, the parts every token uses — attention, embeddings, the routers, the shared expert — come to 1.4 GB. The 10,240 routed experts are the other 18.1 GB, at 1.77 MB each. A token needs at most 8 of them per layer, about 566 MB in all.

With expert streaming turned on, TheYgent loads only the shared part. At each mixture-of-experts layer, it waits for the router's choice, reads just those experts' bytes from the model files on disk — each expert is one contiguous slice of a stacked tensor, so a read is a plain file read — and runs MLX's own expert kernels on them. Recently used experts stay in a per-layer cache in GPU memory, so an expert the next token needs again costs nothing to fetch.

Because the computation is the same kernels on the same bytes, the output doesn't change. We checked that directly: on a tiny random model, every logit is bit-identical to the fully loaded model's, and on the real 35B model, every streamed run produced the same 128 tokens as the fully loaded one.

What it costs

Measured with Qwen3.6-35B-A3B in 4-bit (20 GB) on a 36 GB Apple Silicon Mac, generating 128 tokens:

So a model that needs 20 GB fully loaded answers in under 6 GB at 9 tokens a second, or under 2 GB at 4. That's slower than loading it — and if a model fits, loading it is always the better choice — but it turns "this model doesn't run here" into "this model runs here".

What we learned measuring it

We built this as a measured experiment first, and a few results changed the design.

• The cache matters more than the disk. A 4 GB cache holds 22% of the experts but served 75% of the lookups, because routing is skewed: some experts are far more popular than others. That alone doubled the speed over no cache.
• Per-expert caching beats pinning whole layers. Spending a similar 3.6 GB to keep eight layers fully loaded — no streaming there at all — reached only 5 tokens a second. Caching the popular experts everywhere is worth more than removing work from a few layers.
• Waiting for the router is cheap. Each layer has to stop and learn the router's choice before it can read anything. With every expert cached, that costs about a quarter of a millisecond per layer — real, but not the bottleneck. Misses are.
• The cache has to be read in place. Our first cache assembled the chosen experts into a fresh array at every layer; at 100% hits that still ran at 24 tokens a second. A fixed bank of slots in GPU memory, read directly by the kernels, reached 34 — and updating a slot has to be a slice write, because an indexed write copied the whole 453 MB bank every time.
• Rearranging the files helps only when the disk is the bottleneck. Repacking each expert into one contiguous record made cold reads 26% faster with no cache, and made no difference once the cache served most lookups. We left it out, since it would double the disk space a model takes.

The limits

Expert streaming is for mixture-of-experts chat models in MLX format, on Apple Silicon. A dense model has nothing to stream — every token uses every weight — and a checkpoint the feature can't read is refused when its engine starts, with the reason in the engine log, rather than served wrong.

The first reply to a long prompt is the slow part: processing a long prompt touches nearly every expert, so from a cold disk it can take several seconds per 2,000 tokens before the cache warms up. Several requests at once use more distinct experts, so the cache helps less. And speed depends on your SSD and on how much memory you give the cache.

Turning it on

It's off by default and set per model. In TheYgent's Hugging Face catalog, mixture-of-experts models carry an MoE badge, and the MoE filter shows only those. When you install the MLX variant of one that won't fit your memory, the install dialog offers Stream experts from disk, already ticked. You can turn it on or off later in the model's settings, where Expert cache (GB) sizes the cache — by default, half your memory less the model's shared part. Or set it when registering a model through the API:

curl -X PUT http://localhost:8081/admin/models/big-moe \
  -H 'Content-Type: application/json' \
  -d '{
    "binding": "mlx",
    "source": "hf",
    "model": "mlx-community/Qwen3.6-35B-A3B-4bit",
    "params": {"streamExperts": true, "expertCacheGb": 8}
  }'

Agents don't change: they still call the model by its logical ID, and whether that model streams its experts is the inference plane's business.

Try it

• Code: github.com/al2m4n/TheYgent
• Docs: expert streaming and the rest of the docs
• On a Mac: what runs where, and setting up