Crawler Nodes · Apify Store · New in 1.3

Give Your Agents the Web, on Your Terms.

TheYgent 1.3 adds crawler nodes. Browse the Apify Store, install a hosted scraper or crawler as a node, and wire it into a model as a tool — or run it as a fixed step. You choose which fields the model fills; everything else, including what a call may cost and how much the model reads back, is fixed on the node.

Try the Web Researcher ↓ Read the Docs →

One Node, Thousands of Crawlers

A crawler node runs an actor from the Apify Store: web search that returns the top pages as Markdown, places and reviews from a map, a whole site’s pages, product listings, job posts. TheYgent reads the actor’s input schema and turns it into the node — no code, no request templates.

It wires two ways, like every tool in TheYgent. Drag its use handle into a model’s tools port and the model calls it when a question needs fresh facts. Wire it inline and it runs as a fixed step, its results flowing to the next node. Failures follow the usual tool contract: a clear message on the node’s err port, and the run carries on.

Browse the Store, See Which Actors Work

The Crawlers browser searches the store from inside the editor. Each card shows the pricing model and headline price, how many people ran the actor in the last 30 days, and the share of its public runs that succeeded over those 30 days. That last number matters: one popular web-search actor had timed out on 36 of its 47 runs in a month. Open an actor and its input fields are listed, with the ones a model can usefully fill already ticked.

Of the store’s 100 most popular actors, 97 converted into a usable node with no configuration; the other three need a login cookie, which belongs in a secret. Store searches go from your control plane to Apify’s public API and are cached for a few minutes. Your browser never calls Apify, and browsing needs no account.

You Choose What the Model Fills

An actor’s input is written for a form: dozens of fields, proxies, result counts, paid add-ons. A crawler node offers the model only the fields you tick — usually one or two — and fixes the rest.

Three actors as tools, measured	Every field offered	As crawler nodes
Prompt tokens per call	about 13,500	about 780
Right tool, fact in the answer	6 of 6	6 of 6
Places per map search	50, chosen by the model	5, fixed on the node
Paid reviews add-on	switched on by the model	off, fixed on the node

Qwen3.6-35B-A3B (4-bit, MLX) on a 36 GB Apple Silicon Mac, six questions, October 19, 2026. Actor runs went to a local stand-in for Apify’s run API with invented facts, so a correct answer proves the model read the tool result.

Spend Caps and Result Budgets

Every run is capped. A crawler node sends its own charge cap with every run — $0.25 by default — and TheYgent holds it under a platform ceiling in Settings → Crawlers ($1 by default), so no agent can raise its own spend. Most popular actors bill per event, per page or per place, and the cap is what bounds them.

The model reads what fits. A crawl easily returns megabytes: through a plain HTTP tool, three documentation pages came back as a 50,175-character tool result. A crawler node keeps only the item fields you choose and shares a character budget — 6,000 by default — equally across the results, so every result is seen and a small model’s context isn’t flooded.

Small models get small arguments. Local models write list arguments unreliably — on MLX, a malformed list can drop the request — so list fields such as search terms and URLs are offered as one value per call by default, and TheYgent wraps it into the list the actor wants.

Interactive runs wait for the crawl in one request, up to five minutes. On a durable run, TheYgent starts the crawl, checks on it, and reads the results when it finishes, so a long crawl survives a restart — and the run waterfall shows what each crawl returned and what Apify charged for it.

Try the Web Researcher

The Web researcher template is a local model with a web_search crawler: it searches when a question needs current facts, reads the top three pages as trimmed Markdown, and answers with the URLs it used, for at most $0.05 a search.

Open Settings → Samples, pick a local model with tool calling, and install Web researcher.
Open the agent, select web_search, and use Set token to store your Apify API token.
Ask it something that happened this week.

Or start from any agent: select a model node, press Add crawler, search the store, tick the fields, and the configured node lands on the canvas already wired in.

Or Use Apify’s MCP Server

Apify also publishes an MCP server, and TheYgent’s MCP hub installs it in a few clicks. It hands the model every field of every actor you list, and can let it search the store and pick actors itself. That suits a capable hosted model exploring the store. For an agent that should do one job at a known cost on a local model, use a crawler node.

Apify is a hosted service. A crawler node sends its input — the fixed input and the model’s arguments — to Apify, which crawls from its own servers, and each run bills the Apify account behind your token. You need an Apify account and API token to run actors; per Apify’s pricing page (checked October 19, 2026), its free plan includes $5 of usage a month and needs no credit card. Unlike the rest of TheYgent, a crawler node needs the network. Apify is the only provider today.

Questions

Asked and Answered.

What is a crawler node in TheYgent?

A node that runs a hosted scraper or crawler — an actor from the Apify Store — and returns what it collects: search results with the pages as Markdown, places from a map, a site’s pages. Wire it into a model’s tools and the model calls it when a question needs fresh facts; wire it inline and it runs as a fixed step. You pick the actor in the Crawlers browser, and TheYgent turns its input schema into the node.

Which crawlers can I use?

Actors from the Apify Store. The Crawlers browser searches the store and shows each actor’s pricing, its users, and the share of its public runs that succeeded over the last 30 days. Of the store’s 100 most popular actors, 97 converted into a usable node with no configuration; the other three need a login cookie, which belongs in a secret.

How do I stop a model from running up a bill?

The model fills only the fields you expose, usually one or two; result counts, paid add-ons, and proxy settings stay fixed on the node, and any other argument the model invents is dropped. Every run also carries a charge cap: the node’s own, $0.25 by default, held under a platform ceiling in Settings → Crawlers, $1 by default, that no agent can raise.

Does a crawler node work with small local models?

Yes, and it is built for them. Offered as crawler nodes, three actors took about 780 prompt tokens per call on Qwen3.6-35B-A3B; offered with every field, as an MCP server presents them, about 13,500 — with the same accuracy. List fields are offered as one value per call by default, which local models write more reliably than lists, and the model reads a trimmed result instead of a whole crawl.

Do I need an Apify account?

Yes. Apify runs the actors on its own servers and bills the account behind the API token you store in TheYgent, which keeps it encrypted and sends it only to Apify. Per Apify’s pricing page (checked October 19, 2026), its free plan includes $5 of usage a month and needs no credit card. Browsing the store needs no account.

Should I use a crawler node or Apify’s MCP server?

Both work. Apify’s MCP server, installed from TheYgent’s MCP hub, hands a model every actor field and can let it pick actors on its own — a good fit for a capable model exploring the store. A crawler node fits when you know which actor the agent needs: you fix what each call may do and cost, and the prompt stays small.

Keep Reading

docs