Web Crawl API
Pass a small set of public URLs in one request and get per-page markdown back — same Social Fetch key as markdown, ask, and social routes.
GET /v1/web/crawl1 credit per URL requested. Up to 5 URLs per request (5 credits max).
About this endpoint
Competitor briefs and RAG jobs often start with a short list: pricing page, docs home, about page, changelog. Calling markdown once per URL works, but a five-URL batch is fewer round trips when you already know the set.
`GET /v1/web/crawl` takes one or more `url` query parameters (repeat `url` for each page, max 5). Each value must be a public http or https URL (up to 2083 characters). The response uses the shared `{ data, meta }` envelope. `data.results` is one row per requested URL with `url`, HTTP `status`, `success`, optional `markdown` (`raw` / `fit`), and optional `html`, `metadata`, or `errorMessage`. `data.summary` counts `requestedUrls`, `succeeded`, and `failed`. Billing is 1 credit per URL requested (up to 5 credits). Confirm with `meta.creditsCharged`.
This route is a bounded synchronous batch, not a site spider. It does not follow links or mirror an entire domain. For one page at a time, use `GET /v1/web/markdown` or `GET /v1/web/html`. For a question about one URL, use `GET /v1/web/ask`. To discover candidate URLs first, use `GET /v1/web/search`.
Web Crawl API FAQ
What does the Web Crawl API return?
A data object with results (one row per URL: url, status, success, optional markdown with raw/fit, plus optional html, metadata, or errorMessage) and summary counts (requestedUrls, succeeded, failed). The envelope also includes meta.requestId and meta.creditsCharged.
How do I call web crawl?
Send GET /v1/web/crawl with x-api-key and at least one url query parameter (public http or https, up to 2083 characters). Repeat the url parameter for each page, up to 5 URLs. Full rules and examples are in the API docs.
How much does web crawl cost?
1 credit per URL requested, up to 5 URLs per request (5 credits max), as documented on the operation. Confirm on every response with meta.creditsCharged — that field is the billing source of truth.
Can I crawl an entire website?
No. This route accepts a fixed list of up to five public URLs you already have. It does not discover or follow links across a domain. Split larger jobs into multiple crawl calls or fetch URLs one at a time with markdown/html.
How is crawl different from web markdown?
GET /v1/web/markdown extracts one URL. GET /v1/web/crawl batches up to five URLs in a single synchronous request and returns per-URL rows plus a summary. Use markdown for a single page; use crawl when you already have a short list.
What happens if one URL in the batch fails?
Other URLs can still succeed. Check each row's success flag and optional errorMessage, and read data.summary for succeeded vs failed counts. Do not assume every requested URL produced markdown.
Can web crawl fetch pages behind a login?
No. Only publicly reachable pages without your session cookies. Private dashboards and authenticated app screens are out of scope.