Web Crawl API
Pass a small set of public URLs in one request and get per-page markdown back — same Social Fetch key as markdown, ask, and social routes.
GET /v1/web/crawl1 credit per URL Up to 5 URLs per request Max 5 credits
About this endpoint
GET /v1/web/crawl takes one or more url query parameters (repeat url for each page, max 5). Each value must be a public http or https URL (up to 2083 characters). The response uses the shared { data, meta } envelope. data.results is one row per requested URL with url, HTTP status, success, optional markdown (raw / fit), and optional html, metadata, or errorMessage. data.summary counts requestedUrls, succeeded, and failed. Billing is 1 credit per URL requested (up to 5 credits). Confirm with meta.creditsCharged.
This route is a bounded synchronous batch, not a site spider. It does not follow links or mirror an entire domain. For one page at a time, use GET /v1/web/markdown or GET /v1/web/html. For a question about one URL, use GET /v1/web/ask. To discover candidate URLs first, use GET /v1/web/search.
Web Crawl API FAQ
What does the Web Crawl API return?
A data object with results (one row per URL: url, status, success, optional markdown with raw/fit, plus optional html, metadata, or errorMessage) and summary counts (requestedUrls, succeeded, failed). The envelope also includes meta.requestId and meta.creditsCharged.
How do I call web crawl?
Send GET /v1/web/crawl with x-api-key and at least one url query parameter (public http or https, up to 2083 characters). Repeat the url parameter for each page, up to 5 URLs. Full rules and examples are in the API docs.
How much does web crawl cost?
1 credit per URL requested, up to 5 URLs per request (5 credits max), as documented on the operation. Confirm on every response with meta.creditsCharged — that field is the billing source of truth.
Can I crawl an entire website?
No. This route accepts a fixed list of up to five public URLs you already have. It does not discover or follow links across a domain. Split larger jobs into multiple crawl calls or fetch URLs one at a time with markdown/html.
How is crawl different from web markdown?
GET /v1/web/markdown extracts one URL. GET /v1/web/crawl batches up to five URLs in a single synchronous request and returns per-URL rows plus a summary. Use markdown for a single page; use crawl when you already have a short list.
What happens if one URL in the batch fails?
Other URLs can still succeed. Check each row's success flag and optional errorMessage, and read data.summary for succeeded vs failed counts. Do not assume every requested URL produced markdown.
Can web crawl fetch pages behind a login?
No. Only publicly reachable pages without your session cookies. Private dashboards and authenticated app screens are out of scope.