Crawl web pages
Crawl a small set of web pages synchronously.
Operation
/v1/web/crawlHeaders
API key (`sfk_...`)
Parameters
URLs to crawl. Repeat the `url` query parameter for multiple pages (max 5).
Response fields
Endpoint-specific response payload.
Per-URL crawl results.
Final URL associated with this crawl result.
HTTP status code reported for the page fetch.
Whether the page was crawled successfully.
Markdown extracted from the page when available.
Raw markdown for the crawled page.
Filtered markdown for the crawled page.
HTML content for the page when returned by the crawler.
Page metadata such as title or description when available.
Provider error message when the page crawl failed.
Summary counts for the crawl batch.
Number of URLs requested in the crawl batch.
≥ 0
Number of URLs that crawled successfully.
≥ 0
Number of URLs that failed to crawl.
≥ 0
Metadata describing the request and billing outcome.
Unique request identifier for tracing this API call.
min 1 chars
Credits charged for this request.
≥ 0
Public API version that served the response.
one of: v1
Code example
Responses
Crawl results.
Error codes