Web

Crawl web pages

Crawl a small set of web pages synchronously.

GET/v1/web/crawl
x-api-keystringheader

API key (`sfk_...`)

Parameters
5
urlarrayrequiredquery

URLs to crawl. Repeat the `url` query parameter for multiple pages (max 5).

scanFullPagebooleanoptionalquery

When true, scroll the page to load dynamically appended content (infinite scroll). Default false.

waitForstringoptionalquery

Wait for a CSS selector before extraction. Must be prefixed with "css:" (e.g. css:main). JavaScript wait conditions are not supported.

max 200 chars

proxystringoptionalquery

Most expensive proxy tier to use. The page is tried on `datacenter` first and moves up only if it is blocked, stopping at this tier. Default: every tier (`mobile`).

one of: datacenter, residential, mobile

maxCreditsintegeroptionalup to maxCreditsquery

Most credits each URL may cost (1–25). Default 1. The batch shares one budget of this times the number of URLs; you're charged for what the pages use, at least 1 credit per URL.

≥ 1 · ≤ 25

Response fields
46

Endpoint-specific response payload.

Per-URL crawl results.

dataresults[]url
string

Final URL associated with this crawl result.

dataresults[]status
integer

HTTP status code reported for the page fetch.

dataresults[]success
boolean

Whether the page was crawled successfully.

Markdown extracted from the page when available.

dataresults[]markdownraw
stringoptional

Raw markdown for the crawled page.

dataresults[]markdownfit
stringoptional

Filtered markdown for the crawled page.

dataresults[]html
stringoptional

HTML content for the page when returned by the crawler.

dataresults[]metadata
objectoptional

Page metadata such as title or description when available.

dataresults[]errorMessage
stringoptional

Error message when the page crawl failed.

Links discovered on the page when available.

Same-host links discovered on the page.

dataresults[]linksinternal[]href
string

Absolute or page-relative link href.

dataresults[]linksinternal[]text
stringoptional

Anchor text when available.

dataresults[]linksinternal[]title
stringoptional

Title attribute when available.

Cross-host links discovered on the page.

dataresults[]linksexternal[]href
string

Absolute or page-relative link href.

dataresults[]linksexternal[]text
stringoptional

Anchor text when available.

dataresults[]linksexternal[]title
stringoptional

Title attribute when available.

Media assets discovered on the page when available.

Images on the page.

dataresults[]mediaimages[]src
string

Image source URL.

dataresults[]mediaimages[]alt
stringoptional

Alt text when available.

dataresults[]mediaimages[]score
numberoptional

Optional relevance score from the crawler.

Videos on the page.

dataresults[]mediavideos[]src
string
dataresults[]mediavideos[]alt
stringoptional
dataresults[]mediavideos[]score
numberoptional

Audio elements on the page.

dataresults[]mediaaudios[]src
string
dataresults[]mediaaudios[]alt
stringoptional
dataresults[]mediaaudios[]score
numberoptional

Summary counts for the crawl batch.

datasummaryrequestedUrls
integer

Number of URLs requested in the crawl batch.

≥ 0

datasummarysucceeded
integer

Number of URLs that crawled successfully.

≥ 0

datasummaryfailed
integer

Number of URLs that failed to crawl.

≥ 0

dataproxy
stringnullable

Proxy tier the page was loaded on: the tier that served it, or the last one tried. Null when not measured.

one of: datacenter, residential, mobile, null

databudgetExceeded
boolean

Whether the page needed more than `maxCredits` allows. Loading stopped early, so content may be incomplete.

databytesTransferred
integernullable

Data the page loaded through the proxy, in bytes, across every tier tried. Null when not measured.

≥ 0

Metadata describing the request and billing outcome.

metarequestId
string

Unique request identifier for tracing this API call.

min 1 chars

metacreditsCharged
integer

Credits charged for this request.

≥ 0

metaversion
string

Public API version that served the response.

one of: v1

metastale
booleanoptional

Present, and `true`, only when the source couldn't be reached and this is the last good answer for the same request instead, up to 24 hours old. Absent on every live answer.

one of: true

metafetchedAt
stringoptional

When a stale answer (`stale: true`) was originally fetched (ISO 8601, UTC). Absent on every live answer.

Code example

curl "https://api.socialfetch.dev/v1/web/crawl?url=https://www.socialfetch.dev" \
  -H "x-api-key: $SOCIALFETCH_API_KEY"

Responses

Crawl results.