# Social Fetch API - Web endpoints
# Base URL: https://api.socialfetch.dev
# Auth: x-api-key header (sfk_...)
# Full catalog: https://www.socialfetch.dev/llms-endpoints.txt | JSON: https://www.socialfetch.dev/llms.json
# Docs hub: https://www.socialfetch.dev/docs/api | OpenAPI: https://www.socialfetch.dev/openapi.json

7 documented Web operations. Do not invent paths - only the routes below (and their linked markdown pages) are supported.

## GET /v1/web/search
Search the web
Credit cost: 1 credit per successful request.
Docs: https://www.socialfetch.dev/docs/api/v1/web/search/get.mdx
SDK: client.web.search({ query: "Social media scraping API", region: "US" })
Parameters:
- query (required, string) - Search query text to run against the public web.
- region (optional, string) - ISO 3166-1 country code for localized results (e.g. US, GB, CA).
- datePosted (optional, string, enum: last-hour | last-day | last-week | last-month | last-year) - Optional filter by when results were posted.
- page (optional, integer) - Page number (1-based). Default: 1.
Pagination: send query `page` from prior `data.page.nextCursor`; stop when `data.page.hasMore` is false.
curl "https://api.socialfetch.dev/v1/web/search?query=social media scraping API" \
  -H "x-api-key: YOUR_API_KEY"

## GET /v1/web/markdown
Generate web page markdown
Credit cost: 1 credit per successful request.
Docs: https://www.socialfetch.dev/docs/api/v1/web/markdown/get.mdx
SDK: client.web.getMarkdown({ url: "https://www.socialfetch.dev/" })
Parameters:
- url (required, string) - Web page URL to fetch.
- filter (optional, string, enum: fit | raw | bm25) - Markdown extraction filter. `fit`: strip boilerplate and extract the main readable content. `raw`: full unfiltered page markdown, no content pruning. `bm25`: rank and return only the content most relevant to `query`, using the BM25 keyword-relevance algorithm — requires `query` to be set.
- query (optional, string) - Optional query string used by the bm25 filter to rank relevant content.
- cacheMode (optional, string, enum: enabled | bypass | write_only) - Cache behavior. `enabled`: read from cache if present, else fetch and write to cache. `bypass`: always fetch fresh, ignoring and not updating the cache. `write_only`: always fetch fresh, but write the result to cache without reading from it first. Default: `enabled`.
- scanFullPage (optional, boolean) - When true, scroll the page to load dynamically appended content (infinite scroll). Default false.
- waitFor (optional, string) - Wait for a CSS selector before extraction. Must be prefixed with "css:" (e.g. css:main). JavaScript wait conditions are not supported.
Outcome field: `data.lookupStatus` in `found`, `restricted`
Empty results: `lookupStatus: restricted` means bot/access protection blocked the fetch; content fields are null.
curl "https://api.socialfetch.dev/v1/web/markdown?url=https://www.socialfetch.dev" \
  -H "x-api-key: YOUR_API_KEY"

## GET /v1/web/ask
Ask a question about a web page
Credit cost: 1 credit per successful request.
Docs: https://www.socialfetch.dev/docs/api/v1/web/ask/get.mdx
SDK: client.web.ask({ url: "https://www.socialfetch.dev/", q: "What is this page about?" })
Parameters:
- url (required, string) - Web page URL to fetch.
- q (required, string) - Natural-language question to answer about the page content.
Outcome field: `data.lookupStatus` in `found`, `restricted`
Empty results: `lookupStatus: restricted` means bot/access protection blocked the fetch; `answer` is null.
curl "https://api.socialfetch.dev/v1/web/ask?url=https://www.socialfetch.dev&q=What is this page about?" \
  -H "x-api-key: YOUR_API_KEY"

## GET /v1/web/html
Generate web page HTML
Credit cost: 1 credit per successful request.
Docs: https://www.socialfetch.dev/docs/api/v1/web/html/get.mdx
SDK: client.web.getHtml({ url: "https://www.socialfetch.dev/" })
Parameters:
- url (required, string) - Web page URL to fetch.
- scanFullPage (optional, boolean) - When true, scroll the page to load dynamically appended content (infinite scroll). Default false.
- waitFor (optional, string) - Wait for a CSS selector before extraction. Must be prefixed with "css:" (e.g. css:main). JavaScript wait conditions are not supported.
Outcome field: `data.lookupStatus` in `found`, `restricted`
Empty results: `lookupStatus: restricted` means bot/access protection blocked the fetch; `html` is null.
curl "https://api.socialfetch.dev/v1/web/html?url=https://www.socialfetch.dev" \
  -H "x-api-key: YOUR_API_KEY"

## GET /v1/web/screenshot
Capture website screenshot
Credit cost: 1 credit (2 with region)
Docs: https://www.socialfetch.dev/docs/api/v1/web/screenshot/get.mdx
SDK: client.web.getScreenshot({ url: "https://www.socialfetch.dev/" })
Parameters:
- url (required, string) - Web page URL to fetch.
- fullPage (optional, boolean) - When true, capture the full scrollable page. Default: false (viewport).
- viewportWidth (optional, integer) - Viewport width in CSS pixels. Default: 1280.
- viewportHeight (optional, integer) - Viewport height in CSS pixels. Default: 800.
- deviceScaleFactor (optional, integer) - Device scale factor (1–3). Default: 1.
- selector (optional, string) - CSS selector to clip the screenshot to a single element. Cannot be combined with fullPage.
- format (optional, string, enum: png | jpeg | webp) - Output image format. Default: png.
- quality (optional, integer) - JPEG/WebP quality 1–100. Invalid when format is png.
- delay (optional, integer) - Extra settle delay in milliseconds after load (0–10000).
- waitFor (optional, string) - CSS selector to wait for before capturing.
- waitUntil (optional, string, enum: load | domcontentloaded | networkidle) - Navigation wait condition. `networkidle` is bounded and resolves on idle or a short cap, whichever comes first. Default: load.
- blockCookieBanners (optional, boolean) - Dismiss/block cookie consent banners. Default: true.
- blockAds (optional, boolean) - Block ads and trackers during render. Default: true.
- darkMode (optional, boolean) - Request prefers-color-scheme: dark. Default: false.
- region (optional, string) - Optional ISO 3166-1 alpha-2 country for geo-located rendering (+1 credit).
- cacheMode (optional, string, enum: enabled | bypass | write_only) - Cache behavior. `enabled`: read from cache if present, else fetch and write to cache. `bypass`: always fetch fresh, ignoring and not updating the cache. `write_only`: always fetch fresh, but write the result to cache without reading from it first. Default: `enabled`.
- cacheTtl (optional, integer) - Optional Redis artifact-cache TTL in seconds (60–604800). Must stay strictly below the 7-day object lifetime.
- response (optional, string, enum: url | base64) - Delivery mode. `url` (default) returns a hosted CDN URL valid for 7 days. `base64` returns the image bytes inline when small enough.
Outcome field: `data.lookupStatus` in `found`, `restricted`
Empty results: `lookupStatus: restricted` means bot/access protection blocked the capture; an artifact may still be present.
curl "https://api.socialfetch.dev/v1/web/screenshot?url=https://www.socialfetch.dev" \
  -H "x-api-key: YOUR_API_KEY"

## GET /v1/web/crawl
Crawl web pages
Credit cost: 1 credit per URL requested. Up to 5 URLs per request (5 credits max).
Docs: https://www.socialfetch.dev/docs/api/v1/web/crawl/get.mdx
SDK: client.web.crawl({ urls: ["https://www.socialfetch.dev/"] })
Parameters:
- url (required, array) - URLs to crawl. Repeat the `url` query parameter for multiple pages (max 5).
- scanFullPage (optional, boolean) - When true, scroll the page to load dynamically appended content (infinite scroll). Default false.
- waitFor (optional, string) - Wait for a CSS selector before extraction. Must be prefixed with "css:" (e.g. css:main). JavaScript wait conditions are not supported.
curl "https://api.socialfetch.dev/v1/web/crawl?url=https://www.socialfetch.dev" \
  -H "x-api-key: YOUR_API_KEY"

## POST /v1/web/extract
Extract structured data from a web page
Credit cost: 2 credits per successful request.
Docs: https://www.socialfetch.dev/docs/api/v1/web/extract/post.mdx
SDK: client.web.extract({ url: "https://example.com/products", schema: { name: "products", baseSelector: "div.product", fields: [{ name: "name", selector: "h2", type: "text" }] } })
Outcome field: `data.lookupStatus` in `found`, `restricted`
Empty results: `lookupStatus: restricted` means bot/access protection blocked the fetch; `extracted` is null.
curl "https://api.socialfetch.dev/v1/web/extract" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/products","schema":{"name":"products","baseSelector":"div.product","fields":[{"name":"name","selector":"h2","type":"text"},{"name":"price","selector":".price","type":"text"}]}}' \
  -X POST