Web extraction API
Search the open web, fetch public pages as markdown or cleaned HTML, ask a question about one URL, or crawl a short URL list. Same API key and `{ data, meta }` envelope as TikTok and LinkedIn.
5 endpointsWhy teams use Social Fetch for Web
Social pipelines rarely stop at the profile card. A TikTok bio links to a Shopify store. A competitor tweet points at a pricing page. An agent brief cites a docs article that never lived on Instagram. Homegrown Puppeteer jobs and a second scraping vendor fix that gap until auth, billing, and response shapes diverge from the social stack you already ship.
Dedicated web-crawl products (Firecrawl-class scrape / crawl / map APIs) are the right tool when the job is a whole docs site, sitemap discovery, or long-running crawls. Social Fetch Web is narrower on purpose: five HTTP routes under `/v1/web/...` for the public pages that sit next to social lookups (keyword discovery, single-URL markdown or HTML, grounded Q&A on one page, and a synchronous batch of up to five known URLs). You do not get a site spider or a browser automation IDE.
`GET /v1/web/search` returns ranked title / URL / snippet rows. `GET /v1/web/markdown` and `GET /v1/web/html` extract one URL you already have. `GET /v1/web/ask` takes `url` plus `q` and returns an LLM answer grounded in that page. `GET /v1/web/crawl` repeats `url` up to five times and returns per-URL rows plus a summary. Responses use the shared envelope (`lookupStatus` where applicable, `meta.requestId`, `meta.creditsCharged`) so a landing-page extract sits in the same warehouse schema as a LinkedIn company row.
The same API key covers the rest of the marketplace catalog (17 platforms, 137 endpoints). Use this hub when social APIs are not enough and you still want one billing wallet. Use a purpose-built crawl farm when you need unbounded site mirrors or map-first discovery at crawl-product depth.
What Web data can you get?
- Public web keyword search: title, URL, and snippet rows with optional region and datePosted filters.
- Single-URL markdown extract with filter modes (`fit`, `raw`, `bm25`) for RAG and archives.
- Single-URL cleaned HTML for DOM parsers, selectors, and markup diffs.
- Page ask: natural-language question about one public URL, grounded answer back.
- Bounded crawl: up to five known URLs in one synchronous request with per-URL results.
- Shared `{ data, meta }` envelope and credit wallet with social marketplace routes.
- Public pages only: no session cookies, no login walls, no Playwright fleet.
Web API endpoints
Grouped by capability from the live API catalog. Open an endpoint page for parameters and examples — credit notes come from OpenAPI pricing extensions.
Search & discovery
1Page fetch
4How to call the Web API
1. Create a key and smoke-test one URL
Sign up at Social Fetch and copy an `sfk_…` key. New accounts get 100 free credits. Start with `GET /v1/web/markdown?url=…` (or search with a real query) so you see `lookupStatus`, content fields, and `meta.creditsCharged` before you wire agents or cron.
2. Pick search → extract → ask → crawl
Discover candidates with `GET /v1/web/search`. Hydrate keepers with `GET /v1/web/markdown` or `GET /v1/web/html`. Use `GET /v1/web/ask` when you want one grounded answer without shipping full page text. Pass a short known list to `GET /v1/web/crawl` (repeat `url`, max 5). Pass `x-api-key` on every call.
3. Combine with social tools
Branch on `lookupStatus` / per-URL `success` before writing rows. Store markdown or answers beside social entity ids. Log `meta.requestId` on bad rows. In MCP or agent setups, register Web tools next to profile tools so the model can fetch a bio link after a social lookup.
How much does the Web API cost?
Web routes are credit-metered. Single-page markdown, HTML, ask, and search amounts are shown on each endpoint row on this hub (and in the API registry); do not assume every Web call costs the same as every social call. Crawl is metered per URL requested: 1 credit per URL, up to 5 URLs per request (5 credits max). Confirm every response with `meta.creditsCharged`. Credits never expire on pay-as-you-go packs. Signup includes 100 free credits.
Social Fetch vs the official Web API
There is no single official "web API." Browser automation fleets, proxy scrape APIs, and Firecrawl-style LLM crawl products all solve overlapping open-web jobs with different depth and billing shapes. Social Fetch Web is for teams that already use the social marketplace and need public page extract / search / ask / small-batch crawl on the same key and envelope. It is not a replacement for unbounded site crawl, map, or authenticated browser sessions.
Web API FAQ
What is the Social Fetch Web API?
Five REST routes under the Web tag: public keyword search (`GET /v1/web/search`), URL to markdown (`GET /v1/web/markdown`), URL to cleaned HTML (`GET /v1/web/html`), page Q&A (`GET /v1/web/ask`), and a bounded multi-URL crawl (`GET /v1/web/crawl`). Authenticate with a Social Fetch API key. Exact parameters and examples live under `/docs/api` and on this hub's endpoint list.
Is this a Firecrawl alternative or a full site crawler?
It is Firecrawl-adjacent for single-page markdown, HTML, search, and small batches, not a full crawl-product replacement. Social Fetch crawl accepts up to five URLs you already have and does not follow links or mirror a domain. If you need sitemap map, deep recursive crawl, or crawl-as-a-product depth, use a dedicated web crawl vendor. Use Social Fetch when those pages sit next to social lookups on one marketplace key.
How do I turn a URL into markdown for RAG?
Call `GET /v1/web/markdown` with a required public `url`. Optional `filter` is `fit` (default, readability-oriented), `raw`, or `bm25` (pass `query` when using bm25). On `lookupStatus: found`, read `data.markdown`. Prefer this route for LLM-ready text; use `GET /v1/web/html` when you need markup.
How is GET /v1/web/ask different from POST /v1/ask?
`GET /v1/web/ask` answers a natural-language question about one public page URL (`url` + `q`). `POST /v1/ask` routes a plain-English prompt to social and marketplace operations (profiles, posts, search, and similar). Use web ask when you already have a page URL; use the Ask product when you need a platform lookup routed for you.
Can I search the web before I have a URL?
Yes. `GET /v1/web/search` takes a required `query` and returns ranked organic-style results with title, url, and content snippets. Optional `region` (ISO country code) and `datePosted` filters localize or narrow by recency. Paginate with 1-based `page`. Then extract or ask on the URLs you keep.
How much do Web routes cost?
You buy credit packs (or spend the 100 free signup credits). Amounts for markdown, HTML, ask, and search are shown on each endpoint page linked from this hub; bill from `meta.creditsCharged`. Crawl is 1 credit per URL requested, up to 5 URLs per request (5 credits max). There is no required monthly subscription for pay-as-you-go packs.
Can I crawl an entire website?
No. `GET /v1/web/crawl` is a bounded synchronous batch: repeat the `url` query parameter for up to five public URLs you already have. It does not discover or follow links across a domain. Split larger jobs into multiple crawl calls or fetch URLs one at a time with markdown/html.
What does lookupStatus restricted mean?
On markdown, HTML, and ask, `restricted` means bot or access protection blocked automated fetching. Content fields (`markdown`, `html`, or `answer`) are null. The request can still bill as a completed lookup; read `meta.creditsCharged` and treat restricted as a hard stop for that URL.
Can Web routes read pages behind a login?
No. Only publicly reachable pages without your session cookies. Private dashboards, paywalled articles, and authenticated app screens are out of scope.
Do Web tools work with MCP and agents?
Yes. The same Web operations appear as typed MCP tools beside social tools when you connect Social Fetch MCP with a key that has credits. For production jobs, pin the typed GET paths so routing stays explicit. `POST /v1/ask` can still route social lookups; page Q&A stays on `GET /v1/web/ask`.
Start with a public URL extract
Create an account, spend the 100 free credits on a markdown fetch or a web search, then buy a pack when the JSON fits your RAG or research pipeline. Credits do not expire on pay-as-you-go packs.