> **For coding agents and LLMs:** This is one published Social Fetch blog post (markdown export). Product docs and API orientation live in [`/llms.txt`](https://www.socialfetch.dev/llms.txt). The HTML article is at the on-site URL below.

## This page

- **On-site (HTML):** <https://www.socialfetch.dev/blog/web-extraction-api>
- **Markdown (.mdx) URL:** <https://www.socialfetch.dev/blog/web-extraction-api.mdx>
- **Blog:** <https://www.socialfetch.dev/blog>

---

# Web scraping API: markdown & HTML for LLMs

Teams already using Social Fetch kept hitting the same wall: the pipeline does not stay inside social platforms.

You pull a TikTok profile and then need the Linktree in the bio. You monitor a competitor on X and then need the pricing page they linked. A research agent spends half its context on blog posts and docs, not Instagram posts.

Every time, the fix was a second stack: a Puppeteer container to babysit, a generic scraping API with different auth and billing, or a `/utils/scrape.ts` that broke when a site moved a CSS class.

> **Same API key, same envelope**
>
> Pass a public URL, get structured content back. Same `x-api-key`, same `{ data, meta }` response shape, same credit wallet as TikTok and Instagram lookups.

## The problem with traditional web scraping

Social Fetch already covered the social side. The painful part was everything linked from it: a creator's Shopify store, a competitor's pricing page, docs your agent needs to read, press releases attached to viral posts.

Handling those URLs meant a second vendor, a second billing model, and usually headless browsers you did not want to operate. One integration path for "I need the text from this URL" is simpler than evaluating another scraping product every time a bio link shows up.

## How the web extraction API works

Four routes live under `/v1/web/\*`, using the credentials and JSON envelope you already have:

- [Extract to Markdown](/docs/api/v1/web/markdown/get): page text with optional `fit`, `raw`, or `bm25` filters
- [Extract to HTML](/docs/api/v1/web/html/get): sanitized HTML when downstream code expects markup
- [Ask a Web Page](/docs/api/v1/web/ask/get): pass a URL and a question, get a grounded answer
- [Crawl Multiple URLs](/docs/api/v1/web/crawl/get): up to five URLs in one synchronous request, with per-page rows

Each route is documented in the [API reference](/docs/api). Pricing per operation is on that page; confirm what actually billed with `meta.creditsCharged` on every response.

```ts
// Extract clean markdown for RAG pipelines
const page = await client.web.getMarkdown({
  url: "https://competitor.com/pricing",
  filter: "fit",
});

if (page.ok) {
  console.log(page.value.data.markdown?.fit);
  console.log(page.value.data.lookupStatus, page.value.meta.creditsCharged);
}

// Ask a direct question of a web page
const answer = await client.web.ask({
  url: "https://competitor.com/pricing",
  q: "What is the cheapest plan?",
});

if (answer.ok) {
  console.log(answer.value.data.answer);
  console.log(answer.value.meta.creditsCharged);
}
```

Markdown, HTML, and ask responses include `data.lookupStatus`. On `found`, content fields are populated. On `restricted`, the page hit bot or access protection and content fields come back null. That is still HTTP `200`, not a retry signal. See [Errors](/docs/errors).

> **Markdown extraction filters**
>
> `fit` (default): readability-oriented text, nav stripped out.
> `raw`: fuller DOM-to-markdown conversion.
> `bm25`: passages ranked against a `query` you pass (required when `filter=bm25`).

Crawl takes one or more `url` query parameters (repeat the param for each page, max five). You get `data.results` with a row per URL plus `data.summary` counts. It does not follow links or spider a domain. Only the URLs you pass.

## Billing on completed lookups

Credits are charged when we complete a lookup attempt, same as profile and post routes. Read [Credits](/docs/credits) for the full model.

For markdown, HTML, and ask:

| Outcome | Charged? |
| --- | --- |
| Pre-send validation error (malformed URL, missing `q`) | No |
| `lookup_failed` / HTTP`503` | No |
| HTTP `200` +`lookupStatus: "found"` | Yes |
| HTTP `200` +`lookupStatus: "restricted"` | Yes (lookup completed, page blocked) |

`restricted` is not empty markdown you can ignore. Treat it as a hard stop for that URL. Do not retry it expecting a different outcome.

Crawl bills 1 credit per URL requested (up to five per call). Check each row's `success` flag and `data.summary`. A batch can partially succeed. `meta.creditsCharged` is the billing source of truth, not your own URL count.

Retry `lookup_failed` and HTTP `503` with backoff. Fix validation errors instead of retrying them.

## Use cases for LLM pipelines

For RAG, `GET /v1/web/markdown` with `filter=fit` strips navigation chrome and returns text you can chunk and embed. If you only need passages about one topic, `filter=bm25` with a `query` returns ranked sections from a single page without building a search index first.

Product teams use crawl to fetch a pricing page, a features page, and an about page in one call. Three URLs requested means three credits and three markdown blobs you can diff weekly.

The ask route takes a URL and a natural-language question and returns an answer grounded in what is on the page: return policies, plan tiers, feature lists. You can chain it beside social calls: pull a profile, read a bio URL, fetch the linked page with the same SDK client.

To discover candidate URLs by keyword first, use `[GET /v1/web/search](/docs/api/v1/web/search/get)`. Search finds pages; markdown, HTML, ask, and crawl extract them.

## What this API will not do

Public pages only. If a human needs a login, session token, or cookie to see the page, this API will not see it either.

Bot-protected sites return `lookupStatus: "restricted"` on markdown, HTML, and ask. That is not a transient error you should hammer with retries.

Live fetch, not a long-lived cache. Expect seconds, not milliseconds. You get the page as it exists when the request runs.

Crawl is bounded: five URLs max, synchronous, no link following. Built for a short list you already have, not mirroring an entire site.

Ask reads what is publicly visible on the URL. It is not open-ended chat. It answers from page content.

## Get started

If you already have a Social Fetch API key, these routes are live. No opt-in or waitlist.

Open the [playground](https://app.socialfetch.dev/playground) under **Web**, paste a URL, and inspect the response. Wire it into your backend with `client.web.\*` from the [TypeScript SDK](/docs/sdk). Estimate volume on [Pricing](/pricing). Flat credits per operation, same wallet as social lookups.

Quickstart Guide Auth, first request, and understanding the response envelope. /docs/quickstart
API Reference Parameters and response shapes for all Web routes. /docs/api

[Quickstart Guide](/docs/quickstart)
[API Reference](/docs/api)

Social Fetch covers 23 platforms in one integration. Web extraction applies the same API key and JSON discipline to the URLs outside those networks: pages linked from bios, tweets, and posts that the rest of your pipeline still has to read.