Web scraping API: markdown & HTML for LLMs

Teams already using Social Fetch kept hitting the same wall: the pipeline does not stay inside social platforms.

You pull a TikTok profile and then need the Linktree in the bio. You monitor a competitor on X and then need the pricing page they linked. A research agent spends half its context on blog posts and docs, not Instagram posts.

Every time, the fix was a second stack: a Puppeteer container to babysit, a generic scraping API with different auth and billing, or a /utils/scrape.ts that broke when a site moved a CSS class.

Same API key, same envelope

Pass a public URL, get structured content back. Same x-api-key, same { data, meta } response shape, same credit wallet as TikTok and Instagram lookups.

The problem with traditional web scraping

Social Fetch already covered the social side. The painful part was everything linked from it: a creator's Shopify store, a competitor's pricing page, docs your agent needs to read, press releases attached to viral posts.

Handling those URLs meant a second vendor, a second billing model, and usually headless browsers you did not want to operate. One integration path for "I need the text from this URL" is simpler than evaluating another scraping product every time a bio link shows up.

How the web extraction API works

Four routes live under /v1/web/*, using the credentials and JSON envelope you already have:

Extract to Markdown: page text with optional fit, raw, or bm25 filters
Extract to HTML: sanitized HTML when downstream code expects markup
Ask a Web Page: pass a URL and a question, get a grounded answer
Crawl Multiple URLs: up to five URLs in one synchronous request, with per-page rows

Each route is documented in the API reference. Pricing per operation is on that page; confirm what actually billed with meta.creditsCharged on every response.

typescript

// Extract clean markdown for RAG pipelines const page = await client.web.getMarkdown({ url: "https://competitor.com/pricing", filter: "fit", }); if (page.ok) { console.log(page.value.data.markdown?.fit); console.log(page.value.data.lookupStatus, page.value.meta.creditsCharged); } // Ask a direct question of a web page const answer = await client.web.ask({ url: "https://competitor.com/pricing", q: "What is the cheapest plan?", }); if (answer.ok) { console.log(answer.value.data.answer); console.log(answer.value.meta.creditsCharged); }

Markdown, HTML, and ask responses include data.lookupStatus. On found, content fields are populated. On restricted, the page hit bot or access protection and content fields come back null. That is still HTTP 200, not a retry signal. See Errors.

Markdown extraction filters

fit (default): readability-oriented text, nav stripped out. raw: fuller DOM-to-markdown conversion. bm25: passages ranked against a query you pass (required when filter=bm25).

Crawl takes one or more url query parameters (repeat the param for each page, max five). You get data.results with a row per URL plus data.summary counts. It does not follow links or spider a domain. Only the URLs you pass.

Billing on completed lookups

Credits are charged when we complete a lookup attempt, same as profile and post routes. Read Credits for the full model.

For markdown, HTML, and ask:

Outcome	Charged?
Pre-send validation error (malformed URL, missing `q`)	No
`lookup_failed` / HTTP `503`	No
HTTP `200` + `lookupStatus: "found"`	Yes
HTTP `200` + `lookupStatus: "restricted"`	Yes (lookup completed, page blocked)

restricted is not empty markdown you can ignore. Treat it as a hard stop for that URL. Do not retry it expecting a different outcome.

Crawl bills 1 credit per URL requested (up to five per call). Check each row's success flag and data.summary. A batch can partially succeed. meta.creditsCharged is the billing source of truth, not your own URL count.

Retry lookup_failed and HTTP 503 with backoff. Fix validation errors instead of retrying them.

Use cases for LLM pipelines

For RAG, GET /v1/web/markdown with filter=fit strips navigation chrome and returns text you can chunk and embed. If you only need passages about one topic, filter=bm25 with a query returns ranked sections from a single page without building a search index first.

Product teams use crawl to fetch a pricing page, a features page, and an about page in one call. Three URLs requested means three credits and three markdown blobs you can diff weekly.

The ask route takes a URL and a natural-language question and returns an answer grounded in what is on the page: return policies, plan tiers, feature lists. You can chain it beside social calls: pull a profile, read a bio URL, fetch the linked page with the same SDK client.

To discover candidate URLs by keyword first, use GET /v1/web/search. Search finds pages; markdown, HTML, ask, and crawl extract them.

What this API will not do

Public pages only. If a human needs a login, session token, or cookie to see the page, this API will not see it either.

Bot-protected sites return lookupStatus: "restricted" on markdown, HTML, and ask. That is not a transient error you should hammer with retries.

Live fetch, not a long-lived cache. Expect seconds, not milliseconds. You get the page as it exists when the request runs.

Crawl is bounded: five URLs max, synchronous, no link following. Built for a short list you already have, not mirroring an entire site.

Ask reads what is publicly visible on the URL. It is not open-ended chat. It answers from page content.