> **For coding agents and LLMs:** This is one page from the Social Fetch docs (markdown export). For curated orientation and workflow guidance, start with [`/llms.txt`](https://www.socialfetch.dev/llms.txt); for agent onboarding and crawl rules, use [`/agents.txt`](https://www.socialfetch.dev/agents.txt); for the full endpoint list with links to pages like this one, use [`/llms-endpoints.txt`](https://www.socialfetch.dev/llms-endpoints.txt); for one platform's parameters and curls, use [`/llms-{platform}.txt`](https://www.socialfetch.dev/llms-tiktok.txt); use [`/llms.json`](https://www.socialfetch.dev/llms.json) when you need structured JSON for tool registration. ## This page - **On-site (HTML):** - **Markdown (.mdx) URL:** ## API base URL and authentication - **API origin (from OpenAPI `servers`):** `https://api.socialfetch.dev` - **Authentication:** send `x-api-key: sfk_...` on `/v1/**` and `/v2/**` routes unless the operation is explicitly anonymous (check OpenAPI `security`, the [API reference hub](https://www.socialfetch.dev/docs/api.mdx), [`/llms.txt`](https://www.socialfetch.dev/llms.txt), or [`/llms.json`](https://www.socialfetch.dev/llms.json) for each route). - **OpenAPI JSON:** ## Recommended docs entrypoints (this site) - [Documentation overview](https://www.socialfetch.dev/docs.mdx) — top-level orientation (markdown). - [Quickstart](https://www.socialfetch.dev/docs/quickstart.mdx) — authenticate with `x-api-key`, validate auth with `whoami`, and understand the JSON envelope. - [SDK](https://www.socialfetch.dev/docs/sdk.mdx) — official TypeScript SDK guide, including `SocialFetchClient`, `Result`, and `unwrap()`. - [Capability matrix](https://www.socialfetch.dev/docs/capability-matrix.mdx) — every operation with identifiers, pagination, outcomes, media download, credits, and its SDK method. Generated from OpenAPI, so use it for route selection instead of scanning individual pages. - [Recipes](https://www.socialfetch.dev/docs/recipes.mdx) — copyable workflows (brand monitoring, transcripts, Ad Library, creator scoring, Reddit research) with credit callouts and SDK examples. - [Integrations](https://www.socialfetch.dev/docs/integrations.mdx) — MCP for AI clients, n8n verified node, Apify Store Actors, Make custom app, SDK, and REST API connection paths. - [MCP product page](https://www.socialfetch.dev/mcp) — hosted MCP overview, OAuth, Skills install. - [MCP integration](https://www.socialfetch.dev/docs/integrations/mcp.mdx) — hosted `/mcp` server, OAuth, Cursor/VS Code/Claude install snippets, 910 endpoint tools, plus docs_search/docs_read for implementation help. - [n8n integration](https://www.socialfetch.dev/docs/integrations/n8n.mdx) — install `n8n-nodes-socialfetch`, credentials, and workflow examples. - [Apify integration](https://www.socialfetch.dev/docs/integrations/apify.mdx) — Store Actors under @social-fetch, PPE billing, dataset export, and quick start. - [Make integration](https://www.socialfetch.dev/docs/integrations/make.mdx) — custom app modules for Make scenarios, API key credentials, and module catalog. - [`/llms-endpoints.txt`](https://www.socialfetch.dev/llms-endpoints.txt) — every documented operation with a direct link to that route's agent-readable markdown page (prefer this over parsing OpenAPI). - [`/llms-{platform}.txt`](https://www.socialfetch.dev/llms-tiktok.txt) — per-platform endpoint files generated from OpenAPI (parameters, credits, curls). - [`/agents.txt`](https://www.socialfetch.dev/agents.txt) — agent crawl/onboarding file with capabilities, auth rules, and allowlist. - [`/llms.json`](https://www.socialfetch.dev/llms.json) — structured machine-readable operation inventory with parameter names, pagination, outcomes, credits, and SDK mapping. - [API reference hub](https://www.socialfetch.dev/docs/api.mdx) — human-friendly index of operations with links into generated pages. - [Errors](https://www.socialfetch.dev/docs/errors.mdx) — shared error envelope and HTTP status guidance. - [Credits](https://www.socialfetch.dev/docs/credits.mdx) — metering, `402`, and planning batch jobs. - Outcome semantics such as `found`, `not_found`, and `private` are documented in [Errors](https://www.socialfetch.dev/docs/errors.mdx) and on operation pages when present in the OpenAPI contract. ## Markdown docs convention - Every docs page has a markdown twin: append **`.mdx`** to the docs pathname (for example `/docs/quickstart` → `/docs/quickstart.mdx`). - Agents that send `Accept: text/markdown` on `/docs/**` HTML URLs may receive markdown directly (same URL, `Vary: Accept`). - Published blog posts use the same convention: `/blog/{slug}` → `/blog/{slug}.mdx`, or `Accept: text/markdown` on the HTML URL (`Vary: Accept`). --- # Capture a web page's network responses (https://www.socialfetch.dev/docs/api/v1/web/network/get) ## Summary Load a web page and return the JSON and other responses its own scripts fetch while it loads. **Tags:** `Web` ## HTTP - **Method:** GET - **Path:** `/v1/web/network` - **Base URL:** `https://api.socialfetch.dev` ## Capability summary - **SDK mapping:** `client.web.captureNetwork({ url: "https://www.socialfetch.dev/pricing", match: "/api/" })` - **Accepted identifiers:** `url` (query) - **Pagination:** none - **Business outcome field:** `data.lookupStatus` with values `found`, `restricted` ## Agent notes - **Disambiguation:** Use when the data a page shows is loaded by its own JavaScript (prices, listings, search results) and is missing from `/v1/web/html`. Only responses the page requests itself are returned; nothing is replayed. - **Empty results:** An empty `requests` list with `lookupStatus: found` means the page made no fetch or XHR request containing `match`: check the substring, or raise `waitMs` for slow pages. `lookupStatus: restricted` means bot protection blocked the page on every allowed proxy tier (don't set `proxy=datacenter` on protected sites). `budgetExceeded: true` means the page needed more data than `maxCredits` allows: retry with a higher `maxCredits`. ## Credits - **Base:** 1 credit per successful lookup. - **Add-on (`maxCredits`):** +24 credits. _Lets a heavy page use more data, up to the cap you set (at most 25). Charged for what the page uses._ - **Maximum on success (200):** 25 credits. - **Failed requests (`lookup_failed`, `503`):** not charged. - **Authoritative field:** `meta.creditsCharged`. ## Authentication - **`x-api-key`**: API key (`sfk_...`) ## Parameters ### `url` (query) - **Required:** yes - **Constraints:** type `string`; minLength: 1; maxLength: 2083 - **Description:** Web page URL to load. ### `match` (query) - **Required:** yes - **Constraints:** type `string`; minLength: 1; maxLength: 200 - **Description:** Part of the request URL to keep, for example `/api/` or `/graphql`. Only fetch and XHR requests whose URL contains it are returned. ### `waitMs` (query) - **Required:** no - **Constraints:** type `integer`; minimum: 0; maximum: 15000 - **Description:** How long to keep listening after the page loads, in milliseconds (0–15000). Default 4000. ### `proxy` (query) - **Required:** no - **Constraints:** type `string`; enum: datacenter, residential, mobile - **Description:** Most expensive proxy tier to use. The page is tried on `datacenter` first and moves up only if it is blocked, stopping at this tier. Default: every tier (`mobile`). ### `maxCredits` (query) - **Required:** no - **Constraints:** type `integer`; minimum: 1; maximum: 25 - **Description:** Most credits this call may cost (1–25). Default 1, which loads about 1.6 MB on the datacenter proxy or 190 KB on residential or mobile. Raise it for heavy pages; you're charged only for what the page uses. ## Responses (status codes) - **200**: Matching fetch and XHR responses. - **400**: Invalid query parameters or disallowed URL - **401**: Missing or invalid API key - **402**: Insufficient credits - **500**: Unexpected or billing error - **502**: Capture could not be completed. - **503**: Service temporarily unavailable; safe to retry with backoff. ## Response body (200) Matching fetch and XHR responses. ### Field outline - **data** (required) — type `object`. Endpoint-specific response payload. - **lookupStatus** (required) — type `string`; enum: found, restricted. Whether page content could be extracted. Restricted means bot protection or similar access controls blocked automated fetching. - **url** (required) — type `string`. URL of the page after redirects. - **status** (required) — type `integer`; nullable. HTTP status of the page, or null when unknown. - **proxy** (required) — type `string`; enum: datacenter, residential, mobile. Proxy tier the page was loaded on: the tier that served it, or the last one tried when it was blocked. - **budgetExceeded** (required) — type `boolean`. Whether the page needed more data than `maxCredits` allows. Loading stopped early, so `requests` may be incomplete. - **bytesTransferred** (required) — type `integer`; minimum: 0. Data the page loaded through the proxy, in bytes, across every tier tried. - **requests** (required) — type `array`. Matching fetch and XHR requests, in the order their responses arrived (at most 20). Empty when nothing matched or the page was restricted. - _items:_ - **url** (required) — type `string`. Request URL. - **method** (required) — type `string`. HTTP method, for example `GET` or `POST`. - **status** (required) — type `integer`. HTTP status of the response. - **contentType** (required) — type `string`; nullable. Response `Content-Type` header, or null when absent. - **body** (optional) — nullable. Response body: parsed JSON when the response is JSON, otherwise text. Null when the body is binary, unavailable or not kept. - **bodyBytes** (required) — type `integer`; minimum: 0. Size of the full response body in bytes. - **truncated** (required) — type `boolean`. Whether `body` was cut short. Bodies over 512 KB, or past 2 MB across one call, are cut; a cut JSON body is returned as text. - **meta** (required) — type `object`. Metadata describing the request and billing outcome. - **requestId** (required) — type `string`; minLength: 1. Unique request identifier for tracing this API call. - **creditsCharged** (required) — type `integer`; minimum: 0. Credits charged for this request. - **version** (required) — type `string`; enum: v1. Public API version that served the response. ### Example JSON (found) ```json { "data": { "lookupStatus": "found", "url": "https://shop.example.com/products/coffee-beans", "status": 200, "proxy": "datacenter", "budgetExceeded": false, "bytesTransferred": 88271, "requests": [ { "url": "https://shop.example.com/api/prices", "method": "POST", "status": 200, "contentType": "application/json", "body": { "presets": [ 250 ], "currency": "USD", "prices": { "250": { "price": 3, "discount": 0 }, "500": { "price": 6, "discount": 0 }, "1000": { "price": 11.99, "discount": 0 } } }, "bodyBytes": 152, "truncated": false } ] }, "meta": { "requestId": "req_01example", "creditsCharged": 1, "version": "v1" } } ``` ### Machine-readable error codes When an error JSON body is returned, it may include one of these `error.code` values (derived from the OpenAPI schemas for this operation; additional codes may exist at runtime): - `bad_request` ## Error handling & retries Interpret HTTP status codes using the descriptions below. Do not assume a JSON body unless the OpenAPI schema defines one for that status. - **400**: Invalid query parameters or disallowed URL **Retry:** Fix the request; retrying the same invalid payload will not help. - **401**: Missing or invalid API key **Retry:** Fix the API key first; retrying without changes will not help. - **402**: Insufficient credits **Retry:** Do not retry without resolving billing/credits (retrying the same request will not help). - **500**: Unexpected or billing error - **502**: Capture could not be completed. **Retry:** May be transient; a few retries with backoff are reasonable. - **503**: Service temporarily unavailable; safe to retry with backoff. **Retry:** Usually safe to retry with exponential backoff and jitter. ### Suggested client defaults - Send the API key using the `x-api-key` header on every request. - On `503` (and sometimes `502`), retry with backoff; cap retries and surface a clear error to the user. - On `402`, surface an actionable billing message rather than blind retries. ## Examples ### TypeScript SDK ```typescript import { SocialFetchClient } from "@socialfetch/sdk"; const client = new SocialFetchClient({ apiKey: process.env.SOCIALFETCH_API_KEY!, }); const result = await client.web.captureNetwork({ url: "https://www.socialfetch.dev", match: "/api/", }); if (!result.ok) { console.error(result.error); } else { console.log(result.value.data); } ``` ### Node.js ```javascript const response = await fetch( "https://api.socialfetch.dev/v1/web/network?url=https://www.socialfetch.dev&match=/api/", { headers: { "x-api-key": "YOUR_API_KEY" } } ); const data = await response.json(); console.log(data); ``` ### cURL ```bash curl "https://api.socialfetch.dev/v1/web/network?url=https://www.socialfetch.dev&match=/api/" \ -H "x-api-key: YOUR_API_KEY" ``` ### Python ```python import requests response = requests.get( "https://api.socialfetch.dev/v1/web/network?url=https://www.socialfetch.dev&match=/api/", headers={"x-api-key": "YOUR_API_KEY"}, ) data = response.json() print(data) ```