> **For coding agents and LLMs:** This is one page from the Social Fetch docs (markdown export). The sections below mirror the orientation block in [`/llms.txt`](https://www.socialfetch.dev/llms.txt); use [`/llms.json`](https://www.socialfetch.dev/llms.json) when you need a structured operation inventory. The catalog covers documented operations with on-site reference pages.

## This page

- **On-site (HTML):** [https://www.socialfetch.dev/docs/api/v1/web/crawl/get](https://www.socialfetch.dev/docs/api/v1/web/crawl/get)
- **Markdown (.mdx) URL:** [https://www.socialfetch.dev/docs/api/v1/web/crawl/get.mdx](https://www.socialfetch.dev/docs/api/v1/web/crawl/get.mdx)

## API base URL and authentication

- **API origin (from OpenAPI `servers`):** `https://api.socialfetch.dev`
- **Authentication:** send `x-api-key: sfk_...` on `/v1/**` routes unless the operation is explicitly anonymous (check OpenAPI `security`, the [API reference hub](https://www.socialfetch.dev/docs/api.mdx), [`/llms.txt`](https://www.socialfetch.dev/llms.txt), or [`/llms.json`](https://www.socialfetch.dev/llms.json) for each route).
- **OpenAPI JSON:** [https://www.socialfetch.dev/openapi.json](https://www.socialfetch.dev/openapi.json)

## Recommended docs entrypoints (this site)

- [Documentation overview](https://www.socialfetch.dev/docs.mdx) — top-level orientation (markdown).
- [Quickstart](https://www.socialfetch.dev/docs/quickstart.mdx) — authenticate with `x-api-key`, validate auth with `whoami`, and understand the JSON envelope.
- [SDK](https://www.socialfetch.dev/docs/sdk.mdx) — official TypeScript SDK guide, including `SocialFetchClient`, `Result`, and `unwrap()`.
- [SDK reference](https://www.socialfetch.dev/docs/sdk-reference.mdx) — exhaustive SDK method inventory and route mapping for agents, tooling, and power users.
- [Choose the right endpoint](https://www.socialfetch.dev/docs/choose-endpoint.mdx) — task-oriented route selection for smoke tests, profiles, list endpoints, and single-item lookups.
- [Capability matrix](https://www.socialfetch.dev/docs/capability-matrix.mdx) — fast comparison of identifiers, pagination, outcomes, media download, and SDK coverage.
- [`/llms.json`](https://www.socialfetch.dev/llms.json) — structured machine-readable operation inventory with parameter names, pagination, outcomes, credits, and SDK mapping.
- [API reference hub](https://www.socialfetch.dev/docs/api.mdx) — human-friendly index of operations with links into generated pages.
- [Errors](https://www.socialfetch.dev/docs/errors.mdx) — shared error envelope and HTTP status guidance.
- [Credits](https://www.socialfetch.dev/docs/credits.mdx) — metering, `402`, and planning batch jobs.
- Outcome semantics such as `found`, `not_found`, and `private` are documented in [Errors](https://www.socialfetch.dev/docs/errors.mdx) and on operation pages when present in the OpenAPI contract.

## Markdown docs convention

- Every docs page has a markdown twin: append **`.mdx`** to the docs pathname (for example `/docs/quickstart` → `/docs/quickstart.mdx`).
- Agents that send `Accept: text/markdown` on `/docs/**` HTML URLs may receive markdown directly (same URL, `Vary: Accept`).

---
# Crawl web pages (https://www.socialfetch.dev/docs/api/v1/web/crawl/get)

## Summary

Crawl a small set of web pages synchronously.

**Tags:** `Web`

## HTTP

- **Method:** GET
- **Path:** `/v1/web/crawl`
- **Base URL:** `https://api.socialfetch.dev`

## Capability summary

- **SDK mapping:** `client.web.crawl({ urls: ["https://www.socialfetch.dev/"] })`
- **Accepted identifiers:** `url` (query)
- **Pagination:** none

## Authentication

- **`x-api-key`**: API key (`sfk_...`)

## Parameters

### `url` (query)

- **Required:** yes
- **Constraints:** type `array`
- **Description:** URLs to crawl. Repeat the `url` query parameter for multiple pages (max 5).

## Responses (status codes)

- **200**: Crawl results.
- **400**: Invalid query parameters or disallowed URL
- **401**: Missing or invalid API key
- **402**: Insufficient credits
- **500**: Unexpected or billing error
- **502**: Crawl could not be completed from the upstream response.
- **503**: Service temporarily unavailable; safe to retry with backoff.

## Response body (200)

Crawl results.

### Field outline

- **data** (required) — type `object`. Endpoint-specific response payload.
  - **results** (required) — type `array`. Per-URL crawl results.
    - _items:_
      - **url** (required) — type `string`. Final URL associated with this crawl result.
      - **status** (required) — type `integer`. HTTP status code reported for the page fetch.
      - **success** (required) — type `boolean`. Whether the page was crawled successfully.
      - **markdown** (optional) — type `object`. Markdown extracted from the page when available.
        - **raw** (optional) — type `string`. Raw markdown for the crawled page.
        - **fit** (optional) — type `string`. Filtered markdown for the crawled page.
      - **html** (optional) — type `string`. HTML content for the page when returned by the crawler.
      - **metadata** (optional) — type `object`. Page metadata such as title or description when available.
      - **errorMessage** (optional) — type `string`. Provider error message when the page crawl failed.
  - **summary** (required) — type `object`. Summary counts for the crawl batch.
    - **requestedUrls** (required) — type `integer`; minimum: 0. Number of URLs requested in the crawl batch.
    - **succeeded** (required) — type `integer`; minimum: 0. Number of URLs that crawled successfully.
    - **failed** (required) — type `integer`; minimum: 0. Number of URLs that failed to crawl.
- **meta** (required) — type `object`. Metadata describing the request and billing outcome.
  - **requestId** (required) — type `string`; minLength: 1. Unique request identifier for tracing this API call.
  - **creditsCharged** (required) — type `integer`; minimum: 0. Credits charged for this request.
  - **version** (required) — type `string`; enum: v1. Public API version that served the response.

### Example JSON (OpenAPI example)

```json
{
  "data": {
    "results": [
      {
        "url": "https://www.socialfetch.dev/",
        "status": 200,
        "success": true,
        "markdown": {
          "raw": "# Social media scraper API for every major platform.",
          "fit": "Social media scraper API for every major platform."
        }
      }
    ],
    "summary": {
      "requestedUrls": 1,
      "succeeded": 1,
      "failed": 0
    }
  },
  "meta": {
    "requestId": "req_01example",
    "creditsCharged": 1,
    "version": "v1"
  }
}
```

### Machine-readable error codes

When an error JSON body is returned, it may include one of these `error.code` values (derived from the OpenAPI schemas for this operation; additional codes may exist at runtime):

- `bad_request`

## Error handling & retries

Interpret HTTP status codes using the descriptions below. Do not assume a JSON body unless the OpenAPI schema defines one for that status.

- **400**: Invalid query parameters or disallowed URL **Retry:** Fix the request; retrying the same invalid payload will not help.
- **401**: Missing or invalid API key **Retry:** Fix the API key first; retrying without changes will not help.
- **402**: Insufficient credits **Retry:** Do not retry without resolving billing/credits (retrying the same request will not help).
- **500**: Unexpected or billing error
- **502**: Crawl could not be completed from the upstream response. **Retry:** May be transient; a few retries with backoff are reasonable.
- **503**: Service temporarily unavailable; safe to retry with backoff. **Retry:** Usually safe to retry with exponential backoff and jitter.

### Suggested client defaults

- Send the API key using the `x-api-key` header on every request.
- On `503` (and sometimes `502`), retry with backoff; cap retries and surface a clear error to the user.
- On `402`, surface an actionable billing message rather than blind retries.

## Examples

### TypeScript SDK

```typescript
import { SocialFetchClient } from "@socialfetch/sdk";

const client = new SocialFetchClient({
  apiKey: process.env.SOCIALFETCH_API_KEY!,
});

const result = await client.web.crawl({
  url: "value",
});

if (!result.ok) {
  console.error(result.error);
} else {
  console.log(result.value.data);
}
```

### Node.js

```javascript
const response = await fetch(
  "https://api.socialfetch.dev/v1/web/crawl?url=value",
  {
    headers: {
      "x-api-key": "YOUR_API_KEY",
    }
  }
);

const data = await response.json();
console.log(data);
```

### cURL

```bash
curl "https://api.socialfetch.dev/v1/web/crawl?url=value" \
  -H "x-api-key: YOUR_API_KEY"
```

### Python

```python
import requests

response = requests.get(
    "https://api.socialfetch.dev/v1/web/crawl?url=value",
    headers={"x-api-key": "YOUR_API_KEY"},
)
data = response.json()
print(data)
```