> **For coding agents and LLMs:** This is one page from the Social Fetch docs (markdown export). For curated orientation and workflow guidance, start with [`/llms.txt`](https://www.socialfetch.dev/llms.txt); for agent onboarding and crawl rules, use [`/agents.txt`](https://www.socialfetch.dev/agents.txt); for the full endpoint list with links to pages like this one, use [`/llms-endpoints.txt`](https://www.socialfetch.dev/llms-endpoints.txt); for one platform's parameters and curls, use [`/llms-{platform}.txt`](https://www.socialfetch.dev/llms-tiktok.txt); use [`/llms.json`](https://www.socialfetch.dev/llms.json) when you need structured JSON for tool registration.

## This page

- **On-site (HTML):** <https://www.socialfetch.dev/docs/api/v1/web/map/get>
- **Markdown (.mdx) URL:** <https://www.socialfetch.dev/docs/api/v1/web/map/get.mdx>

## API base URL and authentication

- **API origin (from OpenAPI `servers`):** `https://api.socialfetch.dev`
- **Authentication:** send `x-api-key: sfk_...` on `/v1/**` and `/v2/**` routes unless the operation is explicitly anonymous (check OpenAPI `security`, the [API reference hub](https://www.socialfetch.dev/docs/api.mdx), [`/llms.txt`](https://www.socialfetch.dev/llms.txt), or [`/llms.json`](https://www.socialfetch.dev/llms.json) for each route).
- **OpenAPI JSON:** <https://www.socialfetch.dev/openapi.json>

## Recommended docs entrypoints (this site)

- [Documentation overview](https://www.socialfetch.dev/docs.mdx) — top-level orientation (markdown).
- [Quickstart](https://www.socialfetch.dev/docs/quickstart.mdx) — authenticate with `x-api-key`, validate auth with `whoami`, and understand the JSON envelope.
- [SDK](https://www.socialfetch.dev/docs/sdk.mdx) — official TypeScript SDK guide, including `SocialFetchClient`, `Result`, and `unwrap()`.
- [Capability matrix](https://www.socialfetch.dev/docs/capability-matrix.mdx) — every operation with identifiers, pagination, outcomes, media download, credits, and its SDK method. Generated from OpenAPI, so use it for route selection instead of scanning individual pages.
- [Recipes](https://www.socialfetch.dev/docs/recipes.mdx) — copyable workflows (brand monitoring, transcripts, Ad Library, creator scoring, Reddit research) with credit callouts and SDK examples.
- [Integrations](https://www.socialfetch.dev/docs/integrations.mdx) — MCP for AI clients, n8n verified node, Apify Store Actors, Make custom app, SDK, and REST API connection paths.
- [MCP product page](https://www.socialfetch.dev/mcp) — hosted MCP overview, OAuth, Skills install.
- [MCP integration](https://www.socialfetch.dev/docs/integrations/mcp.mdx) — hosted `/mcp` server, OAuth, Cursor/VS Code/Claude install snippets, 1112 endpoint tools, plus docs_search/docs_read for implementation help.
- [n8n integration](https://www.socialfetch.dev/docs/integrations/n8n.mdx) — install `n8n-nodes-socialfetch`, credentials, and workflow examples.
- [Apify integration](https://www.socialfetch.dev/docs/integrations/apify.mdx) — Store Actors under @social-fetch, PPE billing, dataset export, and quick start.
- [Make integration](https://www.socialfetch.dev/docs/integrations/make.mdx) — custom app modules for Make scenarios, API key credentials, and module catalog.
- [`/llms-endpoints.txt`](https://www.socialfetch.dev/llms-endpoints.txt) — every documented operation with a direct link to that route's agent-readable markdown page (prefer this over parsing OpenAPI).
- [`/llms-{platform}.txt`](https://www.socialfetch.dev/llms-tiktok.txt) — per-platform endpoint files generated from OpenAPI (parameters, credits, curls).
- [`/agents.txt`](https://www.socialfetch.dev/agents.txt) — agent crawl/onboarding file with capabilities, auth rules, and allowlist.
- [`/llms.json`](https://www.socialfetch.dev/llms.json) — structured machine-readable operation inventory with parameter names, pagination, outcomes, credits, and SDK mapping.
- [API reference hub](https://www.socialfetch.dev/docs/api.mdx) — human-friendly index of operations with links into generated pages.
- [Errors](https://www.socialfetch.dev/docs/errors.mdx) — shared error envelope and HTTP status guidance.
- [Credits](https://www.socialfetch.dev/docs/credits.mdx) — metering, `402`, and planning batch jobs.
- Outcome semantics such as `found`, `not_found`, and `private` are documented in [Errors](https://www.socialfetch.dev/docs/errors.mdx) and on operation pages when present in the OpenAPI contract.

## Markdown docs convention

- Every docs page has a markdown twin: append **`.mdx`** to the docs pathname (for example `/docs/quickstart` → `/docs/quickstart.mdx`).
- Agents that send `Accept: text/markdown` on `/docs/**` HTML URLs may receive markdown directly (same URL, `Vary: Accept`).
- Published blog posts use the same convention: `/blog/{slug}` → `/blog/{slug}.mdx`, or `Accept: text/markdown` on the HTML URL (`Vary: Accept`).

---
# List website URLs (https://www.socialfetch.dev/docs/api/v1/web/map/get)

## Summary

List a website's URLs from its sitemaps, or from the page's own links when it has none.

**Tags:** `Web`

## HTTP

- **Method:** GET
- **Path:** `/v1/web/map`
- **Base URL:** `https://api.socialfetch.dev`

## Capability summary

- **SDK mapping:** `client.web.map({ url: "https://www.socialfetch.dev/" })`
- **Accepted identifiers:** `url` (query), `query` (query)
- **Pagination:** none
- **Business outcome field:** `data.lookupStatus` with values `found`, `restricted`, `budget_exceeded`

## Agent notes

- **Disambiguation:** Lists URLs from the site's sitemaps (found through robots.txt, then `/sitemap.xml`), following sitemap indexes. A site without a sitemap is mapped from the links on the page itself (`source: links`). It doesn't load the pages: pass the URLs to `GET /v1/web/markdown` or `GET /v1/web/crawl` for their content.
- **Empty results:** `source: none` with no URLs means neither the sitemaps nor the page listed any. `truncated: true` means more URLs exist than were returned: raise `limit`, or `maxCredits` when `budgetExceeded` is true. `lookupStatus: restricted` means bot protection blocked the site on every proxy tier allowed.

## Credits

- **Base:** 1 credit per successful lookup.
- **Add-on (`maxCredits`):** +24 credits. _Lets a heavy page use more data, up to the cap you set (at most 25). Charged for what the page uses._
- **Maximum on success (200):** 25 credits.
- **Failed requests (`lookup_failed`, `503`):** not charged.
- **Authoritative field:** `meta.creditsCharged`.

## Authentication

- **`x-api-key`**: API key (`sfk_...`)

## Parameters

### `url` (query)

- **Required:** yes
- **Constraints:** type `string`; minLength: 1; maxLength: 2083
- **Description:** Site to map: its home page, any page on it, or a sitemap URL such as `https://www.socialfetch.dev/sitemap.xml`.

### `limit` (query)

- **Required:** no
- **Constraints:** type `integer`; minimum: 1; maximum: 5000
- **Description:** Most URLs to return (1–5000). Default 1000.

### `query` (query)

- **Required:** no
- **Constraints:** type `string`; minLength: 1; maxLength: 200
- **Description:** Only return URLs containing this text (case-insensitive), for example `/blog/`.

### `sitemap` (query)

- **Required:** no
- **Constraints:** type `string`; enum: include, only, skip
- **Description:** How to find URLs: `include` reads the sitemaps and falls back to the page's links when they list nothing, `only` reads the sitemaps alone, `skip` reads only the page's links. Default `include`.

### `includeSubdomains` (query)

- **Required:** no
- **Constraints:** type `boolean`
- **Description:** Whether to keep URLs on subdomains of the site, such as `docs.` or `blog.`. Default true.

### `ignoreQueryParameters` (query)

- **Required:** no
- **Constraints:** type `boolean`
- **Description:** Whether URLs that differ only by their query string count as one URL. Default true.

### `proxy` (query)

- **Required:** no
- **Constraints:** type `string`; enum: datacenter, residential, mobile
- **Description:** Most expensive proxy tier to use. The page is tried on `datacenter` first and moves up only if it is blocked, stopping at this tier. Default: every tier (`mobile`).

### `maxCredits` (query)

- **Required:** no
- **Constraints:** type `integer`; minimum: 1; maximum: 25
- **Description:** Most credits this call may cost (1–25). Default 1, which loads about 1.6 MB on the datacenter proxy or 190 KB on residential or mobile. Raise it for heavy pages; you're charged only for what the page uses.

## Responses (status codes)

- **200**: Website's URLs.
- **400**: Invalid query parameters or disallowed URL
- **401**: Missing or invalid API key
- **402**: Insufficient credits
- **500**: Unexpected or billing error
- **502**: Site map could not be completed.
- **503**: Service temporarily unavailable; safe to retry with backoff.

## Response body (200)

Website's URLs.

### Field outline

- **data** (required) — type `object`. Endpoint-specific response payload.
  - **lookupStatus** (required) — type `string`; enum: found, restricted, budget_exceeded. Whether page content could be extracted. `restricted`: bot protection blocked the page on every proxy tier allowed. `budget_exceeded`: the page needed more than `maxCredits` allows before any content loaded.
  - **proxy** (required) — type `string`; enum: datacenter, residential, mobile, null; nullable. Proxy tier the page was loaded on: the tier that served it, or the last one tried. Null when not measured.
  - **budgetExceeded** (required) — type `boolean`. Whether the page needed more than `maxCredits` allows. Loading stopped early, so content may be incomplete.
  - **bytesTransferred** (required) — type `integer`; minimum: 0; nullable. Data the page loaded through the proxy, in bytes, across every tier tried. Null when not measured.
  - **url** (required) — type `string`. URL that was requested.
  - **source** (required) — type `string`; enum: sitemap, links, none. Where the URLs came from: the site's sitemaps, the page's own links, or nowhere (no URLs found).
  - **urls** (required) — type `array`. The site's URLs, in the order found, without duplicates, at most `limit`.
    - _items:_
      - **url** (required) — type `string`. Page URL.
      - **lastmod** (required) — type `string`; nullable. Last modified date the sitemap gives, as written there (usually ISO 8601); null when absent.
      - **changefreq** (required) — type `string`; nullable. Change frequency the sitemap gives, for example `weekly`; null when absent.
      - **priority** (required) — type `number`; nullable. Priority the sitemap gives (0–1); null when absent.
      - **sitemap** (required) — type `string`; nullable. Sitemap that listed the URL; null when it came from the page's links.
  - **truncated** (required) — type `boolean`. Whether more URLs were left unread: `limit` was reached, the sitemap or `maxCredits` caps stopped reading, or time ran out (`timedOut`).
  - **sitemaps** (required) — type `array`. Sitemap files read, in order, at most 25.
    - _items:_
      - **url** (required) — type `string`. Sitemap URL (after redirects).
      - **status** (required) — type `integer`; nullable. HTTP status it answered with; null when it didn't answer.
      - **kind** (required) — type `string`; enum: urlset, index, null; nullable. `urlset` lists pages, `index` lists more sitemaps; null when unreadable.
      - **urlCount** (required) — type `integer`; minimum: 0. Entries it lists: pages for a urlset, sitemaps for an index (counted up to where reading stopped).
      - **error** (required) — type `string`; nullable. Why it couldn't be read, for example `not_found`, `blocked`, `not_xml`, `index_too_deep`, `blocked_destination` (it points at a non-public address, so it was skipped) or `unsupported_encoding`; null when read.
      - **truncated** (required) — type `boolean`. Whether reading stopped early (size cap, `maxCredits` or `limit`).
  - **robotsTxt** (required) — type `object`; nullable. The site's robots.txt, when it was read (not for a sitemap URL or `sitemap=skip`).
    - **url** (required) — type `string`. robots.txt URL (after redirects).
    - **status** (required) — type `integer`; nullable. HTTP status it answered with; null when it didn't.
    - **sitemaps** (required) — type `array`. Sitemap URLs it lists, in order.
      - _items:_
        - type `string`
  - **timedOut** (required) — type `boolean`. Whether reading stopped at the service's time limit; `urls` holds what was found by then, and `truncated` is true.
  - **warnings** (required) — type `array`. Things skipped along the way, for example a sitemap that points at a non-public address; empty when nothing was.
    - _items:_
      - type `string`
- **meta** (required) — type `object`. Metadata describing the request and billing outcome.
  - **requestId** (required) — type `string`; minLength: 1. Unique request identifier for tracing this API call.
  - **creditsCharged** (required) — type `integer`; minimum: 0. Credits charged for this request.
  - **version** (required) — type `string`; enum: v1. Public API version that served the response.
  - **stale** (optional) — type `boolean`; enum: true. Present, and `true`, only when the source couldn't be reached and this is the last good answer for the same request instead, up to 24 hours old. Absent on every live answer.
  - **fetchedAt** (optional) — type `string`. When a stale answer (`stale: true`) was originally fetched (ISO 8601, UTC). Absent on every live answer.

### Example JSON (OpenAPI example)

```json
{
  "data": {
    "lookupStatus": "found",
    "proxy": "datacenter",
    "budgetExceeded": false,
    "bytesTransferred": 7125,
    "url": "https://www.socialfetch.dev/",
    "source": "sitemap",
    "urls": [
      {
        "url": "https://www.socialfetch.dev/",
        "lastmod": null,
        "changefreq": null,
        "priority": null,
        "sitemap": "https://www.socialfetch.dev/sitemap.xml"
      }
    ],
    "truncated": true,
    "sitemaps": [
      {
        "url": "https://www.socialfetch.dev/sitemap.xml",
        "status": 200,
        "kind": "urlset",
        "urlCount": 313,
        "error": null,
        "truncated": true
      }
    ],
    "robotsTxt": {
      "url": "https://www.socialfetch.dev/robots.txt",
      "status": 200,
      "sitemaps": [
        "https://www.socialfetch.dev/sitemap.xml"
      ]
    },
    "timedOut": false,
    "warnings": []
  },
  "meta": {
    "requestId": "req_01example",
    "creditsCharged": 1,
    "version": "v1"
  }
}
```

### Machine-readable error codes

When an error JSON body is returned, it may include one of these `error.code` values (derived from the OpenAPI schemas for this operation; additional codes may exist at runtime):

- `bad_request`

## Error handling & retries

Interpret HTTP status codes using the descriptions below. Do not assume a JSON body unless the OpenAPI schema defines one for that status.

- **400**: Invalid query parameters or disallowed URL **Retry:** Fix the request; retrying the same invalid payload will not help.
- **401**: Missing or invalid API key **Retry:** Fix the API key first; retrying without changes will not help.
- **402**: Insufficient credits **Retry:** Do not retry without resolving billing/credits (retrying the same request will not help).
- **500**: Unexpected or billing error
- **502**: Site map could not be completed. **Retry:** May be transient; a few retries with backoff are reasonable.
- **503**: Service temporarily unavailable; safe to retry with backoff. **Retry:** Usually safe to retry with exponential backoff and jitter.

### Suggested client defaults

- Send the API key using the `x-api-key` header on every request.
- On `503` (and sometimes `502`), retry with backoff; cap retries and surface a clear error to the user.
- On `402`, surface an actionable billing message rather than blind retries.

## Examples

### TypeScript SDK

```typescript
import { SocialFetchClient } from "@socialfetch/sdk";

const client = new SocialFetchClient({
  apiKey: process.env.SOCIALFETCH_API_KEY!,
});

const result = await client.web.map({
  url: "https://www.socialfetch.dev",
});

if (!result.ok) {
  console.error(result.error);
} else {
  console.log(result.value.data);
}
```

### Node.js

```javascript
const response = await fetch(
  "https://api.socialfetch.dev/v1/web/map?url=https://www.socialfetch.dev",
  {
    headers: {
      "x-api-key": "YOUR_API_KEY"
    }
  }
);

const data = await response.json();
console.log(data);
```

### cURL

```bash
curl "https://api.socialfetch.dev/v1/web/map?url=https://www.socialfetch.dev" \
  -H "x-api-key: YOUR_API_KEY"
```

### Python

```python
import requests

response = requests.get(
    "https://api.socialfetch.dev/v1/web/map?url=https://www.socialfetch.dev",
    headers={"x-api-key": "YOUR_API_KEY"},
)
data = response.json()
print(data)
```