Web

List website URLs

List a website's URLs from its sitemaps, or from the page's own links when it has none.

GET/v1/web/map
x-api-keystringheader

API key (`sfk_...`)

Parameters
8
urlstringrequiredquery

Site to map: its home page, any page on it, or a sitemap URL such as `https://www.socialfetch.dev/sitemap.xml`.

min 1 chars · max 2083 chars

limitintegeroptionalquery

Most URLs to return (1–5000). Default 1000.

≥ 1 · ≤ 5000

querystringoptionalquery

Only return URLs containing this text (case-insensitive), for example `/blog/`.

min 1 chars · max 200 chars

sitemapstringoptionalquery

How to find URLs: `include` reads the sitemaps and falls back to the page's links when they list nothing, `only` reads the sitemaps alone, `skip` reads only the page's links. Default `include`.

one of: include, only, skip

includeSubdomainsbooleanoptionalquery

Whether to keep URLs on subdomains of the site, such as `docs.` or `blog.`. Default true.

ignoreQueryParametersbooleanoptionalquery

Whether URLs that differ only by their query string count as one URL. Default true.

proxystringoptionalquery

Most expensive proxy tier to use. The page is tried on `datacenter` first and moves up only if it is blocked, stopping at this tier. Default: every tier (`mobile`).

one of: datacenter, residential, mobile

maxCreditsintegeroptionalup to maxCreditsquery

Most credits this call may cost (1–25). Default 1, which loads about 1.6 MB on the datacenter proxy or 190 KB on residential or mobile. Raise it for heavy pages; you're charged only for what the page uses.

≥ 1 · ≤ 25

Response fields
35

Endpoint-specific response payload.

datalookupStatus
string

Whether page content could be extracted. `restricted`: bot protection blocked the page on every proxy tier allowed. `budget_exceeded`: the page needed more than `maxCredits` allows before any content loaded.

one of: found, restricted, budget_exceeded

dataproxy
stringnullable

Proxy tier the page was loaded on: the tier that served it, or the last one tried. Null when not measured.

one of: datacenter, residential, mobile, null

databudgetExceeded
boolean

Whether the page needed more than `maxCredits` allows. Loading stopped early, so content may be incomplete.

databytesTransferred
integernullable

Data the page loaded through the proxy, in bytes, across every tier tried. Null when not measured.

≥ 0

dataurl
string

URL that was requested.

datasource
string

Where the URLs came from: the site's sitemaps, the page's own links, or nowhere (no URLs found).

one of: sitemap, links, none

The site's URLs, in the order found, without duplicates, at most `limit`.

dataurls[]url
string

Page URL.

dataurls[]lastmod
stringnullable

Last modified date the sitemap gives, as written there (usually ISO 8601); null when absent.

dataurls[]changefreq
stringnullable

Change frequency the sitemap gives, for example `weekly`; null when absent.

dataurls[]priority
numbernullable

Priority the sitemap gives (0–1); null when absent.

dataurls[]sitemap
stringnullable

Sitemap that listed the URL; null when it came from the page's links.

datatruncated
boolean

Whether more URLs were left unread: `limit` was reached, the sitemap or `maxCredits` caps stopped reading, or time ran out (`timedOut`).

Sitemap files read, in order, at most 25.

datasitemaps[]url
string

Sitemap URL (after redirects).

datasitemaps[]status
integernullable

HTTP status it answered with; null when it didn't answer.

datasitemaps[]kind
stringnullable

`urlset` lists pages, `index` lists more sitemaps; null when unreadable.

one of: urlset, index, null

datasitemaps[]urlCount
integer

Entries it lists: pages for a urlset, sitemaps for an index (counted up to where reading stopped).

≥ 0

datasitemaps[]error
stringnullable

Why it couldn't be read, for example `not_found`, `blocked`, `not_xml`, `index_too_deep`, `blocked_destination` (it points at a non-public address, so it was skipped) or `unsupported_encoding`; null when read.

datasitemaps[]truncated
boolean

Whether reading stopped early (size cap, `maxCredits` or `limit`).

The site's robots.txt, when it was read (not for a sitemap URL or `sitemap=skip`).

datarobotsTxturl
string

robots.txt URL (after redirects).

datarobotsTxtstatus
integernullable

HTTP status it answered with; null when it didn't.

Sitemap URLs it lists, in order.

datarobotsTxtsitemaps[]
stringoptional
datatimedOut
boolean

Whether reading stopped at the service's time limit; `urls` holds what was found by then, and `truncated` is true.

Things skipped along the way, for example a sitemap that points at a non-public address; empty when nothing was.

datawarnings[]
stringoptional

Metadata describing the request and billing outcome.

metarequestId
string

Unique request identifier for tracing this API call.

min 1 chars

metacreditsCharged
integer

Credits charged for this request.

≥ 0

metaversion
string

Public API version that served the response.

one of: v1

metastale
booleanoptional

Present, and `true`, only when the source couldn't be reached and this is the last good answer for the same request instead, up to 24 hours old. Absent on every live answer.

one of: true

metafetchedAt
stringoptional

When a stale answer (`stale: true`) was originally fetched (ISO 8601, UTC). Absent on every live answer.

Code example

curl "https://api.socialfetch.dev/v1/web/map?url=https://www.socialfetch.dev" \
  -H "x-api-key: YOUR_API_KEY"

Responses

Website's URLs.