List website URLs
List a website's URLs from its sitemaps, or from the page's own links when it has none.
Headers
Get your API keyAPI key (`sfk_...`)
Site to map: its home page, any page on it, or a sitemap URL such as `https://www.socialfetch.dev/sitemap.xml`.
min 1 chars · max 2083 chars
Most URLs to return (1–5000). Default 1000.
≥ 1 · ≤ 5000
Only return URLs containing this text (case-insensitive), for example `/blog/`.
min 1 chars · max 200 chars
How to find URLs: `include` reads the sitemaps and falls back to the page's links when they list nothing, `only` reads the sitemaps alone, `skip` reads only the page's links. Default `include`.
one of: include, only, skip
Whether to keep URLs on subdomains of the site, such as `docs.` or `blog.`. Default true.
Whether URLs that differ only by their query string count as one URL. Default true.
Most expensive proxy tier to use. The page is tried on `datacenter` first and moves up only if it is blocked, stopping at this tier. Default: every tier (`mobile`).
one of: datacenter, residential, mobile
Most credits this call may cost (1–25). Default 1, which loads about 1.6 MB on the datacenter proxy or 190 KB on residential or mobile. Raise it for heavy pages; you're charged only for what the page uses.
≥ 1 · ≤ 25
Endpoint-specific response payload.
Whether page content could be extracted. `restricted`: bot protection blocked the page on every proxy tier allowed. `budget_exceeded`: the page needed more than `maxCredits` allows before any content loaded.
one of: found, restricted, budget_exceeded
Proxy tier the page was loaded on: the tier that served it, or the last one tried. Null when not measured.
one of: datacenter, residential, mobile, null
Whether the page needed more than `maxCredits` allows. Loading stopped early, so content may be incomplete.
Data the page loaded through the proxy, in bytes, across every tier tried. Null when not measured.
≥ 0
URL that was requested.
Where the URLs came from: the site's sitemaps, the page's own links, or nowhere (no URLs found).
one of: sitemap, links, none
The site's URLs, in the order found, without duplicates, at most `limit`.
Page URL.
Last modified date the sitemap gives, as written there (usually ISO 8601); null when absent.
Change frequency the sitemap gives, for example `weekly`; null when absent.
Priority the sitemap gives (0–1); null when absent.
Sitemap that listed the URL; null when it came from the page's links.
Whether more URLs were left unread: `limit` was reached, the sitemap or `maxCredits` caps stopped reading, or time ran out (`timedOut`).
Sitemap files read, in order, at most 25.
Sitemap URL (after redirects).
HTTP status it answered with; null when it didn't answer.
`urlset` lists pages, `index` lists more sitemaps; null when unreadable.
one of: urlset, index, null
Entries it lists: pages for a urlset, sitemaps for an index (counted up to where reading stopped).
≥ 0
Why it couldn't be read, for example `not_found`, `blocked`, `not_xml`, `index_too_deep`, `blocked_destination` (it points at a non-public address, so it was skipped) or `unsupported_encoding`; null when read.
Whether reading stopped early (size cap, `maxCredits` or `limit`).
The site's robots.txt, when it was read (not for a sitemap URL or `sitemap=skip`).
robots.txt URL (after redirects).
HTTP status it answered with; null when it didn't.
Sitemap URLs it lists, in order.
Whether reading stopped at the service's time limit; `urls` holds what was found by then, and `truncated` is true.
Things skipped along the way, for example a sitemap that points at a non-public address; empty when nothing was.
Metadata describing the request and billing outcome.
Unique request identifier for tracing this API call.
min 1 chars
Credits charged for this request.
≥ 0
Public API version that served the response.
one of: v1
Present, and `true`, only when the source couldn't be reached and this is the last good answer for the same request instead, up to 24 hours old. Absent on every live answer.
one of: true
When a stale answer (`stale: true`) was originally fetched (ISO 8601, UTC). Absent on every live answer.
Code example
curl "https://api.socialfetch.dev/v1/web/map?url=https://www.socialfetch.dev" \
-H "x-api-key: YOUR_API_KEY"Responses
Website's URLs.