Web Map API
Pass a site; get back its URLs, ready to filter and feed into markdown, crawl or SEO calls.
GET /v1/web/map1 credit +24 credits for maxCredits Max 25 credits You're only charged for what's returned
About this endpoint
GET /v1/web/map finds a site's sitemaps through its robots.txt (or at /sitemap.xml), follows sitemap indexes, and returns the URLs they list with lastmod, changefreq and priority where the sitemap gives them. Gzipped sitemaps work too. You can also pass a sitemap URL directly. A site with no sitemap is mapped from the links on the page you pass (source: links).
Narrow the list with query (only URLs containing that text, such as /blog/), keep or drop subdomains with includeSubdomains, and set limit (default 1000, up to 5000). truncated: true says more URLs were left; sitemaps lists every file read, with its status and how many entries it holds. A slow site stops at the time limit with the URLs found so far (timedOut: true), and a sitemap that points at a non-public address is skipped and named in warnings.
A call costs 1 credit, which reads about 1.6 MB of sitemaps on the standard proxy, tens of thousands of URLs. A larger site stops early (budgetExceeded: true) with what was read unless you raise maxCredits (up to 25); then you're charged only for the data used. Read meta.creditsCharged on every response.
Price
From $1.35 per 1,000 requests
1 credit per request. Failed requests are free.
- Scale pack
- $1.64 / 1,000
- Growth pack
- $2.36 / 1,000
The lowest rate is on annual plans; packs are pay as you go.
Example request
Send your API key in the x-api-key header. New accounts get 100 free credits, no card needed.
Example response
A 200 response. Live responses have the same shape with current data.
Web Map API FAQ
What does the Web Map API return?
lookupStatus, the requested url, source (sitemap, links or none), urls (each with url, lastmod, changefreq, priority and the sitemap that listed it), truncated, sitemaps (each file read, with status, kind, urlCount and error), robotsTxt, timedOut, warnings, and the proxy tier, budgetExceeded and bytesTransferred.
How do I call it?
Send GET /v1/web/map with x-api-key and a site URL, a page on it, or a sitemap URL. Optional: limit, query, sitemap (include, only or skip), includeSubdomains, ignoreQueryParameters, proxy and maxCredits.
How much does it cost?
1 credit per call by default, which loads about 1.6 MB on the standard proxy. A page that needs more stops early (budgetExceeded: true) unless you raise maxCredits (up to 25); then you're charged only for the data it uses. Blocked sites move up to residential or mobile proxies automatically; those use the budget about nine times faster. meta.creditsCharged is the billing source of truth.
What if a site has no sitemap?
With the default sitemap=include, the page you pass is read once and its links on the same site become the map, marked source: links. Use sitemap=only to get sitemap URLs alone, or sitemap=skip to use only the page's links.
Does it load the pages it lists?
No. It reads robots.txt and sitemaps (or one page's links). To get the pages' content, pass the URLs to GET /v1/web/markdown, GET /v1/web/crawl or GET /v1/web/seo.
What does lookupStatus restricted mean?
Bot protection blocked the site on every proxy tier allowed, so urls is empty. It bills like the other web routes. If you set proxy=datacenter, try without it.