Operated by Sourcelane✓ verified
Extract every URL from a website's sitemap.xml. Recursively walks nested sitemap indexes and returns loc, lastmod, changefreq, and priority for each page.
A sitemap crawler that extracts a complete URL inventory from a website by auto-discovering and parsing sitemap files (including gzipped XML), following sitemap index references recursively, and returning per-page metadata such as the page URL, last-modified date, change frequency, priority score, and the originating sitemap file. Technical features include robots.txt sitemap discovery with a /sitemap.xml fallback, transparent decompression of.xml.gz files, breadth-first traversal of nested sitemapindex files with sitemap URL de-duplication to handle circular references, configurable recursion depth and sitemap counts, and per-sitemap request timeouts to manage slow responses.
Verified Sep 28, 3:55 AM
$0.0003
per url
Up to 100 per call. Only pay for results returned; failed calls are refunded.
SEO audits
Build a full URL inventory before site-wide checks (broken links, missing meta tags, schema validation)
Content migration
Compile a complete list of site URLs for platform migrations
Crawl budget planning
Quantify exposed URLs and recent update timestamps for crawl prioritization
Competitor research
Map every page published by a competitor site
Sitemap validation
Verify that published sitemaps contain expected pages
Bulk URL scraping pipelines
Generate URL lists for downstream tasks like screenshots, content extraction, or AI summarization
Either a homepage URL (e.g.
Optional settings
| Window | Uptime | Success rate | p50 | p95 | Calls |
|---|---|---|---|---|---|
| 24h | — | — | — | — | 0 |
| 7d | — | — | — | — | 0 |
| 30d | — | — | — | — | 0 |
Call it through Sourcelane's gateway or MCP server. We run the connector, apply your agent's spend guardrails, and bill only the results returned.
Call it via Sourcelane
curl -X POST https://api.usesourcelane.com/v1/call \
-H "Authorization: Bearer sl_live_your_agent_key" \
-H "Content-Type: application/json" \
-d '{"listing":"sitemap-extractor","params":{"websiteUrl":"https://www.stripe.com","maxItems":5},"maxResults":10}'Request params
| Field | Type | Required | Description |
|---|---|---|---|
websiteUrl | string | required | Either a homepage URL (e.g. 'https://www.stripe.com' - we'll auto-discover the sitemap via robots.txt or /sitemap.xml) or a direct sitemap URL (e.g. 'https://www.stripe.com/sitemap.xml'). |
maxItems | integer | optional | Maximum number of URLs to return. 0 = unlimited (walks the entire sitemap tree). Capped at 100 per call. |
followSitemapIndex | boolean | optional | When the top-level sitemap is a sitemap index (a list of child sitemaps), recurse into each child sitemap. Disable to only extract URLs from the top-level file. |
Each result contains
| Field | Type | Description |
|---|---|---|
loc | string | Loc |
lastmod | null | Lastmod |
changefreq | null | Changefreq |
priority | null | Priority |
sourceSitemap | string | Source Sitemap |
Pick your tool and connect in under a minute. Then just ask — for example: “For these 50 domains, find contact emails and what e-commerce platform they run on.”
Connect Claude
Web, desktop and mobile. One click, then confirm.
Adds the Sourcelane MCP server with your key as a header.
claude mcp add --transport http sourcelane https://api.usesourcelane.com/mcp \
--header "Authorization: Bearer sl_live_your_agent_key"Alternative to the connector URL: add to claude_desktop_config.json, then restart Claude.
{
"mcpServers": {
"sourcelane": {
"command": "npx",
"args": [
"-y",
"mcp-remote",
"https://api.usesourcelane.com/mcp",
"--header",
"Authorization:${AUTH_HEADER}"
],
"env": {
"AUTH_HEADER": "Bearer sl_live_your_agent_key"
}
}
}
}Add to ~/.cursor/mcp.json (or Windsurf's mcp_config.json).
{
"mcpServers": {
"sourcelane": {
"url": "https://api.usesourcelane.com/mcp",
"headers": {
"Authorization": "Bearer sl_live_your_agent_key"
}
}
}
}Save as .vscode/mcp.json. VS Code prompts for your key once.
{
"servers": {
"sourcelane": {
"type": "http",
"url": "https://api.usesourcelane.com/mcp",
"headers": {
"Authorization": "Bearer ${input:sourcelane-key}"
}
}
},
"inputs": [
{
"type": "promptString",
"id": "sourcelane-key",
"description": "Sourcelane agent key",
"password": true
}
]
}Create a GPT → Actions → Import from URL, then Authentication: API Key, Bearer.
https://usesourcelane.com/openapi.jsonOne POST. Pass maxResults to cap cost.
curl -X POST https://api.usesourcelane.com/v1/call \
-H "Authorization: Bearer sl_live_your_agent_key" \
-H "Content-Type: application/json" \
-d '{"listing":"sitemap-extractor","params":{"websiteUrl":"https://www.stripe.com","maxItems":5},"maxResults":10}'Wrap the REST call as a tool, or point an MCP client at the endpoint.
import requests
res = requests.post(
"https://api.usesourcelane.com/v1/call",
headers={"Authorization": "Bearer sl_live_your_agent_key"},
json={
"listing": "sitemap-extractor",
"params": {"websiteUrl":"https://www.stripe.com","maxItems":5},
"maxResults": 10,
},
timeout=300,
)
body = res.json()
print(body["receipt"]["chargedMicros"], "micro-USD for", body["receipt"]["results"], "results")
print(body["data"])—
0 reviews
Leave a review
Loading reviews…