Substack Post Scraper

News & Research

Operated by Sourcelane✓ verified

A web-scraping connector that extracts complete post-level metadata from Substack publications (supports Substack subdomains and custom domains). It collects publication information (name, URL, description), per-post identifiers and URLs, content metadata (titles, subtitles, descriptions, content type, cover image), publishing data (publish timestamps and word counts), paywall status, engagement metrics (likes, comments, restacks), author details (name and handle), and a scrape timestamp for each post; outputs one record per post optimized for downstream analysis and ranking by engagement or cadence.

Verified Sep 28, 12:10 AM

$0.0018

per post

Up to 100 per call. Only pay for results returned; failed calls are refunded.

What people use it for

Content research — analyze topics and formats published by a newsletter
Competitor analysis — benchmark newsletters within a niche
Publishing cadence tracking — measure posting frequency and timing
Newsletter sponsorship research — assess potential sponsorship targets by audience engagement
Top-performing post discovery — identify posts with high likes, comments, or restacks
Creator and influencer research — build lists of active Substack writers in any vertical

Trust & reliability

7d uptime trend

Window	Uptime	Success rate	p50	p95
24h	—	—	—	—
7d	—	—	—	—
30d	—	—	—	—

Calling contract

Call it through Sourcelane's gateway or MCP server. We run the connector, apply your agent's spend guardrails, and bill only the results returned.

Call it via Sourcelane

curl -X POST https://cool-puffin-608.convex.site/v1/call \
  -H "Authorization: Bearer sl_live_your_agent_key" \
  -H "Content-Type: application/json" \
  -d '{"listing":"substack-post-scraper","params":{"urls":["https://astralcodexten.substack.com","https://www.thefitzwilliam.com"],"maxItems":5},"maxResults":10}'

Request params

Field	Type	Required	Description
`urls`	array	required	Substack publication URLs to scrape. Accepts substack.com subdomains (https://platformer.substack.com) and custom domains (https://platformer.news). All posts are returned newest-first.
`maxItems`	integer	optional	Maximum number of posts to collect across all publications. Set to 0 for unlimited. Capped at 100 per call.

Each result contains

Field	Type	Description
`publicationName`	null	Publication Name
`publicationUrl`	string	Publication Url
`publicationDescription`	null	Publication Description
`postId`	integer	Post Id
`postUrl`	string	Post Url
`title`	string	Title
`subtitle`	string	Subtitle
`description`	string	Description
`publishedAt`	string	Published At
`type`	string	Type
`isPaywalled`	boolean	Is Paywalled
`likesCount`	integer	Likes Count
`commentsCount`	integer	—
`restacksCount`	integer	—
`wordCount`	integer	—
`coverImageUrl`	string	—
`authorName`	string	—
`authorHandle`	null	—
`scrapedAt`	string	—

Use Substack Post Scraper from Claude, ChatGPT or Cursor

Pick your tool and connect in under a minute. Then just ask — for example: “What has been reported about solid-state batteries in the last week? Cite sources.”

Connect Claude

Web, desktop and mobile. Paste one URL.

1Copy your personal connector URL
2In Claude open Settings → Connectors → Add custom connector
3Paste the URL and click Add — done

Manual setup (config files, REST, Python) +

Claude Code

Adds the Sourcelane MCP server with your key as a header.

terminal

claude mcp add --transport http sourcelane https://cool-puffin-608.convex.site/mcp \
  --header "Authorization: Bearer sl_live_your_agent_key"

Claude Desktop (config file)

Alternative to the connector URL: add to claude_desktop_config.json, then restart Claude.

claude_desktop_config.json

{
  "mcpServers": {
    "sourcelane": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://cool-puffin-608.convex.site/mcp",
        "--header",
        "Authorization:${AUTH_HEADER}"
      ],
      "env": {
        "AUTH_HEADER": "Bearer sl_live_your_agent_key"
      }
    }
  }
}

Cursor & Windsurf

Add to ~/.cursor/mcp.json (or Windsurf's mcp_config.json).

mcp.json

{
  "mcpServers": {
    "sourcelane": {
      "url": "https://cool-puffin-608.convex.site/mcp",
      "headers": {
        "Authorization": "Bearer sl_live_your_agent_key"
      }
    }
  }
}

VS Code

Save as .vscode/mcp.json. VS Code prompts for your key once.

.vscode/mcp.json

{
  "servers": {
    "sourcelane": {
      "type": "http",
      "url": "https://cool-puffin-608.convex.site/mcp",
      "headers": {
        "Authorization": "Bearer ${input:sourcelane-key}"
      }
    }
  },
  "inputs": [
    {
      "type": "promptString",
      "id": "sourcelane-key",
      "description": "Sourcelane agent key",
      "password": true
    }
  ]
}

ChatGPT Custom GPT (Actions)

Create a GPT → Actions → Import from URL, then Authentication: API Key, Bearer.

OpenAPI schema URL

https://sourcelane-amber.vercel.app/openapi.json

REST

One POST. Pass maxResults to cap cost.

curl

curl -X POST https://cool-puffin-608.convex.site/v1/call \
  -H "Authorization: Bearer sl_live_your_agent_key" \
  -H "Content-Type: application/json" \
  -d '{"listing":"substack-post-scraper","params":{"urls":["https://astralcodexten.substack.com","https://www.thefitzwilliam.com"],"maxItems":5},"maxResults":10}'

Python, LangChain, CrewAI, OpenAI Agents SDK…

Wrap the REST call as a tool, or point an MCP client at the endpoint.

python

import requests

res = requests.post(
    "https://cool-puffin-608.convex.site/v1/call",
    headers={"Authorization": "Bearer sl_live_your_agent_key"},
    json={
        "listing": "substack-post-scraper",
        "params": {"urls":["https://astralcodexten.substack.com","https://www.thefitzwilliam.com"],"maxItems":5},
        "maxResults": 10,
    },
    timeout=300,
)
body = res.json()
print(body["receipt"]["chargedMicros"], "micro-USD for", body["receipt"]["results"], "results")
print(body["data"])

Reviews

—

0 reviews

Leave a review

Loading reviews…