Agent-Reach: One CLI to Wire Twitter, Reddit, YouTube, and Bilibili Into Your AI Agent

AI agents can write code, refactor documentation, and manage projects. Ask them to read a Twitter thread, summarize a YouTube video, or check Reddit sentiment, and they hit a wall. Every platform has different authentication, rate limits, scraping defenses, and data schemas. Agent-Reach solves this with a single CLI command that wires Twitter, Reddit, YouTube, Bilibili, XiaoHongShu, and GitHub into any MCP-compatible agent without paid APIs or brittle CSS selectors.

The project hit GitHub Trending #9 for Python with 63K+ stars and 5K+ forks. It abstracts platform-specific scraping into a unified MCP server, so agents can query social data without knowing each platform’s internal structure.

The Problem: Agents Can’t Read the Internet

Agents fail at social platform access for predictable reasons:

Authentication barriers: Twitter requires login for search, Reddit blocks server IPs, XiaoHongShu demands cookies.
API costs: Twitter’s API starts at hundreds of dollars per month for basic access.
Anti-bot defenses: Platforms rotate selectors, fingerprint browsers, and block common scraping tools.
Schema heterogeneity: Each platform returns data in different formats (JSON, HTML, XML feeds).

Agent-Reach routes around these by maintaining local authentication state, swapping backend scrapers when one gets blocked, and exposing a single MCP interface that agents can query without platform-specific logic.

Architecture: Multi-Backend Routing with Local Auth

Agent-Reach is a Python CLI that runs as an MCP server. It handles three layers:

Authentication state management: Stores cookies and tokens locally, never uploads them.
Backend routing: Each platform has a primary scraper and fallback options. When one fails, the router switches to the next.
MCP interface: Exposes platform data through standardized tool calls that agents can invoke.

Authentication State

Agent-Reach stores authentication credentials in local files:

Twitter: Cookie-based session stored in ~/.agent-reach/twitter_cookies.json
Reddit: OAuth token cached after initial login
YouTube: No auth required for public videos, optional login for private playlists
Bilibili: Session cookies required for most content
XiaoHongShu: Login mandatory, cookies stored locally

The CLI never sends credentials to external servers. When an agent calls a tool like search_twitter, the MCP server reads the local cookie file, attaches it to the HTTP request, and returns the scraped data.

Backend Routing

Each platform has multiple scraping backends ranked by reliability. When the primary backend fails (rate limit, IP block, selector change), the router tries the next option.

Example for Bilibili:

Primary: bili-cli (official-ish tool, harder to block)
Fallback 1: yt-dlp with Bilibili extractor
Fallback 2: Direct HTML scraping with rotating user agents

In June 2026, Bilibili’s anti-bot system blocked yt-dlp entirely. Agent-Reach automatically switched to bili-cli without requiring user intervention. The routing logic is in agent_reach/platforms/bilibili.py:

class BilibiliPlatform:
    def __init__(self):
        self.backends = [
            BiliCliBackend(),
            YtDlpBackend(),
            DirectScraperBackend()
        ]
    
    def fetch_video(self, video_id):
        for backend in self.backends:
            try:
                return backend.fetch(video_id)
            except (RateLimitError, BlockedError) as e:
                logger.warning(f"{backend.name} failed: {e}")
                continue
        raise AllBackendsFailedError("All Bilibili backends exhausted")

MCP Interface

Agent-Reach exposes platform capabilities as MCP tools. Agents call these tools without knowing which backend is running:

search_twitter(query, limit) → returns list of tweets with text, author, timestamp
get_reddit_thread(url) → returns post content plus top comments
get_youtube_transcript(video_id) → returns timestamped transcript
search_bilibili(keyword) → returns video metadata and transcripts
get_xiaohongshu_post(url) → returns post text and image URLs

Each tool returns JSON with a consistent schema. The agent doesn’t need to parse HTML or handle platform-specific quirks.

Deployment Shape

Agent-Reach runs in two modes:

Local Mode (Default)

Install via pip, run as a background MCP server:

pip install agent-reach
agent-reach start

The server listens on localhost:8765. Agents connect via MCP protocol. Authentication happens through local cookie files. No proxy required.

Server Mode (Optional)

For agents running on cloud VMs or containers, some platforms (Reddit, Twitter) block datacenter IPs. Agent-Reach supports an optional residential proxy:

agent-reach start --proxy https://proxy.example.com

The proxy costs around $1/month for typical usage. It’s only needed when the agent runs on a server, not for local development.

Diagnostics and Failure Modes

Agent-Reach includes a doctor command that tests each platform and reports status:

agent-reach doctor

Output example:

✓ Twitter: authenticated, 500 requests remaining today
✗ Reddit: IP blocked, switch to proxy mode
✓ YouTube: no auth required, working
✓ Bilibili: bili-cli backend active
⚠ XiaoHongShu: cookies expired, re-login required

Common failure modes:

Failure	Cause	Fix
`CookieExpiredError`	Platform logged you out	Re-run `agent-reach login <platform>`
`RateLimitError`	Too many requests in short window	Wait 15 minutes or switch to proxy
`AllBackendsFailedError`	All scrapers blocked for a platform	Check GitHub issues, update to latest version
`SchemaChangedError`	Platform changed HTML structure	Update Agent-Reach, maintainers patch selectors

Security Boundaries

Agent-Reach stores credentials locally but exposes them to any process that can call the MCP server. Security considerations:

Local-only by default: MCP server binds to 127.0.0.1, not 0.0.0.0.
No credential upload: Cookies never leave your machine unless you explicitly configure a proxy.
Agent trust model: Any agent with MCP access can read your social feeds. Don’t run untrusted agents with Agent-Reach enabled.
Proxy risk: If you use the optional proxy, your cookies pass through a third-party server. Use only if you trust the proxy provider.

Observability

Agent-Reach logs all tool calls and backend switches:

2026-07-31 10:15:23 INFO [twitter] search_twitter(query="AI agents", limit=10) → 10 results
2026-07-31 10:16:45 WARN [bilibili] yt-dlp backend failed (403), switching to bili-cli
2026-07-31 10:17:02 INFO [bilibili] bili-cli backend succeeded

Logs go to ~/.agent-reach/logs/. Useful for debugging rate limits and backend failures.

Trade-Offs

Aspect	Agent-Reach	Platform APIs	Custom Scrapers
Cost	Free (optional $1/mo proxy)	$100-500/mo per platform	Free but high maintenance
Reliability	Medium (depends on backend updates)	High (official)	Low (breaks on selector changes)
Setup time	5 minutes	Hours (API keys, OAuth, docs)	Days (per platform)
Rate limits	Platform-dependent, no hard caps	Strict, enforced by API	Platform-dependent
Auth complexity	One-time login per platform	OAuth flows, token refresh	Manual cookie management

Technical Verdict

Use Agent-Reach when:

You need agents to read social platforms without paying for APIs.
You want a single interface for heterogeneous data sources.
You’re building local agents (Claude Code, Cursor) that can run a background MCP server.
You trust the maintainers to keep scrapers updated when platforms change.

Avoid Agent-Reach when:

You need guaranteed uptime and SLAs (use official APIs).
You’re scraping at scale (thousands of requests per hour) and need rate limit predictability.
You can’t tolerate occasional breakage when platforms update their anti-bot defenses.
You need real-time data (scrapers add 1-3 second latency per request).

Agent-Reach is plumbing for agents that need eyes on the internet. It won’t replace official APIs for production systems, but it removes the friction of wiring social platforms into agentic workflows. The multi-backend routing is the key insight: when one scraper dies, the next one takes over, and agents keep working.

Agentic AI, minus the noise.