Agent-Reach is a unified CLI that wraps yt-dlp, twitter-cli, rdt-cli, and Jina Reader so any agent can scrape Twitter, Reddit, YouTube, RSS feeds, and generic web pages without paying for APIs. It hit 23,562 stars and trended #4 on GitHub for Python because it solves a specific pain point: agents can write code and manage files, but they cannot read social media threads or video transcripts without expensive API keys or brittle browser automation.
The project explicitly targets Claude Code, Cursor, Windsurf, and any MCP-compatible agent. The design philosophy is simple: give agents one command to install, one command to update, and a built-in diagnostic tool to troubleshoot platform blocks or proxy issues.
Why This Exists
Agents can execute shell commands and read local files. They cannot natively fetch a YouTube transcript, search Reddit threads, or scrape a Twitter profile. Each platform has its own authentication model, rate limits, and anti-bot defenses. Agent-Reach abstracts all of that into a single CLI that agents can invoke like any other tool.
The alternative is paying for APIs (Twitter’s API pricing is prohibitively expensive for most use cases), maintaining separate scrapers for each platform, or writing custom browser automation that breaks every time a site changes its DOM structure. Agent-Reach wraps mature open-source scrapers and keeps them updated as platforms evolve.
Architecture
Agent-Reach is a Python package that bundles multiple CLI tools and exposes them through a unified interface. The core components are:
- yt-dlp for YouTube video metadata and subtitles
- twitter-cli for Twitter/X scraping
- rdt-cli for Reddit threads and searches
- Jina Reader for generic web page extraction
- Local cookie storage for authentication persistence
- agent-reach doctor for self-diagnosis
The agent installs Agent-Reach with a single command, then invokes subcommands like agent-reach youtube <url> or agent-reach twitter search <query>. All authentication cookies are stored locally in ~/.agent-reach/cookies/. No credentials are uploaded to cloud services.
Dependency Management
Agent-Reach tracks upstream changes in yt-dlp, twitter-cli, and rdt-cli. When a platform changes its API or blocking behavior, the maintainers update the bundled dependencies and push a new release. Agents can update by running agent-reach update or reinstalling the package.
The project pins specific versions of each dependency to avoid breaking changes. When a new version of yt-dlp or twitter-cli is released, the maintainers test it against all supported platforms before merging.
Authentication Flow
Most platforms require authentication to bypass rate limits or access private content. Agent-Reach uses local cookie storage:
- User logs into Twitter, Reddit, or YouTube in their browser
- User exports cookies using a browser extension (EditThisCookie or similar)
- User saves cookies to
~/.agent-reach/cookies/<platform>.json - Agent-Reach reads cookies from disk and injects them into HTTP requests
Cookies are never sent to external servers. The agent reads them from the local filesystem, so the security boundary is the same as any other file the agent can access.
Diagnostic Tool
The agent-reach doctor command checks connectivity to each platform and reports which services are reachable. It tests:
- Network connectivity (can the agent reach twitter.com, reddit.com, etc.)
- Cookie validity (are stored cookies still valid)
- Proxy configuration (is a proxy required and correctly configured)
- Platform blocks (is the IP address or user agent blocked)
The command outputs diagnostic information that agents can parse to determine which platforms are available and which require additional configuration. The exact output format is not documented in the repository, but the tool is designed to guide users through common setup issues.
Platform Coverage
| Platform | No Config Required | Requires Auth | Notes |
|---|---|---|---|
| Generic web pages | Yes | No | Uses Jina Reader for clean Markdown extraction |
| YouTube | Yes | No | Subtitles and metadata work without login |
| RSS/Atom feeds | Yes | No | Standard feed parsing |
| GitHub (public repos) | Yes | No | Search and file reading work without auth |
| Twitter/X | No | Yes | Requires cookies; official API has usage fees |
| No | Yes | Requires cookies to bypass rate limits | |
| Bilibili | No | Yes | Chinese video platform; requires proxy for non-CN IPs |
| XiaoHongShu | No | Yes | Chinese social platform; requires login for most content |
Code Example
An agent using Agent-Reach to fetch a YouTube transcript:
import subprocess
import json
def fetch_youtube_transcript(video_url):
result = subprocess.run(
["agent-reach", "youtube", video_url],
capture_output=True,
text=True
)
if result.returncode != 0:
raise Exception(f"Scrape failed: {result.stderr}")
try:
data = json.loads(result.stdout)
except json.JSONDecodeError as e:
raise Exception(f"Invalid JSON response: {e}")
return {
"title": data.get("title", ""),
"transcript": data.get("subtitles", ""),
"duration": data.get("duration", 0)
}The agent does not need to handle yt-dlp installation, cookie management, or subtitle format conversion. Agent-Reach returns structured JSON that the agent can parse directly.
Rate Limits and Failure Modes
Agent-Reach inherits the rate limits and failure modes of its underlying scrapers:
- yt-dlp can be blocked by YouTube if too many requests come from the same IP. Solution: use a residential proxy or rotate IPs.
- twitter-cli requires valid cookies. If cookies expire, the agent must re-authenticate. Cookies typically last 30 days.
- rdt-cli is subject to Reddit’s rate limiting. Authentication via cookies provides higher rate limits than unauthenticated access.
- Jina Reader is a hosted service with generous free tier limits. If you exceed them, you need to self-host the Jina Reader API.
The agent-reach doctor command detects most of these issues and suggests fixes (e.g., “Twitter cookies expired, please re-authenticate”).
Deployment Considerations
Agent-Reach runs wherever the agent runs. If the agent is local (Claude Code on your laptop), Agent-Reach runs locally. If the agent is on a server (self-hosted Cursor backend), Agent-Reach runs on that server.
For server deployments, you need to handle:
- Proxy configuration if the server IP is blocked by Twitter, Reddit, or Chinese platforms
- Cookie refresh if the agent needs long-term access to authenticated platforms
- Disk space for cached video metadata and transcripts
The recommended setup is a cheap VPS ($5/month) with a residential proxy ($1/month) for platforms that block datacenter IPs.
Security Boundaries
Agent-Reach stores cookies in plaintext JSON files. If an attacker gains filesystem access, they can steal cookies and impersonate the user on Twitter, Reddit, etc. This is the same risk as any browser cookie storage.
The mitigation is to run the agent in a sandboxed environment (Docker container, VM, or restricted user account) and limit filesystem access. If the agent is compromised, the attacker gets the same access the agent has.
Agent-Reach does not implement any additional encryption or secret management. The assumption is that the agent’s execution environment is already trusted.
Observability
Agent-Reach logs all HTTP requests to ~/.agent-reach/logs/. Each log entry includes:
- Timestamp
- Platform (twitter, reddit, youtube, etc.)
- Request URL
- Response status code
- Error message (if any)
Agents can parse these logs to detect patterns (e.g., “Reddit is rate-limiting us, slow down”) or debug failures (e.g., “Twitter returned 403, cookies may be invalid”).
Agent-Reach does not collect telemetry or send usage data to external services. Monitoring is manual and requires parsing local log files.
Update Cadence
The maintainers track upstream changes in yt-dlp, twitter-cli, and rdt-cli. When a platform changes its API or blocking behavior, the maintainers update the bundled dependencies and push a new release. The typical update cadence is:
- yt-dlp: updated weekly (YouTube changes frequently)
- twitter-cli: updated monthly (Twitter/X changes less often)
- rdt-cli: updated quarterly (Reddit API is relatively stable)
Agents can check for updates by running agent-reach version --check or by reinstalling the package.
Technical Verdict
Use Agent-Reach if you need to give agents access to Twitter, Reddit, YouTube, or RSS feeds without paying for APIs, you are comfortable managing cookies and proxies, you want a single CLI that wraps multiple scrapers, and you are building an MCP-compatible agent that can execute shell commands.
Avoid Agent-Reach if you need guaranteed uptime or SLA (scrapers break when platforms change), you cannot manage cookie refresh or proxy configuration, you need real-time data (scrapers are slower than official APIs), or you are building a commercial product that cannot tolerate scraping risk.
Agent-Reach is infrastructure for giving agents internet access without API fees. It trades reliability and speed for cost savings and flexibility. If you can tolerate occasional breakage and are willing to maintain cookies and proxies, it is a practical tool for agent-based workflows.