Basic
Core scraping and crawling patterns to get started.Basic Scrape
Scrape a single URL and get markdown, HTML, and metadata
Batch Scrape
Scrape multiple URLs concurrently with progress tracking
Large Batch Scrape
Handle 1000+ URLs with pool sizing and memory management
Browser Pool Config
Configure pool size, recycling, and queue management
Single Proxy
Route scraping through a proxy with geo-targeting
Proxy Rotation
Rotate through multiple proxies with round-robin or random strategies
Cloudflare Bypass
Scrape Cloudflare-protected sites with automatic challenge handling
Crawl Website
Crawl a site with depth control and optional content scraping
All Formats
Output content in both markdown and HTML formats
AI Tools
Integrate Reader with LLMs, RAG frameworks, and vector stores.OpenAI Summary
Scrape pages and summarize with GPT, with token tracking
Anthropic Summary
Scrape pages and summarize with Claude, with token tracking
Vercel AI Streaming
Stream summaries in real-time using the Vercel AI SDK
LangChain Loader
Custom LangChain document loader for scraping and crawling
LlamaIndex Loader
Custom LlamaIndex document loader for RAG pipelines
Pinecone Ingest
Scrape, embed with OpenAI, and ingest into Pinecone
Qdrant Ingest
Scrape, embed with OpenAI, and ingest into Qdrant
Production
Patterns for running Reader in production environments.Express Server
REST API with health checks, scrape/crawl endpoints, and graceful shutdown
Browser Pool Scaling
Advanced pool management with metrics, auto-recovery, and graceful degradation
Job Queue (BullMQ)
Async job queue with Redis, webhooks, retries, and horizontal scaling
Deployment
Deploy Reader to various platforms.Docker
Docker and docker-compose with health checks and graceful shutdown
AWS Lambda
Serverless deployment with container images and API Gateway
Vercel Functions
Serverless functions with remote browser service recommendations