Skip to main content
Reader has two proxy modes: standard (the default, 1 credit per page) and premium (3 credits per page, uses residential proxies to bypass bot walls). Most scrapes work fine with standard. This guide is for deciding when to upgrade to premium.

The default: standard

standard uses datacenter proxies and works for the vast majority of the web: blogs, documentation sites, news, public APIs, marketing pages, and most government or academic sites. If you have no reason to think a site is hostile, leave proxyMode unset and it defaults to standard.

When to force premium

Force proxyMode: "premium" when:
  • The target is known-hostile. Amazon, LinkedIn, booking sites, ticket resellers, most aggressive e-commerce. These sites block datacenter IP ranges entirely, so standard mode will always fail. Go straight to premium and get clean data on the first try.
  • Standard is returning thin or wrong content. If metadata.statusCode is 200 but the markdown is very short, or contains phrases like “checking your browser” or “access denied”, the site is blocking standard mode. Switch to premium.
  • You need consistent results across a batch. Some sites block standard inconsistently: some pages work, others don’t. Forcing premium gives you a uniform experience across the run.

When to stick with standard

Use standard explicitly (or leave proxyMode unset) when:
  • You know the target is friendly. Your own blog, a partner’s public docs, a government site, a content API. No need to pay 3x.
  • You want a cost guarantee. With standard explicit, every scrape is exactly 1 credit. For a 10,000-URL batch where budget certainty matters, lock the mode.
  • You want an error signal for blocks. If you force standard and the target blocks you, Reader returns an error rather than silently costing more. You find out immediately that something changed.

Per-site cheat sheet

Per-URL split in a batch

proxyMode applies to the whole batch, not per URL. If you have a mixed list of friendly and hostile URLs, two options:
  1. Pre-split the batch. Route friendly URLs to a standard batch and hostile URLs to a premium batch.
  2. Run all hostile URLs at premium. If you know certain domains always need premium, use a lookup to route them to a separate call.
Option 1 keeps costs predictable. For small mixed batches, running everything at premium is simpler if cost is less of a concern.

Checking what mode was used

Every response includes metadata.proxyMode so you can see exactly what ran:
For a batch, check the distribution across results to understand which sites needed premium:

Next