Skip to main content
The #1 reason scrapes come back with thin or wrong content is the target site sitting behind a bot-detection layer: Cloudflare, Akamai, PerimeterX, Datadome, or similar. The site serves real content to browsers and a “checking your browser” page to everything else. Reader handles most of these cases with premium mode.

​
How Reader handles it

standard mode uses datacenter proxies, which many anti-bot systems detect and block. premium mode uses residential proxy IPs that are indistinguishable from real users, bypassing the common bot detection vendors. For sites that are known to be hostile (Amazon, LinkedIn, booking sites, ticket resellers), use premium mode directly:

​
Detecting a block

Some blocks are subtle: the HTTP status code is 200 but the content isn’t what you’d see in a browser. Symptoms:
  • metadata.statusCode is 200 but the markdown is very short
  • The markdown contains phrases like “please enable JavaScript”, “checking your browser”, or “access denied”
  • The same URL in a browser shows totally different content
Reader thinks the scrape succeeded, but what you got isn’t the real page. Switch to premium when you see this.

​
Force premium explicitly

If standard mode is returning thin or challenge content, switch to premium:
This uses residential proxies and goes straight to the bypass strategy. 3 credits per page.

​
When premium doesn’t work either

If premium-mode scrapes also come back thin, the site is beyond Reader’s reach. Common offenders:
  • Sites with CAPTCHAs that require human interaction
  • Sites using very new or custom bot detection that our premium mode hasn’t learned
  • Sites that require a logged-in session
For these, your options are:
  1. Use the site’s official API if they have one
  2. Scrape with your own session (cookies, auth, manual CAPTCHA solving) and only use Reader for the public parts
  3. File an issue. Sometimes we can tune premium for a specific site

​
The waitForSelector trick

If a page loads a shell and then hydrates content client-side, even premium won’t give you the real markdown unless the page has finished hydrating. Combine premium with waitForSelector:
Reader waits for .product-price to appear before capturing, ensuring you see the fully-rendered post-hydration DOM.

​
Detecting thin results in code

Set a minimum content length and flag anything below it:
The exact threshold depends on the type of content you’re scraping. A product detail page under 500 chars is almost certainly a block; a news headline page might legitimately be shorter.

​
Next