Skip to main content
Reader enforces two kinds of limits, and they’re independent. Understanding both will save you from confusing 429 errors.

​
The two limits

A request can hit both, but usually you’ll hit rate limits first if you’re firing sync scrapes in a loop, and concurrency first if you’re kicking off batches or crawls.

​
Limits

​
Rate limit errors

When you exceed your RPM quota, Reader returns 429 with:
Reader also sets the Retry-After response header to the same retryAfterSeconds value. Honor it. The SDKs do this automatically: they back off and retry once the window reopens.

​
Concurrency errors

When you already have N active async jobs (where N is your tier limit) and try to create another, you get 429 with a different code:
concurrency_limited doesn’t include a Retry-After because Reader can’t predict when your existing jobs will finish. Poll them, cancel stale ones, or wait for your webhook.

​
What counts as an active job

A job is “active” as long as its status is queued or processing. Once a job reaches completed, failed, or cancelled, it no longer counts against your concurrency limit, even though it stays in your history. Sync scrapes (single-URL POST /v1/read) are not jobs. They don’t count toward concurrency.

​
Per-key rate limits

You can set a custom RPM override on individual API keys, useful when you want different keys to share a workspace quota but cap each one. This is managed in the dashboard.

​
Strategies for hitting limits

  • Use batch instead of a loop. One POST /v1/read with 1,000 URLs counts as one request toward your RPM and one job toward your concurrency, instead of 1,000 sync scrapes firing at your rate limit.
  • Wait on webhooks, not polling. Polling GET /v1/jobs/{id} every second is 60 RPM per job. Subscribe a webhook instead.
  • Stagger job creation. If you need to run 100 crawls, don’t kick them off in parallel. Queue them yourself and start new ones as old ones finish.
See the Rate limits guide for code patterns.

​
Next