> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reader.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Scrape, batch, crawl, or discover

> Unified endpoint for all read operations. Pass `url` for a single scrape, `urls` for a batch, or `url` + `maxPages` to crawl a site.

`POST /v1/read` is the unified content-extraction endpoint. It auto-detects the operation from the body:

* **Single `url`** - synchronous scrape, returned immediately in the response
* **Multiple `urls`** - async batch job
* **`url` + `maxDepth` or `maxPages`** - async crawl job (discover + scrape)
* **`url` + `maxDepth` or `maxPages` + `scrape: false`** - async discover job (URLs only, no content)

See [The read primitive](/home/concepts/read-primitive) for a narrative overview.

## Proxy mode

Set `proxyMode` to `"standard"` (1 credit, fast, default) or `"premium"` (3 credits, residential proxies, bypasses bot walls). The response metadata tells you which mode ran. See [Proxy modes](/home/concepts/proxy-modes).

## Concurrency

Set `maxConcurrency` to limit how many browser slots a batch or crawl job uses. Prevents one large job from consuming all your plan's concurrent browser slots. Capped to your plan's limit.

Check your current usage with [Queue status](/api-reference/account/queue-status).

## Extract

Add the `extract` parameter to pull structured data from the page alongside markdown. Provide a JSON Schema, a shorthand schema, a natural language prompt, or both.

```bash theme={null}
curl -X POST https://api.reader.dev/v1/read \
  -H "x-api-key: $READER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/product",
    "extract": {
      "schema": { "title": "string", "price": "number", "in_stock": "boolean" }
    }
  }'
```

The response includes an `extracted` field with the structured data and extraction metadata. Adds 2 credits to the scrape cost. Only supported for single-URL scrapes (not batch or crawl). See [Extract](/home/concepts/extract) for details.

## Idempotency

Pass an `x-idempotency-key` header to deduplicate retried POSTs. Reader caches the original response for 24 hours and returns it verbatim on any subsequent request with the same key.

```bash theme={null}
curl -X POST https://api.reader.dev/v1/read \
  -H "x-api-key: $READER_KEY" \
  -H "x-idempotency-key: batch-2026-04-04-run-1" \
  -H "Content-Type: application/json" \
  -d '{ "urls": ["https://example.com/a", "https://example.com/b"] }'
```

## Discover mode

Set `scrape` to `false` on a crawl request to discover URLs without extracting content. Results contain URL, title, and description only. Costs 1 credit per discover job regardless of how many URLs are found.

```bash theme={null}
curl -X POST https://api.reader.dev/v1/read \
  -H "x-api-key: $READER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "maxDepth": 3,
    "maxPages": 500,
    "scrape": false
  }'
```

See [Scrape, crawl, and discover](/home/concepts/scrape-vs-crawl) for usage patterns.


## OpenAPI

````yaml openapi.json POST /v1/read
openapi: 3.1.0
info:
  title: Reader API
  version: 1.0.0
  description: >-
    Reader turns any URL into clean markdown for LLMs. One endpoint (`POST
    /v1/read`) handles scrape, batch, and crawl operations. See
    https://reader.dev/docs for concepts and guides.
servers:
  - url: https://api.reader.dev
    description: Production
security: []
tags:
  - name: Read
    description: Scrape, batch, and crawl operations
  - name: Jobs
    description: Async job management
  - name: Webhooks
    description: Event subscriptions
  - name: Account
    description: API keys, credits, and usage
paths:
  /v1/read:
    post:
      tags:
        - Read
      summary: Scrape, batch, or crawl
      description: >-
        Unified endpoint for all read operations. Pass `url` for a single
        scrape, `urls` for a batch, or `url` + `maxPages` to crawl a site.
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/ReadRequest'
      responses:
        '200':
          description: Sync scrape completed
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ScrapeResponse'
              example:
                success: true
                data:
                  url: https://example.com
                  markdown: >-
                    # Example Domain


                    This domain is for use in illustrative examples in
                    documents. You may use this domain in literature without
                    prior coordination or asking for permission.


                    [More information...](https://www.iana.org/domains/example)
                  metadata:
                    title: Example Domain
                    description: null
                    statusCode: 200
                    duration: 487
                    cached: false
                    proxyMode: standard
                    scrapedAt: '2026-04-04T12:00:00Z'
              examples:
                scrape:
                  summary: Single-URL scrape
                  value:
                    success: true
                    data:
                      url: https://example.com
                      markdown: >-
                        # Example Domain


                        This domain is for use in illustrative examples in
                        documents. You may use this domain in literature without
                        prior coordination or asking for permission.


                        [More
                        information...](https://www.iana.org/domains/example)
                      metadata:
                        title: Example Domain
                        description: null
                        statusCode: 200
                        duration: 487
                        cached: false
                        proxyMode: standard
                        scrapedAt: '2026-04-04T12:00:00Z'
        '201':
          description: Async job created (batch or crawl)
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/JobCreatedResponse'
              example:
                success: true
                data:
                  id: job_9fba2
                  status: queued
                  mode: batch
                  total: 3
                  completed: 0
                  creditsUsed: 0
                  createdAt: '2026-04-04T12:00:00Z'
              examples:
                jobCreated:
                  summary: Batch or crawl job accepted
                  value:
                    success: true
                    data:
                      id: job_9fba2
                      status: queued
                      mode: batch
                      total: 3
                      completed: 0
                      creditsUsed: 0
                      createdAt: '2026-04-04T12:00:00Z'
        '400':
          description: Invalid request
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorEnvelope'
              example:
                success: false
                error:
                  code: invalid_request
                  message: Invalid request body
                  details:
                    issues:
                      - path: url
                        code: invalid_string
                        message: Invalid url
                  docsUrl: https://reader.dev/docs/home/concepts/errors#invalid-request
              examples:
                invalid_request:
                  summary: Invalid request
                  value:
                    success: false
                    error:
                      code: invalid_request
                      message: Invalid request body
                      details:
                        issues:
                          - path: url
                            code: invalid_string
                            message: Invalid url
                      docsUrl: >-
                        https://reader.dev/docs/home/concepts/errors#invalid-request
        '401':
          description: Unauthenticated
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorEnvelope'
              example:
                success: false
                error:
                  code: unauthenticated
                  message: Missing or invalid API key
                  docsUrl: https://reader.dev/docs/home/concepts/errors#unauthenticated
              examples:
                unauthenticated:
                  summary: Unauthenticated
                  value:
                    success: false
                    error:
                      code: unauthenticated
                      message: Missing or invalid API key
                      docsUrl: >-
                        https://reader.dev/docs/home/concepts/errors#unauthenticated
        '402':
          description: Insufficient credits
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorEnvelope'
              example:
                success: false
                error:
                  code: insufficient_credits
                  message: >-
                    You need 50 credits but only 10 are available. Wait for your
                    credits to reset at the next billing cycle.
                  details:
                    required: 50
                    available: 10
                    resetAt: '2026-05-01T00:00:00Z'
                  docsUrl: >-
                    https://reader.dev/docs/home/concepts/errors#insufficient-credits
              examples:
                insufficient_credits:
                  summary: Insufficient credits
                  value:
                    success: false
                    error:
                      code: insufficient_credits
                      message: >-
                        You need 50 credits but only 10 are available. Wait for
                        your credits to reset at the next billing cycle.
                      details:
                        required: 50
                        available: 10
                        resetAt: '2026-05-01T00:00:00Z'
                      docsUrl: >-
                        https://reader.dev/docs/home/concepts/errors#insufficient-credits
        '403':
          description: URL blocked
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorEnvelope'
              example:
                success: false
                error:
                  code: url_blocked
                  message: 'URL blocked: Resolves to private IP'
                  details:
                    url: https://10.0.0.1
                    reason: Resolves to private IP
                  docsUrl: https://reader.dev/docs/home/concepts/errors#url-blocked
              examples:
                url_blocked:
                  summary: URL blocked
                  value:
                    success: false
                    error:
                      code: url_blocked
                      message: 'URL blocked: Resolves to private IP'
                      details:
                        url: https://10.0.0.1
                        reason: Resolves to private IP
                      docsUrl: https://reader.dev/docs/home/concepts/errors#url-blocked
        '429':
          description: Rate limited
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorEnvelope'
              example:
                success: false
                error:
                  code: rate_limited
                  message: Rate limit of 60 requests per 60s exceeded. Retry after 12s.
                  details:
                    limit: 60
                    windowSeconds: 60
                    retryAfterSeconds: 12
                  docsUrl: https://reader.dev/docs/home/concepts/errors#rate-limited
              examples:
                rate_limited:
                  summary: Rate limited
                  value:
                    success: false
                    error:
                      code: rate_limited
                      message: >-
                        Rate limit of 60 requests per 60s exceeded. Retry after
                        12s.
                      details:
                        limit: 60
                        windowSeconds: 60
                        retryAfterSeconds: 12
                      docsUrl: >-
                        https://reader.dev/docs/home/concepts/errors#rate-limited
        '502':
          description: Upstream unavailable
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorEnvelope'
              example:
                success: false
                error:
                  code: upstream_unavailable
                  message: Upstream service is unavailable
                  docsUrl: >-
                    https://reader.dev/docs/home/concepts/errors#upstream-unavailable
              examples:
                upstream_unavailable:
                  summary: Upstream unavailable
                  value:
                    success: false
                    error:
                      code: upstream_unavailable
                      message: Upstream service is unavailable
                      docsUrl: >-
                        https://reader.dev/docs/home/concepts/errors#upstream-unavailable
        '504':
          description: Scrape timed out
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorEnvelope'
              example:
                success: false
                error:
                  code: scrape_timeout
                  message: Scrape exceeded 30000ms timeout
                  details:
                    timeoutMs: 30000
                  docsUrl: https://reader.dev/docs/home/concepts/errors#scrape-timeout
              examples:
                scrape_timeout:
                  summary: Scrape timed out
                  value:
                    success: false
                    error:
                      code: scrape_timeout
                      message: Scrape exceeded 30000ms timeout
                      details:
                        timeoutMs: 30000
                      docsUrl: >-
                        https://reader.dev/docs/home/concepts/errors#scrape-timeout
      security:
        - ApiKeyAuth: []
components:
  schemas:
    ReadRequest:
      type: object
      properties:
        url:
          type: string
          format: uri
          example: https://example.com
        urls:
          type: array
          items:
            type: string
            format: uri
          minItems: 1
          maxItems: 1000
        formats:
          type: array
          items:
            type: string
            enum:
              - markdown
              - html
              - screenshot
          description: Content formats to include in the response.
          example:
            - markdown
        onlyMainContent:
          type: boolean
          description: 'Strip navigation, footers, and boilerplate. Default: true.'
        includeTags:
          type: array
          items:
            type: string
        excludeTags:
          type: array
          items:
            type: string
        waitForSelector:
          type: string
        timeoutMs:
          type: integer
          minimum: 1000
          maximum: 120000
        proxyMode:
          $ref: '#/components/schemas/ProxyMode'
        batchConcurrency:
          type: integer
          minimum: 1
          maximum: 20
        maxConcurrency:
          type: integer
          minimum: 1
          description: >-
            Max concurrent browser slots for this request. Capped to your plan's
            limit.
        maxDepth:
          type: integer
          minimum: 1
          maximum: 10
          description: Crawl depth (when crawling). Omit for single-URL scrape.
        maxPages:
          type: integer
          minimum: 1
          maximum: 10000
          description: Maximum pages to discover during crawl.
        scrape:
          type: boolean
          description: >-
            When false, crawl discovers URLs without scraping their content
            (discover mode). Only applies when maxDepth or maxPages is set.
            Default: true.
        cache:
          type: boolean
          description: 'Reuse cached content within TTL. Default: true.'
        webhook:
          type: object
          properties:
            url:
              type: string
              format: uri
            events:
              type: array
              items:
                type: string
            secret:
              type: string
              minLength: 16
              maxLength: 256
          required:
            - url
    ScrapeResponse:
      type: object
      properties:
        success:
          type: boolean
          enum:
            - true
        data:
          $ref: '#/components/schemas/ScrapeResult'
      required:
        - success
        - data
      example:
        success: true
        data:
          url: https://example.com
          markdown: >-
            # Example Domain


            This domain is for use in illustrative examples in documents. You
            may use this domain in literature without prior coordination or
            asking for permission.


            [More information...](https://www.iana.org/domains/example)
          metadata:
            title: Example Domain
            description: null
            statusCode: 200
            duration: 487
            cached: false
            proxyMode: standard
            scrapedAt: '2026-04-04T12:00:00Z'
    JobCreatedResponse:
      type: object
      properties:
        success:
          type: boolean
          enum:
            - true
        data:
          $ref: '#/components/schemas/JobSummary'
      required:
        - success
        - data
      example:
        success: true
        data:
          id: job_9fba2
          status: queued
          mode: batch
          total: 3
          completed: 0
          creditsUsed: 0
          createdAt: '2026-04-04T12:00:00Z'
    ErrorEnvelope:
      type: object
      properties:
        success:
          type: boolean
          enum:
            - false
        error:
          type: object
          properties:
            code:
              $ref: '#/components/schemas/ErrorCode'
            message:
              type: string
            details:
              type: object
              additionalProperties: {}
            docsUrl:
              type: string
              format: uri
          required:
            - code
            - message
            - docsUrl
      required:
        - success
        - error
    ProxyMode:
      type: string
      enum:
        - standard
        - premium
      description: >-
        Proxy mode for the scrape. `standard` uses datacenter proxies (default,
        1 credit); `premium` uses residential proxies for better bot bypass (3
        credits).
      example: standard
    ScrapeResult:
      type: object
      properties:
        url:
          type: string
          format: uri
        markdown:
          type: string
        html:
          type: string
        screenshot:
          type: string
          description: >-
            Base64-encoded PNG screenshot of the full page. Only present when
            "screenshot" is included in formats.
        metadata:
          $ref: '#/components/schemas/ScrapeMetadata'
      required:
        - url
        - metadata
    JobSummary:
      type: object
      properties:
        id:
          type: string
        status:
          type: string
          enum:
            - queued
            - processing
            - completed
            - failed
            - cancelled
        mode:
          type: string
          enum:
            - scrape
            - batch
            - crawl
        total:
          type: integer
        completed:
          type: integer
        creditsUsed:
          type: integer
        createdAt:
          type: string
          format: date-time
      required:
        - id
        - status
        - mode
        - total
        - completed
        - creditsUsed
        - createdAt
    ErrorCode:
      type: string
      enum:
        - invalid_request
        - unauthenticated
        - insufficient_credits
        - url_blocked
        - not_found
        - conflict
        - rate_limited
        - concurrency_limited
        - internal_error
        - upstream_unavailable
        - scrape_timeout
    ScrapeMetadata:
      type: object
      properties:
        title:
          type:
            - string
            - 'null'
        description:
          type:
            - string
            - 'null'
        statusCode:
          type: integer
        duration:
          type: integer
          description: Scrape duration in ms.
        cached:
          type: boolean
        proxyMode:
          $ref: '#/components/schemas/ResolvedProxyMode'
        scrapedAt:
          type: string
          format: date-time
      required:
        - duration
        - cached
        - scrapedAt
    ResolvedProxyMode:
      type: string
      enum:
        - standard
        - premium
      description: >-
        The proxy mode that actually ran: `standard` (datacenter) or `premium`
        (residential). Omitted on cache hits.
  securitySchemes:
    ApiKeyAuth:
      type: apiKey
      in: header
      name: x-api-key

````