Skip to content

Rate limits and concurrency

Credits

Crawl jobs consume credits based on the number and type of targets:

Job type Credits per target
Search (type: "search") 5 per keyword
Product details (type: "details") 2 per ASIN or URL

Creating or retrying a job reserves credits. If the balance is insufficient, the request fails with 402 INSUFFICIENT_CREDITS. Check current plan and balance in the dashboard.

Queue behavior

When you create a job, AzScraper validates the request, records the job, and adds it to the crawl queue. Workers process jobs asynchronously, so a successful response means the job is queued, not that the crawl has finished.

Poll GET /v1/key/jobs/{jobId} every few seconds until the status is completed, failed, or cancelled. Wait times depend on worker capacity and current workload. Scheduled jobs use the same queue.

Request throttling

Project API keys are throttled according to the organization's plan:

Plan Request limit
Starter 20 requests per second
Growth 100 requests per second

These limits use fixed one-second windows and are shared across all API keys owned by the same user. Creating additional keys does not increase that user's allowance. Users in the same organization have separate counters, while the limit tier comes from the organization's plan. Other plans use their configured per-minute limits, also counted per user.

When rate limiting is enabled, authenticated API-key responses include these headers. The API also exposes them to browser clients through CORS:

Header Meaning
X-RateLimit-Limit Maximum requests allowed in the current window
X-RateLimit-Remaining Requests left in the current window
X-RateLimit-Reset Seconds until the current window resets

When the limit is exceeded, the API returns 429 RATE_LIMITED and a Retry-After header with the number of seconds to wait. Retry transient failures with exponential backoff and jitter, and honor Retry-After when present.

Client patterns

  • Group several targets in one crawl when practical.
  • Poll with increasing intervals, for example 2 seconds, then 5 seconds, then 10 seconds.
  • Fetch result rows or export the data after the job completes. Do not repeatedly request results while it is still running.
  • Do not retry job creation automatically after 402; check the balance in the dashboard first.
  • Use the delivery ID to identify and ignore duplicate webhook deliveries.
  • Get data — create jobs, poll status, and export results.
  • Webhooks — receive event notifications without frequent polling.
  • Error taxonomy — INSUFFICIENT_CREDITS and request errors.