Integrate AzScraper with AI agents¶

Use the HTTP API when your application should control each request. Use MCP when an AI agent should discover AzScraper tools and choose when to call them. Both options use the same project-scoped API key and project data.

Before you connect¶

Create a project and a project API key in the AzScraper dashboard.
Keep the key in your AI client’s secret store or server-side environment. Do not paste it into prompts, source control, or browser code.
Choose one of the connection methods below.

The key is scoped to its project. MCP tools infer the project from the key, so an agent does not need to provide a project ID.

Connect over MCP¶

AzScraper provides a remote MCP server at:

https://api.azscraper.com/mcp

Configure your MCP client to use the Streamable HTTP transport and send the project key as a Bearer token. Client configuration formats vary; this is the common remote-server shape:

{
  "mcpServers": {
    "azscraper": {
      "type": "http",
      "url": "https://api.azscraper.com/mcp",
      "headers": {
        "Authorization": "Bearer <project-api-key>"
      }
    }
  }
}

After connecting, the client discovers the available tools. All calls are authenticated with the configured API key.

Available tools¶

Tool	What it does	Important inputs
`create_crawl`	Starts a search or product-details crawl. This consumes project credits.	`type` (`search` or `details`), `targets`, optional `config` (`marketplace`, `lang`)
`list_crawls`	Lists crawl jobs for the key’s project.	Optional `status`, `q`, `from`, `to`, `limit`, `offset`
`get_crawl`	Returns a crawl’s status and metadata.	`jobId`
`get_crawl_results`	Returns parsed result rows for a crawl.	`jobId`
`get_project_analytics`	Summarizes crawl activity for the key’s project.	Optional `from`, `to` date strings

Review the targets before allowing an agent to call create_crawl: each crawl uses credits and may enqueue work. MCP calls are subject to the same per-user rate limits as API-key requests.

MCP request flow¶

sequenceDiagram
    autonumber
    actor Agent as AI agent
    participant Client as MCP client
    participant MCP as AzScraper MCP server
    participant Crawl as Crawl service
    Agent->>Client: Ask to research products
    Client->>MCP: Streamable HTTP initialize
    Note over Client,MCP: Authorization: Bearer project API key
    MCP-->>Client: Server capabilities
    Client->>MCP: tools/list
    MCP-->>Client: AzScraper tool definitions
    Agent->>Client: Choose create_crawl
    Client->>MCP: tools/call create_crawl
    MCP->>Crawl: Create job in the key's project
    Crawl-->>MCP: Job ID and initial status
    MCP-->>Client: Tool result
    Client-->>Agent: Job details
    Agent->>Client: Ask for status or results
    Client->>MCP: tools/call get_crawl or get_crawl_results
    MCP-->>Client: Status or result rows
    Client-->>Agent: Crawl data

Connect with the HTTP API¶

The REST API is a good fit for deterministic workflows, custom polling, exports, schedules, and webhook-driven applications. Set the API origin and key in your runtime:

export API_BASE="https://api.azscraper.com"
export API_KEY="<project-api-key>"
export PROJECT_ID="<project-id>"

Create a search crawl:

curl -sS -X POST "$API_BASE/v1/key/projects/$PROJECT_ID/jobs" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"type":"search","targets":["wireless earbuds"],"config":{"marketplace":"US","lang":"en-US"}}'

Then poll the returned job ID and fetch its results:

curl -sS "$API_BASE/v1/key/jobs/$JOB_ID" \
  -H "Authorization: Bearer $API_KEY"

curl -sS "$API_BASE/v1/key/jobs/$JOB_ID/results" \
  -H "Authorization: Bearer $API_KEY"

See API overview for the full endpoint list, Get started for request examples, and Rate limits & concurrency for request limits and polling guidance.

Protect your integration¶

Give each agent or integration its own API key so you can revoke it independently.
Store keys in a secret manager or the client’s secure credential store.
Never include a live key in a prompt, model context, frontend bundle, or log.
Use read-only tools when an agent only needs to inspect data. Treat create_crawl as a credit-consuming action.
Rotate a key immediately if it is exposed.