davila7--claude-code-templates/cli-tool/components/skills/web-data/bright-data-best-practices/references/web-scraper-api.md

Web Scraper API Reference

Table of Contents

• Overview
• Authentication
• Choosing Sync vs Async
• Synchronous Requests
• Asynchronous Requests
• Monitor Progress
• Download Results
• Scraper Types
• Output Formats
• Billing Model
• Best Practices

---

Overview

Bright Data Web Scraper API provides pre-built scrapers ("datasets") for 100+ popular websites including Amazon, LinkedIn, Instagram, TikTok, YouTube, Facebook, and more. You provide input (URLs or keywords), and receive clean structured JSON/CSV data without writing any scraping logic.

Supported domains include: Amazon, eBay, Walmart, LinkedIn, Instagram, TikTok, YouTube, Facebook, Reddit, Twitter/X, Crunchbase, ZoomInfo, and many more.

---

Authentication

export BRIGHTDATA_API_KEY="your-api-key"

Get your API key from: https://brightdata.com/cp/setting/users

All requests use Bearer token authentication:

Authorization: Bearer YOUR_API_KEY

---

Choosing Sync vs Async

Factor	Synchronous (/scrape)	Asynchronous (/trigger)
Input size	Up to 20 URLs	Any size — built for bulk
Response time	Immediate (within 1 min)	Background job — poll for completion
Timeout behavior	Returns 202 + snapshot_id if >1 min	N/A — always async
Best for	Real-time single lookups	Large batches, scheduled jobs

---

Synchronous Requests

Endpoint: POST https://api.brightdata.com/datasets/v3/scrape

Results are returned immediately in the response body.

Request Parameters

Parameter	Type	Required	Description
dataset_id	string	Yes	Identifies which scraper to use (from the Scraper Library)
format	string	No	Output format: json (default), ndjson, jsonl, or csv
custom_output_fields	string	No	Pipe-separated field names to filter output (e.g., url|title|price)
include_errors	boolean	No	Include error reporting in results

Request Body

{
  "input": [
    { "url": "https://www.amazon.com/dp/B09X7M8TBQ" },
    { "url": "https://www.amazon.com/dp/B0B7CTCPKN" }
  ]
}

Python Example

import requests

response = requests.post(
    "https://api.brightdata.com/datasets/v3/scrape",
    params={
        "dataset_id": "gd_l7q7dkf244hwjntr0",  # Amazon product dataset_id
        "format": "json"
    },
    headers={
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json"
    },
    json={
        "input": [
            {"url": "https://www.amazon.com/dp/B09X7M8TBQ"},
            {"url": "https://www.amazon.com/dp/B0B7CTCPKN"}
        ]
    }
)

if response.status_code == 200:
    data = response.json()
    for item in data:
        print(item["title"], item["price"])
elif response.status_code == 202:
    # Processing exceeded 1-minute timeout — use snapshot_id for async retrieval
    snapshot_id = response.json().get("snapshot_id")
    print(f"Processing... poll with snapshot_id: {snapshot_id}")

const response = await fetch(
  "https://api.brightdata.com/datasets/v3/scrape?dataset_id=gd_l7q7dkf244hwjntr0&format=json",
  {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${API_KEY}`,
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      input: [
        { url: "https://www.amazon.com/dp/B09X7M8TBQ" }
      ]
    })
  }
);

if (response.status === 200) {
  const data = await response.json();
  console.log(data);
} else if (response.status === 202) {
  const { snapshot_id } = await response.json();
  // Poll for completion
}

Response Codes (Sync)

Code	Meaning
200 OK	Data returned directly in response body
202 Accepted	Processing exceeded 1-minute timeout — response includes snapshot_id for async retrieval

---

Asynchronous Requests

Use /trigger for large batches or when you don't need an immediate response.

Endpoint: POST https://api.brightdata.com/datasets/v3/trigger

Request Parameters (same as sync plus)

Parameter	Type	Required	Description
dataset_id	string	Yes	Scraper identifier
format	string	No	json, ndjson, jsonl, csv
custom_output_fields	string	No	Pipe-separated field names
include_errors	boolean	No	Include errors in output
notify	string	No	Webhook URL to receive completion notification
output	object	No	External storage delivery config (S3, GCS, etc.)

Python Example (Trigger + Poll)

import requests
import time

# Step 1: Trigger the job
trigger_response = requests.post(
    "https://api.brightdata.com/datasets/v3/trigger",
    params={
        "dataset_id": "gd_l7q7dkf244hwjntr0",
        "format": "json"
    },
    headers={
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json"
    },
    json={
        "input": [
            {"url": "https://www.amazon.com/dp/B09X7M8TBQ"},
            # ... hundreds more URLs
        ]
    }
)
snapshot_id = trigger_response.json()["snapshot_id"]

# Step 2: Poll until ready
while True:
    progress = requests.get(
        f"https://api.brightdata.com/datasets/v3/progress/{snapshot_id}",
        headers={"Authorization": f"Bearer {API_KEY}"}
    )
    status = progress.json()["status"]
    print(f"Status: {status}")

    if status == "ready":
        break
    elif status == "failed":
        raise Exception("Scraping job failed")

    time.sleep(10)

# Step 3: Download results
results = requests.get(
    f"https://api.brightdata.com/datasets/v3/snapshot/{snapshot_id}",
    params={"format": "json"},
    headers={"Authorization": f"Bearer {API_KEY}"}
)
data = results.json()

---

Monitor Progress

Endpoint: GET https://api.brightdata.com/datasets/v3/progress/{snapshot_id}

response = requests.get(
    f"https://api.brightdata.com/datasets/v3/progress/{snapshot_id}",
    headers={"Authorization": f"Bearer {API_KEY}"}
)
status = response.json()["status"]

Status Values

Status	Description
starting	Job initialization
running	Data collection in progress
ready	Results available for download
failed	Job failed

Error Responses

Code	Meaning
401	Missing or invalid API key
404	Snapshot ID not found

---

Download Results

Endpoint: GET https://api.brightdata.com/datasets/v3/snapshot/{snapshot_id}

response = requests.get(
    f"https://api.brightdata.com/datasets/v3/snapshot/{snapshot_id}",
    params={"format": "json"},
    headers={"Authorization": f"Bearer {API_KEY}"}
)
data = response.json()

Snapshot Lifecycle

• Snapshots are available for 30 days after collection
• Download in JSON, NDJSON, JSONL, or CSV format

---

Scraper Types

The Scraper Library contains pre-built scrapers organized by type:

PDP Scrapers (Product/Profile Detail)

• Accept one or more URLs
• Return detailed data for each URL
• Example: Amazon product page → price, title, reviews, specs

Discovery Scrapers

• Accept search terms, keywords, or category URLs
• Return lists of results to explore
• Example: Amazon search → list of matching products

Finding Dataset IDs

1. Go to https://brightdata.com/cp/datasets (Scraper Library)
2. Select the platform and data type you need
3. Each scraper has a unique dataset_id shown in the API reference

---

Output Formats

Format	Description
json	Standard JSON array (default)
ndjson	Newline-delimited JSON (one object per line) — good for streaming large results
jsonl	Same as ndjson
csv	CSV format

Custom Output Fields

Filter returned fields to reduce payload size:

params = {
    "dataset_id": "gd_l7q7dkf244hwjntr0",
    "format": "json",
    "custom_output_fields": "url|title|price|rating"  # pipe-separated
}

Nested fields use dot notation: about.updated_on

---

Billing Model

Scenario	Billing
Standard	Per delivered record — starting from $0.70/1,000 records
Failed due to user input error	Billable — resources were consumed processing the invalid input
Sync timeout (202) → async retrieval	Single charge for the records, not double
Real-time mode	Up to 20 URL inputs per call

Data retention: Collected snapshots available for 30 days.

---

Best Practices

1. Use sync for ≤20 URLs, async for larger batches

Sync is simpler for small jobs. For anything larger, use /trigger with polling.

if len(urls) <= 20:
    # Use /scrape for immediate results
    endpoint = "https://api.brightdata.com/datasets/v3/scrape"
else:
    # Use /trigger for bulk
    endpoint = "https://api.brightdata.com/datasets/v3/trigger"

2. Handle 202 responses in sync mode

If your sync request takes >1 minute, you'll get a 202 with snapshot_id. Always handle this case:

if response.status_code == 202:
    snapshot_id = response.json()["snapshot_id"]
    # Fall through to polling logic

3. Use webhooks for production async workflows

Polling is fine for development. In production, configure notify URL to receive push notifications:

json={
    "input": [...],
    "notify": "https://your-server.com/webhook/brightdata"
}

4. Use custom_output_fields to reduce payload

Only request fields you need. This reduces bandwidth and response size:

params={"custom_output_fields": "url|title|price|availability"}

5. Use ndjson format for large result sets

NDJSON is more memory-efficient for large datasets since you can stream-process line by line:

for line in response.iter_lines():
    record = json.loads(line)
    process(record)

6. Check data retention (30 days)

Download your snapshots within 30 days. After that, the data is gone.

7. Validate inputs before submitting

Submitting invalid URLs/inputs that fail due to user error is still billable. Validate URLs before sending:

from urllib.parse import urlparse

def is_valid_url(url: str) -> bool:
    parsed = urlparse(url)
    return parsed.scheme in ("http", "https") and bool(parsed.netloc)

urls = [u for u in raw_urls if is_valid_url(u)]

8. Use delivery to external storage for large jobs

Instead of downloading via the API, configure delivery to S3/GCS in the trigger request for large datasets:

json={
    "input": [...],
    "output": {
        "type": "s3",
        "bucket": "your-bucket",
        "prefix": "brightdata/results/"
    }
}