davila7--claude-code-templates/cli-tool/components/skills/web-data/bright-data-best-practices/references/browser-api.md

Browser API (Scraping Browser) Reference

Table of Contents

• Overview
• Authentication
• Connection Strings
• Supported Frameworks
• Session Rules
• Quick Start Examples
• Custom CDP Functions
• Geolocation Targeting
• Bandwidth Optimization
• CAPTCHA Handling
• Debugging
• Premium Domains
• Error Codes
• Billing Model
• Best Practices
• Anti-Patterns

---

Overview

Bright Data Browser API (also called Scraping Browser) is a managed cloud browser service. It handles proxy management, fingerprinting, CAPTCHA solving, and bot detection bypass automatically. Use it when you need full browser automation: clicking, scrolling, form filling, JavaScript execution, or working with SPAs.

When to use Browser API vs other APIs:

Need	Use
Simple HTTP scraping, no interaction	Web Unlocker
Google/Bing search results	SERP API
Structured data from known platforms	Web Scraper API
Click, scroll, fill forms, run JS	Browser API
Intercept XHR/fetch calls from a page	Browser API
Handle complex anti-bot that requires browser	Browser API
Puppeteer/Playwright/Selenium automation	Browser API

---

Authentication

Get credentials from your Browser API zone's Overview tab in the Control Panel.

• Username: brd-customer-{CUSTOMER_ID}-zone-{ZONE_NAME}
• Password: Your zone password

export BROWSER_AUTH="brd-customer-CUSTOMER_ID-zone-ZONE_NAME:PASSWORD"

---

Connection Strings

Framework	Connection Type	Endpoint
Playwright	WebSocket	wss://${AUTH}@brd.superproxy.io:9222
Puppeteer	WebSocket	wss://${AUTH}@brd.superproxy.io:9222
Selenium	HTTPS	https://${AUTH}@brd.superproxy.io:9515

Replace ${AUTH} with username:password.

Critical: Wrong port = 407 error. Playwright/Puppeteer = port 9222. Selenium = port 9515.

---

Supported Frameworks

• Node.js: Puppeteer, Playwright, Selenium WebDriver
• Python: Playwright, Selenium
• C# (.NET): PuppeteerSharp, Playwright, Selenium
• Other languages: Any language with CDP or WebDriver support (Ruby, Go, Java, etc.)

---

Session Rules

• One initial navigation per session. After page.goto(url), you can interact (click, scroll, evaluate) within the same page, but cannot navigate to a different URL in the same session. Start a new session for a new target.
• Idle timeout: Sessions inactive for 5+ minutes automatically close.
• Maximum duration: Sessions cannot exceed 30 minutes.
• Password entry: Disabled by default to protect non-public data. Contact Bright Data to enable.

---

Quick Start Examples

Playwright (Node.js)

const { chromium } = require("playwright-core");

const AUTH = process.env.BROWSER_AUTH || "brd-customer-CUSTOMER_ID-zone-ZONE_NAME:PASSWORD";
const TARGET_URL = "https://example.com";

(async () => {
  const browser = await chromium.connectOverCDP(
    `wss://${AUTH}@brd.superproxy.io:9222`
  );
  const page = await browser.newPage();

  // Set navigation timeout to 2 minutes (recommended for complex unlocking)
  page.setDefaultNavigationTimeout(120000);

  await page.goto(TARGET_URL, { waitUntil: "domcontentloaded" });
  const html = await page.content();
  console.log(html);

  await browser.close();
})();

Playwright (Python)

import asyncio
from playwright.async_api import async_playwright

AUTH = "brd-customer-CUSTOMER_ID-zone-ZONE_NAME:PASSWORD"
TARGET_URL = "https://example.com"

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.connect_over_cdp(
            f"wss://{AUTH}@brd.superproxy.io:9222"
        )
        page = await browser.new_page()
        page.set_default_navigation_timeout(120000)

        await page.goto(TARGET_URL, wait_until="domcontentloaded")
        html = await page.content()
        print(html)

        await browser.close()

asyncio.run(main())

Puppeteer (Node.js)

const puppeteer = require("puppeteer-core");

const AUTH = process.env.BROWSER_AUTH || "brd-customer-CUSTOMER_ID-zone-ZONE_NAME:PASSWORD";

(async () => {
  const browser = await puppeteer.connect({
    browserWSEndpoint: `wss://${AUTH}@brd.superproxy.io:9222`
  });
  const page = await browser.newPage();
  page.setDefaultNavigationTimeout(120000);

  await page.goto("https://example.com", { waitUntil: "domcontentloaded" });
  const html = await page.content();
  console.log(html);

  await browser.close();
})();

Selenium (Python)

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

AUTH = "brd-customer-CUSTOMER_ID-zone-ZONE_NAME:PASSWORD"

options = Options()
options.add_argument("--ignore-certificate-errors")

driver = webdriver.Remote(
    command_executor=f"https://{AUTH}@brd.superproxy.io:9515",
    options=options
)
driver.get("https://example.com")
print(driver.page_source)
driver.quit()

---

Custom CDP Functions

Bright Data extends standard CDP with specialized commands. Use page.evaluate or client.send to invoke them.

CAPTCHA Handling

Captcha.setAutoSolve

Control automatic CAPTCHA solving.

// Disable auto-solve (use manual control instead)
const client = await page.target().createCDPSession();
await client.send("Captcha.setAutoSolve", { autoSolve: false });

Captcha.solve

Manually trigger CAPTCHA solving and wait for completion.

const client = await page.target().createCDPSession();
const result = await client.send("Captcha.solve", { timeout: 30000 });
console.log(result.status); // "solved" | "not_detected" | "timeout"

# Python equivalent
client = await page.context.new_cdp_session(page)
result = await client.send("Captcha.solve", {"timeout": 30000})
print(result["status"])

Geolocation

Proxy.setLocation

Set precise proxy location by coordinates. Must be called before navigating to the site.

const client = await page.target().createCDPSession();
await client.send("Proxy.setLocation", {
  latitude: 37.7749,
  longitude: -122.4194,
  distance: 10,   // Search radius in km
  strict: true    // true = only peers within distance; false = expand if none found
});
await page.goto("https://example.com");

client = await page.context.new_cdp_session(page)
await client.send("Proxy.setLocation", {
    "latitude": 37.7749,
    "longitude": -122.4194,
    "distance": 10,
    "strict": True
})
await page.goto("https://example.com")

Session Management

Proxy.useSession

Maintain consistent proxy peer across multiple browsing sessions (same IP continuity).

const client = await page.target().createCDPSession();
await client.send("Proxy.useSession", { sessionId: "my-session-123" });

Device Emulation

Emulation.getSupportedDevices

Get list of available device profiles.

const client = await page.target().createCDPSession();
const { devices } = await client.send("Emulation.getSupportedDevices");
console.log(devices); // ["iPhone 14", "Samsung Galaxy S21", ...]

Emulation.setDevice

Apply a specific device profile (user agent, screen size, touch).

await client.send("Emulation.setDevice", {
  deviceName: "iPhone 14"
});
// Optional: landscape orientation
await client.send("Emulation.setDevice", {
  deviceName: "iPhone 14",
  landscape: true
});

Ad Blocking

Unblocker.enableAdBlock / Unblocker.disableAdBlock

Block/unblock ads to reduce bandwidth on content-heavy sites.

const client = await page.target().createCDPSession();
await client.send("Unblocker.enableAdBlock");
// ... scrape page ...
await client.send("Unblocker.disableAdBlock"); // if needed later

Input Acceleration

Input.type

Faster text input than standard keyboard simulation. Useful for bulk form-filling.

const client = await page.target().createCDPSession();
await client.send("Input.type", {
  text: "search query here",
  selector: "#search-input"
});

File Downloads

Download.*

Control file downloads with content-type filtering.

const client = await page.target().createCDPSession();
await client.send("Download.setDownloadBehavior", {
  behavior: "allow",
  downloadPath: "/tmp/downloads"
});
// Retrieve file as base64
const { data } = await client.send("Download.getDownloadedFile", {
  guid: "download-guid-here"
});

Security / Client Certificates

Browser.addCertificate

Install a client SSL/TLS certificate for authenticated domain access. Certificate is automatically removed when the session ends.

await client.send("Browser.addCertificate", {
  host: "example.com",
  certificate: "base64-encoded-cert",
  privateKey: "base64-encoded-key"
});

Debugging

Page.inspect

Get a Chrome DevTools debugger URL to connect and inspect the live session.

const { url } = await client.send("Page.inspect");
console.log(`DevTools: ${url}`);
// Open in Chrome or use programmatically

---

Geolocation Targeting

Country-Level (via credentials)

Append -country-XX to your username (2-letter ISO code):

const AUTH = "brd-customer-ID-zone-NAME-country-us:PASSWORD";
const browser = await chromium.connectOverCDP(`wss://${AUTH}@brd.superproxy.io:9222`);

EU targeting — automatically routes through 29+ European countries:

const AUTH = "brd-customer-ID-zone-NAME-country-eu:PASSWORD";

Precise Location (via CDP)

Use Proxy.setLocation for city/neighborhood-level targeting. Call before page.goto():

const client = await page.target().createCDPSession();
await client.send("Proxy.setLocation", {
  latitude: 51.5074,   // London
  longitude: -0.1278,
  distance: 5,          // 5km radius
  strict: false         // expand search if no peers found nearby
});
await page.goto("https://example.com");

---

Bandwidth Optimization

Browser sessions are billed by traffic. These techniques reduce costs:

Block Unnecessary Resources

// Puppeteer
await page.setRequestInterception(true);
page.on("request", (req) => {
  const blocked = ["image", "stylesheet", "font", "media"];
  if (blocked.includes(req.resourceType())) {
    req.abort();
  } else {
    req.continue();
  }
});

# Playwright
async def block_resources(route, request):
    blocked = ["image", "stylesheet", "font", "media"]
    if request.resource_type in blocked:
        await route.abort()
    else:
        await route.continue_()

await page.route("**/*", block_resources)

Block Specific URLs (CDP)

const client = await page.target().createCDPSession();
await client.send("Network.setBlockedURLs", {
  urls: [
    "*google-analytics*",
    "*facebook.com/tr*",
    "*.doubleclick.net*"
  ]
});

Enable Ad Blocker

await client.send("Unblocker.enableAdBlock");

Browser Caching

Browser API automatically caches resources across multiple navigations to the same domain within a session. No extra configuration needed.

---

CAPTCHA Handling

Automatic (Default)

CAPTCHA solving is enabled by default. No code needed — the browser solves CAPTCHAs transparently.

Detect and Wait for CAPTCHA

// Navigate and wait for CAPTCHA to be solved automatically
const client = await page.target().createCDPSession();
await page.goto("https://example.com");

// If CAPTCHA appears, explicitly wait for it
const captchaResult = await client.send("Captcha.solve", { timeout: 60000 });
if (captchaResult.status === "solved") {
  console.log("CAPTCHA solved");
}

Disable Auto-Solve

const client = await page.target().createCDPSession();
await client.send("Captcha.setAutoSolve", { autoSolve: false });
// Now you control when/if to solve CAPTCHAs

---

Debugging

Via Control Panel

Navigate to your Browser API zone → Overview tab → Click "Chrome Dev Tools Debugger".

Via CDP Command

const { url } = await client.send("Page.inspect");
// Opens Chrome DevTools for the live session

Programmatic Inspection (Playwright)

Playwright provides a slowMo and debug URL option. Use Page.inspect to get the debug URL and connect with Chrome.

---

Premium Domains

Certain websites classified as "premium" require more Browser API resources:

• Enable during zone creation: toggle "Premium Domains" ON
• Only traffic to premium-classified domains incurs the higher rate
• The list of premium domains updates regularly based on Bright Data's algorithms
• Check current premium domain list in your zone settings

---

Error Codes

Code	Issue	Solution
407	Wrong port	Playwright/Puppeteer = port 9222, Selenium = port 9515
403	Authentication failure	Verify username format and password; confirm you're using a Browser API zone (not proxy zone)
503	Service unavailable (scaling)	Reconnect after 1 minute

---

Billing Model

Factor	Detail
Pricing unit	Traffic-based only (bandwidth consumed)
Time/instance fees	None
Session idle timeout	5 minutes — sessions close automatically
Session max duration	30 minutes
Resource blocking	Reduces bandwidth = reduces cost

---

Best Practices

1. Always set navigation timeout to 2 minutes

Complex anti-bot procedures take time. Default timeouts (30s) are too short.

page.setDefaultNavigationTimeout(120000); // 2 minutes

2. Call Proxy.setLocation before navigation

Location selection must happen before page.goto() to ensure the correct proxy is selected.

// CORRECT: Set location first
await client.send("Proxy.setLocation", { latitude: 37.7749, longitude: -122.4194, distance: 10 });
await page.goto("https://example.com");

// WRONG: Location set after navigation has no effect on proxy selection
await page.goto("https://example.com");
await client.send("Proxy.setLocation", { ... }); // too late

3. Block unnecessary resources to cut bandwidth costs

Images, stylesheets, and fonts are often not needed for data extraction. Block them.

await page.route("**/*.{png,jpg,jpeg,gif,svg,css,woff,woff2}", route => route.abort());

4. Use waitUntil: "domcontentloaded" over networkidle

networkidle waits for all network activity to stop, which is slow and often unnecessary. domcontentloaded is faster and sufficient for most pages.

await page.goto(url, { waitUntil: "domcontentloaded" });

5. Start a fresh session for each new target URL

Since only one initial navigation per session is allowed, create a new browser connection for each independent scraping task.

async function scrapeUrl(url) {
  const browser = await chromium.connectOverCDP(`wss://${AUTH}@brd.superproxy.io:9222`);
  try {
    const page = await browser.newPage();
    page.setDefaultNavigationTimeout(120000);
    await page.goto(url, { waitUntil: "domcontentloaded" });
    return await page.content();
  } finally {
    await browser.close();
  }
}

6. Use Proxy.useSession for IP continuity across sessions

When you need the same IP for multiple requests (e.g., login → scrape → logout flow across sessions):

await client.send("Proxy.useSession", { sessionId: "user-session-abc123" });

7. Use Unblocker.enableAdBlock for content-heavy sites

Ad networks add significant bandwidth. Blocking ads reduces costs on news sites, blogs, and similar pages.

8. Use Emulation.setDevice for mobile-specific pages

Some sites serve different content to mobile users. Set the device profile instead of manually setting user agent.

await client.send("Emulation.setDevice", { deviceName: "iPhone 14" });
await page.goto("https://example.com/mobile");

9. Monitor sessions via Page.inspect during development

During development, use Page.inspect to get a live DevTools URL to debug what the browser actually sees.

10. Use Input.type for bulk form filling

Input.type is faster than simulating individual keystrokes — important when filling many fields at scale.

---

Anti-Patterns

DO NOT use proxy networks with Playwright/Puppeteer/Selenium. Browser automation libraries should connect to Browser API, not route through a proxy. The Browser API endpoint (wss://...) replaces proxy configuration entirely.

DO NOT reuse sessions for different target URLs. One session = one page.goto(). After that, interact only with the loaded page. For new URLs, create new sessions.

DO NOT set navigation timeout below 60 seconds. Bright Data's unlocking procedures can take time. Anything below 60 seconds risks premature timeouts on difficult sites. Recommended: 120 seconds.

DO NOT enable custom headers unless required. Custom headers/cookies shift billing from success-only to 100% of all requests.

DO NOT rely on networkidle for SPA pages. Single-page applications may never reach true networkidle state. Use specific element waiters instead:

await page.waitForSelector(".product-data", { timeout: 30000 });