chore: import upstream snapshot with attribution
Test Browser Use CLI Install / uv pip install (ubuntu-latest) (push) Failing after 1s
Test Browser Use CLI Install / uvx browser-use from local wheel (push) Failing after 1s
Test Browser Use CLI Install / uvx browser-use[cli] from PyPI (push) Failing after 1s
package / pip-install-on-macos-latest-py-3.11 (push) Has been skipped
package / pip-install-on-macos-latest-py-3.13 (push) Has been skipped
package / pip-install-on-ubuntu-latest-py-3.11 (push) Has been skipped
package / pip-install-on-windows-latest-py-3.13 (push) Has been skipped
cloud_evals / trigger_cloud_eval_image_build (push) Failing after 1s
docker / build_publish_image (push) Failing after 1s
Test Browser Use CLI Install / browser-use skill sync (push) Failing after 1s
lint / code-style (push) Failing after 0s
lint / type-checker (push) Failing after 1s
package / pip-build (push) Failing after 1s
lint / syntax-errors (push) Failing after 3s
package / pip-install-on-ubuntu-latest-py-3.13 (push) Has been skipped
package / pip-install-on-windows-latest-py-3.11 (push) Has been skipped
test / ${{ matrix.test_filename }} (push) Has been skipped
test / evaluate-tasks (push) Has been skipped
test / setup-chromium (push) Failing after 2s
test / find_tests (push) Failing after 2s
Test Browser Use CLI Install / uv pip install (windows-latest) (push) Has been cancelled
Test Browser Use CLI Install / uv pip install (macos-latest) (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 12:02:32 +08:00
commit 4cd2d4af2b
475 changed files with 121829 additions and 0 deletions
+33
View File
@@ -0,0 +1,33 @@
# Contributing Agent Tasks
Contribute your own agent tasks and we test if the agent solves them for CI testing!
## How to Add a Task
1. Create a new `.yaml` file in this directory (`tests/agent_tasks/`).
2. Use the following format:
```yaml
name: My Task Name
task: Describe the task for the agent to perform
judge_context:
- List criteria for success, one per line
max_steps: 10
```
## Guidelines
- Be specific in your task and criteria.
- The `judge_context` should list what counts as a successful result.
- The agent's output will be judged by an LLM using these criteria.
## Running the Tests
To run all agent tasks:
```bash
pytest tests/ci/test_agent_real_tasks.py
```
---
Happy contributing!
+7
View File
@@ -0,0 +1,7 @@
name: Amazon Laptop Search
task: Go to amazon.com, search for 'laptop', and return the first result
judge_context:
- The agent must navigate to amazon.com
- The agent must search for 'laptop'
- The agent must return name of the first laptop
max_steps: 10
+5
View File
@@ -0,0 +1,5 @@
name: Find pip install command for browser-use
task: Find the pip installation command for the browser-use repo
judge_context:
- The output must include the command ('pip install browser-use')
max_steps: 10
+21
View File
@@ -0,0 +1,21 @@
<!DOCTYPE html>
<html>
<head>
<title>Same-Origin Iframe</title>
</head>
<body style="padding: 10px; background: #fff;">
<h3>Same-Origin Iframe Content</h3>
<button id="iframe-btn">Iframe Button</button>
<input type="text" id="iframe-input" placeholder="Iframe input" />
<script>
// When button is clicked, increment counter in parent page using addEventListener
document.getElementById('iframe-btn').addEventListener('click', function() {
if (window.parent && window.parent !== window) {
// Call parent's incrementCounter function
window.parent.incrementCounter('Same-Origin Iframe');
}
});
</script>
</body>
</html>
+177
View File
@@ -0,0 +1,177 @@
"""
Test that headers are properly passed to CDPClient for authenticated remote browser connections.
This tests the fix for: When using browser-use with remote browser services that require
authentication headers, these headers need to be included in the WebSocket handshake.
"""
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from browser_use.browser.profile import BrowserProfile
from browser_use.browser.session import BrowserSession
def test_browser_profile_headers_attribute():
"""Test that BrowserProfile correctly stores headers attribute."""
test_headers = {'Authorization': 'Bearer token123', 'X-API-Key': 'key456'}
profile = BrowserProfile(headers=test_headers)
# Verify headers are stored correctly
assert profile.headers == test_headers
# Test with profile without headers
profile_no_headers = BrowserProfile()
assert profile_no_headers.headers is None
def test_browser_profile_headers_inherited():
"""Test that BrowserSession can access headers from its profile."""
test_headers = {'Authorization': 'Bearer test-token'}
session = BrowserSession(cdp_url='wss://example.com/cdp', headers=test_headers)
assert session.browser_profile.headers == test_headers
@pytest.mark.asyncio
async def test_cdp_client_headers_passed_on_connect():
"""Test that headers from BrowserProfile are passed to CDPClient on connect()."""
test_headers = {
'Authorization': 'AWS4-HMAC-SHA256 Credential=test...',
'X-Amz-Date': '20250914T163733Z',
'X-Amz-Security-Token': 'test-token',
'Host': 'remote-browser.example.com',
}
session = BrowserSession(cdp_url='wss://remote-browser.example.com/cdp', headers=test_headers)
with patch('browser_use.browser.session.TimeoutWrappedCDPClient') as mock_cdp_client_class:
# Setup mock CDPClient instance
mock_cdp_client = AsyncMock()
mock_cdp_client_class.return_value = mock_cdp_client
mock_cdp_client.start = AsyncMock()
mock_cdp_client.stop = AsyncMock()
# Mock CDP methods
mock_cdp_client.send = MagicMock()
mock_cdp_client.send.Target = MagicMock()
mock_cdp_client.send.Target.setAutoAttach = AsyncMock()
mock_cdp_client.send.Target.getTargets = AsyncMock(return_value={'targetInfos': []})
mock_cdp_client.send.Target.createTarget = AsyncMock(return_value={'targetId': 'test-target-id'})
# Mock SessionManager (imported inside connect() from browser_use.browser.session_manager)
with patch('browser_use.browser.session_manager.SessionManager') as mock_session_manager_class:
mock_session_manager = MagicMock()
mock_session_manager_class.return_value = mock_session_manager
mock_session_manager.start_monitoring = AsyncMock()
mock_session_manager.get_all_page_targets = MagicMock(return_value=[])
try:
await session.connect()
except Exception:
# May fail due to incomplete mocking, but we can still verify the key assertion
pass
# Verify CDPClient was instantiated with the headers
mock_cdp_client_class.assert_called_once()
call_kwargs = mock_cdp_client_class.call_args
# Check positional args and keyword args
assert call_kwargs[0][0] == 'wss://remote-browser.example.com/cdp', 'CDP URL should be first arg'
actual_headers = call_kwargs[1].get('additional_headers')
# All user-provided headers must be present
for key, value in test_headers.items():
assert actual_headers[key] == value, f'Header {key} should be passed as additional_headers'
# User-Agent should be injected for remote connections
assert 'User-Agent' in actual_headers, 'User-Agent should be injected for remote connections'
assert actual_headers['User-Agent'].startswith('browser-use/'), 'User-Agent should start with browser-use/'
assert call_kwargs[1].get('max_ws_frame_size') == 200 * 1024 * 1024, 'max_ws_frame_size should be set'
@pytest.mark.asyncio
async def test_cdp_client_no_headers_when_none():
"""Test that CDPClient is created with None headers when profile has no headers."""
session = BrowserSession(cdp_url='wss://example.com/cdp')
assert session.browser_profile.headers is None
with patch('browser_use.browser.session.TimeoutWrappedCDPClient') as mock_cdp_client_class:
mock_cdp_client = AsyncMock()
mock_cdp_client_class.return_value = mock_cdp_client
mock_cdp_client.start = AsyncMock()
mock_cdp_client.stop = AsyncMock()
mock_cdp_client.send = MagicMock()
mock_cdp_client.send.Target = MagicMock()
mock_cdp_client.send.Target.setAutoAttach = AsyncMock()
mock_cdp_client.send.Target.getTargets = AsyncMock(return_value={'targetInfos': []})
mock_cdp_client.send.Target.createTarget = AsyncMock(return_value={'targetId': 'test-target-id'})
with patch('browser_use.browser.session_manager.SessionManager') as mock_session_manager_class:
mock_session_manager = MagicMock()
mock_session_manager_class.return_value = mock_session_manager
mock_session_manager.start_monitoring = AsyncMock()
mock_session_manager.get_all_page_targets = MagicMock(return_value=[])
try:
await session.connect()
except Exception:
pass
# Verify CDPClient was called with User-Agent even when no user headers are set (remote connection)
call_kwargs = mock_cdp_client_class.call_args
actual_headers = call_kwargs[1].get('additional_headers')
assert actual_headers is not None, 'Remote connections should always have headers with User-Agent'
assert actual_headers['User-Agent'].startswith('browser-use/'), 'User-Agent should be injected for remote connections'
@pytest.mark.asyncio
async def test_headers_used_for_json_version_endpoint():
"""Test that headers are also used when fetching WebSocket URL from /json/version."""
test_headers = {'Authorization': 'Bearer test-token'}
# Use HTTP URL (not ws://) to trigger /json/version fetch
session = BrowserSession(cdp_url='http://remote-browser.example.com:9222', headers=test_headers)
with patch('browser_use.browser.session.httpx.AsyncClient') as mock_client_class:
mock_client = AsyncMock()
mock_client_class.return_value.__aenter__ = AsyncMock(return_value=mock_client)
mock_client_class.return_value.__aexit__ = AsyncMock()
# Mock the /json/version response
mock_response = MagicMock()
mock_response.json.return_value = {'webSocketDebuggerUrl': 'ws://remote-browser.example.com:9222/devtools/browser/abc'}
mock_client.get = AsyncMock(return_value=mock_response)
with patch('browser_use.browser.session.TimeoutWrappedCDPClient') as mock_cdp_client_class:
mock_cdp_client = AsyncMock()
mock_cdp_client_class.return_value = mock_cdp_client
mock_cdp_client.start = AsyncMock()
mock_cdp_client.send = MagicMock()
mock_cdp_client.send.Target = MagicMock()
mock_cdp_client.send.Target.setAutoAttach = AsyncMock()
with patch('browser_use.browser.session_manager.SessionManager') as mock_sm_class:
mock_sm = MagicMock()
mock_sm_class.return_value = mock_sm
mock_sm.start_monitoring = AsyncMock()
mock_sm.get_all_page_targets = MagicMock(return_value=[])
try:
await session.connect()
except Exception:
pass
# Verify headers were passed to the HTTP GET request
mock_client.get.assert_called_once()
call_kwargs = mock_client.get.call_args
actual_headers = call_kwargs[1].get('headers')
# All user-provided headers must be present
for key, value in test_headers.items():
assert actual_headers[key] == value, f'Header {key} should be passed to /json/version'
# User-Agent should be injected
assert actual_headers['User-Agent'].startswith('browser-use/'), (
'User-Agent should be injected for /json/version fetch'
)
+259
View File
@@ -0,0 +1,259 @@
"""Tests for cloud browser functionality."""
import tempfile
from pathlib import Path
from unittest.mock import AsyncMock, patch
import pytest
from browser_use.browser.cloud.cloud import (
CloudBrowserAuthError,
CloudBrowserClient,
CloudBrowserError,
)
from browser_use.browser.cloud.views import CreateBrowserRequest
from browser_use.browser.profile import BrowserProfile
from browser_use.browser.session import BrowserSession
from browser_use.sync.auth import CloudAuthConfig
@pytest.fixture
def temp_config_dir(monkeypatch):
"""Create temporary config directory."""
with tempfile.TemporaryDirectory() as tmpdir:
temp_dir = Path(tmpdir) / '.config' / 'browseruse'
temp_dir.mkdir(parents=True, exist_ok=True)
# Use monkeypatch to set the environment variable
monkeypatch.setenv('BROWSER_USE_CONFIG_DIR', str(temp_dir))
yield temp_dir
@pytest.fixture
def mock_auth_config(temp_config_dir):
"""Create a mock auth config with valid token."""
auth_config = CloudAuthConfig(api_token='test-token', user_id='test-user-id', authorized_at=None)
auth_config.save_to_file()
return auth_config
class TestCloudBrowserClient:
"""Test CloudBrowserClient class."""
async def test_create_browser_success(self, mock_auth_config, monkeypatch):
"""Test successful cloud browser creation."""
# Clear environment variable so test uses mock_auth_config
monkeypatch.delenv('BROWSER_USE_API_KEY', raising=False)
# Mock response data matching the API
mock_response_data = {
'id': 'test-browser-id',
'status': 'active',
'liveUrl': 'https://live.browser-use.com?wss=test',
'cdpUrl': 'wss://test.proxy.daytona.works',
'timeoutAt': '2025-09-17T04:35:36.049892',
'startedAt': '2025-09-17T03:35:36.049974',
'finishedAt': None,
}
# Mock the httpx client
with patch('httpx.AsyncClient') as mock_client_class:
mock_response = AsyncMock()
mock_response.status_code = 201
mock_response.is_success = True
mock_response.json = lambda: mock_response_data
mock_client = AsyncMock()
mock_client.post.return_value = mock_response
mock_client_class.return_value = mock_client
client = CloudBrowserClient()
client.client = mock_client
result = await client.create_browser(CreateBrowserRequest())
assert result.id == 'test-browser-id'
assert result.status == 'active'
assert result.cdpUrl == 'wss://test.proxy.daytona.works'
# Verify auth headers were included
mock_client.post.assert_called_once()
call_args = mock_client.post.call_args
assert 'X-Browser-Use-API-Key' in call_args.kwargs['headers']
assert call_args.kwargs['headers']['X-Browser-Use-API-Key'] == 'test-token'
async def test_create_browser_auth_error(self, temp_config_dir, monkeypatch):
"""Test cloud browser creation with auth error."""
# Clear environment variable and don't create auth config - should trigger auth error
monkeypatch.delenv('BROWSER_USE_API_KEY', raising=False)
client = CloudBrowserClient()
with pytest.raises(CloudBrowserAuthError) as exc_info:
await client.create_browser(CreateBrowserRequest())
assert 'BROWSER_USE_API_KEY is not set' in str(exc_info.value)
async def test_create_browser_http_401(self, mock_auth_config, monkeypatch):
"""Test cloud browser creation with HTTP 401 response."""
# Clear environment variable so test uses mock_auth_config
monkeypatch.delenv('BROWSER_USE_API_KEY', raising=False)
with patch('httpx.AsyncClient') as mock_client_class:
mock_response = AsyncMock()
mock_response.status_code = 401
mock_response.is_success = False
mock_client = AsyncMock()
mock_client.post.return_value = mock_response
mock_client_class.return_value = mock_client
client = CloudBrowserClient()
client.client = mock_client
with pytest.raises(CloudBrowserAuthError) as exc_info:
await client.create_browser(CreateBrowserRequest())
assert 'BROWSER_USE_API_KEY is invalid' in str(exc_info.value)
async def test_create_browser_with_env_var(self, temp_config_dir, monkeypatch):
"""Test cloud browser creation using BROWSER_USE_API_KEY environment variable."""
# Set environment variable
monkeypatch.setenv('BROWSER_USE_API_KEY', 'env-test-token')
# Mock response data matching the API
mock_response_data = {
'id': 'test-browser-id',
'status': 'active',
'liveUrl': 'https://live.browser-use.com?wss=test',
'cdpUrl': 'wss://test.proxy.daytona.works',
'timeoutAt': '2025-09-17T04:35:36.049892',
'startedAt': '2025-09-17T03:35:36.049974',
'finishedAt': None,
}
with patch('httpx.AsyncClient') as mock_client_class:
mock_response = AsyncMock()
mock_response.status_code = 201
mock_response.is_success = True
mock_response.json = lambda: mock_response_data
mock_client = AsyncMock()
mock_client.post.return_value = mock_response
mock_client_class.return_value = mock_client
client = CloudBrowserClient()
client.client = mock_client
result = await client.create_browser(CreateBrowserRequest())
assert result.id == 'test-browser-id'
assert result.status == 'active'
assert result.cdpUrl == 'wss://test.proxy.daytona.works'
# Verify environment variable was used
mock_client.post.assert_called_once()
call_args = mock_client.post.call_args
assert 'X-Browser-Use-API-Key' in call_args.kwargs['headers']
assert call_args.kwargs['headers']['X-Browser-Use-API-Key'] == 'env-test-token'
async def test_stop_browser_success(self, mock_auth_config, monkeypatch):
"""Test successful cloud browser session stop."""
# Clear environment variable so test uses mock_auth_config
monkeypatch.delenv('BROWSER_USE_API_KEY', raising=False)
# Mock response data for stop
mock_response_data = {
'id': 'test-browser-id',
'status': 'stopped',
'liveUrl': 'https://live.browser-use.com?wss=test',
'cdpUrl': 'wss://test.proxy.daytona.works',
'timeoutAt': '2025-09-17T04:35:36.049892',
'startedAt': '2025-09-17T03:35:36.049974',
'finishedAt': '2025-09-17T04:35:36.049892',
}
with patch('httpx.AsyncClient') as mock_client_class:
mock_response = AsyncMock()
mock_response.status_code = 200
mock_response.is_success = True
mock_response.json = lambda: mock_response_data
mock_client = AsyncMock()
mock_client.patch.return_value = mock_response
mock_client_class.return_value = mock_client
client = CloudBrowserClient()
client.client = mock_client
client.current_session_id = 'test-browser-id'
result = await client.stop_browser()
assert result.id == 'test-browser-id'
assert result.status == 'stopped'
assert result.finishedAt is not None
# Verify correct API call
mock_client.patch.assert_called_once()
call_args = mock_client.patch.call_args
assert 'test-browser-id' in call_args.args[0] # URL contains session ID
assert call_args.kwargs['json'] == {'action': 'stop'}
assert 'X-Browser-Use-API-Key' in call_args.kwargs['headers']
async def test_stop_browser_session_not_found(self, mock_auth_config, monkeypatch):
"""Test stopping a browser session that doesn't exist."""
# Clear environment variable so test uses mock_auth_config
monkeypatch.delenv('BROWSER_USE_API_KEY', raising=False)
with patch('httpx.AsyncClient') as mock_client_class:
mock_response = AsyncMock()
mock_response.status_code = 404
mock_response.is_success = False
mock_client = AsyncMock()
mock_client.patch.return_value = mock_response
mock_client_class.return_value = mock_client
client = CloudBrowserClient()
client.client = mock_client
with pytest.raises(CloudBrowserError) as exc_info:
await client.stop_browser('nonexistent-session')
assert 'not found' in str(exc_info.value)
class TestBrowserSessionCloudIntegration:
"""Test BrowserSession integration with cloud browsers."""
async def test_cloud_browser_profile_property(self):
"""Test that cloud_browser property works correctly."""
# Just test the profile and session properties without connecting
profile = BrowserProfile(use_cloud=True)
session = BrowserSession(browser_profile=profile, cdp_url='ws://mock-url') # Provide CDP URL to avoid connection
assert session.cloud_browser is True
assert session.browser_profile.use_cloud is True
async def test_browser_session_cloud_browser_logic(self, mock_auth_config, monkeypatch):
"""Test that cloud browser profile settings work correctly."""
# Clear environment variable so test uses mock_auth_config
monkeypatch.delenv('BROWSER_USE_API_KEY', raising=False)
# Test cloud browser profile creation
profile = BrowserProfile(use_cloud=True)
assert profile.use_cloud is True
# Test that BrowserSession respects cloud_browser setting
# Provide CDP URL to avoid actual connection attempts
session = BrowserSession(browser_profile=profile, cdp_url='ws://mock-url')
assert session.cloud_browser is True
+138
View File
@@ -0,0 +1,138 @@
"""Test clicking elements inside cross-origin iframes."""
import asyncio
import pytest
from browser_use.browser.profile import BrowserProfile, ViewportSize
from browser_use.browser.session import BrowserSession
from browser_use.tools.service import Tools
@pytest.fixture
async def browser_session():
"""Create browser session with cross-origin iframe support."""
session = BrowserSession(
browser_profile=BrowserProfile(
headless=True,
user_data_dir=None,
keep_alive=True,
window_size=ViewportSize(width=1920, height=1400),
cross_origin_iframes=True, # Enable cross-origin iframe extraction
)
)
await session.start()
yield session
await session.kill()
class TestCrossOriginIframeClick:
"""Test clicking elements inside cross-origin iframes."""
async def test_click_element_in_cross_origin_iframe(self, httpserver, browser_session: BrowserSession):
"""Verify that elements inside iframes in different CDP targets can be clicked."""
# Create iframe content with clickable elements
iframe_html = """
<!DOCTYPE html>
<html>
<head><title>Iframe Page</title></head>
<body>
<h1>Iframe Content</h1>
<a href="https://test-domain.example/page" id="iframe-link">Test Link</a>
<button id="iframe-button">Iframe Button</button>
</body>
</html>
"""
# Create main page with iframe pointing to our test server
main_html = """
<!DOCTYPE html>
<html>
<head><title>Multi-Target Test</title></head>
<body>
<h1>Main Page</h1>
<button id="main-button">Main Button</button>
<iframe id="test-iframe" src="/iframe-content" style="width: 800px; height: 600px;"></iframe>
</body>
</html>
"""
# Serve both pages
httpserver.expect_request('/multi-target-test').respond_with_data(main_html, content_type='text/html')
httpserver.expect_request('/iframe-content').respond_with_data(iframe_html, content_type='text/html')
url = httpserver.url_for('/multi-target-test')
# Navigate to the page
await browser_session.navigate_to(url)
# Wait for iframe to load
await asyncio.sleep(2)
# Get DOM state with cross-origin iframe extraction enabled
# Use browser_session.get_browser_state_summary() instead of directly creating DomService
# This goes through the proper event bus and watchdog system
browser_state = await browser_session.get_browser_state_summary(
include_screenshot=False,
include_recent_events=False,
)
assert browser_state.dom_state is not None
state = browser_state.dom_state
print(f'\n📊 Found {len(state.selector_map)} total elements')
# Find elements from different targets
targets_found = set()
main_page_elements = []
iframe_elements = []
for idx, element in state.selector_map.items():
target_id = element.target_id
targets_found.add(target_id)
# Check if element is from iframe (identified by id attributes we set)
# Iframe elements will have a different target_id when cross_origin_iframes=True
if element.attributes:
element_id = element.attributes.get('id', '')
if element_id in ('iframe-link', 'iframe-button'):
iframe_elements.append((idx, element))
print(f' ✅ Found iframe element: [{idx}] {element.tag_name} id={element_id}')
elif element_id == 'main-button':
main_page_elements.append((idx, element))
# Verify we found elements from at least 2 different targets
print(f'\n🎯 Found elements from {len(targets_found)} different CDP targets')
# Check if iframe elements were found
if len(iframe_elements) == 0:
pytest.fail('Expected to find at least one element from iframe, but found none')
# Verify we found at least one element from the iframe
assert len(iframe_elements) > 0, 'Expected to find at least one element from iframe'
# Try clicking the iframe element
print('\n🖱️ Testing Click on Iframe Element:')
tools = Tools()
link_idx, link_element = iframe_elements[0]
print(f' Attempting to click element [{link_idx}] from iframe...')
try:
result = await tools.click(index=link_idx, browser_session=browser_session)
# Check for errors
if result.error:
pytest.fail(f'Click on iframe element [{link_idx}] failed with error: {result.error}')
if result.extracted_content and (
'not available' in result.extracted_content.lower() or 'failed' in result.extracted_content.lower()
):
pytest.fail(f'Click on iframe element [{link_idx}] failed: {result.extracted_content}')
print(f' ✅ Click succeeded on iframe element [{link_idx}]!')
print(' 🎉 Iframe element clicking works!')
except Exception as e:
pytest.fail(f'Exception while clicking iframe element [{link_idx}]: {e}')
print('\n✅ Test passed: Iframe elements can be clicked')
+585
View File
@@ -0,0 +1,585 @@
"""
Test DOM serializer with complex scenarios: shadow DOM, same-origin and cross-origin iframes.
This test verifies that the DOM serializer correctly:
1. Extracts interactive elements from shadow DOM
2. Processes same-origin iframes
3. Handles cross-origin iframes (should be blocked)
4. Generates correct selector_map with expected element counts
Usage:
uv run pytest tests/ci/browser/test_dom_serializer.py -v -s
"""
import pytest
from pytest_httpserver import HTTPServer
from browser_use.agent.service import Agent
from browser_use.browser import BrowserSession
from browser_use.browser.profile import BrowserProfile, ViewportSize
from tests.ci.conftest import create_mock_llm
@pytest.fixture(scope='session')
def http_server():
"""Create and provide a test HTTP server for DOM serializer tests."""
from pathlib import Path
server = HTTPServer()
server.start()
# Load HTML templates from files
test_dir = Path(__file__).parent
main_page_html = (test_dir / 'test_page_template.html').read_text()
iframe_html = (test_dir / 'iframe_template.html').read_text()
stacked_page_html = (test_dir / 'test_page_stacked_template.html').read_text()
# Route 1: Main page with shadow DOM and iframes
server.expect_request('/dom-test-main').respond_with_data(main_page_html, content_type='text/html')
# Route 2: Same-origin iframe content
server.expect_request('/iframe-same-origin').respond_with_data(iframe_html, content_type='text/html')
# Route 3: Stacked complex scenarios test page
server.expect_request('/stacked-test').respond_with_data(stacked_page_html, content_type='text/html')
yield server
server.stop()
@pytest.fixture(scope='session')
def base_url(http_server):
"""Return the base URL for the test HTTP server."""
return f'http://{http_server.host}:{http_server.port}'
@pytest.fixture(scope='function')
async def browser_session():
"""Create a browser session for DOM serializer tests."""
session = BrowserSession(
browser_profile=BrowserProfile(
headless=True,
user_data_dir=None,
keep_alive=True,
window_size=ViewportSize(width=1920, height=1400), # Taller window to fit all stacked elements
cross_origin_iframes=True, # Enable cross-origin iframe extraction via CDP target switching
)
)
await session.start()
yield session
await session.kill()
class TestDOMSerializer:
"""Test DOM serializer with complex scenarios."""
async def test_dom_serializer_with_shadow_dom_and_iframes(self, browser_session, base_url):
"""Test DOM serializer extracts elements from shadow DOM, same-origin iframes, and cross-origin iframes.
This test verifies:
1. Elements are in the serializer (selector_map)
2. We can click elements using click(index)
Expected interactive elements:
- Regular DOM: 3 elements (button, input, link on main page)
- Shadow DOM: 3 elements (2 buttons, 1 input inside shadow root)
- Same-origin iframe: 2 elements (button, input inside iframe)
- Cross-origin iframe placeholder: about:blank (no interactive elements)
- Iframe tags: 2 elements (the iframe elements themselves)
Total: ~10 interactive elements
"""
from browser_use.tools.service import Tools
tools = Tools()
# Create mock LLM actions that will click elements from each category
# We'll generate actions dynamically after we know the indices
actions = [
f"""
{{
"thinking": "I'll navigate to the DOM test page",
"evaluation_previous_goal": "Starting task",
"memory": "Navigating to test page",
"next_goal": "Navigate to test page",
"action": [
{{
"navigate": {{
"url": "{base_url}/dom-test-main",
"new_tab": false
}}
}}
]
}}
"""
]
await tools.navigate(url=f'{base_url}/dom-test-main', new_tab=False, browser_session=browser_session)
import asyncio
await asyncio.sleep(1)
# Get the browser state to access selector_map
browser_state_summary = await browser_session.get_browser_state_summary(
include_screenshot=False,
include_recent_events=False,
)
assert browser_state_summary is not None, 'Browser state summary should not be None'
assert browser_state_summary.dom_state is not None, 'DOM state should not be None'
selector_map = browser_state_summary.dom_state.selector_map
print(f' Selector map: {selector_map.keys()}')
print('\n📊 DOM Serializer Analysis:')
print(f' Total interactive elements found: {len(selector_map)}')
serilized_text = browser_state_summary.dom_state.llm_representation()
print(f' Serialized text: {serilized_text}')
# assume all selector map keys are as text in the serialized text
# for idx, element in selector_map.items():
# assert str(idx) in serilized_text, f'Element {idx} should be in serialized text'
# print(f' ✓ Element {idx} found in serialized text')
# assume at least 10 interactive elements are in the selector map
assert len(selector_map) >= 10, f'Should find at least 10 interactive elements, found {len(selector_map)}'
# assert all interactive elements marked with [123] from serialized text are in selector map
# find all [index] from serialized text with regex
import re
indices = re.findall(r'\[(\d+)\]', serilized_text)
for idx in indices:
assert int(idx) in selector_map.keys(), f'Element {idx} should be in selector map'
print(f' ✓ Element {idx} found in selector map')
regular_elements = []
shadow_elements = []
iframe_content_elements = []
iframe_tags = []
# Categorize elements by their IDs (more stable than hardcoded indices)
# Check element attributes to identify their location
for idx, element in selector_map.items():
# Check if this is an iframe tag (not content inside iframe)
if element.tag_name == 'iframe':
iframe_tags.append((idx, element))
# Check if element has an ID attribute
elif hasattr(element, 'attributes') and 'id' in element.attributes:
elem_id = element.attributes['id'].lower()
# Shadow DOM elements have IDs starting with "shadow-"
if elem_id.startswith('shadow-'):
shadow_elements.append((idx, element))
# Iframe content elements have IDs starting with "iframe-"
elif elem_id.startswith('iframe-'):
iframe_content_elements.append((idx, element))
# Everything else is regular DOM
else:
regular_elements.append((idx, element))
# Elements without IDs are regular DOM
else:
regular_elements.append((idx, element))
# Verify element counts based on our test page structure:
# - Regular DOM: 3-4 elements (button, input, link on main page + possible cross-origin content)
# - Shadow DOM: 3 elements (2 buttons, 1 input inside shadow root)
# - Iframe content: 2 elements (button, input from same-origin iframe)
# - Iframe tags: 2 elements (the iframe elements themselves)
# Total: ~10-11 interactive elements depending on cross-origin iframe extraction
print('\n✅ DOM Serializer Test Summary:')
print(f' • Regular DOM: {len(regular_elements)} elements {"" if len(regular_elements) >= 3 else ""}')
print(f' • Shadow DOM: {len(shadow_elements)} elements {"" if len(shadow_elements) >= 3 else ""}')
print(
f' • Same-origin iframe content: {len(iframe_content_elements)} elements {"" if len(iframe_content_elements) >= 2 else ""}'
)
print(f' • Iframe tags: {len(iframe_tags)} elements {"" if len(iframe_tags) >= 2 else ""}')
print(f' • Total elements: {len(selector_map)}')
# Verify we found elements from all sources
assert len(selector_map) >= 8, f'Should find at least 8 interactive elements, found {len(selector_map)}'
assert len(regular_elements) >= 1, f'Should find at least 1 regular DOM element, found {len(regular_elements)}'
assert len(shadow_elements) >= 1, f'Should find at least 1 shadow DOM element, found {len(shadow_elements)}'
assert len(iframe_content_elements) >= 1, (
f'Should find at least 1 iframe content element, found {len(iframe_content_elements)}'
)
# Now test clicking elements from each category using tools.click(index)
print('\n🖱️ Testing Click Functionality:')
# Helper to call tools.click(index) and verify it worked
async def click(index: int, element_description: str, browser_session: BrowserSession):
result = await tools.click(index=index, browser_session=browser_session)
# Check both error field and extracted_content for failure messages
if result.error:
raise AssertionError(f'Click on {element_description} [{index}] failed: {result.error}')
if result.extracted_content and (
'not available' in result.extracted_content.lower() or 'failed' in result.extracted_content.lower()
):
raise AssertionError(f'Click on {element_description} [{index}] failed: {result.extracted_content}')
print(f'{element_description} [{index}] clicked successfully')
return result
# Test clicking a regular DOM element (button)
if regular_elements:
regular_button_idx = next((idx for idx, el in regular_elements if 'regular-btn' in el.attributes.get('id', '')), None)
if regular_button_idx:
await click(regular_button_idx, 'Regular DOM button', browser_session)
# Test clicking a shadow DOM element (button)
if shadow_elements:
shadow_button_idx = next((idx for idx, el in shadow_elements if 'btn' in el.attributes.get('id', '')), None)
if shadow_button_idx:
await click(shadow_button_idx, 'Shadow DOM button', browser_session)
# Test clicking a same-origin iframe element (button)
if iframe_content_elements:
iframe_button_idx = next((idx for idx, el in iframe_content_elements if 'btn' in el.attributes.get('id', '')), None)
if iframe_button_idx:
await click(iframe_button_idx, 'Same-origin iframe button', browser_session)
# Validate click counter - verify all 3 clicks actually executed JavaScript
print('\n✅ Validating click counter...')
# Get the CDP session for the main page (use target from a regular DOM element)
# Note: browser_session.agent_focus_target_id may point to a different target than the page
if regular_elements and regular_elements[0][1].target_id:
cdp_session = await browser_session.get_or_create_cdp_session(target_id=regular_elements[0][1].target_id)
else:
cdp_session = await browser_session.get_or_create_cdp_session()
result = await cdp_session.cdp_client.send.Runtime.evaluate(
params={
'expression': 'window.getClickCount()',
'returnByValue': True,
},
session_id=cdp_session.session_id,
)
click_count = result.get('result', {}).get('value', 0)
print(f' Click counter value: {click_count}')
assert click_count == 3, (
f'Expected 3 clicks (Regular DOM + Shadow DOM + Iframe), but counter shows {click_count}. '
f'This means some clicks did not execute JavaScript properly.'
)
print('\n🎉 DOM Serializer test completed successfully!')
async def test_dom_serializer_element_counts_detailed(self, browser_session, base_url):
"""Detailed test to verify specific element types are captured correctly."""
actions = [
f"""
{{
"thinking": "Navigating to test page",
"evaluation_previous_goal": "Starting",
"memory": "Navigate",
"next_goal": "Navigate",
"action": [
{{
"navigate": {{
"url": "{base_url}/dom-test-main",
"new_tab": false
}}
}}
]
}}
""",
"""
{
"thinking": "Done",
"evaluation_previous_goal": "Navigated",
"memory": "Complete",
"next_goal": "Done",
"action": [
{
"done": {
"text": "Done",
"success": true
}
}
]
}
""",
]
mock_llm = create_mock_llm(actions=actions)
agent = Agent(
task=f'Navigate to {base_url}/dom-test-main',
llm=mock_llm,
browser_session=browser_session,
)
history = await agent.run(max_steps=2)
# Get current browser state to access selector_map
browser_state_summary = await browser_session.get_browser_state_summary(
include_screenshot=False,
include_recent_events=False,
)
selector_map = browser_state_summary.dom_state.selector_map
# Count different element types
buttons = 0
inputs = 0
links = 0
for idx, element in selector_map.items():
element_str = str(element).lower()
if 'button' in element_str or '<button' in element_str:
buttons += 1
elif 'input' in element_str or '<input' in element_str:
inputs += 1
elif 'link' in element_str or '<a' in element_str or 'href' in element_str:
links += 1
print('\n📊 Element Type Counts:')
print(f' Buttons: {buttons}')
print(f' Inputs: {inputs}')
print(f' Links: {links}')
print(f' Total: {len(selector_map)}')
# We should have at least some of each type from the regular DOM
assert buttons >= 1, f'Should find at least 1 button, found {buttons}'
assert inputs >= 1, f'Should find at least 1 input, found {inputs}'
print('\n✅ Element type verification passed!')
async def test_stacked_complex_scenarios(self, browser_session, base_url):
"""Test clicking through stacked complex scenarios and verify cross-origin iframe extraction.
This test verifies:
1. Open shadow DOM element interaction
2. Closed shadow DOM element interaction (nested inside open shadow)
3. Same-origin iframe element interaction (inside closed shadow)
4. Cross-origin iframe placeholder with about:blank (no external dependencies)
5. Truly nested structure: Open Shadow → Closed Shadow → Iframe
"""
from browser_use.tools.service import Tools
tools = Tools()
# Navigate to stacked test page
await tools.navigate(url=f'{base_url}/stacked-test', new_tab=False, browser_session=browser_session)
import asyncio
await asyncio.sleep(1)
# Get browser state
browser_state_summary = await browser_session.get_browser_state_summary(
include_screenshot=False,
include_recent_events=False,
)
selector_map = browser_state_summary.dom_state.selector_map
print(f'\n📊 Stacked Test - Found {len(selector_map)} elements')
# Debug: Show all elements
print('\n🔍 All elements found:')
for idx, element in selector_map.items():
elem_id = element.attributes.get('id', 'NO_ID') if hasattr(element, 'attributes') else 'NO_ATTR'
print(f' [{idx}] {element.tag_name} id={elem_id} target={element.target_id[-4:] if element.target_id else "None"}')
# Categorize elements
open_shadow_elements = []
closed_shadow_elements = []
iframe_elements = []
final_button = None
for idx, element in selector_map.items():
if hasattr(element, 'attributes') and 'id' in element.attributes:
elem_id = element.attributes['id'].lower()
if 'open-shadow' in elem_id:
open_shadow_elements.append((idx, element))
elif 'closed-shadow' in elem_id:
closed_shadow_elements.append((idx, element))
elif 'iframe' in elem_id and element.tag_name != 'iframe':
iframe_elements.append((idx, element))
elif 'final-button' in elem_id:
final_button = (idx, element)
print('\n📋 Element Distribution:')
print(f' Open Shadow: {len(open_shadow_elements)} elements')
print(f' Closed Shadow: {len(closed_shadow_elements)} elements')
print(f' Iframe content: {len(iframe_elements)} elements')
print(f' Final button: {"Found" if final_button else "Not found"}')
# Test clicking through each stacked layer
print('\n🖱️ Testing Click Functionality Through Stacked Layers:')
async def click(index: int, element_description: str, browser_session: BrowserSession):
result = await tools.click(index=index, browser_session=browser_session)
if result.error:
raise AssertionError(f'Click on {element_description} [{index}] failed: {result.error}')
if result.extracted_content and (
'not available' in result.extracted_content.lower() or 'failed' in result.extracted_content.lower()
):
raise AssertionError(f'Click on {element_description} [{index}] failed: {result.extracted_content}')
print(f'{element_description} [{index}] clicked successfully')
return result
clicks_performed = 0
# 1. Click open shadow button
if open_shadow_elements:
open_shadow_btn = next((idx for idx, el in open_shadow_elements if 'btn' in el.attributes.get('id', '')), None)
if open_shadow_btn:
await click(open_shadow_btn, 'Open Shadow DOM button', browser_session)
clicks_performed += 1
# 2. Click closed shadow button
if closed_shadow_elements:
closed_shadow_btn = next((idx for idx, el in closed_shadow_elements if 'btn' in el.attributes.get('id', '')), None)
if closed_shadow_btn:
await click(closed_shadow_btn, 'Closed Shadow DOM button', browser_session)
clicks_performed += 1
# 3. Click iframe button
if iframe_elements:
iframe_btn = next((idx for idx, el in iframe_elements if 'btn' in el.attributes.get('id', '')), None)
if iframe_btn:
await click(iframe_btn, 'Same-origin iframe button', browser_session)
clicks_performed += 1
# 4. Try clicking cross-origin iframe tag (can click the tag, but not elements inside)
cross_origin_iframe_tag = None
for idx, element in selector_map.items():
if (
element.tag_name == 'iframe'
and hasattr(element, 'attributes')
and 'cross-origin' in element.attributes.get('id', '').lower()
):
cross_origin_iframe_tag = (idx, element)
break
# Verify cross-origin iframe extraction is working
# Check the full DOM tree (not just selector_map which only has interactive elements)
def count_targets_in_tree(node, targets=None):
if targets is None:
targets = set()
# SimplifiedNode has original_node which is an EnhancedDOMTreeNode
if hasattr(node, 'original_node') and node.original_node and node.original_node.target_id:
targets.add(node.original_node.target_id)
# Recursively check children
if hasattr(node, 'children') and node.children:
for child in node.children:
count_targets_in_tree(child, targets)
return targets
all_targets = count_targets_in_tree(browser_state_summary.dom_state._root)
print('\n📊 Cross-Origin Iframe Extraction:')
print(f' Found elements from {len(all_targets)} different CDP targets in full DOM tree')
if len(all_targets) >= 2:
print(' ✅ Multi-target iframe extraction IS WORKING!')
print(' ✓ Successfully extracted DOM from multiple CDP targets')
print(' ✓ CDP target switching feature is enabled and functional')
else:
print(' ⚠️ Only found elements from 1 target (cross-origin extraction may not be working)')
if cross_origin_iframe_tag:
print(f'\n 📌 Found cross-origin iframe tag [{cross_origin_iframe_tag[0]}]')
# Note: We don't increment clicks_performed since this doesn't trigger our counter
# await click(cross_origin_iframe_tag[0], 'Cross-origin iframe tag (scroll)', browser_session)
# 5. Click final button (after all stacked elements)
if final_button:
await click(final_button[0], 'Final button (after stack)', browser_session)
clicks_performed += 1
# Validate click counter
print('\n✅ Validating click counter...')
# Get CDP session from a non-iframe element (open shadow or final button)
if open_shadow_elements:
cdp_session = await browser_session.get_or_create_cdp_session(target_id=open_shadow_elements[0][1].target_id)
elif final_button:
cdp_session = await browser_session.get_or_create_cdp_session(target_id=final_button[1].target_id)
else:
cdp_session = await browser_session.get_or_create_cdp_session()
result = await cdp_session.cdp_client.send.Runtime.evaluate(
params={
'expression': 'window.getClickCount()',
'returnByValue': True,
},
session_id=cdp_session.session_id,
)
click_count = result.get('result', {}).get('value', 0)
print(f' Click counter value: {click_count}')
print(f' Expected clicks: {clicks_performed}')
assert click_count == clicks_performed, (
f'Expected {clicks_performed} clicks, but counter shows {click_count}. '
f'Some clicks did not execute JavaScript properly.'
)
print('\n🎉 Stacked scenario test completed successfully!')
print(' ✓ Open shadow DOM clicks work')
print(' ✓ Closed shadow DOM clicks work')
print(' ✓ Same-origin iframe clicks work (can access elements inside)')
print(' ✓ Cross-origin iframe extraction works (CDP target switching enabled)')
print(' ✓ Truly nested structure works: Open Shadow → Closed Shadow → Iframe')
if __name__ == '__main__':
"""Run test in debug mode with manual fixture setup."""
import asyncio
import logging
# Set up debug logging
logging.basicConfig(
level=logging.DEBUG,
format='%(levelname)-8s [%(name)s] %(message)s',
)
async def main():
# Set up HTTP server fixture
from pathlib import Path
from pytest_httpserver import HTTPServer
server = HTTPServer()
server.start()
# Load HTML templates from files (same as http_server fixture)
test_dir = Path(__file__).parent
main_page_html = (test_dir / 'test_page_stacked_template.html').read_text()
# Set up routes using templates
server.expect_request('/stacked-test').respond_with_data(main_page_html, content_type='text/html')
base_url = f'http://{server.host}:{server.port}'
print(f'\n🌐 HTTP Server running at {base_url}')
# Set up browser session
from browser_use.browser import BrowserSession
from browser_use.browser.profile import BrowserProfile
session = BrowserSession(
browser_profile=BrowserProfile(
headless=False, # Set to False to see browser in action
user_data_dir=None,
keep_alive=True,
)
)
try:
await session.start()
print('🚀 Browser session started\n')
# Run the test
test = TestDOMSerializer()
await test.test_stacked_complex_scenarios(session, base_url)
print('\n✅ Test completed successfully!')
finally:
# Cleanup
await session.kill()
server.stop()
print('\n🧹 Cleanup complete')
asyncio.run(main())
+396
View File
@@ -0,0 +1,396 @@
"""
Test navigation edge cases: broken pages, slow loading, non-existing pages.
Tests verify that:
1. Agent can handle navigation to broken/malformed HTML pages
2. Agent can handle slow-loading pages without hanging
3. Agent can handle non-existing pages (404, connection refused, etc.)
4. Agent can recover and continue making LLM calls after encountering these issues
All tests use:
- max_steps=3 to limit agent actions
- 120s timeout to fail if test takes too long
- Mock LLM to verify agent can still make decisions after navigation errors
Usage:
uv run pytest tests/ci/browser/test_navigation.py -v -s
"""
import asyncio
import time
import pytest
from pytest_httpserver import HTTPServer
from werkzeug import Response
from browser_use.agent.service import Agent
from browser_use.browser import BrowserSession
from browser_use.browser.profile import BrowserProfile
from tests.ci.conftest import create_mock_llm
@pytest.fixture(scope='session')
def http_server():
"""Create and provide a test HTTP server for navigation tests."""
server = HTTPServer()
server.start()
# Route 1: Broken/malformed HTML page
server.expect_request('/broken').respond_with_data(
'<html><head><title>Broken Page</title></head><body><h1>Incomplete HTML',
content_type='text/html',
)
# Route 2: Valid page for testing navigation after error recovery
server.expect_request('/valid').respond_with_data(
'<html><head><title>Valid Page</title></head><body><h1>Valid Page</h1><p>This page loaded successfully</p></body></html>',
content_type='text/html',
)
# Route 3: Slow loading page - delays 10 seconds before responding
def slow_handler(request):
time.sleep(10)
return Response(
'<html><head><title>Slow Page</title></head><body><h1>Slow Loading Page</h1><p>This page took 10 seconds to load</p></body></html>',
content_type='text/html',
)
server.expect_request('/slow').respond_with_handler(slow_handler)
# Route 4: 404 page
server.expect_request('/notfound').respond_with_data(
'<html><head><title>404 Not Found</title></head><body><h1>404 - Page Not Found</h1></body></html>',
status=404,
content_type='text/html',
)
yield server
server.stop()
@pytest.fixture(scope='session')
def base_url(http_server):
"""Return the base URL for the test HTTP server."""
return f'http://{http_server.host}:{http_server.port}'
@pytest.fixture(scope='function')
async def browser_session():
"""Create a browser session for navigation tests."""
session = BrowserSession(
browser_profile=BrowserProfile(
headless=True,
user_data_dir=None,
keep_alive=True,
)
)
await session.start()
yield session
await session.kill()
class TestNavigationEdgeCases:
"""Test navigation error handling and recovery."""
async def test_broken_page_navigation(self, browser_session, base_url):
"""Test that agent can handle broken/malformed HTML and still make LLM calls."""
# Create actions for the agent:
# 1. Navigate to broken page
# 2. Check if page exists
# 3. Done
actions = [
f"""
{{
"thinking": "I need to navigate to the broken page",
"evaluation_previous_goal": "Starting task",
"memory": "Navigating to broken page",
"next_goal": "Navigate to broken page",
"action": [
{{
"navigate": {{
"url": "{base_url}/broken"
}}
}}
]
}}
""",
"""
{
"thinking": "I should check if the page loaded",
"evaluation_previous_goal": "Navigated to page",
"memory": "Checking page state",
"next_goal": "Verify page exists",
"action": [
{
"done": {
"text": "Page exists despite broken HTML",
"success": true
}
}
]
}
""",
]
mock_llm = create_mock_llm(actions=actions)
agent = Agent(
task=f'Navigate to {base_url}/broken and check if page exists',
llm=mock_llm,
browser_session=browser_session,
)
# Run with timeout - should complete within 2 minutes
try:
history = await asyncio.wait_for(agent.run(max_steps=3), timeout=120)
assert len(history) > 0, 'Agent should have completed at least one step'
# If agent completes successfully, it means LLM was called and functioning
final_result = history.final_result()
assert final_result is not None, 'Agent should return a final result'
except TimeoutError:
pytest.fail('Test timed out after 2 minutes - agent hung on broken page')
async def test_slow_loading_page(self, browser_session, base_url):
"""Test that agent can handle slow-loading pages without hanging."""
actions = [
f"""
{{
"thinking": "I need to navigate to the slow page",
"evaluation_previous_goal": "Starting task",
"memory": "Navigating to slow page",
"next_goal": "Navigate to slow page",
"action": [
{{
"navigate": {{
"url": "{base_url}/slow"
}}
}}
]
}}
""",
"""
{
"thinking": "The page loaded, even though it was slow",
"evaluation_previous_goal": "Successfully navigated",
"memory": "Page loaded after delay",
"next_goal": "Complete task",
"action": [
{
"done": {
"text": "Slow page loaded successfully",
"success": true
}
}
]
}
""",
]
mock_llm = create_mock_llm(actions=actions)
agent = Agent(
task=f'Navigate to {base_url}/slow and wait for it to load',
llm=mock_llm,
browser_session=browser_session,
)
# Run with timeout - should complete within 2 minutes
start_time = time.time()
try:
history = await asyncio.wait_for(agent.run(max_steps=3), timeout=120)
elapsed = time.time() - start_time
assert len(history) > 0, 'Agent should have completed at least one step'
assert elapsed >= 10, f'Agent should have waited for slow page (10s delay), but only took {elapsed:.1f}s'
final_result = history.final_result()
assert final_result is not None, 'Agent should return a final result'
except TimeoutError:
pytest.fail('Test timed out after 2 minutes - agent hung on slow page')
async def test_nonexisting_page_404(self, browser_session, base_url):
"""Test that agent can handle 404 pages and still make LLM calls."""
actions = [
f"""
{{
"thinking": "I need to navigate to the non-existing page",
"evaluation_previous_goal": "Starting task",
"memory": "Navigating to 404 page",
"next_goal": "Navigate to non-existing page",
"action": [
{{
"navigate": {{
"url": "{base_url}/notfound"
}}
}}
]
}}
""",
"""
{
"thinking": "I got a 404 error but the browser still works",
"evaluation_previous_goal": "Navigated to 404 page",
"memory": "Page not found",
"next_goal": "Report that page does not exist",
"action": [
{
"done": {
"text": "Page does not exist (404 error)",
"success": false
}
}
]
}
""",
]
mock_llm = create_mock_llm(actions=actions)
agent = Agent(
task=f'Navigate to {base_url}/notfound and check if page exists',
llm=mock_llm,
browser_session=browser_session,
)
# Run with timeout - should complete within 2 minutes
try:
history = await asyncio.wait_for(agent.run(max_steps=3), timeout=120)
assert len(history) > 0, 'Agent should have completed at least one step'
final_result = history.final_result()
assert final_result is not None, 'Agent should return a final result'
except TimeoutError:
pytest.fail('Test timed out after 2 minutes - agent hung on 404 page')
async def test_nonexisting_domain(self, browser_session):
"""Test that agent can handle completely non-existing domains (connection refused)."""
# Use a localhost port that's not listening
nonexisting_url = 'http://localhost:59999/page'
actions = [
f"""
{{
"thinking": "I need to navigate to a non-existing domain",
"evaluation_previous_goal": "Starting task",
"memory": "Attempting to navigate",
"next_goal": "Navigate to non-existing domain",
"action": [
{{
"navigate": {{
"url": "{nonexisting_url}"
}}
}}
]
}}
""",
"""
{
"thinking": "The connection failed but I can still proceed",
"evaluation_previous_goal": "Connection failed",
"memory": "Domain does not exist",
"next_goal": "Report failure",
"action": [
{
"done": {
"text": "Domain does not exist (connection refused)",
"success": false
}
}
]
}
""",
]
mock_llm = create_mock_llm(actions=actions)
agent = Agent(
task=f'Navigate to {nonexisting_url} and check if it exists',
llm=mock_llm,
browser_session=browser_session,
)
# Run with timeout - should complete within 2 minutes
try:
history = await asyncio.wait_for(agent.run(max_steps=3), timeout=120)
assert len(history) > 0, 'Agent should have completed at least one step'
final_result = history.final_result()
assert final_result is not None, 'Agent should return a final result'
except TimeoutError:
pytest.fail('Test timed out after 2 minutes - agent hung on non-existing domain')
async def test_recovery_after_navigation_error(self, browser_session, base_url):
"""Test that agent can recover and navigate to valid page after encountering error."""
actions = [
f"""
{{
"thinking": "First, I'll try the broken page",
"evaluation_previous_goal": "Starting task",
"memory": "Navigating to broken page",
"next_goal": "Navigate to broken page first",
"action": [
{{
"navigate": {{
"url": "{base_url}/broken"
}}
}}
]
}}
""",
f"""
{{
"thinking": "That page was broken, let me try a valid page now",
"evaluation_previous_goal": "Broken page loaded",
"memory": "Now navigating to valid page",
"next_goal": "Navigate to valid page",
"action": [
{{
"navigate": {{
"url": "{base_url}/valid"
}}
}}
]
}}
""",
"""
{
"thinking": "The valid page loaded successfully after the broken one",
"evaluation_previous_goal": "Valid page loaded",
"memory": "Successfully recovered from error",
"next_goal": "Complete task",
"action": [
{
"done": {
"text": "Successfully navigated to valid page after broken page",
"success": true
}
}
]
}
""",
]
mock_llm = create_mock_llm(actions=actions)
agent = Agent(
task=f'First navigate to {base_url}/broken, then navigate to {base_url}/valid',
llm=mock_llm,
browser_session=browser_session,
)
# Run with timeout - should complete within 2 minutes
try:
history = await asyncio.wait_for(agent.run(max_steps=3), timeout=120)
assert len(history) >= 2, 'Agent should have completed at least 2 steps (broken -> valid)'
# Verify final page is the valid one
final_url = await browser_session.get_current_page_url()
assert final_url.endswith('/valid'), f'Final URL should be /valid, got {final_url}'
# Verify agent completed successfully
final_result = history.final_result()
assert final_result is not None, 'Agent should return a final result'
except TimeoutError:
pytest.fail('Test timed out after 2 minutes - agent could not recover from broken page')
@@ -0,0 +1,139 @@
"""Regression tests: navigation readiness detection must actually detect page load.
cdp-use's event registry is single-slot per CDP method: registering a per-session
Page.lifecycleEvent closure replaces the previous one, so only the most recently
attached target recorded lifecycle events. Any earlier tab's event deque went silent,
and every navigation on it burned the full readiness timeout (3s same-domain /
8s cross-domain) before proceeding on a page in unknown load state.
Lifecycle events are now stored per-target in SessionManager, fed by one global
handler registered once on the root CDP client.
"""
import asyncio
import time
import pytest
from browser_use.browser.events import NavigateToUrlEvent
from browser_use.browser.profile import BrowserProfile
from browser_use.browser.session import BrowserSession
SIMPLE_HTML = '<html><head><title>fast page</title></head><body>hello</body></html>'
# Generous bound for a local page load: well above real load time (~100ms),
# well below the 3s that a readiness-timeout fallback burns.
FAST_NAVIGATION_BOUND_S = 2.5
@pytest.fixture
async def browser_session():
session = BrowserSession(browser_profile=BrowserProfile(headless=True, user_data_dir=None, keep_alive=True))
await session.start()
yield session
await session.kill()
async def test_navigation_detects_readiness_without_burning_timeout(httpserver, browser_session: BrowserSession):
"""A local page load must complete as soon as the load event fires, not after the fallback timeout."""
httpserver.expect_request('/fast').respond_with_data(SIMPLE_HTML, content_type='text/html')
start = time.monotonic()
await browser_session.navigate_to(httpserver.url_for('/fast'))
elapsed = time.monotonic() - start
assert elapsed < FAST_NAVIGATION_BOUND_S, (
f'navigation took {elapsed:.2f}s — readiness detection failed and the fallback timeout was burned'
)
async def test_first_tab_navigation_still_works_after_second_tab_opens(httpserver, browser_session: BrowserSession):
"""Opening a new tab must not disable lifecycle monitoring on existing tabs.
Pre-fix: the second tab's per-session handler registration replaced the first
tab's on the shared root client, freezing the first tab's event deque — every
subsequent navigation on tab A hit the full readiness timeout.
"""
httpserver.expect_request('/a').respond_with_data(SIMPLE_HTML, content_type='text/html')
httpserver.expect_request('/b').respond_with_data(SIMPLE_HTML, content_type='text/html')
httpserver.expect_request('/a2').respond_with_data(SIMPLE_HTML, content_type='text/html')
await browser_session.navigate_to(httpserver.url_for('/a'))
tab_a = browser_session.agent_focus_target_id
assert tab_a is not None
# Open a second tab — its target attach re-runs page monitoring setup
event = browser_session.event_bus.dispatch(NavigateToUrlEvent(url=httpserver.url_for('/b'), new_tab=True))
await event
await asyncio.sleep(0.5)
# Navigate the FIRST tab again (same domain -> 3s fallback timeout pre-fix)
start = time.monotonic()
await browser_session._navigate_and_wait(httpserver.url_for('/a2'), tab_a, wait_until='load')
elapsed = time.monotonic() - start
assert elapsed < FAST_NAVIGATION_BOUND_S, (
f'tab A navigation took {elapsed:.2f}s after tab B opened — its lifecycle monitoring was clobbered'
)
async def test_readiness_timeout_is_reported_not_swallowed(httpserver, browser_session: BrowserSession):
"""When readiness genuinely times out, _navigate_and_wait must say so instead of
returning indistinguishably from success (NavigationCompleteEvent.loading_status
exists for exactly this)."""
def slow_image(request):
time.sleep(5)
from werkzeug.wrappers import Response
return Response(b'', content_type='image/png')
# DOMContentLoaded fires immediately; 'load' is held hostage by the stalled image.
httpserver.expect_request('/hanging').respond_with_data(
'<html><body><img src="/slow-img">never finishes loading</body></html>', content_type='text/html'
)
httpserver.expect_request('/slow-img').respond_with_handler(slow_image)
target_id = browser_session.agent_focus_target_id
assert target_id is not None
status = await browser_session._navigate_and_wait(httpserver.url_for('/hanging'), target_id, timeout=1.0, wait_until='load')
assert status is not None and 'timeout' in status, f'readiness timeout was swallowed, got status={status!r}'
async def test_successful_navigation_returns_no_timeout_status(httpserver, browser_session: BrowserSession):
httpserver.expect_request('/ok').respond_with_data(SIMPLE_HTML, content_type='text/html')
target_id = browser_session.agent_focus_target_id
assert target_id is not None
status = await browser_session._navigate_and_wait(httpserver.url_for('/ok'), target_id, wait_until='load')
assert status is None
async def test_same_document_navigation_completes_immediately(httpserver, browser_session: BrowserSession):
"""Fragment/History-API navigations must not burn the readiness timeout.
Page.navigate returns no loaderId for same-document navigations and Chrome
emits no new load/DOMContentLoaded lifecycle events for them — the navigation
is already committed when Page.navigate returns.
"""
httpserver.expect_request('/page').respond_with_data(
'<html><body><a id="anchor" name="section">s</a></body></html>', content_type='text/html'
)
await browser_session.navigate_to(httpserver.url_for('/page'))
target_id = browser_session.agent_focus_target_id
assert target_id is not None
# Let the first load's trailing lifecycle events (networkIdle fires ~1s after
# load) drain, so a stale event can't accidentally satisfy the fragment wait.
await asyncio.sleep(1.5)
start = time.monotonic()
status = await browser_session._navigate_and_wait(httpserver.url_for('/page') + '#section', target_id, wait_until='load')
elapsed = time.monotonic() - start
assert elapsed < FAST_NAVIGATION_BOUND_S, f'same-document navigation took {elapsed:.2f}s — burned the readiness timeout'
assert status is None, f'same-document navigation reported a bogus timeout: {status!r}'
@@ -0,0 +1,188 @@
"""
Test navigation on heavy/slow-loading pages (e.g. e-commerce PDPs).
Reproduces the issue where navigating to heavy pages like stevemadden.com PDPs
fails due to NavigateToUrlEvent timing out.
Usage:
uv run pytest tests/ci/browser/test_navigation_slow_pages.py -v -s
"""
import asyncio
import time
import pytest
from pytest_httpserver import HTTPServer
from werkzeug import Response
from browser_use.agent.service import Agent
from browser_use.browser import BrowserSession
from browser_use.browser.events import NavigateToUrlEvent
from browser_use.browser.profile import BrowserProfile
from tests.ci.conftest import create_mock_llm
HEAVY_PDP_HTML = """
<!DOCTYPE html>
<html>
<head><title>Frosting Black Velvet - Steve Madden</title></head>
<body>
<h1>FROSTING</h1>
<p class="price">$129.95</p>
<button id="add-to-cart">ADD TO BAG</button>
</body>
</html>
"""
@pytest.fixture(scope='session')
def heavy_page_server():
server = HTTPServer()
server.start()
def slow_initial_response(request):
time.sleep(6)
return Response(HEAVY_PDP_HTML, content_type='text/html')
server.expect_request('/slow-server-pdp').respond_with_handler(slow_initial_response)
def redirect_step1(request):
return Response('', status=302, headers={'Location': f'http://{server.host}:{server.port}/redirect-step2'})
def redirect_step2(request):
return Response('', status=302, headers={'Location': f'http://{server.host}:{server.port}/redirect-final'})
def redirect_final(request):
time.sleep(3)
return Response(HEAVY_PDP_HTML, content_type='text/html')
server.expect_request('/redirect-step1').respond_with_handler(redirect_step1)
server.expect_request('/redirect-step2').respond_with_handler(redirect_step2)
server.expect_request('/redirect-final').respond_with_handler(redirect_final)
server.expect_request('/fast-dom-slow-load').respond_with_data(HEAVY_PDP_HTML, content_type='text/html')
server.expect_request('/quick-page').respond_with_data(
'<html><body><h1>Quick Page</h1></body></html>', content_type='text/html'
)
yield server
server.stop()
@pytest.fixture(scope='session')
def heavy_base_url(heavy_page_server):
return f'http://{heavy_page_server.host}:{heavy_page_server.port}'
@pytest.fixture(scope='function')
async def browser_session():
session = BrowserSession(browser_profile=BrowserProfile(headless=True, user_data_dir=None, keep_alive=True))
await session.start()
yield session
await session.kill()
def _nav_actions(url: str, msg: str = 'Done') -> list[str]:
"""Helper to build a navigate-then-done action sequence."""
return [
f"""
{{
"thinking": "Navigate to the page",
"evaluation_previous_goal": "Starting task",
"memory": "Navigating",
"next_goal": "Navigate",
"action": [{{"navigate": {{"url": "{url}"}}}}]
}}
""",
f"""
{{
"thinking": "Page loaded",
"evaluation_previous_goal": "Navigation completed",
"memory": "Page loaded",
"next_goal": "Done",
"action": [{{"done": {{"text": "{msg}", "success": true}}}}]
}}
""",
]
class TestHeavyPageNavigation:
async def test_slow_server_response_completes(self, browser_session, heavy_base_url):
"""Navigation succeeds even when server takes 6s to respond."""
url = f'{heavy_base_url}/slow-server-pdp'
agent = Agent(
task=f'Navigate to {url}',
llm=create_mock_llm(actions=_nav_actions(url)),
browser_session=browser_session,
)
start = time.time()
history = await asyncio.wait_for(agent.run(max_steps=3), timeout=60)
assert len(history) > 0
assert history.final_result() is not None
assert time.time() - start >= 5, 'Should have waited for slow server'
async def test_redirect_chain_completes(self, browser_session, heavy_base_url):
"""Navigation handles multi-step redirects + slow final response."""
url = f'{heavy_base_url}/redirect-step1'
agent = Agent(
task=f'Navigate to {url}',
llm=create_mock_llm(actions=_nav_actions(url)),
browser_session=browser_session,
)
history = await asyncio.wait_for(agent.run(max_steps=3), timeout=60)
assert len(history) > 0
assert history.final_result() is not None
async def test_navigate_event_accepts_domcontentloaded(self, browser_session, heavy_base_url):
"""NavigateToUrlEvent with fast page should complete quickly via DOMContentLoaded/load."""
url = f'{heavy_base_url}/fast-dom-slow-load'
event = browser_session.event_bus.dispatch(NavigateToUrlEvent(url=url))
await asyncio.wait_for(event, timeout=15)
await event.event_result(raise_if_any=True, raise_if_none=False)
async def test_recovery_after_slow_navigation(self, browser_session, heavy_base_url):
"""Agent recovers and navigates to a fast page after a slow one."""
slow_url = f'{heavy_base_url}/slow-server-pdp'
quick_url = f'{heavy_base_url}/quick-page'
actions = [
f"""
{{
"thinking": "Navigate to slow page",
"evaluation_previous_goal": "Starting",
"memory": "Going to slow page",
"next_goal": "Navigate",
"action": [{{"navigate": {{"url": "{slow_url}"}}}}]
}}
""",
f"""
{{
"thinking": "Now navigate to quick page",
"evaluation_previous_goal": "Slow page loaded",
"memory": "Trying quick page",
"next_goal": "Navigate",
"action": [{{"navigate": {{"url": "{quick_url}"}}}}]
}}
""",
"""
{
"thinking": "Both done",
"evaluation_previous_goal": "Quick page loaded",
"memory": "Recovery successful",
"next_goal": "Done",
"action": [{"done": {"text": "Recovery succeeded", "success": true}}]
}
""",
]
agent = Agent(
task='Navigate to slow then quick page',
llm=create_mock_llm(actions=actions),
browser_session=browser_session,
)
history = await asyncio.wait_for(agent.run(max_steps=4), timeout=90)
assert len(history) >= 2
assert history.final_result() is not None
async def test_event_timeout_sufficient_for_heavy_pages(self, browser_session):
"""event_timeout should be >= 30s to handle slow servers + redirect chains."""
event = NavigateToUrlEvent(url='http://example.com')
assert event.event_timeout is not None
assert event.event_timeout >= 30.0, f'event_timeout={event.event_timeout}s is too low for heavy pages (need >= 30s)'
+219
View File
@@ -0,0 +1,219 @@
"""Test all recording and save functionality for Agent and BrowserSession."""
from pathlib import Path
import pytest
from browser_use import Agent, AgentHistoryList
from browser_use.browser import BrowserProfile, BrowserSession
from tests.ci.conftest import create_mock_llm
@pytest.fixture
def test_dir(tmp_path):
"""Create a test directory that gets cleaned up after each test."""
test_path = tmp_path / 'test_recordings'
test_path.mkdir(exist_ok=True)
yield test_path
@pytest.fixture
async def httpserver_url(httpserver):
"""Simple test page."""
# Use expect_ordered_request with multiple handlers to handle repeated requests
for _ in range(10): # Allow up to 10 requests to the same URL
httpserver.expect_ordered_request('/').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head>
<title>Test Page</title>
</head>
<body>
<h1>Test Recording Page</h1>
<input type="text" id="search" placeholder="Search here" />
<button type="button" id="submit">Submit</button>
</body>
</html>
""",
content_type='text/html',
)
return httpserver.url_for('/')
@pytest.fixture
def llm():
"""Create mocked LLM instance for tests."""
return create_mock_llm()
@pytest.fixture
def interactive_llm(httpserver_url):
"""Create mocked LLM that navigates to page and interacts with elements."""
actions = [
# First action: Navigate to the page
f"""
{{
"thinking": "null",
"evaluation_previous_goal": "Starting the task",
"memory": "Need to navigate to the test page",
"next_goal": "Navigate to the URL",
"action": [
{{
"navigate": {{
"url": "{httpserver_url}",
"new_tab": false
}}
}}
]
}}
""",
# Second action: Click in the search box
"""
{
"thinking": "null",
"evaluation_previous_goal": "Successfully navigated to the page",
"memory": "Page loaded, can see search box and submit button",
"next_goal": "Click on the search box to focus it",
"action": [
{
"click": {
"index": 0
}
}
]
}
""",
# Third action: Type text in the search box
"""
{
"thinking": "null",
"evaluation_previous_goal": "Clicked on search box",
"memory": "Search box is focused and ready for input",
"next_goal": "Type 'test' in the search box",
"action": [
{
"input_text": {
"index": 0,
"text": "test"
}
}
]
}
""",
# Fourth action: Click submit button
"""
{
"thinking": "null",
"evaluation_previous_goal": "Typed 'test' in search box",
"memory": "Text 'test' has been entered successfully",
"next_goal": "Click the submit button to complete the task",
"action": [
{
"click": {
"index": 1
}
}
]
}
""",
# Fifth action: Done - task completed
"""
{
"thinking": "null",
"evaluation_previous_goal": "Clicked the submit button",
"memory": "Successfully navigated to the page, typed 'test' in the search box, and clicked submit",
"next_goal": "Task completed",
"action": [
{
"done": {
"text": "Task completed - typed 'test' in search box and clicked submit",
"success": true
}
}
]
}
""",
]
return create_mock_llm(actions)
class TestAgentRecordings:
"""Test Agent save_conversation_path and generate_gif parameters."""
@pytest.mark.parametrize('path_type', ['with_slash', 'without_slash', 'deep_directory'])
async def test_save_conversation_path(self, test_dir, httpserver_url, llm, path_type):
"""Test saving conversation with different path types."""
if path_type == 'with_slash':
conversation_path = test_dir / 'logs' / 'conversation'
elif path_type == 'without_slash':
conversation_path = test_dir / 'logs'
else: # deep_directory
conversation_path = test_dir / 'logs' / 'deep' / 'directory' / 'conversation'
browser_session = BrowserSession(browser_profile=BrowserProfile(headless=True, disable_security=True, user_data_dir=None))
await browser_session.start()
try:
agent = Agent(
task=f'go to {httpserver_url} and type "test" in the search box',
llm=llm,
browser_session=browser_session,
save_conversation_path=str(conversation_path),
)
history: AgentHistoryList = await agent.run(max_steps=2)
result = history.final_result()
assert result is not None
# Check that the conversation directory and files were created
assert conversation_path.exists(), f'{path_type}: conversation directory was not created'
# Files are now always created as conversation_<agent_id>_<step>.txt inside the directory
conversation_files = list(conversation_path.glob('conversation_*.txt'))
assert len(conversation_files) > 0, f'{path_type}: conversation file was not created in {conversation_path}'
finally:
await browser_session.kill()
@pytest.mark.skip(reason='TODO: fix')
@pytest.mark.parametrize('generate_gif', [False, True, 'custom_path'])
async def test_generate_gif(self, test_dir, httpserver_url, llm, generate_gif):
"""Test GIF generation with different settings."""
# Clean up any existing GIFs first
for gif in Path.cwd().glob('agent_*.gif'):
gif.unlink()
gif_param = generate_gif
expected_gif_path = None
if generate_gif == 'custom_path':
expected_gif_path = test_dir / 'custom_agent.gif'
gif_param = str(expected_gif_path)
browser_session = BrowserSession(browser_profile=BrowserProfile(headless=True, disable_security=True, user_data_dir=None))
await browser_session.start()
try:
agent = Agent(
task=f'go to {httpserver_url}',
llm=llm,
browser_session=browser_session,
generate_gif=gif_param,
)
history: AgentHistoryList = await agent.run(max_steps=2)
result = history.final_result()
assert result is not None
# Check GIF creation
if generate_gif is False:
gif_files = list(Path.cwd().glob('*.gif'))
assert len(gif_files) == 0, 'GIF file was created when generate_gif=False'
elif generate_gif is True:
# With mock LLM that doesn't navigate, all screenshots will be about:blank placeholders
# So no GIF will be created (this is expected behavior)
gif_files = list(Path.cwd().glob('agent_history.gif'))
assert len(gif_files) == 0, 'GIF should not be created when all screenshots are placeholders'
else: # custom_path
assert expected_gif_path is not None, 'expected_gif_path should be set for custom_path'
# With mock LLM that doesn't navigate, no GIF will be created
assert not expected_gif_path.exists(), 'GIF should not be created when all screenshots are placeholders'
finally:
await browser_session.kill()
@@ -0,0 +1,129 @@
<!DOCTYPE html>
<html>
<head>
<title>Stacked DOM Elements Test</title>
<style>
body { font-family: Arial; padding: 20px; min-height: 3000px; }
.section {
margin: 20px 0;
padding: 15px;
border: 2px solid #333;
background: #f9f9f9;
}
#click-counter {
position: fixed;
top: 20px;
right: 20px;
background: #4CAF50;
color: white;
padding: 30px 50px;
border-radius: 15px;
font-size: 48px;
font-weight: bold;
box-shadow: 0 4px 20px rgba(0,0,0,0.3);
transition: all 0.2s ease;
z-index: 9999;
}
#counter-value {
font-size: 64px;
display: inline-block;
min-width: 60px;
text-align: center;
}
@keyframes flash {
0% { transform: scale(1); }
50% { transform: scale(1.3); background: #FFC107; }
100% { transform: scale(1); }
}
.flash {
animation: flash 0.3s ease;
}
.final-button {
margin-top: 50px;
padding: 20px 40px;
font-size: 24px;
background: #2196F3;
color: white;
border: none;
border-radius: 8px;
cursor: pointer;
}
</style>
</head>
<body>
<div id="click-counter">Clicks: <span id="counter-value">0</span></div>
<h1>Nested DOM Test</h1>
<!-- Root: Open Shadow DOM (contains everything else) -->
<div class="section">
<div id="open-shadow-host"></div>
</div>
<script>
// Global click counter
let clickCount = 0;
function incrementCounter(source) {
clickCount++;
const counter = document.getElementById('click-counter');
const counterValue = document.getElementById('counter-value');
counterValue.textContent = clickCount;
console.log(`Click #${clickCount} from: ${source}`);
// Add flash animation
counter.classList.remove('flash');
void counter.offsetWidth; // Trigger reflow
counter.classList.add('flash');
}
// Expose counter for testing
window.getClickCount = function() {
return clickCount;
};
// Build nested structure: Open Shadow → Closed Shadow → Iframe → Final Button
// 1. Create Open Shadow DOM (contains everything else)
const openShadowHost = document.getElementById('open-shadow-host');
const openShadowRoot = openShadowHost.attachShadow({mode: 'open'});
openShadowRoot.innerHTML = `
<style>
.shadow-content { padding: 15px; background: #e3f2fd; border: 2px solid #2196F3; margin: 10px 0; }
button { padding: 10px 20px; font-size: 16px; margin: 10px 0; display: block; }
.nested-info { font-weight: bold; color: #1976D2; }
</style>
<div class="shadow-content">
<button id="open-shadow-btn">Open Shadow Button</button>
<div id="closed-shadow-host"></div>
</div>
`;
openShadowRoot.getElementById('open-shadow-btn').addEventListener('click', function() {
incrementCounter('Open Shadow DOM');
});
// 2. Create Closed Shadow DOM INSIDE Open Shadow (nested!)
const closedShadowHost = openShadowRoot.getElementById('closed-shadow-host');
const closedShadowRoot = closedShadowHost.attachShadow({mode: 'closed'});
closedShadowRoot.innerHTML = `
<style>
.shadow-content { padding: 15px; background: #fff3e0; border: 2px solid #FF9800; margin: 10px 0; }
button { padding: 10px 20px; font-size: 16px; margin: 10px 0; display: block; }
iframe { width: 100%; height: 250px; border: 2px solid #4CAF50; margin: 10px 0; }
.nested-info { font-weight: bold; color: #F57C00; }
.iframe-label { font-size: 14px; color: #666; margin-top: 10px; }
</style>
<div class="shadow-content">
<button id="closed-shadow-btn">Closed Shadow Button</button>
<iframe id="cross-origin-iframe" src="about:blank"></iframe>
<iframe id="nested-iframe" src="/iframe-same-origin"></iframe>
</div>
`;
closedShadowRoot.getElementById('closed-shadow-btn').addEventListener('click', function() {
incrementCounter('Closed Shadow DOM');
});
</script>
</body>
</html>
+118
View File
@@ -0,0 +1,118 @@
<!DOCTYPE html>
<html>
<head>
<title>DOM Serializer Test - Main Page</title>
<style>
body { font-family: Arial; padding: 20px; }
.section { margin: 20px 0; padding: 15px; border: 1px solid #ccc; }
#click-counter {
position: fixed;
top: 20px;
right: 20px;
background: #4CAF50;
color: white;
padding: 30px 50px;
border-radius: 15px;
font-size: 48px;
font-weight: bold;
box-shadow: 0 4px 20px rgba(0,0,0,0.3);
transition: all 0.2s ease;
z-index: 9999;
}
#counter-value {
font-size: 64px;
display: inline-block;
min-width: 60px;
text-align: center;
}
@keyframes flash {
0% { transform: scale(1); }
50% { transform: scale(1.3); background: #FFC107; }
100% { transform: scale(1); }
}
.flash {
animation: flash 0.3s ease;
}
</style>
</head>
<body>
<div id="click-counter">Clicks: <span id="counter-value">0</span></div>
<h1>DOM Serializer Test Page</h1>
<!-- Regular DOM elements (3 interactive elements) -->
<div class="section">
<h2>Regular DOM Elements</h2>
<button id="regular-btn-1">Regular Button 1</button>
<input type="text" id="regular-input" placeholder="Regular input" />
<a href="#test" id="regular-link">Regular Link</a>
</div>
<!-- Shadow DOM elements (3 interactive elements inside shadow) -->
<div class="section">
<h2>Shadow DOM Elements</h2>
<div id="shadow-host"></div>
</div>
<!-- Same-origin iframe (2 interactive elements inside) -->
<div class="section">
<h2>Same-Origin Iframe</h2>
<iframe id="same-origin-iframe" src="/iframe-same-origin" style="width:100%; height:200px; border:1px solid #999;"></iframe>
</div>
<!-- Cross-origin iframe placeholder (external domain removed for test isolation) -->
<div class="section">
<h2>Cross-Origin Iframe (Placeholder)</h2>
<iframe id="cross-origin-iframe" src="about:blank" style="width:100%; height:200px; border:1px solid #999;"></iframe>
</div>
<script>
// Global click counter
let clickCount = 0;
function incrementCounter(source) {
clickCount++;
const counter = document.getElementById('click-counter');
const counterValue = document.getElementById('counter-value');
counterValue.textContent = clickCount;
console.log(`Click #${clickCount} from: ${source}`);
// Add flash animation
counter.classList.remove('flash');
void counter.offsetWidth; // Trigger reflow
counter.classList.add('flash');
}
// Expose counter for testing
window.getClickCount = function() {
return clickCount;
};
// Add click handler to regular button using addEventListener
document.getElementById('regular-btn-1').addEventListener('click', function() {
incrementCounter('Regular DOM');
});
// Create shadow DOM with interactive elements
const shadowHost = document.getElementById('shadow-host');
const shadowRoot = shadowHost.attachShadow({mode: 'open'});
shadowRoot.innerHTML = `
<style>
.shadow-content { padding: 10px; background: #f0f0f0; }
</style>
<div class="shadow-content">
<p>Content inside Shadow DOM:</p>
<button id="shadow-btn-1">Shadow Button 1</button>
<input type="text" id="shadow-input" placeholder="Shadow input" />
<button id="shadow-btn-2">Shadow Button 2</button>
</div>
`;
// Add click handler to shadow DOM button using addEventListener
shadowRoot.getElementById('shadow-btn-1').addEventListener('click', function() {
incrementCounter('Shadow DOM');
});
</script>
</body>
</html>
+63
View File
@@ -0,0 +1,63 @@
import shutil
from pathlib import Path
import pytest
from browser_use.browser import profile as profile_module
from browser_use.browser.profile import BrowserChannel, BrowserProfile
def _create_chrome_user_data_dir(tmp_path: Path) -> Path:
user_data_dir = tmp_path / 'Chrome User Data'
default_profile = user_data_dir / 'Default'
default_profile.mkdir(parents=True)
(default_profile / 'Preferences').write_text('{"profile": "default"}')
(user_data_dir / 'Local State').write_text('{"browser": "chrome"}')
return user_data_dir
def test_chrome_profile_copy_skips_transient_lock_files(tmp_path: Path) -> None:
user_data_dir = _create_chrome_user_data_dir(tmp_path)
default_profile = user_data_dir / 'Default'
(default_profile / 'SingletonLock').write_text('locked')
(default_profile / 'Cookies-journal').write_text('journal')
browser_profile = BrowserProfile(
user_data_dir=user_data_dir,
channel=BrowserChannel.CHROME,
headless=True,
)
assert browser_profile.user_data_dir is not None
temp_user_data_dir = Path(browser_profile.user_data_dir)
try:
assert (temp_user_data_dir / 'Default' / 'Preferences').read_text() == '{"profile": "default"}'
assert (temp_user_data_dir / 'Local State').read_text() == '{"browser": "chrome"}'
assert not (temp_user_data_dir / 'Default' / 'SingletonLock').exists()
assert not (temp_user_data_dir / 'Default' / 'Cookies-journal').exists()
finally:
shutil.rmtree(temp_user_data_dir, ignore_errors=True)
def test_chrome_profile_copy_lock_error_is_actionable(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
user_data_dir = _create_chrome_user_data_dir(tmp_path)
temp_user_data_dir = tmp_path / 'browser-use-user-data-dir-test'
def fake_mkdtemp(prefix: str) -> str:
temp_user_data_dir.mkdir()
return str(temp_user_data_dir)
def fake_copytree(*_args: object, **_kwargs: object) -> None:
raise PermissionError(13, 'The process cannot access the file because it is being used by another process')
monkeypatch.setattr(profile_module.tempfile, 'mkdtemp', fake_mkdtemp)
monkeypatch.setattr(shutil, 'copytree', fake_copytree)
with pytest.raises(RuntimeError, match='Close any Chrome windows using this profile.*--cdp-url'):
BrowserProfile(
user_data_dir=user_data_dir,
channel=BrowserChannel.CHROME,
headless=True,
)
assert not temp_user_data_dir.exists()
+113
View File
@@ -0,0 +1,113 @@
import asyncio
from typing import Any
import pytest
from browser_use.browser import BrowserProfile, BrowserSession
from browser_use.browser.profile import ProxySettings
from browser_use.config import CONFIG
def test_chromium_args_include_proxy_flags():
profile = BrowserProfile(
headless=True,
user_data_dir=str(CONFIG.BROWSER_USE_PROFILES_DIR / 'proxy-smoke'),
proxy=ProxySettings(
server='http://proxy.local:8080',
bypass='localhost,127.0.0.1',
),
)
args = profile.get_args()
assert any(a == '--proxy-server=http://proxy.local:8080' for a in args), args
assert any(a == '--proxy-bypass-list=localhost,127.0.0.1' for a in args), args
@pytest.mark.asyncio
async def test_cdp_proxy_auth_handler_registers_and_responds():
# Create profile with proxy auth credentials
profile = BrowserProfile(
headless=True,
user_data_dir=str(CONFIG.BROWSER_USE_PROFILES_DIR / 'proxy-smoke'),
proxy=ProxySettings(username='user', password='pass'),
)
session = BrowserSession(browser_profile=profile)
# Stub CDP client with minimal Fetch support
class StubCDP:
def __init__(self) -> None:
self.enabled = False
self.last_auth: dict[str, Any] | None = None
self.last_default: dict[str, Any] | None = None
self.auth_callback = None
self.request_paused_callback = None
class _FetchSend:
def __init__(self, outer: 'StubCDP') -> None:
self._outer = outer
async def enable(self, params: dict, session_id: str | None = None) -> None:
self._outer.enabled = True
async def continueWithAuth(self, params: dict, session_id: str | None = None) -> None:
self._outer.last_auth = {'params': params, 'session_id': session_id}
async def continueRequest(self, params: dict, session_id: str | None = None) -> None:
# no-op; included to mirror CDP API surface used by impl
pass
class _Send:
def __init__(self, outer: 'StubCDP') -> None:
self.Fetch = _FetchSend(outer)
class _FetchRegister:
def __init__(self, outer: 'StubCDP') -> None:
self._outer = outer
def authRequired(self, callback) -> None:
self._outer.auth_callback = callback
def requestPaused(self, callback) -> None:
self._outer.request_paused_callback = callback
class _Register:
def __init__(self, outer: 'StubCDP') -> None:
self.Fetch = _FetchRegister(outer)
self.send = _Send(self)
self.register = _Register(self)
root = StubCDP()
# Attach stubs to session
session._cdp_client_root = root # type: ignore[attr-defined]
# No need to attach a real CDPSession; _setup_proxy_auth works with root client
# Should register Fetch handler and enable auth handling without raising
await session._setup_proxy_auth()
assert root.enabled is True
assert callable(root.auth_callback)
# Simulate proxy auth required event
ev = {'requestId': 'r1', 'authChallenge': {'source': 'Proxy'}}
root.auth_callback(ev, session_id='s1') # type: ignore[misc]
# Let scheduled task run
await asyncio.sleep(0.05)
assert root.last_auth is not None
params = root.last_auth['params']
assert params['authChallengeResponse']['response'] == 'ProvideCredentials'
assert params['authChallengeResponse']['username'] == 'user'
assert params['authChallengeResponse']['password'] == 'pass'
assert root.last_auth['session_id'] == 's1'
# Now simulate a non-proxy auth challenge and ensure default handling
ev2 = {'requestId': 'r2', 'authChallenge': {'source': 'Server'}}
root.auth_callback(ev2, session_id='s2') # type: ignore[misc]
await asyncio.sleep(0.05)
# After non-proxy challenge, last_auth should reflect Default response
assert root.last_auth is not None
params2 = root.last_auth['params']
assert params2['requestId'] == 'r2'
assert params2['authChallengeResponse']['response'] == 'Default'
+163
View File
@@ -0,0 +1,163 @@
import pytest
from pytest_httpserver import HTTPServer
from browser_use.agent.service import Agent
from browser_use.browser.events import NavigateToUrlEvent
from browser_use.browser.profile import BrowserProfile
from browser_use.browser.session import BrowserSession
from tests.ci.conftest import create_mock_llm
@pytest.fixture(scope='session')
def http_server():
"""Create and provide a test HTTP server for screenshot tests."""
server = HTTPServer()
server.start()
# Route: Page with visible content for screenshot testing
server.expect_request('/screenshot-page').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head>
<title>Screenshot Test Page</title>
<style>
body { font-family: Arial; padding: 20px; background: #f0f0f0; }
h1 { color: #333; font-size: 32px; }
.content { background: white; padding: 20px; border-radius: 8px; margin: 10px 0; }
</style>
</head>
<body>
<h1>Screenshot Test Page</h1>
<div class="content">
<p>This page is used to test screenshot capture with vision enabled.</p>
<p>The agent should capture a screenshot when navigating to this page.</p>
</div>
</body>
</html>
""",
content_type='text/html',
)
yield server
server.stop()
@pytest.fixture(scope='session')
def base_url(http_server):
"""Return the base URL for the test HTTP server."""
return f'http://{http_server.host}:{http_server.port}'
@pytest.fixture(scope='function')
async def browser_session():
session = BrowserSession(browser_profile=BrowserProfile(headless=True))
await session.start()
yield session
await session.kill()
@pytest.mark.asyncio
async def test_basic_screenshots(browser_session: BrowserSession, httpserver):
"""Navigate to a local page and ensure screenshot helpers return bytes."""
html = """
<html><body><h1 id='title'>Hello</h1><p>Screenshot demo.</p></body></html>
"""
httpserver.expect_request('/demo').respond_with_data(html, content_type='text/html')
url = httpserver.url_for('/demo')
nav = browser_session.event_bus.dispatch(NavigateToUrlEvent(url=url, new_tab=False))
await nav
data = await browser_session.take_screenshot(full_page=False)
assert data, 'Viewport screenshot returned no data'
element = await browser_session.screenshot_element('h1')
assert element, 'Element screenshot returned no data'
async def test_agent_screenshot_with_vision_enabled(browser_session, base_url):
"""Test that agent captures screenshots when vision is enabled.
This integration test verifies that:
1. Agent with vision=True navigates to a page
2. After prepare_context/update message manager, screenshot is captured
3. Screenshot is included in the agent's history state
"""
# Create mock LLM actions
actions = [
f"""
{{
"thinking": "I'll navigate to the screenshot test page",
"evaluation_previous_goal": "Starting task",
"memory": "Navigating to page",
"next_goal": "Navigate to test page",
"action": [
{{
"navigate": {{
"url": "{base_url}/screenshot-page",
"new_tab": false
}}
}}
]
}}
""",
"""
{
"thinking": "Page loaded, completing task",
"evaluation_previous_goal": "Page loaded",
"memory": "Task completed",
"next_goal": "Complete task",
"action": [
{
"done": {
"text": "Successfully navigated and captured screenshot",
"success": true
}
}
]
}
""",
]
mock_llm = create_mock_llm(actions=actions)
# Create agent with vision enabled
agent = Agent(
task=f'Navigate to {base_url}/screenshot-page',
llm=mock_llm,
browser_session=browser_session,
use_vision=True, # Enable vision/screenshots
)
# Run agent
history = await agent.run(max_steps=2)
# Verify agent completed successfully
assert len(history) >= 1, 'Agent should have completed at least 1 step'
final_result = history.final_result()
assert final_result is not None, 'Agent should return a final result'
# Verify screenshots were captured in the history
screenshot_found = False
for i, step in enumerate(history.history):
# Check if browser state has screenshot path
if step.state and hasattr(step.state, 'screenshot_path') and step.state.screenshot_path:
screenshot_found = True
print(f'\n✅ Step {i + 1}: Screenshot captured at {step.state.screenshot_path}')
# Verify screenshot file exists (it should be saved to disk)
import os
assert os.path.exists(step.state.screenshot_path), f'Screenshot file should exist at {step.state.screenshot_path}'
# Verify screenshot file has content
screenshot_size = os.path.getsize(step.state.screenshot_path)
assert screenshot_size > 0, f'Screenshot file should have content, got {screenshot_size} bytes'
print(f' Screenshot size: {screenshot_size} bytes')
assert screenshot_found, 'At least one screenshot should be captured when vision is enabled'
print('\n🎉 Integration test passed: Screenshots are captured correctly with vision enabled')
+436
View File
@@ -0,0 +1,436 @@
"""
Test script for BrowserSession.start() method to ensure proper initialization,
concurrency handling, and error handling.
Tests cover:
- Calling .start() on a session that's already started
- Simultaneously calling .start() from two parallel coroutines
- Calling .start() on a session that's started but has a closed browser connection
- Calling .close() on a session that hasn't been started yet
"""
import asyncio
import logging
import pytest
from browser_use.browser.profile import (
BROWSERUSE_DEFAULT_CHANNEL,
BrowserChannel,
BrowserProfile,
)
from browser_use.browser.session import BrowserSession
from browser_use.config import CONFIG
# Set up test logging
logger = logging.getLogger('browser_session_start_tests')
# logger.setLevel(logging.DEBUG)
# run with pytest -k test_user_data_dir_not_allowed_to_corrupt_default_profile
class TestBrowserSessionStart:
"""Tests for BrowserSession.start() method initialization and concurrency."""
@pytest.fixture(scope='module')
async def browser_profile(self):
"""Create and provide a BrowserProfile with headless mode."""
profile = BrowserProfile(headless=True, user_data_dir=None, keep_alive=False)
yield profile
@pytest.fixture(scope='function')
async def browser_session(self, browser_profile):
"""Create a BrowserSession instance without starting it."""
session = BrowserSession(browser_profile=browser_profile)
yield session
await session.kill()
async def test_start_already_started_session(self, browser_session):
"""Test calling .start() on a session that's already started."""
# logger.info('Testing start on already started session')
# Start the session for the first time
await browser_session.start()
assert browser_session._cdp_client_root is not None
# Start the session again - should return immediately without re-initialization
await browser_session.start()
assert browser_session._cdp_client_root is not None
# @pytest.mark.skip(reason="Race condition - DOMWatchdog tries to inject scripts into tab that's being closed")
# async def test_page_lifecycle_management(self, browser_session: BrowserSession):
# """Test session handles page lifecycle correctly."""
# # logger.info('Testing page lifecycle management')
# # Start the session and get initial state
# await browser_session.start()
# initial_tabs = await browser_session.get_tabs()
# initial_count = len(initial_tabs)
# # Get current tab info
# current_url = await browser_session.get_current_page_url()
# assert current_url is not None
# # Get current tab ID
# current_tab_id = browser_session.agent_focus.target_id if browser_session.agent_focus else None
# assert current_tab_id is not None
# # Close the current tab using the event system
# from browser_use.browser.events import CloseTabEvent
# close_event = browser_session.event_bus.dispatch(CloseTabEvent(target_id=current_tab_id))
# await close_event
# # Operations should still work - may create new page or use existing
# tabs_after_close = await browser_session.get_tabs()
# assert isinstance(tabs_after_close, list)
# # Create a new tab explicitly
# event = browser_session.event_bus.dispatch(NavigateToUrlEvent(url='about:blank', new_tab=True))
# await event
# await event.event_result(raise_if_any=True, raise_if_none=False)
# # Should have at least one tab now
# final_tabs = await browser_session.get_tabs()
# assert len(final_tabs) >= 1
async def test_user_data_dir_not_allowed_to_corrupt_default_profile(self):
"""Test user_data_dir handling for different browser channels and version mismatches."""
# Test 1: Chromium with default user_data_dir and default channel should work fine
session = BrowserSession(
browser_profile=BrowserProfile(
headless=True,
user_data_dir=CONFIG.BROWSER_USE_DEFAULT_USER_DATA_DIR,
channel=BROWSERUSE_DEFAULT_CHANNEL, # chromium
keep_alive=False,
),
)
try:
await session.start()
assert session._cdp_client_root is not None
# Verify the user_data_dir wasn't changed
assert session.browser_profile.user_data_dir == CONFIG.BROWSER_USE_DEFAULT_USER_DATA_DIR
finally:
await session.kill()
# Test 2: Chrome with default user_data_dir should change dir AND copy to temp
profile2 = BrowserProfile(
headless=True,
user_data_dir=CONFIG.BROWSER_USE_DEFAULT_USER_DATA_DIR,
channel=BrowserChannel.CHROME,
keep_alive=False,
)
# The validator should have changed the user_data_dir to avoid corruption
# And then _copy_profile copies it to a temp directory (Chrome only)
assert profile2.user_data_dir != CONFIG.BROWSER_USE_DEFAULT_USER_DATA_DIR
assert 'browser-use-user-data-dir-' in str(profile2.user_data_dir)
# Test 3: Edge with default user_data_dir should also change
profile3 = BrowserProfile(
headless=True,
user_data_dir=CONFIG.BROWSER_USE_DEFAULT_USER_DATA_DIR,
channel=BrowserChannel.MSEDGE,
keep_alive=False,
)
assert profile3.user_data_dir != CONFIG.BROWSER_USE_DEFAULT_USER_DATA_DIR
assert profile3.user_data_dir == CONFIG.BROWSER_USE_DEFAULT_USER_DATA_DIR.parent / 'default-msedge'
assert 'browser-use-user-data-dir-' not in str(profile3.user_data_dir)
class TestBrowserSessionReusePatterns:
"""Tests for all browser re-use patterns documented in docs/customize/real-browser.mdx"""
async def test_sequential_agents_same_profile_different_browser(self, mock_llm):
"""Test Sequential Agents, Same Profile, Different Browser pattern"""
from browser_use import Agent
from browser_use.browser.profile import BrowserProfile
# Create a reusable profile
reused_profile = BrowserProfile(
user_data_dir=None, # Use temp dir for testing
headless=True,
)
# First agent
agent1 = Agent(
task='The first task...',
llm=mock_llm,
browser_profile=reused_profile,
)
await agent1.run()
# Verify first agent's session is closed
assert agent1.browser_session is not None
assert not agent1.browser_session._cdp_client_root is not None
# Second agent with same profile
agent2 = Agent(
task='The second task...',
llm=mock_llm,
browser_profile=reused_profile,
# Disable memory for tests
)
await agent2.run()
# Verify second agent created a new session
assert agent2.browser_session is not None
assert agent1.browser_session is not agent2.browser_session
assert not agent2.browser_session._cdp_client_root is not None
async def test_sequential_agents_same_profile_same_browser(self, mock_llm):
"""Test Sequential Agents, Same Profile, Same Browser pattern"""
from browser_use import Agent, BrowserSession
# Create a reusable session with keep_alive
reused_session = BrowserSession(
browser_profile=BrowserProfile(
user_data_dir=None, # Use temp dir for testing
headless=True,
keep_alive=True, # Don't close browser after agent.run()
),
)
try:
# Start the session manually (agents will reuse this initialized session)
await reused_session.start()
# First agent
agent1 = Agent(
task='The first task...',
llm=mock_llm,
browser_session=reused_session,
# Disable memory for tests
)
await agent1.run()
# Verify session is still alive
assert reused_session._cdp_client_root is not None
# Second agent reusing the same session
agent2 = Agent(
task='The second task...',
llm=mock_llm,
browser_session=reused_session,
# Disable memory for tests
)
await agent2.run()
# Verify same browser was used (using __eq__ to check browser_pid, cdp_url)
assert agent1.browser_session == agent2.browser_session
assert agent1.browser_session == reused_session
assert reused_session._cdp_client_root is not None
finally:
await reused_session.kill()
class TestBrowserSessionEventSystem:
"""Tests for the new event system integration in BrowserSession."""
@pytest.fixture(scope='function')
async def browser_session(self):
"""Create a BrowserSession instance for event system testing."""
profile = BrowserProfile(headless=True, user_data_dir=None, keep_alive=False)
session = BrowserSession(browser_profile=profile)
yield session
await session.kill()
async def test_event_bus_initialization(self, browser_session):
"""Test that event bus is properly initialized with unique name."""
# Event bus should be created during __init__
assert browser_session.event_bus is not None
assert browser_session.event_bus.name.startswith('EventBus_')
# Event bus name format may vary, just check it exists
async def test_event_handlers_registration(self, browser_session: BrowserSession):
"""Test that event handlers are properly registered."""
# Attach all watchdogs to register their handlers
await browser_session.attach_all_watchdogs()
# Check that handlers are registered in the event bus
from browser_use.browser.events import (
BrowserStartEvent,
BrowserStateRequestEvent,
BrowserStopEvent,
ClickElementEvent,
CloseTabEvent,
ScreenshotEvent,
ScrollEvent,
TypeTextEvent,
)
# These event types should have handlers registered
event_types_with_handlers = [
BrowserStartEvent,
BrowserStopEvent,
ClickElementEvent,
TypeTextEvent,
ScrollEvent,
CloseTabEvent,
BrowserStateRequestEvent,
ScreenshotEvent,
]
for event_type in event_types_with_handlers:
handlers = browser_session.event_bus.handlers.get(event_type.__name__, [])
assert len(handlers) > 0, f'No handlers registered for {event_type.__name__}'
async def test_direct_event_dispatching(self, browser_session):
"""Test direct event dispatching without using the public API."""
from browser_use.browser.events import BrowserConnectedEvent, BrowserStartEvent
# Dispatch BrowserStartEvent directly
start_event = browser_session.event_bus.dispatch(BrowserStartEvent())
# Wait for event to complete
await start_event
# Check if BrowserConnectedEvent was dispatched
assert browser_session._cdp_client_root is not None
# Check event history
event_history = list(browser_session.event_bus.event_history.values())
assert len(event_history) >= 2 # BrowserStartEvent + BrowserConnectedEvent + others
# Find the BrowserConnectedEvent in history
started_events = [e for e in event_history if isinstance(e, BrowserConnectedEvent)]
assert len(started_events) >= 1
assert started_events[0].cdp_url is not None
async def test_event_system_error_handling(self, browser_session):
"""Test error handling in event system."""
from browser_use.browser.events import BrowserStartEvent
# Create session with invalid CDP URL to trigger error
error_session = BrowserSession(
browser_profile=BrowserProfile(headless=True),
cdp_url='http://localhost:99999', # Invalid port
)
try:
# Dispatch start event directly - should trigger error handling
start_event = error_session.event_bus.dispatch(BrowserStartEvent())
# The event bus catches and logs the error, but the event awaits successfully
await start_event
# The session should not be initialized due to the error
assert error_session._cdp_client_root is None, 'Session should not be initialized after connection error'
# Verify the error was logged in the event history (good enough for error handling test)
assert len(error_session.event_bus.event_history) > 0, 'Event should be tracked even with errors'
finally:
await error_session.kill()
async def test_concurrent_event_dispatching(self, browser_session: BrowserSession):
"""Test that concurrent events are handled properly."""
from browser_use.browser.events import ScreenshotEvent
# Start browser first
await browser_session.start()
# Dispatch multiple events concurrently
screenshot_event1 = browser_session.event_bus.dispatch(ScreenshotEvent())
screenshot_event2 = browser_session.event_bus.dispatch(ScreenshotEvent())
# Both should complete successfully
results = await asyncio.gather(screenshot_event1, screenshot_event2, return_exceptions=True)
# Check that no exceptions were raised
for result in results:
assert not isinstance(result, Exception), f'Event failed with: {result}'
# async def test_many_parallel_browser_sessions(self):
# """Test spawning 12 parallel browser_sessions with different settings and ensure they all work"""
# from browser_use import BrowserSession
# browser_sessions = []
# for i in range(3):
# browser_sessions.append(
# BrowserSession(
# browser_profile=BrowserProfile(
# user_data_dir=None,
# headless=True,
# keep_alive=True,
# ),
# )
# )
# for i in range(3):
# browser_sessions.append(
# BrowserSession(
# browser_profile=BrowserProfile(
# user_data_dir=Path(tempfile.mkdtemp(prefix=f'browseruse-tmp-{i}')),
# headless=True,
# keep_alive=True,
# ),
# )
# )
# for i in range(3):
# browser_sessions.append(
# BrowserSession(
# browser_profile=BrowserProfile(
# user_data_dir=None,
# headless=True,
# keep_alive=False,
# ),
# )
# )
# for i in range(3):
# browser_sessions.append(
# BrowserSession(
# browser_profile=BrowserProfile(
# user_data_dir=Path(tempfile.mkdtemp(prefix=f'browseruse-tmp-{i}')),
# headless=True,
# keep_alive=False,
# ),
# )
# )
# print('Starting many parallel browser sessions...')
# await asyncio.gather(*[browser_session.start() for browser_session in browser_sessions])
# print('Ensuring all parallel browser sessions are connected and usable...')
# new_tab_tasks = []
# for browser_session in browser_sessions:
# assert browser_session._cdp_client_root is not None
# assert browser_session._cdp_client_root is not None
# new_tab_tasks.append(browser_session.create_new_tab('chrome://version'))
# await asyncio.gather(*new_tab_tasks)
# print('killing every 3rd browser_session to test parallel shutdown')
# kill_tasks = []
# for i in range(0, len(browser_sessions), 3):
# kill_tasks.append(browser_sessions[i].kill())
# browser_sessions[i] = None
# results = await asyncio.gather(*kill_tasks, return_exceptions=True)
# # Check that no exceptions were raised during cleanup
# for i, result in enumerate(results):
# if isinstance(result, Exception):
# print(f'Warning: Browser session kill raised exception: {type(result).__name__}: {result}')
# print('ensuring the remaining browser_sessions are still connected and usable')
# new_tab_tasks = []
# screenshot_tasks = []
# for browser_session in filter(bool, browser_sessions):
# assert browser_session._cdp_client_root is not None
# assert browser_session._cdp_client_root is not None
# new_tab_tasks.append(browser_session.create_new_tab('chrome://version'))
# screenshot_tasks.append(browser_session.take_screenshot())
# await asyncio.gather(*new_tab_tasks)
# await asyncio.gather(*screenshot_tasks)
# kill_tasks = []
# print('killing the remaining browser_sessions')
# for browser_session in filter(bool, browser_sessions):
# kill_tasks.append(browser_session.kill())
# results = await asyncio.gather(*kill_tasks, return_exceptions=True)
# # Check that no exceptions were raised during cleanup
# for i, result in enumerate(results):
# if isinstance(result, Exception):
# print(f'Warning: Browser session kill raised exception: {type(result).__name__}: {result}')
+671
View File
@@ -0,0 +1,671 @@
"""
Test multi-tab operations: creation, switching, closing, and background tabs.
Tests verify that:
1. Agent can create multiple tabs (3) and switch between them
2. Agent can close tabs with vision=True
3. Agent can handle buttons that open new tabs in background
4. Agent can continue and call done() after each tab operation
5. Browser state doesn't timeout during background tab operations
All tests use:
- max_steps=5 to allow multiple tab operations
- 120s timeout to fail if test takes too long
- Mock LLM to verify agent can still make decisions after tab operations
Usage:
uv run pytest tests/ci/browser/test_tabs.py -v -s
"""
import asyncio
import time
import pytest
from pytest_httpserver import HTTPServer
from browser_use.agent.service import Agent
from browser_use.browser import BrowserSession
from browser_use.browser.profile import BrowserProfile
from tests.ci.conftest import create_mock_llm
@pytest.fixture(scope='session')
def http_server():
"""Create and provide a test HTTP server for tab tests."""
server = HTTPServer()
server.start()
# Route 1: Home page
server.expect_request('/home').respond_with_data(
'<html><head><title>Home Page</title></head><body><h1>Home Page</h1><p>This is the home page</p></body></html>',
content_type='text/html',
)
# Route 2: Page 1
server.expect_request('/page1').respond_with_data(
'<html><head><title>Page 1</title></head><body><h1>Page 1</h1><p>First test page</p></body></html>',
content_type='text/html',
)
# Route 3: Page 2
server.expect_request('/page2').respond_with_data(
'<html><head><title>Page 2</title></head><body><h1>Page 2</h1><p>Second test page</p></body></html>',
content_type='text/html',
)
# Route 4: Page 3
server.expect_request('/page3').respond_with_data(
'<html><head><title>Page 3</title></head><body><h1>Page 3</h1><p>Third test page</p></body></html>',
content_type='text/html',
)
# Route 5: Background tab page - has a link that opens a new tab in the background
server.expect_request('/background-tab-test').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head><title>Background Tab Test</title></head>
<body style="padding: 20px; font-family: Arial;">
<h1>Background Tab Test</h1>
<p>Click the link below to open a new tab in the background:</p>
<a href="/page3" target="_blank" id="open-tab-link">Open New Tab (link)</a>
<br><br>
<button id="open-tab-btn" onclick="window.open('/page3', '_blank'); document.getElementById('status').textContent='Tab opened!'">
Open New Tab (button)
</button>
<p id="status" style="margin-top: 20px; color: green;"></p>
</body>
</html>
""",
content_type='text/html',
)
yield server
server.stop()
@pytest.fixture(scope='session')
def base_url(http_server):
"""Return the base URL for the test HTTP server."""
return f'http://{http_server.host}:{http_server.port}'
@pytest.fixture(scope='function')
async def browser_session():
"""Create a browser session for tab tests."""
session = BrowserSession(
browser_profile=BrowserProfile(
headless=True,
user_data_dir=None,
keep_alive=True,
)
)
await session.start()
yield session
await session.kill()
class TestMultiTabOperations:
"""Test multi-tab creation, switching, and closing."""
async def test_create_and_switch_three_tabs(self, browser_session, base_url):
"""Test that agent can create 3 tabs, switch between them, and call done().
This test verifies that browser state is retrieved between each step.
"""
start_time = time.time()
actions = [
# Action 1: Navigate to home page
f"""
{{
"thinking": "I'll start by navigating to the home page",
"evaluation_previous_goal": "Starting task",
"memory": "Navigating to home page",
"next_goal": "Navigate to home page",
"action": [
{{
"navigate": {{
"url": "{base_url}/home",
"new_tab": false
}}
}}
]
}}
""",
# Action 2: Open page1 in new tab
f"""
{{
"thinking": "Now I'll open page 1 in a new tab",
"evaluation_previous_goal": "Home page loaded",
"memory": "Opening page 1 in new tab",
"next_goal": "Open page 1 in new tab",
"action": [
{{
"navigate": {{
"url": "{base_url}/page1",
"new_tab": true
}}
}}
]
}}
""",
# Action 3: Open page2 in new tab
f"""
{{
"thinking": "Now I'll open page 2 in a new tab",
"evaluation_previous_goal": "Page 1 opened in new tab",
"memory": "Opening page 2 in new tab",
"next_goal": "Open page 2 in new tab",
"action": [
{{
"navigate": {{
"url": "{base_url}/page2",
"new_tab": true
}}
}}
]
}}
""",
# Action 4: Switch to first tab
"""
{
"thinking": "Now I'll switch back to the first tab",
"evaluation_previous_goal": "Page 2 opened in new tab",
"memory": "Switching to first tab",
"next_goal": "Switch to first tab",
"action": [
{
"switch": {
"tab_id": "0000"
}
}
]
}
""",
# Action 5: Done
"""
{
"thinking": "I've successfully created 3 tabs and switched between them",
"evaluation_previous_goal": "Switched to first tab",
"memory": "All tabs created and switched",
"next_goal": "Complete task",
"action": [
{
"done": {
"text": "Successfully created 3 tabs and switched between them",
"success": true
}
}
]
}
""",
]
mock_llm = create_mock_llm(actions=actions)
agent = Agent(
task=f'Navigate to {base_url}/home, then open {base_url}/page1 and {base_url}/page2 in new tabs, then switch back to the first tab',
llm=mock_llm,
browser_session=browser_session,
)
# Run with timeout - should complete within 2 minutes
try:
history = await asyncio.wait_for(agent.run(max_steps=5), timeout=120)
elapsed = time.time() - start_time
print(f'\n⏱️ Test completed in {elapsed:.2f} seconds')
print(f'📊 Completed {len(history)} steps')
# Verify each step has browser state
for i, step in enumerate(history.history):
assert step.state is not None, f'Step {i} should have browser state'
assert step.state.url is not None, f'Step {i} should have URL in browser state'
print(f' Step {i + 1}: URL={step.state.url}, tabs={len(step.state.tabs) if step.state.tabs else 0}')
assert len(history) >= 4, 'Agent should have completed at least 4 steps'
# Verify we have 3 tabs open
tabs = await browser_session.get_tabs()
assert len(tabs) >= 3, f'Should have at least 3 tabs open, got {len(tabs)}'
# Verify agent completed successfully
final_result = history.final_result()
assert final_result is not None, 'Agent should return a final result'
assert 'Successfully' in final_result, 'Agent should report success'
# Note: Test is fast (< 1s) because mock LLM returns instantly and pages are simple,
# but browser state IS being retrieved correctly between steps as verified above
except TimeoutError:
pytest.fail('Test timed out after 2 minutes - agent hung during tab operations')
async def test_close_tab_with_vision(self, browser_session, base_url):
"""Test that agent can close a tab with vision=True and call done()."""
actions = [
# Action 1: Navigate to home page
f"""
{{
"thinking": "I'll start by navigating to the home page",
"evaluation_previous_goal": "Starting task",
"memory": "Navigating to home page",
"next_goal": "Navigate to home page",
"action": [
{{
"navigate": {{
"url": "{base_url}/home",
"new_tab": false
}}
}}
]
}}
""",
# Action 2: Open page1 in new tab
f"""
{{
"thinking": "Now I'll open page 1 in a new tab",
"evaluation_previous_goal": "Home page loaded",
"memory": "Opening page 1 in new tab",
"next_goal": "Open page 1 in new tab",
"action": [
{{
"navigate": {{
"url": "{base_url}/page1",
"new_tab": true
}}
}}
]
}}
""",
# Action 3: Close the current tab
"""
{
"thinking": "Now I'll close the current tab (page1)",
"evaluation_previous_goal": "Page 1 opened in new tab",
"memory": "Closing current tab",
"next_goal": "Close current tab",
"action": [
{
"close": {
"tab_id": "0001"
}
}
]
}
""",
# Action 4: Done
"""
{
"thinking": "I've successfully closed the tab",
"evaluation_previous_goal": "Tab closed",
"memory": "Tab closed successfully",
"next_goal": "Complete task",
"action": [
{
"done": {
"text": "Successfully closed the tab",
"success": true
}
}
]
}
""",
]
mock_llm = create_mock_llm(actions=actions)
agent = Agent(
task=f'Navigate to {base_url}/home, then open {base_url}/page1 in a new tab, then close the page1 tab',
llm=mock_llm,
browser_session=browser_session,
use_vision=True, # Enable vision for this test
)
# Run with timeout - should complete within 2 minutes
try:
history = await asyncio.wait_for(agent.run(max_steps=5), timeout=120)
assert len(history) >= 3, 'Agent should have completed at least 3 steps'
# Verify agent completed successfully
final_result = history.final_result()
assert final_result is not None, 'Agent should return a final result'
assert 'Successfully' in final_result, 'Agent should report success'
except TimeoutError:
pytest.fail('Test timed out after 2 minutes - agent hung during tab closing with vision')
async def test_background_tab_open_no_timeout(self, browser_session, base_url):
"""Test that browser state doesn't timeout when a new tab opens in the background."""
start_time = time.time()
actions = [
# Action 1: Navigate to home page
f"""
{{
"thinking": "I'll navigate to the home page first",
"evaluation_previous_goal": "Starting task",
"memory": "Navigating to home page",
"next_goal": "Navigate to home page",
"action": [
{{
"navigate": {{
"url": "{base_url}/home",
"new_tab": false
}}
}}
]
}}
""",
# Action 2: Open page1 in new background tab (stay on home page)
f"""
{{
"thinking": "I'll open page1 in a new background tab",
"evaluation_previous_goal": "Home page loaded",
"memory": "Opening background tab",
"next_goal": "Open background tab without switching to it",
"action": [
{{
"navigate": {{
"url": "{base_url}/page1",
"new_tab": true
}}
}}
]
}}
""",
# Action 3: Immediately check browser state after background tab opens
"""
{
"thinking": "After opening background tab, browser state should still be accessible",
"evaluation_previous_goal": "Background tab opened",
"memory": "Verifying browser state works",
"next_goal": "Complete task",
"action": [
{
"done": {
"text": "Successfully opened background tab, browser state remains accessible",
"success": true
}
}
]
}
""",
]
mock_llm = create_mock_llm(actions=actions)
agent = Agent(
task=f'Navigate to {base_url}/home and open {base_url}/page1 in a new tab',
llm=mock_llm,
browser_session=browser_session,
)
# Run with timeout - this tests if browser state times out when new tabs open
try:
history = await asyncio.wait_for(agent.run(max_steps=3), timeout=120)
elapsed = time.time() - start_time
print(f'\n⏱️ Test completed in {elapsed:.2f} seconds')
print(f'📊 Completed {len(history)} steps')
# Verify each step has browser state (the key test - no timeouts)
for i, step in enumerate(history.history):
assert step.state is not None, f'Step {i} should have browser state'
assert step.state.url is not None, f'Step {i} should have URL in browser state'
print(f' Step {i + 1}: URL={step.state.url}, tabs={len(step.state.tabs) if step.state.tabs else 0}')
assert len(history) >= 2, 'Agent should have completed at least 2 steps'
# Verify agent completed successfully
final_result = history.final_result()
assert final_result is not None, 'Agent should return a final result'
assert 'Successfully' in final_result, 'Agent should report success'
# Verify we have at least 2 tabs
tabs = await browser_session.get_tabs()
print(f' Final tab count: {len(tabs)}')
assert len(tabs) >= 2, f'Should have at least 2 tabs after opening background tab, got {len(tabs)}'
except TimeoutError:
pytest.fail('Test timed out after 2 minutes - browser state timed out after opening background tab')
async def test_rapid_tab_operations_no_timeout(self, browser_session, base_url):
"""Test that browser state doesn't timeout during rapid tab operations."""
actions = [
# Action 1: Navigate to home page
f"""
{{
"thinking": "I'll navigate to the home page",
"evaluation_previous_goal": "Starting task",
"memory": "Navigating to home page",
"next_goal": "Navigate to home page",
"action": [
{{
"navigate": {{
"url": "{base_url}/home",
"new_tab": false
}}
}}
]
}}
""",
# Action 2: Open page1 in new tab
f"""
{{
"thinking": "Opening page1 in new tab",
"evaluation_previous_goal": "Home page loaded",
"memory": "Opening page1",
"next_goal": "Open page1",
"action": [
{{
"navigate": {{
"url": "{base_url}/page1",
"new_tab": true
}}
}}
]
}}
""",
# Action 3: Open page2 in new tab
f"""
{{
"thinking": "Opening page2 in new tab",
"evaluation_previous_goal": "Page1 opened",
"memory": "Opening page2",
"next_goal": "Open page2",
"action": [
{{
"navigate": {{
"url": "{base_url}/page2",
"new_tab": true
}}
}}
]
}}
""",
# Action 4: Open page3 in new tab
f"""
{{
"thinking": "Opening page3 in new tab",
"evaluation_previous_goal": "Page2 opened",
"memory": "Opening page3",
"next_goal": "Open page3",
"action": [
{{
"navigate": {{
"url": "{base_url}/page3",
"new_tab": true
}}
}}
]
}}
""",
# Action 5: Verify browser state is still accessible
"""
{
"thinking": "All tabs opened rapidly, browser state should still be accessible",
"evaluation_previous_goal": "Page3 opened",
"memory": "All tabs opened",
"next_goal": "Complete task",
"action": [
{
"done": {
"text": "Successfully opened 4 tabs rapidly without timeout",
"success": true
}
}
]
}
""",
]
mock_llm = create_mock_llm(actions=actions)
agent = Agent(
task='Open multiple tabs rapidly and verify browser state remains accessible',
llm=mock_llm,
browser_session=browser_session,
)
# Run with timeout - should complete within 2 minutes
try:
history = await asyncio.wait_for(agent.run(max_steps=5), timeout=120)
assert len(history) >= 4, 'Agent should have completed at least 4 steps'
# Verify we have 4 tabs open
tabs = await browser_session.get_tabs()
assert len(tabs) >= 4, f'Should have at least 4 tabs open, got {len(tabs)}'
# Verify agent completed successfully
final_result = history.final_result()
assert final_result is not None, 'Agent should return a final result'
assert 'Successfully' in final_result, 'Agent should report success'
except TimeoutError:
pytest.fail('Test timed out after 2 minutes - browser state timed out during rapid tab operations')
async def test_multiple_tab_switches_and_close(self, browser_session, base_url):
"""Test that agent can switch between multiple tabs and close one."""
actions = [
# Action 1: Navigate to home page
f"""
{{
"thinking": "I'll start by navigating to the home page",
"evaluation_previous_goal": "Starting task",
"memory": "Navigating to home page",
"next_goal": "Navigate to home page",
"action": [
{{
"navigate": {{
"url": "{base_url}/home",
"new_tab": false
}}
}}
]
}}
""",
# Action 2: Open page1 in new tab
f"""
{{
"thinking": "Opening page 1 in new tab",
"evaluation_previous_goal": "Home page loaded",
"memory": "Opening page 1",
"next_goal": "Open page 1",
"action": [
{{
"navigate": {{
"url": "{base_url}/page1",
"new_tab": true
}}
}}
]
}}
""",
# Action 3: Open page2 in new tab
f"""
{{
"thinking": "Opening page 2 in new tab",
"evaluation_previous_goal": "Page 1 opened",
"memory": "Opening page 2",
"next_goal": "Open page 2",
"action": [
{{
"navigate": {{
"url": "{base_url}/page2",
"new_tab": true
}}
}}
]
}}
""",
# Action 4: Switch to tab 1
"""
{
"thinking": "Switching to tab 1 (page1)",
"evaluation_previous_goal": "Page 2 opened",
"memory": "Switching to page 1",
"next_goal": "Switch to page 1",
"action": [
{
"switch": {
"tab_id": "0001"
}
}
]
}
""",
# Action 5: Close current tab
"""
{
"thinking": "Closing the current tab (page1)",
"evaluation_previous_goal": "Switched to page 1",
"memory": "Closing page 1",
"next_goal": "Close page 1",
"action": [
{
"close": {
"tab_id": "0001"
}
}
]
}
""",
# Action 6: Done
"""
{
"thinking": "Successfully completed all tab operations",
"evaluation_previous_goal": "Tab closed",
"memory": "All operations completed",
"next_goal": "Complete task",
"action": [
{
"done": {
"text": "Successfully created, switched, and closed tabs",
"success": true
}
}
]
}
""",
]
mock_llm = create_mock_llm(actions=actions)
agent = Agent(
task='Create 3 tabs, switch to the second one, then close it',
llm=mock_llm,
browser_session=browser_session,
)
# Run with timeout - should complete within 2 minutes
try:
history = await asyncio.wait_for(agent.run(max_steps=6), timeout=120)
assert len(history) >= 5, 'Agent should have completed at least 5 steps'
# Verify agent completed successfully
final_result = history.final_result()
assert final_result is not None, 'Agent should return a final result'
assert 'Successfully' in final_result, 'Agent should report success'
except TimeoutError:
pytest.fail('Test timed out after 2 minutes - agent hung during multiple tab operations')
@@ -0,0 +1,136 @@
"""Test clicking elements inside TRUE cross-origin iframes (external domains)."""
import asyncio
import pytest
from browser_use.browser.profile import BrowserProfile, ViewportSize
from browser_use.browser.session import BrowserSession
from browser_use.tools.service import Tools
@pytest.fixture
async def browser_session():
"""Create browser session with cross-origin iframe support."""
session = BrowserSession(
browser_profile=BrowserProfile(
headless=True,
user_data_dir=None,
keep_alive=True,
window_size=ViewportSize(width=1920, height=1400),
cross_origin_iframes=True, # Enable cross-origin iframe extraction
)
)
await session.start()
yield session
await session.kill()
class TestTrueCrossOriginIframeClick:
"""Test clicking elements inside true cross-origin iframes."""
async def test_click_element_in_true_cross_origin_iframe(self, httpserver, browser_session: BrowserSession):
"""Verify that elements inside TRUE cross-origin iframes (example.com) can be clicked.
This test uses example.com which is a real external domain, testing actual cross-origin
iframe extraction and clicking via CDP target switching.
"""
# Create main page with TRUE cross-origin iframe pointing to example.com
main_html = """
<!DOCTYPE html>
<html>
<head><title>True Cross-Origin Test</title></head>
<body>
<h1>Main Page</h1>
<button id="main-button">Main Button</button>
<iframe id="cross-origin" src="https://example.com" style="width: 800px; height: 600px;"></iframe>
</body>
</html>
"""
# Serve the main page
httpserver.expect_request('/true-cross-origin-test').respond_with_data(main_html, content_type='text/html')
url = httpserver.url_for('/true-cross-origin-test')
# Navigate to the page
await browser_session.navigate_to(url)
# Wait for cross-origin iframe to load (network request)
await asyncio.sleep(5)
# Get DOM state with cross-origin iframe extraction enabled
browser_state = await browser_session.get_browser_state_summary(
include_screenshot=False,
include_recent_events=False,
)
assert browser_state.dom_state is not None
state = browser_state.dom_state
print(f'\n📊 Found {len(state.selector_map)} total elements')
# Find elements from different targets
targets_found = set()
main_page_elements = []
cross_origin_elements = []
for idx, element in state.selector_map.items():
target_id = element.target_id
targets_found.add(target_id)
# Check if element is from cross-origin iframe (example.com)
# Look for links - example.com has a link to iana.org/domains/reserved
if element.attributes:
href = element.attributes.get('href', '')
element_id = element.attributes.get('id', '')
# example.com has a link to iana.org/domains/reserved
if 'iana.org' in href:
cross_origin_elements.append((idx, element))
print(f' ✅ Found cross-origin element: [{idx}] {element.tag_name} href={href}')
elif element_id == 'main-button':
main_page_elements.append((idx, element))
# Verify we found elements from at least 2 different targets
print(f'\n🎯 Found elements from {len(targets_found)} different CDP targets')
# Check if cross-origin iframe loaded
if len(targets_found) < 2:
print('⚠️ Warning: Cross-origin iframe did not create separate CDP target')
print(' This may indicate cross_origin_iframes feature is not working as expected')
pytest.skip('Cross-origin iframe did not create separate CDP target - skipping test')
if len(cross_origin_elements) == 0:
print('⚠️ Warning: No elements found from example.com iframe')
print(' Network may be restricted in CI environment')
pytest.skip('No elements extracted from example.com - skipping click test')
# Verify we found at least one element from the cross-origin iframe
assert len(cross_origin_elements) > 0, 'Expected to find at least one element from cross-origin iframe (example.com)'
# Try clicking the cross-origin element
print('\n🖱️ Testing Click on True Cross-Origin Iframe Element:')
tools = Tools()
link_idx, link_element = cross_origin_elements[0]
print(f' Attempting to click element [{link_idx}] from example.com iframe...')
try:
result = await tools.click(index=link_idx, browser_session=browser_session)
# Check for errors
if result.error:
pytest.fail(f'Click on cross-origin element [{link_idx}] failed with error: {result.error}')
if result.extracted_content and (
'not available' in result.extracted_content.lower() or 'failed' in result.extracted_content.lower()
):
pytest.fail(f'Click on cross-origin element [{link_idx}] failed: {result.extracted_content}')
print(f' ✅ Click succeeded on cross-origin element [{link_idx}]!')
print(' 🎉 True cross-origin iframe element clicking works!')
except Exception as e:
pytest.fail(f'Exception while clicking cross-origin element [{link_idx}]: {e}')
print('\n✅ Test passed: True cross-origin iframe elements can be clicked')
+238
View File
@@ -0,0 +1,238 @@
"""
Pytest configuration for browser-use CI tests.
Sets up environment variables to ensure tests never connect to production services.
"""
import os
import socketserver
import tempfile
from unittest.mock import AsyncMock
import pytest
from dotenv import load_dotenv
from pytest_httpserver import HTTPServer
# Fix for httpserver hanging on shutdown - prevent blocking on socket close
# This prevents tests from hanging when shutting down HTTP servers
socketserver.ThreadingMixIn.block_on_close = False
# Also set daemon threads to prevent hanging
socketserver.ThreadingMixIn.daemon_threads = True
from browser_use.agent.views import AgentOutput
from browser_use.llm import BaseChatModel
from browser_use.llm.views import ChatInvokeCompletion
from browser_use.tools.service import Tools
# Load environment variables before any imports
load_dotenv()
# Skip LLM API key verification for tests
os.environ['SKIP_LLM_API_KEY_VERIFICATION'] = 'true'
from bubus import BaseEvent
from browser_use import Agent
from browser_use.browser import BrowserProfile, BrowserSession
from browser_use.sync.service import CloudSync
@pytest.fixture(autouse=True)
def setup_test_environment():
"""
Automatically set up test environment for all tests.
"""
# Create a temporary directory for test config (but not for extensions)
config_dir = tempfile.mkdtemp(prefix='browseruse_tests_')
original_env = {}
test_env_vars = {
'SKIP_LLM_API_KEY_VERIFICATION': 'true',
'ANONYMIZED_TELEMETRY': 'false',
'BROWSER_USE_CLOUD_SYNC': 'true',
'BROWSER_USE_CLOUD_API_URL': 'http://placeholder-will-be-replaced-by-specific-test-fixtures',
'BROWSER_USE_CLOUD_UI_URL': 'http://placeholder-will-be-replaced-by-specific-test-fixtures',
# Don't set BROWSER_USE_CONFIG_DIR anymore - let it use the default ~/.config/browseruse
# This way extensions will be cached in ~/.config/browseruse/extensions
}
for key, value in test_env_vars.items():
original_env[key] = os.environ.get(key)
os.environ[key] = value
yield
# Restore original environment
for key, value in original_env.items():
if value is None:
os.environ.pop(key, None)
else:
os.environ[key] = value
# not a fixture, mock_llm() provides this in a fixture below, this is a helper so that it can accept args
def create_mock_llm(actions: list[str] | None = None) -> BaseChatModel:
"""Create a mock LLM that returns specified actions or a default done action.
Args:
actions: Optional list of JSON strings representing actions to return in sequence.
If not provided, returns a single done action.
After all actions are exhausted, returns a done action.
Returns:
Mock LLM that will return the actions in order, or just a done action if no actions provided.
"""
tools = Tools()
ActionModel = tools.registry.create_action_model()
AgentOutputWithActions = AgentOutput.type_with_custom_actions(ActionModel)
llm = AsyncMock(spec=BaseChatModel)
llm.model = 'mock-llm'
llm._verified_api_keys = True
# Add missing properties from BaseChatModel protocol
llm.provider = 'mock'
llm.name = 'mock-llm'
llm.model_name = 'mock-llm' # Ensure this returns a string, not a mock
# Default done action
default_done_action = """
{
"thinking": "null",
"evaluation_previous_goal": "Successfully completed the task",
"memory": "Task completed",
"next_goal": "Task completed",
"action": [
{
"done": {
"text": "Task completed successfully",
"success": true
}
}
]
}
"""
# Unified logic for both cases
action_index = 0
def get_next_action() -> str:
nonlocal action_index
if actions is not None and action_index < len(actions):
action = actions[action_index]
action_index += 1
return action
else:
return default_done_action
async def mock_ainvoke(*args, **kwargs):
# Check if output_format is provided (2nd argument or in kwargs)
output_format = None
if len(args) >= 2:
output_format = args[1]
elif 'output_format' in kwargs:
output_format = kwargs['output_format']
action_json = get_next_action()
if output_format is None:
# Return string completion
return ChatInvokeCompletion(completion=action_json, usage=None)
else:
# Parse with provided output_format (could be AgentOutputWithActions or another model)
if output_format == AgentOutputWithActions:
parsed = AgentOutputWithActions.model_validate_json(action_json)
else:
# For other output formats, try to parse the JSON with that model
parsed = output_format.model_validate_json(action_json)
return ChatInvokeCompletion(completion=parsed, usage=None)
llm.ainvoke.side_effect = mock_ainvoke
return llm
@pytest.fixture(scope='module')
async def browser_session():
"""Create a real browser session for testing"""
session = BrowserSession(
browser_profile=BrowserProfile(
headless=True,
user_data_dir=None, # Use temporary directory
keep_alive=True,
enable_default_extensions=True, # Enable extensions during tests
)
)
await session.start()
yield session
await session.kill()
# Ensure event bus is properly stopped
await session.event_bus.stop(clear=True, timeout=5)
@pytest.fixture(scope='function')
def cloud_sync(httpserver: HTTPServer):
"""
Create a CloudSync instance configured for testing.
This fixture creates a real CloudSync instance and sets up the test environment
to use the httpserver URLs.
"""
# Set up test environment
test_http_server_url = httpserver.url_for('')
os.environ['BROWSER_USE_CLOUD_API_URL'] = test_http_server_url
os.environ['BROWSER_USE_CLOUD_UI_URL'] = test_http_server_url
os.environ['BROWSER_USE_CLOUD_SYNC'] = 'true'
# Create CloudSync with test server URL
cloud_sync = CloudSync(
base_url=test_http_server_url,
)
return cloud_sync
@pytest.fixture(scope='function')
def mock_llm():
"""Create a mock LLM that just returns the done action if queried"""
return create_mock_llm(actions=None)
@pytest.fixture(scope='function')
def agent_with_cloud(browser_session, mock_llm, cloud_sync):
"""Create agent (cloud_sync parameter removed)."""
agent = Agent(
task='Test task',
llm=mock_llm,
browser_session=browser_session,
)
return agent
@pytest.fixture(scope='function')
def event_collector():
"""Helper to collect all events emitted during tests"""
events = []
event_order = []
class EventCollector:
def __init__(self):
self.events = events
self.event_order = event_order
async def collect_event(self, event: BaseEvent):
self.events.append(event)
self.event_order.append(event.event_type)
return 'collected'
def get_events_by_type(self, event_type: str) -> list[BaseEvent]:
return [e for e in self.events if e.event_type == event_type]
def clear(self):
self.events.clear()
self.event_order.clear()
return EventCollector()
+372
View File
@@ -0,0 +1,372 @@
"""
Runs all agent tasks in parallel (up to 10 at a time) using separate subprocesses.
Each task gets its own Python process, preventing browser session interference.
Fails with exit code 1 if 0% of tasks pass.
"""
import argparse
import asyncio
import glob
import json
import logging
import os
import sys
import warnings
import anyio
import yaml
from dotenv import load_dotenv
from pydantic import BaseModel
load_dotenv()
from browser_use import Agent, AgentHistoryList, BrowserProfile, BrowserSession, ChatBrowserUse
from browser_use.llm.google.chat import ChatGoogle
from browser_use.llm.messages import UserMessage
# --- CONFIG ---
MAX_PARALLEL = 10
TASK_DIR = (
sys.argv[1]
if len(sys.argv) > 1 and not sys.argv[1].startswith('--')
else os.path.join(os.path.dirname(__file__), '../agent_tasks')
)
TASK_FILES = glob.glob(os.path.join(TASK_DIR, '*.yaml'))
class JudgeResponse(BaseModel):
success: bool
explanation: str
async def run_single_task(task_file):
"""Run a single task in the current process (called by subprocess)"""
try:
print(f'[DEBUG] Starting task: {os.path.basename(task_file)}', file=sys.stderr)
# Suppress all logging in subprocess to avoid interfering with JSON output
logging.getLogger().setLevel(logging.CRITICAL)
for logger_name in ['browser_use', 'telemetry', 'message_manager']:
logging.getLogger(logger_name).setLevel(logging.CRITICAL)
warnings.filterwarnings('ignore')
print('[DEBUG] Loading task file...', file=sys.stderr)
content = await anyio.Path(task_file).read_text()
task_data = yaml.safe_load(content)
task = task_data['task']
judge_context = task_data.get('judge_context', ['The agent must solve the task'])
max_steps = task_data.get('max_steps', 15)
print(f'[DEBUG] Task: {task[:100]}...', file=sys.stderr)
print(f'[DEBUG] Max steps: {max_steps}', file=sys.stderr)
api_key = os.getenv('BROWSER_USE_API_KEY')
if not api_key:
print('[SKIP] BROWSER_USE_API_KEY is not set - skipping task evaluation', file=sys.stderr)
return {
'file': os.path.basename(task_file),
'success': True, # Mark as success so it doesn't fail CI
'explanation': 'Skipped - API key not available (fork PR or missing secret)',
}
agent_llm = ChatBrowserUse(api_key=api_key)
# Check if Google API key is available for judge LLM
google_api_key = os.getenv('GOOGLE_API_KEY')
if not google_api_key:
print('[SKIP] GOOGLE_API_KEY is not set - skipping task evaluation', file=sys.stderr)
return {
'file': os.path.basename(task_file),
'success': True, # Mark as success so it doesn't fail CI
'explanation': 'Skipped - Google API key not available (fork PR or missing secret)',
}
judge_llm = ChatGoogle(model='gemini-3.1-flash-lite')
print('[DEBUG] LLMs initialized', file=sys.stderr)
# Each subprocess gets its own profile and session
print('[DEBUG] Creating browser session...', file=sys.stderr)
profile = BrowserProfile(
headless=True,
user_data_dir=None,
chromium_sandbox=False, # Disable sandbox for CI environment (GitHub Actions)
)
session = BrowserSession(browser_profile=profile)
print('[DEBUG] Browser session created', file=sys.stderr)
# Test if browser is working
try:
await session.start()
from browser_use.browser.events import NavigateToUrlEvent
event = session.event_bus.dispatch(NavigateToUrlEvent(url='https://httpbin.org/get', new_tab=True))
await event
print('[DEBUG] Browser test: navigation successful', file=sys.stderr)
title = await session.get_current_page_title()
print(f"[DEBUG] Browser test: got title '{title}'", file=sys.stderr)
except Exception as browser_error:
print(f'[DEBUG] Browser test failed: {str(browser_error)}', file=sys.stderr)
print(
f'[DEBUG] Browser error type: {type(browser_error).__name__}',
file=sys.stderr,
)
print('[DEBUG] Starting agent execution...', file=sys.stderr)
agent = Agent(task=task, llm=agent_llm, browser_session=session)
try:
history: AgentHistoryList = await agent.run(max_steps=max_steps)
print('[DEBUG] Agent.run() returned successfully', file=sys.stderr)
except Exception as agent_error:
print(
f'[DEBUG] Agent.run() failed with error: {str(agent_error)}',
file=sys.stderr,
)
print(f'[DEBUG] Error type: {type(agent_error).__name__}', file=sys.stderr)
# Re-raise to be caught by outer try-catch
raise agent_error
agent_output = history.final_result() or ''
print('[DEBUG] Agent execution completed', file=sys.stderr)
# Test if LLM is working by making a simple call
try:
response = await agent_llm.ainvoke([UserMessage(content="Say 'test'")])
print(
f'[DEBUG] LLM test call successful: {response.completion[:50]}',
file=sys.stderr,
)
except Exception as llm_error:
print(f'[DEBUG] LLM test call failed: {str(llm_error)}', file=sys.stderr)
# Debug: capture more details about the agent execution
total_steps = len(history.history) if hasattr(history, 'history') else 0
last_action = history.history[-1] if hasattr(history, 'history') and history.history else None
debug_info = f'Steps: {total_steps}, Final result length: {len(agent_output)}'
if last_action:
debug_info += f', Last action: {type(last_action).__name__}'
# Log to stderr so it shows up in GitHub Actions (won't interfere with JSON output to stdout)
print(f'[DEBUG] Task {os.path.basename(task_file)}: {debug_info}', file=sys.stderr)
if agent_output:
print(
f'[DEBUG] Agent output preview: {agent_output[:200]}...',
file=sys.stderr,
)
else:
print('[DEBUG] Agent produced no output!', file=sys.stderr)
criteria = '\n- '.join(judge_context)
judge_prompt = f"""
You are a evaluator of a browser agent task inside a ci/cd pipeline. Here was the agent's task:
{task}
Here is the agent's output:
{agent_output if agent_output else '[No output provided]'}
Debug info: {debug_info}
Criteria for success:
- {criteria}
Reply in JSON with keys: success (true/false), explanation (string).
If the agent provided no output, explain what might have gone wrong.
"""
response = await judge_llm.ainvoke([UserMessage(content=judge_prompt)], output_format=JudgeResponse)
judge_response = response.completion
result = {
'file': os.path.basename(task_file),
'success': judge_response.success,
'explanation': judge_response.explanation,
}
# Clean up session before returning
await session.kill()
return result
except Exception as e:
# Ensure session cleanup even on error
try:
await session.kill()
except Exception:
pass
return {
'file': os.path.basename(task_file),
'success': False,
'explanation': f'Task failed with error: {str(e)}',
}
async def run_task_subprocess(task_file, semaphore):
"""Run a task in a separate subprocess"""
async with semaphore:
try:
# Set environment to reduce noise in subprocess
env = os.environ.copy()
env['PYTHONPATH'] = os.pathsep.join(sys.path)
proc = await asyncio.create_subprocess_exec(
sys.executable,
__file__,
'--task',
task_file,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
env=env,
)
stdout, stderr = await proc.communicate()
if proc.returncode == 0:
try:
# Parse JSON result from subprocess
stdout_text = stdout.decode().strip()
stderr_text = stderr.decode().strip()
# Display subprocess debug logs
if stderr_text:
print(f'[SUBPROCESS {os.path.basename(task_file)}] Debug output:')
for line in stderr_text.split('\n'):
if line.strip():
print(f' {line}')
# Find the JSON line (should be the last line that starts with {)
lines = stdout_text.split('\n')
json_line = None
for line in reversed(lines):
line = line.strip()
if line.startswith('{') and line.endswith('}'):
json_line = line
break
if json_line:
result = json.loads(json_line)
print(f'[PARENT] Task {os.path.basename(task_file)} completed: {result["success"]}')
else:
raise ValueError(f'No JSON found in output: {stdout_text}')
except (json.JSONDecodeError, ValueError) as e:
result = {
'file': os.path.basename(task_file),
'success': False,
'explanation': f'Failed to parse subprocess result: {str(e)[:100]}',
}
print(f'[PARENT] Task {os.path.basename(task_file)} failed to parse: {str(e)}')
print(f'[PARENT] Full stdout was: {stdout.decode()[:500]}')
else:
stderr_text = stderr.decode().strip()
result = {
'file': os.path.basename(task_file),
'success': False,
'explanation': f'Subprocess failed (code {proc.returncode}): {stderr_text[:200]}',
}
print(f'[PARENT] Task {os.path.basename(task_file)} subprocess failed with code {proc.returncode}')
if stderr_text:
print(f'[PARENT] stderr: {stderr_text[:1000]}')
stdout_text = stdout.decode().strip()
if stdout_text:
print(f'[PARENT] stdout: {stdout_text[:1000]}')
except Exception as e:
result = {
'file': os.path.basename(task_file),
'success': False,
'explanation': f'Failed to start subprocess: {str(e)}',
}
print(f'[PARENT] Failed to start subprocess for {os.path.basename(task_file)}: {str(e)}')
return result
async def main():
"""Run all tasks in parallel using subprocesses"""
semaphore = asyncio.Semaphore(MAX_PARALLEL)
print(f'Found task files: {TASK_FILES}')
if not TASK_FILES:
print('No task files found!')
return 0, 0
# Run all tasks in parallel subprocesses
tasks = [run_task_subprocess(task_file, semaphore) for task_file in TASK_FILES]
results = await asyncio.gather(*tasks)
passed = sum(1 for r in results if r['success'])
total = len(results)
print('\n' + '=' * 60)
print(f'{"RESULTS":^60}\n')
# Prepare table data
headers = ['Task', 'Success', 'Reason']
rows = []
for r in results:
status = '' if r['success'] else ''
rows.append([r['file'], status, r['explanation']])
# Calculate column widths
col_widths = [max(len(str(row[i])) for row in ([headers] + rows)) for i in range(3)]
# Print header
header_row = ' | '.join(headers[i].ljust(col_widths[i]) for i in range(3))
print(header_row)
print('-+-'.join('-' * w for w in col_widths))
# Print rows
for row in rows:
print(' | '.join(str(row[i]).ljust(col_widths[i]) for i in range(3)))
print('\n' + '=' * 60)
print(f'\n{"SCORE":^60}')
print(f'\n{"=" * 60}\n')
print(f'\n{"*" * 10} {passed}/{total} PASSED {"*" * 10}\n')
print('=' * 60 + '\n')
# Output results for GitHub Actions
print(f'PASSED={passed}')
print(f'TOTAL={total}')
# Output detailed results as JSON for GitHub Actions
detailed_results = []
for r in results:
detailed_results.append(
{
'task': r['file'].replace('.yaml', ''),
'success': r['success'],
'reason': r['explanation'],
}
)
print('DETAILED_RESULTS=' + json.dumps(detailed_results))
return passed, total
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--task', type=str, help='Path to a single task YAML file (for subprocess mode)')
args = parser.parse_args()
if args.task:
# Subprocess mode: run a single task and output ONLY JSON
try:
result = asyncio.run(run_single_task(args.task))
# Output ONLY the JSON result, nothing else
print(json.dumps(result))
except Exception as e:
# Even on critical failure, output valid JSON
error_result = {
'file': os.path.basename(args.task),
'success': False,
'explanation': f'Critical subprocess error: {str(e)}',
}
print(json.dumps(error_result))
else:
# Parent process mode: run all tasks in parallel subprocesses
passed, total = asyncio.run(main())
# Results already printed by main() function
# Fail if 0% pass rate (all tasks failed)
if total > 0 and passed == 0:
print('\n❌ CRITICAL: 0% pass rate - all tasks failed!')
sys.exit(1)
+120
View File
@@ -0,0 +1,120 @@
"""Tests for lazy loading configuration system."""
import os
from browser_use.config import CONFIG
class TestLazyConfig:
"""Test lazy loading of environment variables through CONFIG object."""
def test_config_reads_env_vars_lazily(self):
"""Test that CONFIG reads environment variables each time they're accessed."""
# Set an env var
original_value = os.environ.get('BROWSER_USE_LOGGING_LEVEL', '')
try:
os.environ['BROWSER_USE_LOGGING_LEVEL'] = 'debug'
assert CONFIG.BROWSER_USE_LOGGING_LEVEL == 'debug'
# Change the env var
os.environ['BROWSER_USE_LOGGING_LEVEL'] = 'info'
assert CONFIG.BROWSER_USE_LOGGING_LEVEL == 'info'
# Delete the env var to test default
del os.environ['BROWSER_USE_LOGGING_LEVEL']
assert CONFIG.BROWSER_USE_LOGGING_LEVEL == 'info' # default value
finally:
# Restore original value
if original_value:
os.environ['BROWSER_USE_LOGGING_LEVEL'] = original_value
else:
os.environ.pop('BROWSER_USE_LOGGING_LEVEL', None)
def test_boolean_env_vars(self):
"""Test boolean environment variables are parsed correctly."""
original_value = os.environ.get('ANONYMIZED_TELEMETRY', '')
try:
# Test true values
for true_val in ['true', 'True', 'TRUE', 'yes', 'Yes', '1']:
os.environ['ANONYMIZED_TELEMETRY'] = true_val
assert CONFIG.ANONYMIZED_TELEMETRY is True, f'Failed for value: {true_val}'
# Test false values
for false_val in ['false', 'False', 'FALSE', 'no', 'No', '0']:
os.environ['ANONYMIZED_TELEMETRY'] = false_val
assert CONFIG.ANONYMIZED_TELEMETRY is False, f'Failed for value: {false_val}'
finally:
if original_value:
os.environ['ANONYMIZED_TELEMETRY'] = original_value
else:
os.environ.pop('ANONYMIZED_TELEMETRY', None)
def test_api_keys_lazy_loading(self):
"""Test API keys are loaded lazily."""
original_value = os.environ.get('OPENAI_API_KEY', '')
try:
# Test empty default
os.environ.pop('OPENAI_API_KEY', None)
assert CONFIG.OPENAI_API_KEY == ''
# Set a value
os.environ['OPENAI_API_KEY'] = 'test-key-123'
assert CONFIG.OPENAI_API_KEY == 'test-key-123'
# Change the value
os.environ['OPENAI_API_KEY'] = 'new-key-456'
assert CONFIG.OPENAI_API_KEY == 'new-key-456'
finally:
if original_value:
os.environ['OPENAI_API_KEY'] = original_value
else:
os.environ.pop('OPENAI_API_KEY', None)
def test_path_configuration(self):
"""Test path configuration variables."""
original_value = os.environ.get('XDG_CACHE_HOME', '')
try:
# Test custom path
test_path = '/tmp/test-cache'
os.environ['XDG_CACHE_HOME'] = test_path
# Use Path().resolve() to handle symlinks (e.g., /tmp -> /private/tmp on macOS)
from pathlib import Path
assert CONFIG.XDG_CACHE_HOME == Path(test_path).resolve()
# Test default path expansion
os.environ.pop('XDG_CACHE_HOME', None)
assert '/.cache' in str(CONFIG.XDG_CACHE_HOME)
finally:
if original_value:
os.environ['XDG_CACHE_HOME'] = original_value
else:
os.environ.pop('XDG_CACHE_HOME', None)
def test_cloud_sync_inherits_telemetry(self):
"""Test BROWSER_USE_CLOUD_SYNC inherits from ANONYMIZED_TELEMETRY when not set."""
telemetry_original = os.environ.get('ANONYMIZED_TELEMETRY', '')
sync_original = os.environ.get('BROWSER_USE_CLOUD_SYNC', '')
try:
# When BROWSER_USE_CLOUD_SYNC is not set, it should inherit from ANONYMIZED_TELEMETRY
os.environ['ANONYMIZED_TELEMETRY'] = 'true'
os.environ.pop('BROWSER_USE_CLOUD_SYNC', None)
assert CONFIG.BROWSER_USE_CLOUD_SYNC is True
os.environ['ANONYMIZED_TELEMETRY'] = 'false'
os.environ.pop('BROWSER_USE_CLOUD_SYNC', None)
assert CONFIG.BROWSER_USE_CLOUD_SYNC is False
# When explicitly set, it should use its own value
os.environ['ANONYMIZED_TELEMETRY'] = 'false'
os.environ['BROWSER_USE_CLOUD_SYNC'] = 'true'
assert CONFIG.BROWSER_USE_CLOUD_SYNC is True
finally:
if telemetry_original:
os.environ['ANONYMIZED_TELEMETRY'] = telemetry_original
else:
os.environ.pop('ANONYMIZED_TELEMETRY', None)
if sync_original:
os.environ['BROWSER_USE_CLOUD_SYNC'] = sync_original
else:
os.environ.pop('BROWSER_USE_CLOUD_SYNC', None)
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,394 @@
import asyncio
import base64
import socketserver
import pytest
from pytest_httpserver import HTTPServer
from browser_use.browser import BrowserProfile, BrowserSession
# Fix for httpserver hanging on shutdown - prevent blocking on socket close
socketserver.ThreadingMixIn.block_on_close = False
socketserver.ThreadingMixIn.daemon_threads = True
class TestBrowserContext:
"""Tests for browser context functionality using real browser instances."""
@pytest.fixture(scope='session')
def http_server(self):
"""Create and provide a test HTTP server that serves static content."""
server = HTTPServer()
server.start()
# Add routes for test pages
server.expect_request('/').respond_with_data(
'<html><head><title>Test Home Page</title></head><body><h1>Test Home Page</h1><p>Welcome to the test site</p></body></html>',
content_type='text/html',
)
server.expect_request('/scroll_test').respond_with_data(
"""
<html>
<head>
<title>Scroll Test</title>
<style>
body { height: 3000px; }
.marker { position: absolute; }
#top { top: 0; }
#middle { top: 1000px; }
#bottom { top: 2000px; }
</style>
</head>
<body>
<div id="top" class="marker">Top of the page</div>
<div id="middle" class="marker">Middle of the page</div>
<div id="bottom" class="marker">Bottom of the page</div>
</body>
</html>
""",
content_type='text/html',
)
yield server
server.stop()
@pytest.fixture(scope='session')
def base_url(self, http_server):
"""Return the base URL for the test HTTP server."""
return f'http://{http_server.host}:{http_server.port}'
@pytest.fixture(scope='module')
async def browser_session(self):
"""Create and provide a BrowserSession instance with security disabled."""
browser_session = BrowserSession(
browser_profile=BrowserProfile(
headless=True,
user_data_dir=None,
keep_alive=True,
)
)
await browser_session.start()
yield browser_session
await browser_session.kill()
# Ensure event bus is properly stopped
await browser_session.event_bus.stop(clear=True, timeout=5)
@pytest.mark.skip(reason='TODO: fix')
def test_is_url_allowed(self):
"""
Test the _is_url_allowed method to verify that it correctly checks URLs against
the allowed domains configuration.
"""
# Scenario 1: allowed_domains is None, any URL should be allowed.
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
config1 = BrowserProfile(allowed_domains=None, headless=True, user_data_dir=None)
context1 = BrowserSession(browser_profile=config1)
event_bus1 = EventBus()
watchdog1 = SecurityWatchdog(browser_session=context1, event_bus=event_bus1)
assert watchdog1._is_url_allowed('http://anydomain.com') is True
assert watchdog1._is_url_allowed('https://anotherdomain.org/path') is True
# Scenario 2: allowed_domains is provided.
# Note: match_url_with_domain_pattern defaults to https:// scheme when none is specified
allowed = ['https://example.com', 'http://example.com', 'http://*.mysite.org', 'https://*.mysite.org']
config2 = BrowserProfile(allowed_domains=allowed, headless=True, user_data_dir=None)
context2 = BrowserSession(browser_profile=config2)
event_bus2 = EventBus()
watchdog2 = SecurityWatchdog(browser_session=context2, event_bus=event_bus2)
# URL exactly matching
assert watchdog2._is_url_allowed('http://example.com') is True
# URL with subdomain (should not be allowed)
assert watchdog2._is_url_allowed('http://sub.example.com/path') is False
# URL with subdomain for wildcard pattern (should be allowed)
assert watchdog2._is_url_allowed('http://sub.mysite.org') is True
# URL that matches second allowed domain
assert watchdog2._is_url_allowed('https://mysite.org/page') is True
# URL with port number, still allowed (port is stripped)
assert watchdog2._is_url_allowed('http://example.com:8080') is True
assert watchdog2._is_url_allowed('https://example.com:443') is True
# Scenario 3: Malformed URL or empty domain
# urlparse will return an empty netloc for some malformed URLs.
assert watchdog2._is_url_allowed('notaurl') is False
# Method was removed from BrowserSession
def test_enhanced_css_selector_for_element(self):
"""
Test removed: _enhanced_css_selector_for_element method no longer exists.
"""
pass # Method was removed from BrowserSession
@pytest.mark.asyncio
@pytest.mark.skip(reason='TODO: fix')
async def test_navigate_and_get_current_page(self, browser_session, base_url):
"""Test that navigate method changes the URL and get_current_page returns the proper page."""
# Navigate to the test page
from browser_use.browser.events import NavigateToUrlEvent
event = browser_session.event_bus.dispatch(NavigateToUrlEvent(url=f'{base_url}/'))
await event
# Get the current page
url = await browser_session.get_current_page_url()
# Verify the page URL matches what we navigated to
assert f'{base_url}/' in url
# Verify the page title
title = await browser_session.get_current_page_title()
assert title == 'Test Home Page'
@pytest.mark.asyncio
@pytest.mark.skip(reason='TODO: fix')
async def test_refresh_page(self, browser_session, base_url):
"""Test that refresh_page correctly reloads the current page."""
# Navigate to the test page
from browser_use.browser.events import NavigateToUrlEvent
event = browser_session.event_bus.dispatch(NavigateToUrlEvent(url=f'{base_url}/'))
await event
# Get the current page info before refresh
url_before = await browser_session.get_current_page_url()
title_before = await browser_session.get_current_page_title()
# Refresh the page
await browser_session.refresh()
# Get the current page info after refresh
url_after = await browser_session.get_current_page_url()
title_after = await browser_session.get_current_page_title()
# Verify it's still on the same URL
assert url_after == url_before
# Verify the page title is still correct
assert title_after == 'Test Home Page'
@pytest.mark.asyncio
@pytest.mark.skip(reason='TODO: fix')
async def test_execute_javascript(self, browser_session, base_url):
"""Test that execute_javascript correctly executes JavaScript in the current page."""
# Navigate to a test page
from browser_use.browser.events import NavigateToUrlEvent
event = browser_session.event_bus.dispatch(NavigateToUrlEvent(url=f'{base_url}/'))
await event
# Execute a simple JavaScript snippet that returns a value
result = await browser_session.execute_javascript('document.title')
# Verify the result
assert result == 'Test Home Page'
# Execute JavaScript that modifies the page
await browser_session.execute_javascript("document.body.style.backgroundColor = 'red'")
# Verify the change by reading back the value
bg_color = await browser_session.execute_javascript('document.body.style.backgroundColor')
assert bg_color == 'red'
@pytest.mark.asyncio
@pytest.mark.skip(reason='TODO: fix')
@pytest.mark.skip(reason='get_scroll_info API changed - depends on page object that no longer exists')
async def test_get_scroll_info(self, browser_session, base_url):
"""Test that get_scroll_info returns the correct scroll position information."""
# Navigate to the scroll test page
from browser_use.browser.events import NavigateToUrlEvent
event = browser_session.event_bus.dispatch(NavigateToUrlEvent(url=f'{base_url}/scroll_test'))
await event
page = await browser_session.get_current_page()
# Get initial scroll info
pixels_above_initial, pixels_below_initial = await browser_session.get_scroll_info(page)
# Verify initial scroll position
assert pixels_above_initial == 0, 'Initial scroll position should be at the top'
assert pixels_below_initial > 0, 'There should be content below the viewport'
# Scroll down the page
await browser_session.execute_javascript('window.scrollBy(0, 500)')
await asyncio.sleep(0.2) # Brief delay for scroll to complete
# Get new scroll info
pixels_above_after_scroll, pixels_below_after_scroll = await browser_session.get_scroll_info(page)
# Verify new scroll position
assert pixels_above_after_scroll > 0, 'Page should be scrolled down'
assert pixels_above_after_scroll >= 400, 'Page should be scrolled down at least 400px'
assert pixels_below_after_scroll < pixels_below_initial, 'Less content should be below viewport after scrolling'
@pytest.mark.asyncio
@pytest.mark.skip(reason='TODO: fix')
async def test_take_screenshot(self, browser_session, base_url):
"""Test that take_screenshot returns a valid base64 encoded image."""
# Navigate to the test page
from browser_use.browser.events import NavigateToUrlEvent
event = browser_session.event_bus.dispatch(NavigateToUrlEvent(url=f'{base_url}/'))
await event
# Take a screenshot
screenshot_base64 = await browser_session.take_screenshot()
# Verify the screenshot is a valid base64 string
assert isinstance(screenshot_base64, str)
assert len(screenshot_base64) > 0
# Verify it can be decoded as base64
try:
image_data = base64.b64decode(screenshot_base64)
# Verify the data starts with a valid image signature (PNG file header)
assert image_data[:8] == b'\x89PNG\r\n\x1a\n', 'Screenshot is not a valid PNG image'
except Exception as e:
pytest.fail(f'Failed to decode screenshot as base64: {e}')
@pytest.mark.asyncio
@pytest.mark.skip(reason='TODO: fix')
async def test_switch_tab_operations(self, browser_session, base_url):
"""Test tab creation, switching, and closing operations."""
# Navigate to home page in first tab
from browser_use.browser.events import NavigateToUrlEvent
event = browser_session.event_bus.dispatch(NavigateToUrlEvent(url=f'{base_url}/'))
await event
# Create a new tab
await browser_session.create_new_tab(f'{base_url}/scroll_test')
# Verify we have two tabs now
tabs_info = await browser_session.get_tabs()
assert len(tabs_info) == 2, 'Should have two tabs open'
# Verify current tab is the scroll test page
current_url = await browser_session.get_current_page_url()
assert f'{base_url}/scroll_test' in current_url
# Switch back to the first tab
await browser_session.switch_to_tab(0)
# Verify we're back on the home page
current_url = await browser_session.get_current_page_url()
assert f'{base_url}/' in current_url
# Close the second tab
await browser_session.close_tab(1)
# Verify we have the expected number of tabs
# The first tab remains plus any about:blank tabs created by AboutBlankWatchdog
tabs_info = await browser_session.get_tabs_info()
# Filter out about:blank tabs created by the watchdog
non_blank_tabs = [tab for tab in tabs_info if 'about:blank' not in tab.url]
assert len(non_blank_tabs) == 1, (
f'Should have one non-blank tab open after closing the second, but got {len(non_blank_tabs)}: {non_blank_tabs}'
)
assert base_url in non_blank_tabs[0].url, 'The remaining tab should be the home page'
# TODO: highlighting doesn't exist anymore
# @pytest.mark.asyncio
# async def test_remove_highlights(self, browser_session, base_url):
# """Test that remove_highlights successfully removes highlight elements."""
# # Navigate to a test page
# from browser_use.browser.events import NavigateToUrlEvent; event = browser_session.event_bus.dispatch(NavigateToUrlEvent(url=f'{base_url}/')
# # Add a highlight via JavaScript
# await browser_session.execute_javascript("""
# const container = document.createElement('div');
# container.id = 'playwright-highlight-container';
# document.body.appendChild(container);
# const highlight = document.createElement('div');
# highlight.id = 'playwright-highlight-1';
# container.appendChild(highlight);
# const element = document.querySelector('h1');
# element.setAttribute('browser-user-highlight-id', 'playwright-highlight-1');
# """)
# # Verify the highlight container exists
# container_exists = await browser_session.execute_javascript(
# "document.getElementById('playwright-highlight-container') !== null"
# )
# assert container_exists, 'Highlight container should exist before removal'
# # Call remove_highlights
# await browser_session.remove_highlights()
# # Verify the highlight container was removed
# container_exists_after = await browser_session.execute_javascript(
# "document.getElementById('playwright-highlight-container') !== null"
# )
# assert not container_exists_after, 'Highlight container should be removed'
# # Verify the highlight attribute was removed from the element
# attribute_exists = await browser_session.execute_javascript(
# "document.querySelector('h1').hasAttribute('browser-user-highlight-id')"
# )
# assert not attribute_exists, 'browser-user-highlight-id attribute should be removed'
@pytest.mark.asyncio
@pytest.mark.skip(reason='TODO: fix')
async def test_custom_action_with_no_arguments(self, browser_session, base_url):
"""Test that custom actions with no arguments are handled correctly"""
from browser_use.agent.views import ActionResult
from browser_use.tools.registry.service import Registry
# Create a registry
registry = Registry()
# Register a custom action with no arguments
@registry.action('Some custom action with no args')
def simple_action():
return ActionResult(extracted_content='return some result')
# Navigate to a test page
from browser_use.browser.events import NavigateToUrlEvent
event = browser_session.event_bus.dispatch(NavigateToUrlEvent(url=f'{base_url}/'))
await event
# Execute the action
result = await registry.execute_action('simple_action', {})
# Verify the result
assert isinstance(result, ActionResult)
assert result.extracted_content == 'return some result'
# Test that the action model is created correctly
action_model = registry.create_action_model()
# The action should be in the model fields
assert 'simple_action' in action_model.model_fields
# Create an instance with the simple_action
action_instance = action_model(simple_action={}) # type: ignore[call-arg]
# Test that model_dump works correctly
dumped = action_instance.model_dump(exclude_unset=True)
assert 'simple_action' in dumped
assert dumped['simple_action'] == {}
# Test async version as well
@registry.action('Async custom action with no args')
async def async_simple_action():
return ActionResult(extracted_content='async result')
result = await registry.execute_action('async_simple_action', {})
assert result.extracted_content == 'async result'
# Test with special parameters but no regular arguments
@registry.action('Action with only special params')
async def special_params_only(browser_session):
current_url = await browser_session.get_current_page_url()
return ActionResult(extracted_content=f'Page URL: {current_url}')
result = await registry.execute_action('special_params_only', {}, browser_session=browser_session)
assert 'Page URL:' in result.extracted_content
assert base_url in result.extracted_content
@@ -0,0 +1,544 @@
"""
Comprehensive tests for the action registry system - Core functionality.
Tests cover:
1. Existing parameter patterns (individual params, pydantic models)
2. Special parameter injection (browser_session, page_extraction_llm, etc.)
3. Action-to-action calling scenarios
4. Mixed parameter patterns
5. Registry execution edge cases
"""
import asyncio
import logging
import pytest
from pydantic import Field
from pytest_httpserver import HTTPServer
from pytest_httpserver.httpserver import HandlerType
from browser_use.agent.views import ActionResult
from browser_use.browser import BrowserSession
from browser_use.browser.profile import BrowserProfile
from browser_use.llm.messages import UserMessage
from browser_use.tools.registry.service import Registry
from browser_use.tools.registry.views import ActionModel as BaseActionModel
from browser_use.tools.views import (
ClickElementAction,
InputTextAction,
NoParamsAction,
SearchAction,
)
from tests.ci.conftest import create_mock_llm
# Configure logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
class TestContext:
"""Simple context for testing"""
pass
# Test parameter models
class SimpleParams(BaseActionModel):
"""Simple parameter model"""
value: str = Field(description='Test value')
class ComplexParams(BaseActionModel):
"""Complex parameter model with multiple fields"""
text: str = Field(description='Text input')
number: int = Field(description='Number input', default=42)
optional_flag: bool = Field(description='Optional boolean', default=False)
# Test fixtures
@pytest.fixture(scope='session')
def http_server():
"""Create and provide a test HTTP server that serves static content."""
server = HTTPServer()
server.start()
# Add a simple test page that can handle multiple requests
server.expect_request('/test', handler_type=HandlerType.PERMANENT).respond_with_data(
'<html><head><title>Test Page</title></head><body><h1>Test Page</h1><p>Hello from test page</p></body></html>',
content_type='text/html',
)
yield server
server.stop()
@pytest.fixture(scope='session')
def base_url(http_server):
"""Return the base URL for the test HTTP server."""
return f'http://{http_server.host}:{http_server.port}'
@pytest.fixture(scope='module')
def mock_llm():
"""Create a mock LLM"""
return create_mock_llm()
@pytest.fixture(scope='function')
def registry():
"""Create a fresh registry for each test"""
return Registry[TestContext]()
@pytest.fixture(scope='function')
async def browser_session(base_url):
"""Create a real BrowserSession for testing"""
browser_session = BrowserSession(
browser_profile=BrowserProfile(
headless=True,
user_data_dir=None,
keep_alive=True,
)
)
await browser_session.start()
from browser_use.browser.events import NavigateToUrlEvent
browser_session.event_bus.dispatch(NavigateToUrlEvent(url=f'{base_url}/test'))
await asyncio.sleep(0.5) # Wait for navigation
yield browser_session
await browser_session.kill()
class TestActionRegistryParameterPatterns:
"""Test different parameter patterns that should all continue to work"""
async def test_individual_parameters_no_browser(self, registry):
"""Test action with individual parameters, no special injection"""
@registry.action('Simple action with individual params')
async def simple_action(text: str, number: int = 10):
return ActionResult(extracted_content=f'Text: {text}, Number: {number}')
# Test execution
result = await registry.execute_action('simple_action', {'text': 'hello', 'number': 42})
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'Text: hello, Number: 42' in result.extracted_content
async def test_individual_parameters_with_browser(self, registry, browser_session, base_url):
"""Test action with individual parameters plus browser_session injection"""
@registry.action('Action with individual params and browser')
async def action_with_browser(text: str, browser_session: BrowserSession):
url = await browser_session.get_current_page_url()
return ActionResult(extracted_content=f'Text: {text}, URL: {url}')
# Navigate to test page first
from browser_use.browser.events import NavigateToUrlEvent
event = browser_session.event_bus.dispatch(NavigateToUrlEvent(url=f'{base_url}/test', new_tab=True))
await event
# Test execution
result = await registry.execute_action('action_with_browser', {'text': 'hello'}, browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'Text: hello, URL:' in result.extracted_content
assert base_url in result.extracted_content
async def test_pydantic_model_parameters(self, registry, browser_session, base_url):
"""Test action that takes a pydantic model as first parameter"""
@registry.action('Action with pydantic model', param_model=ComplexParams)
async def pydantic_action(params: ComplexParams, browser_session: BrowserSession):
url = await browser_session.get_current_page_url()
return ActionResult(
extracted_content=f'Text: {params.text}, Number: {params.number}, Flag: {params.optional_flag}, URL: {url}'
)
# Navigate to test page first
from browser_use.browser.events import NavigateToUrlEvent
event = browser_session.event_bus.dispatch(NavigateToUrlEvent(url=f'{base_url}/test', new_tab=True))
await event
# Test execution
result = await registry.execute_action(
'pydantic_action', {'text': 'test', 'number': 100, 'optional_flag': True}, browser_session=browser_session
)
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'Text: test, Number: 100, Flag: True' in result.extracted_content
assert base_url in result.extracted_content
async def test_mixed_special_parameters(self, registry, browser_session, base_url, mock_llm):
"""Test action with multiple special injected parameters"""
from browser_use.llm.base import BaseChatModel
@registry.action('Action with multiple special params')
async def multi_special_action(
text: str,
browser_session: BrowserSession,
page_extraction_llm: BaseChatModel,
available_file_paths: list,
):
llm_response = await page_extraction_llm.ainvoke([UserMessage(content='test')])
files = available_file_paths or []
url = await browser_session.get_current_page_url()
return ActionResult(
extracted_content=f'Text: {text}, URL: {url}, LLM: {llm_response.completion}, Files: {len(files)}'
)
# Navigate to test page first
from browser_use.browser.events import NavigateToUrlEvent
event = browser_session.event_bus.dispatch(NavigateToUrlEvent(url=f'{base_url}/test', new_tab=True))
await event
# Test execution
result = await registry.execute_action(
'multi_special_action',
{'text': 'hello'},
browser_session=browser_session,
page_extraction_llm=mock_llm,
available_file_paths=['file1.txt', 'file2.txt'],
)
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'Text: hello' in result.extracted_content
assert base_url in result.extracted_content
# The mock LLM returns a JSON response
assert '"Task completed successfully"' in result.extracted_content
assert 'Files: 2' in result.extracted_content
async def test_no_params_action(self, registry, browser_session):
"""Test action with NoParamsAction model"""
@registry.action('No params action', param_model=NoParamsAction)
async def no_params_action(params: NoParamsAction, browser_session: BrowserSession):
url = await browser_session.get_current_page_url()
return ActionResult(extracted_content=f'No params action executed on {url}')
# Test execution with any parameters (should be ignored)
result = await registry.execute_action(
'no_params_action', {'random': 'data', 'should': 'be', 'ignored': True}, browser_session=browser_session
)
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'No params action executed on' in result.extracted_content
assert '/test' in result.extracted_content
class TestActionToActionCalling:
"""Test scenarios where actions call other actions"""
async def test_action_calling_action_with_kwargs(self, registry, browser_session):
"""Test action calling another action using kwargs (current problematic pattern)"""
# Helper function that actions can call
async def helper_function(browser_session: BrowserSession, data: str):
url = await browser_session.get_current_page_url()
return f'Helper processed: {data} on {url}'
@registry.action('First action')
async def first_action(text: str, browser_session: BrowserSession):
# This should work without parameter conflicts
result = await helper_function(browser_session=browser_session, data=text)
return ActionResult(extracted_content=f'First: {result}')
@registry.action('Calling action')
async def calling_action(message: str, browser_session: BrowserSession):
# Call the first action through the registry (simulates action-to-action calling)
intermediate_result = await registry.execute_action(
'first_action', {'text': message}, browser_session=browser_session
)
return ActionResult(extracted_content=f'Called result: {intermediate_result.extracted_content}')
# Test the calling chain
result = await registry.execute_action('calling_action', {'message': 'test'}, browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'Called result: First: Helper processed: test on' in result.extracted_content
assert '/test' in result.extracted_content
async def test_google_sheets_style_calling_pattern(self, registry, browser_session):
"""Test the specific pattern from Google Sheets actions that causes the error"""
# Simulate the _select_cell_or_range helper function
async def _select_cell_or_range(browser_session: BrowserSession, cell_or_range: str):
url = await browser_session.get_current_page_url()
return ActionResult(extracted_content=f'Selected cell {cell_or_range} on {url}')
@registry.action('Select cell or range')
async def select_cell_or_range(cell_or_range: str, browser_session: BrowserSession):
# This pattern now works with kwargs
return await _select_cell_or_range(browser_session=browser_session, cell_or_range=cell_or_range)
@registry.action('Select cell or range (fixed)')
async def select_cell_or_range_fixed(cell_or_range: str, browser_session: BrowserSession):
# This pattern also works
return await _select_cell_or_range(browser_session, cell_or_range)
@registry.action('Update range contents')
async def update_range_contents(range_name: str, new_contents: str, browser_session: BrowserSession):
# This action calls select_cell_or_range, simulating the real Google Sheets pattern
# Get the action's param model to call it properly
action = registry.registry.actions['select_cell_or_range_fixed']
params = action.param_model(cell_or_range=range_name)
await select_cell_or_range_fixed(cell_or_range=range_name, browser_session=browser_session)
return ActionResult(extracted_content=f'Updated range {range_name} with {new_contents}')
# Test the fixed version (should work)
result_fixed = await registry.execute_action(
'select_cell_or_range_fixed', {'cell_or_range': 'A1:F100'}, browser_session=browser_session
)
assert result_fixed.extracted_content is not None
assert 'Selected cell A1:F100 on' in result_fixed.extracted_content
assert '/test' in result_fixed.extracted_content
# Test the chained calling pattern
result_chain = await registry.execute_action(
'update_range_contents', {'range_name': 'B2:D4', 'new_contents': 'test data'}, browser_session=browser_session
)
assert result_chain.extracted_content is not None
assert 'Updated range B2:D4 with test data' in result_chain.extracted_content
# Test the problematic version (should work with enhanced registry)
result_problematic = await registry.execute_action(
'select_cell_or_range', {'cell_or_range': 'A1:F100'}, browser_session=browser_session
)
# With the enhanced registry, this should succeed
assert result_problematic.extracted_content is not None
assert 'Selected cell A1:F100 on' in result_problematic.extracted_content
assert '/test' in result_problematic.extracted_content
async def test_complex_action_chain(self, registry, browser_session):
"""Test a complex chain of actions calling other actions"""
@registry.action('Base action')
async def base_action(value: str, browser_session: BrowserSession):
url = await browser_session.get_current_page_url()
return ActionResult(extracted_content=f'Base: {value} on {url}')
@registry.action('Middle action')
async def middle_action(input_val: str, browser_session: BrowserSession):
# Call base action
base_result = await registry.execute_action(
'base_action', {'value': f'processed-{input_val}'}, browser_session=browser_session
)
return ActionResult(extracted_content=f'Middle: {base_result.extracted_content}')
@registry.action('Top action')
async def top_action(original: str, browser_session: BrowserSession):
# Call middle action
middle_result = await registry.execute_action(
'middle_action', {'input_val': f'enhanced-{original}'}, browser_session=browser_session
)
return ActionResult(extracted_content=f'Top: {middle_result.extracted_content}')
# Test the full chain
result = await registry.execute_action('top_action', {'original': 'test'}, browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'Top: Middle: Base: processed-enhanced-test on' in result.extracted_content
assert '/test' in result.extracted_content
class TestRegistryEdgeCases:
"""Test edge cases and error conditions"""
async def test_decorated_action_rejects_positional_args(self, registry, browser_session):
"""Test that decorated actions reject positional arguments"""
@registry.action('Action that should reject positional args')
async def test_action(cell_or_range: str, browser_session: BrowserSession):
url = await browser_session.get_current_page_url()
return ActionResult(extracted_content=f'Selected cell {cell_or_range} on {url}')
# Test that calling with positional arguments raises TypeError
with pytest.raises(
TypeError, match='test_action\\(\\) does not accept positional arguments, only keyword arguments are allowed'
):
await test_action('A1:B2', browser_session)
# Test that calling with keyword arguments works
result = await test_action(browser_session=browser_session, cell_or_range='A1:B2')
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'Selected cell A1:B2 on' in result.extracted_content
async def test_missing_required_browser_session(self, registry):
"""Test that actions requiring browser_session fail appropriately when not provided"""
@registry.action('Requires browser')
async def requires_browser(text: str, browser_session: BrowserSession):
url = await browser_session.get_current_page_url()
return ActionResult(extracted_content=f'Text: {text}, URL: {url}')
# Should raise RuntimeError when browser_session is required but not provided
with pytest.raises(RuntimeError, match='requires browser_session but none provided'):
await registry.execute_action(
'requires_browser',
{'text': 'test'},
# No browser_session provided
)
async def test_missing_required_llm(self, registry, browser_session):
"""Test that actions requiring page_extraction_llm fail appropriately when not provided"""
from browser_use.llm.base import BaseChatModel
@registry.action('Requires LLM')
async def requires_llm(text: str, browser_session: BrowserSession, page_extraction_llm: BaseChatModel):
url = await browser_session.get_current_page_url()
llm_response = await page_extraction_llm.ainvoke([UserMessage(content='test')])
return ActionResult(extracted_content=f'Text: {text}, LLM: {llm_response.completion}')
# Should raise RuntimeError when page_extraction_llm is required but not provided
with pytest.raises(RuntimeError, match='requires page_extraction_llm but none provided'):
await registry.execute_action(
'requires_llm',
{'text': 'test'},
browser_session=browser_session,
# No page_extraction_llm provided
)
async def test_invalid_parameters(self, registry, browser_session):
"""Test handling of invalid parameters"""
@registry.action('Typed action')
async def typed_action(number: int, browser_session: BrowserSession):
return ActionResult(extracted_content=f'Number: {number}')
# Should raise RuntimeError when parameter validation fails
with pytest.raises(RuntimeError, match='Invalid parameters'):
await registry.execute_action(
'typed_action',
{'number': 'not a number'}, # Invalid type
browser_session=browser_session,
)
async def test_nonexistent_action(self, registry, browser_session):
"""Test calling a non-existent action"""
with pytest.raises(ValueError, match='Action nonexistent_action not found'):
await registry.execute_action('nonexistent_action', {'param': 'value'}, browser_session=browser_session)
async def test_sync_action_wrapper(self, registry, browser_session):
"""Test that sync functions are properly wrapped to be async"""
@registry.action('Sync action')
def sync_action(text: str, browser_session: BrowserSession):
# This is a sync function that should be wrapped
return ActionResult(extracted_content=f'Sync: {text}')
# Should work even though the original function is sync
result = await registry.execute_action('sync_action', {'text': 'test'}, browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'Sync: test' in result.extracted_content
async def test_excluded_actions(self, browser_session):
"""Test that excluded actions are not registered"""
registry_with_exclusions = Registry[TestContext](exclude_actions=['excluded_action'])
@registry_with_exclusions.action('Excluded action')
async def excluded_action(text: str):
return ActionResult(extracted_content=f'Should not execute: {text}')
@registry_with_exclusions.action('Included action')
async def included_action(text: str):
return ActionResult(extracted_content=f'Should execute: {text}')
# Excluded action should not be in registry
assert 'excluded_action' not in registry_with_exclusions.registry.actions
assert 'included_action' in registry_with_exclusions.registry.actions
# Should raise error when trying to execute excluded action
with pytest.raises(ValueError, match='Action excluded_action not found'):
await registry_with_exclusions.execute_action('excluded_action', {'text': 'test'})
# Included action should work
result = await registry_with_exclusions.execute_action('included_action', {'text': 'test'})
assert result.extracted_content is not None
assert 'Should execute: test' in result.extracted_content
class TestExistingToolsActions:
"""Test that existing tools actions continue to work"""
async def test_existing_action_models(self, registry, browser_session):
"""Test that existing action parameter models work correctly"""
@registry.action('Test search', param_model=SearchAction)
async def test_search(params: SearchAction, browser_session: BrowserSession):
return ActionResult(extracted_content=f'Searched for: {params.query}')
@registry.action('Test click', param_model=ClickElementAction)
async def test_click(params: ClickElementAction, browser_session: BrowserSession):
return ActionResult(extracted_content=f'Clicked element: {params.index}')
@registry.action('Test input', param_model=InputTextAction)
async def test_input(params: InputTextAction, browser_session: BrowserSession):
return ActionResult(extracted_content=f'Input text: {params.text} at index: {params.index}')
# Test SearchGoogleAction
result1 = await registry.execute_action('test_search', {'query': 'python testing'}, browser_session=browser_session)
assert result1.extracted_content is not None
assert 'Searched for: python testing' in result1.extracted_content
# Test ClickElementAction
result2 = await registry.execute_action('test_click', {'index': 42}, browser_session=browser_session)
assert result2.extracted_content is not None
assert 'Clicked element: 42' in result2.extracted_content
# Test InputTextAction
result3 = await registry.execute_action('test_input', {'index': 5, 'text': 'test input'}, browser_session=browser_session)
assert result3.extracted_content is not None
assert 'Input text: test input at index: 5' in result3.extracted_content
async def test_pydantic_vs_individual_params_consistency(self, registry, browser_session):
"""Test that pydantic and individual parameter patterns produce consistent results"""
# Action using individual parameters
@registry.action('Individual params')
async def individual_params_action(text: str, number: int, browser_session: BrowserSession):
return ActionResult(extracted_content=f'Individual: {text}-{number}')
# Action using pydantic model
class TestParams(BaseActionModel):
text: str
number: int
@registry.action('Pydantic params', param_model=TestParams)
async def pydantic_params_action(params: TestParams, browser_session: BrowserSession):
return ActionResult(extracted_content=f'Pydantic: {params.text}-{params.number}')
# Both should produce similar results
test_data = {'text': 'hello', 'number': 42}
result1 = await registry.execute_action('individual_params_action', test_data, browser_session=browser_session)
result2 = await registry.execute_action('pydantic_params_action', test_data, browser_session=browser_session)
# Both should extract the same content (just different prefixes)
assert result1.extracted_content is not None
assert 'hello-42' in result1.extracted_content
assert result2.extracted_content is not None
assert 'hello-42' in result2.extracted_content
assert 'Individual:' in result1.extracted_content
assert 'Pydantic:' in result2.extracted_content
@@ -0,0 +1,547 @@
"""
Comprehensive tests for the action registry system - Validation and patterns.
Tests cover:
1. Type 1 and Type 2 patterns
2. Validation rules
3. Decorated function behavior
4. Parameter model generation
5. Parameter ordering
"""
import asyncio
import logging
import pytest
from pydantic import Field
from browser_use.agent.views import ActionResult
from browser_use.browser import BrowserSession
from browser_use.tools.registry.service import Registry
from browser_use.tools.registry.views import ActionModel as BaseActionModel
from tests.ci.conftest import create_mock_llm
# Configure logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
class TestType1Pattern:
"""Test Type 1 Pattern: Pydantic model first (from normalization tests)"""
def test_type1_with_param_model(self):
"""Type 1: action(params: Model, special_args...) should work"""
registry = Registry()
class ClickAction(BaseActionModel):
index: int
delay: float = 0.0
@registry.action('Click element', param_model=ClickAction)
async def click_element(params: ClickAction, browser_session: BrowserSession):
return ActionResult(extracted_content=f'Clicked {params.index}')
# Verify registration
assert 'click_element' in registry.registry.actions
action = registry.registry.actions['click_element']
assert action.param_model == ClickAction
# Verify decorated function signature (should be kwargs-only)
import inspect
sig = inspect.signature(click_element)
params = list(sig.parameters.values())
# Should have no positional-only or positional-or-keyword params
for param in params:
assert param.kind in (inspect.Parameter.KEYWORD_ONLY, inspect.Parameter.VAR_KEYWORD)
def test_type1_with_multiple_special_params(self):
"""Type 1 with multiple special params should work"""
registry = Registry()
class ExtractAction(BaseActionModel):
goal: str
include_links: bool = False
from browser_use.llm.base import BaseChatModel
@registry.action('Extract content', param_model=ExtractAction)
async def extract_content(params: ExtractAction, browser_session: BrowserSession, page_extraction_llm: BaseChatModel):
return ActionResult(extracted_content=params.goal)
assert 'extract_content' in registry.registry.actions
class TestType2Pattern:
"""Test Type 2 Pattern: loose parameters (from normalization tests)"""
def test_type2_simple_action(self):
"""Type 2: action(arg1, arg2, special_args...) should work"""
registry = Registry()
@registry.action('Fill field')
async def fill_field(index: int, text: str, browser_session: BrowserSession):
return ActionResult(extracted_content=f'Filled {index} with {text}')
# Verify registration
assert 'fill_field' in registry.registry.actions
action = registry.registry.actions['fill_field']
# Should auto-generate param model
assert action.param_model is not None
assert 'index' in action.param_model.model_fields
assert 'text' in action.param_model.model_fields
def test_type2_with_defaults(self):
"""Type 2 with default values should preserve defaults"""
registry = Registry()
@registry.action('Scroll page')
async def scroll_page(direction: str = 'down', amount: int = 100, browser_session: BrowserSession = None): # type: ignore
return ActionResult(extracted_content=f'Scrolled {direction} by {amount}')
action = registry.registry.actions['scroll_page']
# Check that defaults are preserved in generated model
schema = action.param_model.model_json_schema()
assert schema['properties']['direction']['default'] == 'down'
assert schema['properties']['amount']['default'] == 100
def test_type2_no_action_params(self):
"""Type 2 with only special params should work"""
registry = Registry()
@registry.action('Save PDF')
async def save_pdf(browser_session: BrowserSession):
return ActionResult(extracted_content='Saved PDF')
action = registry.registry.actions['save_pdf']
# Should have empty or minimal param model
fields = action.param_model.model_fields
assert len(fields) == 0 or all(f in ['title'] for f in fields)
def test_no_special_params_action(self):
"""Test action with no special params (like wait action in Tools)"""
registry = Registry()
@registry.action('Wait for x seconds default 3')
async def wait(seconds: int = 3):
await asyncio.sleep(seconds)
return ActionResult(extracted_content=f'Waited {seconds} seconds')
# Should register successfully
assert 'wait' in registry.registry.actions
action = registry.registry.actions['wait']
# Should have seconds in param model
assert 'seconds' in action.param_model.model_fields
# Should preserve default value
schema = action.param_model.model_json_schema()
assert schema['properties']['seconds']['default'] == 3
class TestValidationRules:
"""Test validation rules for action registration (from normalization tests)"""
def test_error_on_kwargs_in_original_function(self):
"""Should error if original function has kwargs"""
registry = Registry()
with pytest.raises(ValueError, match='kwargs.*not allowed'):
@registry.action('Bad action')
async def bad_action(index: int, browser_session: BrowserSession, **kwargs):
pass
def test_error_on_special_param_name_with_wrong_type(self):
"""Should error if special param name used with wrong type"""
registry = Registry()
# Using 'browser_session' with wrong type should error
with pytest.raises(ValueError, match='conflicts with special argument.*browser_session: BrowserSession'):
@registry.action('Bad session')
async def bad_session(browser_session: str):
pass
def test_special_params_must_match_type(self):
"""Special params with correct types should work"""
registry = Registry()
@registry.action('Good action')
async def good_action(
index: int,
browser_session: BrowserSession, # Correct type
):
return ActionResult()
assert 'good_action' in registry.registry.actions
class TestDecoratedFunctionBehavior:
"""Test behavior of decorated action functions (from normalization tests)"""
async def test_decorated_function_only_accepts_kwargs(self):
"""Decorated functions should only accept kwargs, no positional args"""
registry = Registry()
class MockBrowserSession:
async def get_current_page(self):
return None
@registry.action('Click')
async def click(index: int, browser_session: BrowserSession):
return ActionResult()
# Should raise error when called with positional args
with pytest.raises(TypeError, match='positional arguments'):
await click(5, MockBrowserSession())
async def test_decorated_function_accepts_params_model(self):
"""Decorated function should accept params as model"""
registry = Registry()
class MockBrowserSession:
async def get_current_page(self):
return None
@registry.action('Input text')
async def input_text(index: int, text: str, browser_session: BrowserSession):
return ActionResult(extracted_content=f'{index}:{text}')
# Get the generated param model class
action = registry.registry.actions['input_text']
ParamsModel = action.param_model
# Should work with params model
result = await input_text(params=ParamsModel(index=5, text='hello'), browser_session=MockBrowserSession())
assert result.extracted_content == '5:hello'
async def test_decorated_function_ignores_extra_kwargs(self):
"""Decorated function should ignore extra kwargs for easy unpacking"""
registry = Registry()
@registry.action('Simple action')
async def simple_action(value: int):
return ActionResult(extracted_content=str(value))
# Should work even with extra kwargs
special_context = {
'browser_session': None,
'page_extraction_llm': create_mock_llm(),
'context': {'extra': 'data'},
'unknown_param': 'ignored',
}
action = registry.registry.actions['simple_action']
ParamsModel = action.param_model
result = await simple_action(params=ParamsModel(value=42), **special_context)
assert result.extracted_content == '42'
class TestParamsModelGeneration:
"""Test automatic parameter model generation (from normalization tests)"""
def test_generates_model_from_non_special_args(self):
"""Should generate param model from non-special positional args"""
registry = Registry()
@registry.action('Complex action')
async def complex_action(
query: str,
max_results: int,
include_images: bool = True,
browser_session: BrowserSession = None, # type: ignore
):
return ActionResult()
action = registry.registry.actions['complex_action']
model_fields = action.param_model.model_fields
# Should include only non-special params
assert 'query' in model_fields
assert 'max_results' in model_fields
assert 'include_images' in model_fields
# Should NOT include special params
assert 'browser_session' not in model_fields
def test_preserves_type_annotations(self):
"""Generated model should preserve type annotations"""
registry = Registry()
@registry.action('Typed action')
async def typed_action(
count: int,
rate: float,
enabled: bool,
name: str | None = None,
browser_session: BrowserSession = None, # type: ignore
):
return ActionResult()
action = registry.registry.actions['typed_action']
schema = action.param_model.model_json_schema()
# Check types are preserved
assert schema['properties']['count']['type'] == 'integer'
assert schema['properties']['rate']['type'] == 'number'
assert schema['properties']['enabled']['type'] == 'boolean'
# Optional should allow null
assert 'null' in schema['properties']['name']['anyOf'][1]['type']
class TestParameterOrdering:
"""Test mixed ordering of parameters (from normalization tests)"""
def test_mixed_param_ordering(self):
"""Should handle any ordering of action params and special params"""
registry = Registry()
from browser_use.llm.base import BaseChatModel
# Special params mixed throughout
@registry.action('Mixed params')
async def mixed_action(
first: str,
browser_session: BrowserSession,
second: int,
third: bool = True,
page_extraction_llm: BaseChatModel = None, # type: ignore
):
return ActionResult()
action = registry.registry.actions['mixed_action']
model_fields = action.param_model.model_fields
# Only action params in model
assert set(model_fields.keys()) == {'first', 'second', 'third'}
assert model_fields['third'].default is True
def test_extract_content_pattern_registration(self):
"""Test that the extract_content pattern with mixed params registers correctly"""
registry = Registry()
# This is the problematic pattern: positional arg, then special args, then kwargs with defaults
@registry.action('Extract content from page')
async def extract_content(
goal: str,
page_extraction_llm,
include_links: bool = False,
):
return ActionResult(extracted_content=f'Goal: {goal}, include_links: {include_links}')
# Verify registration
assert 'extract_content' in registry.registry.actions
action = registry.registry.actions['extract_content']
# Check that the param model only includes user-facing params
model_fields = action.param_model.model_fields
assert 'goal' in model_fields
assert 'include_links' in model_fields
assert model_fields['include_links'].default is False
# Special params should NOT be in the model
assert 'page' not in model_fields
assert 'page_extraction_llm' not in model_fields
# Verify the action was properly registered
assert action.name == 'extract_content'
assert action.description == 'Extract content from page'
class TestParamsModelArgsAndKwargs:
async def test_browser_session_double_kwarg(self):
"""Run the test to diagnose browser_session parameter issue
This test demonstrates the problem and our fix. The issue happens because:
1. In tools/service.py, we have:
```python
@registry.action('Google Sheets: Select a specific cell or range of cells')
async def select_cell_or_range(browser_session: BrowserSession, cell_or_range: str):
return await _select_cell_or_range(browser_session=browser_session, cell_or_range=cell_or_range)
```
2. When registry.execute_action calls this function, it adds browser_session to extra_args:
```python
# In registry/service.py
if 'browser_session' in parameter_names:
extra_args['browser_session'] = browser_session
```
3. Then later, when calling action.function:
```python
return await action.function(**params_dict, **extra_args)
```
4. This effectively means browser_session is passed twice:
- Once through extra_args['browser_session']
- And again through params_dict['browser_session'] (from the original function)
The fix is to pass browser_session positionally in select_cell_or_range:
```python
return await _select_cell_or_range(browser_session, cell_or_range)
```
This test confirms that this approach works.
"""
from browser_use.tools.registry.service import Registry
from browser_use.tools.registry.views import ActionModel
# Simple context for testing
class TestContext:
pass
class MockBrowserSession:
async def get_current_page(self):
return None
browser_session = MockBrowserSession()
# Create registry
registry = Registry[TestContext]()
# Model that doesn't include browser_session (renamed to avoid pytest collecting it)
class CellActionParams(ActionModel):
value: str = Field(description='Test value')
# Model that includes browser_session
class ModelWithBrowser(ActionModel):
value: str = Field(description='Test value')
browser_session: BrowserSession = None # type: ignore
# Create a custom param model for select_cell_or_range
class CellRangeParams(ActionModel):
cell_or_range: str = Field(description='Cell or range to select')
# Use the provided real browser session
# Test with the real issue: select_cell_or_range
# logger.info('\n\n=== Test: Simulating select_cell_or_range issue with correct model ===')
# Define the function without using our registry - this will be a helper function
async def _select_cell_or_range(browser_session, cell_or_range):
"""Helper function for select_cell_or_range"""
return f'Selected cell {cell_or_range}'
# This simulates the actual issue we're seeing in the real code
# The browser_session parameter is in both the function signature and passed as a named arg
@registry.action('Google Sheets: Select a cell or range', param_model=CellRangeParams)
async def select_cell_or_range(browser_session: BrowserSession, cell_or_range: str):
# logger.info(f'select_cell_or_range called with browser_session={browser_session}, cell_or_range={cell_or_range}')
# PROBLEMATIC LINE: browser_session is passed by name, matching the parameter name
# This is what causes the "got multiple values" error in the real code
return await _select_cell_or_range(browser_session=browser_session, cell_or_range=cell_or_range)
# Fix attempt: Register a version that uses positional args instead
@registry.action('Google Sheets: Select a cell or range (fixed)', param_model=CellRangeParams)
async def select_cell_or_range_fixed(browser_session: BrowserSession, cell_or_range: str):
# logger.info(f'select_cell_or_range_fixed called with browser_session={browser_session}, cell_or_range={cell_or_range}')
# FIXED LINE: browser_session is passed positionally, avoiding the parameter name conflict
return await _select_cell_or_range(browser_session, cell_or_range)
# Another attempt: explicitly call using **kwargs to simulate what the registry does
@registry.action('Google Sheets: Select with kwargs', param_model=CellRangeParams)
async def select_with_kwargs(browser_session: BrowserSession, cell_or_range: str):
# logger.info(f'select_with_kwargs called with browser_session={browser_session}, cell_or_range={cell_or_range}')
# Get params and extra_args, like in Registry.execute_action
params = {'cell_or_range': cell_or_range, 'browser_session': browser_session}
extra_args = {'browser_session': browser_session}
# Try to call _select_cell_or_range with both params and extra_args
# This will fail with "got multiple values for keyword argument 'browser_session'"
try:
# logger.info('Attempting to call with both params and extra_args (should fail):')
await _select_cell_or_range(**params, **extra_args)
except TypeError as e:
# logger.info(f'Expected error: {e}')
# Remove browser_session from params to avoid the conflict
params_fixed = dict(params)
del params_fixed['browser_session']
# logger.info(f'Fixed params: {params_fixed}')
# This should work
result = await _select_cell_or_range(**params_fixed, **extra_args)
# logger.info(f'Success after fix: {result}')
return result
# Test the original problematic version
# logger.info('\n--- Testing original problematic version ---')
try:
result1 = await registry.execute_action(
'select_cell_or_range',
{'cell_or_range': 'A1:F100'},
browser_session=browser_session, # type: ignore
)
# logger.info(f'Success! Result: {result1}')
except Exception as e:
logger.error(f'Error: {str(e)}')
# Test the fixed version (using positional args)
# logger.info('\n--- Testing fixed version (positional args) ---')
try:
result2 = await registry.execute_action(
'select_cell_or_range_fixed',
{'cell_or_range': 'A1:F100'},
browser_session=browser_session, # type: ignore
)
# logger.info(f'Success! Result: {result2}')
except Exception as e:
logger.error(f'Error: {str(e)}')
# Test with kwargs version that simulates what Registry.execute_action does
# logger.info('\n--- Testing kwargs simulation version ---')
try:
result3 = await registry.execute_action(
'select_with_kwargs',
{'cell_or_range': 'A1:F100'},
browser_session=browser_session, # type: ignore
)
# logger.info(f'Success! Result: {result3}')
except Exception as e:
logger.error(f'Error: {str(e)}')
# Manual test of our theory: browser_session is passed twice
# logger.info('\n--- Direct test of our theory ---')
try:
# Create the model instance
params = CellRangeParams(cell_or_range='A1:F100')
# First check if the extra_args approach works
# logger.info('Checking if extra_args approach works:')
extra_args = {'browser_session': browser_session}
# If we were to modify Registry.execute_action:
# 1. Check if the function parameter needs browser_session
parameter_names = ['browser_session', 'cell_or_range']
browser_keys = ['browser_session', 'browser', 'browser_context']
# Create params dict
param_dict = params.model_dump()
# logger.info(f'params dict before: {param_dict}')
# Apply our fix: remove browser_session from params dict
for key in browser_keys:
if key in param_dict and key in extra_args:
# logger.info(f'Removing {key} from params dict')
del param_dict[key]
# logger.info(f'params dict after: {param_dict}')
# logger.info(f'extra_args: {extra_args}')
# This would be the fixed code:
# return await action.function(**param_dict, **extra_args)
# Call directly to test
result3 = await select_cell_or_range(**param_dict, **extra_args)
# logger.info(f'Success with our fix! Result: {result3}')
except Exception as e:
logger.error(f'Error with our manual test: {str(e)}')
@@ -0,0 +1,161 @@
"""
Simplified tests for URL shortening functionality in Agent service.
Three focused tests:
1. Input message processing with URL shortening
2. Output processing with custom actions and URL restoration
3. End-to-end pipeline test
"""
import json
import pytest
from browser_use.agent.service import Agent
from browser_use.agent.views import AgentOutput
from browser_use.llm.messages import AssistantMessage, BaseMessage, UserMessage
# Super long URL to reuse across tests - much longer than the 25 character limit
# Includes both query params (?...) and fragment params (#...)
SUPER_LONG_URL = 'https://documentation.example-company.com/api/v3/enterprise/user-management/endpoints/administration/create-new-user-account-with-permissions/advanced-settings?format=detailed-json&version=3.2.1&timestamp=1699123456789&session_id=abc123def456ghi789&authentication_token=very_long_authentication_token_string_here&include_metadata=true&expand_relationships=user_groups,permissions,roles&sort_by=created_at&order=desc&page_size=100&include_deprecated_fields=false&api_key=super_long_api_key_that_exceeds_normal_limits#section=user_management&tab=advanced&view=detailed&scroll_to=permissions_table&highlight=admin_settings&filter=active_users&expand_all=true&debug_mode=enabled'
@pytest.fixture
def agent():
"""Create an agent instance for testing URL shortening functionality."""
from tests.ci.conftest import create_mock_llm
return Agent(task='Test URL shortening', llm=create_mock_llm(), url_shortening_limit=25)
class TestUrlShorteningInputProcessing:
"""Test URL shortening for input messages."""
def test_process_input_messages_with_url_shortening(self, agent: Agent):
"""Test that long URLs in input messages are shortened and mappings stored."""
original_content = f'Please visit {SUPER_LONG_URL} and extract information'
messages: list[BaseMessage] = [UserMessage(content=original_content)]
# Process messages (modifies messages in-place and returns URL mappings)
url_mappings = agent._process_messsages_and_replace_long_urls_shorter_ones(messages)
# Verify URL was shortened in the message (modified in-place)
processed_content = messages[0].content or ''
assert processed_content != original_content
assert 'https://documentation.example-company.com' in processed_content
assert len(processed_content) < len(original_content)
# Verify URL mapping was returned
assert len(url_mappings) == 1
shortened_url = next(iter(url_mappings.keys()))
assert url_mappings[shortened_url] == SUPER_LONG_URL
def test_process_user_and_assistant_messages_with_url_shortening(self, agent: Agent):
"""Test URL shortening in both UserMessage and AssistantMessage."""
user_content = f'I need to access {SUPER_LONG_URL} for the API documentation'
assistant_content = f'I will help you navigate to {SUPER_LONG_URL} to retrieve the documentation'
messages: list[BaseMessage] = [UserMessage(content=user_content), AssistantMessage(content=assistant_content)]
# Process messages (modifies messages in-place and returns URL mappings)
url_mappings = agent._process_messsages_and_replace_long_urls_shorter_ones(messages)
# Verify URL was shortened in both messages
user_processed_content = messages[0].content or ''
assistant_processed_content = messages[1].content or ''
assert user_processed_content != user_content
assert assistant_processed_content != assistant_content
assert 'https://documentation.example-company.com' in user_processed_content
assert 'https://documentation.example-company.com' in assistant_processed_content
assert len(user_processed_content) < len(user_content)
assert len(assistant_processed_content) < len(assistant_content)
# Verify URL mapping was returned (should be same shortened URL for both occurrences)
assert len(url_mappings) == 1
shortened_url = next(iter(url_mappings.keys()))
assert url_mappings[shortened_url] == SUPER_LONG_URL
class TestUrlShorteningOutputProcessing:
"""Test URL restoration for output processing with custom actions."""
def test_process_output_with_custom_actions_and_url_restoration(self, agent: Agent):
"""Test that shortened URLs in AgentOutput with custom actions are restored."""
# Set up URL mapping (simulating previous shortening)
shortened_url: str = agent._replace_urls_in_text(SUPER_LONG_URL)[0]
url_mappings = {shortened_url: SUPER_LONG_URL}
# Create AgentOutput with shortened URLs using JSON parsing
output_json = {
'thinking': f'I need to navigate to {shortened_url} for documentation',
'evaluation_previous_goal': 'Successfully processed the request',
'memory': f'Found useful info at {shortened_url}',
'next_goal': 'Complete the documentation review',
'action': [{'navigate': {'url': shortened_url, 'new_tab': False}}],
}
# Create properly typed AgentOutput with custom actions
tools = agent.tools
ActionModel = tools.registry.create_action_model()
AgentOutputWithActions = AgentOutput.type_with_custom_actions(ActionModel)
agent_output = AgentOutputWithActions.model_validate_json(json.dumps(output_json))
# Process the output to restore URLs (modifies agent_output in-place)
agent._recursive_process_all_strings_inside_pydantic_model(agent_output, url_mappings)
# Verify URLs were restored in all locations
assert SUPER_LONG_URL in (agent_output.thinking or '')
assert SUPER_LONG_URL in (agent_output.memory or '')
action_data = agent_output.action[0].model_dump()
assert action_data['navigate']['url'] == SUPER_LONG_URL
class TestUrlShorteningEndToEnd:
"""Test complete URL shortening pipeline end-to-end."""
def test_complete_url_shortening_pipeline(self, agent: Agent):
"""Test the complete pipeline: input shortening -> processing -> output restoration."""
# Step 1: Input processing with URL shortening
original_content = f'Navigate to {SUPER_LONG_URL} and extract the API documentation'
messages: list[BaseMessage] = [UserMessage(content=original_content)]
url_mappings = agent._process_messsages_and_replace_long_urls_shorter_ones(messages)
# Verify URL was shortened in input
assert len(url_mappings) == 1
shortened_url = next(iter(url_mappings.keys()))
assert url_mappings[shortened_url] == SUPER_LONG_URL
assert shortened_url in (messages[0].content or '')
# Step 2: Simulate agent output with shortened URL
output_json = {
'thinking': f'I will navigate to {shortened_url} to get the documentation',
'evaluation_previous_goal': 'Starting documentation extraction',
'memory': f'Target URL: {shortened_url}',
'next_goal': 'Extract API documentation',
'action': [{'navigate': {'url': shortened_url, 'new_tab': True}}],
}
# Create AgentOutput with custom actions
tools = agent.tools
ActionModel = tools.registry.create_action_model()
AgentOutputWithActions = AgentOutput.type_with_custom_actions(ActionModel)
agent_output = AgentOutputWithActions.model_validate_json(json.dumps(output_json))
# Step 3: Output processing with URL restoration (modifies agent_output in-place)
agent._recursive_process_all_strings_inside_pydantic_model(agent_output, url_mappings)
# Verify complete pipeline worked correctly
assert SUPER_LONG_URL in (agent_output.thinking or '')
assert SUPER_LONG_URL in (agent_output.memory or '')
action_data = agent_output.action[0].model_dump()
assert action_data['navigate']['url'] == SUPER_LONG_URL
assert action_data['navigate']['new_tab'] is True
# Verify original shortened content is no longer present
assert shortened_url not in (agent_output.thinking or '')
assert shortened_url not in (agent_output.memory or '')
@@ -0,0 +1,13 @@
from browser_use.utils import _is_newer_browser_use_version
def test_prerelease_is_newer_than_previous_stable():
assert _is_newer_browser_use_version('0.12.9', '0.13.0rc3') is False
def test_stable_release_is_newer_than_same_release_candidate():
assert _is_newer_browser_use_version('0.13.0', '0.13.0rc3') is True
def test_later_release_candidate_is_newer():
assert _is_newer_browser_use_version('0.13.0rc4', '0.13.0rc3') is True
@@ -0,0 +1,382 @@
"""Test autocomplete/combobox field detection, value readback, and input clearing.
Tests cover:
- Value mismatch detection when JS rewrites input value
- Combobox field detection (role=combobox + aria-autocomplete)
- Datalist field detection (input with list attribute)
- No false positives on plain inputs
- Sensitive data skips value verification
- Pre-filled input clearing (clear=True default)
- Pre-filled input appending (clear=False)
- Concatenation auto-retry when clear fails
- Autocomplete delay before next action
"""
import asyncio
import pytest
from pytest_httpserver import HTTPServer
from browser_use.agent.views import ActionResult
from browser_use.browser import BrowserSession
from browser_use.browser.profile import BrowserProfile
from browser_use.tools.service import Tools
@pytest.fixture(scope='session')
def http_server():
"""Create and provide a test HTTP server with autocomplete test pages."""
server = HTTPServer()
server.start()
# Page 1: Input with JS that rewrites value on change (simulates autocomplete replacing text)
server.expect_request('/autocomplete-rewrite').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head><title>Autocomplete Rewrite Test</title></head>
<body>
<input id="search" type="text" />
<script>
const input = document.getElementById('search');
input.addEventListener('change', function() {
// Simulate autocomplete rewriting the value
this.value = 'REWRITTEN_' + this.value;
});
</script>
</body>
</html>
""",
content_type='text/html',
)
# Page 2: Input with role=combobox + aria-autocomplete=list + aria-controls + listbox
server.expect_request('/combobox-field').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head><title>Combobox Field Test</title></head>
<body>
<div>
<input id="combo" type="text" role="combobox"
aria-autocomplete="list" aria-controls="suggestions-list"
aria-expanded="false" />
<ul id="suggestions-list" role="listbox" style="display:none;">
<li role="option">Option A</li>
<li role="option">Option B</li>
</ul>
</div>
</body>
</html>
""",
content_type='text/html',
)
# Page 3: Input with list attribute pointing to a datalist
server.expect_request('/datalist-field').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head><title>Datalist Field Test</title></head>
<body>
<input id="city" type="text" list="suggestions" />
<datalist id="suggestions">
<option value="New York">
<option value="Los Angeles">
<option value="Chicago">
</datalist>
</body>
</html>
""",
content_type='text/html',
)
# Page 4: Plain input with no autocomplete attributes
server.expect_request('/normal-input').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head><title>Normal Input Test</title></head>
<body>
<input id="plain" type="text" placeholder="Just a normal input" />
</body>
</html>
""",
content_type='text/html',
)
# Page 5: Pre-filled input to test clear=True behavior
server.expect_request('/prefilled-input').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head><title>Pre-filled Input Test</title></head>
<body>
<input id="prefilled" type="text" value="old value" />
</body>
</html>
""",
content_type='text/html',
)
# Page 6: Input where clear fails — input event listener restores old text
# Simulates a framework-controlled input where clearing triggers re-render with old state
server.expect_request('/sticky-input').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head><title>Sticky Input Test</title></head>
<body>
<input id="sticky" type="text" value="prefix_" />
<script>
var el = document.getElementById('sticky');
var clearAttempts = 0;
// Intercept value clears: restore old value on first two clears
// (simulates framework re-rendering with stale state)
el.addEventListener('input', function() {
if (el.value === '' && clearAttempts < 2) {
clearAttempts++;
el.value = 'prefix_';
}
});
</script>
</body>
</html>
""",
content_type='text/html',
)
yield server
server.stop()
@pytest.fixture(scope='session')
def base_url(http_server):
"""Return the base URL for the test HTTP server."""
return f'http://{http_server.host}:{http_server.port}'
@pytest.fixture(scope='module')
async def browser_session():
"""Create and provide a Browser instance for testing."""
browser_session = BrowserSession(
browser_profile=BrowserProfile(
headless=True,
user_data_dir=None,
keep_alive=True,
chromium_sandbox=False,
)
)
await browser_session.start()
yield browser_session
await browser_session.kill()
@pytest.fixture(scope='function')
def tools():
"""Create and provide a Tools instance."""
return Tools()
class TestAutocompleteInteraction:
"""Test autocomplete/combobox detection and value readback."""
async def test_value_mismatch_detected(self, tools: Tools, browser_session: BrowserSession, base_url: str):
"""Type into a field whose JS rewrites the value on change. Assert the ActionResult notes the mismatch."""
await tools.navigate(url=f'{base_url}/autocomplete-rewrite', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.3)
await browser_session.get_browser_state_summary()
input_index = await browser_session.get_index_by_id('search')
assert input_index is not None, 'Could not find search input'
result = await tools.input(index=input_index, text='hello', browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'differs from typed text' in result.extracted_content, (
f'Expected mismatch note in extracted_content, got: {result.extracted_content}'
)
async def test_combobox_field_detected(self, tools: Tools, browser_session: BrowserSession, base_url: str):
"""Type into a combobox field. Assert the ActionResult includes autocomplete guidance."""
await tools.navigate(url=f'{base_url}/combobox-field', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.3)
await browser_session.get_browser_state_summary()
combo_index = await browser_session.get_index_by_id('combo')
assert combo_index is not None, 'Could not find combobox input'
result = await tools.input(index=combo_index, text='test', browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'autocomplete field' in result.extracted_content, (
f'Expected autocomplete guidance in extracted_content, got: {result.extracted_content}'
)
async def test_datalist_field_detected(self, tools: Tools, browser_session: BrowserSession, base_url: str):
"""Type into a datalist-backed field. Assert the ActionResult includes autocomplete guidance."""
await tools.navigate(url=f'{base_url}/datalist-field', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.3)
await browser_session.get_browser_state_summary()
city_index = await browser_session.get_index_by_id('city')
assert city_index is not None, 'Could not find datalist input'
result = await tools.input(index=city_index, text='New', browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'autocomplete field' in result.extracted_content, (
f'Expected autocomplete guidance in extracted_content, got: {result.extracted_content}'
)
async def test_normal_input_no_false_positive(self, tools: Tools, browser_session: BrowserSession, base_url: str):
"""Type into a plain input. Assert the ActionResult does NOT contain autocomplete guidance."""
await tools.navigate(url=f'{base_url}/normal-input', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.3)
await browser_session.get_browser_state_summary()
plain_index = await browser_session.get_index_by_id('plain')
assert plain_index is not None, 'Could not find plain input'
result = await tools.input(index=plain_index, text='hello', browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'autocomplete field' not in result.extracted_content, (
f'Got false positive autocomplete guidance on plain input: {result.extracted_content}'
)
async def test_sensitive_data_skips_value_verification(self, tools: Tools, browser_session: BrowserSession, base_url: str):
"""Type sensitive data into the rewrite field. Assert no 'differs from typed text' note appears."""
await tools.navigate(url=f'{base_url}/autocomplete-rewrite', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.3)
await browser_session.get_browser_state_summary()
input_index = await browser_session.get_index_by_id('search')
assert input_index is not None, 'Could not find search input'
# Use tools.act() with sensitive_data to trigger the sensitive code path
result = await tools.input(
index=input_index,
text='secret123',
browser_session=browser_session,
sensitive_data={'password': 'secret123'},
)
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'differs from typed text' not in result.extracted_content, (
f'Sensitive data should not show value mismatch: {result.extracted_content}'
)
async def test_prefilled_input_cleared_by_default(self, tools: Tools, browser_session: BrowserSession, base_url: str):
"""Type into a pre-filled input with clear=True (default). Field should contain only the new text."""
await tools.navigate(url=f'{base_url}/prefilled-input', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.3)
await browser_session.get_browser_state_summary()
idx = await browser_session.get_index_by_id('prefilled')
assert idx is not None, 'Could not find prefilled input'
result = await tools.input(index=idx, text='new value', browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.error is None, f'Input action failed: {result.error}'
# Read back the actual DOM value via CDP
cdp_session = await browser_session.get_or_create_cdp_session()
readback = await cdp_session.cdp_client.send.Runtime.evaluate(
params={'expression': "document.getElementById('prefilled').value"},
session_id=cdp_session.session_id,
)
actual = readback.get('result', {}).get('value', '')
assert actual == 'new value', f'Expected "new value", got "{actual}" — clear=True did not remove old text'
async def test_prefilled_input_append_with_clear_false(self, tools: Tools, browser_session: BrowserSession, base_url: str):
"""Type into a pre-filled input with clear=False. Field should contain old + new text."""
await tools.navigate(url=f'{base_url}/prefilled-input', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.3)
await browser_session.get_browser_state_summary()
idx = await browser_session.get_index_by_id('prefilled')
assert idx is not None, 'Could not find prefilled input'
result = await tools.input(index=idx, text=' appended', clear=False, browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.error is None, f'Input action failed: {result.error}'
# Read back the actual DOM value via CDP
cdp_session = await browser_session.get_or_create_cdp_session()
readback = await cdp_session.cdp_client.send.Runtime.evaluate(
params={'expression': "document.getElementById('prefilled').value"},
session_id=cdp_session.session_id,
)
actual = readback.get('result', {}).get('value', '')
assert 'old value' in actual and 'appended' in actual, f'Expected old text + appended text, got "{actual}"'
async def test_concatenation_retry_on_sticky_field(self, tools: Tools, browser_session: BrowserSession, base_url: str):
"""Type into a field where clearing is resisted by JS. The retry should fix the value."""
await tools.navigate(url=f'{base_url}/sticky-input', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.3)
await browser_session.get_browser_state_summary()
idx = await browser_session.get_index_by_id('sticky')
assert idx is not None, 'Could not find sticky input'
result = await tools.input(index=idx, text='typed_text', browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.error is None, f'Input action failed: {result.error}'
# The retry mechanism uses a native setter to bypass the event listener.
# Read back the final DOM value.
cdp_session = await browser_session.get_or_create_cdp_session()
readback = await cdp_session.cdp_client.send.Runtime.evaluate(
params={'expression': "document.getElementById('sticky').value"},
session_id=cdp_session.session_id,
)
actual = readback.get('result', {}).get('value', '')
# The retry should have set the value to just "typed_text" via the native setter.
# Even if the event listener fires on the retry's dispatched events, the native setter
# bypasses instance-level interception. The value may or may not be perfect depending
# on how the JS listener interacts, but it should not be "prefix_typed_text" (raw concatenation).
assert actual != 'prefix_typed_text', f'Got raw concatenation "{actual}" — retry should have prevented this'
async def test_combobox_field_adds_delay(self, tools: Tools, browser_session: BrowserSession, base_url: str):
"""Typing into a combobox (role=combobox) field should take >= 400ms due to the mechanical delay."""
import time
await tools.navigate(url=f'{base_url}/combobox-field', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.3)
await browser_session.get_browser_state_summary()
combo_idx = await browser_session.get_index_by_id('combo')
assert combo_idx is not None
t0 = time.monotonic()
await tools.input(index=combo_idx, text='hi', browser_session=browser_session)
duration = time.monotonic() - t0
# The 400ms sleep is a hard floor — total duration must exceed it
assert duration >= 0.4, f'Combobox delay not present: input took only {duration:.3f}s (expected >= 0.4s)'
async def test_datalist_field_no_delay(self, tools: Tools, browser_session: BrowserSession, base_url: str):
"""Native datalist fields should NOT get the 400ms delay — browser handles them instantly."""
import time
await tools.navigate(url=f'{base_url}/datalist-field', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.3)
await browser_session.get_browser_state_summary()
city_idx = await browser_session.get_index_by_id('city')
assert city_idx is not None
t0 = time.monotonic()
await tools.input(index=city_idx, text='Chi', browser_session=browser_session)
duration = time.monotonic() - t0
# Datalist fields should complete without the 400ms tax.
# Normal typing for 3 chars takes well under 400ms.
assert duration < 0.4, f'Datalist field got unexpected delay: {duration:.3f}s (should be < 0.4s)'
@@ -0,0 +1,275 @@
import pytest
from pytest_httpserver import HTTPServer
from browser_use.agent.views import ActionResult
from browser_use.browser import BrowserSession
from browser_use.browser.profile import BrowserProfile
from browser_use.tools.service import Tools
@pytest.fixture(scope='session')
def http_server():
"""Create and provide a test HTTP server that serves static content."""
server = HTTPServer()
server.start()
# Add route for ARIA menu test page
server.expect_request('/aria-menu').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head>
<title>ARIA Menu Test</title>
<style>
.menu {
list-style: none;
padding: 0;
margin: 0;
border: 1px solid #ccc;
background: white;
width: 200px;
}
.menu-item {
padding: 10px 20px;
border-bottom: 1px solid #eee;
}
.menu-item:hover {
background: #f0f0f0;
}
.menu-item-anchor {
text-decoration: none;
color: #333;
display: block;
}
#result {
margin-top: 20px;
padding: 10px;
border: 1px solid #ddd;
min-height: 20px;
}
</style>
</head>
<body>
<h1>ARIA Menu Test</h1>
<p>This menu uses ARIA roles instead of native select elements</p>
<!-- Exactly like the HTML provided in the issue -->
<ul class="menu menu-format-standard menu-regular" role="menu" id="pyNavigation1752753375773" style="display: block;">
<li class="menu-item menu-item-enabled" role="presentation">
<a href="#" onclick="pd(event);" class="menu-item-anchor" tabindex="0" role="menuitem">
<span class="menu-item-title-wrap"><span class="menu-item-title">Filter</span></span>
</a>
</li>
<li class="menu-item menu-item-enabled" role="presentation" id="menu-item-$PpyNavigation1752753375773$ppyElements$l2">
<a href="#" onclick="pd(event);" class="menu-item-anchor menu-item-expand" tabindex="0" role="menuitem" aria-haspopup="true">
<span class="menu-item-title-wrap"><span class="menu-item-title">Sort</span></span>
</a>
<div class="menu-panel-wrapper">
<ul class="menu menu-format-standard menu-regular" role="menu" id="$PpyNavigation1752753375773$ppyElements$l2">
<li class="menu-item menu-item-enabled" role="presentation">
<a href="#" onclick="pd(event);" class="menu-item-anchor" tabindex="0" role="menuitem">
<span class="menu-item-title-wrap"><span class="menu-item-title">Lowest to highest</span></span>
</a>
</li>
<li class="menu-item menu-item-enabled" role="presentation">
<a href="#" onclick="pd(event);" class="menu-item-anchor" tabindex="0" role="menuitem">
<span class="menu-item-title-wrap"><span class="menu-item-title">Highest to lowest</span></span>
</a>
</li>
</ul>
</div>
</li>
<li class="menu-item menu-item-enabled" role="presentation">
<a href="#" onclick="pd(event);" class="menu-item-anchor" tabindex="0" role="menuitem">
<span class="menu-item-title-wrap"><span class="menu-item-title">Appearance</span></span>
</a>
</li>
<li class="menu-item menu-item-enabled" role="presentation">
<a href="#" onclick="pd(event);" class="menu-item-anchor" tabindex="0" role="menuitem">
<span class="menu-item-title-wrap"><span class="menu-item-title">Summarize</span></span>
</a>
</li>
<li class="menu-item menu-item-enabled" role="presentation">
<a href="#" onclick="pd(event);" class="menu-item-anchor" tabindex="0" role="menuitem">
<span class="menu-item-title-wrap"><span class="menu-item-title">Delete</span></span>
</a>
</li>
</ul>
<div id="result">Click an option to see the result</div>
<script>
// Mock the pd function that prevents default
function pd(event) {
event.preventDefault();
const text = event.target.closest('[role="menuitem"]').textContent.trim();
document.getElementById('result').textContent = 'Clicked: ' + text;
}
</script>
</body>
</html>
""",
content_type='text/html',
)
yield server
server.stop()
@pytest.fixture(scope='session')
def base_url(http_server):
"""Return the base URL for the test HTTP server."""
return f'http://{http_server.host}:{http_server.port}'
@pytest.fixture(scope='module')
async def browser_session():
"""Create and provide a Browser instance with security disabled."""
browser_session = BrowserSession(
browser_profile=BrowserProfile(
headless=True,
user_data_dir=None,
keep_alive=True,
chromium_sandbox=False, # Disable sandbox for CI environment
)
)
await browser_session.start()
yield browser_session
await browser_session.kill()
@pytest.fixture(scope='function')
def tools():
"""Create and provide a Tools instance."""
return Tools()
class TestARIAMenuDropdown:
"""Test ARIA menu support for get_dropdown_options and select_dropdown_option."""
@pytest.mark.skip(reason='TODO: fix')
async def test_get_dropdown_options_with_aria_menu(self, tools, browser_session: BrowserSession, base_url):
"""Test that get_dropdown_options can retrieve options from ARIA menus."""
# Navigate to the ARIA menu test page
await tools.navigate(url=f'{base_url}/aria-menu', new_tab=False, browser_session=browser_session)
# Wait for the page to load
from browser_use.browser.events import NavigationCompleteEvent
await browser_session.event_bus.expect(NavigationCompleteEvent, timeout=10.0)
# Initialize the DOM state to populate the selector map
await browser_session.get_browser_state_summary()
# Find the ARIA menu element by ID
menu_index = await browser_session.get_index_by_id('pyNavigation1752753375773')
assert menu_index is not None, 'Could not find ARIA menu element'
# Execute the action with the menu index
result = await tools.dropdown_options(index=menu_index, browser_session=browser_session)
# Verify the result structure
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
# Expected ARIA menu options
expected_options = ['Filter', 'Sort', 'Appearance', 'Summarize', 'Delete']
# Verify all options are returned
for option in expected_options:
assert option in result.extracted_content, f"Option '{option}' not found in result content"
# Verify the instruction for using the text in select_dropdown is included
assert 'Use the exact text string in select_dropdown' in result.extracted_content
@pytest.mark.skip(reason='TODO: fix')
async def test_select_dropdown_option_with_aria_menu(self, tools, browser_session: BrowserSession, base_url):
"""Test that select_dropdown_option can select an option from ARIA menus."""
# Navigate to the ARIA menu test page
await tools.navigate(url=f'{base_url}/aria-menu', new_tab=False, browser_session=browser_session)
# Wait for the page to load
from browser_use.browser.events import NavigationCompleteEvent
await browser_session.event_bus.expect(NavigationCompleteEvent, timeout=10.0)
# Initialize the DOM state to populate the selector map
await browser_session.get_browser_state_summary()
# Find the ARIA menu element by ID
menu_index = await browser_session.get_index_by_id('pyNavigation1752753375773')
assert menu_index is not None, 'Could not find ARIA menu element'
# Execute the action with the menu index to select "Filter"
result = await tools.select_dropdown(index=menu_index, text='Filter', browser_session=browser_session)
# Verify the result structure
assert isinstance(result, ActionResult)
# Core logic validation: Verify selection was successful
assert result.extracted_content is not None
assert 'selected option' in result.extracted_content.lower() or 'clicked' in result.extracted_content.lower()
assert 'Filter' in result.extracted_content
# Verify the click actually had an effect on the page using CDP
cdp_session = await browser_session.get_or_create_cdp_session()
result = await cdp_session.cdp_client.send.Runtime.evaluate(
params={'expression': "document.getElementById('result').textContent", 'returnByValue': True},
session_id=cdp_session.session_id,
)
result_text = result.get('result', {}).get('value', '')
assert 'Filter' in result_text, f"Expected 'Filter' in result text, got '{result_text}'"
@pytest.mark.skip(reason='TODO: fix')
async def test_get_dropdown_options_with_nested_aria_menu(self, tools, browser_session: BrowserSession, base_url):
"""Test that get_dropdown_options can handle nested ARIA menus (like Sort submenu)."""
# Navigate to the ARIA menu test page
await tools.navigate(url=f'{base_url}/aria-menu', new_tab=False, browser_session=browser_session)
# Wait for the page to load
from browser_use.browser.events import NavigationCompleteEvent
await browser_session.event_bus.expect(NavigationCompleteEvent, timeout=10.0)
# Initialize the DOM state to populate the selector map
await browser_session.get_browser_state_summary()
# Get the selector map
selector_map = await browser_session.get_selector_map()
# Find the nested ARIA menu element in the selector map
nested_menu_index = None
for idx, element in selector_map.items():
# Look for the nested UL with id containing "$PpyNavigation"
if (
element.tag_name.lower() == 'ul'
and '$PpyNavigation' in str(element.attributes.get('id', ''))
and element.attributes.get('role') == 'menu'
):
nested_menu_index = idx
break
# The nested menu might not be in the selector map initially if it's hidden
# In that case, we should test the main menu
if nested_menu_index is None:
# Find the main menu instead
for idx, element in selector_map.items():
if element.tag_name.lower() == 'ul' and element.attributes.get('id') == 'pyNavigation1752753375773':
nested_menu_index = idx
break
assert nested_menu_index is not None, (
f'Could not find any ARIA menu element in selector map. Available elements: {[f"{idx}: {element.tag_name}" for idx, element in selector_map.items()]}'
)
# Execute the action with the menu index
result = await tools.dropdown_options(index=nested_menu_index, browser_session=browser_session)
# Verify the result structure
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
# The action should return some menu options
assert 'Use the exact text string in select_dropdown' in result.extracted_content
@@ -0,0 +1,517 @@
"""Test GetDropdownOptionsEvent and SelectDropdownOptionEvent functionality.
This file consolidates all tests related to dropdown functionality including:
- Native <select> dropdowns
- ARIA role="menu" dropdowns
- Custom dropdown implementations
"""
import pytest
from pytest_httpserver import HTTPServer
from browser_use.agent.views import ActionResult
from browser_use.browser import BrowserSession
from browser_use.browser.events import GetDropdownOptionsEvent, NavigationCompleteEvent, SelectDropdownOptionEvent
from browser_use.browser.profile import BrowserProfile
from browser_use.tools.service import Tools
@pytest.fixture(scope='session')
def http_server():
"""Create and provide a test HTTP server that serves static content."""
server = HTTPServer()
server.start()
# Add route for native dropdown test page
server.expect_request('/native-dropdown').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head>
<title>Native Dropdown Test</title>
</head>
<body>
<h1>Native Dropdown Test</h1>
<select id="test-dropdown" name="test-dropdown">
<option value="">Please select</option>
<option value="option1">First Option</option>
<option value="option2">Second Option</option>
<option value="option3">Third Option</option>
</select>
<div id="result">No selection made</div>
<script>
document.getElementById('test-dropdown').addEventListener('change', function(e) {
document.getElementById('result').textContent = 'Selected: ' + e.target.options[e.target.selectedIndex].text;
});
</script>
</body>
</html>
""",
content_type='text/html',
)
# Add route for ARIA menu test page
server.expect_request('/aria-menu').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head>
<title>ARIA Menu Test</title>
<style>
.menu {
list-style: none;
padding: 0;
margin: 0;
border: 1px solid #ccc;
background: white;
width: 200px;
}
.menu-item {
padding: 10px 20px;
border-bottom: 1px solid #eee;
}
.menu-item:hover {
background: #f0f0f0;
}
.menu-item-anchor {
text-decoration: none;
color: #333;
display: block;
}
#result {
margin-top: 20px;
padding: 10px;
border: 1px solid #ddd;
min-height: 20px;
}
</style>
</head>
<body>
<h1>ARIA Menu Test</h1>
<p>This menu uses ARIA roles instead of native select elements</p>
<ul class="menu menu-format-standard menu-regular" role="menu" id="pyNavigation1752753375773" style="display: block;">
<li class="menu-item menu-item-enabled" role="presentation">
<a href="#" onclick="pd(event);" class="menu-item-anchor" tabindex="0" role="menuitem">
<span class="menu-item-title-wrap"><span class="menu-item-title">Filter</span></span>
</a>
</li>
<li class="menu-item menu-item-enabled" role="presentation" id="menu-item-$PpyNavigation1752753375773$ppyElements$l2">
<a href="#" onclick="pd(event);" class="menu-item-anchor menu-item-expand" tabindex="0" role="menuitem" aria-haspopup="true">
<span class="menu-item-title-wrap"><span class="menu-item-title">Sort</span></span>
</a>
<div class="menu-panel-wrapper">
<ul class="menu menu-format-standard menu-regular" role="menu" id="$PpyNavigation1752753375773$ppyElements$l2">
<li class="menu-item menu-item-enabled" role="presentation">
<a href="#" onclick="pd(event);" class="menu-item-anchor" tabindex="0" role="menuitem">
<span class="menu-item-title-wrap"><span class="menu-item-title">Lowest to highest</span></span>
</a>
</li>
<li class="menu-item menu-item-enabled" role="presentation">
<a href="#" onclick="pd(event);" class="menu-item-anchor" tabindex="0" role="menuitem">
<span class="menu-item-title-wrap"><span class="menu-item-title">Highest to lowest</span></span>
</a>
</li>
</ul>
</div>
</li>
<li class="menu-item menu-item-enabled" role="presentation">
<a href="#" onclick="pd(event);" class="menu-item-anchor" tabindex="0" role="menuitem">
<span class="menu-item-title-wrap"><span class="menu-item-title">Appearance</span></span>
</a>
</li>
<li class="menu-item menu-item-enabled" role="presentation">
<a href="#" onclick="pd(event);" class="menu-item-anchor" tabindex="0" role="menuitem">
<span class="menu-item-title-wrap"><span class="menu-item-title">Summarize</span></span>
</a>
</li>
<li class="menu-item menu-item-enabled" role="presentation">
<a href="#" onclick="pd(event);" class="menu-item-anchor" tabindex="0" role="menuitem">
<span class="menu-item-title-wrap"><span class="menu-item-title">Delete</span></span>
</a>
</li>
</ul>
<div id="result">Click an option to see the result</div>
<script>
// Mock the pd function that prevents default
function pd(event) {
event.preventDefault();
const text = event.target.closest('[role="menuitem"]').textContent.trim();
document.getElementById('result').textContent = 'Clicked: ' + text;
}
</script>
</body>
</html>
""",
content_type='text/html',
)
# Add route for custom dropdown test page
server.expect_request('/custom-dropdown').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head>
<title>Custom Dropdown Test</title>
<style>
.dropdown {
position: relative;
display: inline-block;
width: 200px;
}
.dropdown-button {
padding: 10px;
border: 1px solid #ccc;
background: white;
cursor: pointer;
width: 100%;
}
.dropdown-menu {
position: absolute;
top: 100%;
left: 0;
right: 0;
border: 1px solid #ccc;
background: white;
display: block;
z-index: 1000;
}
.dropdown-menu.hidden {
display: none;
}
.dropdown .item {
padding: 10px;
cursor: pointer;
}
.dropdown .item:hover {
background: #f0f0f0;
}
.dropdown .item.selected {
background: #e0e0e0;
}
#result {
margin-top: 20px;
padding: 10px;
border: 1px solid #ddd;
}
</style>
</head>
<body>
<h1>Custom Dropdown Test</h1>
<p>This is a custom dropdown implementation (like Semantic UI)</p>
<div class="dropdown ui" id="custom-dropdown">
<div class="dropdown-button" onclick="toggleDropdown()">
<span id="selected-text">Choose an option</span>
</div>
<div class="dropdown-menu" id="dropdown-menu">
<div class="item" data-value="red" onclick="selectOption('Red', 'red')">Red</div>
<div class="item" data-value="green" onclick="selectOption('Green', 'green')">Green</div>
<div class="item" data-value="blue" onclick="selectOption('Blue', 'blue')">Blue</div>
<div class="item" data-value="yellow" onclick="selectOption('Yellow', 'yellow')">Yellow</div>
</div>
</div>
<div id="result">No selection made</div>
<script>
function toggleDropdown() {
const menu = document.getElementById('dropdown-menu');
menu.classList.toggle('hidden');
}
function selectOption(text, value) {
document.getElementById('selected-text').textContent = text;
document.getElementById('result').textContent = 'Selected: ' + text + ' (value: ' + value + ')';
// Mark as selected
document.querySelectorAll('.item').forEach(item => item.classList.remove('selected'));
event.target.classList.add('selected');
// Close dropdown
document.getElementById('dropdown-menu').classList.add('hidden');
}
</script>
</body>
</html>
""",
content_type='text/html',
)
yield server
server.stop()
@pytest.fixture(scope='session')
def base_url(http_server):
"""Return the base URL for the test HTTP server."""
return f'http://{http_server.host}:{http_server.port}'
@pytest.fixture(scope='module')
async def browser_session():
"""Create and provide a Browser instance with security disabled."""
browser_session = BrowserSession(
browser_profile=BrowserProfile(
headless=True,
user_data_dir=None,
keep_alive=True,
chromium_sandbox=False, # Disable sandbox for CI environment
)
)
await browser_session.start()
yield browser_session
await browser_session.kill()
@pytest.fixture(scope='function')
def tools():
"""Create and provide a Tools instance."""
return Tools()
class TestGetDropdownOptionsEvent:
"""Test GetDropdownOptionsEvent functionality for various dropdown types."""
@pytest.mark.skip(reason='Dropdown text assertion issue - test expects specific text format')
async def test_native_select_dropdown(self, tools, browser_session: BrowserSession, base_url):
"""Test get_dropdown_options with native HTML select element."""
# Navigate to the native dropdown test page
await tools.navigate(url=f'{base_url}/native-dropdown', new_tab=False, browser_session=browser_session)
# Initialize the DOM state to populate the selector map
await browser_session.get_browser_state_summary()
# Find the select element by ID
dropdown_index = await browser_session.get_index_by_id('test-dropdown')
assert dropdown_index is not None, 'Could not find select element'
# Test via tools action
result = await tools.dropdown_options(index=dropdown_index, browser_session=browser_session)
# Verify the result
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
# Verify all expected options are present
expected_options = ['Please select', 'First Option', 'Second Option', 'Third Option']
for option in expected_options:
assert option in result.extracted_content, f"Option '{option}' not found in result content"
# Verify instruction is included
assert 'Use the exact text string' in result.extracted_content and 'select_dropdown' in result.extracted_content
# Also test direct event dispatch
node = await browser_session.get_element_by_index(dropdown_index)
assert node is not None
event = browser_session.event_bus.dispatch(GetDropdownOptionsEvent(node=node))
dropdown_data = await event.event_result(timeout=3.0)
assert dropdown_data is not None
assert 'options' in dropdown_data
assert 'type' in dropdown_data
assert dropdown_data['type'] == 'select'
@pytest.mark.skip(reason='ARIA menu detection issue - element not found in selector map')
async def test_aria_menu_dropdown(self, tools, browser_session: BrowserSession, base_url):
"""Test get_dropdown_options with ARIA role='menu' element."""
# Navigate to the ARIA menu test page
await tools.navigate(url=f'{base_url}/aria-menu', new_tab=False, browser_session=browser_session)
# Initialize the DOM state
await browser_session.get_browser_state_summary()
# Find the ARIA menu by ID
menu_index = await browser_session.get_index_by_id('pyNavigation1752753375773')
assert menu_index is not None, 'Could not find ARIA menu element'
# Test via tools action
result = await tools.dropdown_options(index=menu_index, browser_session=browser_session)
# Verify the result
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
# Verify expected ARIA menu options are present
expected_options = ['Filter', 'Sort', 'Appearance', 'Summarize', 'Delete']
for option in expected_options:
assert option in result.extracted_content, f"Option '{option}' not found in result content"
# Also test direct event dispatch
node = await browser_session.get_element_by_index(menu_index)
assert node is not None
event = browser_session.event_bus.dispatch(GetDropdownOptionsEvent(node=node))
dropdown_data = await event.event_result(timeout=3.0)
assert dropdown_data is not None
assert 'options' in dropdown_data
assert 'type' in dropdown_data
assert dropdown_data['type'] == 'aria'
@pytest.mark.skip(reason='Custom dropdown detection issue - element not found in selector map')
async def test_custom_dropdown(self, tools, browser_session: BrowserSession, base_url):
"""Test get_dropdown_options with custom dropdown implementation."""
# Navigate to the custom dropdown test page
await tools.navigate(url=f'{base_url}/custom-dropdown', new_tab=False, browser_session=browser_session)
# Initialize the DOM state
await browser_session.get_browser_state_summary()
# Find the custom dropdown by ID
dropdown_index = await browser_session.get_index_by_id('custom-dropdown')
assert dropdown_index is not None, 'Could not find custom dropdown element'
# Test via tools action
result = await tools.dropdown_options(index=dropdown_index, browser_session=browser_session)
# Verify the result
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
# Verify expected custom dropdown options are present
expected_options = ['Red', 'Green', 'Blue', 'Yellow']
for option in expected_options:
assert option in result.extracted_content, f"Option '{option}' not found in result content"
# Also test direct event dispatch
node = await browser_session.get_element_by_index(dropdown_index)
assert node is not None
event = browser_session.event_bus.dispatch(GetDropdownOptionsEvent(node=node))
dropdown_data = await event.event_result(timeout=3.0)
assert dropdown_data is not None
assert 'options' in dropdown_data
assert 'type' in dropdown_data
assert dropdown_data['type'] == 'custom'
class TestSelectDropdownOptionEvent:
"""Test SelectDropdownOptionEvent functionality for various dropdown types."""
@pytest.mark.skip(reason='Timeout issue - test takes too long to complete')
async def test_select_native_dropdown_option(self, tools, browser_session: BrowserSession, base_url):
"""Test select_dropdown_option with native HTML select element."""
# Navigate to the native dropdown test page
await tools.navigate(url=f'{base_url}/native-dropdown', new_tab=False, browser_session=browser_session)
await browser_session.event_bus.expect(NavigationCompleteEvent, timeout=10.0)
# Initialize the DOM state
await browser_session.get_browser_state_summary()
# Find the select element by ID
dropdown_index = await browser_session.get_index_by_id('test-dropdown')
assert dropdown_index is not None
# Test via tools action
result = await tools.select_dropdown(index=dropdown_index, text='Second Option', browser_session=browser_session)
# Verify the result
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'Second Option' in result.extracted_content
# Verify the selection actually worked using CDP
cdp_session = await browser_session.get_or_create_cdp_session()
result = await cdp_session.cdp_client.send.Runtime.evaluate(
params={'expression': "document.getElementById('test-dropdown').selectedIndex", 'returnByValue': True},
session_id=cdp_session.session_id,
)
selected_index = result.get('result', {}).get('value', -1)
assert selected_index == 2, f'Expected selected index 2, got {selected_index}'
@pytest.mark.skip(reason='Timeout issue - test takes too long to complete')
async def test_select_aria_menu_option(self, tools, browser_session: BrowserSession, base_url):
"""Test select_dropdown_option with ARIA menu."""
# Navigate to the ARIA menu test page
await tools.navigate(url=f'{base_url}/aria-menu', new_tab=False, browser_session=browser_session)
await browser_session.event_bus.expect(NavigationCompleteEvent, timeout=10.0)
# Initialize the DOM state
await browser_session.get_browser_state_summary()
# Find the ARIA menu by ID
menu_index = await browser_session.get_index_by_id('pyNavigation1752753375773')
assert menu_index is not None
# Test via tools action
result = await tools.select_dropdown(index=menu_index, text='Filter', browser_session=browser_session)
# Verify the result
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'Filter' in result.extracted_content
# Verify the click had an effect using CDP
cdp_session = await browser_session.get_or_create_cdp_session()
result = await cdp_session.cdp_client.send.Runtime.evaluate(
params={'expression': "document.getElementById('result').textContent", 'returnByValue': True},
session_id=cdp_session.session_id,
)
result_text = result.get('result', {}).get('value', '')
assert 'Filter' in result_text, f"Expected 'Filter' in result text, got '{result_text}'"
@pytest.mark.skip(reason='Timeout issue - test takes too long to complete')
async def test_select_custom_dropdown_option(self, tools, browser_session: BrowserSession, base_url):
"""Test select_dropdown_option with custom dropdown."""
# Navigate to the custom dropdown test page
await tools.navigate(url=f'{base_url}/custom-dropdown', new_tab=False, browser_session=browser_session)
await browser_session.event_bus.expect(NavigationCompleteEvent, timeout=10.0)
# Initialize the DOM state
await browser_session.get_browser_state_summary()
# Find the custom dropdown by ID
dropdown_index = await browser_session.get_index_by_id('custom-dropdown')
assert dropdown_index is not None
# Test via tools action
result = await tools.select_dropdown(index=dropdown_index, text='Blue', browser_session=browser_session)
# Verify the result
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'Blue' in result.extracted_content
# Verify the selection worked using CDP
cdp_session = await browser_session.get_or_create_cdp_session()
result = await cdp_session.cdp_client.send.Runtime.evaluate(
params={'expression': "document.getElementById('result').textContent", 'returnByValue': True},
session_id=cdp_session.session_id,
)
result_text = result.get('result', {}).get('value', '')
assert 'Blue' in result_text, f"Expected 'Blue' in result text, got '{result_text}'"
@pytest.mark.skip(reason='Timeout issue - test takes too long to complete')
async def test_select_invalid_option_error(self, tools, browser_session: BrowserSession, base_url):
"""Test select_dropdown_option with non-existent option text."""
# Navigate to the native dropdown test page
await tools.navigate(url=f'{base_url}/native-dropdown', new_tab=False, browser_session=browser_session)
await browser_session.event_bus.expect(NavigationCompleteEvent, timeout=10.0)
# Initialize the DOM state
await browser_session.get_browser_state_summary()
# Find the select element by ID
dropdown_index = await browser_session.get_index_by_id('test-dropdown')
assert dropdown_index is not None
# Try to select non-existent option via direct event
node = await browser_session.get_element_by_index(dropdown_index)
assert node is not None
event = browser_session.event_bus.dispatch(SelectDropdownOptionEvent(node=node, text='Non-existent Option'))
try:
selection_data = await event.event_result(timeout=3.0)
# Should have an error in the result
assert selection_data is not None
assert 'error' in selection_data or 'not found' in str(selection_data).lower()
except Exception as e:
# Or raise an exception
assert 'not found' in str(e).lower() or 'no option' in str(e).lower()
+274
View File
@@ -0,0 +1,274 @@
"""Test radio button click interactions with various label association patterns.
Verifies that the occlusion check correctly identifies label-input associations
so that CDP click dispatch works for radio buttons whose labels visually overlap them.
"""
import pytest
from pytest_httpserver import HTTPServer
from browser_use.browser import BrowserSession
from browser_use.browser.profile import BrowserProfile
from browser_use.tools.service import Tools
# -- HTML fixtures --
RADIO_SIBLING_HTML = """
<!DOCTYPE html>
<html>
<head><title>Radio Sibling Label Test</title>
<style>
.radio-group { display: flex; flex-direction: column; gap: 8px; }
.radio-group input[type="radio"] { width: 16px; height: 16px; }
.radio-group label { cursor: pointer; padding: 4px 8px; }
</style>
</head>
<body>
<div class="radio-group">
<div>
<input type="radio" name="color" value="red" id="radio-red">
<label for="radio-red">Red</label>
</div>
<div>
<input type="radio" name="color" value="blue" id="radio-blue">
<label for="radio-blue">Blue</label>
</div>
<div>
<input type="radio" name="color" value="green" id="radio-green">
<label for="radio-green">Green</label>
</div>
</div>
<div id="result"></div>
<script>
document.querySelectorAll('input[name="color"]').forEach(r => {
r.addEventListener('change', () => {
document.getElementById('result').textContent = 'selected:' + r.value;
});
});
</script>
</body>
</html>
"""
RADIO_WRAPPED_HTML = """
<!DOCTYPE html>
<html>
<head><title>Radio Wrapped Label Test</title>
<style>
.radio-group label { display: block; cursor: pointer; padding: 8px; }
</style>
</head>
<body>
<div class="radio-group">
<label><input type="radio" name="fruit" value="apple" id="radio-apple"> Apple</label>
<label><input type="radio" name="fruit" value="banana" id="radio-banana"> Banana</label>
<label><input type="radio" name="fruit" value="cherry" id="radio-cherry"> Cherry</label>
</div>
<div id="result"></div>
<script>
document.querySelectorAll('input[name="fruit"]').forEach(r => {
r.addEventListener('change', () => {
document.getElementById('result').textContent = 'selected:' + r.value;
});
});
</script>
</body>
</html>
"""
RADIO_CUSTOM_HTML = """
<!DOCTYPE html>
<html>
<head><title>Radio Custom Styled Test</title>
<style>
.radio-group { display: flex; flex-direction: column; gap: 8px; }
.radio-group input[type="radio"] {
appearance: none;
-webkit-appearance: none;
width: 20px;
height: 20px;
border: 2px solid #999;
border-radius: 50%;
position: relative;
}
.radio-group input[type="radio"]:checked {
border-color: #007bff;
background: #007bff;
}
.radio-group label {
cursor: pointer;
padding: 4px 8px;
display: inline-flex;
align-items: center;
gap: 6px;
}
</style>
</head>
<body>
<div class="radio-group">
<div>
<input type="radio" name="size" value="small" id="radio-small">
<label for="radio-small">Small</label>
</div>
<div>
<input type="radio" name="size" value="medium" id="radio-medium">
<label for="radio-medium">Medium</label>
</div>
<div>
<input type="radio" name="size" value="large" id="radio-large">
<label for="radio-large">Large</label>
</div>
</div>
<div id="result"></div>
<script>
document.querySelectorAll('input[name="size"]').forEach(r => {
r.addEventListener('change', () => {
document.getElementById('result').textContent = 'selected:' + r.value;
});
});
</script>
</body>
</html>
"""
@pytest.fixture(scope='session')
def http_server():
server = HTTPServer()
server.start()
server.expect_request('/radio-sibling').respond_with_data(RADIO_SIBLING_HTML, content_type='text/html')
server.expect_request('/radio-wrapped').respond_with_data(RADIO_WRAPPED_HTML, content_type='text/html')
server.expect_request('/radio-custom').respond_with_data(RADIO_CUSTOM_HTML, content_type='text/html')
yield server
server.stop()
@pytest.fixture(scope='session')
def base_url(http_server):
return f'http://{http_server.host}:{http_server.port}'
@pytest.fixture(scope='module')
async def browser_session():
browser_session = BrowserSession(
browser_profile=BrowserProfile(
headless=True,
user_data_dir=None,
keep_alive=True,
chromium_sandbox=False,
)
)
await browser_session.start()
yield browser_session
await browser_session.kill()
@pytest.fixture(scope='function')
def tools():
return Tools()
async def _get_checked_and_result(browser_session: BrowserSession, input_id: str) -> tuple[bool, str]:
"""Helper: returns (is_checked, result_div_text) via CDP."""
cdp_session = await browser_session.get_or_create_cdp_session()
sid = cdp_session.session_id
checked_result = await cdp_session.cdp_client.send.Runtime.evaluate(
params={
'expression': f"document.getElementById('{input_id}').checked",
'returnByValue': True,
},
session_id=sid,
)
is_checked = checked_result.get('result', {}).get('value', False)
text_result = await cdp_session.cdp_client.send.Runtime.evaluate(
params={
'expression': "document.getElementById('result').textContent",
'returnByValue': True,
},
session_id=sid,
)
result_text = text_result.get('result', {}).get('value', '')
return is_checked, result_text
class TestRadioButtons:
"""Test radio button clicks across label association patterns."""
async def test_sibling_label_radio_click(self, tools: Tools, browser_session: BrowserSession, base_url: str):
"""Click a radio whose sibling <label for=...> may occlude it."""
await tools.navigate(url=f'{base_url}/radio-sibling', new_tab=False, browser_session=browser_session)
await browser_session.get_browser_state_summary()
idx = await browser_session.get_index_by_id('radio-blue')
assert idx is not None, 'Could not find radio-blue in selector map'
result = await tools.click(index=idx, browser_session=browser_session)
assert result.error is None, f'Click failed: {result.error}'
is_checked, result_text = await _get_checked_and_result(browser_session, 'radio-blue')
assert is_checked, 'radio-blue should be checked after click'
assert 'selected:blue' in result_text, f'Change event not fired, result: {result_text}'
async def test_wrapped_label_radio_click(self, tools: Tools, browser_session: BrowserSession, base_url: str):
"""Click a radio wrapped inside its <label>."""
await tools.navigate(url=f'{base_url}/radio-wrapped', new_tab=False, browser_session=browser_session)
await browser_session.get_browser_state_summary()
idx = await browser_session.get_index_by_id('radio-banana')
assert idx is not None, 'Could not find radio-banana in selector map'
result = await tools.click(index=idx, browser_session=browser_session)
assert result.error is None, f'Click failed: {result.error}'
is_checked, result_text = await _get_checked_and_result(browser_session, 'radio-banana')
assert is_checked, 'radio-banana should be checked after click'
assert 'selected:banana' in result_text, f'Change event not fired, result: {result_text}'
async def test_custom_styled_radio_click(self, tools: Tools, browser_session: BrowserSession, base_url: str):
"""Click a CSS-customized radio (appearance:none) with sibling label."""
await tools.navigate(url=f'{base_url}/radio-custom', new_tab=False, browser_session=browser_session)
await browser_session.get_browser_state_summary()
idx = await browser_session.get_index_by_id('radio-large')
assert idx is not None, 'Could not find radio-large in selector map'
result = await tools.click(index=idx, browser_session=browser_session)
assert result.error is None, f'Click failed: {result.error}'
is_checked, result_text = await _get_checked_and_result(browser_session, 'radio-large')
assert is_checked, 'radio-large should be checked after click'
assert 'selected:large' in result_text, f'Change event not fired, result: {result_text}'
async def test_radio_group_switching(self, tools: Tools, browser_session: BrowserSession, base_url: str):
"""Click one radio then another in the same group; first should uncheck."""
await tools.navigate(url=f'{base_url}/radio-sibling', new_tab=False, browser_session=browser_session)
await browser_session.get_browser_state_summary()
# Click red first
red_idx = await browser_session.get_index_by_id('radio-red')
assert red_idx is not None
result = await tools.click(index=red_idx, browser_session=browser_session)
assert result.error is None, f'Click red failed: {result.error}'
is_red_checked, _ = await _get_checked_and_result(browser_session, 'radio-red')
assert is_red_checked, 'radio-red should be checked'
# Re-fetch state so indices are current, then click green
await browser_session.get_browser_state_summary()
green_idx = await browser_session.get_index_by_id('radio-green')
assert green_idx is not None
result = await tools.click(index=green_idx, browser_session=browser_session)
assert result.error is None, f'Click green failed: {result.error}'
is_green_checked, result_text = await _get_checked_and_result(browser_session, 'radio-green')
assert is_green_checked, 'radio-green should be checked'
assert 'selected:green' in result_text
# Verify red is now unchecked
is_red_checked, _ = await _get_checked_and_result(browser_session, 'radio-red')
assert not is_red_checked, 'radio-red should be unchecked after selecting green'
+109
View File
@@ -0,0 +1,109 @@
"""Shared test helper for LLM model tests."""
import os
import pytest
from browser_use.agent.service import Agent
from browser_use.browser.profile import BrowserProfile
from browser_use.browser.session import BrowserSession
async def run_model_button_click_test(
model_class,
model_name: str,
api_key_env: str | None,
extra_kwargs: dict,
httpserver,
):
"""Test that an LLM model can click a button.
This test verifies:
1. Model can be initialized with API key
2. Agent can navigate and click a button
3. Button click is verified by checking page state change
4. Completes within max 2 steps
"""
# Handle API key validation - skip test if not available
if api_key_env is not None:
api_key = os.getenv(api_key_env)
if not api_key:
pytest.skip(f'{api_key_env} not set - skipping test')
else:
api_key = None
# Handle Azure-specific endpoint validation
from browser_use.llm.azure.chat import ChatAzureOpenAI
if model_class is ChatAzureOpenAI:
azure_endpoint = os.getenv('AZURE_OPENAI_ENDPOINT')
if not azure_endpoint:
pytest.skip('AZURE_OPENAI_ENDPOINT not set - skipping test')
# Add the azure_endpoint to extra_kwargs
extra_kwargs = {**extra_kwargs, 'azure_endpoint': azure_endpoint}
# Create HTML page with a button that changes page content when clicked
html = """
<!DOCTYPE html>
<html>
<head><title>Button Test</title></head>
<body>
<h1>Button Click Test</h1>
<button id="test-button" onclick="document.getElementById('result').innerText='SUCCESS'">
Click Me
</button>
<div id="result">NOT_CLICKED</div>
</body>
</html>
"""
httpserver.expect_request('/').respond_with_data(html, content_type='text/html')
# Create LLM instance with extra kwargs if provided
llm_kwargs = {'model': model_name}
if api_key is not None:
llm_kwargs['api_key'] = api_key
llm_kwargs.update(extra_kwargs)
llm = model_class(**llm_kwargs) # type: ignore[arg-type]
# Create browser session
browser = BrowserSession(
browser_profile=BrowserProfile(
headless=True,
user_data_dir=None, # Use temporary directory
)
)
try:
# Start browser
await browser.start()
# Create agent with button click task (URL in task triggers auto-navigation)
test_url = httpserver.url_for('/')
agent = Agent(
task=f'{test_url} - Click the button',
llm=llm,
browser_session=browser,
max_steps=2, # Max 2 steps as per requirements
)
# Run the agent
result = await agent.run()
# Verify task completed
assert result is not None
assert len(result.history) > 0
# Verify button was clicked by checking page state across any step
button_clicked = False
for step in result.history:
# Check state_message which contains browser state with page text
if step.state_message and 'SUCCESS' in step.state_message:
button_clicked = True
break
# Check if SUCCESS appears in any step (indicating button was clicked)
assert button_clicked, 'Button was not clicked - SUCCESS not found in any page state'
finally:
# Clean up browser session
await browser.kill()
+266
View File
@@ -0,0 +1,266 @@
"""Tests for Azure OpenAI Responses API support."""
import os
import pytest
from browser_use.llm.azure.chat import RESPONSES_API_ONLY_MODELS, ChatAzureOpenAI
from browser_use.llm.messages import (
AssistantMessage,
ContentPartImageParam,
ContentPartTextParam,
Function,
ImageURL,
SystemMessage,
ToolCall,
UserMessage,
)
from browser_use.llm.openai.responses_serializer import ResponsesAPIMessageSerializer
class TestResponsesAPIMessageSerializer:
"""Tests for the ResponsesAPIMessageSerializer class."""
def test_serialize_user_message_string_content(self):
"""Test serializing a user message with string content."""
message = UserMessage(content='Hello, world!')
result = ResponsesAPIMessageSerializer.serialize(message)
assert result['role'] == 'user'
assert result['content'] == 'Hello, world!'
def test_serialize_user_message_text_parts(self):
"""Test serializing a user message with text content parts."""
message = UserMessage(
content=[
ContentPartTextParam(type='text', text='First part'),
ContentPartTextParam(type='text', text='Second part'),
]
)
result = ResponsesAPIMessageSerializer.serialize(message)
assert result['role'] == 'user'
assert isinstance(result['content'], list)
assert len(result['content']) == 2
assert result['content'][0]['type'] == 'input_text'
assert result['content'][0]['text'] == 'First part'
assert result['content'][1]['type'] == 'input_text'
assert result['content'][1]['text'] == 'Second part'
def test_serialize_user_message_with_image(self):
"""Test serializing a user message with image content."""
message = UserMessage(
content=[
ContentPartTextParam(type='text', text='What is in this image?'),
ContentPartImageParam(
type='image_url',
image_url=ImageURL(url='https://example.com/image.png', detail='auto'),
),
]
)
result = ResponsesAPIMessageSerializer.serialize(message)
assert result['role'] == 'user'
assert isinstance(result['content'], list)
assert len(result['content']) == 2
assert result['content'][0]['type'] == 'input_text'
assert result['content'][1]['type'] == 'input_image'
assert result['content'][1].get('image_url') == 'https://example.com/image.png'
assert result['content'][1].get('detail') == 'auto'
def test_serialize_system_message_string_content(self):
"""Test serializing a system message with string content."""
message = SystemMessage(content='You are a helpful assistant.')
result = ResponsesAPIMessageSerializer.serialize(message)
assert result['role'] == 'system'
assert result['content'] == 'You are a helpful assistant.'
def test_serialize_system_message_text_parts(self):
"""Test serializing a system message with text content parts."""
message = SystemMessage(content=[ContentPartTextParam(type='text', text='System instruction')])
result = ResponsesAPIMessageSerializer.serialize(message)
assert result['role'] == 'system'
assert isinstance(result['content'], list)
assert len(result['content']) == 1
assert result['content'][0]['type'] == 'input_text'
def test_serialize_assistant_message_string_content(self):
"""Test serializing an assistant message with string content."""
message = AssistantMessage(content='Here is my response.')
result = ResponsesAPIMessageSerializer.serialize(message)
assert result['role'] == 'assistant'
assert result['content'] == 'Here is my response.'
def test_serialize_assistant_message_none_content_with_tool_calls(self):
"""Test serializing an assistant message with None content and tool calls."""
message = AssistantMessage(
content=None,
tool_calls=[
ToolCall(
id='call_123',
type='function',
function=Function(name='search', arguments='{"query": "test"}'),
)
],
)
result = ResponsesAPIMessageSerializer.serialize(message)
assert result['role'] == 'assistant'
assert '[Tool call: search({"query": "test"})]' in result['content']
def test_serialize_assistant_message_none_content_no_tool_calls(self):
"""Test serializing an assistant message with None content and no tool calls."""
message = AssistantMessage(content=None)
result = ResponsesAPIMessageSerializer.serialize(message)
assert result['role'] == 'assistant'
assert result['content'] == ''
def test_serialize_messages_list(self):
"""Test serializing a list of messages."""
messages = [
SystemMessage(content='You are helpful.'),
UserMessage(content='Hello!'),
AssistantMessage(content='Hi there!'),
]
results = ResponsesAPIMessageSerializer.serialize_messages(messages)
assert len(results) == 3
assert results[0]['role'] == 'system'
assert results[1]['role'] == 'user'
assert results[2]['role'] == 'assistant'
class TestChatAzureOpenAIShouldUseResponsesAPI:
"""Tests for the _should_use_responses_api method."""
def test_use_responses_api_true(self):
"""Test that use_responses_api=True forces Responses API."""
llm = ChatAzureOpenAI(
model='gpt-4o',
api_key='test',
azure_endpoint='https://test.openai.azure.com',
use_responses_api=True,
)
assert llm._should_use_responses_api() is True
def test_use_responses_api_false(self):
"""Test that use_responses_api=False forces Chat Completions API."""
llm = ChatAzureOpenAI(
model='gpt-5.1-codex-mini', # Even with a Responses-only model
api_key='test',
azure_endpoint='https://test.openai.azure.com',
use_responses_api=False,
)
assert llm._should_use_responses_api() is False
def test_use_responses_api_auto_with_responses_only_model(self):
"""Test that auto mode detects Responses-only models."""
for model_name in RESPONSES_API_ONLY_MODELS:
llm = ChatAzureOpenAI(
model=model_name,
api_key='test',
azure_endpoint='https://test.openai.azure.com',
use_responses_api='auto',
)
assert llm._should_use_responses_api() is True, f'Expected Responses API for {model_name}'
def test_use_responses_api_auto_with_regular_model(self):
"""Test that auto mode uses Chat Completions for regular models."""
regular_models = ['gpt-4o', 'gpt-4.1-mini', 'gpt-3.5-turbo', 'gpt-4']
for model_name in regular_models:
llm = ChatAzureOpenAI(
model=model_name,
api_key='test',
azure_endpoint='https://test.openai.azure.com',
use_responses_api='auto',
)
assert llm._should_use_responses_api() is False, f'Expected Chat Completions for {model_name}'
def test_use_responses_api_auto_is_default(self):
"""Test that 'auto' is the default value for use_responses_api."""
llm = ChatAzureOpenAI(
model='gpt-4o',
api_key='test',
azure_endpoint='https://test.openai.azure.com',
)
assert llm.use_responses_api == 'auto'
def test_responses_api_only_models_list(self):
"""Test that the RESPONSES_API_ONLY_MODELS list contains expected models."""
expected_models = [
'gpt-5.1-codex',
'gpt-5.1-codex-mini',
'gpt-5.1-codex-max',
'gpt-5-codex',
'codex-mini-latest',
'computer-use-preview',
]
for model in expected_models:
assert model in RESPONSES_API_ONLY_MODELS, f'{model} should be in RESPONSES_API_ONLY_MODELS'
class TestChatAzureOpenAIIntegration:
"""Integration tests for Azure OpenAI with Responses API.
These tests require valid Azure OpenAI credentials and are skipped if not available.
"""
@pytest.fixture
def azure_credentials(self):
"""Get Azure OpenAI credentials from environment."""
api_key = os.getenv('AZURE_OPENAI_KEY') or os.getenv('AZURE_OPENAI_API_KEY')
endpoint = os.getenv('AZURE_OPENAI_ENDPOINT')
if not api_key or not endpoint:
pytest.skip('Azure OpenAI credentials not available')
return {'api_key': api_key, 'azure_endpoint': endpoint}
async def test_chat_completions_api_basic_call(self, azure_credentials):
"""Test basic call using Chat Completions API."""
llm = ChatAzureOpenAI(
model='gpt-4.1-mini',
api_key=azure_credentials['api_key'],
azure_endpoint=azure_credentials['azure_endpoint'],
use_responses_api=False, # Force Chat Completions API
)
messages = [
SystemMessage(content='You are a helpful assistant.'),
UserMessage(content='Say "hello" and nothing else.'),
]
result = await llm.ainvoke(messages)
assert result.completion is not None
assert 'hello' in result.completion.lower()
async def test_responses_api_basic_call(self, azure_credentials):
"""Test basic call using Responses API.
This test only runs if the Azure deployment supports the Responses API
(api_version >= 2025-03-01-preview).
"""
llm = ChatAzureOpenAI(
model='gpt-4.1-mini',
api_key=azure_credentials['api_key'],
azure_endpoint=azure_credentials['azure_endpoint'],
api_version='2025-03-01-preview', # Required for Responses API
use_responses_api=True, # Force Responses API
)
messages = [
SystemMessage(content='You are a helpful assistant.'),
UserMessage(content='Say "hello" and nothing else.'),
]
try:
result = await llm.ainvoke(messages)
assert result.completion is not None
assert 'hello' in result.completion.lower()
except Exception as e:
# Skip if Responses API is not supported
if 'Responses API' in str(e) or '404' in str(e):
pytest.skip('Responses API not supported by this Azure deployment')
raise
+15
View File
@@ -0,0 +1,15 @@
"""Test Anthropic model button click."""
from browser_use.llm.anthropic.chat import ChatAnthropic
from tests.ci.models.model_test_helper import run_model_button_click_test
async def test_anthropic_claude_sonnet_4_6(httpserver):
"""Test Anthropic claude-sonnet-4-6 can click a button."""
await run_model_button_click_test(
model_class=ChatAnthropic,
model_name='claude-sonnet-4-6',
api_key_env='ANTHROPIC_API_KEY',
extra_kwargs={},
httpserver=httpserver,
)
+15
View File
@@ -0,0 +1,15 @@
"""Test Azure OpenAI model button click."""
from browser_use.llm.azure.chat import ChatAzureOpenAI
from tests.ci.models.model_test_helper import run_model_button_click_test
async def test_azure_gpt_4_1_mini(httpserver):
"""Test Azure OpenAI gpt-4.1-mini can click a button."""
await run_model_button_click_test(
model_class=ChatAzureOpenAI,
model_name='gpt-4.1-mini',
api_key_env='AZURE_OPENAI_KEY',
extra_kwargs={}, # Azure endpoint will be added by helper
httpserver=httpserver,
)
+118
View File
@@ -0,0 +1,118 @@
"""Tests for the ChatBrowserUse cloud client."""
import pytest
from browser_use.llm.browser_use.chat import ChatBrowserUse
from browser_use.llm.messages import UserMessage
from tests.ci.models.model_test_helper import run_model_button_click_test
# A syntactically-valid key so the constructor doesn't bail before we reach the
# code under test. These unit tests never hit the network.
TEST_API_KEY = 'test-key-not-real'
async def test_browseruse_bu_latest(httpserver):
"""Test Browser Use bu-latest can click a button."""
await run_model_button_click_test(
model_class=ChatBrowserUse,
model_name='bu-latest',
api_key_env='BROWSER_USE_API_KEY',
extra_kwargs={},
httpserver=httpserver,
)
# --- Model validation -------------------------------------------------------
def test_default_model_is_bu_2_0():
chat = ChatBrowserUse(api_key=TEST_API_KEY)
assert chat.model == 'bu-2-0'
assert chat.provider == 'browser-use'
@pytest.mark.parametrize('alias', ['bu-1-0', 'bu-2-0'])
def test_bu_aliases_are_accepted(alias):
chat = ChatBrowserUse(model=alias, api_key=TEST_API_KEY)
assert chat.model == alias
assert chat.name == alias
assert chat.provider == 'browser-use'
def test_bu_latest_normalizes_to_bu_2_0():
chat = ChatBrowserUse(model='bu-latest', api_key=TEST_API_KEY)
assert chat.model == 'bu-2-0'
assert chat.name == 'bu-2-0'
@pytest.mark.parametrize(
'model',
[
'anthropic/claude-sonnet-4-6',
'openai/gpt-5.5',
'google/gemini-3-pro',
'browser-use/bu-30b-a3b-preview',
],
)
def test_provider_prefixed_models_are_accepted(model):
"""Provider-prefixed ids are accepted and forwarded verbatim (the gateway resolves them)."""
chat = ChatBrowserUse(model=model, api_key=TEST_API_KEY)
assert chat.model == model
assert chat.name == model
# Always routes through the browser-use gateway, whatever the target model.
assert chat.provider == 'browser-use'
@pytest.mark.parametrize('model', ['gpt-5', 'claude-sonnet-4-6', 'bu-9-9', 'random-model'])
def test_bare_model_ids_are_rejected(model):
"""Bare ids (no bu-* alias, no provider/ prefix) are rejected."""
with pytest.raises(ValueError, match='Invalid model'):
ChatBrowserUse(model=model, api_key=TEST_API_KEY)
async def test_provider_prefixed_model_forwarded_in_payload(httpserver):
"""The provider-prefixed id must be sent verbatim in the request body."""
httpserver.expect_request('/v1/chat/completions', method='POST').respond_with_json({'completion': 'hello from gateway'})
chat = ChatBrowserUse(
model='anthropic/claude-sonnet-4-6',
api_key=TEST_API_KEY,
base_url=httpserver.url_for('/').rstrip('/'),
)
result = await chat.ainvoke([UserMessage(content='hi')])
assert result.completion == 'hello from gateway'
# Check the posted body.
request, _ = httpserver.log[-1]
body = request.get_json()
assert body['model'] == 'anthropic/claude-sonnet-4-6'
assert body['request_type'] == 'browser_agent'
# --- Agent screenshot auto-config -------------------------------------------
# Both the classic and beta agents auto-config the Claude screenshot size, so both
# must strip the provider prefix for gateway ids.
@pytest.mark.parametrize('agent_path', ['classic', 'beta'])
@pytest.mark.parametrize(
'model,expected_size',
[
# Claude Sonnet via the gateway keeps the auto-config; the prefix must not break detection.
('anthropic/claude-sonnet-4-6', (1400, 850)),
# Non-Claude models keep the default.
('bu-2-0', None),
('openai/gpt-5.5', None),
],
)
def test_claude_sonnet_screenshot_autoconfig_through_gateway(agent_path, model, expected_size):
if agent_path == 'classic':
from browser_use.agent.service import Agent
else:
from browser_use.beta import Agent
llm = ChatBrowserUse(model=model, api_key=TEST_API_KEY)
agent = Agent(task='test', llm=llm)
assert agent.browser_session is not None
assert agent.browser_session.llm_screenshot_size == expected_size
+71
View File
@@ -0,0 +1,71 @@
"""Test Google model button click."""
from browser_use.llm.google.chat import ChatGoogle
from tests.ci.models.model_test_helper import run_model_button_click_test
async def test_google_gemini_3_flash_preview(httpserver):
"""Test Google gemini-3-flash-preview can click a button."""
await run_model_button_click_test(
model_class=ChatGoogle,
model_name='gemini-3-flash-preview',
api_key_env='GOOGLE_API_KEY',
extra_kwargs={},
httpserver=httpserver,
)
def test_x_goog_api_client_header_is_set():
"""Test that the x-goog-api-client header is correctly set in the HTTP options."""
chat = ChatGoogle(model='gemini-flash-latest', api_key='fake')
# Generate the params used for genai.Client
params = chat._get_client_params()
# Extract the header
http_options = params.get('http_options', {})
headers = http_options.get('headers', {})
assert 'x-goog-api-client' in headers, 'x-goog-api-client header missing'
assert 'browser-use/' in headers['x-goog-api-client'], 'browser-use not found in x-goog-api-client header'
def test_x_goog_api_client_header_with_none_http_options():
"""Test setting header when http_options is None."""
chat = ChatGoogle(model='gemini-flash-latest', api_key='fake', http_options=None)
params = chat._get_client_params()
http_opts = params.get('http_options', {})
assert http_opts.get('headers', {}).get('x-goog-api-client', '').startswith('browser-use/')
def test_x_goog_api_client_header_with_pydantic_http_options():
"""Test setting header when http_options is a types.HttpOptions Pydantic model."""
from google.genai import types
pydantic_opts = types.HttpOptions(timeout=30, headers={'custom-header': 'value'})
chat = ChatGoogle(model='gemini-flash-latest', api_key='fake', http_options=pydantic_opts)
params = chat._get_client_params()
http_opts = params.get('http_options', {})
# Verify it extracts and preserves timeout and custom-header
assert http_opts.get('timeout') == 30
assert http_opts.get('headers', {}).get('custom-header') == 'value'
assert http_opts.get('headers', {}).get('x-goog-api-client', '').startswith('browser-use/')
def test_x_goog_api_client_header_with_dict_http_options():
"""Test setting header when http_options is a dictionary (types.HttpOptionsDict)."""
from google.genai import types
dict_opts: types.HttpOptionsDict = {
'timeout': 45,
'headers': {'another-header': 'another-value'},
}
chat = ChatGoogle(model='gemini-flash-latest', api_key='fake', http_options=dict_opts)
params = chat._get_client_params()
http_opts = params.get('http_options', {})
# Verify it preserves dictionary values and appends the tracking header
assert http_opts.get('timeout') == 45
assert http_opts.get('headers', {}).get('another-header') == 'another-value'
assert http_opts.get('headers', {}).get('x-goog-api-client', '').startswith('browser-use/')
+12
View File
@@ -0,0 +1,12 @@
from browser_use.llm.anthropic.chat import ChatAnthropic
from browser_use.llm.models import get_llm_by_name
def test_get_llm_by_name_resolves_anthropic_from_env(monkeypatch):
monkeypatch.setenv('ANTHROPIC_API_KEY', 'anthropic-test-key')
llm = get_llm_by_name('anthropic_claude_sonnet_4_0')
assert isinstance(llm, ChatAnthropic)
assert llm.model == 'claude-sonnet-4-0'
assert llm.api_key == 'anthropic-test-key'
+15
View File
@@ -0,0 +1,15 @@
"""Test OpenAI model button click."""
from browser_use.llm.openai.chat import ChatOpenAI
from tests.ci.models.model_test_helper import run_model_button_click_test
async def test_openai_gpt_4_1_mini(httpserver):
"""Test OpenAI gpt-4.1-mini can click a button."""
await run_model_button_click_test(
model_class=ChatOpenAI,
model_name='gpt-4.1-mini',
api_key_env='OPENAI_API_KEY',
extra_kwargs={},
httpserver=httpserver,
)
@@ -0,0 +1,76 @@
"""
Tests for the SchemaOptimizer to ensure it correctly processes and
optimizes the schemas for agent actions without losing information.
"""
from pydantic import BaseModel
from browser_use.agent.views import AgentOutput
from browser_use.llm.schema import SchemaOptimizer
from browser_use.tools.service import Tools
class ProductInfo(BaseModel):
"""A sample structured output model with multiple fields."""
price: str
title: str
rating: float | None = None
def test_optimizer_preserves_all_fields_in_structured_done_action():
"""
Ensures the SchemaOptimizer does not drop fields from a custom structured
output model when creating the schema for the 'done' action.
This test specifically checks for a bug where fields were being lost
during the optimization process.
"""
# 1. Setup a tools with a custom output model, simulating an Agent
# being created with an `output_model_schema`.
tools = Tools(output_model=ProductInfo)
# 2. Get the dynamically created AgentOutput model, which includes all registered actions.
ActionModel = tools.registry.create_action_model()
agent_output_model = AgentOutput.type_with_custom_actions(ActionModel)
# 3. Run the schema optimizer on the agent's output model.
optimized_schema = SchemaOptimizer.create_optimized_json_schema(agent_output_model)
# 4. Find the 'done' action schema within the optimized output.
# The path is properties -> action -> items -> anyOf -> [schema with 'done'].
done_action_schema = None
actions_schemas = optimized_schema.get('properties', {}).get('action', {}).get('items', {}).get('anyOf', [])
for action_schema in actions_schemas:
if 'done' in action_schema.get('properties', {}):
done_action_schema = action_schema
break
# 5. Assert that the 'done' action schema was successfully found.
assert done_action_schema is not None, "Could not find 'done' action in the optimized schema."
# 6. Navigate to the schema for our custom data model within the 'done' action.
# The path is properties -> done -> properties -> data -> properties.
done_params_schema = done_action_schema.get('properties', {}).get('done', {})
structured_data_schema = done_params_schema.get('properties', {}).get('data', {})
final_properties = structured_data_schema.get('properties', {})
# 7. Assert that the set of fields in the optimized schema matches the original model's fields.
original_fields = set(ProductInfo.model_fields.keys())
optimized_fields = set(final_properties.keys())
assert original_fields == optimized_fields, (
f"Field mismatch between original and optimized structured 'done' action schema.\n"
f'Missing from optimized: {original_fields - optimized_fields}\n'
f'Unexpected in optimized: {optimized_fields - original_fields}'
)
def test_gemini_schema_retains_required_fields():
"""Gemini schema should keep explicit required arrays for mandatory fields."""
schema = SchemaOptimizer.create_gemini_optimized_schema(ProductInfo)
assert 'required' in schema, 'Gemini schema removed required fields.'
required_fields = set(schema['required'])
assert {'price', 'title'}.issubset(required_fields), 'Mandatory fields must stay required for Gemini.'
+569
View File
@@ -0,0 +1,569 @@
from browser_use.browser import BrowserProfile, BrowserSession
class TestUrlAllowlistSecurity:
"""Tests for URL allowlist security bypass prevention and URL allowlist glob pattern matching."""
def test_authentication_bypass_prevention(self):
"""Test that the URL allowlist cannot be bypassed using authentication credentials."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
# Create a context config with a sample allowed domain
browser_profile = BrowserProfile(allowed_domains=['example.com'], headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Security vulnerability test cases
# These should all be detected as malicious despite containing "example.com"
assert watchdog._is_url_allowed('https://example.com:password@malicious.com') is False
assert watchdog._is_url_allowed('https://example.com@malicious.com') is False
assert watchdog._is_url_allowed('https://example.com%20@malicious.com') is False
assert watchdog._is_url_allowed('https://example.com%3A@malicious.com') is False
# Make sure legitimate auth credentials still work
assert watchdog._is_url_allowed('https://user:password@example.com') is True
def test_glob_pattern_matching(self):
"""Test that glob patterns in allowed_domains work correctly."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
# Test *.example.com pattern (should match subdomains and main domain)
browser_profile = BrowserProfile(allowed_domains=['*.example.com'], headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Should match subdomains
assert watchdog._is_url_allowed('https://sub.example.com') is True
assert watchdog._is_url_allowed('https://deep.sub.example.com') is True
# Should also match main domain
assert watchdog._is_url_allowed('https://example.com') is True
# Should not match other domains
assert watchdog._is_url_allowed('https://notexample.com') is False
assert watchdog._is_url_allowed('https://example.org') is False
# Test more complex glob patterns
browser_profile = BrowserProfile(
allowed_domains=[
'*.google.com',
'https://wiki.org',
'https://good.com',
'https://*.test.com',
'chrome://version',
'brave://*',
],
headless=True,
user_data_dir=None,
)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Should match domains ending with google.com
assert watchdog._is_url_allowed('https://google.com') is True
assert watchdog._is_url_allowed('https://www.google.com') is True
assert (
watchdog._is_url_allowed('https://evilgood.com') is False
) # make sure we dont allow *good.com patterns, only *.good.com
# Should match domains starting with wiki
assert watchdog._is_url_allowed('http://wiki.org') is False
assert watchdog._is_url_allowed('https://wiki.org') is True
# Should not match internal domains because scheme was not provided
assert watchdog._is_url_allowed('chrome://google.com') is False
assert watchdog._is_url_allowed('chrome://abc.google.com') is False
# Test browser internal URLs
assert watchdog._is_url_allowed('chrome://settings') is False
assert watchdog._is_url_allowed('chrome://version') is True
assert watchdog._is_url_allowed('chrome-extension://version/') is False
assert watchdog._is_url_allowed('brave://anything/') is True
assert watchdog._is_url_allowed('about:blank') is True
assert watchdog._is_url_allowed('chrome://new-tab-page/') is True
assert watchdog._is_url_allowed('chrome://new-tab-page') is True
# Test security for glob patterns (authentication credentials bypass attempts)
# These should all be detected as malicious despite containing allowed domain patterns
assert watchdog._is_url_allowed('https://allowed.example.com:password@notallowed.com') is False
assert watchdog._is_url_allowed('https://subdomain.example.com@evil.com') is False
assert watchdog._is_url_allowed('https://sub.example.com%20@malicious.org') is False
assert watchdog._is_url_allowed('https://anygoogle.com@evil.org') is False
# Test pattern matching
assert watchdog._is_url_allowed('https://www.test.com') is True
assert watchdog._is_url_allowed('https://www.testx.com') is False
def test_glob_pattern_edge_cases(self):
"""Test edge cases for glob pattern matching to ensure proper behavior."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
# Test with domains containing glob pattern in the middle
browser_profile = BrowserProfile(allowed_domains=['*.google.com', 'https://wiki.org'], headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Verify that 'wiki*' pattern doesn't match domains that merely contain 'wiki' in the middle
assert watchdog._is_url_allowed('https://notawiki.com') is False
assert watchdog._is_url_allowed('https://havewikipages.org') is False
assert watchdog._is_url_allowed('https://my-wiki-site.com') is False
# Verify that '*google.com' doesn't match domains that have 'google' in the middle
assert watchdog._is_url_allowed('https://mygoogle.company.com') is False
# Create context with potentially risky glob pattern that demonstrates security concerns
browser_profile = BrowserProfile(allowed_domains=['*.google.com', '*.google.co.uk'], headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Should match legitimate Google domains
assert watchdog._is_url_allowed('https://www.google.com') is True
assert watchdog._is_url_allowed('https://mail.google.co.uk') is True
# Shouldn't match potentially malicious domains with a similar structure
# This demonstrates why the previous pattern was risky and why it's now rejected
assert watchdog._is_url_allowed('https://www.google.evil.com') is False
def test_automatic_www_subdomain_addition(self):
"""Test that root domains automatically allow www subdomain."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
# Test with simple root domains
browser_profile = BrowserProfile(allowed_domains=['example.com', 'test.org'], headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Root domain should allow itself
assert watchdog._is_url_allowed('https://example.com') is True
assert watchdog._is_url_allowed('https://test.org') is True
# Root domain should automatically allow www subdomain
assert watchdog._is_url_allowed('https://www.example.com') is True
assert watchdog._is_url_allowed('https://www.test.org') is True
# Should not allow other subdomains
assert watchdog._is_url_allowed('https://mail.example.com') is False
assert watchdog._is_url_allowed('https://sub.test.org') is False
# Should not allow unrelated domains
assert watchdog._is_url_allowed('https://notexample.com') is False
assert watchdog._is_url_allowed('https://www.notexample.com') is False
def test_www_subdomain_not_added_for_country_tlds(self):
"""Test www subdomain is NOT automatically added for country-specific TLDs (2+ dots)."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
# Test with country-specific TLDs - these should NOT get automatic www
browser_profile = BrowserProfile(
allowed_domains=['example.co.uk', 'test.com.au', 'site.co.jp'], headless=True, user_data_dir=None
)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Root domains should work exactly as specified
assert watchdog._is_url_allowed('https://example.co.uk') is True
assert watchdog._is_url_allowed('https://test.com.au') is True
assert watchdog._is_url_allowed('https://site.co.jp') is True
# www subdomains should NOT work automatically (user must specify explicitly)
assert watchdog._is_url_allowed('https://www.example.co.uk') is False
assert watchdog._is_url_allowed('https://www.test.com.au') is False
assert watchdog._is_url_allowed('https://www.site.co.jp') is False
# Other subdomains should not work
assert watchdog._is_url_allowed('https://mail.example.co.uk') is False
assert watchdog._is_url_allowed('https://api.test.com.au') is False
def test_www_subdomain_not_added_for_existing_subdomains(self):
"""Test that www is not automatically added for domains that already have subdomains."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
# Test with existing subdomains - should NOT get automatic www
browser_profile = BrowserProfile(allowed_domains=['mail.example.com', 'api.test.org'], headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Exact subdomain should work
assert watchdog._is_url_allowed('https://mail.example.com') is True
assert watchdog._is_url_allowed('https://api.test.org') is True
# www should NOT be automatically added to subdomains
assert watchdog._is_url_allowed('https://www.mail.example.com') is False
assert watchdog._is_url_allowed('https://www.api.test.org') is False
# Root domains should not work either
assert watchdog._is_url_allowed('https://example.com') is False
assert watchdog._is_url_allowed('https://test.org') is False
def test_www_subdomain_not_added_for_wildcard_patterns(self):
"""Test that www is not automatically added for wildcard patterns."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
# Test with wildcard patterns - should NOT get automatic www logic
browser_profile = BrowserProfile(allowed_domains=['*.example.com'], headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Wildcard should match everything including root and www
assert watchdog._is_url_allowed('https://example.com') is True
assert watchdog._is_url_allowed('https://www.example.com') is True
assert watchdog._is_url_allowed('https://mail.example.com') is True
def test_www_subdomain_not_added_for_url_patterns(self):
"""Test that www is not automatically added for full URL patterns."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
# Test with full URL patterns - should NOT get automatic www logic
browser_profile = BrowserProfile(
allowed_domains=['https://example.com', 'http://test.org'], headless=True, user_data_dir=None
)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Exact URL should work
assert watchdog._is_url_allowed('https://example.com/path') is True
assert watchdog._is_url_allowed('http://test.org/page') is True
# www should NOT be automatically added for full URL patterns
assert watchdog._is_url_allowed('https://www.example.com') is False
assert watchdog._is_url_allowed('http://www.test.org') is False
def test_is_root_domain_helper(self):
"""Test the _is_root_domain helper method logic."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
browser_profile = BrowserProfile(allowed_domains=['example.com'], headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Simple root domains (1 dot) - should return True
assert watchdog._is_root_domain('example.com') is True
assert watchdog._is_root_domain('test.org') is True
assert watchdog._is_root_domain('site.net') is True
# Subdomains (more than 1 dot) - should return False
assert watchdog._is_root_domain('www.example.com') is False
assert watchdog._is_root_domain('mail.example.com') is False
assert watchdog._is_root_domain('example.co.uk') is False
assert watchdog._is_root_domain('test.com.au') is False
# Wildcards - should return False
assert watchdog._is_root_domain('*.example.com') is False
assert watchdog._is_root_domain('*example.com') is False
# Full URLs - should return False
assert watchdog._is_root_domain('https://example.com') is False
assert watchdog._is_root_domain('http://test.org') is False
# Invalid domains - should return False
assert watchdog._is_root_domain('example') is False
assert watchdog._is_root_domain('') is False
class TestUrlProhibitlistSecurity:
"""Tests for URL prohibitlist (blocked domains) behavior and matching semantics."""
def test_simple_prohibited_domains(self):
"""Domain-only patterns block exact host and www, but not other subdomains."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
browser_profile = BrowserProfile(prohibited_domains=['example.com', 'test.org'], headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Block exact and www
assert watchdog._is_url_allowed('https://example.com') is False
assert watchdog._is_url_allowed('https://www.example.com') is False
assert watchdog._is_url_allowed('https://test.org') is False
assert watchdog._is_url_allowed('https://www.test.org') is False
# Allow other subdomains when only root is prohibited
assert watchdog._is_url_allowed('https://mail.example.com') is True
assert watchdog._is_url_allowed('https://api.test.org') is True
# Allow unrelated domains
assert watchdog._is_url_allowed('https://notexample.com') is True
def test_glob_pattern_prohibited(self):
"""Wildcard patterns block subdomains and main domain for http/https only."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
browser_profile = BrowserProfile(prohibited_domains=['*.example.com'], headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Block subdomains and main domain
assert watchdog._is_url_allowed('https://example.com') is False
assert watchdog._is_url_allowed('https://www.example.com') is False
assert watchdog._is_url_allowed('https://mail.example.com') is False
# Allow other domains
assert watchdog._is_url_allowed('https://notexample.com') is True
# Wildcard with domain-only should not apply to non-http(s)
assert watchdog._is_url_allowed('chrome://abc.example.com') is True
def test_full_url_prohibited_patterns(self):
"""Full URL patterns block only matching scheme/host/prefix."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
browser_profile = BrowserProfile(prohibited_domains=['https://wiki.org', 'brave://*'], headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Scheme-specific blocking
assert watchdog._is_url_allowed('http://wiki.org') is True
assert watchdog._is_url_allowed('https://wiki.org') is False
assert watchdog._is_url_allowed('https://wiki.org/path') is False
# Internal URL prefix blocking
assert watchdog._is_url_allowed('brave://anything/') is False
assert watchdog._is_url_allowed('chrome://settings') is True
def test_internal_urls_allowed_even_when_prohibited(self):
"""Internal new-tab/blank URLs are always allowed regardless of prohibited list."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
browser_profile = BrowserProfile(prohibited_domains=['*'], headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
assert watchdog._is_url_allowed('about:blank') is True
assert watchdog._is_url_allowed('chrome://new-tab-page/') is True
assert watchdog._is_url_allowed('chrome://new-tab-page') is True
assert watchdog._is_url_allowed('chrome://newtab/') is True
def test_prohibited_ignored_when_allowlist_present(self):
"""When allowlist is set, prohibited list is ignored by design."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
browser_profile = BrowserProfile(
allowed_domains=['*.example.com'],
prohibited_domains=['https://example.com'],
headless=True,
user_data_dir=None,
)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Allowed by allowlist even though exact URL is in prohibited list
assert watchdog._is_url_allowed('https://example.com') is True
assert watchdog._is_url_allowed('https://www.example.com') is True
# Not in allowlist => blocked (prohibited list is not consulted in this mode)
assert watchdog._is_url_allowed('https://api.example.com') is True # wildcard allowlist includes this
# A domain outside the allowlist should be blocked
assert watchdog._is_url_allowed('https://notexample.com') is False
def test_auth_credentials_do_not_cause_false_block(self):
"""Credentials injection with prohibited domain in username should not block unrelated hosts."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
browser_profile = BrowserProfile(prohibited_domains=['example.com'], headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Host is malicious.com, should not be blocked just because username contains example.com
assert watchdog._is_url_allowed('https://example.com:password@malicious.com') is True
assert watchdog._is_url_allowed('https://example.com@malicious.com') is True
assert watchdog._is_url_allowed('https://example.com%20@malicious.com') is True
assert watchdog._is_url_allowed('https://example.com%3A@malicious.com') is True
# Legitimate credentials to a prohibited host should be blocked
assert watchdog._is_url_allowed('https://user:password@example.com') is False
def test_case_insensitive_prohibited_domains(self):
"""Prohibited domain matching should be case-insensitive."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
browser_profile = BrowserProfile(prohibited_domains=['Example.COM'], headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
assert watchdog._is_url_allowed('https://example.com') is False
assert watchdog._is_url_allowed('https://WWW.EXAMPLE.COM') is False
assert watchdog._is_url_allowed('https://mail.example.com') is True
class TestDomainListOptimization:
"""Tests for domain list optimization (set conversion for large lists)."""
def test_small_list_keeps_pattern_support(self):
"""Test that lists < 100 items keep pattern matching support."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
browser_profile = BrowserProfile(
prohibited_domains=['*.google.com', 'x.com', 'facebook.com'], headless=True, user_data_dir=None
)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Should still be a list
assert isinstance(browser_session.browser_profile.prohibited_domains, list)
# Pattern matching should work
assert watchdog._is_url_allowed('https://www.google.com') is False
assert watchdog._is_url_allowed('https://mail.google.com') is False
assert watchdog._is_url_allowed('https://google.com') is False
# Exact matches should work
assert watchdog._is_url_allowed('https://x.com') is False
assert watchdog._is_url_allowed('https://facebook.com') is False
# Other domains should be allowed
assert watchdog._is_url_allowed('https://example.com') is True
def test_large_list_converts_to_set(self):
"""Test that lists >= 100 items are converted to sets."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
# Create a list of 100 domains
large_list = [f'blocked{i}.com' for i in range(100)]
browser_profile = BrowserProfile(prohibited_domains=large_list, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Should be converted to set
assert isinstance(browser_session.browser_profile.prohibited_domains, set)
assert len(browser_session.browser_profile.prohibited_domains) == 100
# Exact matches should work
assert watchdog._is_url_allowed('https://blocked0.com') is False
assert watchdog._is_url_allowed('https://blocked50.com') is False
assert watchdog._is_url_allowed('https://blocked99.com') is False
# Other domains should be allowed
assert watchdog._is_url_allowed('https://example.com') is True
assert watchdog._is_url_allowed('https://blocked100.com') is True # Not in list
def test_www_variant_matching_with_sets(self):
"""Test that www variants are checked in set-based lookups."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
# Create a list with 100 domains (some with www, some without)
large_list = [f'site{i}.com' for i in range(50)] + [f'www.domain{i}.org' for i in range(50)]
browser_profile = BrowserProfile(prohibited_domains=large_list, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Should be converted to set
assert isinstance(browser_session.browser_profile.prohibited_domains, set)
# Test www variant matching for domains without www prefix
assert watchdog._is_url_allowed('https://site0.com') is False
assert watchdog._is_url_allowed('https://www.site0.com') is False # Should also be blocked
# Test www variant matching for domains with www prefix
assert watchdog._is_url_allowed('https://www.domain0.org') is False
assert watchdog._is_url_allowed('https://domain0.org') is False # Should also be blocked
# Test that unrelated domains are allowed
assert watchdog._is_url_allowed('https://example.com') is True
assert watchdog._is_url_allowed('https://www.example.com') is True
def test_allowed_domains_with_sets(self):
"""Test that allowed_domains also works with set optimization."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
# Create a large allowlist
large_list = [f'allowed{i}.com' for i in range(100)]
browser_profile = BrowserProfile(allowed_domains=large_list, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Should be converted to set
assert isinstance(browser_session.browser_profile.allowed_domains, set)
# Allowed domains should work
assert watchdog._is_url_allowed('https://allowed0.com') is True
assert watchdog._is_url_allowed('https://www.allowed0.com') is True
assert watchdog._is_url_allowed('https://allowed99.com') is True
# Other domains should be blocked
assert watchdog._is_url_allowed('https://example.com') is False
assert watchdog._is_url_allowed('https://notallowed.com') is False
def test_manual_set_input(self):
"""Test that users can directly provide a set."""
from bubus import EventBus
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
blocked_set = {f'blocked{i}.com' for i in range(50)}
browser_profile = BrowserProfile(prohibited_domains=blocked_set, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Should remain a set
assert isinstance(browser_session.browser_profile.prohibited_domains, set)
# Should work correctly
assert watchdog._is_url_allowed('https://blocked0.com') is False
assert watchdog._is_url_allowed('https://example.com') is True
@@ -0,0 +1,116 @@
"""Tests for download filename sanitization (GHSA-rv9j-wqjp-2fv4,
GHSA-66xh-g88g-2h8j, GHSA-hpr4-fqgr-xhj9).
`DownloadsWatchdog` historically joined attacker-controlled filenames from CDP
(`Page.downloadWillBegin.suggestedFilename`) and `Content-Disposition` headers
directly into the configured `downloads_path`. Strings like `../../escape.bin`
or `/etc/shadow.bak` would `os.path.join` outside the downloads directory,
writing the fetched bytes (also attacker-controlled — the response body is the
exploit content) to an arbitrary location with the agent's process privileges.
`download_file_from_url` triggers passively for any
`Content-Disposition: attachment` response, so this is reachable from any
visited site — `allowed_domains` does not mitigate it.
The fix funnels every attacker-controlled filename through
`DownloadsWatchdog._sanitize_download_filename`, which keeps only the basename
and rejects pure-traversal names. Each on-disk sink additionally verifies
containment via `os.path.realpath`.
"""
from __future__ import annotations
from pathlib import Path
from browser_use.browser.watchdogs.downloads_watchdog import DownloadsWatchdog
class TestSanitizeDownloadFilename:
def test_strips_relative_traversal(self) -> None:
assert DownloadsWatchdog._sanitize_download_filename('../../etc/passwd') == 'passwd'
def test_strips_absolute_unix_path(self) -> None:
assert DownloadsWatchdog._sanitize_download_filename('/etc/shadow') == 'shadow'
def test_strips_windows_backslash_paths(self) -> None:
assert DownloadsWatchdog._sanitize_download_filename('..\\..\\Windows\\System32\\config.txt') == 'config.txt'
def test_strips_mixed_separators(self) -> None:
assert DownloadsWatchdog._sanitize_download_filename('a/b\\c/../d.pdf') == 'd.pdf'
def test_pure_traversal_falls_back_to_download(self) -> None:
for malicious in ('..', '.', '/', '\\', '../', '..\\', '/.', '\\.', '/..'):
assert DownloadsWatchdog._sanitize_download_filename(malicious) == 'download', (
f'{malicious!r} should fall back to default'
)
def test_null_byte_stripped(self) -> None:
# Null bytes can be used to confuse path handling on some platforms.
assert DownloadsWatchdog._sanitize_download_filename('file.txt\x00.exe') == 'file.txt.exe'
def test_empty_or_none_falls_back(self) -> None:
assert DownloadsWatchdog._sanitize_download_filename('') == 'download'
assert DownloadsWatchdog._sanitize_download_filename(None) == 'download'
def test_normal_filenames_preserved(self) -> None:
# Make sure we don't over-sanitize legitimate names.
assert DownloadsWatchdog._sanitize_download_filename('report.pdf') == 'report.pdf'
assert DownloadsWatchdog._sanitize_download_filename('file with spaces.pdf') == 'file with spaces.pdf'
assert DownloadsWatchdog._sanitize_download_filename('file-with_underscores.csv') == 'file-with_underscores.csv'
# Dotfiles are allowed as long as they're not just dots.
assert DownloadsWatchdog._sanitize_download_filename('.bashrc') == '.bashrc'
def test_unicode_preserved(self) -> None:
# Filenames with non-ASCII characters should survive (common for i18n filenames).
assert DownloadsWatchdog._sanitize_download_filename('résumé.pdf') == 'résumé.pdf'
assert DownloadsWatchdog._sanitize_download_filename('文档.pdf') == '文档.pdf'
class TestIsPathContained:
def test_file_inside_dir_returns_true(self, tmp_path: Path) -> None:
f = tmp_path / 'a.txt'
f.write_text('x')
assert DownloadsWatchdog._is_path_contained(f, tmp_path) is True
def test_nested_file_inside_dir_returns_true(self, tmp_path: Path) -> None:
nested = tmp_path / 'sub' / 'a.txt'
nested.parent.mkdir()
nested.write_text('x')
assert DownloadsWatchdog._is_path_contained(nested, tmp_path) is True
def test_escaping_path_returns_false(self, tmp_path: Path) -> None:
escape = tmp_path / '..' / 'a.txt'
assert DownloadsWatchdog._is_path_contained(escape, tmp_path) is False
def test_dir_itself_returns_true(self, tmp_path: Path) -> None:
assert DownloadsWatchdog._is_path_contained(tmp_path, tmp_path) is True
def test_sibling_dir_returns_false(self, tmp_path: Path) -> None:
sibling = tmp_path.parent / (tmp_path.name + '_sibling')
sibling.mkdir(exist_ok=True)
try:
f = sibling / 'a.txt'
f.write_text('x')
assert DownloadsWatchdog._is_path_contained(f, tmp_path) is False
finally:
f.unlink(missing_ok=True)
sibling.rmdir()
class TestUniqueFilenameOperatesOnSanitizedBasename:
"""`_get_unique_filename` must only ever receive a sanitized basename; if a
traversal string slips through, the (1)/(2) collision-avoidance logic
silently writes outside the intended directory."""
async def test_unique_filename_on_basename_stays_inside_dir(self, tmp_path: Path) -> None:
# Pre-sanitized name — what the caller should always pass.
result = await DownloadsWatchdog._get_unique_filename(str(tmp_path), 'report.pdf')
assert result == 'report.pdf'
# The resolved path lives inside tmp_path.
assert DownloadsWatchdog._is_path_contained(tmp_path / result, tmp_path)
async def test_unique_filename_collision_handling_stays_inside_dir(self, tmp_path: Path) -> None:
(tmp_path / 'report.pdf').write_text('x')
result = await DownloadsWatchdog._get_unique_filename(str(tmp_path), 'report.pdf')
assert result == 'report (1).pdf'
assert DownloadsWatchdog._is_path_contained(tmp_path / result, tmp_path)
+675
View File
@@ -0,0 +1,675 @@
"""
Comprehensive tests for IP address blocking in SecurityWatchdog.
Tests cover IPv4, IPv6, localhost, private networks, edge cases, and interactions
with allowed_domains and prohibited_domains configurations.
"""
from bubus import EventBus
from browser_use.browser import BrowserProfile, BrowserSession
from browser_use.browser.watchdogs.security_watchdog import SecurityWatchdog
class TestIPv4Blocking:
"""Test blocking of IPv4 addresses."""
def test_block_public_ipv4_addresses(self):
"""Test that public IPv4 addresses are blocked when block_ip_addresses=True."""
browser_profile = BrowserProfile(block_ip_addresses=True, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Public IPv4 addresses should be blocked
assert watchdog._is_url_allowed('http://180.1.1.1/supersafe.txt') is False
assert watchdog._is_url_allowed('https://8.8.8.8/') is False
assert watchdog._is_url_allowed('http://1.1.1.1:8080/api') is False
assert watchdog._is_url_allowed('https://142.250.185.46/search') is False
assert watchdog._is_url_allowed('http://93.184.216.34/') is False
def test_block_private_ipv4_networks(self):
"""Test that private network IPv4 addresses are blocked."""
browser_profile = BrowserProfile(block_ip_addresses=True, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Private network ranges (RFC 1918)
assert watchdog._is_url_allowed('http://192.168.1.1/') is False
assert watchdog._is_url_allowed('http://192.168.0.100/admin') is False
assert watchdog._is_url_allowed('http://10.0.0.1/') is False
assert watchdog._is_url_allowed('http://10.255.255.255/') is False
assert watchdog._is_url_allowed('http://172.16.0.1/') is False
assert watchdog._is_url_allowed('http://172.31.255.254/') is False
def test_block_localhost_ipv4(self):
"""Test that localhost IPv4 addresses are blocked."""
browser_profile = BrowserProfile(block_ip_addresses=True, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Localhost/loopback addresses
assert watchdog._is_url_allowed('http://127.0.0.1/') is False
assert watchdog._is_url_allowed('http://127.0.0.1:8080/') is False
assert watchdog._is_url_allowed('https://127.0.0.1:3000/api/test') is False
assert watchdog._is_url_allowed('http://127.1.2.3/') is False # Any 127.x.x.x
def test_block_ipv4_with_ports_and_paths(self):
"""Test that IPv4 addresses with ports and paths are blocked."""
browser_profile = BrowserProfile(block_ip_addresses=True, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# With various ports
assert watchdog._is_url_allowed('http://8.8.8.8:80/') is False
assert watchdog._is_url_allowed('https://8.8.8.8:443/') is False
assert watchdog._is_url_allowed('http://192.168.1.1:8080/') is False
assert watchdog._is_url_allowed('http://10.0.0.1:3000/api') is False
# With paths and query strings
assert watchdog._is_url_allowed('http://1.2.3.4/path/to/resource') is False
assert watchdog._is_url_allowed('http://5.6.7.8/api?key=value') is False
assert watchdog._is_url_allowed('https://9.10.11.12/path/to/file.html#anchor') is False
def test_allow_ipv4_when_blocking_disabled(self):
"""Test that IPv4 addresses are allowed when block_ip_addresses=False (default)."""
browser_profile = BrowserProfile(block_ip_addresses=False, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# All IP addresses should be allowed when blocking is disabled
assert watchdog._is_url_allowed('http://180.1.1.1/supersafe.txt') is True
assert watchdog._is_url_allowed('http://192.168.1.1/') is True
assert watchdog._is_url_allowed('http://127.0.0.1:8080/') is True
assert watchdog._is_url_allowed('http://8.8.8.8/') is True
class TestIPv6Blocking:
"""Test blocking of IPv6 addresses."""
def test_block_ipv6_addresses(self):
"""Test that IPv6 addresses are blocked when block_ip_addresses=True."""
browser_profile = BrowserProfile(block_ip_addresses=True, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Public IPv6 addresses (with brackets as per URL standard)
assert watchdog._is_url_allowed('http://[2001:db8::1]/') is False
assert watchdog._is_url_allowed('https://[2001:4860:4860::8888]/') is False
assert watchdog._is_url_allowed('http://[2606:4700:4700::1111]/path') is False
assert watchdog._is_url_allowed('https://[2001:db8:85a3::8a2e:370:7334]/api') is False
def test_block_ipv6_localhost(self):
"""Test that IPv6 localhost addresses are blocked."""
browser_profile = BrowserProfile(block_ip_addresses=True, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# IPv6 loopback
assert watchdog._is_url_allowed('http://[::1]/') is False
assert watchdog._is_url_allowed('http://[::1]:8080/') is False
assert watchdog._is_url_allowed('https://[::1]:3000/api') is False
assert watchdog._is_url_allowed('http://[0:0:0:0:0:0:0:1]/') is False # Expanded form
def test_block_ipv6_with_ports_and_paths(self):
"""Test that IPv6 addresses with ports and paths are blocked."""
browser_profile = BrowserProfile(block_ip_addresses=True, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# IPv6 with ports
assert watchdog._is_url_allowed('http://[2001:db8::1]:80/') is False
assert watchdog._is_url_allowed('https://[2001:db8::1]:443/') is False
assert watchdog._is_url_allowed('http://[::1]:8080/api') is False
# IPv6 with paths
assert watchdog._is_url_allowed('http://[2001:db8::1]/path/to/resource') is False
assert watchdog._is_url_allowed('https://[2001:db8::1]/api?key=value') is False
def test_allow_ipv6_when_blocking_disabled(self):
"""Test that IPv6 addresses are allowed when block_ip_addresses=False."""
browser_profile = BrowserProfile(block_ip_addresses=False, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# All IPv6 addresses should be allowed
assert watchdog._is_url_allowed('http://[2001:db8::1]/') is True
assert watchdog._is_url_allowed('http://[::1]:8080/') is True
assert watchdog._is_url_allowed('https://[2001:4860:4860::8888]/') is True
class TestDomainNamesStillAllowed:
"""Test that regular domain names are not affected by IP blocking."""
def test_domain_names_allowed_with_ip_blocking(self):
"""Test that domain names continue to work when IP blocking is enabled."""
browser_profile = BrowserProfile(block_ip_addresses=True, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Regular domain names should still be allowed
assert watchdog._is_url_allowed('https://example.com') is True
assert watchdog._is_url_allowed('https://www.google.com') is True
assert watchdog._is_url_allowed('http://subdomain.example.org/path') is True
assert watchdog._is_url_allowed('https://api.github.com/repos') is True
assert watchdog._is_url_allowed('http://localhost/') is True # "localhost" is a domain name, not IP
assert watchdog._is_url_allowed('http://localhost:8080/api') is True
def test_domains_with_numbers_allowed(self):
"""Test that domain names containing numbers are still allowed."""
browser_profile = BrowserProfile(block_ip_addresses=True, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Domains with numbers (but not valid IP addresses)
assert watchdog._is_url_allowed('https://example123.com') is True
assert watchdog._is_url_allowed('https://123example.com') is True
assert watchdog._is_url_allowed('https://server1.example.com') is True
assert watchdog._is_url_allowed('http://web2.site.org') is True
class TestIPBlockingWithAllowedDomains:
"""Test interaction between IP blocking and allowed_domains."""
def test_ip_blocked_even_in_allowed_domains(self):
"""Test that IPs are blocked even if they're in allowed_domains list."""
# Note: It doesn't make sense to add IPs to allowed_domains, but if someone does,
# IP blocking should take precedence
browser_profile = BrowserProfile(
block_ip_addresses=True,
allowed_domains=['example.com', '192.168.1.1'], # IP in allowlist
headless=True,
user_data_dir=None,
)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# IP should be blocked despite being in allowed_domains
assert watchdog._is_url_allowed('http://192.168.1.1/') is False
# Regular domain should work as expected
assert watchdog._is_url_allowed('https://example.com') is True
# Other domains not in allowed_domains should be blocked
assert watchdog._is_url_allowed('https://other.com') is False
def test_allowed_domains_with_ip_blocking_enabled(self):
"""Test that allowed_domains works normally with IP blocking enabled."""
browser_profile = BrowserProfile(
block_ip_addresses=True, allowed_domains=['example.com', '*.google.com'], headless=True, user_data_dir=None
)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Allowed domains should work
assert watchdog._is_url_allowed('https://example.com') is True
assert watchdog._is_url_allowed('https://www.google.com') is True
# Not allowed domains should be blocked
assert watchdog._is_url_allowed('https://other.com') is False
# IPs should be blocked regardless
assert watchdog._is_url_allowed('http://8.8.8.8/') is False
assert watchdog._is_url_allowed('http://192.168.1.1/') is False
class TestIPBlockingWithProhibitedDomains:
"""Test interaction between IP blocking and prohibited_domains."""
def test_ip_blocked_regardless_of_prohibited_domains(self):
"""Test that IPs are blocked when IP blocking is on, independent of prohibited_domains."""
browser_profile = BrowserProfile(
block_ip_addresses=True, prohibited_domains=['example.com'], headless=True, user_data_dir=None
)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# IPs should be blocked due to IP blocking
assert watchdog._is_url_allowed('http://192.168.1.1/') is False
assert watchdog._is_url_allowed('http://8.8.8.8/') is False
# Prohibited domain should be blocked
assert watchdog._is_url_allowed('https://example.com') is False
# Other domains should be allowed
assert watchdog._is_url_allowed('https://other.com') is True
def test_prohibited_domains_without_ip_blocking(self):
"""Test that prohibited_domains works normally when IP blocking is disabled."""
browser_profile = BrowserProfile(
block_ip_addresses=False, prohibited_domains=['example.com', '8.8.8.8'], headless=True, user_data_dir=None
)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Prohibited domain should be blocked
assert watchdog._is_url_allowed('https://example.com') is False
# IP in prohibited list should be blocked (by prohibited_domains, not IP blocking)
assert watchdog._is_url_allowed('http://8.8.8.8/') is False
# Other IPs should be allowed (IP blocking is off)
assert watchdog._is_url_allowed('http://192.168.1.1/') is True
# Other domains should be allowed
assert watchdog._is_url_allowed('https://other.com') is True
class TestEdgeCases:
"""Test edge cases and invalid inputs."""
def test_invalid_urls_handled_gracefully(self):
"""Test that invalid URLs don't cause crashes."""
browser_profile = BrowserProfile(block_ip_addresses=True, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Invalid URLs should return False
assert watchdog._is_url_allowed('not-a-url') is False
assert watchdog._is_url_allowed('') is False
assert watchdog._is_url_allowed('http://') is False
assert watchdog._is_url_allowed('://example.com') is False
def test_internal_browser_urls_allowed(self):
"""Test that internal browser URLs are still allowed with IP blocking."""
browser_profile = BrowserProfile(block_ip_addresses=True, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Internal URLs should always be allowed
assert watchdog._is_url_allowed('about:blank') is True
assert watchdog._is_url_allowed('chrome://new-tab-page/') is True
assert watchdog._is_url_allowed('chrome://new-tab-page') is True
assert watchdog._is_url_allowed('chrome://newtab/') is True
def test_ipv4_lookalike_domains_allowed(self):
"""Test that strings that look like IPs but cannot be resolved as IPs by
the kernel/browser are still treated as domain names and allowed.
Note: short-form IPv4 strings such as `1.2.3` (which Chromium resolves
as `1.2.0.3`) ARE recognized as IPs — see TestNonStandardIPv4Representations.
"""
browser_profile = BrowserProfile(block_ip_addresses=True, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# These look like IPs but cannot resolve as IPv4 in any form the browser accepts:
# - 999.999.999.999: out of range for every octet
# - 1.2.3.4.5: too many octets
assert watchdog._is_url_allowed('http://999.999.999.999/') is True
assert watchdog._is_url_allowed('http://1.2.3.4.5/') is True
def test_different_schemes_with_ips(self):
"""Test that IP blocking works across different URL schemes."""
browser_profile = BrowserProfile(block_ip_addresses=True, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# HTTP and HTTPS
assert watchdog._is_url_allowed('http://192.168.1.1/') is False
assert watchdog._is_url_allowed('https://192.168.1.1/') is False
# FTP (if browser supports it)
assert watchdog._is_url_allowed('ftp://192.168.1.1/') is False
# WebSocket (parsed as regular URL)
assert watchdog._is_url_allowed('ws://192.168.1.1:8080/') is False
assert watchdog._is_url_allowed('wss://192.168.1.1:8080/') is False
class TestIsIPAddressHelper:
"""Test the _is_ip_address helper method directly."""
def test_valid_ipv4_detection(self):
"""Test that valid IPv4 addresses are correctly detected."""
browser_profile = BrowserProfile(block_ip_addresses=True, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Valid IPv4 addresses
assert watchdog._is_ip_address('127.0.0.1') is True
assert watchdog._is_ip_address('192.168.1.1') is True
assert watchdog._is_ip_address('8.8.8.8') is True
assert watchdog._is_ip_address('255.255.255.255') is True
assert watchdog._is_ip_address('0.0.0.0') is True
def test_valid_ipv6_detection(self):
"""Test that valid IPv6 addresses are correctly detected."""
browser_profile = BrowserProfile(block_ip_addresses=True, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Valid IPv6 addresses (without brackets - those are URL-specific)
assert watchdog._is_ip_address('::1') is True
assert watchdog._is_ip_address('2001:db8::1') is True
assert watchdog._is_ip_address('2001:4860:4860::8888') is True
assert watchdog._is_ip_address('fe80::1') is True
assert watchdog._is_ip_address('2001:db8:85a3::8a2e:370:7334') is True
def test_invalid_ip_detection(self):
"""Test that non-IP strings are correctly identified as not IPs."""
browser_profile = BrowserProfile(block_ip_addresses=True, headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Domain names (not IPs)
assert watchdog._is_ip_address('example.com') is False
assert watchdog._is_ip_address('www.google.com') is False
assert watchdog._is_ip_address('localhost') is False
# Invalid IPs (rejected by both ipaddress and inet_aton)
assert watchdog._is_ip_address('999.999.999.999') is False
assert watchdog._is_ip_address('1.2.3.4.5') is False
assert watchdog._is_ip_address('not-an-ip') is False
assert watchdog._is_ip_address('') is False
# Short-form IPv4 strings (1.2.3 == 1.2.0.3) ARE valid IPs that
# browsers/kernel resolve — covered by TestNonStandardIPv4Representations.
# IPs with ports or paths (not valid for the helper - it only checks hostnames)
assert watchdog._is_ip_address('192.168.1.1:8080') is False
assert watchdog._is_ip_address('192.168.1.1/path') is False
class TestDefaultBehavior:
"""Test that default behavior (no IP blocking) is maintained."""
def test_default_block_ip_addresses_is_false(self):
"""Test that block_ip_addresses defaults to False."""
browser_profile = BrowserProfile(headless=True, user_data_dir=None)
# Default should be False
assert browser_profile.block_ip_addresses is False
def test_no_blocking_by_default(self):
"""Test that IPs are not blocked by default."""
browser_profile = BrowserProfile(headless=True, user_data_dir=None)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# All IPs should be allowed by default
assert watchdog._is_url_allowed('http://180.1.1.1/supersafe.txt') is True
assert watchdog._is_url_allowed('http://192.168.1.1/') is True
assert watchdog._is_url_allowed('http://127.0.0.1:8080/') is True
assert watchdog._is_url_allowed('http://[::1]/') is True
assert watchdog._is_url_allowed('https://8.8.8.8/') is True
class TestComplexScenarios:
"""Test complex real-world scenarios."""
def test_mixed_configuration_comprehensive(self):
"""Test a complex configuration with multiple security settings."""
browser_profile = BrowserProfile(
block_ip_addresses=True,
allowed_domains=['example.com', '*.google.com'],
prohibited_domains=['bad.example.com'], # Should be ignored when allowlist is set
headless=True,
user_data_dir=None,
)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Allowed domains should work
assert watchdog._is_url_allowed('https://example.com') is True
assert watchdog._is_url_allowed('https://www.google.com') is True
assert watchdog._is_url_allowed('https://mail.google.com') is True
# IPs should be blocked
assert watchdog._is_url_allowed('http://8.8.8.8/') is False
assert watchdog._is_url_allowed('http://192.168.1.1/') is False
# Domains not in allowlist should be blocked
assert watchdog._is_url_allowed('https://other.com') is False
def test_localhost_development_scenario(self):
"""Test typical local development scenario."""
# Developer wants to block external IPs but allow domain names
browser_profile = BrowserProfile(
block_ip_addresses=True,
headless=True,
user_data_dir=None, # No domain restrictions
)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Domain names should work (including localhost as a name)
assert watchdog._is_url_allowed('http://localhost:3000/') is True
assert watchdog._is_url_allowed('http://localhost:8080/api') is True
# But localhost IP should be blocked
assert watchdog._is_url_allowed('http://127.0.0.1:3000/') is False
# External domains should work
assert watchdog._is_url_allowed('https://api.example.com') is True
# External IPs should be blocked
assert watchdog._is_url_allowed('http://8.8.8.8/') is False
def test_security_hardening_scenario(self):
"""Test maximum security scenario with IP blocking and domain restrictions."""
browser_profile = BrowserProfile(
block_ip_addresses=True,
allowed_domains=['example.com', 'api.example.com'],
headless=True,
user_data_dir=None,
)
browser_session = BrowserSession(browser_profile=browser_profile)
event_bus = EventBus()
watchdog = SecurityWatchdog(browser_session=browser_session, event_bus=event_bus)
# Only specified domains allowed
assert watchdog._is_url_allowed('https://example.com') is True
assert watchdog._is_url_allowed('https://api.example.com') is True
# IPs blocked
assert watchdog._is_url_allowed('http://192.168.1.1/') is False
# Other domains blocked
assert watchdog._is_url_allowed('https://other.com') is False
# Even localhost blocked
assert watchdog._is_url_allowed('http://127.0.0.1/') is False
class TestNonStandardIPv4Representations:
"""Regression tests for GHSA-xrfv-gg9f-wwjp / GHSA-g27c-8gp4-28cv.
`ipaddress.ip_address()` only accepts the canonical dotted-quad form, but
Chromium also resolves decimal, hex, octal, and short-form IPv4 strings.
Without canonicalization, `block_ip_addresses=True` could be bypassed via:
http://2130706433/ → 127.0.0.1 (decimal)
http://0x7f000001/ → 127.0.0.1 (hex)
http://0177.0.0.1/ → 127.0.0.1 (octal)
http://127.1/ → 127.0.0.1 (short-form)
http://127.0.1/ → 127.0.0.1 (short-form)
"""
def _watchdog(self) -> SecurityWatchdog:
profile = BrowserProfile(block_ip_addresses=True, headless=True, user_data_dir=None)
session = BrowserSession(browser_profile=profile)
return SecurityWatchdog(browser_session=session, event_bus=EventBus())
def test_decimal_ipv4_blocked(self):
"""Decimal integer IPv4 strings (browser-accepted) must be blocked."""
watchdog = self._watchdog()
assert watchdog._is_url_allowed('http://2130706433/') is False # 127.0.0.1
assert watchdog._is_url_allowed('http://3232235521/') is False # 192.168.0.1
def test_hex_ipv4_blocked(self):
"""Hex IPv4 strings (browser-accepted) must be blocked."""
watchdog = self._watchdog()
assert watchdog._is_url_allowed('http://0x7f000001/') is False # 127.0.0.1
assert watchdog._is_url_allowed('http://0x7F.0x0.0x0.0x1/') is False
def test_octal_ipv4_blocked(self):
"""Octal IPv4 strings (browser-accepted) must be blocked."""
watchdog = self._watchdog()
assert watchdog._is_url_allowed('http://0177.0.0.1/') is False # 127.0.0.1
def test_short_form_ipv4_blocked(self):
"""Short-form IPv4 strings (browser-accepted) must be blocked."""
watchdog = self._watchdog()
assert watchdog._is_url_allowed('http://127.1/') is False
assert watchdog._is_url_allowed('http://127.0.1/') is False
assert watchdog._is_url_allowed('http://10.1/') is False
def test_lookalike_domains_still_allowed(self):
"""Hostnames that look IP-ish but are not (e.g. embedded labels) must
remain unaffected — only browser-resolvable IP forms should be blocked."""
watchdog = self._watchdog()
# Real domains, not IPs — should not be incorrectly blocked.
assert watchdog._is_url_allowed('http://127.0.0.1.evil.com/') is True
assert watchdog._is_url_allowed('http://2130706433.evil.com/') is True
assert watchdog._is_url_allowed('http://example.com/') is True
def test_non_standard_forms_allowed_when_blocking_disabled(self):
"""Without block_ip_addresses, non-standard IPv4 strings are not blocked."""
profile = BrowserProfile(block_ip_addresses=False, headless=True, user_data_dir=None)
session = BrowserSession(browser_profile=profile)
watchdog = SecurityWatchdog(browser_session=session, event_bus=EventBus())
assert watchdog._is_url_allowed('http://2130706433/') is True
assert watchdog._is_url_allowed('http://0x7f000001/') is True
def test_non_standard_forms_blocked_inside_allowed_domains(self):
"""block_ip_addresses must override allowed_domains for non-standard forms
just as it does for the canonical form."""
profile = BrowserProfile(
block_ip_addresses=True,
allowed_domains=['*'],
headless=True,
user_data_dir=None,
)
session = BrowserSession(browser_profile=profile)
watchdog = SecurityWatchdog(browser_session=session, event_bus=EventBus())
# Even with '*' allowlist, the IP block must still fire.
assert watchdog._is_url_allowed('http://2130706433/') is False
assert watchdog._is_url_allowed('http://0x7f000001/') is False
# Sanity: legitimate domains still allowed.
assert watchdog._is_url_allowed('https://example.com/') is True
def test_malformed_unicode_hostnames_do_not_crash_classifier(self):
"""Hostnames with lone surrogates (e.g. `\\udcff` from URL-decoded
malformed UTF-8) must not crash `_is_ip_address` — `socket.inet_aton`
raises `UnicodeEncodeError` (not `OSError`) for surrogates, and the
classifier must treat that as "not an IP" rather than propagating.
Pre-existing defensive behavior in the original code (caught
`Exception`) — must be preserved when extending with `inet_aton`.
"""
watchdog = self._watchdog()
# Lone surrogates — common in URLs containing percent-encoded malformed UTF-8.
assert watchdog._is_ip_address('\udcff') is False
assert watchdog._is_ip_address('\ud800') is False
assert watchdog._is_ip_address('caf\udce9.local') is False
assert watchdog._is_ip_address('\udcff.example.com') is False
def test_percent_encoded_ipv4_blocked(self):
"""Percent-encoded hostnames that decode to IPs must be blocked.
Chromium percent-decodes the host before resolving — so
`http://%30x7f000001/` decodes to `0x7f000001` → `127.0.0.1`, and
`http://%31%32%37.0.0.1/` decodes to `127.0.0.1`. Without
percent-decoding inside the classifier, the IP block is bypassed
whenever the URL contains any `%`-encoded host bytes.
"""
watchdog = self._watchdog()
# Mixed encoding: %30 = '0', rest literal → '0x7f000001'
assert watchdog._is_url_allowed('http://%30x7f000001/') is False
# Fully encoded canonical form: %31%32%37 = '127'
assert watchdog._is_url_allowed('http://%31%32%37.0.0.1/') is False
# Fully encoded decimal form: → '2130706433'
assert watchdog._is_url_allowed('http://%32%31%33%30%37%30%36%34%33%33/') is False
# Direct classifier checks for the same decoded forms.
assert watchdog._is_ip_address('%30x7f000001') is True
assert watchdog._is_ip_address('%31%32%37.0.0.1') is True
assert watchdog._is_ip_address('%32%31%33%30%37%30%36%34%33%33') is True
def test_malformed_percent_encoding_does_not_crash(self):
"""Hostnames with malformed `%` escapes must not crash the classifier."""
watchdog = self._watchdog()
# `unquote` leaves bad `%`-sequences as-is; we must still treat the
# result as a non-IP rather than blowing up.
assert watchdog._is_ip_address('%') is False
assert watchdog._is_ip_address('%zz') is False
assert watchdog._is_ip_address('%2') is False
def test_unicode_normalized_ipv4_blocked(self):
"""Hostnames using fullwidth, circled, or other Unicode digit variants
that NFKC/IDNA-normalize to ASCII IPv4 literals must be blocked.
WHATWG URL canonicalization maps `127...` (fullwidth digits),
`x7f000001` (fullwidth zero + ASCII hex), and `①②⑦.⓪.⓪.①` (circled
digits) all to `127.0.0.1`. Without NFKC normalization in the
classifier, the new non-standard IPv4 blocking can be bypassed with
any equivalent Unicode digit form.
"""
watchdog = self._watchdog()
# Fullwidth digits (U+FF10..U+FF19) — NFKC → ASCII digits.
assert watchdog._is_url_allowed('http://127.../') is False
# Fullwidth zero + ASCII hex 7f000001.
assert watchdog._is_url_allowed('http://x7f000001/') is False
# Circled digits (U+2460+, U+24EA for zero).
assert watchdog._is_url_allowed('http://①②⑦.⓪.⓪.①/') is False
# Direct classifier checks.
assert watchdog._is_ip_address('127...') is True
assert watchdog._is_ip_address('x7f000001') is True
assert watchdog._is_ip_address('①②⑦.⓪.⓪.①') is True
def test_idn_domains_not_misclassified_as_ip(self):
"""Defense against false positives from the new normalization step:
legitimate IDN domains (Unicode letters / punycode) MUST NOT be
classified as IPs after NFKC."""
watchdog = self._watchdog()
assert watchdog._is_ip_address('café.example') is False
assert watchdog._is_ip_address('xn--caf-dma.example') is False
assert watchdog._is_ip_address('日本.example') is False
assert watchdog._is_ip_address('xn--wgv71a.example') is False
def test_idna_dot_separators_blocked(self):
"""Per RFC 3490 / UTS46, four code points act as label separators in
IDNA processing — `.` (U+002E), `。` (U+3002 IDEOGRAPHIC FULL STOP),
`` (U+FF0E FULLWIDTH FULL STOP), `。` (U+FF61 HALFWIDTH IDEOGRAPHIC
FULL STOP). WHATWG URL parsing maps all four to `.` before resolution,
so `http://127。0。0。1/` etc. reach 127.0.0.1.
NFKC alone is insufficient — it maps U+FF0E → U+002E and U+FF61 →
U+3002, but leaves U+3002 (the most common one) untouched. Classifier
must additionally fold U+3002 and U+FF61 to U+002E before IP parsing.
"""
watchdog = self._watchdog()
# All four dot variants must result in the IP being blocked.
assert watchdog._is_url_allowed('http://127。0。0。1/') is False # U+3002
assert watchdog._is_url_allowed('http://127。0。0。1/') is False # U+FF61
assert watchdog._is_url_allowed('http://127001/') is False # U+FF0E
# Combined with circled-digit normalization.
assert watchdog._is_url_allowed('http://①②⑦。⓪。⓪。①/') is False
# Direct classifier checks.
assert watchdog._is_ip_address('127。0。0。1') is True
assert watchdog._is_ip_address('127。0。0。1') is True
assert watchdog._is_ip_address('127001') is True
@@ -0,0 +1,100 @@
"""Tests for MCP server allowed_domains dispatch (GHSA-vfcm-843v-w6v3).
The MCP `retry_with_browser_use_agent` tool used to default `allowed_domains` to
`[]` when the client omitted the argument. That value was then forwarded to
`BrowserProfile(allowed_domains=[])`, which `SecurityWatchdog` interprets as
"no allowlist configured — allow every URL", silently disabling any
admin-configured restrictions on the underlying profile.
The fix is to default to `None` so the admin-configured profile defaults are
preserved when the client omits the argument.
"""
from typing import Any
import pytest
from browser_use.mcp.server import BrowserUseServer
@pytest.fixture
def server() -> BrowserUseServer:
return BrowserUseServer()
async def test_dispatch_passes_none_when_allowed_domains_omitted(server: BrowserUseServer) -> None:
"""Client omits `allowed_domains` → dispatcher must pass `None` to the agent runner."""
captured: dict[str, Any] = {}
async def recording_stub(**kwargs: Any) -> str:
captured.update(kwargs)
return 'ok'
server._retry_with_browser_use_agent = recording_stub # type: ignore[method-assign]
await server._execute_tool('retry_with_browser_use_agent', {'task': 'noop'})
assert 'allowed_domains' in captured, 'dispatcher must forward the allowed_domains kwarg'
assert captured['allowed_domains'] is None, (
f'Expected None to preserve admin-configured profile defaults; '
f'got {captured["allowed_domains"]!r} which would disable SecurityWatchdog '
f'when passed as BrowserProfile(allowed_domains=...).'
)
async def test_dispatch_forwards_explicit_list(server: BrowserUseServer) -> None:
"""Explicit non-empty list from client must reach the agent runner unchanged."""
captured: dict[str, Any] = {}
async def recording_stub(**kwargs: Any) -> str:
captured.update(kwargs)
return 'ok'
server._retry_with_browser_use_agent = recording_stub # type: ignore[method-assign]
await server._execute_tool(
'retry_with_browser_use_agent',
{'task': 'noop', 'allowed_domains': ['example.test']},
)
assert captured['allowed_domains'] == ['example.test']
async def test_explicit_empty_list_does_not_override_profile_defaults(server: BrowserUseServer) -> None:
"""Defense in depth: even when a client explicitly sends `allowed_domains=[]`,
`_retry_with_browser_use_agent` must NOT wipe profile defaults — empty list is
semantically equivalent to "no override" for the security boundary."""
# Patch the boundary just below the override decision: the BrowserProfile constructor.
# We capture the resolved allowed_domains and short-circuit before LLM/agent setup.
captured_profile_kwargs: dict[str, Any] = {}
from browser_use.mcp import server as server_module
original_browser_profile = server_module.BrowserProfile
class CapturingBrowserProfile:
def __init__(self, **kwargs: Any) -> None:
captured_profile_kwargs.update(kwargs)
raise _StopAgentSetup('captured')
server_module.BrowserProfile = CapturingBrowserProfile # type: ignore[misc]
# Stub out the LLM construction path so we don't need real credentials —
# the override decision happens before BrowserProfile() is called regardless,
# but we need _retry_with_browser_use_agent to reach that point.
try:
with pytest.raises(_StopAgentSetup):
await server._retry_with_browser_use_agent(
task='noop',
allowed_domains=[],
)
finally:
server_module.BrowserProfile = original_browser_profile # type: ignore[misc]
# The profile dict should NOT have allowed_domains overridden to [] —
# either the key is absent (profile defaults apply) or it carries whatever
# the admin-configured default was. Either way: not [].
assert captured_profile_kwargs.get('allowed_domains') != [], (
'Empty list from client wiped profile defaults — SecurityWatchdog would be disabled.'
)
class _StopAgentSetup(Exception):
"""Marker exception used to short-circuit agent construction during testing."""
+83
View File
@@ -0,0 +1,83 @@
"""Test that disable_security flag properly merges --disable-features flags without breaking extensions."""
import tempfile
from browser_use.browser.profile import BrowserProfile
class TestBrowserProfileDisableSecurity:
"""Test disable_security flag behavior."""
def test_disable_security_preserves_extension_features(self):
"""Test that disable_security=True doesn't break extension features by properly merging --disable-features flags."""
# Test with disable_security=False (baseline)
profile_normal = BrowserProfile(disable_security=False, user_data_dir=tempfile.mkdtemp(prefix='test-normal-'))
profile_normal.detect_display_configuration()
args_normal = profile_normal.get_args()
# Test with disable_security=True
profile_security_disabled = BrowserProfile(disable_security=True, user_data_dir=tempfile.mkdtemp(prefix='test-security-'))
profile_security_disabled.detect_display_configuration()
args_security_disabled = profile_security_disabled.get_args()
# Extract disable-features args
def extract_disable_features(args):
for arg in args:
if arg.startswith('--disable-features='):
return set(arg.split('=', 1)[1].split(','))
return set()
features_normal = extract_disable_features(args_normal)
features_security_disabled = extract_disable_features(args_security_disabled)
# Check that extension-related features are preserved
extension_features = {
'ExtensionManifestV2Disabled',
'ExtensionDisableUnsupportedDeveloper',
'ExtensionManifestV2Unsupported',
}
security_features = {'IsolateOrigins', 'site-per-process'}
# Verify that security disabled has both extension and security features
missing_extension_features = extension_features - features_security_disabled
missing_security_features = security_features - features_security_disabled
assert not missing_extension_features, (
f'Missing extension features when disable_security=True: {missing_extension_features}'
)
assert not missing_security_features, f'Missing security features when disable_security=True: {missing_security_features}'
# Verify that security disabled profile has more features than normal (due to added security features)
assert len(features_security_disabled) > len(features_normal), (
'Security disabled profile should have more features than normal profile'
)
# Verify all normal features are preserved in security disabled profile
missing_normal_features = features_normal - features_security_disabled
assert not missing_normal_features, f'Normal features missing from security disabled profile: {missing_normal_features}'
def test_disable_features_flag_deduplication(self):
"""Test that duplicate --disable-features values are properly deduplicated."""
profile = BrowserProfile(
disable_security=True,
user_data_dir=tempfile.mkdtemp(prefix='test-dedup-'),
# Add duplicate features to test deduplication
args=['--disable-features=TestFeature1,TestFeature2', '--disable-features=TestFeature2,TestFeature3'],
)
profile.detect_display_configuration()
args = profile.get_args()
# Extract disable-features args
disable_features_args = [arg for arg in args if arg.startswith('--disable-features=')]
# Should only have one consolidated --disable-features flag
assert len(disable_features_args) == 1, f'Expected 1 disable-features flag, got {len(disable_features_args)}'
features = set(disable_features_args[0].split('=', 1)[1].split(','))
# Should have all test features without duplicates
expected_test_features = {'TestFeature1', 'TestFeature2', 'TestFeature3'}
assert expected_test_features.issubset(features), f'Missing test features: {expected_test_features - features}'
+590
View File
@@ -0,0 +1,590 @@
import pytest
from pydantic import BaseModel, Field
from browser_use.agent.message_manager.service import MessageManager
from browser_use.agent.views import ActionResult, AgentOutput, AgentStepInfo, MessageManagerState
from browser_use.browser.views import BrowserStateSummary
from browser_use.dom.views import SerializedDOMState
from browser_use.filesystem.file_system import FileSystem
from browser_use.llm import SystemMessage, UserMessage
from browser_use.llm.messages import ContentPartTextParam
from browser_use.tools.registry.service import Registry
from browser_use.utils import is_new_tab_page, match_url_with_domain_pattern
class SensitiveParams(BaseModel):
"""Test parameter model for sensitive data testing."""
text: str = Field(description='Text with sensitive data placeholders')
@pytest.fixture
def registry():
return Registry()
@pytest.fixture
def message_manager():
import os
import tempfile
import uuid
base_tmp = tempfile.gettempdir() # e.g., /tmp on Unix
file_system_path = os.path.join(base_tmp, str(uuid.uuid4()))
return MessageManager(
task='Test task',
system_message=SystemMessage(content='System message'),
state=MessageManagerState(),
file_system=FileSystem(file_system_path),
)
def test_replace_sensitive_data_with_missing_keys(registry, caplog):
"""Test that _replace_sensitive_data handles missing keys gracefully"""
# Create a simple Pydantic model with sensitive data placeholders
params = SensitiveParams(text='Please enter <secret>username</secret> and <secret>password</secret>')
# Case 1: All keys present - both placeholders should be replaced
sensitive_data = {'username': 'user123', 'password': 'pass456'}
result = registry._replace_sensitive_data(params, sensitive_data)
assert result.text == 'Please enter user123 and pass456'
assert '<secret>' not in result.text # No secret tags should remain
# Case 2: One key missing - only available key should be replaced
sensitive_data = {'username': 'user123'} # password is missing
result = registry._replace_sensitive_data(params, sensitive_data)
assert result.text == 'Please enter user123 and <secret>password</secret>'
assert 'user123' in result.text
assert '<secret>password</secret>' in result.text # Missing key's tag remains
# Case 3: Multiple keys missing - all tags should be preserved
sensitive_data = {} # both keys missing
result = registry._replace_sensitive_data(params, sensitive_data)
assert result.text == 'Please enter <secret>username</secret> and <secret>password</secret>'
assert '<secret>username</secret>' in result.text
assert '<secret>password</secret>' in result.text
# Case 4: One key empty - empty values are treated as missing
sensitive_data = {'username': 'user123', 'password': ''}
result = registry._replace_sensitive_data(params, sensitive_data)
assert result.text == 'Please enter user123 and <secret>password</secret>'
assert 'user123' in result.text
assert '<secret>password</secret>' in result.text # Empty value's tag remains
def test_simple_domain_specific_sensitive_data(registry, caplog):
"""Test the basic functionality of domain-specific sensitive data replacement"""
# Create a simple Pydantic model with sensitive data placeholders
params = SensitiveParams(text='Please enter <secret>username</secret> and <secret>password</secret>')
# Simple test with directly instantiable values
sensitive_data = {
'example.com': {'username': 'example_user'},
'other_data': 'non_secret_value', # Old format mixed with new
}
# Without a URL, domain-specific secrets should NOT be exposed
result = registry._replace_sensitive_data(params, sensitive_data)
assert result.text == 'Please enter <secret>username</secret> and <secret>password</secret>'
assert '<secret>username</secret>' in result.text # Should NOT be replaced without URL
assert '<secret>password</secret>' in result.text # Password is missing in sensitive_data
assert 'example_user' not in result.text # Domain-specific value should not appear
# Test with a matching URL - domain-specific secrets should be exposed
result = registry._replace_sensitive_data(params, sensitive_data, 'https://example.com/login')
assert result.text == 'Please enter example_user and <secret>password</secret>'
assert 'example_user' in result.text # Should be replaced with matching URL
assert '<secret>password</secret>' in result.text # Password is still missing
assert '<secret>username</secret>' not in result.text # Username tag should be replaced
def test_match_url_with_domain_pattern():
"""Test that the domain pattern matching utility works correctly"""
# Test exact domain matches
assert match_url_with_domain_pattern('https://example.com', 'example.com') is True
assert match_url_with_domain_pattern('http://example.com', 'example.com') is False # Default scheme is now https
assert match_url_with_domain_pattern('https://google.com', 'example.com') is False
# Test subdomain pattern matches
assert match_url_with_domain_pattern('https://sub.example.com', '*.example.com') is True
assert match_url_with_domain_pattern('https://example.com', '*.example.com') is True # Base domain should match too
assert match_url_with_domain_pattern('https://sub.sub.example.com', '*.example.com') is True
assert match_url_with_domain_pattern('https://example.org', '*.example.com') is False
# Test protocol pattern matches
assert match_url_with_domain_pattern('https://example.com', 'http*://example.com') is True
assert match_url_with_domain_pattern('http://example.com', 'http*://example.com') is True
assert match_url_with_domain_pattern('ftp://example.com', 'http*://example.com') is False
# Test explicit http protocol
assert match_url_with_domain_pattern('http://example.com', 'http://example.com') is True
assert match_url_with_domain_pattern('https://example.com', 'http://example.com') is False
# Test Chrome extension pattern
assert match_url_with_domain_pattern('chrome-extension://abcdefghijkl', 'chrome-extension://*') is True
assert match_url_with_domain_pattern('chrome-extension://mnopqrstuvwx', 'chrome-extension://abcdefghijkl') is False
# Test new tab page handling
assert match_url_with_domain_pattern('about:blank', 'example.com') is False
assert match_url_with_domain_pattern('about:blank', '*://*') is False
assert match_url_with_domain_pattern('chrome://new-tab-page/', 'example.com') is False
assert match_url_with_domain_pattern('chrome://new-tab-page/', '*://*') is False
assert match_url_with_domain_pattern('chrome://new-tab-page', 'example.com') is False
assert match_url_with_domain_pattern('chrome://new-tab-page', '*://*') is False
def test_unsafe_domain_patterns():
"""Test that unsafe domain patterns are rejected"""
# These are unsafe patterns that could match too many domains
assert match_url_with_domain_pattern('https://evil.com', '*google.com') is False
assert match_url_with_domain_pattern('https://google.com.evil.com', '*.*.com') is False
assert match_url_with_domain_pattern('https://google.com', '**google.com') is False
assert match_url_with_domain_pattern('https://google.com', 'g*e.com') is False
assert match_url_with_domain_pattern('https://google.com', '*com*') is False
# Test with patterns that have multiple asterisks in different positions
assert match_url_with_domain_pattern('https://subdomain.example.com', '*domain*example*') is False
assert match_url_with_domain_pattern('https://sub.domain.example.com', '*.*.example.com') is False
# Test patterns with wildcards in TLD part
assert match_url_with_domain_pattern('https://example.com', 'example.*') is False
assert match_url_with_domain_pattern('https://example.org', 'example.*') is False
def test_malformed_urls_and_patterns():
"""Test handling of malformed URLs or patterns"""
# Malformed URLs
assert match_url_with_domain_pattern('not-a-url', 'example.com') is False
assert match_url_with_domain_pattern('http://', 'example.com') is False
assert match_url_with_domain_pattern('https://', 'example.com') is False
assert match_url_with_domain_pattern('ftp:/example.com', 'example.com') is False # Missing slash
# Empty URLs or patterns
assert match_url_with_domain_pattern('', 'example.com') is False
assert match_url_with_domain_pattern('https://example.com', '') is False
# URLs with no hostname
assert match_url_with_domain_pattern('file:///path/to/file.txt', 'example.com') is False
# Invalid pattern formats
assert match_url_with_domain_pattern('https://example.com', '..example.com') is False
assert match_url_with_domain_pattern('https://example.com', '.*.example.com') is False
assert match_url_with_domain_pattern('https://example.com', '**') is False
# Nested URL attacks in path, query or fragments
assert match_url_with_domain_pattern('https://example.com/redirect?url=https://evil.com', 'example.com') is True
assert match_url_with_domain_pattern('https://example.com/path/https://evil.com', 'example.com') is True
assert match_url_with_domain_pattern('https://example.com#https://evil.com', 'example.com') is True
# These should match example.com, not evil.com since urlparse extracts the hostname correctly
# Complex URL obfuscation attempts
assert match_url_with_domain_pattern('https://example.com/path?next=//evil.com/attack', 'example.com') is True
assert match_url_with_domain_pattern('https://example.com@evil.com', 'example.com') is False
assert match_url_with_domain_pattern('https://evil.com?example.com', 'example.com') is False
assert match_url_with_domain_pattern('https://user:example.com@evil.com', 'example.com') is False
# urlparse correctly identifies evil.com as the hostname in these cases
def test_url_components():
"""Test handling of URL components like credentials, ports, fragments, etc."""
# URLs with credentials (username:password@)
assert match_url_with_domain_pattern('https://user:pass@example.com', 'example.com') is True
assert match_url_with_domain_pattern('https://user:pass@example.com', '*.example.com') is True
# URLs with ports
assert match_url_with_domain_pattern('https://example.com:8080', 'example.com') is True
assert match_url_with_domain_pattern('https://example.com:8080', 'example.com:8080') is True # Port is stripped from pattern
# URLs with paths
assert match_url_with_domain_pattern('https://example.com/path/to/page', 'example.com') is True
assert (
match_url_with_domain_pattern('https://example.com/path/to/page', 'example.com/path') is False
) # Paths in patterns are not supported
# URLs with query parameters
assert match_url_with_domain_pattern('https://example.com?param=value', 'example.com') is True
# URLs with fragments
assert match_url_with_domain_pattern('https://example.com#section', 'example.com') is True
# URLs with all components
assert match_url_with_domain_pattern('https://user:pass@example.com:8080/path?query=val#fragment', 'example.com') is True
def test_filter_sensitive_data(message_manager):
"""Test that _filter_sensitive_data handles all sensitive data scenarios correctly"""
# Set up a message with sensitive information
message = UserMessage(content='My username is admin and password is secret123')
# Case 1: No sensitive data provided
message_manager.sensitive_data = None
result = message_manager._filter_sensitive_data(message)
assert result.content == 'My username is admin and password is secret123'
# Case 2: All sensitive data is properly replaced
message_manager.sensitive_data = {'username': 'admin', 'password': 'secret123'}
result = message_manager._filter_sensitive_data(message)
assert '<secret>username</secret>' in result.content
assert '<secret>password</secret>' in result.content
# Case 3: Make sure it works with nested content
nested_message = UserMessage(content=[ContentPartTextParam(text='My username is admin and password is secret123')])
result = message_manager._filter_sensitive_data(nested_message)
assert '<secret>username</secret>' in result.content[0].text
assert '<secret>password</secret>' in result.content[0].text
# Case 4: Test with empty values
message_manager.sensitive_data = {'username': 'admin', 'password': ''}
result = message_manager._filter_sensitive_data(message)
assert '<secret>username</secret>' in result.content
# Only username should be replaced since password is empty
# Case 5: Test with domain-specific sensitive data format
message_manager.sensitive_data = {
'example.com': {'username': 'admin', 'password': 'secret123'},
'google.com': {'email': 'user@example.com', 'password': 'google_pass'},
}
# Update the message to include the values we're going to test
message = UserMessage(content='My username is admin, email is user@example.com and password is secret123 or google_pass')
result = message_manager._filter_sensitive_data(message)
# All sensitive values should be replaced regardless of domain
assert '<secret>username</secret>' in result.content
assert '<secret>password</secret>' in result.content
assert '<secret>email</secret>' in result.content
def test_is_new_tab_page():
"""Test is_new_tab_page function"""
# Test about:blank
assert is_new_tab_page('about:blank') is True
# Test chrome://new-tab-page variations
assert is_new_tab_page('chrome://new-tab-page/') is True
assert is_new_tab_page('chrome://new-tab-page') is True
# Test regular URLs
assert is_new_tab_page('https://example.com') is False
assert is_new_tab_page('http://google.com') is False
assert is_new_tab_page('') is False
assert is_new_tab_page('chrome://settings') is False
def test_sensitive_data_filtered_from_action_results():
"""
Test that sensitive data in action results is filtered before being sent to the LLM.
This tests the full flow:
1. Agent outputs actions with <secret>password</secret> placeholder
2. Placeholder gets replaced with real value 'secret_pass123' during action execution
3. Action result contains: "Typed 'secret_pass123' into password field"
4. When state messages are created, the real value should be replaced back to placeholder
5. The LLM should never see the real password value
"""
import os
import tempfile
import uuid
base_tmp = tempfile.gettempdir()
file_system_path = os.path.join(base_tmp, str(uuid.uuid4()))
sensitive_data: dict[str, str | dict[str, str]] = {'username': 'admin_user', 'password': 'secret_pass123'}
message_manager = MessageManager(
task='Login to the website',
system_message=SystemMessage(content='You are a browser automation agent'),
state=MessageManagerState(),
file_system=FileSystem(file_system_path),
sensitive_data=sensitive_data,
)
# Create browser state
dom_state = SerializedDOMState(_root=None, selector_map={})
browser_state = BrowserStateSummary(
dom_state=dom_state,
url='https://example.com/login',
title='Login Page',
tabs=[],
)
# Simulate action result containing sensitive data after placeholder replacement
# This represents what happens after typing a password into a form field
action_results = [
ActionResult(
long_term_memory="Successfully typed 'secret_pass123' into the password field",
error=None,
)
]
# Create model output for step 1
model_output = AgentOutput(
evaluation_previous_goal='Navigated to login page',
memory='On login page, need to enter credentials',
next_goal='Submit login form',
action=[],
)
step_info = AgentStepInfo(step_number=1, max_steps=10)
# Create state messages - this should filter sensitive data
message_manager.create_state_messages(
browser_state_summary=browser_state,
model_output=model_output,
result=action_results,
step_info=step_info,
use_vision=False,
)
# Get messages that would be sent to LLM
messages = message_manager.get_messages()
# Extract all text content from messages
all_text = []
for msg in messages:
if isinstance(msg.content, str):
all_text.append(msg.content)
elif isinstance(msg.content, list):
for part in msg.content:
if isinstance(part, ContentPartTextParam):
all_text.append(part.text)
combined_text = '\n'.join(all_text)
# Verify the bug is fixed: plaintext password should NOT appear in messages
assert 'secret_pass123' not in combined_text, (
'Sensitive data leaked! Real password value found in LLM messages. '
'The _filter_sensitive_data method should replace it with <secret>password</secret>'
)
# Verify the filtered placeholder IS present (proves filtering happened)
assert '<secret>password</secret>' in combined_text, (
'Filtering did not work correctly. Expected <secret>password</secret> placeholder in messages.'
)
def test_sensitive_data_filtered_with_domain_specific_format():
"""Test that domain-specific sensitive data format is also filtered from action results."""
import os
import tempfile
import uuid
base_tmp = tempfile.gettempdir()
file_system_path = os.path.join(base_tmp, str(uuid.uuid4()))
# Use domain-specific format
sensitive_data: dict[str, str | dict[str, str]] = {
'example.com': {'api_key': 'sk-secret-api-key-12345'},
}
message_manager = MessageManager(
task='Use the API',
system_message=SystemMessage(content='You are a browser automation agent'),
state=MessageManagerState(),
file_system=FileSystem(file_system_path),
sensitive_data=sensitive_data,
)
dom_state = SerializedDOMState(_root=None, selector_map={})
browser_state = BrowserStateSummary(
dom_state=dom_state,
url='https://example.com/api',
title='API Page',
tabs=[],
)
# Action result with API key that should be filtered
action_results = [
ActionResult(
long_term_memory="Set API key to 'sk-secret-api-key-12345' in the input field",
error=None,
)
]
model_output = AgentOutput(
evaluation_previous_goal='Opened API settings',
memory='Need to configure API key',
next_goal='Save settings',
action=[],
)
step_info = AgentStepInfo(step_number=1, max_steps=10)
message_manager.create_state_messages(
browser_state_summary=browser_state,
model_output=model_output,
result=action_results,
step_info=step_info,
use_vision=False,
)
messages = message_manager.get_messages()
all_text = []
for msg in messages:
if isinstance(msg.content, str):
all_text.append(msg.content)
elif isinstance(msg.content, list):
for part in msg.content:
if isinstance(part, ContentPartTextParam):
all_text.append(part.text)
combined_text = '\n'.join(all_text)
# API key should be filtered out
assert 'sk-secret-api-key-12345' not in combined_text, 'API key leaked into LLM messages!'
assert '<secret>api_key</secret>' in combined_text, 'API key placeholder not found in messages'
# ─── Tests for password field value redaction in DOM snapshots ────────────────
def _make_dom_node(
tag_name: str,
attributes: dict[str, str],
ax_value: str | None = None,
):
"""Create a minimal EnhancedDOMTreeNode for serializer testing."""
from browser_use.dom.views import (
EnhancedAXNode,
EnhancedAXProperty,
EnhancedDOMTreeNode,
NodeType,
)
ax_node = None
if ax_value is not None:
ax_node = EnhancedAXNode(
ax_node_id='ax-1',
ignored=False,
role='textbox',
name=None,
description=None,
properties=[
EnhancedAXProperty(name='valuetext', value=ax_value),
],
child_ids=None,
)
return EnhancedDOMTreeNode(
node_id=1,
backend_node_id=1,
node_type=NodeType.ELEMENT_NODE,
node_name=tag_name.upper(),
node_value='',
attributes=attributes,
is_scrollable=None,
is_visible=True,
absolute_position=None,
target_id='target-1',
frame_id=None,
session_id=None,
content_document=None,
shadow_root_type=None,
shadow_roots=None,
parent_node=None,
children_nodes=None,
ax_node=ax_node,
snapshot_node=None,
)
def test_password_field_value_excluded_from_dom_snapshot():
"""
Password field values must never appear in DOM snapshots sent to the LLM.
When a user types a password into <input type="password">, the accessibility tree
stores the real value. The serializer extracts AX tree values for all form elements
to show the LLM what's been typed. Without filtering, the password appears in the
DOM text representation sent to the LLM on every subsequent step.
Attack scenario:
1. Agent types password into <input type="password"> via sensitive_data placeholder
2. Next step: DOM snapshot extracts the typed value from the AX tree
3. Password appears in plaintext in the LLM context
4. Prompt injection on a later page can exfiltrate it
"""
from browser_use.dom.serializer.serializer import DOMTreeSerializer
from browser_use.dom.views import DEFAULT_INCLUDE_ATTRIBUTES
secret_password = 'hubble_space_telescope'
node = _make_dom_node(
tag_name='input',
attributes={'type': 'password', 'name': 'password', 'id': 'pw-field'},
ax_value=secret_password,
)
attrs_str = DOMTreeSerializer._build_attributes_string(node, list(DEFAULT_INCLUDE_ATTRIBUTES), '')
assert secret_password not in attrs_str, (
f'Password "{secret_password}" leaked into DOM serialization! '
'Password field values must be excluded from DOM snapshots sent to the LLM.'
)
# The type attribute should still be present so the LLM knows it's a password field
assert 'type=password' in attrs_str, 'Password field type attribute should be preserved'
def test_password_field_value_excluded_even_from_html_attributes():
"""
Even if the DOM attribute 'value' is set (e.g. <input type="password" value="preset">),
the serializer must strip it for password fields.
"""
from browser_use.dom.serializer.serializer import DOMTreeSerializer
from browser_use.dom.views import DEFAULT_INCLUDE_ATTRIBUTES
preset_password = 'hubble_space_telescope'
node = _make_dom_node(
tag_name='input',
attributes={'type': 'password', 'name': 'password', 'value': preset_password},
ax_value=None, # no AX value, but HTML attribute has it
)
attrs_str = DOMTreeSerializer._build_attributes_string(node, list(DEFAULT_INCLUDE_ATTRIBUTES), '')
assert preset_password not in attrs_str, (
f'Preset password "{preset_password}" leaked via HTML value attribute! '
'Password field values must be stripped regardless of source.'
)
def test_text_input_value_preserved():
"""Non-password input values should still be included (backward compatibility)."""
from browser_use.dom.serializer.serializer import DOMTreeSerializer
from browser_use.dom.views import DEFAULT_INCLUDE_ATTRIBUTES
username = 'john.doe@example.com'
node = _make_dom_node(
tag_name='input',
attributes={'type': 'text', 'name': 'username'},
ax_value=username,
)
attrs_str = DOMTreeSerializer._build_attributes_string(node, list(DEFAULT_INCLUDE_ATTRIBUTES), '')
assert username in attrs_str, 'Non-password input values should be preserved in DOM snapshots'
def test_password_field_without_type_attribute():
"""
An input without an explicit type attribute defaults to 'text' — its value
should NOT be stripped. Only explicit type="password" fields are protected.
"""
from browser_use.dom.serializer.serializer import DOMTreeSerializer
from browser_use.dom.views import DEFAULT_INCLUDE_ATTRIBUTES
value = 'some_text_value'
node = _make_dom_node(
tag_name='input',
attributes={'name': 'search'},
ax_value=value,
)
attrs_str = DOMTreeSerializer._build_attributes_string(node, list(DEFAULT_INCLUDE_ATTRIBUTES), '')
assert value in attrs_str, 'Input without type attribute should preserve its value'
@@ -0,0 +1,196 @@
"""Tests for upload_file FileSystem path containment (GHSA-j9hj-92j8-jv9h).
The `upload_file` action used to construct the absolute upload path by joining
`file_system.get_dir()` with the agent-controlled `params.path`. Because
`FileSystem.get_file()` matches by basename (it does `os.path.basename` first),
an agent-controlled path like `../note.md` would:
1. Pass `get_file()` lookup if a file named `note.md` exists in the FileSystem.
2. Be naively joined to `data_dir` producing `data_dir/../note.md`, which
resolves *outside* the FileSystem directory.
3. Be uploaded to the browser as the resolved escaped file.
The fix uses `file_obj.full_name` (the FileSystem-owned basename) for the join
and additionally verifies via `os.path.realpath` that the result is contained
in `data_dir`.
"""
from __future__ import annotations
import os
from typing import Any
import pytest
from browser_use.agent.views import ActionResult
from browser_use.filesystem.file_system import FileSystem
from browser_use.tools.service import Tools
class _StubBrowserSession:
"""Minimal stub exposing only the attributes `upload_file` touches before it
would hand off to CDP. We stop the action just after path resolution.
Per the project rule "use real objects", this stub stands in only for the
browser/CDP boundary that has no bearing on the security property under test
(file path containment); the FileSystem and Tools registry are real.
"""
is_local = True
downloaded_files: list[str] = []
agent_focus_target_id: str | None = None
session_manager: Any = None
cdp_client: Any = None
# Capture the params.path passed in by the action so the test can inspect it.
captured_resolved_path: str | None = None
async def get_current_page_url(self) -> str:
return 'about:blank'
async def get_selector_map(self) -> dict:
# Returning an empty map causes upload_file to bail with an
# "Element ... does not exist" error AFTER it has resolved the path.
return {}
@pytest.fixture
def stub_session() -> _StubBrowserSession:
return _StubBrowserSession()
async def test_traversal_in_agent_path_does_not_escape_filesystem_dir(
tmp_path,
stub_session: _StubBrowserSession,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""GHSA-j9hj-92j8-jv9h: agent-controlled `../note.md` must not cause
the upload path to escape `file_system.get_dir()`."""
fs = FileSystem(base_dir=tmp_path)
# Legitimate FileSystem-owned file.
await fs.write_file('note.md', 'safe content')
data_dir_real = os.path.realpath(str(fs.get_dir()))
# Capture every path that the action checks for existence — the first one
# is the resolved upload path. Pre-fix this is the escape; post-fix it must
# be the inside path.
exists_calls: list[str] = []
real_exists = os.path.exists
def capturing_exists(path: str) -> bool:
exists_calls.append(str(path))
return real_exists(path)
monkeypatch.setattr('browser_use.tools.service.os.path.exists', capturing_exists)
tools = Tools()
result = await tools.registry.execute_action(
'upload_file',
{'index': 0, 'path': '../note.md'},
browser_session=stub_session, # type: ignore[arg-type]
file_system=fs,
available_file_paths=[],
)
# Whatever the eventual ActionResult, the first os.path.exists call inside
# upload_file is on the resolved upload path. It MUST be contained in data_dir.
assert exists_calls, 'upload_file should call os.path.exists on the resolved path'
resolved = os.path.realpath(exists_calls[0])
assert resolved == data_dir_real or resolved.startswith(data_dir_real + os.sep), (
f'Upload path escaped FileSystem directory.\n'
f' data_dir : {data_dir_real}\n'
f' resolved : {resolved}\n'
f' raw : {exists_calls[0]}'
)
# Sanity: the resolved path should point at the FileSystem-owned file.
assert resolved == os.path.realpath(str(fs.get_dir() / 'note.md'))
# The action itself will fail with an "Element does not exist" error from
# the stub selector map — that's fine; we're only checking the path.
assert isinstance(result, ActionResult)
async def test_traversal_with_no_basename_match_still_fails_safely(
tmp_path,
stub_session: _StubBrowserSession,
) -> None:
"""Sanity: if the basename does not match any FileSystem-owned file, the
action must reject with an availability error — never reach the join sink."""
fs = FileSystem(base_dir=tmp_path)
# Note: no `note.md` registered in the FileSystem.
tools = Tools()
result = await tools.registry.execute_action(
'upload_file',
{'index': 0, 'path': '../note.md'},
browser_session=stub_session, # type: ignore[arg-type]
file_system=fs,
available_file_paths=[],
)
assert isinstance(result, ActionResult)
assert result.error is not None
# Should be the "not available" rejection, not a path-resolution success.
assert 'not available' in result.error.lower() or 'does not exist' in result.error.lower()
class _StubRemoteBrowserSession(_StubBrowserSession):
"""Stub for a remote (non-local) browser session — the upload action's
rules differ here. Remote paths are passed through to the browser process."""
is_local = False
async def test_remote_session_does_not_rewrite_to_local_filesystem_on_basename_collision(
tmp_path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""For remote sessions, an absolute remote path that happens to share a
basename with a local FileSystem-managed file MUST be passed through
unchanged. Rewriting to the local FileSystem path would silently upload
the wrong file (the local managed one) instead of the remote-machine file
the agent referenced.
Pre-existing issue surfaced by the GHSA-j9hj-92j8-jv9h review (codex bot).
"""
fs = FileSystem(base_dir=tmp_path)
# Create a local managed file with a basename that collides with the
# remote path the agent will reference.
await fs.write_file('note.md', 'LOCAL content the agent did NOT ask for')
# What the remote-session agent would pass — a path valid on the remote
# machine, NOT in the local FileSystem.
remote_path = '/tmp/note.md'
# Capture every path that the action checks for existence. For remote
# sessions the local-existence check is skipped, so we instead capture
# the resolved path via the same os.path.exists hook used by the local
# test — if the rewrite-to-local-filesystem branch fires, the next call
# will be against the data_dir path; if not, no call should reference it.
exists_calls: list[str] = []
real_exists = os.path.exists
def capturing_exists(path: str) -> bool:
exists_calls.append(str(path))
return real_exists(path)
monkeypatch.setattr('browser_use.tools.service.os.path.exists', capturing_exists)
tools = Tools()
await tools.registry.execute_action(
'upload_file',
{'index': 0, 'path': remote_path},
browser_session=_StubRemoteBrowserSession(), # type: ignore[arg-type]
file_system=fs,
available_file_paths=[],
)
# The action should NOT have rewritten to a local FileSystem path. If it
# had, we'd see the data_dir/note.md path appear in the captured calls.
data_dir_str = str(fs.get_dir())
rewrites = [p for p in exists_calls if p.startswith(data_dir_str)]
assert not rewrites, (
f'Remote session upload silently rewrote to local FileSystem path: {rewrites}. '
f'The agent intended to upload {remote_path!r}; the rewrite would have uploaded the local managed file instead.'
)
+191
View File
@@ -0,0 +1,191 @@
"""Tests for AGI-497: SPA/JS-heavy page renders blank.
Covers:
1. Skeleton screen detection — many elements but near-zero text triggers a warning in page_stats.
2. Navigate reload fallback — empty-body page triggers retry cycle but ultimately succeeds
(no error) because the DOM root exists. Error is only returned when _root is None.
"""
import asyncio
import tempfile
import pytest
from pytest_httpserver import HTTPServer
from browser_use.agent.views import ActionResult
from browser_use.browser import BrowserProfile, BrowserSession
from browser_use.tools.service import Tools
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture(scope='session')
def http_server():
"""Session-scoped HTTP server for blank-page tests."""
server = HTTPServer()
server.start()
# --- skeleton page: 30 empty divs, essentially no text ---
skeleton_html = '<!DOCTYPE html><html><head><title>Loading...</title></head><body>'
skeleton_html += ''.join(f'<div class="skeleton-item skeleton-{i}"></div>' for i in range(30))
skeleton_html += '</body></html>'
server.expect_request('/skeleton').respond_with_data(skeleton_html, content_type='text/html')
# --- rich page: real text content (should NOT be flagged as skeleton) ---
server.expect_request('/products').respond_with_data(
"""<!DOCTYPE html>
<html><head><title>Products</title></head>
<body>
<h1>Product Catalog</h1>
<div class="product"><h2>Widget A</h2><p>Price: $29.99 - A sturdy widget for everyday use.</p></div>
<div class="product"><h2>Widget B</h2><p>Price: $49.99 - Premium widget with extended warranty.</p></div>
<div class="product"><h2>Gadget C</h2><p>Price: $19.50 - Compact gadget, fits in your pocket.</p></div>
<div class="product"><h2>Gadget D</h2><p>Price: $99.00 - Professional-grade gadget for power users.</p></div>
</body>
</html>""",
content_type='text/html',
)
# --- empty body page: body is always empty ---
# Used to test the navigate retry cycle. _root is NOT None (body element exists),
# so navigate retries but ultimately succeeds (no error).
server.expect_request('/always-empty').respond_with_data(
'<html><body></body></html>',
content_type='text/html',
)
yield server
server.stop()
@pytest.fixture(scope='session')
def base_url(http_server):
return f'http://{http_server.host}:{http_server.port}'
@pytest.fixture(scope='module')
async def browser_session():
session = BrowserSession(
browser_profile=BrowserProfile(
headless=True,
user_data_dir=None,
keep_alive=True,
)
)
await session.start()
yield session
await session.kill()
@pytest.fixture(scope='function')
def tools():
return Tools()
# ---------------------------------------------------------------------------
# Helper
# ---------------------------------------------------------------------------
async def _navigate(tools, browser_session, url):
"""Navigate to url and give the page a moment to settle."""
await tools.navigate(url=url, new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
def _make_prompt(state):
"""Build an AgentMessagePrompt from a BrowserStateSummary using a temp dir for FileSystem."""
from browser_use.agent.prompts import AgentMessagePrompt
from browser_use.filesystem.file_system import FileSystem
tmp_dir = tempfile.mkdtemp(prefix='browseruse_test_')
file_system = FileSystem(base_dir=tmp_dir, create_default_files=False)
return AgentMessagePrompt(
browser_state_summary=state,
file_system=file_system,
)
# ---------------------------------------------------------------------------
# Test 1: Skeleton screen detection in _extract_page_statistics()
# ---------------------------------------------------------------------------
class TestSkeletonScreenDetection:
"""_extract_page_statistics() should flag pages with many elements but very little text."""
async def test_skeleton_page_low_text_chars(self, tools, browser_session, base_url):
"""Skeleton page: total_elements > 20 but text_chars < total_elements * 5."""
await _navigate(tools, browser_session, f'{base_url}/skeleton')
state = await browser_session.get_browser_state_summary(include_screenshot=False)
prompt = _make_prompt(state)
page_stats = prompt._extract_page_statistics()
assert page_stats['total_elements'] > 20, f'Expected >20 elements for skeleton page, got {page_stats["total_elements"]}'
assert page_stats['text_chars'] < page_stats['total_elements'] * 5, (
f'Expected text_chars ({page_stats["text_chars"]}) < total_elements*5 '
f'({page_stats["total_elements"] * 5}) for skeleton page'
)
async def test_skeleton_page_description_contains_warning(self, tools, browser_session, base_url):
"""_get_browser_state_description() includes skeleton warning for placeholder pages."""
await _navigate(tools, browser_session, f'{base_url}/skeleton')
state = await browser_session.get_browser_state_summary(include_screenshot=False)
prompt = _make_prompt(state)
description = prompt._get_browser_state_description()
assert 'skeleton' in description.lower() or 'placeholder' in description.lower(), (
f'Expected skeleton/placeholder warning in description, got:\n{description[:500]}'
)
async def test_rich_page_not_flagged_as_skeleton(self, tools, browser_session, base_url):
"""A page with real text content should NOT be flagged as skeleton."""
await _navigate(tools, browser_session, f'{base_url}/products')
state = await browser_session.get_browser_state_summary(include_screenshot=False)
prompt = _make_prompt(state)
page_stats = prompt._extract_page_statistics()
description = prompt._get_browser_state_description()
# Rich page should have substantial text relative to element count
assert page_stats['text_chars'] >= page_stats['total_elements'] * 5, (
f'Rich page should have text_chars ({page_stats["text_chars"]}) >= total_elements*5 '
f'({page_stats["total_elements"] * 5})'
)
# No skeleton warning in description
assert 'skeleton' not in description.lower() and 'placeholder' not in description.lower(), (
'Rich page should NOT produce a skeleton warning'
)
# ---------------------------------------------------------------------------
# Test 2: Navigate reload fallback — empty-body page triggers retry but succeeds
# ---------------------------------------------------------------------------
class TestNavigateReloadFallback:
"""Navigate retries when llm_representation() is empty, but only errors when _root is None."""
async def test_empty_body_page_retries_then_succeeds(self, tools, browser_session, base_url):
"""
Navigating to a page with an empty body triggers the health-check retry cycle
(empty llm_representation) but ultimately succeeds (no error) because the page
HAS a DOM root (_root is not None — the body element exists and is visible).
The error path only fires when _root is None (truly unloadable pages like those
blocked by anti-bot or returning empty HTTP responses), which avoids false positives
on image-only or other non-interactive-but-valid pages.
"""
empty_url = f'{base_url}/always-empty'
# Triggers: health check -> 3s wait -> reload -> 5s wait -> check _root -> not None -> success
result = await tools.navigate(url=empty_url, new_tab=False, browser_session=browser_session)
assert isinstance(result, ActionResult)
# No error — the body IS a valid DOM root, just visually empty.
# Skeleton detection (in AgentMessagePrompt._get_browser_state_description) warns the LLM.
assert result.error is None, f'Expected no error for empty-body page, got: {result.error}'
+404
View File
@@ -0,0 +1,404 @@
"""Tests for action loop detection — behavioral cycle breaking (PR #4)."""
from browser_use.agent.service import Agent
from browser_use.agent.views import (
ActionLoopDetector,
PageFingerprint,
compute_action_hash,
)
from browser_use.llm.messages import UserMessage
from tests.ci.conftest import create_mock_llm
def _get_context_messages(agent: Agent) -> list[str]:
"""Extract text content from the agent's context messages."""
msgs = agent._message_manager.state.history.context_messages
return [m.content for m in msgs if isinstance(m, UserMessage) and isinstance(m.content, str)]
# ─── Action hash normalization tests ─────────────────────────────────────────
def test_search_normalization_ignores_keyword_order():
"""Two searches with the same keywords in different order should produce the same hash."""
h1 = compute_action_hash('search', {'query': 'site:example.com answers votes'})
h2 = compute_action_hash('search', {'query': 'votes answers site:example.com'})
assert h1 == h2
def test_search_normalization_ignores_case():
"""Search normalization is case-insensitive."""
h1 = compute_action_hash('search', {'query': 'Python Tutorial'})
h2 = compute_action_hash('search', {'query': 'python tutorial'})
assert h1 == h2
def test_search_normalization_ignores_punctuation():
"""Search normalization strips punctuation."""
h1 = compute_action_hash('search', {'query': 'site:hinative.com "answers" votes'})
h2 = compute_action_hash('search', {'query': 'site:hinative.com answers, votes'})
assert h1 == h2
def test_search_normalization_deduplicates_tokens():
"""Duplicate tokens in a search query produce the same hash as single tokens."""
h1 = compute_action_hash('search', {'query': 'python python tutorial'})
h2 = compute_action_hash('search', {'query': 'python tutorial'})
assert h1 == h2
def test_search_different_queries_produce_different_hashes():
"""Fundamentally different search queries should NOT match."""
h1 = compute_action_hash('search', {'query': 'python web scraping'})
h2 = compute_action_hash('search', {'query': 'javascript testing framework'})
assert h1 != h2
def test_click_same_index_same_hash():
"""Clicking the same element index produces the same hash."""
h1 = compute_action_hash('click', {'index': 5})
h2 = compute_action_hash('click', {'index': 5})
assert h1 == h2
def test_click_different_index_different_hash():
"""Clicking different element indices produces different hashes."""
h1 = compute_action_hash('click', {'index': 5})
h2 = compute_action_hash('click', {'index': 12})
assert h1 != h2
def test_input_same_element_same_text():
"""Same element + same text = same hash."""
h1 = compute_action_hash('input', {'index': 3, 'text': 'hello world', 'clear': True})
h2 = compute_action_hash('input', {'index': 3, 'text': 'hello world', 'clear': False})
assert h1 == h2 # clear flag doesn't affect hash
def test_input_different_text_different_hash():
"""Same element but different text = different hash."""
h1 = compute_action_hash('input', {'index': 3, 'text': 'hello'})
h2 = compute_action_hash('input', {'index': 3, 'text': 'goodbye'})
assert h1 != h2
def test_navigate_same_url_same_hash():
"""Navigating to the exact same URL produces the same hash."""
h1 = compute_action_hash('navigate', {'url': 'https://example.com/page1'})
h2 = compute_action_hash('navigate', {'url': 'https://example.com/page1'})
assert h1 == h2
def test_navigate_different_paths_different_hash():
"""Navigating to different paths on the same domain produces different hashes — this is genuine exploration."""
h1 = compute_action_hash('navigate', {'url': 'https://example.com/page1'})
h2 = compute_action_hash('navigate', {'url': 'https://example.com/page2'})
assert h1 != h2
def test_navigate_different_domain_different_hash():
"""Navigate to different domains produces different hashes."""
h1 = compute_action_hash('navigate', {'url': 'https://example.com/page1'})
h2 = compute_action_hash('navigate', {'url': 'https://other.com/page1'})
assert h1 != h2
def test_scroll_direction_matters():
"""Scroll up and scroll down are different actions."""
h1 = compute_action_hash('scroll', {'down': True, 'index': None})
h2 = compute_action_hash('scroll', {'down': False, 'index': None})
assert h1 != h2
def test_scroll_different_elements_different_hash():
"""Scrolling different elements produces different hashes."""
h1 = compute_action_hash('scroll', {'down': True, 'index': 5})
h2 = compute_action_hash('scroll', {'down': True, 'index': 10})
assert h1 != h2
def test_scroll_same_element_same_hash():
"""Scrolling the same element in the same direction produces the same hash."""
h1 = compute_action_hash('scroll', {'down': True, 'index': 5})
h2 = compute_action_hash('scroll', {'down': True, 'index': 5})
assert h1 == h2
def test_different_action_types_different_hashes():
"""Different action types always produce different hashes."""
h1 = compute_action_hash('click', {'index': 5})
h2 = compute_action_hash('scroll', {'down': True, 'index': None})
h3 = compute_action_hash('search', {'query': 'test'})
assert len({h1, h2, h3}) == 3
# ─── ActionLoopDetector unit tests ───────────────────────────────────────────
def test_detector_no_nudge_for_diverse_actions():
"""No nudge when actions are all different."""
detector = ActionLoopDetector(window_size=20)
detector.record_action('click', {'index': 1})
detector.record_action('scroll', {'down': True, 'index': None})
detector.record_action('click', {'index': 2})
detector.record_action('search', {'query': 'something'})
detector.record_action('navigate', {'url': 'https://example.com'})
assert detector.get_nudge_message() is None
def test_detector_nudge_at_5_repeats():
"""Nudge triggers at 5 repetitions of the same action."""
detector = ActionLoopDetector(window_size=20)
for _ in range(5):
detector.record_action('search', {'query': 'site:hinative.com answers votes'})
msg = detector.get_nudge_message()
assert msg is not None
assert 'repeated a similar action' in msg
assert '5 times' in msg
def test_detector_no_nudge_at_4_repeats():
"""No nudge at only 4 repetitions (below threshold)."""
detector = ActionLoopDetector(window_size=20)
for _ in range(4):
detector.record_action('search', {'query': 'site:hinative.com answers votes'})
assert detector.get_nudge_message() is None
def test_detector_nudge_escalates_at_8_repeats():
"""Stronger nudge at 8 repetitions."""
detector = ActionLoopDetector(window_size=20)
for _ in range(8):
detector.record_action('search', {'query': 'site:hinative.com answers votes'})
msg = detector.get_nudge_message()
assert msg is not None
assert 'still making progress' in msg
assert '8 times' in msg
def test_detector_nudge_escalates_at_12_repeats():
"""Most urgent nudge at 12 repetitions."""
detector = ActionLoopDetector(window_size=20)
for _ in range(12):
detector.record_action('search', {'query': 'site:hinative.com answers votes'})
msg = detector.get_nudge_message()
assert msg is not None
assert 'making progress with each repetition' in msg
assert '12 times' in msg
def test_detector_critical_message_no_done_directive():
"""Critical nudge should NOT tell the agent to call done — just a gentle heads up."""
detector = ActionLoopDetector(window_size=20)
for _ in range(12):
detector.record_action('search', {'query': 'site:hinative.com answers votes'})
msg = detector.get_nudge_message()
assert msg is not None
assert 'done action' not in msg
assert 'different approach' in msg
def test_detector_first_nudge_no_cannot_complete():
"""First nudge should NOT say task 'cannot be completed' — just raise awareness."""
detector = ActionLoopDetector(window_size=20)
for _ in range(5):
detector.record_action('search', {'query': 'site:hinative.com answers votes'})
msg = detector.get_nudge_message()
assert msg is not None
assert 'cannot be completed' not in msg
assert 'reconsidering your approach' in msg
def test_detector_window_slides():
"""Old actions fall out of the window."""
detector = ActionLoopDetector(window_size=10)
# Fill window with repeated actions
for _ in range(5):
detector.record_action('click', {'index': 7})
assert detector.max_repetition_count == 5
# Push them out with diverse actions
for i in range(10):
detector.record_action('click', {'index': 100 + i})
# The 5 old repeated actions should have been pushed out
assert detector.max_repetition_count < 5
assert detector.get_nudge_message() is None
def test_detector_search_variations_detected_as_same():
"""Minor variations of the same search (the hinative pattern) are detected as repetition."""
detector = ActionLoopDetector(window_size=20)
# These are the kind of variations the agent produces
queries = [
'site:hinative.com answers votes questions',
'site:hinative.com questions answers votes',
'site:hinative.com votes answers questions',
'site:hinative.com questions votes answers',
'site:hinative.com answers questions votes',
]
for q in queries:
detector.record_action('search', {'query': q})
assert detector.max_repetition_count == 5
assert detector.get_nudge_message() is not None
# ─── Page stagnation detection tests ─────────────────────────────────────────
def test_page_stagnation_no_nudge_when_pages_change():
"""No stagnation nudge when page content changes each step."""
detector = ActionLoopDetector(window_size=20)
detector.record_page_state('https://example.com', 'page content 1', 50)
detector.record_page_state('https://example.com', 'page content 2', 55)
detector.record_page_state('https://example.com', 'page content 3', 60)
assert detector.consecutive_stagnant_pages == 0
assert detector.get_nudge_message() is None
def test_page_stagnation_nudge_at_5_identical_pages():
"""Stagnation nudge fires after 5 consecutive identical page states."""
detector = ActionLoopDetector(window_size=20)
# First recording establishes baseline (doesn't count as stagnant)
for _ in range(6):
detector.record_page_state('https://example.com', 'same content', 50)
assert detector.consecutive_stagnant_pages >= 5
msg = detector.get_nudge_message()
assert msg is not None
assert 'page content has not changed' in msg
def test_page_stagnation_no_nudge_at_4_identical_pages():
"""No stagnation nudge at only 4 consecutive identical pages (below threshold)."""
detector = ActionLoopDetector(window_size=20)
# First recording establishes baseline, then 4 stagnant = 5 total recordings
for _ in range(5):
detector.record_page_state('https://example.com', 'same content', 50)
assert detector.consecutive_stagnant_pages == 4
assert detector.get_nudge_message() is None
def test_page_stagnation_resets_on_change():
"""Stagnation counter resets when page content changes."""
detector = ActionLoopDetector(window_size=20)
detector.record_page_state('https://example.com', 'same content', 50)
detector.record_page_state('https://example.com', 'same content', 50)
detector.record_page_state('https://example.com', 'same content', 50)
assert detector.consecutive_stagnant_pages == 2
# Page changes
detector.record_page_state('https://example.com', 'different content', 55)
assert detector.consecutive_stagnant_pages == 0
def test_combined_loop_and_stagnation():
"""Both action loop and page stagnation messages appear together."""
detector = ActionLoopDetector(window_size=20)
# Create action repetition (8 for STRONG LOOP WARNING)
for _ in range(8):
detector.record_action('click', {'index': 7})
# Create page stagnation (need 5 consecutive stagnant)
detector.record_page_state('https://example.com', 'same', 50)
for _ in range(5):
detector.record_page_state('https://example.com', 'same', 50)
msg = detector.get_nudge_message()
assert msg is not None
assert 'still making progress' in msg
assert 'page content has not changed' in msg
# ─── PageFingerprint tests ───────────────────────────────────────────────────
def test_page_fingerprint_same_content_equal():
"""Same content produces equal fingerprints."""
fp1 = PageFingerprint.from_browser_state('https://example.com', 'hello world', 50)
fp2 = PageFingerprint.from_browser_state('https://example.com', 'hello world', 50)
assert fp1 == fp2
def test_page_fingerprint_different_content_not_equal():
"""Different content produces different fingerprints."""
fp1 = PageFingerprint.from_browser_state('https://example.com', 'hello world', 50)
fp2 = PageFingerprint.from_browser_state('https://example.com', 'goodbye world', 50)
assert fp1 != fp2
def test_page_fingerprint_different_url_not_equal():
"""Different URL produces different fingerprint even with same content."""
fp1 = PageFingerprint.from_browser_state('https://example.com', 'hello world', 50)
fp2 = PageFingerprint.from_browser_state('https://other.com', 'hello world', 50)
assert fp1 != fp2
def test_page_fingerprint_different_element_count_not_equal():
"""Different element count produces different fingerprint."""
fp1 = PageFingerprint.from_browser_state('https://example.com', 'hello world', 50)
fp2 = PageFingerprint.from_browser_state('https://example.com', 'hello world', 51)
assert fp1 != fp2
# ─── Agent integration tests ─────────────────────────────────────────────────
async def test_loop_nudge_injected_into_context():
"""Loop detection nudge is injected as a context message in the agent."""
llm = create_mock_llm()
agent = Agent(task='Test task', llm=llm)
# Simulate 5 repeated actions (new threshold)
for _ in range(5):
agent.state.loop_detector.record_action('search', {'query': 'site:example.com answers'})
agent._inject_loop_detection_nudge()
messages = _get_context_messages(agent)
assert len(messages) == 1
assert 'repeated a similar action' in messages[0]
async def test_no_loop_nudge_when_disabled():
"""No loop nudge when loop_detection_enabled is False."""
llm = create_mock_llm()
agent = Agent(
task='Test task',
llm=llm,
loop_detection_enabled=False,
)
# Simulate 8 repeated actions
for _ in range(8):
agent.state.loop_detector.record_action('search', {'query': 'site:example.com answers'})
agent._inject_loop_detection_nudge()
messages = _get_context_messages(agent)
assert len(messages) == 0
async def test_no_loop_nudge_for_diverse_actions():
"""No loop nudge when actions are diverse."""
llm = create_mock_llm()
agent = Agent(task='Test task', llm=llm)
agent.state.loop_detector.record_action('click', {'index': 1})
agent.state.loop_detector.record_action('scroll', {'down': True, 'index': None})
agent.state.loop_detector.record_action('click', {'index': 2})
agent._inject_loop_detection_nudge()
messages = _get_context_messages(agent)
assert len(messages) == 0
async def test_loop_detector_initialized_from_settings():
"""Loop detector window size is set from agent settings."""
llm = create_mock_llm()
agent = Agent(task='Test task', llm=llm, loop_detection_window=30)
assert agent.state.loop_detector.window_size == 30
async def test_loop_detector_default_window_size():
"""Loop detection default window size is 20."""
llm = create_mock_llm()
agent = Agent(task='Test task', llm=llm)
assert agent.settings.loop_detection_enabled is True
assert agent.state.loop_detector.window_size == 20
+152
View File
@@ -0,0 +1,152 @@
"""Tests for the RecordingWatchdog start/stop API and the `browser-use record` CLI command.
The watchdog drives CDP screencast (`Page.startScreencast`/`stopScreencast`) and
`VideoRecorderService` (imageio+ffmpeg) to produce an MP4. These tests exercise
the full stack against a real headless browser.
"""
from __future__ import annotations
import asyncio
from pathlib import Path
from typing import Any
import pytest
try:
import imageio.v2 as iio # type: ignore[import-not-found]
IMAGEIO_AVAILABLE = True
except ImportError:
IMAGEIO_AVAILABLE = False
from browser_use.browser.events import NavigateToUrlEvent
from browser_use.browser.profile import BrowserProfile
from browser_use.browser.session import BrowserSession
pytestmark = pytest.mark.skipif(
not IMAGEIO_AVAILABLE,
reason='Recording requires the [video] extra: pip install "browser-use[video]"',
)
@pytest.fixture
async def browser_session():
session = BrowserSession(browser_profile=BrowserProfile(headless=True))
await session.start()
yield session
await session.kill()
@pytest.fixture
def page_url(httpserver):
httpserver.expect_request('/recpage').respond_with_data(
"""
<html>
<body style='background:#f0f;padding:40px;'>
<h1 id='title'>Recording test</h1>
<p>This content should appear in the captured video.</p>
</body>
</html>
""",
content_type='text/html',
)
return httpserver.url_for('/recpage')
async def _drive_browser_briefly(bs: BrowserSession, url: str, ticks: int = 8) -> None:
"""Navigate + poke the page so screencast emits a few frames."""
await bs.event_bus.dispatch(NavigateToUrlEvent(url=url, new_tab=False))
# Screencast emits frames as the page changes; give it enough time to collect some
for _ in range(ticks):
await asyncio.sleep(0.15)
async def test_start_stop_recording_produces_video(browser_session: BrowserSession, page_url: str, tmp_path: Path):
"""start_recording → activity → stop_recording should write a valid MP4."""
watchdog = browser_session._recording_watchdog
assert watchdog is not None, 'BrowserSession should always attach a RecordingWatchdog'
out_path = tmp_path / 'session.mp4'
assert not watchdog.is_recording
saved = await watchdog.start_recording(out_path)
assert saved == out_path
assert watchdog.is_recording
await _drive_browser_briefly(browser_session, page_url)
final = await watchdog.stop_recording()
assert final == out_path
assert not watchdog.is_recording
assert out_path.exists(), 'recording stop should leave a file on disk'
assert out_path.stat().st_size > 0, 'recorded video must be non-empty'
# Confirm the file is actually a decodable video with at least one frame.
reader: Any = iio.get_reader(str(out_path))
try:
frame: Any = reader.get_next_data()
assert frame is not None and frame.size > 0
finally:
reader.close()
async def test_start_recording_twice_raises(browser_session: BrowserSession, tmp_path: Path):
watchdog = browser_session._recording_watchdog
assert watchdog is not None
await watchdog.start_recording(tmp_path / 'first.mp4')
try:
with pytest.raises(RuntimeError, match='already in progress'):
await watchdog.start_recording(tmp_path / 'second.mp4')
finally:
await watchdog.stop_recording()
async def test_stop_without_start_returns_none(browser_session: BrowserSession):
watchdog = browser_session._recording_watchdog
assert watchdog is not None
assert await watchdog.stop_recording() is None
async def test_on_browser_connected_degrades_gracefully_when_recording_fails(
browser_session: BrowserSession, tmp_path: Path, monkeypatch
):
"""If start_recording() raises (e.g. missing [video] deps), profile-driven recording
must degrade to a warning instead of breaking BrowserSession startup (see PR #4710 review)."""
from browser_use.browser.events import BrowserConnectedEvent
from browser_use.browser.watchdogs import recording_watchdog as rw_mod
watchdog = browser_session._recording_watchdog
assert watchdog is not None
async def fake_start_recording(self: Any, *_args: Any, **_kwargs: Any) -> Path:
raise RuntimeError('simulated missing video deps')
monkeypatch.setattr(rw_mod.RecordingWatchdog, 'start_recording', fake_start_recording)
browser_session.browser_profile.record_video_dir = tmp_path
# Must not raise — watchdog should catch the RuntimeError and just log a warning.
await watchdog.on_BrowserConnectedEvent(BrowserConnectedEvent(cdp_url=browser_session.cdp_url or ''))
assert not watchdog.is_recording
async def test_profile_record_video_dir_still_works(page_url: str, tmp_path: Path):
"""The existing event-driven flow (profile.record_video_dir) must keep working."""
session = BrowserSession(
browser_profile=BrowserProfile(headless=True, record_video_dir=tmp_path),
)
await session.start()
try:
watchdog = session._recording_watchdog
assert watchdog is not None
# on_BrowserConnectedEvent should have auto-started recording via the watchdog
assert watchdog.is_recording, 'profile.record_video_dir should have auto-started recording'
await _drive_browser_briefly(session, page_url)
finally:
await session.kill()
# After kill, BrowserStopEvent should have finalized the video file into tmp_path
videos = list(tmp_path.glob('*.mp4'))
assert videos, f'expected at least one recorded mp4 in {tmp_path}'
assert videos[0].stat().st_size > 0
+384
View File
@@ -0,0 +1,384 @@
import asyncio
import tempfile
import anyio
import pytest
from pytest_httpserver import HTTPServer
from browser_use.agent.views import ActionResult
from browser_use.browser import BrowserSession
from browser_use.browser.profile import BrowserProfile
from browser_use.filesystem.file_system import FileSystem
from browser_use.tools.service import Tools
@pytest.fixture(scope='session')
def http_server():
server = HTTPServer()
server.start()
server.expect_request('/pdf-test').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head><title>PDF Test Page</title></head>
<body>
<h1>Hello PDF</h1>
<p>This page should be saved as a PDF document.</p>
<ul>
<li>Item 1</li>
<li>Item 2</li>
<li>Item 3</li>
</ul>
</body>
</html>
""",
content_type='text/html',
)
server.expect_request('/pdf-styled').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head>
<title>Styled PDF Page</title>
<style>
body { background-color: #f0f0f0; font-family: sans-serif; }
h1 { color: navy; }
.highlight { background-color: yellow; padding: 10px; }
</style>
</head>
<body>
<h1>Styled Content</h1>
<div class="highlight">This has a background color that should appear when print_background=True.</div>
</body>
</html>
""",
content_type='text/html',
)
yield server
server.stop()
@pytest.fixture(scope='session')
def base_url(http_server):
return f'http://{http_server.host}:{http_server.port}'
@pytest.fixture(scope='module')
async def browser_session():
browser_session = BrowserSession(
browser_profile=BrowserProfile(
headless=True,
user_data_dir=None,
keep_alive=True,
)
)
await browser_session.start()
yield browser_session
await browser_session.kill()
@pytest.fixture(scope='function')
def tools():
return Tools()
def _get_attachments(result: ActionResult) -> list[str]:
"""Helper to extract attachments with pyright-safe narrowing."""
assert result.attachments is not None
return result.attachments
class TestSaveAsPdf:
"""Tests for the save_as_pdf action."""
async def test_save_as_pdf_registered(self, tools):
"""save_as_pdf action is in the default action registry."""
assert 'save_as_pdf' in tools.registry.registry.actions
action = tools.registry.registry.actions['save_as_pdf']
assert action.function is not None
assert 'PDF' in action.description
async def test_save_as_pdf_default_filename(self, tools, browser_session, base_url):
"""save_as_pdf with no filename uses the page title."""
await tools.navigate(url=f'{base_url}/pdf-test', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
with tempfile.TemporaryDirectory() as temp_dir:
file_system = FileSystem(temp_dir)
result = await tools.save_as_pdf(browser_session=browser_session, file_system=file_system)
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'Saved page as PDF' in result.extracted_content
attachments = _get_attachments(result)
assert len(attachments) == 1
pdf_path = attachments[0]
assert pdf_path.endswith('.pdf')
assert await anyio.Path(pdf_path).exists()
# Verify it's actually a PDF (starts with %PDF magic bytes)
header = await anyio.Path(pdf_path).read_bytes()
assert header[:5] == b'%PDF-', f'File does not start with PDF magic bytes: {header[:5]!r}'
async def test_save_as_pdf_custom_filename(self, tools, browser_session, base_url):
"""save_as_pdf with a custom filename uses that name."""
await tools.navigate(url=f'{base_url}/pdf-test', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
with tempfile.TemporaryDirectory() as temp_dir:
file_system = FileSystem(temp_dir)
result = await tools.save_as_pdf(
file_name='my-report',
browser_session=browser_session,
file_system=file_system,
)
assert isinstance(result, ActionResult)
attachments = _get_attachments(result)
assert len(attachments) == 1
pdf_path = attachments[0]
assert 'my-report.pdf' in pdf_path
assert await anyio.Path(pdf_path).exists()
async def test_save_as_pdf_custom_filename_with_extension(self, tools, browser_session, base_url):
"""save_as_pdf doesn't double the .pdf extension."""
await tools.navigate(url=f'{base_url}/pdf-test', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
with tempfile.TemporaryDirectory() as temp_dir:
file_system = FileSystem(temp_dir)
result = await tools.save_as_pdf(
file_name='already.pdf',
browser_session=browser_session,
file_system=file_system,
)
assert isinstance(result, ActionResult)
attachments = _get_attachments(result)
pdf_path = attachments[0]
# Should not be "already.pdf.pdf"
assert pdf_path.endswith('already.pdf')
assert not pdf_path.endswith('.pdf.pdf')
async def test_save_as_pdf_duplicate_filename(self, tools, browser_session, base_url):
"""save_as_pdf increments filename when a duplicate exists."""
await tools.navigate(url=f'{base_url}/pdf-test', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
with tempfile.TemporaryDirectory() as temp_dir:
file_system = FileSystem(temp_dir)
# Save first PDF
result1 = await tools.save_as_pdf(
file_name='duplicate',
browser_session=browser_session,
file_system=file_system,
)
attachments1 = _get_attachments(result1)
assert await anyio.Path(attachments1[0]).exists()
assert attachments1[0].endswith('duplicate.pdf')
# Save second PDF with same name
result2 = await tools.save_as_pdf(
file_name='duplicate',
browser_session=browser_session,
file_system=file_system,
)
attachments2 = _get_attachments(result2)
assert await anyio.Path(attachments2[0]).exists()
assert 'duplicate (1).pdf' in attachments2[0]
async def test_save_as_pdf_landscape(self, tools, browser_session, base_url):
"""save_as_pdf with landscape=True produces a valid PDF."""
await tools.navigate(url=f'{base_url}/pdf-test', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
with tempfile.TemporaryDirectory() as temp_dir:
file_system = FileSystem(temp_dir)
result = await tools.save_as_pdf(
file_name='landscape-test',
landscape=True,
browser_session=browser_session,
file_system=file_system,
)
assert isinstance(result, ActionResult)
attachments = _get_attachments(result)
assert await anyio.Path(attachments[0]).exists()
header = await anyio.Path(attachments[0]).read_bytes()
assert header[:5] == b'%PDF-'
async def test_save_as_pdf_a4_format(self, tools, browser_session, base_url):
"""save_as_pdf with paper_format='A4' produces a valid PDF."""
await tools.navigate(url=f'{base_url}/pdf-test', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
with tempfile.TemporaryDirectory() as temp_dir:
file_system = FileSystem(temp_dir)
result = await tools.save_as_pdf(
file_name='a4-test',
paper_format='A4',
browser_session=browser_session,
file_system=file_system,
)
assert isinstance(result, ActionResult)
attachments = _get_attachments(result)
assert await anyio.Path(attachments[0]).exists()
async def test_save_as_pdf_with_background(self, tools, browser_session, base_url):
"""save_as_pdf with print_background=True on a styled page produces a valid PDF."""
await tools.navigate(url=f'{base_url}/pdf-styled', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
with tempfile.TemporaryDirectory() as temp_dir:
file_system = FileSystem(temp_dir)
result = await tools.save_as_pdf(
file_name='styled-with-bg',
print_background=True,
browser_session=browser_session,
file_system=file_system,
)
assert isinstance(result, ActionResult)
attachments = _get_attachments(result)
pdf_path = attachments[0]
assert await anyio.Path(pdf_path).exists()
# Verify file size is non-trivial (has actual rendered content)
stat = await anyio.Path(pdf_path).stat()
assert stat.st_size > 1000, f'PDF seems too small ({stat.st_size} bytes), may be empty'
async def test_save_as_pdf_header_footer_renders_url(self, tools, browser_session, http_server, base_url):
"""display_header_footer=True (the default) prints the page URL into the footer."""
import pypdf
page_url = f'{base_url}/pdf-test'
await tools.navigate(url=page_url, new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
with tempfile.TemporaryDirectory() as temp_dir:
file_system = FileSystem(temp_dir)
result = await tools.save_as_pdf(
file_name='with-metadata',
browser_session=browser_session,
file_system=file_system,
)
pdf_path = _get_attachments(result)[0]
assert await anyio.Path(pdf_path).exists()
reader = pypdf.PdfReader(pdf_path)
text_no_ws = ''.join(reader.pages[0].extract_text().split())
# The footer metadata (page URL) should be embedded in the rendered PDF.
expected = f'{http_server.host}:{http_server.port}/pdf-test'
assert expected in text_no_ws, f'URL footer metadata not found in PDF text: {text_no_ws!r}'
async def test_save_as_pdf_without_header_footer(self, tools, browser_session, http_server, base_url):
"""display_header_footer=False omits the URL/date metadata from the PDF."""
import pypdf
page_url = f'{base_url}/pdf-test'
await tools.navigate(url=page_url, new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
with tempfile.TemporaryDirectory() as temp_dir:
file_system = FileSystem(temp_dir)
result = await tools.save_as_pdf(
file_name='no-metadata',
display_header_footer=False,
browser_session=browser_session,
file_system=file_system,
)
pdf_path = _get_attachments(result)[0]
assert await anyio.Path(pdf_path).exists()
reader = pypdf.PdfReader(pdf_path)
text_no_ws = ''.join(reader.pages[0].extract_text().split())
# Page body never contains the URL, so it must not appear without the footer.
netloc = f'{http_server.host}:{http_server.port}'
assert netloc not in text_no_ws, f'URL leaked into PDF without header/footer: {text_no_ws!r}'
async def test_save_as_pdf_custom_templates(self, tools, browser_session, base_url):
"""Custom header/footer templates are honored over the defaults."""
import pypdf
await tools.navigate(url=f'{base_url}/pdf-test', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
with tempfile.TemporaryDirectory() as temp_dir:
file_system = FileSystem(temp_dir)
result = await tools.save_as_pdf(
file_name='custom-templates',
header_template='<div style="font-size:12px;">CONFIDENTIAL DRAFT</div>',
footer_template='<div style="font-size:12px;"><span class="title"></span></div>',
browser_session=browser_session,
file_system=file_system,
)
pdf_path = _get_attachments(result)[0]
reader = pypdf.PdfReader(pdf_path)
text_no_ws = ''.join(reader.pages[0].extract_text().split())
assert 'CONFIDENTIALDRAFT' in text_no_ws, f'Custom header not found in PDF text: {text_no_ws!r}'
async def test_save_as_pdf_long_url_keeps_page_number(self, tools, browser_session, base_url):
"""A long footer URL truncates instead of pushing the page number off the printable area."""
import pypdf
# Long, unbroken URL — without min-width:0 + ellipsis on the url span this
# overflows the footer and pushes the page count past the page edge.
long_url = f'{base_url}/pdf-test?q={"x" * 400}'
await tools.navigate(url=long_url, new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
with tempfile.TemporaryDirectory() as temp_dir:
file_system = FileSystem(temp_dir)
result = await tools.save_as_pdf(
file_name='long-url',
# Suppress the header date so the page-number check below is unambiguous.
header_template='<span></span>',
browser_session=browser_session,
file_system=file_system,
)
pdf_path = _get_attachments(result)[0]
reader = pypdf.PdfReader(pdf_path)
text_no_ws = ''.join(reader.pages[0].extract_text().split())
# The footer page count must still render despite the very long URL.
assert '1/1' in text_no_ws, f'page number pushed off-page by long URL: {text_no_ws!r}'
async def test_save_as_pdf_param_model_schema(self):
"""SaveAsPdfAction schema exposes the right fields with defaults."""
from browser_use.tools.views import SaveAsPdfAction
schema = SaveAsPdfAction.model_json_schema()
props = schema['properties']
assert 'file_name' in props
assert 'print_background' in props
assert 'landscape' in props
assert 'scale' in props
assert 'paper_format' in props
assert 'display_header_footer' in props
assert 'header_template' in props
assert 'footer_template' in props
# Check defaults
assert props['print_background']['default'] is True
assert props['landscape']['default'] is False
assert props['scale']['default'] == 1.0
assert props['paper_format']['default'] == 'Letter'
# Header/footer metadata is on by default, matching the browser Print dialog
assert props['display_header_footer']['default'] is True
+176
View File
@@ -0,0 +1,176 @@
"""Per-action timeout regression test.
When a CDP WebSocket goes silent (common failure mode with remote / cloud browsers),
action handlers can await event-bus dispatches that never resolve — individual CDP
calls like Page.navigate() have their own timeouts, but the surrounding event
plumbing does not. Without a per-action cap, `tools.act()` hangs indefinitely and
agents never emit a step, producing empty history traces.
This test replaces `registry.execute_action` with a coroutine that sleeps longer
than the per-action cap, then asserts that `tools.act()` returns within the cap
with an ActionResult(error=...) instead of hanging.
"""
import asyncio
import time
from typing import Any
import pytest
from browser_use.agent.views import ActionModel, ActionResult
from browser_use.tools.service import Tools
class _StubActionModel(ActionModel):
"""ActionModel with two arbitrary named slots for tools.act() plumbing tests.
Tests target tools.act() behaviour (timeout wrapping, error handling), not any
registered action — so we declare fixed slots here and stub out execute_action.
"""
hung_action: dict[str, Any] | None = None
fast_action: dict[str, Any] | None = None
@pytest.mark.asyncio
async def test_act_enforces_per_action_timeout_on_hung_handler():
"""tools.act() must return within action_timeout even if the handler hangs."""
tools = Tools()
# Replace the action executor with one that hangs far past the timeout.
sleep_seconds = 30.0
call_count = {'n': 0}
async def _hanging_execute_action(**_kwargs):
call_count['n'] += 1
await asyncio.sleep(sleep_seconds)
return ActionResult(extracted_content='should never be reached')
tools.registry.execute_action = _hanging_execute_action # type: ignore[assignment]
# Build an ActionModel with a single slot — act() iterates model_dump(exclude_unset=True).
action = _StubActionModel(hung_action={'url': 'https://example.com'})
# Use a tight timeout so the test runs in under a second.
action_timeout = 0.5
start = time.monotonic()
result = await tools.act(action=action, browser_session=None, action_timeout=action_timeout) # type: ignore[arg-type]
elapsed = time.monotonic() - start
# Handler got invoked exactly once.
assert call_count['n'] == 1
# Returned well before the sleep would have finished.
assert elapsed < sleep_seconds / 2, f'act() did not honor timeout; took {elapsed:.2f}s'
# And returned close to the timeout itself (with a reasonable grace margin).
assert elapsed < action_timeout + 2.0, f'act() overshot timeout; took {elapsed:.2f}s'
# Returned a proper ActionResult describing the timeout.
assert isinstance(result, ActionResult)
assert result.error is not None
assert 'timed out' in result.error.lower()
assert 'hung_action' in result.error
@pytest.mark.asyncio
async def test_act_passes_through_fast_handler():
"""When the handler finishes fast, act() returns its result unchanged."""
tools = Tools()
async def _fast_execute_action(**_kwargs):
return ActionResult(extracted_content='done')
tools.registry.execute_action = _fast_execute_action # type: ignore[assignment]
action = _StubActionModel(fast_action={'x': 1})
result = await tools.act(action=action, browser_session=None, action_timeout=5.0) # type: ignore[arg-type]
assert isinstance(result, ActionResult)
assert result.error is None
assert result.extracted_content == 'done'
@pytest.mark.asyncio
async def test_act_rejects_invalid_action_timeout_override():
"""An invalid action_timeout override (nan / inf / <=0) must fall back to
the default, not silently defeat the timeout (nan → immediate timeout,
inf → no timeout at all)."""
tools = Tools()
calls = {'n': 0}
async def _fast_execute_action(**_kwargs):
calls['n'] += 1
return ActionResult(extracted_content='done')
tools.registry.execute_action = _fast_execute_action # type: ignore[assignment]
# nan would otherwise produce an immediate TimeoutError; we expect the
# coercion to fall back to the default, so the fast handler runs to
# completion and returns the success result.
action = _StubActionModel(fast_action={'x': 1})
result = await tools.act(action=action, browser_session=None, action_timeout=float('nan')) # type: ignore[arg-type]
assert calls['n'] == 1
assert result.error is None
assert result.extracted_content == 'done'
# inf / non-positive values also fall back cleanly.
for bad in (float('inf'), 0.0, -5.0):
result = await tools.act(action=action, browser_session=None, action_timeout=bad) # type: ignore[arg-type]
assert result.error is None, f'override {bad!r} should have fallen back'
def test_default_action_timeout_accommodates_extract_action():
"""The module-level default must sit above extract's 120s LLM inner cap."""
from browser_use.tools.service import _DEFAULT_ACTION_TIMEOUT_S
# extract action uses page_extraction_llm.ainvoke(..., timeout=120.0); the
# outer per-action cap must not truncate it.
assert _DEFAULT_ACTION_TIMEOUT_S >= 150.0, (
f'Default action cap ({_DEFAULT_ACTION_TIMEOUT_S}s) is below the 120s '
f'extract timeout + grace — slow but valid extractions would be killed.'
)
@pytest.fixture
def _restore_service_module():
"""Reload browser_use.tools.service without any env override on teardown.
Tests in this file intentionally reload the module with BROWSER_USE_ACTION_TIMEOUT_S
set to various values; without this fixture, the last reload's default leaks into
every later test in the same worker.
"""
import importlib
import os
import browser_use.tools.service as svc_module
yield svc_module
os.environ.pop('BROWSER_USE_ACTION_TIMEOUT_S', None)
importlib.reload(svc_module)
def test_malformed_env_timeout_does_not_break_import(monkeypatch, _restore_service_module):
"""Bad BROWSER_USE_ACTION_TIMEOUT_S values must fall back, not crash or misbehave.
Covers three failure modes:
- Non-numeric / empty (ValueError from float()): would crash module import.
- NaN: parses fine but makes asyncio.wait_for time out immediately for every action.
- Infinity / negative / zero: parses fine but effectively disables the hang guard.
"""
import importlib
svc_module = _restore_service_module
bad_values = ('', 'not-a-number', 'abc', 'nan', 'NaN', 'inf', '-inf', '0', '-5')
for bad_value in bad_values:
monkeypatch.setenv('BROWSER_USE_ACTION_TIMEOUT_S', bad_value)
reloaded = importlib.reload(svc_module)
assert reloaded._DEFAULT_ACTION_TIMEOUT_S == 180.0, (
f'Expected fallback 180.0 for bad env {bad_value!r}, got {reloaded._DEFAULT_ACTION_TIMEOUT_S}'
)
# Valid finite positive values still take effect.
monkeypatch.setenv('BROWSER_USE_ACTION_TIMEOUT_S', '45')
reloaded = importlib.reload(svc_module)
assert reloaded._DEFAULT_ACTION_TIMEOUT_S == 45.0
+387
View File
@@ -0,0 +1,387 @@
"""Tests for inline task planning feature.
Covers: plan generation, step advancement, replanning, rendering,
disabled planning, replan nudge, flash mode schema, and edge cases.
"""
import json
from browser_use.agent.views import (
AgentOutput,
PlanItem,
)
from browser_use.tools.service import Tools
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _make_agent_output(**overrides) -> AgentOutput:
"""Build a minimal AgentOutput with plan fields."""
tools = Tools()
ActionModel = tools.registry.create_action_model()
OutputType = AgentOutput.type_with_custom_actions(ActionModel)
action_json = json.dumps(
{
'evaluation_previous_goal': 'Success',
'memory': 'mem',
'next_goal': 'goal',
**{k: v for k, v in overrides.items() if k in ('current_plan_item', 'plan_update')},
'action': [{'done': {'text': 'ok', 'success': True}}],
}
)
return OutputType.model_validate_json(action_json)
def _make_agent(browser_session, mock_llm, **kwargs):
"""Create an Agent with defaults suitable for unit tests."""
from browser_use import Agent
return Agent(task='Test task', llm=mock_llm, browser_session=browser_session, **kwargs)
# ---------------------------------------------------------------------------
# 1. Plan generation from plan_update on step 1
# ---------------------------------------------------------------------------
async def test_plan_generation_from_plan_update(browser_session, mock_llm):
agent = _make_agent(browser_session, mock_llm)
output = _make_agent_output(plan_update=['Navigate to page', 'Search for item', 'Extract price'])
agent._update_plan_from_model_output(output)
assert agent.state.plan is not None
assert len(agent.state.plan) == 3
assert agent.state.plan[0].status == 'current'
assert agent.state.plan[1].status == 'pending'
assert agent.state.plan[2].status == 'pending'
assert agent.state.current_plan_item_index == 0
assert agent.state.plan_generation_step == agent.state.n_steps
# ---------------------------------------------------------------------------
# 2. Plan step advancement via current_plan_item
# ---------------------------------------------------------------------------
async def test_plan_step_advancement(browser_session, mock_llm):
agent = _make_agent(browser_session, mock_llm)
# Seed a plan
agent.state.plan = [
PlanItem(text='Step A', status='current'),
PlanItem(text='Step B'),
PlanItem(text='Step C'),
]
agent.state.current_plan_item_index = 0
output = _make_agent_output(current_plan_item=2)
agent._update_plan_from_model_output(output)
assert agent.state.plan[0].status == 'done'
assert agent.state.plan[1].status == 'done'
assert agent.state.plan[2].status == 'current'
assert agent.state.current_plan_item_index == 2
# ---------------------------------------------------------------------------
# 3. Replanning replaces old plan
# ---------------------------------------------------------------------------
async def test_replanning_replaces_old_plan(browser_session, mock_llm):
agent = _make_agent(browser_session, mock_llm)
agent.state.plan = [
PlanItem(text='Old step 1', status='done'),
PlanItem(text='Old step 2', status='current'),
]
agent.state.current_plan_item_index = 1
agent.state.plan_generation_step = 1
output = _make_agent_output(plan_update=['New step A', 'New step B', 'New step C'])
agent._update_plan_from_model_output(output)
assert len(agent.state.plan) == 3
assert agent.state.plan[0].text == 'New step A'
assert agent.state.plan[0].status == 'current'
assert agent.state.current_plan_item_index == 0
# ---------------------------------------------------------------------------
# 4. _render_plan_description output format
# ---------------------------------------------------------------------------
async def test_render_plan_description(browser_session, mock_llm):
agent = _make_agent(browser_session, mock_llm)
agent.state.plan = [
PlanItem(text='Navigate to search page', status='done'),
PlanItem(text='Search for "laptop"', status='current'),
PlanItem(text='Extract price from results', status='pending'),
PlanItem(text='Skipped step', status='skipped'),
]
result = agent._render_plan_description()
assert result is not None
lines = result.split('\n')
assert lines[0] == '[x] 0: Navigate to search page'
assert lines[1] == '[>] 1: Search for "laptop"'
assert lines[2] == '[ ] 2: Extract price from results'
assert lines[3] == '[-] 3: Skipped step'
# ---------------------------------------------------------------------------
# 5. Planning disabled returns None
# ---------------------------------------------------------------------------
async def test_planning_disabled_returns_none(browser_session, mock_llm):
agent = _make_agent(browser_session, mock_llm, enable_planning=False)
agent.state.plan = [PlanItem(text='Should not render')]
assert agent._render_plan_description() is None
# Also verify update is a no-op
output = _make_agent_output(plan_update=['New plan'])
agent._update_plan_from_model_output(output)
# Plan should remain unchanged (the method returns early)
assert agent.state.plan[0].text == 'Should not render'
# ---------------------------------------------------------------------------
# 6. Replan nudge injection at threshold
# ---------------------------------------------------------------------------
async def test_replan_nudge_injected_at_threshold(browser_session, mock_llm):
agent = _make_agent(browser_session, mock_llm, planning_replan_on_stall=3)
agent.state.plan = [PlanItem(text='Step 1', status='current')]
agent.state.consecutive_failures = 3
# Track context messages
initial_count = len(agent._message_manager.state.history.context_messages)
agent._inject_replan_nudge()
after_count = len(agent._message_manager.state.history.context_messages)
assert after_count == initial_count + 1
msg = agent._message_manager.state.history.context_messages[-1]
assert isinstance(msg.content, str) and 'REPLAN SUGGESTED' in msg.content
# ---------------------------------------------------------------------------
# 7. No nudge below threshold
# ---------------------------------------------------------------------------
async def test_no_replan_nudge_below_threshold(browser_session, mock_llm):
agent = _make_agent(browser_session, mock_llm, planning_replan_on_stall=3)
agent.state.plan = [PlanItem(text='Step 1', status='current')]
agent.state.consecutive_failures = 2
initial_count = len(agent._message_manager.state.history.context_messages)
agent._inject_replan_nudge()
after_count = len(agent._message_manager.state.history.context_messages)
assert after_count == initial_count
# ---------------------------------------------------------------------------
# 8. Flash mode schema excludes plan fields
# ---------------------------------------------------------------------------
async def test_flash_mode_schema_excludes_plan_fields():
tools = Tools()
ActionModel = tools.registry.create_action_model()
FlashOutput = AgentOutput.type_with_custom_actions_flash_mode(ActionModel)
schema = FlashOutput.model_json_schema()
assert 'current_plan_item' not in schema['properties']
assert 'plan_update' not in schema['properties']
assert 'thinking' not in schema['properties']
# ---------------------------------------------------------------------------
# 9. Full mode schema includes plan fields as optional
# ---------------------------------------------------------------------------
async def test_full_mode_schema_includes_plan_fields_optional():
tools = Tools()
ActionModel = tools.registry.create_action_model()
FullOutput = AgentOutput.type_with_custom_actions(ActionModel)
schema = FullOutput.model_json_schema()
assert 'current_plan_item' in schema['properties']
assert 'plan_update' in schema['properties']
# They should NOT be in required
assert 'current_plan_item' not in schema.get('required', [])
assert 'plan_update' not in schema.get('required', [])
# ---------------------------------------------------------------------------
# 10. Out-of-bounds current_plan_item handled gracefully
# ---------------------------------------------------------------------------
async def test_out_of_bounds_plan_step_clamped(browser_session, mock_llm):
agent = _make_agent(browser_session, mock_llm)
agent.state.plan = [
PlanItem(text='Step A', status='current'),
PlanItem(text='Step B'),
]
agent.state.current_plan_item_index = 0
# Way out of bounds high
output = _make_agent_output(current_plan_item=999)
agent._update_plan_from_model_output(output)
assert agent.state.current_plan_item_index == 1 # clamped to last valid index
assert agent.state.plan[0].status == 'done'
assert agent.state.plan[1].status == 'current'
# Negative index
agent.state.plan = [
PlanItem(text='Step X', status='current'),
PlanItem(text='Step Y'),
]
agent.state.current_plan_item_index = 1
output2 = _make_agent_output(current_plan_item=-5)
agent._update_plan_from_model_output(output2)
assert agent.state.current_plan_item_index == 0 # clamped to 0
assert agent.state.plan[0].status == 'current'
# ---------------------------------------------------------------------------
# 11. No plan means render returns None
# ---------------------------------------------------------------------------
async def test_no_plan_render_returns_none(browser_session, mock_llm):
agent = _make_agent(browser_session, mock_llm)
assert agent.state.plan is None
assert agent._render_plan_description() is None
# ---------------------------------------------------------------------------
# 12. Replan nudge disabled when planning_replan_on_stall=0
# ---------------------------------------------------------------------------
async def test_replan_nudge_disabled_when_zero(browser_session, mock_llm):
agent = _make_agent(browser_session, mock_llm, planning_replan_on_stall=0)
agent.state.plan = [PlanItem(text='Step 1', status='current')]
agent.state.consecutive_failures = 100 # high but doesn't matter
initial_count = len(agent._message_manager.state.history.context_messages)
agent._inject_replan_nudge()
after_count = len(agent._message_manager.state.history.context_messages)
assert after_count == initial_count
# ---------------------------------------------------------------------------
# 13. No nudge when no plan exists
# ---------------------------------------------------------------------------
async def test_no_replan_nudge_without_plan(browser_session, mock_llm):
agent = _make_agent(browser_session, mock_llm, planning_replan_on_stall=1)
agent.state.consecutive_failures = 5 # above threshold
initial_count = len(agent._message_manager.state.history.context_messages)
agent._inject_replan_nudge()
after_count = len(agent._message_manager.state.history.context_messages)
assert after_count == initial_count
# ---------------------------------------------------------------------------
# 14. Exploration nudge fires when no plan exists after N steps
# ---------------------------------------------------------------------------
async def test_exploration_nudge_fires_after_limit(browser_session, mock_llm):
agent = _make_agent(browser_session, mock_llm, planning_exploration_limit=3)
agent.state.plan = None
agent.state.n_steps = 3 # at the limit
initial_count = len(agent._message_manager.state.history.context_messages)
agent._inject_exploration_nudge()
after_count = len(agent._message_manager.state.history.context_messages)
assert after_count == initial_count + 1
msg = agent._message_manager.state.history.context_messages[-1]
assert isinstance(msg.content, str) and 'PLANNING NUDGE' in msg.content
# ---------------------------------------------------------------------------
# 15. No exploration nudge when plan already exists
# ---------------------------------------------------------------------------
async def test_no_exploration_nudge_when_plan_exists(browser_session, mock_llm):
agent = _make_agent(browser_session, mock_llm, planning_exploration_limit=3)
agent.state.plan = [PlanItem(text='Step 1', status='current')]
agent.state.n_steps = 10 # well above limit
initial_count = len(agent._message_manager.state.history.context_messages)
agent._inject_exploration_nudge()
after_count = len(agent._message_manager.state.history.context_messages)
assert after_count == initial_count
# ---------------------------------------------------------------------------
# 16. No exploration nudge below the limit
# ---------------------------------------------------------------------------
async def test_no_exploration_nudge_below_limit(browser_session, mock_llm):
agent = _make_agent(browser_session, mock_llm, planning_exploration_limit=5)
agent.state.plan = None
agent.state.n_steps = 4 # below the limit
initial_count = len(agent._message_manager.state.history.context_messages)
agent._inject_exploration_nudge()
after_count = len(agent._message_manager.state.history.context_messages)
assert after_count == initial_count
# ---------------------------------------------------------------------------
# 17. Exploration nudge disabled when planning_exploration_limit=0
# ---------------------------------------------------------------------------
async def test_exploration_nudge_disabled_when_zero(browser_session, mock_llm):
agent = _make_agent(browser_session, mock_llm, planning_exploration_limit=0)
agent.state.plan = None
agent.state.n_steps = 100 # high but doesn't matter
initial_count = len(agent._message_manager.state.history.context_messages)
agent._inject_exploration_nudge()
after_count = len(agent._message_manager.state.history.context_messages)
assert after_count == initial_count
# ---------------------------------------------------------------------------
# 18. Exploration nudge disabled when enable_planning=False
# ---------------------------------------------------------------------------
async def test_exploration_nudge_disabled_when_planning_off(browser_session, mock_llm):
agent = _make_agent(browser_session, mock_llm, enable_planning=False, planning_exploration_limit=3)
agent.state.plan = None
agent.state.n_steps = 10 # above limit
initial_count = len(agent._message_manager.state.history.context_messages)
agent._inject_exploration_nudge()
after_count = len(agent._message_manager.state.history.context_messages)
assert after_count == initial_count
# ---------------------------------------------------------------------------
# 19. Flash mode forces enable_planning=False
# ---------------------------------------------------------------------------
async def test_flash_mode_disables_planning(browser_session, mock_llm):
agent = _make_agent(browser_session, mock_llm, flash_mode=True)
assert agent.settings.enable_planning is False
+120
View File
@@ -0,0 +1,120 @@
"""Tests for AI step private method used during rerun"""
from unittest.mock import AsyncMock
from browser_use.agent.service import Agent
from browser_use.agent.views import ActionResult
from tests.ci.conftest import create_mock_llm
async def test_execute_ai_step_basic():
"""Test that _execute_ai_step extracts content with AI"""
# Create mock LLM that returns text response
async def custom_ainvoke(*args, **kwargs):
from browser_use.llm.views import ChatInvokeCompletion
return ChatInvokeCompletion(completion='Extracted: Test content from page', usage=None)
mock_llm = AsyncMock()
mock_llm.ainvoke.side_effect = custom_ainvoke
mock_llm.model = 'mock-model'
llm = create_mock_llm(actions=None)
agent = Agent(task='Test task', llm=llm)
await agent.browser_session.start()
try:
# Execute _execute_ai_step with mock LLM
result = await agent._execute_ai_step(
query='Extract the main heading',
include_screenshot=False,
extract_links=False,
ai_step_llm=mock_llm,
)
# Verify result
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'Extracted: Test content from page' in result.extracted_content
assert result.long_term_memory is not None
finally:
await agent.close()
async def test_execute_ai_step_with_screenshot():
"""Test that _execute_ai_step includes screenshot when requested"""
# Create mock LLM
async def custom_ainvoke(*args, **kwargs):
from browser_use.llm.views import ChatInvokeCompletion
# Verify that we received a message with image content
messages = args[0] if args else []
assert len(messages) >= 1, 'Should have at least one message'
# Check if any message has image content
has_image = False
for msg in messages:
if hasattr(msg, 'content') and isinstance(msg.content, list):
for part in msg.content:
if hasattr(part, 'type') and part.type == 'image_url':
has_image = True
break
assert has_image, 'Should include screenshot in message'
return ChatInvokeCompletion(completion='Extracted content with screenshot analysis', usage=None)
mock_llm = AsyncMock()
mock_llm.ainvoke.side_effect = custom_ainvoke
mock_llm.model = 'mock-model'
llm = create_mock_llm(actions=None)
agent = Agent(task='Test task', llm=llm)
await agent.browser_session.start()
try:
# Execute _execute_ai_step with screenshot
result = await agent._execute_ai_step(
query='Analyze this page',
include_screenshot=True,
extract_links=False,
ai_step_llm=mock_llm,
)
# Verify result
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'Extracted content with screenshot analysis' in result.extracted_content
finally:
await agent.close()
async def test_execute_ai_step_error_handling():
"""Test that _execute_ai_step handles errors gracefully"""
# Create mock LLM that raises an error
mock_llm = AsyncMock()
mock_llm.ainvoke.side_effect = Exception('LLM service unavailable')
mock_llm.model = 'mock-model'
llm = create_mock_llm(actions=None)
agent = Agent(task='Test task', llm=llm)
await agent.browser_session.start()
try:
# Execute _execute_ai_step - should return ActionResult with error
result = await agent._execute_ai_step(
query='Extract data',
include_screenshot=False,
ai_step_llm=mock_llm,
)
# Verify error is in result (not raised)
assert isinstance(result, ActionResult)
assert result.error is not None
assert 'AI step failed' in result.error
finally:
await agent.close()
+558
View File
@@ -0,0 +1,558 @@
"""Tests for ax_name (accessible name) element matching in history rerun.
This tests Level 4 matching which uses the accessibility tree's name property
to match elements when hash, stable_hash, and xpath all fail.
This is particularly useful for dynamic SPAs where DOM structure changes
but accessible names remain stable.
Also tests dropdown/menu re-opening behavior when menu items can't be found
because the dropdown closed during the wait between steps.
"""
from unittest.mock import AsyncMock
from browser_use.agent.service import Agent
from browser_use.agent.views import ActionResult, AgentHistory, AgentHistoryList, RerunSummaryAction, StepMetadata
from browser_use.browser.views import BrowserStateHistory
from browser_use.dom.views import DOMInteractedElement, DOMRect, MatchLevel, NodeType
from tests.ci.conftest import create_mock_llm
async def test_ax_name_matching_succeeds_when_hash_fails(httpserver):
"""Test that ax_name matching finds elements when hash/xpath matching fails.
This simulates a dynamic SPA where the element hash and xpath change between
sessions, but the accessible name (ax_name) remains stable.
"""
# Set up a test page with a menu item that has an aria-label
# The aria-label becomes the accessible name (ax_name)
test_html = """<!DOCTYPE html>
<html>
<body>
<div role="menuitem" aria-label="New Contact" id="menu-1">New Contact</div>
<div role="menuitem" aria-label="Search" id="menu-2">Search</div>
</body>
</html>"""
httpserver.expect_request('/test').respond_with_data(test_html, content_type='text/html')
test_url = httpserver.url_for('/test')
# Create a mock LLM for summary
summary_action = RerunSummaryAction(
summary='Rerun completed',
success=True,
completion_status='complete',
)
async def custom_ainvoke(*args, **kwargs):
output_format = args[1] if len(args) > 1 else kwargs.get('output_format')
if output_format is RerunSummaryAction:
from browser_use.llm.views import ChatInvokeCompletion
return ChatInvokeCompletion(completion=summary_action, usage=None)
raise ValueError('Unexpected output_format')
mock_summary_llm = AsyncMock()
mock_summary_llm.ainvoke.side_effect = custom_ainvoke
llm = create_mock_llm(actions=None)
agent = Agent(task='Test task', llm=llm)
AgentOutput = agent.AgentOutput
# Create an element with DIFFERENT hash/xpath but SAME ax_name as the real element
# This simulates what happens in dynamic SPAs where the DOM changes but
# accessible names remain stable
historical_element = DOMInteractedElement(
node_id=9999, # Different node_id
backend_node_id=9999, # Different backend_node_id
frame_id=None,
node_type=NodeType.ELEMENT_NODE,
node_value='',
node_name='DIV', # Same node type
# Note: aria-label is NOT in attributes - this tests that ax_name matching
# is used as a fallback when attribute matching fails
attributes={'role': 'menuitem', 'class': 'dynamic-class-12345'},
x_path='html/body/div[1]/div[4]/div[4]/div[1]', # Different xpath
element_hash=123456789, # Different hash (won't match)
stable_hash=987654321, # Different stable_hash (won't match)
bounds=DOMRect(x=0, y=0, width=100, height=50),
ax_name='New Contact', # SAME ax_name - this should match!
)
# Step 1: Navigate to test page
navigate_step = AgentHistory(
model_output=AgentOutput(
evaluation_previous_goal=None,
memory='Navigate to test page',
next_goal=None,
action=[{'navigate': {'url': test_url}}], # type: ignore[arg-type]
),
result=[ActionResult(long_term_memory='Navigated')],
state=BrowserStateHistory(
url=test_url,
title='Test Page',
tabs=[],
interacted_element=[None],
),
metadata=StepMetadata(
step_start_time=0,
step_end_time=1,
step_number=1,
step_interval=0.1,
),
)
# Step 2: Click on element that has different hash/xpath but same ax_name
click_step = AgentHistory(
model_output=AgentOutput(
evaluation_previous_goal=None,
memory='Click New Contact menu',
next_goal=None,
action=[{'click': {'index': 100}}], # type: ignore[arg-type] # Original index doesn't matter
),
result=[ActionResult(long_term_memory='Clicked New Contact')],
state=BrowserStateHistory(
url=test_url,
title='Test Page',
tabs=[],
interacted_element=[historical_element],
),
metadata=StepMetadata(
step_start_time=1,
step_end_time=2,
step_number=2,
step_interval=0.1,
),
)
history = AgentHistoryList(history=[navigate_step, click_step])
try:
# Run rerun - should succeed because ax_name matching finds the element
results = await agent.rerun_history(
history,
skip_failures=False,
max_retries=1,
summary_llm=mock_summary_llm,
)
# Should have 3 results: navigate + click + AI summary
assert len(results) == 3
# First result should be navigation success
nav_result = results[0]
assert nav_result.error is None
# Second result should be click success (matched via ax_name)
click_result = results[1]
assert click_result.error is None, f'Click should succeed via ax_name matching, got error: {click_result.error}'
# Third result should be AI summary
summary_result = results[2]
assert summary_result.is_done is True
finally:
await agent.close()
async def test_ax_name_matching_requires_same_node_type(httpserver):
"""Test that ax_name matching also requires matching node type.
Even if ax_name matches, the node type (DIV, BUTTON, etc.) must also match.
"""
test_html = """<!DOCTYPE html>
<html>
<body>
<button aria-label="Submit">Submit</button>
<div aria-label="Submit">Submit Label</div>
</body>
</html>"""
httpserver.expect_request('/test').respond_with_data(test_html, content_type='text/html')
test_url = httpserver.url_for('/test')
llm = create_mock_llm(actions=None)
agent = Agent(task='Test task', llm=llm)
AgentOutput = agent.AgentOutput
# Historical element is a SPAN with ax_name "Submit"
# Page has BUTTON and DIV with same ax_name, but no SPAN
historical_element = DOMInteractedElement(
node_id=1,
backend_node_id=1,
frame_id=None,
node_type=NodeType.ELEMENT_NODE,
node_value='',
node_name='SPAN', # SPAN - won't match BUTTON or DIV
attributes={},
x_path='html/body/span',
element_hash=111,
stable_hash=111,
bounds=DOMRect(x=0, y=0, width=100, height=50),
ax_name='Submit', # Same ax_name, but wrong node type
)
navigate_step = AgentHistory(
model_output=AgentOutput(
evaluation_previous_goal=None,
memory='Navigate',
next_goal=None,
action=[{'navigate': {'url': test_url}}], # type: ignore[arg-type]
),
result=[ActionResult(long_term_memory='Navigated')],
state=BrowserStateHistory(
url=test_url,
title='Test',
tabs=[],
interacted_element=[None],
),
metadata=StepMetadata(step_start_time=0, step_end_time=1, step_number=1, step_interval=0.1),
)
click_step = AgentHistory(
model_output=AgentOutput(
evaluation_previous_goal=None,
memory='Click Submit',
next_goal=None,
action=[{'click': {'index': 1}}], # type: ignore[arg-type]
),
result=[ActionResult(long_term_memory='Clicked')],
state=BrowserStateHistory(
url=test_url,
title='Test',
tabs=[],
interacted_element=[historical_element],
),
metadata=StepMetadata(step_start_time=1, step_end_time=2, step_number=2, step_interval=0.1),
)
history = AgentHistoryList(history=[navigate_step, click_step])
try:
# Should fail because no SPAN with ax_name "Submit" exists
await agent.rerun_history(
history,
skip_failures=False,
max_retries=1,
)
assert False, 'Expected RuntimeError - no matching SPAN element'
except RuntimeError as e:
# Expected - no SPAN element with ax_name "Submit"
assert 'failed after 1 attempts' in str(e)
finally:
await agent.close()
def test_match_level_enum_includes_ax_name():
"""Test that MatchLevel enum includes AX_NAME level."""
assert hasattr(MatchLevel, 'AX_NAME')
assert MatchLevel.AX_NAME.value == 4
assert MatchLevel.ATTRIBUTE.value == 5 # AX_NAME comes before ATTRIBUTE
async def test_ax_name_matching_before_attribute_matching(httpserver):
"""Test that ax_name matching (Level 4) is tried before attribute matching (Level 5).
This ensures the correct matching order: EXACT -> STABLE -> XPATH -> AX_NAME -> ATTRIBUTE
"""
# Page has element with text content that becomes its ax_name
# The DIV has role="menuitem" and text "Contact" which becomes its accessible name
# but NO aria-label/id/name attributes - so attribute matching will fail but ax_name should work
test_html = """<!DOCTYPE html>
<html>
<body>
<div role="menuitem">Contact</div>
</body>
</html>"""
httpserver.expect_request('/test').respond_with_data(test_html, content_type='text/html')
test_url = httpserver.url_for('/test')
summary_action = RerunSummaryAction(
summary='Rerun completed',
success=True,
completion_status='complete',
)
async def custom_ainvoke(*args, **kwargs):
output_format = args[1] if len(args) > 1 else kwargs.get('output_format')
if output_format is RerunSummaryAction:
from browser_use.llm.views import ChatInvokeCompletion
return ChatInvokeCompletion(completion=summary_action, usage=None)
raise ValueError('Unexpected output_format')
mock_summary_llm = AsyncMock()
mock_summary_llm.ainvoke.side_effect = custom_ainvoke
llm = create_mock_llm(actions=None)
agent = Agent(task='Test task', llm=llm)
AgentOutput = agent.AgentOutput
# Historical element has NO aria-label attribute (attribute matching will fail)
# but HAS ax_name (ax_name matching should work)
historical_element = DOMInteractedElement(
node_id=1,
backend_node_id=1,
frame_id=None,
node_type=NodeType.ELEMENT_NODE,
node_value='',
node_name='DIV',
# No aria-label, id, or name - attribute matching will fail
attributes={'role': 'menuitem'},
x_path='html/body/div[99]', # Wrong xpath
element_hash=12345, # Wrong hash
stable_hash=12345, # Wrong stable hash
bounds=DOMRect(x=0, y=0, width=100, height=50),
ax_name='Contact', # ax_name from accessibility tree
)
navigate_step = AgentHistory(
model_output=AgentOutput(
evaluation_previous_goal=None,
memory='Navigate',
next_goal=None,
action=[{'navigate': {'url': test_url}}], # type: ignore[arg-type]
),
result=[ActionResult(long_term_memory='Navigated')],
state=BrowserStateHistory(
url=test_url,
title='Test',
tabs=[],
interacted_element=[None],
),
metadata=StepMetadata(step_start_time=0, step_end_time=1, step_number=1, step_interval=0.1),
)
click_step = AgentHistory(
model_output=AgentOutput(
evaluation_previous_goal=None,
memory='Click Contact',
next_goal=None,
action=[{'click': {'index': 1}}], # type: ignore[arg-type]
),
result=[ActionResult(long_term_memory='Clicked')],
state=BrowserStateHistory(
url=test_url,
title='Test',
tabs=[],
interacted_element=[historical_element],
),
metadata=StepMetadata(step_start_time=1, step_end_time=2, step_number=2, step_interval=0.1),
)
history = AgentHistoryList(history=[navigate_step, click_step])
try:
# Should succeed via ax_name matching (Level 4)
# since hash, stable_hash, xpath all fail but ax_name matches
results = await agent.rerun_history(
history,
skip_failures=False,
max_retries=1,
summary_llm=mock_summary_llm,
)
# Navigation + click + summary
assert len(results) == 3
# Click should succeed (matched via ax_name)
click_result = results[1]
assert click_result.error is None, f'Expected ax_name match to succeed, got: {click_result.error}'
finally:
await agent.close()
# Tests for dropdown/menu re-opening behavior
def test_is_menu_opener_step_detects_aria_haspopup():
"""Test that _is_menu_opener_step detects aria-haspopup elements."""
llm = create_mock_llm(actions=None)
agent = Agent(task='Test task', llm=llm)
AgentOutput = agent.AgentOutput
# Element with aria-haspopup="true" should be detected as menu opener
opener_element = DOMInteractedElement(
node_id=1,
backend_node_id=1,
frame_id=None,
node_type=NodeType.ELEMENT_NODE,
node_value='',
node_name='DIV',
attributes={'aria-haspopup': 'true', 'class': 'dropdown-trigger'},
x_path='html/body/div',
element_hash=12345,
stable_hash=12345,
bounds=DOMRect(x=0, y=0, width=100, height=50),
ax_name='Contact',
)
history_item = AgentHistory(
model_output=AgentOutput(
evaluation_previous_goal=None,
memory='Click dropdown',
next_goal=None,
action=[{'click': {'index': 1}}], # type: ignore[arg-type]
),
result=[ActionResult(long_term_memory='Clicked')],
state=BrowserStateHistory(
url='http://test.com',
title='Test',
tabs=[],
interacted_element=[opener_element],
),
metadata=StepMetadata(step_start_time=0, step_end_time=1, step_number=1, step_interval=0.1),
)
assert agent._is_menu_opener_step(history_item) is True
def test_is_menu_opener_step_detects_guidewire_toggle():
"""Test that _is_menu_opener_step detects Guidewire toggleSubMenu pattern."""
llm = create_mock_llm(actions=None)
agent = Agent(task='Test task', llm=llm)
AgentOutput = agent.AgentOutput
# Element with data-gw-click="toggleSubMenu" should be detected
opener_element = DOMInteractedElement(
node_id=1,
backend_node_id=1,
frame_id=None,
node_type=NodeType.ELEMENT_NODE,
node_value='',
node_name='DIV',
attributes={'data-gw-click': 'toggleSubMenu', 'class': 'gw-action--expand-button'},
x_path='html/body/div',
element_hash=12345,
stable_hash=12345,
bounds=DOMRect(x=0, y=0, width=100, height=50),
ax_name=None,
)
history_item = AgentHistory(
model_output=AgentOutput(
evaluation_previous_goal=None,
memory='Toggle menu',
next_goal=None,
action=[{'click': {'index': 1}}], # type: ignore[arg-type]
),
result=[ActionResult(long_term_memory='Toggled')],
state=BrowserStateHistory(
url='http://test.com',
title='Test',
tabs=[],
interacted_element=[opener_element],
),
metadata=StepMetadata(step_start_time=0, step_end_time=1, step_number=1, step_interval=0.1),
)
assert agent._is_menu_opener_step(history_item) is True
def test_is_menu_opener_step_returns_false_for_regular_element():
"""Test that _is_menu_opener_step returns False for non-menu elements."""
llm = create_mock_llm(actions=None)
agent = Agent(task='Test task', llm=llm)
AgentOutput = agent.AgentOutput
# Regular button without menu attributes
regular_element = DOMInteractedElement(
node_id=1,
backend_node_id=1,
frame_id=None,
node_type=NodeType.ELEMENT_NODE,
node_value='',
node_name='BUTTON',
attributes={'class': 'submit-btn', 'type': 'submit'},
x_path='html/body/button',
element_hash=12345,
stable_hash=12345,
bounds=DOMRect(x=0, y=0, width=100, height=50),
ax_name='Submit',
)
history_item = AgentHistory(
model_output=AgentOutput(
evaluation_previous_goal=None,
memory='Click submit',
next_goal=None,
action=[{'click': {'index': 1}}], # type: ignore[arg-type]
),
result=[ActionResult(long_term_memory='Clicked')],
state=BrowserStateHistory(
url='http://test.com',
title='Test',
tabs=[],
interacted_element=[regular_element],
),
metadata=StepMetadata(step_start_time=0, step_end_time=1, step_number=1, step_interval=0.1),
)
assert agent._is_menu_opener_step(history_item) is False
def test_is_menu_item_element_detects_role_menuitem():
"""Test that _is_menu_item_element detects role=menuitem."""
llm = create_mock_llm(actions=None)
agent = Agent(task='Test task', llm=llm)
menu_item = DOMInteractedElement(
node_id=1,
backend_node_id=1,
frame_id=None,
node_type=NodeType.ELEMENT_NODE,
node_value='',
node_name='DIV',
attributes={'role': 'menuitem', 'class': 'menu-option'},
x_path='html/body/div/div',
element_hash=12345,
stable_hash=12345,
bounds=DOMRect(x=0, y=0, width=100, height=50),
ax_name='New Contact',
)
assert agent._is_menu_item_element(menu_item) is True
def test_is_menu_item_element_detects_guidewire_class():
"""Test that _is_menu_item_element detects Guidewire gw-action--inner class."""
llm = create_mock_llm(actions=None)
agent = Agent(task='Test task', llm=llm)
menu_item = DOMInteractedElement(
node_id=1,
backend_node_id=1,
frame_id=None,
node_type=NodeType.ELEMENT_NODE,
node_value='',
node_name='DIV',
attributes={'class': 'gw-action--inner gw-hasDivider', 'aria-haspopup': 'true'},
x_path='html/body/div/div',
element_hash=12345,
stable_hash=12345,
bounds=DOMRect(x=0, y=0, width=100, height=50),
ax_name='New Contact',
)
assert agent._is_menu_item_element(menu_item) is True
def test_is_menu_item_element_returns_false_for_regular_element():
"""Test that _is_menu_item_element returns False for non-menu elements."""
llm = create_mock_llm(actions=None)
agent = Agent(task='Test task', llm=llm)
regular_element = DOMInteractedElement(
node_id=1,
backend_node_id=1,
frame_id=None,
node_type=NodeType.ELEMENT_NODE,
node_value='',
node_name='BUTTON',
attributes={'class': 'submit-btn', 'type': 'submit'},
x_path='html/body/button',
element_hash=12345,
stable_hash=12345,
bounds=DOMRect(x=0, y=0, width=100, height=50),
ax_name='Submit',
)
assert agent._is_menu_item_element(regular_element) is False
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,46 @@
"""Regression test: BrowserError raised inside an action must reach Tools.act with its
structured short/long-term memory intact, instead of being flattened into a generic
'Error executing action ...' RuntimeError by execute_action's catch-all handler."""
from browser_use.browser.views import BrowserError
from browser_use.tools.service import Tools
async def test_browser_error_memory_survives_execute_action():
tools = Tools()
@tools.registry.action(description='Test action that raises a structured BrowserError')
async def raise_structured_error():
raise BrowserError(
message='element is a select, not clickable',
short_term_memory='Available options: alpha, beta, gamma',
long_term_memory='Tried to click a dropdown; use select_dropdown instead',
)
ActionModel = tools.registry.create_action_model()
action = ActionModel(**{'raise_structured_error': {}})
result = await tools.act(action, browser_session=None) # type: ignore[arg-type] -- action doesn't touch the browser
assert result.error == 'Tried to click a dropdown; use select_dropdown instead', (
f'structured long_term_memory lost: {result.error!r}'
)
assert result.extracted_content == 'Available options: alpha, beta, gamma'
async def test_plain_browser_error_still_returns_recoverable_action_result():
"""A BrowserError without long_term_memory must not escape Tools.act as an
exception — it must still come back as a recoverable ActionResult (as it did
when the generic execute_action handler flattened it)."""
tools = Tools()
@tools.registry.action(description='Test action that raises a plain BrowserError')
async def raise_plain_error():
raise BrowserError(message='element with index 5 does not exist')
ActionModel = tools.registry.create_action_model()
action = ActionModel(**{'raise_plain_error': {}})
result = await tools.act(action, browser_session=None) # type: ignore[arg-type] -- action doesn't touch the browser
assert result.error is not None and 'element with index 5 does not exist' in result.error
+48
View File
@@ -0,0 +1,48 @@
import os
import subprocess
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
def _run_browser_use_cli(*args: str) -> subprocess.CompletedProcess[str]:
env = os.environ.copy()
env['PYTHONPATH'] = os.pathsep.join(part for part in (str(ROOT), env.get('PYTHONPATH', '')) if part)
return subprocess.run(
[sys.executable, '-m', 'browser_use.cli', *args],
cwd=ROOT,
env=env,
capture_output=True,
text=True,
timeout=20,
)
def test_browser_use_doctor_help_prints_browser_use_usage():
result = _run_browser_use_cli('doctor', '--help')
assert result.returncode == 0
assert result.stdout == 'usage: browser-use doctor [--fix-snap]\n'
assert result.stderr == ''
def test_normalize_captured_cli_output_handles_string_system_exit(capsys):
from browser_use.cli import _normalize_captured_cli_output
def exits_with_string(_argv):
raise SystemExit('browser-harness failed')
assert _normalize_captured_cli_output(exits_with_string, []) == 1
captured = capsys.readouterr()
assert captured.out == ''
assert captured.err == 'browser-use failed\n'
def test_browser_use_tui_is_deprecated_alias(monkeypatch, capsys):
import browser_use.cli as browser_use_cli
monkeypatch.setattr(browser_use_cli, 'main', lambda: 0)
assert browser_use_cli.browser_use_tui_main() == 0
assert capsys.readouterr().err == 'browser-use-tui is deprecated; use browser-use instead.\n'
@@ -0,0 +1,119 @@
import os
import subprocess
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
BROWSER_USE_REPO_SKILL_URL = 'https://raw.githubusercontent.com/browser-use/browser-use/main/skills/browser-use/SKILL.md'
EXPECTED_SKILL_INSTALL_PATHS = (
Path('.agents') / 'skills' / 'browser-use' / 'SKILL.md',
Path('.claude') / 'skills' / 'browser-use' / 'SKILL.md',
Path('.codex') / 'skills' / 'browser-use' / 'SKILL.md',
Path('.copilot') / 'skills' / 'browser-use' / 'SKILL.md',
Path('.cursor') / 'skills' / 'browser-use' / 'SKILL.md',
Path('.gemini') / 'skills' / 'browser-use' / 'SKILL.md',
Path('.config') / 'opencode' / 'skills' / 'browser-use' / 'SKILL.md',
)
def _fake_browser_harness_tools(tmp_path: Path, skill_text: str) -> Path:
bin_dir = tmp_path / 'bin'
bin_dir.mkdir()
uv = bin_dir / 'uv'
uv.write_text(
'#!/usr/bin/env python3\n'
'import os, pathlib, sys\n'
'pathlib.Path(os.environ["UV_TOOL_INSTALL_ARGS_FILE"]).write_text(" ".join(sys.argv[1:]), encoding="utf-8")\n',
encoding='utf-8',
)
uv.chmod(0o755)
browser_harness = bin_dir / 'browser-harness'
browser_harness.write_text(
'#!/usr/bin/env python3\n'
'import sys\n'
f'text = {skill_text!r}\n'
'if sys.argv[1:] == ["skill"]:\n'
' print(text, end="")\n'
'else:\n'
' print("usage: browser-harness skill", file=sys.stderr)\n'
' sys.exit(2)\n',
encoding='utf-8',
)
browser_harness.chmod(0o755)
return bin_dir
def test_docs_install_browser_use_skill_from_package_alias():
readme = (ROOT / 'README.md').read_text(encoding='utf-8')
assert 'run `browser-use skill install` to register the skill' in readme
assert 'mkdir -p ~/.claude/skills/browser-use' not in readme
assert 'uv run --with "browser-use[browser-harness]" python -c' not in readme
assert 'from browser_use.skills import browser_use_skill_text' not in readme
assert BROWSER_USE_REPO_SKILL_URL not in readme
assert 'raw.githubusercontent.com/browser-use/browser-harness/main/SKILL.md' not in readme
def test_browser_use_cli_installs_browser_harness_package_skill(tmp_path):
bin_dir = _fake_browser_harness_tools(tmp_path, '---\nname: browser-harness\n---\n\n# Browser Harness\n')
home = tmp_path / 'home'
for stale in (home / path for path in EXPECTED_SKILL_INSTALL_PATHS):
stale.parent.mkdir(parents=True)
stale.write_text('stale browser-use skill', encoding='utf-8')
uv_args = tmp_path / 'uv-args.txt'
env = os.environ.copy()
env['HOME'] = str(home)
env['PATH'] = os.pathsep.join(part for part in (str(bin_dir), env.get('PATH', '')) if part)
env['PYTHONPATH'] = os.pathsep.join(part for part in (str(ROOT), env.get('PYTHONPATH', '')) if part)
env['UV_TOOL_INSTALL_ARGS_FILE'] = str(uv_args)
result = subprocess.run(
[sys.executable, '-m', 'browser_use.cli', 'skill', 'install'],
cwd=ROOT,
env=env,
capture_output=True,
text=True,
timeout=10,
)
assert result.returncode == 0, result.stderr
assert uv_args.read_text(encoding='utf-8') == 'tool install --python 3.12 --upgrade --force browser-use'
expected = (
'---\n'
'name: browser-use\n'
'description: "Direct browser control via CDP for web interaction: automation, scraping, testing, screenshots, and site/app work."\n'
'---\n\n'
'# Browser Use\n'
)
for installed in (home / path for path in EXPECTED_SKILL_INSTALL_PATHS):
assert installed.read_text(encoding='utf-8') == expected
def test_browser_use_cli_validates_destination_before_installing_harness(tmp_path):
bin_dir = _fake_browser_harness_tools(tmp_path, '---\nname: browser-harness\n---\n\n# Browser Harness\n')
blocking_file = tmp_path / 'not-a-directory'
blocking_file.write_text('blocks skill directory creation', encoding='utf-8')
uv_args = tmp_path / 'uv-args.txt'
env = os.environ.copy()
env['HOME'] = str(tmp_path / 'home')
env['PATH'] = os.pathsep.join(part for part in (str(bin_dir), env.get('PATH', '')) if part)
env['PYTHONPATH'] = os.pathsep.join(part for part in (str(ROOT), env.get('PYTHONPATH', '')) if part)
env['UV_TOOL_INSTALL_ARGS_FILE'] = str(uv_args)
result = subprocess.run(
[sys.executable, '-m', 'browser_use.cli', 'skill', 'install', '--path', str(blocking_file / 'nested')],
cwd=ROOT,
env=env,
capture_output=True,
text=True,
timeout=10,
)
assert result.returncode == 1
assert 'is not a directory' in result.stderr
assert not uv_args.exists()
+156
View File
@@ -0,0 +1,156 @@
"""Tests for step budget warning injection (IMP-7a)."""
from browser_use.agent.service import Agent
from browser_use.agent.views import AgentStepInfo
from browser_use.llm.messages import UserMessage
from tests.ci.conftest import create_mock_llm
def _get_context_messages(agent: Agent) -> list[str]:
"""Extract text content from the agent's context messages."""
msgs = agent._message_manager.state.history.context_messages
return [m.content for m in msgs if isinstance(m, UserMessage) and isinstance(m.content, str)]
async def test_budget_warning_injected_at_75_percent():
"""Budget warning should be injected when step >= 75% of max_steps."""
llm = create_mock_llm()
agent = Agent(task='Test task', llm=llm)
step_info = AgentStepInfo(step_number=74, max_steps=100) # step 75/100 = 75%
await agent._inject_budget_warning(step_info)
messages = _get_context_messages(agent)
assert len(messages) == 1
assert 'BUDGET WARNING' in messages[0]
assert '75/100' in messages[0]
assert '25 steps remaining' in messages[0]
async def test_budget_warning_injected_at_90_percent():
"""Budget warning should fire at 90% too."""
llm = create_mock_llm()
agent = Agent(task='Test task', llm=llm)
step_info = AgentStepInfo(step_number=89, max_steps=100) # step 90/100 = 90%
await agent._inject_budget_warning(step_info)
messages = _get_context_messages(agent)
assert len(messages) == 1
assert 'BUDGET WARNING' in messages[0]
assert '90/100' in messages[0]
assert '10 steps remaining' in messages[0]
async def test_no_budget_warning_below_75_percent():
"""No warning should be injected when step < 75% of max_steps."""
llm = create_mock_llm()
agent = Agent(task='Test task', llm=llm)
step_info = AgentStepInfo(step_number=73, max_steps=100) # step 74/100 = 74%
await agent._inject_budget_warning(step_info)
messages = _get_context_messages(agent)
assert len(messages) == 0
async def test_no_budget_warning_on_last_step():
"""No budget warning on the last step — _force_done_after_last_step handles that."""
llm = create_mock_llm()
agent = Agent(task='Test task', llm=llm)
step_info = AgentStepInfo(step_number=99, max_steps=100) # last step
assert step_info.is_last_step()
await agent._inject_budget_warning(step_info)
messages = _get_context_messages(agent)
assert len(messages) == 0
async def test_no_budget_warning_when_step_info_is_none():
"""No warning when step_info is None."""
llm = create_mock_llm()
agent = Agent(task='Test task', llm=llm)
await agent._inject_budget_warning(None)
messages = _get_context_messages(agent)
assert len(messages) == 0
async def test_budget_warning_exact_threshold():
"""The warning should fire at exactly 75% (step 15/20)."""
llm = create_mock_llm()
agent = Agent(task='Test task', llm=llm)
# step_number=14 means step 15 (1-indexed), 15/20 = 75%
step_info = AgentStepInfo(step_number=14, max_steps=20)
await agent._inject_budget_warning(step_info)
messages = _get_context_messages(agent)
assert len(messages) == 1
assert '15/20' in messages[0]
assert '5 steps remaining' in messages[0]
async def test_budget_warning_just_below_threshold():
"""No warning at 74% — just below threshold."""
llm = create_mock_llm()
agent = Agent(task='Test task', llm=llm)
# step_number=13 means step 14 (1-indexed), 14/20 = 70%
step_info = AgentStepInfo(step_number=13, max_steps=20)
await agent._inject_budget_warning(step_info)
messages = _get_context_messages(agent)
assert len(messages) == 0
async def test_budget_warning_small_max_steps():
"""Budget warning works correctly with small max_steps values."""
llm = create_mock_llm()
agent = Agent(task='Test task', llm=llm)
# step_number=3 means step 4 (1-indexed), 4/4 = 100% but is_last_step
step_info = AgentStepInfo(step_number=3, max_steps=4)
assert step_info.is_last_step()
await agent._inject_budget_warning(step_info)
messages = _get_context_messages(agent)
assert len(messages) == 0 # last step, no warning
# step_number=2 means step 3 (1-indexed), 3/4 = 75%
step_info = AgentStepInfo(step_number=2, max_steps=4)
await agent._inject_budget_warning(step_info)
messages = _get_context_messages(agent)
assert len(messages) == 1
assert '3/4' in messages[0]
async def test_budget_warning_percentage_display():
"""The percentage in the warning should be integer (no decimals)."""
llm = create_mock_llm()
agent = Agent(task='Test task', llm=llm)
# step 76/100 = 76%
step_info = AgentStepInfo(step_number=75, max_steps=100)
await agent._inject_budget_warning(step_info)
messages = _get_context_messages(agent)
assert '76%' in messages[0]
# Should not have decimal
assert '76.0%' not in messages[0]
async def test_budget_warning_contains_actionable_guidance():
"""The warning message should include actionable guidance for the agent."""
llm = create_mock_llm()
agent = Agent(task='Test task', llm=llm)
step_info = AgentStepInfo(step_number=74, max_steps=100)
await agent._inject_budget_warning(step_info)
messages = _get_context_messages(agent)
msg = messages[0]
assert 'consolidate your results' in msg.lower()
assert 'done' in msg.lower()
assert 'partial results' in msg.lower()
+116
View File
@@ -0,0 +1,116 @@
"""Regression tests for TimeoutWrappedCDPClient.
cdp_use.CDPClient.send_raw awaits a future that only resolves when the browser
sends a matching response. When the server goes silent (observed against cloud
browsers whose WebSocket stays connected at TCP/keepalive layer but never
replies), send_raw hangs forever. The wrapper turns that hang into a fast
TimeoutError.
"""
from __future__ import annotations
import asyncio
import time
from unittest.mock import patch
import pytest
from browser_use.browser._cdp_timeout import (
DEFAULT_CDP_REQUEST_TIMEOUT_S,
TimeoutWrappedCDPClient,
_coerce_valid_timeout,
_parse_env_cdp_timeout,
)
def _make_wrapped_client_without_websocket(timeout_s: float) -> TimeoutWrappedCDPClient:
"""Build a TimeoutWrappedCDPClient without opening a real WebSocket.
Calling `CDPClient.__init__` directly would try to construct a working
client. We only want to exercise the timeout-wrapper `send_raw` path, so
we construct the object via __new__ and set the single attribute the
wrapper needs.
"""
client = TimeoutWrappedCDPClient.__new__(TimeoutWrappedCDPClient)
client._cdp_request_timeout_s = timeout_s
return client
@pytest.mark.asyncio
async def test_send_raw_times_out_on_silent_server():
"""The production TimeoutWrappedCDPClient.send_raw must cap a hung parent
send_raw within the configured timeout.
We deliberately exercise the real `send_raw` (not an inline copy) so
regressions in the wrapper itself — e.g. accidentally removing the
asyncio.wait_for — fail this test.
"""
client = _make_wrapped_client_without_websocket(timeout_s=0.5)
call_count = {'n': 0}
async def _hanging_super_send_raw(self, method, params=None, session_id=None):
call_count['n'] += 1
await asyncio.sleep(30)
return {}
# Patch the parent class's send_raw so TimeoutWrappedCDPClient.send_raw's
# `super().send_raw(...)` call lands on our hanging stub.
with patch('browser_use.browser._cdp_timeout.CDPClient.send_raw', _hanging_super_send_raw):
start = time.monotonic()
with pytest.raises(TimeoutError) as exc:
await client.send_raw('Target.getTargets')
elapsed = time.monotonic() - start
assert call_count['n'] == 1
# Returned within the cap (plus scheduling margin), not after the full 30s.
assert elapsed < 2.0, f'wrapper did not enforce timeout; took {elapsed:.2f}s'
assert 'Target.getTargets' in str(exc.value)
# Error message mentions "within 0s" (0.5 rounded with %.0f) or "within 1s".
assert 'within' in str(exc.value)
@pytest.mark.asyncio
async def test_send_raw_passes_through_when_fast():
"""A parent send_raw that returns quickly should bubble the result up unchanged."""
client = _make_wrapped_client_without_websocket(timeout_s=5.0)
async def _fast_super_send_raw(self, method, params=None, session_id=None):
return {'ok': True, 'method': method}
with patch('browser_use.browser._cdp_timeout.CDPClient.send_raw', _fast_super_send_raw):
result = await client.send_raw('Target.getTargets')
assert result == {'ok': True, 'method': 'Target.getTargets'}
def test_constructor_rejects_invalid_timeout():
"""Non-finite / non-positive constructor args must fall back to the default,
mirroring the env-var path in _parse_env_cdp_timeout."""
# None → default.
assert _coerce_valid_timeout(None) == DEFAULT_CDP_REQUEST_TIMEOUT_S
# Invalid values → default, with a warning.
for bad in (float('nan'), float('inf'), float('-inf'), 0.0, -5.0, -0.01):
assert _coerce_valid_timeout(bad) == DEFAULT_CDP_REQUEST_TIMEOUT_S, f'Expected fallback for {bad!r}, got something else'
# Valid finite positives are preserved.
assert _coerce_valid_timeout(0.1) == 0.1
assert _coerce_valid_timeout(30.0) == 30.0
def test_default_cdp_timeout_is_reasonable():
"""Default must give headroom above typical slow CDP calls but stay below
the 180s agent step_timeout so hangs surface before step-level kills."""
assert 10.0 <= DEFAULT_CDP_REQUEST_TIMEOUT_S <= 120.0, (
f'Default CDP timeout ({DEFAULT_CDP_REQUEST_TIMEOUT_S}s) is outside the sensible 10120s range'
)
def test_parse_env_rejects_malformed_values():
"""Mirrors the defensive parse used for BROWSER_USE_ACTION_TIMEOUT_S."""
for bad in ('', 'nan', 'NaN', 'inf', '-inf', '0', '-5', 'abc'):
assert _parse_env_cdp_timeout(bad) == 60.0, f'Expected fallback for {bad!r}'
# Finite positive values take effect.
assert _parse_env_cdp_timeout('30') == 30.0
assert _parse_env_cdp_timeout('15.5') == 15.5
# None (env var not set) also falls back.
assert _parse_env_cdp_timeout(None) == 60.0
+54
View File
@@ -0,0 +1,54 @@
import json
from pathlib import Path
from browser_use.browser import chrome
def test_macos_profile_path_matches_detected_browser_variant(monkeypatch, tmp_path):
monkeypatch.setattr(chrome.platform, 'system', lambda: 'Darwin')
monkeypatch.setattr(Path, 'home', lambda: tmp_path)
assert chrome.get_chrome_profile_path(
None,
executable_path='/Applications/Chromium.app/Contents/MacOS/Chromium',
) == str(tmp_path / 'Library' / 'Application Support' / 'Chromium')
assert chrome.get_chrome_profile_path(
None,
executable_path='/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary',
) == str(tmp_path / 'Library' / 'Application Support' / 'Google' / 'Chrome Canary')
assert chrome.get_chrome_profile_path(
None,
executable_path='/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
) == str(tmp_path / 'Library' / 'Application Support' / 'Google' / 'Chrome')
def test_list_chrome_profiles_uses_detected_browser_variant(monkeypatch, tmp_path):
monkeypatch.setattr(chrome.platform, 'system', lambda: 'Darwin')
monkeypatch.setattr(Path, 'home', lambda: tmp_path)
monkeypatch.setattr(chrome, 'find_chrome_executable', lambda: '/Applications/Chromium.app/Contents/MacOS/Chromium')
user_data_dir = tmp_path / 'Library' / 'Application Support' / 'Chromium'
user_data_dir.mkdir(parents=True)
(user_data_dir / 'Local State').write_text(
json.dumps({'profile': {'info_cache': {'Profile 1': {'name': 'Work'}}}}),
encoding='utf-8',
)
assert chrome.list_chrome_profiles() == [{'directory': 'Profile 1', 'name': 'Work'}]
def test_list_chrome_profiles_returns_empty_for_unexpected_json_shapes(monkeypatch, tmp_path):
monkeypatch.setattr(chrome.platform, 'system', lambda: 'Darwin')
monkeypatch.setattr(Path, 'home', lambda: tmp_path)
monkeypatch.setattr(chrome, 'find_chrome_executable', lambda: '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome')
user_data_dir = tmp_path / 'Library' / 'Application Support' / 'Google' / 'Chrome'
user_data_dir.mkdir(parents=True)
for payload in (
[],
{'profile': {'info_cache': []}},
{'profile': {'info_cache': {'Default': []}}},
):
(user_data_dir / 'Local State').write_text(json.dumps(payload), encoding='utf-8')
assert chrome.list_chrome_profiles() == []
+183
View File
@@ -0,0 +1,183 @@
"""Tests for coordinate clicking feature.
This feature allows certain models (Claude Sonnet 4, Claude Opus 4, Gemini 3 Pro, browser-use/* models)
to use coordinate-based clicking, while other models only get index-based clicking.
"""
import pytest
from browser_use.tools.service import Tools
from browser_use.tools.views import ClickElementAction, ClickElementActionIndexOnly
class TestCoordinateClickingTools:
"""Test the Tools class coordinate clicking functionality."""
def test_default_coordinate_clicking_disabled(self):
"""By default, coordinate clicking should be disabled."""
tools = Tools()
assert tools._coordinate_clicking_enabled is False
def test_default_uses_index_only_action(self):
"""Default Tools should use ClickElementActionIndexOnly."""
tools = Tools()
click_action = tools.registry.registry.actions.get('click')
assert click_action is not None
assert click_action.param_model == ClickElementActionIndexOnly
def test_default_click_schema_has_only_index(self):
"""Default click action schema should only have index property."""
tools = Tools()
click_action = tools.registry.registry.actions.get('click')
assert click_action is not None
schema = click_action.param_model.model_json_schema()
assert 'index' in schema['properties']
assert 'coordinate_x' not in schema['properties']
assert 'coordinate_y' not in schema['properties']
def test_enable_coordinate_clicking(self):
"""Enabling coordinate clicking should switch to ClickElementAction."""
tools = Tools()
tools.set_coordinate_clicking(True)
assert tools._coordinate_clicking_enabled is True
click_action = tools.registry.registry.actions.get('click')
assert click_action is not None
assert click_action.param_model == ClickElementAction
def test_enabled_click_schema_has_coordinates(self):
"""Enabled click action schema should have index and coordinate properties."""
tools = Tools()
tools.set_coordinate_clicking(True)
click_action = tools.registry.registry.actions.get('click')
assert click_action is not None
schema = click_action.param_model.model_json_schema()
assert 'index' in schema['properties']
assert 'coordinate_x' in schema['properties']
assert 'coordinate_y' in schema['properties']
def test_disable_coordinate_clicking(self):
"""Disabling coordinate clicking should switch back to index-only."""
tools = Tools()
tools.set_coordinate_clicking(True)
tools.set_coordinate_clicking(False)
assert tools._coordinate_clicking_enabled is False
click_action = tools.registry.registry.actions.get('click')
assert click_action is not None
assert click_action.param_model == ClickElementActionIndexOnly
def test_set_coordinate_clicking_idempotent(self):
"""Setting the same value twice should not cause issues."""
tools = Tools()
# Enable twice
tools.set_coordinate_clicking(True)
tools.set_coordinate_clicking(True)
assert tools._coordinate_clicking_enabled is True
# Disable twice
tools.set_coordinate_clicking(False)
tools.set_coordinate_clicking(False)
assert tools._coordinate_clicking_enabled is False
def test_schema_title_consistent(self):
"""Schema title should be 'ClickElementAction' regardless of mode."""
tools = Tools()
# Check default (disabled)
click_action = tools.registry.registry.actions.get('click')
assert click_action is not None
schema = click_action.param_model.model_json_schema()
assert schema['title'] == 'ClickElementAction'
# Check enabled
tools.set_coordinate_clicking(True)
click_action = tools.registry.registry.actions.get('click')
assert click_action is not None
schema = click_action.param_model.model_json_schema()
assert schema['title'] == 'ClickElementAction'
class TestCoordinateClickingModelDetection:
"""Test the model detection logic for coordinate clicking."""
@pytest.mark.parametrize(
'model_name,expected_coords',
[
# Models that SHOULD have coordinate clicking (claude-sonnet-4*, claude-opus-4*, gemini-3-pro*, browser-use/*)
('claude-sonnet-4-5', True),
('claude-sonnet-4-5-20250101', True),
('claude-sonnet-4-0', True),
('claude-sonnet-4', True),
('claude-opus-4-5', True),
('claude-opus-4-5-latest', True),
('claude-opus-4-0', True),
('claude-opus-4', True),
('gemini-3-pro-preview', True),
('gemini-3-pro', True),
('browser-use/fast', True),
('browser-use/accurate', True),
('CLAUDE-SONNET-4-5', True), # Case insensitive
('CLAUDE-SONNET-4', True), # Case insensitive
('GEMINI-3-PRO', True), # Case insensitive
# Models that should NOT have coordinate clicking
('claude-3-5-sonnet', False),
('claude-sonnet-3-5', False),
('gpt-4o', False),
('gpt-4-turbo', False),
('gemini-2.0-flash', False),
('gemini-1.5-pro', False),
('llama-3.1-70b', False),
('mistral-large', False),
],
)
def test_model_detection_patterns(self, model_name: str, expected_coords: bool):
"""Test that the model detection patterns correctly identify coordinate-capable models."""
model_lower = model_name.lower()
supports_coords = any(
pattern in model_lower for pattern in ['claude-sonnet-4', 'claude-opus-4', 'gemini-3-pro', 'browser-use/']
)
assert supports_coords == expected_coords, f'Model {model_name}: expected {expected_coords}, got {supports_coords}'
class TestCoordinateClickingWithPassedTools:
"""Test that coordinate clicking works correctly when Tools is passed to Agent."""
def test_tools_can_be_modified_after_creation(self):
"""Tools created externally can have coordinate clicking enabled."""
tools = Tools()
assert tools._coordinate_clicking_enabled is False
# Simulate what Agent does for coordinate-capable models
tools.set_coordinate_clicking(True)
click_action = tools.registry.registry.actions.get('click')
assert click_action is not None
assert click_action.param_model == ClickElementAction
def test_tools_state_preserved_after_modification(self):
"""Verify that other tool state is preserved when toggling coordinate clicking."""
tools = Tools(exclude_actions=['search'])
# Search should be excluded
assert 'search' not in tools.registry.registry.actions
# Enable coordinate clicking
tools.set_coordinate_clicking(True)
# Search should still be excluded
assert 'search' not in tools.registry.registry.actions
# Click should have coordinates
click_action = tools.registry.registry.actions.get('click')
assert click_action is not None
assert click_action.param_model == ClickElementAction
+103
View File
@@ -0,0 +1,103 @@
"""Tests for DomService.is_element_visible_according_to_all_parents.
Regression tests for the in-place mutation of shared snapshot bounds: the
visibility check must never modify node.snapshot_node.bounds, and a frame
node must not be transformed by its own bounds when it appears in its own
frame chain (as built by _construct_enhanced_node).
"""
from browser_use.dom.service import DomService
from browser_use.dom.views import DOMRect, EnhancedDOMTreeNode, EnhancedSnapshotNode, NodeType
def _make_snapshot(
bounds: DOMRect | None = None,
client_rects: DOMRect | None = None,
scroll_rects: DOMRect | None = None,
) -> EnhancedSnapshotNode:
return EnhancedSnapshotNode(
is_clickable=None,
cursor_style=None,
bounds=bounds,
clientRects=client_rects,
scrollRects=scroll_rects,
computed_styles={},
paint_order=None,
stacking_contexts=None,
)
def _make_node(node_name: str, snapshot_node: EnhancedSnapshotNode | None) -> EnhancedDOMTreeNode:
return EnhancedDOMTreeNode(
node_id=1,
backend_node_id=1,
node_type=NodeType.ELEMENT_NODE,
node_name=node_name,
node_value='',
attributes={},
is_scrollable=None,
is_visible=None,
absolute_position=None,
target_id='test-target',
frame_id=None,
session_id=None,
content_document=None,
shadow_root_type=None,
shadow_roots=None,
parent_node=None,
children_nodes=None,
ax_node=None,
snapshot_node=snapshot_node,
)
def _make_html_frame(viewport_width: float = 1280, viewport_height: float = 900, scroll_y: float = 0) -> EnhancedDOMTreeNode:
return _make_node(
'HTML',
_make_snapshot(
bounds=DOMRect(x=0, y=0, width=viewport_width, height=viewport_height),
client_rects=DOMRect(x=0, y=0, width=viewport_width, height=viewport_height),
scroll_rects=DOMRect(x=0, y=scroll_y, width=viewport_width, height=5000),
),
)
class TestVisibilityDoesNotMutateBounds:
def test_bounds_unchanged_after_visibility_check(self):
"""The check must not modify the shared snapshot bounds object."""
html = _make_html_frame(scroll_y=100)
element = _make_node('DIV', _make_snapshot(bounds=DOMRect(x=10, y=500, width=100, height=50)))
DomService.is_element_visible_according_to_all_parents(element, [html], viewport_threshold=1000)
assert element.snapshot_node is not None and element.snapshot_node.bounds is not None
assert element.snapshot_node.bounds.x == 10
assert element.snapshot_node.bounds.y == 500
def test_visibility_check_is_idempotent(self):
"""Calling the check twice must return the same result (fails if bounds drift)."""
html = _make_html_frame(viewport_height=900, scroll_y=1000)
# Element exactly inside the top threshold window: with in-place mutation
# the first check returns True, then the drifted second check returns False.
element = _make_node('DIV', _make_snapshot(bounds=DOMRect(x=10, y=0, width=100, height=50)))
first = DomService.is_element_visible_according_to_all_parents(element, [html], viewport_threshold=1000)
second = DomService.is_element_visible_according_to_all_parents(element, [html], viewport_threshold=1000)
assert first is True
assert second is True
def test_iframe_not_double_transformed_by_own_bounds(self):
"""An iframe checked against a frame chain containing itself must not have its
coordinates doubled (y=1200 on a 900px viewport with threshold 1000 is visible;
doubled to 2400 it would not be)."""
html = _make_html_frame(viewport_height=900, scroll_y=0)
iframe = _make_node('IFRAME', _make_snapshot(bounds=DOMRect(x=0, y=1200, width=600, height=400)))
# _construct_enhanced_node appends the iframe to its own frame chain before
# computing its visibility — reproduce that here.
visible = DomService.is_element_visible_according_to_all_parents(iframe, [html, iframe], viewport_threshold=1000)
assert visible is True
assert iframe.snapshot_node is not None and iframe.snapshot_node.bounds is not None
assert iframe.snapshot_node.bounds.y == 1200
+51
View File
@@ -0,0 +1,51 @@
from browser_use.browser.watchdogs.downloads_watchdog import _should_auto_download_network_response
def test_downloads_watchdog_skips_generic_text_attachment_without_file_url():
assert not _should_auto_download_network_response(
url='https://www.google.com/complete/search?q=test&client=gws-wiz&xssi=t',
content_type='text/plain',
is_pdf=False,
is_download_attachment=True,
suggested_filename='f.txt',
)
def test_downloads_watchdog_keeps_pdf_network_response():
assert _should_auto_download_network_response(
url='https://example.com/view?id=123',
content_type='application/pdf',
is_pdf=True,
is_download_attachment=False,
suggested_filename=None,
)
def test_downloads_watchdog_keeps_named_file_attachment():
assert _should_auto_download_network_response(
url='https://example.com/download?id=123',
content_type='text/csv',
is_pdf=False,
is_download_attachment=True,
suggested_filename='report.csv',
)
def test_downloads_watchdog_keeps_text_attachment_with_file_url():
assert _should_auto_download_network_response(
url='https://example.com/files/summary.txt?download=1',
content_type='text/plain',
is_pdf=False,
is_download_attachment=True,
suggested_filename='f.txt',
)
def test_downloads_watchdog_keeps_attachment_without_known_extension():
assert _should_auto_download_network_response(
url='https://example.com/download?id=123',
content_type='application/vnd.example.custom',
is_pdf=False,
is_download_attachment=True,
suggested_filename='statement',
)
+85
View File
@@ -0,0 +1,85 @@
"""Regression tests for ENG-5280: V2 worker crash on warm-Lambda resume.
A reused keep_alive session bus is stopped and nulled by Agent.close(); on resume the worker
can step() it before a dispatch() restarts it, which makes stock bubus assert. ResilientEventBus
no-ops step()/wait_until_idle() in that state while still restarting on the next dispatch().
"""
import asyncio
from bubus import BaseEvent, EventBus
from browser_use.browser import BrowserProfile, BrowserSession
from browser_use.browser.session import ResilientEventBus
class ResiliencePingEvent(BaseEvent):
pass
def _tear_down_like_agent_close(bus: EventBus) -> None:
"""Mimic the keep_alive teardown in Agent.close() that triggers ENG-5280."""
bus.event_queue = None
bus._on_idle = None
def test_browser_session_uses_resilient_event_bus():
"""The session's default event bus must be the resilient subclass."""
session = BrowserSession(browser_profile=BrowserProfile(keep_alive=True))
assert isinstance(session.event_bus, ResilientEventBus)
def test_resilient_event_bus_keeps_event_bus_name_prefix():
"""The subclass must keep the ``EventBus_`` default name (not ``ResilientEventBus_``)."""
assert ResilientEventBus().name.startswith('EventBus_')
# Explicit names are still honored.
assert ResilientEventBus(name='EventBus_custom').name == 'EventBus_custom'
async def test_step_on_torn_down_bus_is_noop_not_assertion():
"""Stepping a stopped+nulled bus returns None instead of raising AssertionError."""
bus = ResilientEventBus(name='ResilientWarmResumeStep')
bus.dispatch(ResiliencePingEvent())
await asyncio.sleep(0.1)
await bus.stop(clear=False, timeout=1.0)
_tear_down_like_agent_close(bus)
# Warm-Lambda resume: worker steps the torn-down bus. Must not raise.
assert await bus.step() is None
assert await bus.wait_until_idle(timeout=0.1) is None
async def test_torn_down_bus_still_restarts_on_dispatch():
"""A later dispatch() must recreate the queue and process events normally."""
processed: list[BaseEvent] = []
bus = ResilientEventBus(name='ResilientWarmResumeRestart')
bus.on('ResiliencePingEvent', lambda event: processed.append(event))
bus.dispatch(ResiliencePingEvent())
await asyncio.sleep(0.1)
await bus.stop(clear=False, timeout=1.0)
_tear_down_like_agent_close(bus)
# No-op step on the torn-down bus, then dispatch again -> bus restarts.
assert await bus.step() is None
processed.clear()
event = bus.dispatch(ResiliencePingEvent())
await asyncio.wait_for(event, timeout=2.0)
assert len(processed) == 1
await bus.stop(timeout=0.5)
async def test_stock_event_bus_reproduces_the_crash():
"""Document the upstream bug the subclass guards against (stock bus still asserts)."""
bus = EventBus(name='StockWarmResume')
bus.dispatch(ResiliencePingEvent())
await asyncio.sleep(0.1)
await bus.stop(clear=False, timeout=1.0)
_tear_down_like_agent_close(bus)
try:
await bus.step()
raised = False
except AssertionError:
raised = True
assert raised, 'stock EventBus.step() should assert on a torn-down bus'
+123
View File
@@ -0,0 +1,123 @@
"""Tests for extension configuration environment variables."""
import os
import pytest
class TestDisableExtensionsEnvVar:
"""Test BROWSER_USE_DISABLE_EXTENSIONS environment variable."""
def test_default_value_is_true(self):
"""Without env var set, enable_default_extensions should default to True."""
# Clear the env var if it exists
original = os.environ.pop('BROWSER_USE_DISABLE_EXTENSIONS', None)
try:
# Import fresh to get the default
from browser_use.browser.profile import _get_enable_default_extensions_default
assert _get_enable_default_extensions_default() is True
finally:
if original is not None:
os.environ['BROWSER_USE_DISABLE_EXTENSIONS'] = original
@pytest.mark.parametrize(
'env_value,expected_enabled',
[
# Truthy values for DISABLE = extensions disabled (False)
('true', False),
('True', False),
('TRUE', False),
('1', False),
('yes', False),
('on', False),
# Falsy values for DISABLE = extensions enabled (True)
('false', True),
('False', True),
('FALSE', True),
('0', True),
('no', True),
('off', True),
('', True),
],
)
def test_env_var_values(self, env_value: str, expected_enabled: bool):
"""Test various env var values are parsed correctly."""
original = os.environ.get('BROWSER_USE_DISABLE_EXTENSIONS')
try:
os.environ['BROWSER_USE_DISABLE_EXTENSIONS'] = env_value
from browser_use.browser.profile import _get_enable_default_extensions_default
result = _get_enable_default_extensions_default()
assert result is expected_enabled, (
f"Expected enable_default_extensions={expected_enabled} for DISABLE_EXTENSIONS='{env_value}', got {result}"
)
finally:
if original is not None:
os.environ['BROWSER_USE_DISABLE_EXTENSIONS'] = original
else:
os.environ.pop('BROWSER_USE_DISABLE_EXTENSIONS', None)
def test_browser_profile_uses_env_var(self):
"""Test that BrowserProfile picks up the env var."""
original = os.environ.get('BROWSER_USE_DISABLE_EXTENSIONS')
try:
# Test with env var set to true (disable extensions)
os.environ['BROWSER_USE_DISABLE_EXTENSIONS'] = 'true'
from browser_use.browser.profile import BrowserProfile
profile = BrowserProfile(headless=True)
assert profile.enable_default_extensions is False, (
'BrowserProfile should disable extensions when BROWSER_USE_DISABLE_EXTENSIONS=true'
)
# Test with env var set to false (enable extensions)
os.environ['BROWSER_USE_DISABLE_EXTENSIONS'] = 'false'
profile2 = BrowserProfile(headless=True)
assert profile2.enable_default_extensions is True, (
'BrowserProfile should enable extensions when BROWSER_USE_DISABLE_EXTENSIONS=false'
)
finally:
if original is not None:
os.environ['BROWSER_USE_DISABLE_EXTENSIONS'] = original
else:
os.environ.pop('BROWSER_USE_DISABLE_EXTENSIONS', None)
def test_explicit_param_overrides_env_var(self):
"""Test that explicit enable_default_extensions parameter overrides env var."""
original = os.environ.get('BROWSER_USE_DISABLE_EXTENSIONS')
try:
os.environ['BROWSER_USE_DISABLE_EXTENSIONS'] = 'true'
from browser_use.browser.profile import BrowserProfile
# Explicitly set to True should override env var
profile = BrowserProfile(headless=True, enable_default_extensions=True)
assert profile.enable_default_extensions is True, 'Explicit param should override env var'
finally:
if original is not None:
os.environ['BROWSER_USE_DISABLE_EXTENSIONS'] = original
else:
os.environ.pop('BROWSER_USE_DISABLE_EXTENSIONS', None)
def test_browser_session_uses_env_var(self):
"""Test that BrowserSession picks up the env var via BrowserProfile."""
original = os.environ.get('BROWSER_USE_DISABLE_EXTENSIONS')
try:
os.environ['BROWSER_USE_DISABLE_EXTENSIONS'] = '1'
from browser_use.browser import BrowserSession
session = BrowserSession(headless=True)
assert session.browser_profile.enable_default_extensions is False, (
'BrowserSession should disable extensions when BROWSER_USE_DISABLE_EXTENSIONS=1'
)
finally:
if original is not None:
os.environ['BROWSER_USE_DISABLE_EXTENSIONS'] = original
else:
os.environ.pop('BROWSER_USE_DISABLE_EXTENSIONS', None)
+296
View File
@@ -0,0 +1,296 @@
"""Tests for extract_images support in extract_clean_markdown.
Root cause of AGI-101: markdownify strips img src URLs when images appear inside table cells
(<td>/<th>) or heading elements, because those contexts set the _inline flag. The extract_images
parameter fixes this by adding those tags to keep_inline_images_in.
Block-level images (direct children of <div>, <figure>, etc.) are ALWAYS included in markdown
regardless of extract_images. The parameter only matters for images inside <td>, <th>, <h1>-<h6>.
"""
import asyncio
import pytest
from pytest_httpserver import HTTPServer
from browser_use.browser import BrowserProfile, BrowserSession
from browser_use.dom.markdown_extractor import extract_clean_markdown
# --- Fixtures ---
@pytest.fixture(scope='session')
def http_server():
"""Test HTTP server serving pages with product images."""
server = HTTPServer()
server.start()
# Table-layout products — images are in <td>, the actual bug scenario.
# With extract_images=False (default), img in td becomes just alt text (no URL).
# With extract_images=True, img in td becomes ![alt](url) with the real URL.
server.expect_request('/products-table').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head><title>Products Table</title></head>
<body>
<h1>Product Catalog</h1>
<table>
<thead>
<tr><th>Image</th><th>Name</th><th>Price</th></tr>
</thead>
<tbody>
<tr>
<td><img src="http://localhost/images/widget-a.jpg" alt="Widget A"></td>
<td>Widget A</td>
<td>$29.99</td>
</tr>
<tr>
<td><img src="http://localhost/images/widget-b.jpg" alt="Widget B"></td>
<td>Widget B</td>
<td>$49.99</td>
</tr>
<tr>
<td><img src="http://localhost/images/gadget-c.png" alt="Gadget C"></td>
<td>Gadget C</td>
<td>$19.50</td>
</tr>
</tbody>
</table>
</body>
</html>
""",
content_type='text/html',
)
# Block-level products — images in <div>/<figure> are ALWAYS included in markdown,
# regardless of extract_images value.
server.expect_request('/products-block').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head><title>Products Block</title></head>
<body>
<div class="product">
<img src="http://localhost/images/widget-a.jpg" alt="Widget A">
<p>Widget A - $29.99</p>
</div>
</body>
</html>
""",
content_type='text/html',
)
server.expect_request('/text-only').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head><title>Text Only</title></head>
<body>
<h1>No Images Here</h1>
<p>Just some text content with no images at all.</p>
</body>
</html>
""",
content_type='text/html',
)
yield server
server.stop()
@pytest.fixture(scope='session')
def base_url(http_server):
return f'http://{http_server.host}:{http_server.port}'
@pytest.fixture(scope='module')
async def browser_session():
session = BrowserSession(
browser_profile=BrowserProfile(
headless=True,
user_data_dir=None,
keep_alive=True,
)
)
await session.start()
yield session
await session.kill()
# --- Helper ---
async def _navigate(browser_session, url: str):
"""Navigate to URL and wait for page load."""
await browser_session.navigate_to(url)
await asyncio.sleep(0.5)
# --- Tests ---
class TestExtractCleanMarkdown:
"""Tests for extract_clean_markdown with extract_images parameter."""
async def test_table_images_excluded_by_default(self, browser_session, base_url):
"""Images inside <td> lose their URL with extract_images=False (default).
This is the AGI-101 root cause: markdownify strips img src in _inline contexts
(td/th/headings) when keep_inline_images_in=[]. The alt text is kept but not the URL.
"""
await _navigate(browser_session, f'{base_url}/products-table')
content, _ = await extract_clean_markdown(browser_session=browser_session, extract_images=False)
# Alt text (product names) should still be present
assert 'Widget A' in content
assert 'Widget B' in content
# But image URLs should NOT appear — img in td is stripped to alt text
assert 'widget-a.jpg' not in content
assert 'widget-b.jpg' not in content
assert 'gadget-c.png' not in content
# No markdown image syntax
assert '![' not in content
async def test_table_images_included_when_enabled(self, browser_session, base_url):
"""Images inside <td> include their URL with extract_images=True."""
await _navigate(browser_session, f'{base_url}/products-table')
content, _ = await extract_clean_markdown(browser_session=browser_session, extract_images=True)
# Image markdown syntax SHOULD be present for td-context images
assert '![' in content
# At least one product image URL should appear
assert 'widget-a.jpg' in content or 'widget-b.jpg' in content or 'gadget-c.png' in content
async def test_block_images_always_included(self, browser_session, base_url):
"""Block-level images (in <div>, <figure>) are always included, extract_images has no effect."""
await _navigate(browser_session, f'{base_url}/products-block')
content_false, _ = await extract_clean_markdown(browser_session=browser_session, extract_images=False)
content_true, _ = await extract_clean_markdown(browser_session=browser_session, extract_images=True)
# Block-level images are always converted to ![alt](src) regardless
assert '![' in content_false
assert 'widget-a.jpg' in content_false
assert '![' in content_true
assert 'widget-a.jpg' in content_true
async def test_false_is_default(self, browser_session, base_url):
"""Calling extract_clean_markdown without extract_images behaves same as extract_images=False."""
await _navigate(browser_session, f'{base_url}/products-table')
content_default, _ = await extract_clean_markdown(browser_session=browser_session)
content_false, _ = await extract_clean_markdown(browser_session=browser_session, extract_images=False)
assert content_default == content_false
async def test_no_images_on_text_only_page(self, browser_session, base_url):
"""extract_images=True on a page with no images returns no image markdown."""
await _navigate(browser_session, f'{base_url}/text-only')
content, _ = await extract_clean_markdown(browser_session=browser_session, extract_images=True)
assert '![' not in content
assert 'No Images Here' in content or 'text content' in content
class TestExtractImagesAutoDetection:
"""Tests for auto-detection of image-related queries in the extract action."""
async def test_auto_detect_image_url_query(self, browser_session, base_url, tmp_path):
"""Query containing 'image url' auto-enables extract_images: table-cell img URLs appear in LLM input."""
from unittest.mock import AsyncMock
from browser_use.filesystem.file_system import FileSystem
from browser_use.llm import BaseChatModel
from browser_use.llm.views import ChatInvokeCompletion
from browser_use.tools.service import Tools
await _navigate(browser_session, f'{base_url}/products-table')
captured_content: list[str] = []
mock_llm = AsyncMock(spec=BaseChatModel)
mock_llm.model = 'mock-llm'
mock_llm._verified_api_keys = True
mock_llm.provider = 'mock'
mock_llm.name = 'mock-llm'
mock_llm.model_name = 'mock-llm'
async def capture_ainvoke(*args, **kwargs):
if args:
for msg in args[0]:
content = getattr(msg, 'content', '')
if isinstance(content, str):
captured_content.append(content)
elif isinstance(content, list):
for part in content:
if isinstance(part, dict) and part.get('type') == 'text':
captured_content.append(part.get('text', ''))
return ChatInvokeCompletion(completion='Widget A image: http://localhost/images/widget-a.jpg', usage=None)
mock_llm.ainvoke.side_effect = capture_ainvoke
tools = Tools()
await tools.extract(
query='get image url for each product',
browser_session=browser_session,
page_extraction_llm=mock_llm,
file_system=FileSystem(base_dir=str(tmp_path)),
)
# The LLM should have received content that includes image markdown (td images with URLs)
all_content = ' '.join(captured_content)
assert '![' in all_content or 'widget-a.jpg' in all_content or 'widget-b.jpg' in all_content, (
f'Expected image URLs in LLM input but got: {all_content[:500]}'
)
async def test_no_auto_detect_without_image_keyword(self, browser_session, base_url, tmp_path):
"""Query without image keywords does NOT auto-enable extract_images: table-cell img URLs absent."""
from unittest.mock import AsyncMock
from browser_use.filesystem.file_system import FileSystem
from browser_use.llm import BaseChatModel
from browser_use.llm.views import ChatInvokeCompletion
from browser_use.tools.service import Tools
await _navigate(browser_session, f'{base_url}/products-table')
captured_content: list[str] = []
mock_llm = AsyncMock(spec=BaseChatModel)
mock_llm.model = 'mock-llm'
mock_llm._verified_api_keys = True
mock_llm.provider = 'mock'
mock_llm.name = 'mock-llm'
mock_llm.model_name = 'mock-llm'
async def capture_ainvoke(*args, **kwargs):
if args:
for msg in args[0]:
content = getattr(msg, 'content', '')
if isinstance(content, str):
captured_content.append(content)
elif isinstance(content, list):
for part in content:
if isinstance(part, dict) and part.get('type') == 'text':
captured_content.append(part.get('text', ''))
return ChatInvokeCompletion(completion='Widget A - $29.99, Widget B - $49.99', usage=None)
mock_llm.ainvoke.side_effect = capture_ainvoke
tools = Tools()
await tools.extract(
query='get product names and prices',
browser_session=browser_session,
page_extraction_llm=mock_llm,
file_system=FileSystem(base_dir=str(tmp_path)),
)
# Table-cell image URLs should NOT appear (extract_images=False, no auto-detect)
all_content = ' '.join(captured_content)
assert 'widget-a.jpg' not in all_content and 'widget-b.jpg' not in all_content, (
f'Did not expect image URLs in LLM input but got: {all_content[:500]}'
)
+439
View File
@@ -0,0 +1,439 @@
"""
Tests for the fallback_llm feature in Agent.
Tests verify that when the primary LLM fails with rate limit (429) or server errors (503, 502, 500, 504),
the agent automatically switches to the fallback LLM and continues execution.
"""
from unittest.mock import AsyncMock
import pytest
from browser_use.agent.views import AgentOutput
from browser_use.llm import BaseChatModel
from browser_use.llm.exceptions import ModelProviderError, ModelRateLimitError
from browser_use.llm.views import ChatInvokeCompletion
from browser_use.tools.service import Tools
def create_mock_llm(
model_name: str = 'mock-llm',
should_fail: bool = False,
fail_with: type[Exception] | None = None,
fail_status_code: int = 429,
fail_message: str = 'Rate limit exceeded',
) -> BaseChatModel:
"""Create a mock LLM for testing.
Args:
model_name: Name of the mock model
should_fail: If True, the LLM will raise an exception
fail_with: Exception type to raise (ModelRateLimitError or ModelProviderError)
fail_status_code: HTTP status code for the error
fail_message: Error message
"""
tools = Tools()
ActionModel = tools.registry.create_action_model()
AgentOutputWithActions = AgentOutput.type_with_custom_actions(ActionModel)
llm = AsyncMock(spec=BaseChatModel)
llm.model = model_name
llm._verified_api_keys = True
llm.provider = 'mock'
llm.name = model_name
llm.model_name = model_name
default_done_action = """
{
"thinking": "null",
"evaluation_previous_goal": "Successfully completed the task",
"memory": "Task completed",
"next_goal": "Task completed",
"action": [
{
"done": {
"text": "Task completed successfully",
"success": true
}
}
]
}
"""
async def mock_ainvoke(*args, **kwargs):
if should_fail:
if fail_with == ModelRateLimitError:
raise ModelRateLimitError(message=fail_message, status_code=fail_status_code, model=model_name)
elif fail_with == ModelProviderError:
raise ModelProviderError(message=fail_message, status_code=fail_status_code, model=model_name)
else:
raise Exception(fail_message)
output_format = kwargs.get('output_format')
if output_format is None:
return ChatInvokeCompletion(completion=default_done_action, usage=None)
else:
parsed = output_format.model_validate_json(default_done_action)
return ChatInvokeCompletion(completion=parsed, usage=None)
llm.ainvoke.side_effect = mock_ainvoke
return llm
class TestFallbackLLMParameter:
"""Test fallback_llm parameter initialization."""
def test_fallback_llm_none_by_default(self):
"""Verify fallback_llm defaults to None."""
from browser_use import Agent
primary = create_mock_llm('primary-model')
agent = Agent(task='Test task', llm=primary)
assert agent._fallback_llm is None
assert agent._using_fallback_llm is False
assert agent._original_llm is primary
def test_fallback_llm_single_model(self):
"""Test passing a fallback LLM."""
from browser_use import Agent
primary = create_mock_llm('primary-model')
fallback = create_mock_llm('fallback-model')
agent = Agent(task='Test task', llm=primary, fallback_llm=fallback)
assert agent._fallback_llm is fallback
assert agent._using_fallback_llm is False
def test_public_properties(self):
"""Test the public properties for fallback status."""
from browser_use import Agent
primary = create_mock_llm('primary-model')
fallback = create_mock_llm('fallback-model')
agent = Agent(task='Test task', llm=primary, fallback_llm=fallback)
# Before fallback
assert agent.is_using_fallback_llm is False
assert agent.current_llm_model == 'primary-model'
# Trigger fallback
error = ModelRateLimitError(message='Rate limit', status_code=429, model='primary')
agent._try_switch_to_fallback_llm(error)
# After fallback
assert agent.is_using_fallback_llm is True
assert agent.current_llm_model == 'fallback-model'
class TestFallbackLLMSwitching:
"""Test the fallback switching logic in _try_switch_to_fallback_llm."""
def test_switch_on_rate_limit_error(self):
"""Test that agent switches to fallback on ModelRateLimitError."""
from browser_use import Agent
primary = create_mock_llm('primary-model')
fallback = create_mock_llm('fallback-model')
agent = Agent(task='Test task', llm=primary, fallback_llm=fallback)
error = ModelRateLimitError(message='Rate limit exceeded', status_code=429, model='primary-model')
result = agent._try_switch_to_fallback_llm(error)
assert result is True
assert agent.llm is fallback
assert agent._using_fallback_llm is True
def test_switch_on_503_error(self):
"""Test that agent switches to fallback on 503 Service Unavailable."""
from browser_use import Agent
primary = create_mock_llm('primary-model')
fallback = create_mock_llm('fallback-model')
agent = Agent(task='Test task', llm=primary, fallback_llm=fallback)
error = ModelProviderError(message='Service unavailable', status_code=503, model='primary-model')
result = agent._try_switch_to_fallback_llm(error)
assert result is True
assert agent.llm is fallback
assert agent._using_fallback_llm is True
def test_switch_on_500_error(self):
"""Test that agent switches to fallback on 500 Internal Server Error."""
from browser_use import Agent
primary = create_mock_llm('primary-model')
fallback = create_mock_llm('fallback-model')
agent = Agent(task='Test task', llm=primary, fallback_llm=fallback)
error = ModelProviderError(message='Internal server error', status_code=500, model='primary-model')
result = agent._try_switch_to_fallback_llm(error)
assert result is True
assert agent.llm is fallback
def test_switch_on_502_error(self):
"""Test that agent switches to fallback on 502 Bad Gateway."""
from browser_use import Agent
primary = create_mock_llm('primary-model')
fallback = create_mock_llm('fallback-model')
agent = Agent(task='Test task', llm=primary, fallback_llm=fallback)
error = ModelProviderError(message='Bad gateway', status_code=502, model='primary-model')
result = agent._try_switch_to_fallback_llm(error)
assert result is True
assert agent.llm is fallback
def test_no_switch_on_400_error(self):
"""Test that agent does NOT switch on 400 Bad Request (not retryable)."""
from browser_use import Agent
primary = create_mock_llm('primary-model')
fallback = create_mock_llm('fallback-model')
agent = Agent(task='Test task', llm=primary, fallback_llm=fallback)
error = ModelProviderError(message='Bad request', status_code=400, model='primary-model')
result = agent._try_switch_to_fallback_llm(error)
assert result is False
assert agent.llm is primary # Still using primary
assert agent._using_fallback_llm is False
def test_switch_on_401_error(self):
"""Test that agent switches to fallback on 401 Unauthorized (API key error)."""
from browser_use import Agent
primary = create_mock_llm('primary-model')
fallback = create_mock_llm('fallback-model')
agent = Agent(task='Test task', llm=primary, fallback_llm=fallback)
error = ModelProviderError(message='Invalid API key', status_code=401, model='primary-model')
result = agent._try_switch_to_fallback_llm(error)
assert result is True
assert agent.llm is fallback
assert agent._using_fallback_llm is True
def test_switch_on_402_error(self):
"""Test that agent switches to fallback on 402 Payment Required (insufficient credits)."""
from browser_use import Agent
primary = create_mock_llm('primary-model')
fallback = create_mock_llm('fallback-model')
agent = Agent(task='Test task', llm=primary, fallback_llm=fallback)
error = ModelProviderError(message='Insufficient credits', status_code=402, model='primary-model')
result = agent._try_switch_to_fallback_llm(error)
assert result is True
assert agent.llm is fallback
assert agent._using_fallback_llm is True
def test_no_switch_when_no_fallback_configured(self):
"""Test that agent returns False when no fallback is configured."""
from browser_use import Agent
primary = create_mock_llm('primary-model')
agent = Agent(task='Test task', llm=primary)
error = ModelRateLimitError(message='Rate limit exceeded', status_code=429, model='primary-model')
result = agent._try_switch_to_fallback_llm(error)
assert result is False
assert agent.llm is primary
def test_no_switch_when_already_using_fallback(self):
"""Test that agent doesn't switch again when already using fallback."""
from browser_use import Agent
primary = create_mock_llm('primary-model')
fallback = create_mock_llm('fallback-model')
agent = Agent(task='Test task', llm=primary, fallback_llm=fallback)
# First switch succeeds
error = ModelRateLimitError(message='Rate limit', status_code=429, model='primary')
result = agent._try_switch_to_fallback_llm(error)
assert result is True
assert agent.llm is fallback
# Second switch fails - already using fallback
result = agent._try_switch_to_fallback_llm(error)
assert result is False
assert agent.llm is fallback # Still on fallback
class TestFallbackLLMIntegration:
"""Integration tests for fallback LLM behavior in get_model_output."""
def _create_failing_mock_llm(
self,
model_name: str,
fail_with: type[Exception],
fail_status_code: int = 429,
fail_message: str = 'Rate limit exceeded',
) -> BaseChatModel:
"""Create a mock LLM that always fails with the specified error."""
llm = AsyncMock(spec=BaseChatModel)
llm.model = model_name
llm._verified_api_keys = True
llm.provider = 'mock'
llm.name = model_name
llm.model_name = model_name
async def mock_ainvoke(*args, **kwargs):
if fail_with == ModelRateLimitError:
raise ModelRateLimitError(message=fail_message, status_code=fail_status_code, model=model_name)
elif fail_with == ModelProviderError:
raise ModelProviderError(message=fail_message, status_code=fail_status_code, model=model_name)
else:
raise Exception(fail_message)
llm.ainvoke.side_effect = mock_ainvoke
return llm
def _create_succeeding_mock_llm(self, model_name: str, agent) -> BaseChatModel:
"""Create a mock LLM that succeeds and returns a valid AgentOutput."""
llm = AsyncMock(spec=BaseChatModel)
llm.model = model_name
llm._verified_api_keys = True
llm.provider = 'mock'
llm.name = model_name
llm.model_name = model_name
default_done_action = """
{
"thinking": "null",
"evaluation_previous_goal": "Successfully completed the task",
"memory": "Task completed",
"next_goal": "Task completed",
"action": [
{
"done": {
"text": "Task completed successfully",
"success": true
}
}
]
}
"""
# Capture the agent reference for use in the closure
captured_agent = agent
async def mock_ainvoke(*args, **kwargs):
# Get the output format from kwargs and use it to parse
output_format = kwargs.get('output_format')
if output_format is not None:
parsed = output_format.model_validate_json(default_done_action)
return ChatInvokeCompletion(completion=parsed, usage=None)
# Fallback: use the agent's AgentOutput type
parsed = captured_agent.AgentOutput.model_validate_json(default_done_action)
return ChatInvokeCompletion(completion=parsed, usage=None)
llm.ainvoke.side_effect = mock_ainvoke
return llm
@pytest.mark.asyncio
async def test_get_model_output_switches_to_fallback_on_rate_limit(self, browser_session):
"""Test that get_model_output automatically switches to fallback on rate limit."""
from browser_use import Agent
# Create agent first with a working mock LLM
placeholder = create_mock_llm('placeholder')
agent = Agent(task='Test task', llm=placeholder, browser_session=browser_session)
# Create a failing primary and succeeding fallback
primary = self._create_failing_mock_llm(
'primary-model',
fail_with=ModelRateLimitError,
fail_status_code=429,
fail_message='Rate limit exceeded',
)
fallback = self._create_succeeding_mock_llm('fallback-model', agent)
# Replace the LLM and set up fallback
agent.llm = primary
agent._original_llm = primary
agent._fallback_llm = fallback
from browser_use.llm.messages import BaseMessage, UserMessage
messages: list[BaseMessage] = [UserMessage(content='Test message')]
# This should switch to fallback and succeed
result = await agent.get_model_output(messages)
assert result is not None
assert agent.llm is fallback
assert agent._using_fallback_llm is True
@pytest.mark.asyncio
async def test_get_model_output_raises_when_no_fallback(self, browser_session):
"""Test that get_model_output raises error when no fallback is configured."""
from browser_use import Agent
# Create agent first with a working mock LLM
placeholder = create_mock_llm('placeholder')
agent = Agent(task='Test task', llm=placeholder, browser_session=browser_session)
# Replace with failing LLM
primary = self._create_failing_mock_llm(
'primary-model',
fail_with=ModelRateLimitError,
fail_status_code=429,
fail_message='Rate limit exceeded',
)
agent.llm = primary
agent._original_llm = primary
agent._fallback_llm = None # No fallback
from browser_use.llm.messages import BaseMessage, UserMessage
messages: list[BaseMessage] = [UserMessage(content='Test message')]
# This should raise since no fallback is configured
with pytest.raises(ModelRateLimitError):
await agent.get_model_output(messages)
@pytest.mark.asyncio
async def test_get_model_output_raises_when_fallback_also_fails(self, browser_session):
"""Test that error is raised when fallback also fails."""
from browser_use import Agent
# Create agent first with a working mock LLM
placeholder = create_mock_llm('placeholder')
agent = Agent(task='Test task', llm=placeholder, browser_session=browser_session)
# Both models fail
primary = self._create_failing_mock_llm('primary', fail_with=ModelRateLimitError, fail_status_code=429)
fallback = self._create_failing_mock_llm('fallback', fail_with=ModelProviderError, fail_status_code=503)
agent.llm = primary
agent._original_llm = primary
agent._fallback_llm = fallback
from browser_use.llm.messages import BaseMessage, UserMessage
messages: list[BaseMessage] = [UserMessage(content='Test message')]
# Should fail after fallback also fails
with pytest.raises((ModelRateLimitError, ModelProviderError)):
await agent.get_model_output(messages)
if __name__ == '__main__':
pytest.main([__file__, '-v'])
+192
View File
@@ -0,0 +1,192 @@
"""Tests for DOCX file support in the FileSystem."""
from pathlib import Path
import pytest
from browser_use.filesystem.file_system import (
DocxFile,
FileSystem,
)
class TestDocxFile:
"""Test DOCX file operations."""
@pytest.mark.asyncio
async def test_create_docx_file(self, tmp_path: Path):
"""Test creating a DOCX file."""
fs = FileSystem(tmp_path)
content = """# Heading 1
## Heading 2
### Heading 3
Regular paragraph text.
Another paragraph."""
result = await fs.write_file('test.docx', content)
assert 'successfully' in result.lower()
assert 'test.docx' in fs.list_files()
@pytest.mark.asyncio
async def test_read_docx_file_internal(self, tmp_path: Path):
"""Test reading internal DOCX file."""
fs = FileSystem(tmp_path)
content = """# Title
Some content here."""
await fs.write_file('test.docx', content)
result = await fs.read_file('test.docx')
assert 'test.docx' in result
assert 'Title' in result or 'content' in result
@pytest.mark.asyncio
async def test_read_docx_file_external(self, tmp_path: Path):
"""Test reading external DOCX file."""
from docx import Document
# Create an external DOCX file
external_file = tmp_path / 'external.docx'
doc = Document()
doc.add_heading('Test Heading', level=1)
doc.add_paragraph('Test paragraph content.')
doc.save(str(external_file))
fs = FileSystem(tmp_path / 'workspace')
structured_result = await fs.read_file_structured(str(external_file), external_file=True)
assert 'message' in structured_result
assert 'Test Heading' in structured_result['message']
assert 'Test paragraph content' in structured_result['message']
def test_docx_file_extension(self):
"""Test DOCX file extension property."""
docx_file = DocxFile(name='test')
assert docx_file.extension == 'docx'
assert docx_file.full_name == 'test.docx'
@pytest.mark.asyncio
async def test_docx_with_unicode_characters(self, tmp_path: Path):
"""Test DOCX with unicode and emoji content."""
fs = FileSystem(tmp_path)
content = """# Unicode Test 🚀
Chinese: 你好世界
Arabic: مرحبا بالعالم
Emoji: 😀 👍 🎉"""
result = await fs.write_file('unicode.docx', content)
assert 'successfully' in result.lower()
read_result = await fs.read_file('unicode.docx')
assert 'Unicode Test' in read_result
# Note: Emoji may not be preserved in all systems
@pytest.mark.asyncio
async def test_empty_docx_file(self, tmp_path: Path):
"""Test creating an empty DOCX file."""
fs = FileSystem(tmp_path)
result = await fs.write_file('empty.docx', '')
assert 'successfully' in result.lower()
@pytest.mark.asyncio
async def test_large_docx_file(self, tmp_path: Path):
"""Test creating a large DOCX file."""
fs = FileSystem(tmp_path)
# Create content with 1000 lines
lines = [f'Line {i}: This is a test line with some content.' for i in range(1000)]
content = '\n'.join(lines)
result = await fs.write_file('large.docx', content)
assert 'successfully' in result.lower()
# Verify it can be read back
read_result = await fs.read_file('large.docx')
assert 'Line 0:' in read_result
assert 'Line 999:' in read_result
@pytest.mark.asyncio
async def test_corrupted_docx_file(self, tmp_path: Path):
"""Test reading a corrupted DOCX file."""
# Create a corrupted DOCX file
external_file = tmp_path / 'corrupted.docx'
external_file.write_bytes(b'This is not a valid DOCX file')
fs = FileSystem(tmp_path / 'workspace')
structured_result = await fs.read_file_structured(str(external_file), external_file=True)
assert 'message' in structured_result
assert 'error' in structured_result['message'].lower() or 'could not' in structured_result['message'].lower()
@pytest.mark.asyncio
async def test_docx_with_multiple_paragraphs(self, tmp_path: Path):
"""Test DOCX with various paragraph styles."""
fs = FileSystem(tmp_path)
content = """# Main Title
## Subtitle
This is a regular paragraph.
This is another paragraph with some text.
### Section 3
Final paragraph here."""
await fs.write_file('multi.docx', content)
result = await fs.read_file('multi.docx')
# Should contain all the text (headings converted to paragraphs)
assert 'Main Title' in result
assert 'Subtitle' in result
assert 'regular paragraph' in result
assert 'Final paragraph' in result
class TestFileSystemDocxIntegration:
"""Integration tests for DOCX file type."""
@pytest.mark.asyncio
async def test_multiple_file_types_with_docx(self, tmp_path: Path):
"""Test working with DOCX alongside other file types."""
fs = FileSystem(tmp_path)
# Create different file types
await fs.write_file('doc.docx', '# Document\nContent here')
await fs.write_file('data.json', '{"key": "value"}')
await fs.write_file('notes.txt', 'Some notes')
# Verify all files exist
files = fs.list_files()
assert 'doc.docx' in files
assert 'data.json' in files
assert 'notes.txt' in files
assert 'todo.md' in files # Default file
@pytest.mark.asyncio
async def test_file_system_state_with_docx(self, tmp_path: Path):
"""Test FileSystem state serialization with DOCX files."""
fs = FileSystem(tmp_path)
# Create files
await fs.write_file('test.docx', '# Title\nContent')
await fs.write_file('data.txt', 'Some text')
# Get state
state = fs.get_state()
assert 'test.docx' in state.files
assert 'data.txt' in state.files
# Restore from state
fs2 = FileSystem.from_state(state)
assert 'test.docx' in fs2.list_files()
assert 'data.txt' in fs2.list_files()
def test_allowed_extensions_include_docx(self, tmp_path: Path):
"""Test that DOCX is in allowed extensions."""
fs = FileSystem(tmp_path)
allowed = fs.get_allowed_extensions()
assert 'docx' in allowed
if __name__ == '__main__':
pytest.main([__file__, '-v'])
+249
View File
@@ -0,0 +1,249 @@
"""Tests for image file support in the FileSystem."""
import base64
import io
from pathlib import Path
import pytest
from PIL import Image
from browser_use.filesystem.file_system import FileSystem
class TestImageFiles:
"""Test image file operations - only external reading supported."""
def create_test_image(self, width: int = 100, height: int = 100, format: str = 'PNG') -> bytes:
"""Create a test image and return bytes."""
img = Image.new('RGB', (width, height), color='red')
buffer = io.BytesIO()
img.save(buffer, format=format)
buffer.seek(0)
return buffer.read()
@pytest.mark.asyncio
async def test_read_external_png_image(self, tmp_path: Path):
"""Test reading external PNG image file."""
# Create an external image file
external_file = tmp_path / 'test.png'
img_bytes = self.create_test_image(width=300, height=200, format='PNG')
external_file.write_bytes(img_bytes)
fs = FileSystem(tmp_path / 'workspace')
structured_result = await fs.read_file_structured(str(external_file), external_file=True)
assert 'message' in structured_result
assert 'Read image file' in structured_result['message']
assert 'images' in structured_result
assert structured_result['images'] is not None
assert len(structured_result['images']) == 1
img_data = structured_result['images'][0]
assert img_data['name'] == 'test.png'
assert 'data' in img_data
# Verify base64 is valid
decoded = base64.b64decode(img_data['data'])
assert decoded == img_bytes
@pytest.mark.asyncio
async def test_read_external_jpg_image(self, tmp_path: Path):
"""Test reading external JPG image file."""
# Create an external image file
external_file = tmp_path / 'photo.jpg'
img_bytes = self.create_test_image(width=150, height=100, format='JPEG')
external_file.write_bytes(img_bytes)
fs = FileSystem(tmp_path / 'workspace')
structured_result = await fs.read_file_structured(str(external_file), external_file=True)
assert 'message' in structured_result
assert 'images' in structured_result
assert structured_result['images'] is not None
img_data = structured_result['images'][0]
assert img_data['name'] == 'photo.jpg'
decoded = base64.b64decode(img_data['data'])
assert len(decoded) > 0
@pytest.mark.asyncio
async def test_read_jpeg_extension(self, tmp_path: Path):
"""Test reading .jpeg extension (not just .jpg)."""
external_file = tmp_path / 'test.jpeg'
img_bytes = self.create_test_image(format='JPEG')
external_file.write_bytes(img_bytes)
fs = FileSystem(tmp_path / 'workspace')
structured_result = await fs.read_file_structured(str(external_file), external_file=True)
assert structured_result['images'] is not None
assert structured_result['images'][0]['name'] == 'test.jpeg'
@pytest.mark.asyncio
async def test_read_nonexistent_image(self, tmp_path: Path):
"""Test reading a nonexistent image file."""
fs = FileSystem(tmp_path / 'workspace')
structured_result = await fs.read_file_structured('/path/to/nonexistent.png', external_file=True)
assert 'message' in structured_result
assert 'not found' in structured_result['message'].lower()
assert structured_result['images'] is None
@pytest.mark.asyncio
async def test_corrupted_image_file(self, tmp_path: Path):
"""Test reading a corrupted image file."""
external_file = tmp_path / 'corrupted.png'
# Write invalid PNG data
external_file.write_bytes(b'Not a valid PNG file')
fs = FileSystem(tmp_path / 'workspace')
structured_result = await fs.read_file_structured(str(external_file), external_file=True)
# Should still return base64 data (we don't validate image format)
assert 'message' in structured_result
assert 'Read image file' in structured_result['message']
# Base64 encoding will succeed even for invalid image data
assert structured_result['images'] is not None
@pytest.mark.asyncio
async def test_large_image_file(self, tmp_path: Path):
"""Test reading a large image file."""
# Create a large image (2000x2000)
external_file = tmp_path / 'large.png'
img = Image.new('RGB', (2000, 2000), color='blue')
img.save(str(external_file), format='PNG')
fs = FileSystem(tmp_path / 'workspace')
structured_result = await fs.read_file_structured(str(external_file), external_file=True)
assert 'images' in structured_result
assert structured_result['images'] is not None
# Verify base64 data is present and substantial
assert len(structured_result['images'][0]['data']) > 10000
@pytest.mark.asyncio
async def test_multiple_images_in_sequence(self, tmp_path: Path):
"""Test reading multiple images in sequence."""
fs = FileSystem(tmp_path / 'workspace')
# Create three different images
for i, color in enumerate(['red', 'green', 'blue']):
img_file = tmp_path / f'image_{i}.png'
img = Image.new('RGB', (100, 100), color=color)
img.save(str(img_file), format='PNG')
# Read them all
results = []
for i in range(3):
img_file = tmp_path / f'image_{i}.png'
result = await fs.read_file_structured(str(img_file), external_file=True)
results.append(result)
# Verify all were read successfully
for i, result in enumerate(results):
assert result['images'] is not None
assert result['images'][0]['name'] == f'image_{i}.png'
@pytest.mark.asyncio
async def test_different_image_formats(self, tmp_path: Path):
"""Test reading different image format variations."""
fs = FileSystem(tmp_path / 'workspace')
# Test .jpg
jpg_file = tmp_path / 'test.jpg'
img = Image.new('RGB', (50, 50), color='yellow')
img.save(str(jpg_file), format='JPEG')
result_jpg = await fs.read_file_structured(str(jpg_file), external_file=True)
assert result_jpg['images'] is not None
# Test .jpeg
jpeg_file = tmp_path / 'test.jpeg'
img.save(str(jpeg_file), format='JPEG')
result_jpeg = await fs.read_file_structured(str(jpeg_file), external_file=True)
assert result_jpeg['images'] is not None
# Test .png
png_file = tmp_path / 'test.png'
img.save(str(png_file), format='PNG')
result_png = await fs.read_file_structured(str(png_file), external_file=True)
assert result_png['images'] is not None
@pytest.mark.asyncio
async def test_image_with_transparency(self, tmp_path: Path):
"""Test reading PNG with transparency (RGBA)."""
external_file = tmp_path / 'transparent.png'
# Create RGBA image with transparency
img = Image.new('RGBA', (100, 100), color=(255, 0, 0, 128))
img.save(str(external_file), format='PNG')
fs = FileSystem(tmp_path / 'workspace')
structured_result = await fs.read_file_structured(str(external_file), external_file=True)
assert structured_result['images'] is not None
assert len(structured_result['images'][0]['data']) > 0
class TestActionResultImages:
"""Test ActionResult with images field."""
def test_action_result_with_images(self):
"""Test creating ActionResult with images."""
from browser_use.agent.views import ActionResult
images = [{'name': 'test.png', 'data': 'base64_encoded_data_here'}]
result = ActionResult(
extracted_content='Read image file test.png',
long_term_memory='Read image file test.png',
images=images,
include_extracted_content_only_once=True,
)
assert result.images is not None
assert len(result.images) == 1
assert result.images[0]['name'] == 'test.png'
assert result.images[0]['data'] == 'base64_encoded_data_here'
def test_action_result_without_images(self):
"""Test ActionResult without images (default behavior)."""
from browser_use.agent.views import ActionResult
result = ActionResult(extracted_content='Some text', long_term_memory='Memory')
assert result.images is None
def test_action_result_with_multiple_images(self):
"""Test ActionResult with multiple images."""
from browser_use.agent.views import ActionResult
images = [
{'name': 'image1.png', 'data': 'base64_data_1'},
{'name': 'image2.jpg', 'data': 'base64_data_2'},
]
result = ActionResult(
extracted_content='Read multiple images',
long_term_memory='Read image files',
images=images,
include_extracted_content_only_once=True,
)
assert result.images is not None
assert len(result.images) == 2
assert result.images[0]['name'] == 'image1.png'
assert result.images[1]['name'] == 'image2.jpg'
def test_action_result_with_empty_images_list(self):
"""Test ActionResult with empty images list."""
from browser_use.agent.views import ActionResult
result = ActionResult(
extracted_content='No images',
images=[],
)
# Empty list is still valid
assert result.images == []
if __name__ == '__main__':
pytest.main([__file__, '-v'])
@@ -0,0 +1,386 @@
"""Integration tests for DOCX and image file support in LLM messages."""
import base64
import io
from pathlib import Path
import pytest
from PIL import Image
from browser_use.agent.message_manager.service import MessageManager
from browser_use.agent.prompts import AgentMessagePrompt
from browser_use.agent.views import ActionResult, AgentStepInfo
from browser_use.browser.views import BrowserStateSummary, TabInfo
from browser_use.dom.views import SerializedDOMState
from browser_use.filesystem.file_system import FileSystem
from browser_use.llm.messages import ContentPartImageParam, ContentPartTextParam, SystemMessage
class TestImageInLLMMessages:
"""Test that images flow correctly through to LLM messages."""
def create_test_image(self, width: int = 100, height: int = 100) -> bytes:
"""Create a test image and return bytes."""
img = Image.new('RGB', (width, height), color='red')
buffer = io.BytesIO()
img.save(buffer, format='PNG')
buffer.seek(0)
return buffer.read()
@pytest.mark.asyncio
async def test_image_stored_in_message_manager(self, tmp_path: Path):
"""Test that images are stored in MessageManager state."""
fs = FileSystem(tmp_path)
system_message = SystemMessage(content='Test system message')
mm = MessageManager(task='test', system_message=system_message, file_system=fs)
# Create ActionResult with images
images = [{'name': 'test.png', 'data': 'base64_test_data'}]
action_results = [
ActionResult(
extracted_content='Read image file test.png',
long_term_memory='Read image file test.png',
images=images,
include_extracted_content_only_once=True,
)
]
# Update message manager with results
step_info = AgentStepInfo(step_number=1, max_steps=10)
mm._update_agent_history_description(model_output=None, result=action_results, step_info=step_info)
# Verify images are stored
assert mm.state.read_state_images is not None
assert len(mm.state.read_state_images) == 1
assert mm.state.read_state_images[0]['name'] == 'test.png'
assert mm.state.read_state_images[0]['data'] == 'base64_test_data'
@pytest.mark.asyncio
async def test_images_cleared_after_step(self, tmp_path: Path):
"""Test that images are cleared after each step."""
fs = FileSystem(tmp_path)
system_message = SystemMessage(content='Test system message')
mm = MessageManager(task='test', system_message=system_message, file_system=fs)
# First step with images
images = [{'name': 'test.png', 'data': 'base64_data'}]
action_results = [ActionResult(images=images, include_extracted_content_only_once=True)]
step_info = AgentStepInfo(step_number=1, max_steps=10)
mm._update_agent_history_description(model_output=None, result=action_results, step_info=step_info)
assert len(mm.state.read_state_images) == 1
# Second step without images - should clear
action_results_2 = [ActionResult(extracted_content='No images')]
step_info_2 = AgentStepInfo(step_number=2, max_steps=10)
mm._update_agent_history_description(model_output=None, result=action_results_2, step_info=step_info_2)
assert len(mm.state.read_state_images) == 0
@pytest.mark.asyncio
async def test_multiple_images_accumulated(self, tmp_path: Path):
"""Test that multiple images in one step are accumulated."""
fs = FileSystem(tmp_path)
system_message = SystemMessage(content='Test system message')
mm = MessageManager(task='test', system_message=system_message, file_system=fs)
# Multiple action results with images
action_results = [
ActionResult(images=[{'name': 'img1.png', 'data': 'data1'}], include_extracted_content_only_once=True),
ActionResult(images=[{'name': 'img2.jpg', 'data': 'data2'}], include_extracted_content_only_once=True),
]
step_info = AgentStepInfo(step_number=1, max_steps=10)
mm._update_agent_history_description(model_output=None, result=action_results, step_info=step_info)
assert len(mm.state.read_state_images) == 2
assert mm.state.read_state_images[0]['name'] == 'img1.png'
assert mm.state.read_state_images[1]['name'] == 'img2.jpg'
def test_agent_message_prompt_includes_images(self, tmp_path: Path):
"""Test that AgentMessagePrompt includes images in message content."""
fs = FileSystem(tmp_path)
# Create browser state
browser_state = BrowserStateSummary(
url='https://example.com',
title='Test',
tabs=[TabInfo(target_id='test-0', url='https://example.com', title='Test')],
screenshot=None,
dom_state=SerializedDOMState(_root=None, selector_map={}),
)
# Create images
read_state_images = [{'name': 'test.png', 'data': 'base64_image_data_here'}]
# Create message prompt
prompt = AgentMessagePrompt(
browser_state_summary=browser_state,
file_system=fs,
read_state_images=read_state_images,
)
# Get user message with vision enabled
user_message = prompt.get_user_message(use_vision=True)
# Verify message has content parts (not just string)
assert isinstance(user_message.content, list)
# Find image content parts
image_parts = [part for part in user_message.content if isinstance(part, ContentPartImageParam)]
text_parts = [part for part in user_message.content if isinstance(part, ContentPartTextParam)]
# Should have at least one image
assert len(image_parts) >= 1
# Should have text label
image_labels = [part.text for part in text_parts if 'test.png' in part.text]
assert len(image_labels) >= 1
# Verify image data URL format
img_part = image_parts[0]
assert 'data:image/' in img_part.image_url.url
assert 'base64,base64_image_data_here' in img_part.image_url.url
def test_agent_message_prompt_png_vs_jpg_media_type(self, tmp_path: Path):
"""Test that AgentMessagePrompt correctly detects PNG vs JPG media types."""
fs = FileSystem(tmp_path)
browser_state = BrowserStateSummary(
url='https://example.com',
title='Test',
tabs=[TabInfo(target_id='test-0', url='https://example.com', title='Test')],
screenshot=None,
dom_state=SerializedDOMState(_root=None, selector_map={}),
)
# Test PNG
read_state_images_png = [{'name': 'test.png', 'data': 'data'}]
prompt_png = AgentMessagePrompt(
browser_state_summary=browser_state,
file_system=fs,
read_state_images=read_state_images_png,
)
message_png = prompt_png.get_user_message(use_vision=True)
image_parts_png = [part for part in message_png.content if isinstance(part, ContentPartImageParam)]
assert 'data:image/png;base64' in image_parts_png[0].image_url.url
# Test JPG
read_state_images_jpg = [{'name': 'photo.jpg', 'data': 'data'}]
prompt_jpg = AgentMessagePrompt(
browser_state_summary=browser_state,
file_system=fs,
read_state_images=read_state_images_jpg,
)
message_jpg = prompt_jpg.get_user_message(use_vision=True)
image_parts_jpg = [part for part in message_jpg.content if isinstance(part, ContentPartImageParam)]
assert 'data:image/jpeg;base64' in image_parts_jpg[0].image_url.url
def test_agent_message_prompt_no_images(self, tmp_path: Path):
"""Test that message works correctly when no images are present."""
fs = FileSystem(tmp_path)
browser_state = BrowserStateSummary(
url='https://example.com',
title='Test',
tabs=[TabInfo(target_id='test-0', url='https://example.com', title='Test')],
screenshot=None,
dom_state=SerializedDOMState(_root=None, selector_map={}),
)
# No images
prompt = AgentMessagePrompt(
browser_state_summary=browser_state,
file_system=fs,
read_state_images=[],
)
# Get user message without vision
user_message = prompt.get_user_message(use_vision=False)
# Should be plain text, not content parts
assert isinstance(user_message.content, str)
def test_agent_message_prompt_empty_base64_skipped(self, tmp_path: Path):
"""Test that images with empty base64 data are skipped."""
fs = FileSystem(tmp_path)
browser_state = BrowserStateSummary(
url='https://example.com',
title='Test',
tabs=[TabInfo(target_id='test-0', url='https://example.com', title='Test')],
screenshot=None,
dom_state=SerializedDOMState(_root=None, selector_map={}),
)
# Image with empty data field
read_state_images = [
{'name': 'empty.png', 'data': ''}, # Empty - should be skipped
{'name': 'valid.png', 'data': 'valid_data'}, # Valid
]
prompt = AgentMessagePrompt(
browser_state_summary=browser_state,
file_system=fs,
read_state_images=read_state_images,
)
user_message = prompt.get_user_message(use_vision=True)
image_parts = [part for part in user_message.content if isinstance(part, ContentPartImageParam)]
# Should only have 1 image (the valid one)
assert len(image_parts) == 1
assert 'valid_data' in image_parts[0].image_url.url
class TestDocxInLLMMessages:
"""Test that DOCX content flows correctly through to LLM messages."""
@pytest.mark.asyncio
async def test_docx_in_extracted_content(self, tmp_path: Path):
"""Test that DOCX text appears in extracted_content."""
fs = FileSystem(tmp_path)
# Create DOCX file
content = """# Title
Some important content here."""
await fs.write_file('test.docx', content)
# Read it
result = await fs.read_file('test.docx')
# Verify content is in the result
assert 'Title' in result
assert 'important content' in result
@pytest.mark.asyncio
async def test_docx_in_message_manager(self, tmp_path: Path):
"""Test that DOCX content appears in message manager state."""
fs = FileSystem(tmp_path)
system_message = SystemMessage(content='Test system message')
mm = MessageManager(task='test', system_message=system_message, file_system=fs)
# Simulate read_file action result
docx_content = """Read from file test.docx.
<content>
Title
Some content here.
</content>"""
action_results = [
ActionResult(
extracted_content=docx_content,
long_term_memory='Read file test.docx',
include_extracted_content_only_once=True,
)
]
step_info = AgentStepInfo(step_number=1, max_steps=10)
mm._update_agent_history_description(model_output=None, result=action_results, step_info=step_info)
# Verify it's in read_state_description
assert 'Title' in mm.state.read_state_description
assert 'Some content' in mm.state.read_state_description
class TestEndToEndIntegration:
"""End-to-end tests for file reading and LLM message creation."""
def create_test_image(self) -> bytes:
"""Create a test image."""
img = Image.new('RGB', (50, 50), color='blue')
buffer = io.BytesIO()
img.save(buffer, format='PNG')
buffer.seek(0)
return buffer.read()
@pytest.mark.asyncio
async def test_image_end_to_end(self, tmp_path: Path):
"""Test complete flow: external image → FileSystem → ActionResult → MessageManager → Prompt."""
# Step 1: Create external image
external_file = tmp_path / 'photo.png'
img_bytes = self.create_test_image()
external_file.write_bytes(img_bytes)
# Step 2: Read via FileSystem
fs = FileSystem(tmp_path / 'workspace')
structured_result = await fs.read_file_structured(str(external_file), external_file=True)
assert structured_result['images'] is not None
# Step 3: Create ActionResult (simulating tools/service.py)
action_result = ActionResult(
extracted_content=structured_result['message'],
long_term_memory='Read image file photo.png',
images=structured_result['images'],
include_extracted_content_only_once=True,
)
# Step 4: Process in MessageManager
system_message = SystemMessage(content='Test system message')
mm = MessageManager(task='test', system_message=system_message, file_system=fs)
step_info = AgentStepInfo(step_number=1, max_steps=10)
mm._update_agent_history_description(model_output=None, result=[action_result], step_info=step_info)
# Verify images stored
assert len(mm.state.read_state_images) == 1
assert mm.state.read_state_images[0]['name'] == 'photo.png'
# Step 5: Create message with AgentMessagePrompt
browser_state = BrowserStateSummary(
url='https://example.com',
title='Test',
tabs=[TabInfo(target_id='test-0', url='https://example.com', title='Test')],
screenshot=None,
dom_state=SerializedDOMState(_root=None, selector_map={}),
)
prompt = AgentMessagePrompt(
browser_state_summary=browser_state,
file_system=fs,
read_state_images=mm.state.read_state_images,
)
user_message = prompt.get_user_message(use_vision=True)
# Verify image is in message
assert isinstance(user_message.content, list)
image_parts = [part for part in user_message.content if isinstance(part, ContentPartImageParam)]
assert len(image_parts) >= 1
# Verify image data is correct
base64_str = base64.b64encode(img_bytes).decode('utf-8')
assert base64_str in image_parts[0].image_url.url
@pytest.mark.asyncio
async def test_docx_end_to_end(self, tmp_path: Path):
"""Test complete flow: DOCX file → FileSystem → ActionResult → MessageManager."""
# Step 1: Create DOCX
fs = FileSystem(tmp_path)
docx_content = """# Important Document
This is critical information."""
await fs.write_file('important.docx', docx_content)
# Step 2: Read it
read_result = await fs.read_file('important.docx')
# Step 3: Create ActionResult (simulating tools/service.py)
action_result = ActionResult(
extracted_content=read_result,
long_term_memory=read_result[:100] if len(read_result) > 100 else read_result,
include_extracted_content_only_once=True,
)
# Step 4: Process in MessageManager
system_message = SystemMessage(content='Test system message')
mm = MessageManager(task='test', system_message=system_message, file_system=fs)
step_info = AgentStepInfo(step_number=1, max_steps=10)
mm._update_agent_history_description(model_output=None, result=[action_result], step_info=step_info)
# Verify content is in read_state
assert 'Important Document' in mm.state.read_state_description
assert 'critical information' in mm.state.read_state_description
if __name__ == '__main__':
pytest.main([__file__, '-v'])
+119
View File
@@ -0,0 +1,119 @@
from browser_use.agent.views import StepMetadata
def test_step_metadata_has_step_interval_field():
"""Test that StepMetadata includes step_interval field"""
metadata = StepMetadata(step_number=1, step_start_time=10.0, step_end_time=12.5, step_interval=2.5)
assert hasattr(metadata, 'step_interval')
assert metadata.step_interval == 2.5
def test_step_metadata_step_interval_optional():
"""Test that step_interval is optional (None for first step)"""
# Explicitly None
metadata_none = StepMetadata(step_number=0, step_start_time=0.0, step_end_time=1.0, step_interval=None)
assert metadata_none.step_interval is None
# Omitted (defaults to None)
metadata_default = StepMetadata(step_number=0, step_start_time=0.0, step_end_time=1.0)
assert metadata_default.step_interval is None
def test_step_interval_calculation():
"""Test step_interval calculation logic (uses previous step's duration)"""
# Previous step (Step 1): runs from 100.0 to 102.5 (duration: 2.5s)
previous_start = 100.0
previous_end = 102.5
previous_duration = previous_end - previous_start
# Current step (Step 2): should have step_interval = previous step's duration
# This tells the rerun system "wait 2.5s before executing Step 2"
expected_step_interval = previous_duration
calculated_step_interval = max(0, previous_end - previous_start)
assert abs(calculated_step_interval - expected_step_interval) < 0.001 # Float comparison
assert calculated_step_interval == 2.5
def test_step_metadata_serialization_with_step_interval():
"""Test that step_interval is included in metadata serialization"""
# With step_interval
metadata_with_wait = StepMetadata(step_number=1, step_start_time=10.0, step_end_time=12.5, step_interval=2.5)
data = metadata_with_wait.model_dump()
assert 'step_interval' in data
assert data['step_interval'] == 2.5
# Without step_interval (None)
metadata_without_wait = StepMetadata(step_number=0, step_start_time=0.0, step_end_time=1.0, step_interval=None)
data = metadata_without_wait.model_dump()
assert 'step_interval' in data
assert data['step_interval'] is None
def test_step_metadata_deserialization_with_step_interval():
"""Test that step_interval can be loaded from dict"""
# Load with step_interval
data_with_wait = {'step_number': 1, 'step_start_time': 10.0, 'step_end_time': 12.5, 'step_interval': 2.5}
metadata = StepMetadata.model_validate(data_with_wait)
assert metadata.step_interval == 2.5
# Load without step_interval (old format)
data_without_wait = {
'step_number': 0,
'step_start_time': 0.0,
'step_end_time': 1.0,
# step_interval is missing
}
metadata = StepMetadata.model_validate(data_without_wait)
assert metadata.step_interval is None # Defaults to None
def test_step_interval_backwards_compatibility():
"""Test that old metadata without step_interval still works"""
# Simulate old format from JSON
old_metadata_dict = {
'step_number': 0,
'step_start_time': 1000.0,
'step_end_time': 1002.5,
# step_interval field doesn't exist (old format)
}
# Should load successfully with step_interval defaulting to None
metadata = StepMetadata.model_validate(old_metadata_dict)
assert metadata.step_number == 0
assert metadata.step_start_time == 1000.0
assert metadata.step_end_time == 1002.5
assert metadata.step_interval is None # Default value
def test_duration_seconds_property_still_works():
"""Test that existing duration_seconds property still works"""
metadata = StepMetadata(step_number=1, step_start_time=10.0, step_end_time=13.5, step_interval=2.0)
# duration_seconds should be 3.5 (13.5 - 10.0)
assert metadata.duration_seconds == 3.5
# step_interval is separate from duration
assert metadata.step_interval == 2.0
def test_step_metadata_json_round_trip():
"""Test that step_interval survives JSON serialization round-trip"""
metadata = StepMetadata(step_number=1, step_start_time=100.0, step_end_time=102.5, step_interval=1.5)
# Serialize to JSON
json_str = metadata.model_dump_json()
# Deserialize from JSON
loaded = StepMetadata.model_validate_json(json_str)
assert loaded.step_interval == 1.5
assert loaded.step_number == 1
assert loaded.step_start_time == 100.0
assert loaded.step_end_time == 102.5
+141
View File
@@ -0,0 +1,141 @@
"""Regression test: structured output cut off at the completion-token cap must raise a
clear truncation error, not a misleading JSON parse error ('Unterminated string...')."""
import pytest
from pydantic import BaseModel
from browser_use.llm.exceptions import ModelProviderError
from browser_use.llm.messages import UserMessage
from browser_use.llm.openai.chat import ChatOpenAI
class AnswerFormat(BaseModel):
answer: str
async def test_openai_truncated_structured_output_raises_clear_error(httpserver):
"""finish_reason='length' with JSON cut mid-string must surface the token cap, not a parse error."""
httpserver.expect_request('/v1/chat/completions', method='POST').respond_with_json(
{
'id': 'chatcmpl-test',
'object': 'chat.completion',
'created': 0,
'model': 'gpt-4o',
'choices': [
{
'index': 0,
'message': {'role': 'assistant', 'content': '{"answer": "this output was cut off mid-sent'},
'finish_reason': 'length',
}
],
'usage': {'prompt_tokens': 10, 'completion_tokens': 4096, 'total_tokens': 4106},
}
)
llm = ChatOpenAI(model='gpt-4o', api_key='test-key', base_url=httpserver.url_for('/v1'))
with pytest.raises(ModelProviderError) as exc_info:
await llm.ainvoke([UserMessage(content='answer at length')], output_format=AnswerFormat)
assert 'truncated' in str(exc_info.value), f'expected a truncation error, got: {exc_info.value}'
assert 'max_completion_tokens' in str(exc_info.value)
async def test_openai_normal_structured_output_still_parses(httpserver):
httpserver.expect_request('/v1/chat/completions', method='POST').respond_with_json(
{
'id': 'chatcmpl-test',
'object': 'chat.completion',
'created': 0,
'model': 'gpt-4o',
'choices': [
{
'index': 0,
'message': {'role': 'assistant', 'content': '{"answer": "complete answer"}'},
'finish_reason': 'stop',
}
],
'usage': {'prompt_tokens': 10, 'completion_tokens': 8, 'total_tokens': 18},
}
)
llm = ChatOpenAI(model='gpt-4o', api_key='test-key', base_url=httpserver.url_for('/v1'))
result = await llm.ainvoke([UserMessage(content='answer briefly')], output_format=AnswerFormat)
assert result.completion.answer == 'complete answer'
async def test_truncation_error_readable_when_cap_unset(httpserver):
"""With max_completion_tokens=None, the error must describe the limit without printing 'None'."""
httpserver.expect_request('/v1/chat/completions', method='POST').respond_with_json(
{
'id': 'chatcmpl-test',
'object': 'chat.completion',
'created': 0,
'model': 'gpt-4o',
'choices': [
{
'index': 0,
'message': {'role': 'assistant', 'content': '{"answer": "cut off mid'},
'finish_reason': 'length',
}
],
'usage': {'prompt_tokens': 10, 'completion_tokens': 16384, 'total_tokens': 16394},
}
)
llm = ChatOpenAI(model='gpt-4o', api_key='test-key', base_url=httpserver.url_for('/v1'), max_completion_tokens=None)
with pytest.raises(ModelProviderError) as exc_info:
await llm.ainvoke([UserMessage(content='answer at length')], output_format=AnswerFormat)
assert 'truncated' in str(exc_info.value)
assert 'None' not in str(exc_info.value), f'error message leaks None: {exc_info.value}'
def test_truncation_error_triggers_fallback_llm_switch(tmp_path):
"""A truncation error must allow switching to a configured fallback LLM — a
fallback with a different output cap can succeed where the primary truncated."""
from browser_use.agent.service import Agent
from browser_use.llm.exceptions import ModelOutputTruncatedError
from tests.ci.conftest import create_mock_llm
agent = Agent(
task='test fallback on truncation',
llm=create_mock_llm(),
fallback_llm=create_mock_llm(),
file_system_path=str(tmp_path / 'agent-files'),
)
error = ModelOutputTruncatedError(message='Model output was truncated at max_completion_tokens=4096', model='gpt-4o')
assert agent._try_switch_to_fallback_llm(error) is True
assert agent._using_fallback_llm is True
async def test_truncation_detected_when_content_is_null(httpserver):
"""Reasoning models can spend the whole completion budget on hidden reasoning:
finish_reason='length' with content=null. That must surface as truncation, not
the generic 'Failed to parse structured output'."""
httpserver.expect_request('/v1/chat/completions', method='POST').respond_with_json(
{
'id': 'chatcmpl-test',
'object': 'chat.completion',
'created': 0,
'model': 'o3',
'choices': [
{
'index': 0,
'message': {'role': 'assistant', 'content': None},
'finish_reason': 'length',
}
],
'usage': {'prompt_tokens': 10, 'completion_tokens': 4096, 'total_tokens': 4106},
}
)
llm = ChatOpenAI(model='o3', api_key='test-key', base_url=httpserver.url_for('/v1'))
with pytest.raises(ModelProviderError) as exc_info:
await llm.ainvoke([UserMessage(content='think hard')], output_format=AnswerFormat)
assert 'truncated' in str(exc_info.value), f'expected truncation signal, got: {exc_info.value}'
+304
View File
@@ -0,0 +1,304 @@
"""
Test retry logic with exponential backoff for LLM clients.
"""
import time
from unittest.mock import AsyncMock, MagicMock, patch
import httpx
import pytest
class TestChatBrowserUseRetries:
"""Test retry logic for ChatBrowserUse."""
@pytest.fixture
def mock_env(self, monkeypatch):
"""Set up environment for ChatBrowserUse."""
monkeypatch.setenv('BROWSER_USE_API_KEY', 'test-api-key')
@pytest.mark.asyncio
async def test_retries_on_503_with_exponential_backoff(self, mock_env):
"""Test that 503 errors trigger retries with exponential backoff."""
from browser_use.llm.browser_use.chat import ChatBrowserUse
from browser_use.llm.messages import UserMessage
# Track timing of each attempt
attempt_times: list[float] = []
attempt_count = 0
async def mock_post(*args, **kwargs):
nonlocal attempt_count
attempt_times.append(time.monotonic())
attempt_count += 1
if attempt_count < 3:
# First 2 attempts fail with 503
response = MagicMock()
response.status_code = 503
response.json.return_value = {'detail': 'Service temporarily unavailable'}
raise httpx.HTTPStatusError('503', request=MagicMock(), response=response)
else:
# Third attempt succeeds
response = MagicMock()
response.json.return_value = {
'completion': 'Success!',
'usage': {
'prompt_tokens': 10,
'completion_tokens': 5,
'total_tokens': 15,
'prompt_cached_tokens': None,
'prompt_cache_creation_tokens': None,
'prompt_image_tokens': None,
},
}
response.raise_for_status = MagicMock()
return response
with patch('httpx.AsyncClient') as mock_client_class:
mock_client = AsyncMock()
mock_client.post = mock_post
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
# Use short delays for testing
client = ChatBrowserUse(retry_base_delay=0.1, retry_max_delay=1.0)
result = await client.ainvoke([UserMessage(content='test')])
# Should have made 3 attempts
assert attempt_count == 3
assert result.completion == 'Success!'
# Verify exponential backoff timing
# base_delay=0.1s, so first retry sleeps ~0.1s, second ~0.2s (2x exponential)
delay_1 = attempt_times[1] - attempt_times[0]
delay_2 = attempt_times[2] - attempt_times[1]
# Allow some tolerance for CI runner scheduling jitter
assert 0.05 <= delay_1 <= 0.3, f'First delay {delay_1:.3f}s not in expected range'
assert 0.1 <= delay_2 <= 0.5, f'Second delay {delay_2:.3f}s not in expected range'
# Second delay should be roughly 2x the first (exponential)
assert delay_2 > delay_1, 'Second delay should be longer than first (exponential backoff)'
@pytest.mark.asyncio
async def test_no_retry_on_401(self, mock_env):
"""Test that 401 errors do NOT trigger retries."""
from browser_use.llm.browser_use.chat import ChatBrowserUse
from browser_use.llm.exceptions import ModelProviderError
from browser_use.llm.messages import UserMessage
attempt_count = 0
async def mock_post(*args, **kwargs):
nonlocal attempt_count
attempt_count += 1
response = MagicMock()
response.status_code = 401
response.json.return_value = {'detail': 'Invalid API key'}
raise httpx.HTTPStatusError('401', request=MagicMock(), response=response)
with patch('httpx.AsyncClient') as mock_client_class:
mock_client = AsyncMock()
mock_client.post = mock_post
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
client = ChatBrowserUse(retry_base_delay=0.01)
with pytest.raises(ModelProviderError, match='Invalid API key'):
await client.ainvoke([UserMessage(content='test')])
# Should only attempt once (no retries for 401)
assert attempt_count == 1
@pytest.mark.asyncio
async def test_retries_on_timeout(self, mock_env):
"""Test that timeouts trigger retries."""
from browser_use.llm.browser_use.chat import ChatBrowserUse
from browser_use.llm.messages import UserMessage
attempt_count = 0
async def mock_post(*args, **kwargs):
nonlocal attempt_count
attempt_count += 1
if attempt_count < 2:
raise httpx.TimeoutException('Request timed out')
# Second attempt succeeds (with no usage data to test None handling)
response = MagicMock()
response.json.return_value = {'completion': 'Success after timeout!', 'usage': None}
response.raise_for_status = MagicMock()
return response
with patch('httpx.AsyncClient') as mock_client_class:
mock_client = AsyncMock()
mock_client.post = mock_post
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
client = ChatBrowserUse(retry_base_delay=0.01)
result = await client.ainvoke([UserMessage(content='test')])
assert attempt_count == 2
assert result.completion == 'Success after timeout!'
@pytest.mark.asyncio
async def test_max_retries_exhausted(self, mock_env):
"""Test that error is raised after max retries exhausted."""
from browser_use.llm.browser_use.chat import ChatBrowserUse
from browser_use.llm.exceptions import ModelProviderError
from browser_use.llm.messages import UserMessage
attempt_count = 0
async def mock_post(*args, **kwargs):
nonlocal attempt_count
attempt_count += 1
response = MagicMock()
response.status_code = 503
response.json.return_value = {'detail': 'Service unavailable'}
raise httpx.HTTPStatusError('503', request=MagicMock(), response=response)
with patch('httpx.AsyncClient') as mock_client_class:
mock_client = AsyncMock()
mock_client.post = mock_post
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
mock_client_class.return_value = mock_client
client = ChatBrowserUse(max_retries=3, retry_base_delay=0.01)
with pytest.raises(ModelProviderError, match='Server error'):
await client.ainvoke([UserMessage(content='test')])
# Should have attempted max_retries times
assert attempt_count == 3
class TestChatGoogleRetries:
"""Test retry logic for ChatGoogle."""
@pytest.fixture
def mock_env(self, monkeypatch):
"""Set up environment for ChatGoogle."""
monkeypatch.setenv('GOOGLE_API_KEY', 'test-api-key')
@pytest.mark.asyncio
async def test_retries_on_503_with_exponential_backoff(self, mock_env):
"""Test that 503 errors trigger retries with exponential backoff."""
from browser_use.llm.exceptions import ModelProviderError
from browser_use.llm.google.chat import ChatGoogle
from browser_use.llm.messages import UserMessage
attempt_times: list[float] = []
attempt_count = 0
# Mock the genai client
with patch('browser_use.llm.google.chat.genai') as mock_genai:
mock_client = MagicMock()
mock_genai.Client.return_value = mock_client
async def mock_generate(*args, **kwargs):
nonlocal attempt_count
attempt_times.append(time.monotonic())
attempt_count += 1
if attempt_count < 3:
raise ModelProviderError(message='Service unavailable', status_code=503, model='gemini-2.0-flash')
else:
# Success on third attempt
mock_response = MagicMock()
mock_response.text = 'Success!'
mock_response.usage_metadata = MagicMock(
prompt_token_count=10, candidates_token_count=5, total_token_count=15, cached_content_token_count=0
)
mock_response.candidates = [MagicMock(content=MagicMock(parts=[MagicMock(text='Success!')]))]
return mock_response
# Mock the aio.models.generate_content path
mock_client.aio.models.generate_content = mock_generate
client = ChatGoogle(model='gemini-2.0-flash', api_key='test', retry_base_delay=0.1, retry_max_delay=1.0)
result = await client.ainvoke([UserMessage(content='test')])
assert attempt_count == 3
assert result.completion == 'Success!'
# Verify exponential backoff
delay_1 = attempt_times[1] - attempt_times[0]
delay_2 = attempt_times[2] - attempt_times[1]
assert 0.05 <= delay_1 <= 0.3, f'First delay {delay_1:.3f}s not in expected range'
assert 0.1 <= delay_2 <= 0.5, f'Second delay {delay_2:.3f}s not in expected range'
assert delay_2 > delay_1, 'Second delay should be longer than first'
@pytest.mark.asyncio
async def test_no_retry_on_400(self, mock_env):
"""Test that 400 errors do NOT trigger retries."""
from browser_use.llm.exceptions import ModelProviderError
from browser_use.llm.google.chat import ChatGoogle
from browser_use.llm.messages import UserMessage
attempt_count = 0
with patch('browser_use.llm.google.chat.genai') as mock_genai:
mock_client = MagicMock()
mock_genai.Client.return_value = mock_client
async def mock_generate(*args, **kwargs):
nonlocal attempt_count
attempt_count += 1
raise ModelProviderError(message='Bad request', status_code=400, model='gemini-2.0-flash')
mock_client.aio.models.generate_content = mock_generate
client = ChatGoogle(model='gemini-2.0-flash', api_key='test', retry_base_delay=0.01)
with pytest.raises(ModelProviderError):
await client.ainvoke([UserMessage(content='test')])
# Should only attempt once (400 is not retryable)
assert attempt_count == 1
@pytest.mark.asyncio
async def test_retries_on_429_rate_limit(self, mock_env):
"""Test that 429 rate limit errors trigger retries."""
from browser_use.llm.exceptions import ModelProviderError
from browser_use.llm.google.chat import ChatGoogle
from browser_use.llm.messages import UserMessage
attempt_count = 0
with patch('browser_use.llm.google.chat.genai') as mock_genai:
mock_client = MagicMock()
mock_genai.Client.return_value = mock_client
async def mock_generate(*args, **kwargs):
nonlocal attempt_count
attempt_count += 1
if attempt_count < 2:
raise ModelProviderError(message='Rate limit exceeded', status_code=429, model='gemini-2.0-flash')
else:
mock_response = MagicMock()
mock_response.text = 'Success after rate limit!'
mock_response.usage_metadata = MagicMock(
prompt_token_count=10, candidates_token_count=5, total_token_count=15, cached_content_token_count=0
)
mock_response.candidates = [MagicMock(content=MagicMock(parts=[MagicMock(text='Success after rate limit!')]))]
return mock_response
mock_client.aio.models.generate_content = mock_generate
client = ChatGoogle(model='gemini-2.0-flash', api_key='test', retry_base_delay=0.01)
result = await client.ainvoke([UserMessage(content='test')])
assert attempt_count == 2
assert result.completion == 'Success after rate limit!'
if __name__ == '__main__':
pytest.main([__file__, '-v'])
+358
View File
@@ -0,0 +1,358 @@
"""Tests for structure-aware markdown chunking."""
from markdownify import markdownify as md
from pytest_httpserver import HTTPServer
from browser_use.dom.markdown_extractor import chunk_markdown_by_structure
# ---------------------------------------------------------------------------
# Unit tests — synchronous, no browser needed
# ---------------------------------------------------------------------------
class TestChunkMarkdownBasic:
"""Basic chunking behaviour."""
def test_short_content_single_chunk(self):
content = '# Hello\n\nSome short content.'
chunks = chunk_markdown_by_structure(content, max_chunk_chars=100_000)
assert len(chunks) == 1
assert chunks[0].content == content
assert chunks[0].chunk_index == 0
assert chunks[0].total_chunks == 1
assert chunks[0].has_more is False
def test_empty_content(self):
chunks = chunk_markdown_by_structure('', max_chunk_chars=100)
assert len(chunks) == 1
assert chunks[0].content == ''
assert chunks[0].has_more is False
def test_chunk_offsets_cover_full_content(self):
"""Chunk offsets should cover the entire original content without gaps."""
content = '# Header\n\nParagraph one.\n\n# Header 2\n\nParagraph two.'
chunks = chunk_markdown_by_structure(content, max_chunk_chars=20)
# Verify no gaps between consecutive chunks
for i in range(1, len(chunks)):
assert chunks[i].char_offset_start == chunks[i - 1].char_offset_end, (
f'Gap between chunk {i - 1} end ({chunks[i - 1].char_offset_end}) '
f'and chunk {i} start ({chunks[i].char_offset_start})'
)
# First chunk starts at 0
assert chunks[0].char_offset_start == 0
# Last chunk ends at content length
assert chunks[-1].char_offset_end == len(content)
class TestChunkMarkdownHeaders:
"""Header boundary splitting."""
def test_splits_at_header_boundary(self):
"""Chunks should prefer splitting at header boundaries."""
section_a = '# Section A\n\n' + 'x' * 50
section_b = '\n\n# Section B\n\n' + 'y' * 50
content = section_a + section_b
# Set limit so section_a fits but section_a + section_b doesn't
chunks = chunk_markdown_by_structure(content, max_chunk_chars=len(section_a) + 5)
assert len(chunks) >= 2
# First chunk should contain Section A header
assert '# Section A' in chunks[0].content
# Second chunk should start with or contain Section B header
assert '# Section B' in chunks[1].content
class TestChunkMarkdownHeaderPreferred:
"""Header-preferred splitting ensures chunks start at semantic boundaries."""
def test_header_preferred_split_moves_header_to_next_chunk(self):
"""When a header sits in the middle of an overflowing chunk, split before it."""
# Build content: para_a (big) + header_b + para_b (small)
para_a = 'A' * 600
header_b = '# Section B'
para_b = 'B' * 100
content = f'{para_a}\n\n{header_b}\n\n{para_b}'
# Limit forces a split; header is near end of first chunk
chunks = chunk_markdown_by_structure(content, max_chunk_chars=700)
assert len(chunks) >= 2
# The header should be the START of the second chunk, not the end of the first
assert chunks[1].content.lstrip().startswith('# Section B')
# First chunk should NOT contain the header
assert '# Section B' not in chunks[0].content
def test_header_preferred_split_doesnt_create_tiny_chunks(self):
"""Don't split at a header that would make the prefix chunk < 50% of limit."""
header_a = '# Section A'
para_a = 'A' * 30 # very small before header
header_b = '# Section B'
para_b = 'B' * 600
content = f'{header_a}\n\n{para_a}\n\n{header_b}\n\n{para_b}'
# With a limit of 700, header_b is near the start — splitting there would
# leave a tiny prefix chunk. The algo should NOT split there.
chunks = chunk_markdown_by_structure(content, max_chunk_chars=700)
# First chunk should contain both headers (no tiny split)
assert '# Section A' in chunks[0].content
assert '# Section B' in chunks[0].content
class TestChunkMarkdownCodeFence:
"""Code fence blocks never split."""
def test_code_fence_not_split(self):
code_block = '```python\n' + 'x = 1\n' * 100 + '```'
content = '# Title\n\n' + code_block + '\n\n# Footer\n\nDone.'
# Limit smaller than the code block — it should still stay in one chunk (soft limit)
chunks = chunk_markdown_by_structure(content, max_chunk_chars=50)
# Find the chunk containing the code block
code_chunks = [c for c in chunks if '```python' in c.content and '```' in c.content.split('```python')[1]]
assert len(code_chunks) >= 1, 'Code fence should appear intact in at least one chunk'
def test_unclosed_code_fence(self):
"""Unclosed code fence should still be kept as one block."""
content = '# Title\n\n```python\nx = 1\ny = 2'
chunks = chunk_markdown_by_structure(content, max_chunk_chars=100_000)
assert len(chunks) == 1
assert '```python' in chunks[0].content
assert 'y = 2' in chunks[0].content
class TestChunkMarkdownTable:
"""Table rows never split mid-row."""
def test_table_not_split_mid_row(self):
header = '| Name | Value |'
separator = '| --- | --- |'
rows = [f'| item{i} | val{i} |' for i in range(50)]
table = '\n'.join([header, separator] + rows)
content = '# Data\n\n' + table
# Use a limit that would fall in the middle of the table
chunks = chunk_markdown_by_structure(content, max_chunk_chars=200)
for chunk in chunks:
lines = chunk.content.split('\n')
for line in lines:
stripped = line.strip()
if stripped.startswith('|') and stripped.endswith('|'):
# Each table row line should be complete (start and end with |)
assert stripped.count('|') >= 3, f'Incomplete table row: {stripped}'
def test_table_header_in_overlap_for_continuation(self):
"""When a table spans multiple chunks, the header should be in the overlap prefix."""
header = '| Col1 | Col2 |'
separator = '| --- | --- |'
rows = [f'| r{i} | d{i} |' for i in range(100)]
table = '\n'.join([header, separator] + rows)
content = table
# Force split within the table
chunks = chunk_markdown_by_structure(content, max_chunk_chars=300)
if len(chunks) > 1:
# Second chunk should have table header in overlap
assert '| Col1 | Col2 |' in chunks[1].overlap_prefix
assert '| --- | --- |' in chunks[1].overlap_prefix
def test_table_header_carried_across_three_plus_chunks(self):
"""Table header must persist in overlap for ALL continuation chunks, not just the second."""
header = '| Col1 | Col2 |'
separator = '| --- | --- |'
rows = [f'| row{i} | data{i} |' for i in range(200)]
table = '\n'.join([header, separator] + rows)
content = table
# Force many small chunks
chunks = chunk_markdown_by_structure(content, max_chunk_chars=200)
assert len(chunks) >= 3, f'Expected >=3 chunks, got {len(chunks)}'
# Every chunk after the first should carry the table header in its overlap
for i in range(1, len(chunks)):
assert '| Col1 | Col2 |' in chunks[i].overlap_prefix, f'Chunk {i} missing table header in overlap'
assert '| --- | --- |' in chunks[i].overlap_prefix, f'Chunk {i} missing table separator in overlap'
class TestChunkMarkdownListItems:
"""List item continuations stay together."""
def test_list_items_not_split(self):
items = '\n'.join([f'- Item {i} with some description text' for i in range(50)])
content = '# List\n\n' + items
chunks = chunk_markdown_by_structure(content, max_chunk_chars=200)
for chunk in chunks:
lines = chunk.content.split('\n')
for line in lines:
stripped = line.strip()
if stripped.startswith('- '):
# Each list item should be a complete item
assert 'Item' in stripped
class TestChunkMarkdownStartFromChar:
"""start_from_char parameter returns correct chunk."""
def test_start_from_char_returns_correct_chunk(self):
section_a = '# A\n\nContent A here.'
section_b = '\n\n# B\n\nContent B here.'
content = section_a + section_b
# Chunk at header boundaries
all_chunks = chunk_markdown_by_structure(content, max_chunk_chars=len(section_a) + 5)
if len(all_chunks) > 1:
# Request from char offset within second chunk
mid = all_chunks[1].char_offset_start + 1
filtered = chunk_markdown_by_structure(content, max_chunk_chars=len(section_a) + 5, start_from_char=mid)
assert len(filtered) >= 1
assert filtered[0].chunk_index == all_chunks[1].chunk_index
def test_start_from_char_past_end_returns_empty(self):
content = '# Hello\n\nWorld.'
chunks = chunk_markdown_by_structure(content, max_chunk_chars=100_000, start_from_char=99999)
assert chunks == []
def test_start_from_char_zero_returns_all(self):
content = '# Hello\n\nWorld.'
chunks = chunk_markdown_by_structure(content, max_chunk_chars=100_000, start_from_char=0)
assert len(chunks) == 1
class TestChunkMarkdownOverlap:
"""Overlap lines carry context."""
def test_overlap_lines_carry_context(self):
lines_content = '\n'.join([f'Line {i}' for i in range(100)])
content = lines_content
chunks = chunk_markdown_by_structure(content, max_chunk_chars=200, overlap_lines=3)
if len(chunks) > 1:
# Second chunk should have overlap from first chunk
assert chunks[1].overlap_prefix != ''
# Overlap should contain lines from the end of the previous chunk
overlap_lines = chunks[1].overlap_prefix.split('\n')
assert len(overlap_lines) <= 3 + 2 # some flexibility for table headers etc.
def test_no_overlap_on_first_chunk(self):
content = '# A\n\nSome content.\n\n# B\n\nMore content.'
chunks = chunk_markdown_by_structure(content, max_chunk_chars=25)
assert chunks[0].overlap_prefix == ''
class TestChunkMarkdownMixed:
"""Mixed content scenarios."""
def test_paragraph_splitting(self):
"""Paragraphs separated by blank lines are separate blocks."""
p1 = 'First paragraph with text.'
p2 = 'Second paragraph with more text.'
content = f'{p1}\n\n{p2}'
chunks = chunk_markdown_by_structure(content, max_chunk_chars=30)
# Should produce multiple chunks
assert len(chunks) >= 2
def test_single_oversized_block_allowed(self):
"""A single block bigger than max_chunk_chars is allowed (soft limit)."""
big_para = 'x' * 200
content = big_para
chunks = chunk_markdown_by_structure(content, max_chunk_chars=50)
assert len(chunks) == 1
assert chunks[0].content == big_para
# ---------------------------------------------------------------------------
# HTML → markdown → chunk pipeline tests
# ---------------------------------------------------------------------------
class TestHTMLToMarkdownChunking:
"""End-to-end: HTML table → markdown → chunks."""
def test_large_table_produces_valid_chunks(self):
"""200-row HTML table → markdown → chunks should produce valid table rows in every chunk."""
rows = ''.join(f'<tr><td>Row {i}</td><td>Val {i}</td></tr>' for i in range(200))
html = f'<table><thead><tr><th>Name</th><th>Value</th></tr></thead><tbody>{rows}</tbody></table>'
markdown = md(html, heading_style='ATX')
chunks = chunk_markdown_by_structure(markdown, max_chunk_chars=500)
assert len(chunks) > 1, 'Should produce multiple chunks for 200 rows'
for chunk in chunks:
lines = chunk.content.strip().split('\n')
for line in lines:
s = line.strip()
if s.startswith('|') and s.endswith('|'):
# Every table line should have consistent column count
assert s.count('|') >= 3
def test_table_without_thead_normalization(self):
"""Table with <th> in first <tr> but no <thead> should still produce proper markdown."""
html = '<table><tr><th>A</th><th>B</th></tr><tr><td>1</td><td>2</td></tr><tr><td>3</td><td>4</td></tr></table>'
markdown = md(html, heading_style='ATX')
# Verify markdownify produced a proper table (with separator row)
assert '---' in markdown or '| A |' in markdown
# ---------------------------------------------------------------------------
# Integration tests — require browser + httpserver
# ---------------------------------------------------------------------------
class TestTableNormalizationIntegration:
"""Integration tests using browser session and httpserver."""
async def test_table_without_thead_normalized_via_serializer(self, browser_session, httpserver: HTTPServer):
"""Tables without <thead> should get normalized by HTMLSerializer during extraction."""
html = """
<html><body>
<table>
<tr><th>Header1</th><th>Header2</th></tr>
<tr><td>data1</td><td>data2</td></tr>
<tr><td>data3</td><td>data4</td></tr>
</table>
</body></html>
"""
httpserver.expect_request('/table-test').respond_with_data(html, content_type='text/html')
url = httpserver.url_for('/table-test')
await browser_session.navigate_to(url)
from browser_use.dom.markdown_extractor import extract_clean_markdown
content, _ = await extract_clean_markdown(browser_session=browser_session)
# Should have proper markdown table with separator
assert '|' in content
# The header should be present
assert 'Header1' in content
assert 'Header2' in content
async def test_large_table_extraction_preserves_structure(self, browser_session, httpserver: HTTPServer):
"""Large table extraction should produce structure-aware chunks."""
rows = ''.join(f'<tr><td>Name{i}</td><td>Value{i}</td></tr>' for i in range(300))
html = f"""
<html><body>
<table>
<tr><th>Name</th><th>Value</th></tr>
{rows}
</table>
</body></html>
"""
httpserver.expect_request('/big-table').respond_with_data(html, content_type='text/html')
url = httpserver.url_for('/big-table')
await browser_session.navigate_to(url)
from browser_use.dom.markdown_extractor import extract_clean_markdown
content, _ = await extract_clean_markdown(browser_session=browser_session)
# Chunk with a small limit to force multiple chunks
chunks = chunk_markdown_by_structure(content, max_chunk_chars=2000)
# Should produce multiple chunks
assert len(chunks) > 1
# Each chunk should have complete table rows
for chunk in chunks:
for line in chunk.content.split('\n'):
s = line.strip()
if s.startswith('|') and s.endswith('|'):
assert s.count('|') >= 3, f'Incomplete table row: {s}'
+146
View File
@@ -0,0 +1,146 @@
"""Tests for markdown extractor preprocessing."""
from browser_use.dom.markdown_extractor import _preprocess_markdown_content
class TestPreprocessMarkdownContent:
"""Tests for _preprocess_markdown_content function."""
def test_preserves_short_lines(self):
"""Short lines (1-2 chars) should be preserved, not removed."""
content = '# Items\na\nb\nc\nOK\nNo'
filtered, _ = _preprocess_markdown_content(content)
assert 'a' in filtered.split('\n')
assert 'b' in filtered.split('\n')
assert 'c' in filtered.split('\n')
assert 'OK' in filtered.split('\n')
assert 'No' in filtered.split('\n')
def test_preserves_single_digit_numbers(self):
"""Single digit page numbers should be preserved."""
content = 'Page navigation:\n1\n2\n3\n10'
filtered, _ = _preprocess_markdown_content(content)
lines = filtered.split('\n')
assert '1' in lines
assert '2' in lines
assert '3' in lines
assert '10' in lines
def test_preserves_markdown_list_items(self):
"""Markdown list items with short content should be preserved."""
content = 'Shopping list:\n- a\n- b\n- OK\n- No'
filtered, _ = _preprocess_markdown_content(content)
assert '- a' in filtered
assert '- b' in filtered
assert '- OK' in filtered
assert '- No' in filtered
def test_preserves_state_codes(self):
"""Two-letter state codes should be preserved."""
content = 'States:\nCA\nNY\nTX'
filtered, _ = _preprocess_markdown_content(content)
lines = filtered.split('\n')
assert 'CA' in lines
assert 'NY' in lines
assert 'TX' in lines
def test_removes_empty_lines(self):
"""Empty and whitespace-only lines should be removed."""
content = 'Header\n\n \n\nContent'
filtered, _ = _preprocess_markdown_content(content)
# Should not have empty lines
for line in filtered.split('\n'):
assert line.strip(), f'Found empty line in output: {repr(line)}'
def test_removes_large_json_blobs(self):
"""Large JSON-like lines (>100 chars) should be removed."""
# Create a JSON blob > 100 chars
json_blob = '{"key": "' + 'x' * 100 + '"}'
content = f'Header\n{json_blob}\nFooter'
filtered, _ = _preprocess_markdown_content(content)
assert json_blob not in filtered
assert 'Header' in filtered
assert 'Footer' in filtered
def test_preserves_small_json(self):
"""Small JSON objects (<100 chars) should be preserved."""
small_json = '{"key": "value"}'
content = f'Header\n{small_json}\nFooter'
filtered, _ = _preprocess_markdown_content(content)
assert small_json in filtered
def test_compresses_multiple_newlines(self):
"""4+ consecutive newlines should be compressed to max_newlines."""
content = 'Header\n\n\n\n\nFooter'
filtered, _ = _preprocess_markdown_content(content, max_newlines=2)
# After filtering empty lines, we should have just Header and Footer
lines = [line for line in filtered.split('\n') if line.strip()]
assert lines == ['Header', 'Footer']
def test_returns_chars_filtered_count(self):
"""Should return count of characters removed."""
content = 'Header\n\n\n\n\nFooter'
_, chars_filtered = _preprocess_markdown_content(content)
assert chars_filtered > 0
def test_strips_result(self):
"""Result should be stripped of leading/trailing whitespace."""
content = ' \n\nContent\n\n '
filtered, _ = _preprocess_markdown_content(content)
assert not filtered.startswith(' ')
assert not filtered.startswith('\n')
assert not filtered.endswith(' ')
assert not filtered.endswith('\n')
class TestPreservesLinksAndEncoding:
"""Regression tests: markdown links and percent-encoded URLs must survive filtering."""
def test_preserves_long_markdown_link_lines(self):
"""A long markdown link line (>100 chars) must not be dropped by the JSON heuristic."""
link = '[Read the full quarterly earnings report for fiscal year 2025](https://example.com/investor-relations/reports/q4-2025-earnings-full.pdf)'
assert len(link) > 100
content = f'Header\n{link}\nFooter'
filtered, _ = _preprocess_markdown_content(content)
assert link in filtered
def test_preserves_long_image_link_lines(self):
"""A long clickable-image line (starts with [![) must not be dropped."""
line = '[![Product photo of the deluxe widget](https://cdn.example.com/images/products/deluxe-widget-hero.jpg)](https://example.com/products/deluxe-widget)'
assert len(line) > 100
content = f'Intro\n{line}\nOutro'
filtered, _ = _preprocess_markdown_content(content)
assert line in filtered
def test_removes_long_json_array_lines(self):
"""A valid JSON array blob >100 chars should still be dropped."""
json_array = '[' + ', '.join(f'{{"id": {i}, "name": "item-{i}"}}' for i in range(10)) + ']'
assert len(json_array) > 100
content = f'Header\n{json_array}\nFooter'
filtered, _ = _preprocess_markdown_content(content)
assert json_array not in filtered
assert 'Header' in filtered
def test_preserves_percent_encoded_urls(self):
"""Percent-encodings in URLs must survive HTML -> markdown conversion."""
from browser_use.dom.markdown_extractor import convert_html_to_markdown
html = '<p>See <a href="https://example.com/my%20file%2Fv2?q=a%26b">the doc</a> here</p>'
content, _, _ = convert_html_to_markdown(html)
assert '%20' in content
assert '%2F' in content
assert '%26' in content
+290
View File
@@ -0,0 +1,290 @@
"""
Tests for multi_act() page-change guards.
Verifies:
1. Metadata: terminates_sequence flags are set correctly on built-in actions
2. Static guard: actions tagged terminates_sequence abort remaining queued actions
3. Runtime guard: URL/focus changes detected after click-on-link abort remaining actions
4. Safe chain: multiple inputs execute without interruption
Usage:
uv run pytest tests/ci/test_multi_act_guards.py -v -s
"""
import asyncio
import pytest
from pytest_httpserver import HTTPServer
from browser_use.agent.service import Agent
from browser_use.browser import BrowserSession
from browser_use.browser.profile import BrowserProfile
from browser_use.tools.service import Tools
from tests.ci.conftest import create_mock_llm
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture(scope='session')
def http_server():
"""Test HTTP server with pages for guard tests."""
server = HTTPServer()
server.start()
server.expect_request('/form').respond_with_data(
"""<html><head><title>Form Page</title></head><body>
<h1>Form</h1>
<input id="field1" type="text" placeholder="Field 1" />
<input id="field2" type="text" placeholder="Field 2" />
<input id="field3" type="text" placeholder="Field 3" />
<button id="submit" type="submit">Submit</button>
</body></html>""",
content_type='text/html',
)
server.expect_request('/page_a').respond_with_data(
"""<html><head><title>Page A</title></head><body>
<h1>Page A</h1>
<a id="link_b" href="/page_b">Go to Page B</a>
</body></html>""",
content_type='text/html',
)
server.expect_request('/page_b').respond_with_data(
"""<html><head><title>Page B</title></head><body>
<h1>Page B</h1>
<p>You arrived at Page B</p>
</body></html>""",
content_type='text/html',
)
server.expect_request('/static').respond_with_data(
"""<html><head><title>Static Page</title></head><body>
<h1>Static</h1>
<p>Nothing changes here</p>
<input id="safe_input" type="text" />
</body></html>""",
content_type='text/html',
)
yield server
server.stop()
@pytest.fixture(scope='session')
def base_url(http_server):
return f'http://{http_server.host}:{http_server.port}'
@pytest.fixture(scope='module')
async def browser_session():
session = BrowserSession(
browser_profile=BrowserProfile(
headless=True,
user_data_dir=None,
keep_alive=True,
)
)
await session.start()
yield session
await session.kill()
await session.event_bus.stop(clear=True, timeout=5)
@pytest.fixture(scope='function')
def tools():
return Tools()
# ---------------------------------------------------------------------------
# 1. Metadata tests — verify terminates_sequence flags
# ---------------------------------------------------------------------------
class TestTerminatesSequenceMetadata:
"""Verify that built-in actions have correct terminates_sequence flags."""
def test_navigate_terminates(self, tools):
action = tools.registry.registry.actions.get('navigate')
assert action is not None
assert action.terminates_sequence is True
def test_search_terminates(self, tools):
action = tools.registry.registry.actions.get('search')
assert action is not None
assert action.terminates_sequence is True
def test_go_back_terminates(self, tools):
action = tools.registry.registry.actions.get('go_back')
assert action is not None
assert action.terminates_sequence is True
def test_switch_terminates(self, tools):
action = tools.registry.registry.actions.get('switch')
assert action is not None
assert action.terminates_sequence is True
def test_click_does_not_terminate(self, tools):
action = tools.registry.registry.actions.get('click')
assert action is not None
assert action.terminates_sequence is False
def test_input_does_not_terminate(self, tools):
action = tools.registry.registry.actions.get('input')
assert action is not None
assert action.terminates_sequence is False
def test_scroll_does_not_terminate(self, tools):
action = tools.registry.registry.actions.get('scroll')
assert action is not None
assert action.terminates_sequence is False
def test_extract_does_not_terminate(self, tools):
action = tools.registry.registry.actions.get('extract')
assert action is not None
assert action.terminates_sequence is False
def test_evaluate_terminates(self, tools):
"""evaluate() can mutate the DOM in unpredictable ways (e.g. dismiss cookie overlays),
so any actions queued after it should be skipped to avoid stale element references."""
action = tools.registry.registry.actions.get('evaluate')
assert action is not None
assert action.terminates_sequence is True
# ---------------------------------------------------------------------------
# 2. Static guard — navigate as non-last action skips remaining
# ---------------------------------------------------------------------------
class TestStaticGuard:
"""Verify that terminates_sequence actions abort the remaining queue."""
async def test_navigate_aborts_remaining_actions(self, browser_session, base_url, tools):
"""When navigate is action 2/3, action 3 should never execute."""
# Start on a known page
await tools.navigate(url=f'{base_url}/static', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
# Build action models: [scroll_down, navigate_to_page_a, scroll_down]
ActionModel = tools.registry.create_action_model()
actions = [
ActionModel.model_validate({'scroll': {'down': True, 'pages': 1}}),
ActionModel.model_validate({'navigate': {'url': f'{base_url}/page_a'}}),
ActionModel.model_validate({'scroll': {'down': True, 'pages': 1}}),
]
mock_llm = create_mock_llm()
agent = Agent(task='test', llm=mock_llm, browser_session=browser_session, tools=tools)
results = await agent.multi_act(actions)
# Should have executed exactly 2 actions (scroll + navigate), third skipped
assert len(results) == 2, f'Expected 2 results but got {len(results)}: {results}'
# Verify we actually navigated
url = await browser_session.get_current_page_url()
assert '/page_a' in url
async def test_go_back_aborts_remaining_actions(self, browser_session, base_url, tools):
"""go_back should abort remaining queued actions."""
# Navigate to page_a then page_b so go_back has somewhere to go
await tools.navigate(url=f'{base_url}/page_a', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.3)
await tools.navigate(url=f'{base_url}/page_b', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.3)
ActionModel = tools.registry.create_action_model()
actions = [
ActionModel.model_validate({'go_back': {}}),
ActionModel.model_validate({'scroll': {'down': True, 'pages': 1}}),
]
mock_llm = create_mock_llm()
agent = Agent(task='test', llm=mock_llm, browser_session=browser_session, tools=tools)
results = await agent.multi_act(actions)
# go_back should terminate the sequence — only 1 result
assert len(results) == 1, f'Expected 1 result but got {len(results)}: {results}'
# ---------------------------------------------------------------------------
# 3. Runtime guard — click on link changes URL, remaining actions skipped
# ---------------------------------------------------------------------------
class TestRuntimeGuard:
"""Verify that URL/focus changes detected at runtime abort remaining actions."""
async def test_click_link_aborts_remaining(self, browser_session, base_url, tools):
"""Click a link that navigates to another page — remaining actions skipped."""
await tools.navigate(url=f'{base_url}/page_a', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
# Get the selector map to find the link index
state = await browser_session.get_browser_state_summary()
assert state.dom_state is not None
selector_map = state.dom_state.selector_map
# Find the link element (a#link_b)
link_index = None
for idx, element in selector_map.items():
if hasattr(element, 'tag_name') and element.tag_name == 'a':
link_index = idx
break
assert link_index is not None, 'Could not find link element in selector map'
ActionModel = tools.registry.create_action_model()
actions = [
ActionModel.model_validate({'click': {'index': link_index}}),
ActionModel.model_validate({'scroll': {'down': True, 'pages': 1}}),
ActionModel.model_validate({'scroll': {'down': True, 'pages': 1}}),
]
mock_llm = create_mock_llm()
agent = Agent(task='test', llm=mock_llm, browser_session=browser_session, tools=tools)
results = await agent.multi_act(actions)
# Click navigated to page_b — runtime guard should stop at 1
assert len(results) == 1, f'Expected 1 result but got {len(results)}: {results}'
# Verify we're on page_b
url = await browser_session.get_current_page_url()
assert '/page_b' in url
# ---------------------------------------------------------------------------
# 4. Safe chain — multiple non-page-changing actions all execute
# ---------------------------------------------------------------------------
class TestSafeChain:
"""Verify that non-page-changing actions execute without interruption."""
async def test_multiple_scrolls_all_execute(self, browser_session, base_url, tools):
"""Multiple scroll actions should all execute."""
await tools.navigate(url=f'{base_url}/static', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
ActionModel = tools.registry.create_action_model()
actions = [
ActionModel.model_validate({'scroll': {'down': True, 'pages': 0.5}}),
ActionModel.model_validate({'scroll': {'down': True, 'pages': 0.5}}),
ActionModel.model_validate({'scroll': {'down': False, 'pages': 0.5}}),
]
mock_llm = create_mock_llm()
agent = Agent(task='test', llm=mock_llm, browser_session=browser_session, tools=tools)
results = await agent.multi_act(actions)
# All 3 scrolls should execute
assert len(results) == 3, f'Expected 3 results but got {len(results)}: {results}'
# None should have errors
for r in results:
assert r.error is None, f'Unexpected error: {r.error}'
+153
View File
@@ -0,0 +1,153 @@
import pytest
from browser_use.llm.openrouter.chat import ChatOpenRouter
from browser_use.llm.views import ChatInvokeUsage
from browser_use.tokens import openrouter_pricing
from browser_use.tokens.openrouter_pricing import model_pricing_from_openrouter_metadata
from browser_use.tokens.service import TokenCost
from browser_use.tokens.views import ModelPricing
def _openrouter_metadata() -> dict:
return {
'id': 'deepseek/deepseek-v4-flash',
'context_length': 1_048_576,
'top_provider': {'max_completion_tokens': 16_384},
'pricing': {
'prompt': '0.0000001',
'completion': '0.0000002',
'input_cache_read': '0.00000002',
'input_cache_write': '0.00000003',
},
}
def test_model_pricing_from_openrouter_metadata() -> None:
pricing = model_pricing_from_openrouter_metadata('deepseek/deepseek-v4-flash', _openrouter_metadata())
assert pricing is not None
assert pricing.model == 'deepseek/deepseek-v4-flash'
assert pricing.input_cost_per_token == pytest.approx(0.10 / 1_000_000)
assert pricing.output_cost_per_token == pytest.approx(0.20 / 1_000_000)
assert pricing.cache_read_input_token_cost == pytest.approx(0.02 / 1_000_000)
assert pricing.cache_creation_input_token_cost == pytest.approx(0.03 / 1_000_000)
assert pricing.max_tokens == 1_048_576
assert pricing.max_input_tokens == 1_048_576
assert pricing.max_output_tokens == 16_384
async def test_openrouter_pricing_accepts_litellm_prefixed_model_ids(monkeypatch: pytest.MonkeyPatch) -> None:
async def fake_get_openrouter_models_metadata(refresh: bool = False) -> dict[str, dict]:
return {'deepseek/deepseek-v4-flash': _openrouter_metadata()}
monkeypatch.setattr(openrouter_pricing, 'get_openrouter_models_metadata', fake_get_openrouter_models_metadata)
pricing = await openrouter_pricing.get_openrouter_model_pricing('openrouter/deepseek/deepseek-v4-flash')
assert pricing is not None
assert pricing.model == 'openrouter/deepseek/deepseek-v4-flash'
assert pricing.input_cost_per_token == pytest.approx(0.10 / 1_000_000)
async def test_token_cost_falls_back_to_openrouter_pricing(monkeypatch: pytest.MonkeyPatch) -> None:
async def fake_openrouter_pricing(model_name: str) -> ModelPricing:
assert model_name == 'deepseek/deepseek-v4-flash'
return ModelPricing(
model=model_name,
input_cost_per_token=0.10 / 1_000_000,
output_cost_per_token=0.20 / 1_000_000,
cache_read_input_token_cost=0.02 / 1_000_000,
cache_creation_input_token_cost=None,
max_tokens=1_048_576,
max_input_tokens=1_048_576,
max_output_tokens=16_384,
)
monkeypatch.setattr('browser_use.tokens.service.get_openrouter_model_pricing', fake_openrouter_pricing)
token_cost = TokenCost(include_cost=True)
token_cost._initialized = True
token_cost._pricing_data = {}
pricing = await token_cost.get_model_pricing('deepseek/deepseek-v4-flash')
assert pricing is not None
assert pricing.input_cost_per_token == pytest.approx(0.10 / 1_000_000)
assert pricing.output_cost_per_token == pytest.approx(0.20 / 1_000_000)
async def test_calculate_cost_uses_openrouter_cache_pricing(monkeypatch: pytest.MonkeyPatch) -> None:
async def fake_openrouter_pricing(model_name: str) -> ModelPricing:
return ModelPricing(
model=model_name,
input_cost_per_token=0.10 / 1_000_000,
output_cost_per_token=0.20 / 1_000_000,
cache_read_input_token_cost=0.02 / 1_000_000,
cache_creation_input_token_cost=None,
max_tokens=None,
max_input_tokens=None,
max_output_tokens=None,
)
monkeypatch.setattr('browser_use.tokens.service.get_openrouter_model_pricing', fake_openrouter_pricing)
token_cost = TokenCost(include_cost=True)
token_cost._initialized = True
token_cost._pricing_data = {}
cost = await token_cost.calculate_cost(
'deepseek/deepseek-v4-flash',
ChatInvokeUsage(
prompt_tokens=110,
prompt_cached_tokens=10,
prompt_cache_creation_tokens=None,
prompt_image_tokens=None,
completion_tokens=20,
total_tokens=130,
),
)
assert cost is not None
assert cost.new_prompt_cost == pytest.approx(100 * 0.10 / 1_000_000)
assert cost.prompt_read_cached_cost == pytest.approx(10 * 0.02 / 1_000_000)
assert cost.completion_cost == pytest.approx(20 * 0.20 / 1_000_000)
async def test_registered_openrouter_llm_forces_openrouter_pricing(monkeypatch: pytest.MonkeyPatch) -> None:
seen_model_names = []
async def fake_openrouter_pricing(model_name: str) -> ModelPricing:
seen_model_names.append(model_name)
return ModelPricing(
model=model_name,
input_cost_per_token=0.10 / 1_000_000,
output_cost_per_token=0.20 / 1_000_000,
cache_read_input_token_cost=None,
cache_creation_input_token_cost=None,
max_tokens=None,
max_input_tokens=None,
max_output_tokens=None,
)
monkeypatch.setattr('browser_use.tokens.service.get_openrouter_model_pricing', fake_openrouter_pricing)
token_cost = TokenCost(include_cost=True)
token_cost._initialized = True
token_cost._pricing_data = {'openai/gpt-4o-mini': {'input_cost_per_token': 99, 'output_cost_per_token': 99}}
token_cost.register_llm(ChatOpenRouter(model='openai/gpt-4o-mini', api_key='test-key'))
cost = await token_cost.calculate_cost(
'openai/gpt-4o-mini',
ChatInvokeUsage(
prompt_tokens=10,
prompt_cached_tokens=None,
prompt_cache_creation_tokens=None,
prompt_image_tokens=None,
completion_tokens=5,
total_tokens=15,
),
)
assert cost is not None
assert seen_model_names == ['openrouter/openai/gpt-4o-mini']
assert cost.total_cost == pytest.approx(10 * 0.10 / 1_000_000 + 5 * 0.20 / 1_000_000)
+106
View File
@@ -0,0 +1,106 @@
"""Lock per-step varying metadata (step counter, wall-clock date) at the tail of the user message.
The agent's user message looks roughly:
<user_request>...</user_request>
<agent_history>...</agent_history>
<agent_state>...</agent_state>
<browser_state>...</browser_state>
[<read_state>...</read_state>]
[<page_specific_actions>...</page_specific_actions>]
[unavailable skills info]
<step_info>...</step_info> <-- new home
`<step_info>` carries a step counter and `datetime.now()`, both of which change between
calls. Keeping it at the very end means everything above it can in principle be cached;
moving it back into `<agent_state>` would silently shrink the cacheable prefix.
"""
from browser_use.agent.prompts import AgentMessagePrompt
from browser_use.agent.views import AgentStepInfo
from browser_use.browser.views import BrowserStateSummary, PageInfo, TabInfo
from browser_use.dom.views import SerializedDOMState
from browser_use.filesystem.file_system import FileSystem
def _make_prompt(tmp_path, step_number: int = 3) -> AgentMessagePrompt:
dom_state = SerializedDOMState(_root=None, selector_map={})
bs = BrowserStateSummary(
url='https://example.test/foo',
title='Test',
tabs=[TabInfo(target_id='abcd1234', url='https://example.test/foo', title='Test')],
page_info=PageInfo(
viewport_width=1280,
viewport_height=720,
page_width=1280,
page_height=1440,
scroll_x=0,
scroll_y=0,
pixels_above=0,
pixels_below=720,
pixels_left=0,
pixels_right=0,
),
dom_state=dom_state,
is_pdf_viewer=False,
recent_events=None,
closed_popup_messages=[],
screenshot=None,
)
fs = FileSystem(base_dir=str(tmp_path), create_default_files=False)
return AgentMessagePrompt(
browser_state_summary=bs,
file_system=fs,
agent_history_description='<step>existing history</step>',
task='Do the test thing',
step_info=AgentStepInfo(step_number=step_number, max_steps=50),
)
def test_step_info_lives_at_suffix(tmp_path):
content = _make_prompt(tmp_path).get_user_message(use_vision=False).content
assert isinstance(content, str)
assert '<step_info>' in content
assert content.index('<user_request>') < content.index('<agent_history>')
assert content.index('</agent_history>') < content.index('<agent_state>')
# Suffix: step_info must come after agent_state and browser_state.
assert content.index('<agent_state>') < content.index('<step_info>')
assert content.index('<browser_state>') < content.index('<step_info>')
# And it must NOT live inside <agent_state> any more.
state_start = content.index('<agent_state>')
state_end = content.index('</agent_state>')
assert '<step_info>' not in content[state_start:state_end], 'per-step metadata leaked into <agent_state> prefix region'
assert '<user_request>' not in content[state_start:state_end], 'user request leaked back into <agent_state>'
def test_agent_history_end_marker_is_present_once(tmp_path):
content = _make_prompt(tmp_path).get_user_message(use_vision=False).content
assert isinstance(content, str)
assert content.count('</agent_history>') == 1, (
'super important: LLM gateway cache splitting expects exactly one </agent_history> marker'
)
def test_prefix_up_to_step_info_is_stable_across_steps(tmp_path):
"""Step number and date can change; the bytes before <step_info> must not."""
a = _make_prompt(tmp_path, step_number=3).get_user_message(use_vision=False).content
b = _make_prompt(tmp_path, step_number=4).get_user_message(use_vision=False).content
assert isinstance(a, str) and isinstance(b, str)
prefix_end = a.index('<step_info>')
assert a[:prefix_end] == b[:prefix_end], 'message bytes diverged before <step_info> — step counter is leaking into the prefix'
# Sanity: the tails do differ (step counter advanced).
assert a[prefix_end:] != b[prefix_end:]
def test_agent_state_block_unaffected_by_step_change(tmp_path):
"""<agent_state> should not include per-step-varying metadata."""
a = _make_prompt(tmp_path, step_number=3).get_user_message(use_vision=False).content
b = _make_prompt(tmp_path, step_number=4).get_user_message(use_vision=False).content
assert isinstance(a, str) and isinstance(b, str)
def agent_state_block(s: str) -> str:
return s[s.index('<agent_state>') : s.index('</agent_state>')]
assert agent_state_block(a) == agent_state_block(b)
File diff suppressed because it is too large Load Diff
+227
View File
@@ -0,0 +1,227 @@
"""
Tests for sandbox structured output handling.
Tests that output_model_schema works correctly when using @sandbox decorator,
specifically that the _output_model_schema private attribute is preserved
through serialization/deserialization.
"""
from pydantic import BaseModel
from browser_use.agent.views import ActionResult, AgentHistory, AgentHistoryList, BrowserStateHistory
from browser_use.sandbox.sandbox import _parse_with_type_annotation
class ExtractedData(BaseModel):
"""Example structured output model"""
title: str
price: float
in_stock: bool
class NestedModel(BaseModel):
"""Nested model for testing complex structures"""
items: list[ExtractedData]
total_count: int
class TestGetStructuredOutput:
"""Tests for AgentHistoryList.get_structured_output method"""
def test_get_structured_output_parses_final_result(self):
"""Test that get_structured_output correctly parses final result with provided schema"""
# Create history with structured JSON as final result
json_result = '{"title": "Test Product", "price": 29.99, "in_stock": true}'
history = AgentHistoryList(
history=[
AgentHistory(
model_output=None,
result=[ActionResult(extracted_content=json_result, is_done=True)],
state=BrowserStateHistory(url='https://example.com', title='Test', tabs=[], interacted_element=[]),
)
]
)
# Use get_structured_output with explicit schema
result = history.get_structured_output(ExtractedData)
assert result is not None
assert isinstance(result, ExtractedData)
assert result.title == 'Test Product'
assert result.price == 29.99
assert result.in_stock is True
def test_get_structured_output_returns_none_when_no_final_result(self):
"""Test that get_structured_output returns None when there's no final result"""
history = AgentHistoryList(
history=[
AgentHistory(
model_output=None,
result=[ActionResult(extracted_content=None)],
state=BrowserStateHistory(url='https://example.com', title='Test', tabs=[], interacted_element=[]),
)
]
)
result = history.get_structured_output(ExtractedData)
assert result is None
def test_get_structured_output_with_nested_model(self):
"""Test get_structured_output works with nested Pydantic models"""
json_result = """
{
"items": [
{"title": "Item 1", "price": 10.0, "in_stock": true},
{"title": "Item 2", "price": 20.0, "in_stock": false}
],
"total_count": 2
}
"""
history = AgentHistoryList(
history=[
AgentHistory(
model_output=None,
result=[ActionResult(extracted_content=json_result, is_done=True)],
state=BrowserStateHistory(url='https://example.com', title='Test', tabs=[], interacted_element=[]),
)
]
)
result = history.get_structured_output(NestedModel)
assert result is not None
assert len(result.items) == 2
assert result.items[0].title == 'Item 1'
assert result.total_count == 2
class TestSandboxStructuredOutputParsing:
"""Tests for _parse_with_type_annotation handling of AgentHistoryList[T]"""
def test_parse_agent_history_list_without_generic(self):
"""Test parsing AgentHistoryList without generic parameter"""
data = {
'history': [
{
'model_output': None,
'result': [{'extracted_content': '{"title": "Test", "price": 9.99, "in_stock": true}', 'is_done': True}],
'state': {'url': 'https://example.com', 'title': 'Test', 'tabs': []},
}
]
}
result = _parse_with_type_annotation(data, AgentHistoryList)
assert isinstance(result, AgentHistoryList)
assert len(result.history) == 1
# Without generic, _output_model_schema should be None
assert result._output_model_schema is None
def test_parse_agent_history_list_with_generic_parameter(self):
"""Test parsing AgentHistoryList[ExtractedData] preserves output model schema"""
data = {
'history': [
{
'model_output': None,
'result': [{'extracted_content': '{"title": "Test", "price": 9.99, "in_stock": true}', 'is_done': True}],
'state': {'url': 'https://example.com', 'title': 'Test', 'tabs': []},
}
]
}
# Parse with generic type annotation
result = _parse_with_type_annotation(data, AgentHistoryList[ExtractedData])
assert isinstance(result, AgentHistoryList)
assert len(result.history) == 1
# With generic, _output_model_schema should be set
assert result._output_model_schema is ExtractedData
# Now structured_output property should work
structured = result.structured_output
assert structured is not None
assert isinstance(structured, ExtractedData)
assert structured.title == 'Test'
assert structured.price == 9.99
assert structured.in_stock is True
def test_parse_agent_history_list_structured_output_after_sandbox(self):
"""Simulate full sandbox round-trip with AgentHistoryList[T]"""
# This simulates what happens when sandbox returns data
json_content = '{"title": "Product", "price": 49.99, "in_stock": false}'
data = {
'history': [
{
'model_output': None,
'result': [{'extracted_content': json_content, 'is_done': True}],
'state': {'url': 'https://shop.com', 'title': 'Shop', 'tabs': []},
}
]
}
# Sandbox parses with return type annotation AgentHistoryList[ExtractedData]
result = _parse_with_type_annotation(data, AgentHistoryList[ExtractedData])
# User accesses structured_output property
output = result.structured_output
assert output is not None
assert output.title == 'Product'
assert output.price == 49.99
assert output.in_stock is False
class TestStructuredOutputPropertyFallback:
"""Tests for structured_output property behavior with and without _output_model_schema"""
def test_structured_output_property_works_when_schema_set(self):
"""Test structured_output property works when _output_model_schema is set"""
json_result = '{"title": "Test", "price": 5.0, "in_stock": true}'
history = AgentHistoryList(
history=[
AgentHistory(
model_output=None,
result=[ActionResult(extracted_content=json_result, is_done=True)],
state=BrowserStateHistory(url='https://example.com', title='Test', tabs=[], interacted_element=[]),
)
]
)
# Manually set the schema (as Agent.run() does)
history._output_model_schema = ExtractedData
result = history.structured_output
assert result is not None
assert isinstance(result, ExtractedData)
assert result.title == 'Test'
def test_structured_output_property_returns_none_without_schema(self):
"""Test structured_output property returns None when _output_model_schema is not set"""
json_result = '{"title": "Test", "price": 5.0, "in_stock": true}'
history = AgentHistoryList(
history=[
AgentHistory(
model_output=None,
result=[ActionResult(extracted_content=json_result, is_done=True)],
state=BrowserStateHistory(url='https://example.com', title='Test', tabs=[], interacted_element=[]),
)
]
)
# Don't set _output_model_schema
result = history.structured_output
# Property returns None because schema is not set
assert result is None
# But get_structured_output with explicit schema works
explicit_result = history.get_structured_output(ExtractedData)
assert explicit_result is not None
assert explicit_result.title == 'Test'
+173
View File
@@ -0,0 +1,173 @@
"""Test that screenshot action is excluded when use_vision != 'auto'."""
import pytest
from browser_use.agent.service import Agent
from browser_use.browser.profile import BrowserProfile
from browser_use.browser.session import BrowserSession
from browser_use.tools.service import Tools
from tests.ci.conftest import create_mock_llm
@pytest.fixture(scope='function')
async def browser_session():
session = BrowserSession(browser_profile=BrowserProfile(headless=True))
await session.start()
yield session
await session.kill()
def test_screenshot_excluded_with_use_vision_false():
"""Test that screenshot action is excluded when use_vision=False."""
mock_llm = create_mock_llm(actions=['{"action": [{"done": {"text": "test", "success": true}}]}'])
agent = Agent(
task='test',
llm=mock_llm,
use_vision=False,
)
# Verify screenshot is not in the registry
assert 'screenshot' not in agent.tools.registry.registry.actions, 'Screenshot should be excluded when use_vision=False'
def test_screenshot_excluded_with_use_vision_true():
"""Test that screenshot action is excluded when use_vision=True."""
mock_llm = create_mock_llm(actions=['{"action": [{"done": {"text": "test", "success": true}}]}'])
agent = Agent(
task='test',
llm=mock_llm,
use_vision=True,
)
# Verify screenshot is not in the registry
assert 'screenshot' not in agent.tools.registry.registry.actions, 'Screenshot should be excluded when use_vision=True'
def test_screenshot_included_with_use_vision_auto():
"""Test that screenshot action is included when use_vision='auto'."""
mock_llm = create_mock_llm(actions=['{"action": [{"done": {"text": "test", "success": true}}]}'])
agent = Agent(
task='test',
llm=mock_llm,
use_vision='auto',
)
# Verify screenshot IS in the registry
assert 'screenshot' in agent.tools.registry.registry.actions, 'Screenshot should be included when use_vision="auto"'
def test_screenshot_excluded_with_custom_tools_and_use_vision_false():
"""Test that screenshot action is excluded even when user passes custom tools and use_vision=False.
This is the critical test case that verifies the fix:
When users pass their own Tools instance with screenshot included,
the Agent should still enforce the exclusion if use_vision != 'auto'.
"""
mock_llm = create_mock_llm(actions=['{"action": [{"done": {"text": "test", "success": true}}]}'])
# Create custom tools that includes screenshot action
custom_tools = Tools()
assert 'screenshot' in custom_tools.registry.registry.actions, 'Custom tools should have screenshot by default'
# Pass custom tools to agent with use_vision=False
agent = Agent(
task='test',
llm=mock_llm,
tools=custom_tools,
use_vision=False,
)
# Verify screenshot is excluded even though user passed custom tools
assert 'screenshot' not in agent.tools.registry.registry.actions, (
'Screenshot should be excluded when use_vision=False, even with custom tools'
)
def test_screenshot_excluded_with_custom_tools_and_use_vision_true():
"""Test that screenshot action is excluded even when user passes custom tools and use_vision=True.
This is another critical test case:
When users pass their own Tools instance with screenshot included,
the Agent should still enforce the exclusion if use_vision != 'auto'.
"""
mock_llm = create_mock_llm(actions=['{"action": [{"done": {"text": "test", "success": true}}]}'])
# Create custom tools - by default Tools() includes screenshot
# (unless exclude_actions is passed)
custom_tools = Tools()
# Note: We check if screenshot exists in the default set, but it might not
# exist if use_vision defaults have changed. The key is that after passing
# to Agent with use_vision=True, it should be excluded.
has_screenshot_before = 'screenshot' in custom_tools.registry.registry.actions
# Pass custom tools to agent with use_vision=True
agent = Agent(
task='test',
llm=mock_llm,
tools=custom_tools,
use_vision=True,
)
# Verify screenshot is excluded even though user passed custom tools
# The key test: screenshot should be excluded after Agent init
assert 'screenshot' not in agent.tools.registry.registry.actions, (
f'Screenshot should be excluded when use_vision=True, even with custom tools (had screenshot before: {has_screenshot_before})'
)
def test_screenshot_included_with_custom_tools_and_use_vision_auto():
"""Test that screenshot action is kept when user passes custom tools and use_vision='auto'."""
mock_llm = create_mock_llm(actions=['{"action": [{"done": {"text": "test", "success": true}}]}'])
# Create custom tools that includes screenshot action
custom_tools = Tools()
assert 'screenshot' in custom_tools.registry.registry.actions, 'Custom tools should have screenshot by default'
# Pass custom tools to agent with use_vision='auto'
agent = Agent(
task='test',
llm=mock_llm,
tools=custom_tools,
use_vision='auto',
)
# Verify screenshot is kept when use_vision='auto'
assert 'screenshot' in agent.tools.registry.registry.actions, (
'Screenshot should be included when use_vision="auto", even with custom tools'
)
def test_tools_exclude_action_method():
"""Test the Tools.exclude_action() method directly."""
tools = Tools()
# Verify screenshot is included initially
assert 'screenshot' in tools.registry.registry.actions, 'Screenshot should be included by default'
# Exclude screenshot
tools.exclude_action('screenshot')
# Verify screenshot is excluded
assert 'screenshot' not in tools.registry.registry.actions, 'Screenshot should be excluded after calling exclude_action()'
assert 'screenshot' in tools.registry.exclude_actions, 'Screenshot should be in exclude_actions list'
def test_exclude_action_prevents_re_registration():
"""Test that excluded actions cannot be re-registered."""
tools = Tools()
# Exclude screenshot
tools.exclude_action('screenshot')
assert 'screenshot' not in tools.registry.registry.actions
# Try to re-register screenshot (simulating what happens in __init__)
# The decorator should skip registration since it's in exclude_actions
@tools.registry.action('Test screenshot action')
async def screenshot():
return 'test'
# Verify it was not re-registered
assert 'screenshot' not in tools.registry.registry.actions, 'Excluded action should not be re-registered'
+465
View File
@@ -0,0 +1,465 @@
"""Tests for search_page and find_elements actions."""
import asyncio
import pytest
from pytest_httpserver import HTTPServer
from browser_use.agent.views import ActionResult
from browser_use.browser import BrowserProfile, BrowserSession
from browser_use.tools.service import Tools
# --- Fixtures ---
@pytest.fixture(scope='session')
def http_server():
"""Test HTTP server serving pages for search/find tests."""
server = HTTPServer()
server.start()
server.expect_request('/products').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head><title>Products</title></head>
<body>
<h1>Product Catalog</h1>
<div id="main">
<table class="products">
<thead>
<tr><th>Name</th><th>Price</th><th>Rating</th></tr>
</thead>
<tbody>
<tr class="product-row"><td>Widget A</td><td>$29.99</td><td>4.5 stars</td></tr>
<tr class="product-row"><td>Widget B</td><td>$49.99</td><td>4.2 stars</td></tr>
<tr class="product-row"><td>Gadget C</td><td>$19.50</td><td>3.8 stars</td></tr>
<tr class="product-row"><td>Gadget D</td><td>$99.00</td><td>4.9 stars</td></tr>
</tbody>
</table>
<div class="pagination">
<a href="/products?page=1" class="page-link active">1</a>
<a href="/products?page=2" class="page-link">2</a>
<a href="/products?page=3" class="page-link">3</a>
</div>
</div>
<footer id="footer">
<p>Best price guarantee on all items.</p>
<p>Contact us at support@example.com</p>
</footer>
</body>
</html>
""",
content_type='text/html',
)
server.expect_request('/articles').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head><title>Articles</title></head>
<body>
<article id="post-1">
<h2>Introduction to Python</h2>
<p>Python is a versatile programming language used in web development, data science, and automation.</p>
<a href="/articles/python" class="read-more">Read more</a>
</article>
<article id="post-2">
<h2>JavaScript for Beginners</h2>
<p>JavaScript powers the interactive web. Learn about DOM manipulation and event handling.</p>
<a href="/articles/javascript" class="read-more">Read more</a>
</article>
<article id="post-3">
<h2>Advanced CSS Techniques</h2>
<p>Master CSS Grid, Flexbox, and custom properties for modern web layouts.</p>
<a href="/articles/css" class="read-more">Read more</a>
</article>
</body>
</html>
""",
content_type='text/html',
)
server.expect_request('/empty').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head><title>Empty</title></head>
<body>
<div id="content"></div>
</body>
</html>
""",
content_type='text/html',
)
server.expect_request('/case-test').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head><title>Case Test</title></head>
<body>
<p>The Quick Brown Fox jumps over the lazy dog.</p>
<p>QUICK BROWN FOX is an uppercase variant.</p>
<p>quick brown fox is a lowercase variant.</p>
</body>
</html>
""",
content_type='text/html',
)
# /images-page route is registered dynamically in base_url fixture once port is known
yield server
server.stop()
@pytest.fixture(scope='session')
def base_url(http_server):
url = f'http://{http_server.host}:{http_server.port}'
# Register images page here so we can embed the absolute img src URL
http_server.expect_request('/images-page').respond_with_data(
f"""
<!DOCTYPE html>
<html>
<head><title>Images Page</title></head>
<body>
<img src="{url}/images/product.jpg" alt="Product">
</body>
</html>
""",
content_type='text/html',
)
return url
@pytest.fixture(scope='module')
async def browser_session():
session = BrowserSession(
browser_profile=BrowserProfile(
headless=True,
user_data_dir=None,
keep_alive=True,
)
)
await session.start()
yield session
await session.kill()
@pytest.fixture(scope='function')
def tools():
return Tools()
# --- Helper ---
async def _navigate_and_wait(tools, browser_session, url):
"""Navigate to URL and wait for page load."""
await tools.navigate(url=url, new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
# --- search_page tests ---
class TestSearchPage:
"""Tests for the search_page action."""
async def test_literal_text_search(self, tools, browser_session, base_url):
"""Literal text search finds matches with context."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/products')
result = await tools.search_page(pattern='Widget A', browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.error is None
assert result.extracted_content is not None
assert 'Widget A' in result.extracted_content
assert '1 match' in result.extracted_content
async def test_regex_search_prices(self, tools, browser_session, base_url):
"""Regex search finds all price patterns on the page."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/products')
result = await tools.search_page(pattern=r'\$\d+\.\d{2}', regex=True, browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.error is None
assert result.extracted_content is not None
# Should find $29.99, $49.99, $19.50, $99.00
assert '4 matches' in result.extracted_content
assert '$29.99' in result.extracted_content
assert '$49.99' in result.extracted_content
async def test_css_scope_limits_search(self, tools, browser_session, base_url):
"""css_scope limits search to elements within the selector."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/products')
# Search only in footer
result = await tools.search_page(pattern='price', css_scope='#footer', browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.error is None
assert result.extracted_content is not None
# "Best price guarantee" is in footer
assert '1 match' in result.extracted_content
assert 'guarantee' in result.extracted_content
async def test_case_insensitive_default(self, tools, browser_session, base_url):
"""Search is case-insensitive by default."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/case-test')
result = await tools.search_page(pattern='quick brown fox', browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.error is None
assert result.extracted_content is not None
# Should match all three variants (Quick, QUICK, quick)
assert '3 matches' in result.extracted_content
async def test_case_sensitive(self, tools, browser_session, base_url):
"""case_sensitive=True restricts to exact case."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/case-test')
result = await tools.search_page(pattern='QUICK BROWN FOX', case_sensitive=True, browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.error is None
assert result.extracted_content is not None
assert '1 match' in result.extracted_content
async def test_max_results(self, tools, browser_session, base_url):
"""max_results limits the number of returned matches."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/products')
result = await tools.search_page(pattern=r'\$\d+\.\d{2}', regex=True, max_results=2, browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.error is None
assert result.extracted_content is not None
# Total should still show 4, but only 2 are displayed
assert '4 matches' in result.extracted_content
assert 'Increase max_results' in result.extracted_content
async def test_no_matches(self, tools, browser_session, base_url):
"""No matches returns a clean message, not an error."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/products')
result = await tools.search_page(pattern='xyznonexistent', browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.error is None
assert result.extracted_content is not None
assert 'No matches found' in result.extracted_content
async def test_element_path_in_results(self, tools, browser_session, base_url):
"""Matches include the element path for context."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/products')
result = await tools.search_page(pattern='guarantee', browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.error is None
assert result.extracted_content is not None
# Should show element path containing footer
assert '(in' in result.extracted_content
async def test_invalid_css_scope(self, tools, browser_session, base_url):
"""Invalid css_scope returns a clear error."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/products')
result = await tools.search_page(pattern='test', css_scope='#nonexistent-scope', browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.error is not None
assert 'scope' in result.error.lower() or 'not found' in result.error.lower()
async def test_memory_set(self, tools, browser_session, base_url):
"""long_term_memory is set with match count summary."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/products')
result = await tools.search_page(pattern='Widget', browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.long_term_memory is not None
assert 'Widget' in result.long_term_memory
assert 'match' in result.long_term_memory
# --- find_elements tests ---
class TestFindElements:
"""Tests for the find_elements action."""
async def test_basic_selector(self, tools, browser_session, base_url):
"""Basic CSS selector returns correct elements."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/products')
result = await tools.find_elements(selector='tr.product-row', browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.error is None
assert result.extracted_content is not None
assert '4 elements' in result.extracted_content
assert 'Widget A' in result.extracted_content
assert 'Gadget D' in result.extracted_content
async def test_attribute_extraction(self, tools, browser_session, base_url):
"""attributes parameter extracts specific attributes from elements."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/products')
result = await tools.find_elements(
selector='a.page-link',
attributes=['href', 'class'],
browser_session=browser_session,
)
assert isinstance(result, ActionResult)
assert result.error is None
assert result.extracted_content is not None
assert '3 elements' in result.extracted_content
assert 'href=' in result.extracted_content
assert '/products?page=' in result.extracted_content
async def test_max_results_limiting(self, tools, browser_session, base_url):
"""max_results limits displayed elements while showing total count."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/products')
result = await tools.find_elements(selector='tr.product-row', max_results=2, browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.error is None
assert result.extracted_content is not None
assert '4 elements' in result.extracted_content
assert 'Showing 2 of 4' in result.extracted_content
async def test_no_matching_elements(self, tools, browser_session, base_url):
"""No matches returns a clean message, not an error."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/products')
result = await tools.find_elements(selector='div.nonexistent', browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.error is None
assert result.extracted_content is not None
assert 'No elements found' in result.extracted_content
async def test_invalid_selector(self, tools, browser_session, base_url):
"""Invalid CSS selector returns a clear error, not a crash."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/products')
result = await tools.find_elements(selector='[[[invalid', browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.error is not None
assert 'selector' in result.error.lower() or 'invalid' in result.error.lower()
async def test_include_text_false(self, tools, browser_session, base_url):
"""include_text=False omits text content from results."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/articles')
result = await tools.find_elements(selector='article', include_text=False, browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.error is None
assert result.extracted_content is not None
assert '3 elements' in result.extracted_content
# Text content should not appear (no article body text)
# But the tag and children count should still be present
assert '<article>' in result.extracted_content
async def test_nested_selectors(self, tools, browser_session, base_url):
"""Nested CSS selectors (child combinator) work correctly."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/articles')
result = await tools.find_elements(
selector='article a.read-more',
attributes=['href'],
browser_session=browser_session,
)
assert isinstance(result, ActionResult)
assert result.error is None
assert result.extracted_content is not None
assert '3 elements' in result.extracted_content
assert '/articles/python' in result.extracted_content
assert '/articles/javascript' in result.extracted_content
assert '/articles/css' in result.extracted_content
async def test_children_count(self, tools, browser_session, base_url):
"""Elements show children count."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/products')
result = await tools.find_elements(selector='table.products thead tr', browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.error is None
assert result.extracted_content is not None
assert '1 element' in result.extracted_content
# The header row has 3 <th> children
assert '3 children' in result.extracted_content
async def test_memory_set(self, tools, browser_session, base_url):
"""long_term_memory is set with element count summary."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/products')
result = await tools.find_elements(selector='tr.product-row', browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.long_term_memory is not None
assert '4 element' in result.long_term_memory
async def test_empty_page(self, tools, browser_session, base_url):
"""Works on a nearly empty page without errors."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/empty')
result = await tools.find_elements(selector='p', browser_session=browser_session)
assert isinstance(result, ActionResult)
assert result.error is None
assert result.extracted_content is not None
assert 'No elements found' in result.extracted_content
async def test_img_src_attribute_resolved(self, tools, browser_session, base_url):
"""find_elements with attributes=['src'] returns absolute URLs for img elements."""
await _navigate_and_wait(tools, browser_session, f'{base_url}/images-page')
result = await tools.find_elements(
selector='img',
attributes=['src'],
browser_session=browser_session,
)
assert isinstance(result, ActionResult)
assert result.error is None
assert result.extracted_content is not None
assert '1 element' in result.extracted_content
assert 'src=' in result.extracted_content
# The resolved DOM property should give the absolute URL (including the httpserver base URL)
assert base_url in result.extracted_content
assert 'product.jpg' in result.extracted_content
# --- Registration tests ---
class TestRegistration:
"""Test that new actions are properly registered."""
async def test_search_page_registered(self, tools):
"""search_page is in the default action registry."""
assert 'search_page' in tools.registry.registry.actions
async def test_find_elements_registered(self, tools):
"""find_elements is in the default action registry."""
assert 'find_elements' in tools.registry.registry.actions
async def test_excluded_actions(self):
"""New actions can be excluded via exclude_actions."""
excluded_tools = Tools(exclude_actions=['search_page', 'find_elements'])
assert 'search_page' not in excluded_tools.registry.registry.actions
assert 'find_elements' not in excluded_tools.registry.registry.actions
# Other actions still present
assert 'navigate' in excluded_tools.registry.registry.actions
+631
View File
@@ -0,0 +1,631 @@
"""Tests for schema-enforced structured extraction."""
import asyncio
import json
import tempfile
from unittest.mock import AsyncMock
import pytest
from pydantic import ValidationError
from pytest_httpserver import HTTPServer
from browser_use.agent.views import ActionResult
from browser_use.browser import BrowserProfile, BrowserSession
from browser_use.filesystem.file_system import FileSystem
from browser_use.llm.base import BaseChatModel
from browser_use.llm.views import ChatInvokeCompletion
from browser_use.tools.extraction.schema_utils import schema_dict_to_pydantic_model
from browser_use.tools.extraction.views import ExtractionResult
from browser_use.tools.service import Tools
# ---------------------------------------------------------------------------
# Unit tests: schema_dict_to_pydantic_model
# ---------------------------------------------------------------------------
class TestSchemaDictToPydanticModel:
"""Unit tests for the JSON-Schema → Pydantic model converter."""
def test_flat_object(self):
schema = {
'type': 'object',
'properties': {
'name': {'type': 'string'},
'age': {'type': 'integer'},
},
'required': ['name', 'age'],
}
Model = schema_dict_to_pydantic_model(schema)
instance = Model(name='Alice', age=30)
assert instance.name == 'Alice' # type: ignore[attr-defined]
assert instance.age == 30 # type: ignore[attr-defined]
def test_nested_object(self):
schema = {
'type': 'object',
'properties': {
'person': {
'type': 'object',
'properties': {
'first': {'type': 'string'},
'last': {'type': 'string'},
},
'required': ['first'],
},
},
'required': ['person'],
}
Model = schema_dict_to_pydantic_model(schema)
instance = Model(person={'first': 'Bob', 'last': 'Smith'})
assert instance.person.first == 'Bob' # type: ignore[attr-defined]
def test_array_of_objects(self):
schema = {
'type': 'object',
'properties': {
'items': {
'type': 'array',
'items': {
'type': 'object',
'properties': {
'id': {'type': 'integer'},
'label': {'type': 'string'},
},
'required': ['id', 'label'],
},
},
},
'required': ['items'],
}
Model = schema_dict_to_pydantic_model(schema)
instance = Model(items=[{'id': 1, 'label': 'a'}, {'id': 2, 'label': 'b'}])
assert len(instance.items) == 2 # type: ignore[attr-defined]
assert instance.items[0].id == 1 # type: ignore[attr-defined]
def test_array_of_primitives(self):
schema = {
'type': 'object',
'properties': {
'tags': {'type': 'array', 'items': {'type': 'string'}},
},
'required': ['tags'],
}
Model = schema_dict_to_pydantic_model(schema)
instance = Model(tags=['a', 'b', 'c'])
assert instance.tags == ['a', 'b', 'c'] # type: ignore[attr-defined]
def test_enum_field(self):
schema = {
'type': 'object',
'properties': {
'status': {'type': 'string', 'enum': ['active', 'inactive']},
},
'required': ['status'],
}
Model = schema_dict_to_pydantic_model(schema)
instance = Model(status='active')
assert instance.status == 'active' # type: ignore[attr-defined]
def test_optional_enum_defaults_to_none(self):
"""Non-required enum fields default to None, not an out-of-set empty string."""
schema = {
'type': 'object',
'properties': {
'name': {'type': 'string'},
'priority': {'type': 'string', 'enum': ['low', 'medium', 'high']},
},
'required': ['name'],
}
Model = schema_dict_to_pydantic_model(schema)
instance = Model(name='task1')
assert instance.priority is None # type: ignore[attr-defined]
# Serialized output must not contain an out-of-set value
dumped = instance.model_dump(mode='json')
assert dumped['priority'] is None
# When provided, value still works
instance2 = Model(name='task2', priority='high')
assert instance2.priority == 'high' # type: ignore[attr-defined]
def test_optional_fields_get_type_appropriate_defaults(self):
schema = {
'type': 'object',
'properties': {
'name': {'type': 'string'},
'nickname': {'type': 'string'},
'score': {'type': 'number'},
'rank': {'type': 'integer'},
'active': {'type': 'boolean'},
'tags': {'type': 'array', 'items': {'type': 'string'}},
},
'required': ['name'],
}
Model = schema_dict_to_pydantic_model(schema)
instance = Model(name='Alice')
assert instance.name == 'Alice' # type: ignore[attr-defined]
assert instance.nickname == '' # type: ignore[attr-defined]
assert instance.score == 0.0 # type: ignore[attr-defined]
assert instance.rank == 0 # type: ignore[attr-defined]
assert instance.active is False # type: ignore[attr-defined]
assert instance.tags == [] # type: ignore[attr-defined]
def test_optional_non_nullable_rejects_null(self):
"""Non-required fields that aren't nullable must reject explicit null."""
schema = {
'type': 'object',
'properties': {
'name': {'type': 'string'},
'nickname': {'type': 'string'},
},
'required': ['name'],
}
Model = schema_dict_to_pydantic_model(schema)
with pytest.raises(ValidationError):
Model(name='Alice', nickname=None)
def test_optional_with_explicit_default(self):
schema = {
'type': 'object',
'properties': {
'name': {'type': 'string'},
'color': {'type': 'string', 'default': 'blue'},
},
'required': ['name'],
}
Model = schema_dict_to_pydantic_model(schema)
instance = Model(name='Alice')
assert instance.color == 'blue' # type: ignore[attr-defined]
def test_optional_nested_object_defaults_to_none(self):
"""Non-required nested objects fall back to None since constructing a default is not feasible."""
schema = {
'type': 'object',
'properties': {
'name': {'type': 'string'},
'address': {
'type': 'object',
'properties': {'city': {'type': 'string'}},
'required': ['city'],
},
},
'required': ['name'],
}
Model = schema_dict_to_pydantic_model(schema)
instance = Model(name='Alice')
assert instance.address is None # type: ignore[attr-defined]
def test_model_name_from_title(self):
schema = {
'title': 'ProductInfo',
'type': 'object',
'properties': {'sku': {'type': 'string'}},
'required': ['sku'],
}
Model = schema_dict_to_pydantic_model(schema)
assert Model.__name__ == 'ProductInfo'
def test_model_validate_json_roundtrip(self):
schema = {
'type': 'object',
'properties': {
'x': {'type': 'number'},
'y': {'type': 'boolean'},
},
'required': ['x', 'y'],
}
Model = schema_dict_to_pydantic_model(schema)
instance = Model(x=3.14, y=True)
raw = instance.model_dump_json()
restored = Model.model_validate_json(raw)
assert restored.x == instance.x # type: ignore[attr-defined]
assert restored.y == instance.y # type: ignore[attr-defined]
def test_rejects_ref(self):
schema = {
'type': 'object',
'properties': {'item': {'$ref': '#/$defs/Item'}},
'$defs': {'Item': {'type': 'object', 'properties': {'name': {'type': 'string'}}}},
}
with pytest.raises(ValueError, match='Unsupported JSON Schema keyword'):
schema_dict_to_pydantic_model(schema)
def test_rejects_allOf(self):
schema = {
'type': 'object',
'properties': {'x': {'allOf': [{'type': 'string'}]}},
}
with pytest.raises(ValueError, match='Unsupported JSON Schema keyword'):
schema_dict_to_pydantic_model(schema)
def test_rejects_non_object_toplevel(self):
with pytest.raises(ValueError, match='type "object"'):
schema_dict_to_pydantic_model({'type': 'array', 'items': {'type': 'string'}})
def test_rejects_empty_properties(self):
with pytest.raises(ValueError, match='at least one property'):
schema_dict_to_pydantic_model({'type': 'object', 'properties': {}})
def test_extra_fields_forbidden(self):
schema = {
'type': 'object',
'properties': {'name': {'type': 'string'}},
'required': ['name'],
}
Model = schema_dict_to_pydantic_model(schema)
with pytest.raises(ValidationError):
Model(name='ok', bogus='nope')
def test_nullable_field(self):
schema = {
'type': 'object',
'properties': {
'value': {'type': 'string', 'nullable': True},
},
'required': ['value'],
}
Model = schema_dict_to_pydantic_model(schema)
instance = Model(value=None)
assert instance.value is None # type: ignore[attr-defined]
def test_field_descriptions_preserved(self):
schema = {
'type': 'object',
'properties': {
'price': {'type': 'number', 'description': 'The price in USD'},
},
'required': ['price'],
}
Model = schema_dict_to_pydantic_model(schema)
field_info = Model.model_fields['price']
assert field_info.description == 'The price in USD'
# ---------------------------------------------------------------------------
# Unit tests: ExtractionResult
# ---------------------------------------------------------------------------
class TestExtractionResult:
def test_construction(self):
er = ExtractionResult(
data={'name': 'Alice'},
schema_used={'type': 'object', 'properties': {'name': {'type': 'string'}}},
)
assert er.data == {'name': 'Alice'}
assert er.is_partial is False
assert er.source_url is None
def test_serialization_roundtrip(self):
er = ExtractionResult(
data={'items': [1, 2]},
schema_used={'type': 'object', 'properties': {'items': {'type': 'array'}}},
is_partial=True,
source_url='http://example.com',
content_stats={'original_html_chars': 5000},
)
raw = er.model_dump_json()
restored = ExtractionResult.model_validate_json(raw)
assert restored == er
# ---------------------------------------------------------------------------
# Integration tests: extract action via Tools
# ---------------------------------------------------------------------------
def _make_extraction_llm(structured_response: dict | None = None, freetext_response: str = 'free text result') -> BaseChatModel:
"""Create a mock LLM that handles both structured and freetext extraction calls."""
llm = AsyncMock(spec=BaseChatModel)
llm.model = 'mock-extraction-llm'
llm._verified_api_keys = True
llm.provider = 'mock'
llm.name = 'mock-extraction-llm'
llm.model_name = 'mock-extraction-llm'
async def mock_ainvoke(messages, output_format=None, **kwargs):
if output_format is not None and structured_response is not None:
# Structured path: parse the dict through the model
instance = output_format.model_validate(structured_response)
return ChatInvokeCompletion(completion=instance, usage=None)
# Freetext path
return ChatInvokeCompletion(completion=freetext_response, usage=None)
llm.ainvoke.side_effect = mock_ainvoke
return llm
@pytest.fixture(scope='module')
async def browser_session():
session = BrowserSession(browser_profile=BrowserProfile(headless=True, user_data_dir=None, keep_alive=True))
await session.start()
yield session
await session.kill()
await session.event_bus.stop(clear=True, timeout=5)
@pytest.fixture(scope='session')
def http_server():
server = HTTPServer()
server.start()
server.expect_request('/products').respond_with_data(
"""<html><body>
<h1>Products</h1>
<ul>
<li>Widget A - $9.99</li>
<li>Widget B - $19.99</li>
</ul>
</body></html>""",
content_type='text/html',
)
yield server
server.stop()
@pytest.fixture(scope='session')
def base_url(http_server):
return f'http://{http_server.host}:{http_server.port}'
class TestExtractStructured:
"""Integration tests for the extract action's structured extraction path."""
async def test_structured_extraction_returns_json(self, browser_session, base_url):
"""When output_schema is provided, extract returns structured JSON in <structured_result> tags."""
tools = Tools()
await tools.navigate(url=f'{base_url}/products', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
output_schema = {
'type': 'object',
'properties': {
'products': {
'type': 'array',
'items': {
'type': 'object',
'properties': {
'name': {'type': 'string'},
'price': {'type': 'number'},
},
'required': ['name', 'price'],
},
},
},
'required': ['products'],
}
mock_data = {'products': [{'name': 'Widget A', 'price': 9.99}, {'name': 'Widget B', 'price': 19.99}]}
extraction_llm = _make_extraction_llm(structured_response=mock_data)
with tempfile.TemporaryDirectory() as tmp:
fs = FileSystem(tmp)
result = await tools.extract(
query='List all products with prices',
output_schema=output_schema,
browser_session=browser_session,
page_extraction_llm=extraction_llm,
file_system=fs,
)
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert '<structured_result>' in result.extracted_content
assert '</structured_result>' in result.extracted_content
# Parse the JSON out of the tags
start = result.extracted_content.index('<structured_result>') + len('<structured_result>')
end = result.extracted_content.index('</structured_result>')
parsed = json.loads(result.extracted_content[start:end].strip())
assert parsed == mock_data
# Metadata
assert result.metadata is not None
assert result.metadata['structured_extraction'] is True
meta = result.metadata['extraction_result']
assert meta['data'] == mock_data
assert meta['schema_used'] == output_schema
async def test_freetext_extraction_unchanged(self, browser_session, base_url):
"""When output_schema is None, extract returns free-text in <result> tags (backward compat)."""
tools = Tools()
await tools.navigate(url=f'{base_url}/products', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
extraction_llm = _make_extraction_llm(freetext_response='Widget A costs $9.99, Widget B costs $19.99')
with tempfile.TemporaryDirectory() as tmp:
fs = FileSystem(tmp)
result = await tools.extract(
query='What products are listed?',
browser_session=browser_session,
page_extraction_llm=extraction_llm,
file_system=fs,
)
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert '<result>' in result.extracted_content
assert '</result>' in result.extracted_content
assert '<structured_result>' not in result.extracted_content
assert result.metadata is None
async def test_invalid_schema_falls_back_to_freetext(self, browser_session, base_url):
"""When output_schema contains unsupported keywords, fall back to free-text gracefully."""
tools = Tools()
await tools.navigate(url=f'{base_url}/products', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
bad_schema = {
'type': 'object',
'properties': {'item': {'$ref': '#/$defs/Item'}},
'$defs': {'Item': {'type': 'object', 'properties': {'name': {'type': 'string'}}}},
}
extraction_llm = _make_extraction_llm(freetext_response='fallback text')
with tempfile.TemporaryDirectory() as tmp:
fs = FileSystem(tmp)
result = await tools.extract(
query='Get products',
output_schema=bad_schema,
browser_session=browser_session,
page_extraction_llm=extraction_llm,
file_system=fs,
)
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
# Should have used the free-text path
assert '<result>' in result.extracted_content
assert '<structured_result>' not in result.extracted_content
assert result.metadata is None
# ---------------------------------------------------------------------------
# Integration tests: extraction_schema injection via special parameter
# ---------------------------------------------------------------------------
PRODUCT_SCHEMA = {
'type': 'object',
'properties': {
'products': {
'type': 'array',
'items': {
'type': 'object',
'properties': {
'name': {'type': 'string'},
'price': {'type': 'number'},
},
'required': ['name', 'price'],
},
},
},
'required': ['products'],
}
MOCK_PRODUCTS = {'products': [{'name': 'Widget A', 'price': 9.99}, {'name': 'Widget B', 'price': 19.99}]}
class TestExtractionSchemaInjection:
"""Tests that extraction_schema injected as a special parameter triggers structured extraction."""
async def test_injected_extraction_schema_triggers_structured_path(self, browser_session, base_url):
"""extraction_schema passed via act() triggers structured extraction even without output_schema in params."""
tools = Tools()
await tools.navigate(url=f'{base_url}/products', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
extraction_llm = _make_extraction_llm(structured_response=MOCK_PRODUCTS)
with tempfile.TemporaryDirectory() as tmp:
fs = FileSystem(tmp)
result = await tools.extract(
query='List all products with prices',
browser_session=browser_session,
page_extraction_llm=extraction_llm,
file_system=fs,
extraction_schema=PRODUCT_SCHEMA,
)
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert '<structured_result>' in result.extracted_content
# Parse and verify JSON
start = result.extracted_content.index('<structured_result>') + len('<structured_result>')
end = result.extracted_content.index('</structured_result>')
parsed = json.loads(result.extracted_content[start:end].strip())
assert parsed == MOCK_PRODUCTS
assert result.metadata is not None
assert result.metadata['structured_extraction'] is True
async def test_output_schema_takes_precedence_over_extraction_schema(self, browser_session, base_url):
"""When the LLM provides output_schema in params, it should take precedence over injected extraction_schema."""
tools = Tools()
await tools.navigate(url=f'{base_url}/products', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
# Different schema than the injected one — just a name list
param_schema = {
'type': 'object',
'properties': {
'names': {'type': 'array', 'items': {'type': 'string'}},
},
'required': ['names'],
}
param_response = {'names': ['Widget A', 'Widget B']}
extraction_llm = _make_extraction_llm(structured_response=param_response)
with tempfile.TemporaryDirectory() as tmp:
fs = FileSystem(tmp)
result = await tools.extract(
query='List product names',
output_schema=param_schema,
browser_session=browser_session,
page_extraction_llm=extraction_llm,
file_system=fs,
extraction_schema=PRODUCT_SCHEMA, # should be ignored
)
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert '<structured_result>' in result.extracted_content
start = result.extracted_content.index('<structured_result>') + len('<structured_result>')
end = result.extracted_content.index('</structured_result>')
parsed = json.loads(result.extracted_content[start:end].strip())
# Should match param_schema response, NOT PRODUCT_SCHEMA
assert parsed == param_response
assert result.metadata is not None
assert result.metadata['extraction_result']['schema_used'] == param_schema
async def test_no_schema_uses_freetext_path(self, browser_session, base_url):
"""When neither output_schema nor extraction_schema is provided, free-text path is used (backward compat)."""
tools = Tools()
await tools.navigate(url=f'{base_url}/products', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
extraction_llm = _make_extraction_llm(freetext_response='Widget A costs $9.99')
with tempfile.TemporaryDirectory() as tmp:
fs = FileSystem(tmp)
result = await tools.extract(
query='What products are listed?',
browser_session=browser_session,
page_extraction_llm=extraction_llm,
file_system=fs,
# No extraction_schema, no output_schema
)
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert '<result>' in result.extracted_content
assert '<structured_result>' not in result.extracted_content
assert result.metadata is None
async def test_extraction_schema_threads_through_act(self, browser_session, base_url):
"""extraction_schema passed to act() reaches extract() via the registry's special parameter injection."""
tools = Tools()
await tools.navigate(url=f'{base_url}/products', new_tab=False, browser_session=browser_session)
await asyncio.sleep(0.5)
extraction_llm = _make_extraction_llm(structured_response=MOCK_PRODUCTS)
with tempfile.TemporaryDirectory() as tmp:
fs = FileSystem(tmp)
# Build an ActionModel for the extract action
action_model = tools.registry.create_action_model()
action = action_model.model_validate({'extract': {'query': 'List products'}})
result = await tools.act(
action=action,
browser_session=browser_session,
page_extraction_llm=extraction_llm,
file_system=fs,
extraction_schema=PRODUCT_SCHEMA,
)
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert '<structured_result>' in result.extracted_content
+668
View File
@@ -0,0 +1,668 @@
import asyncio
import json
import os
import tempfile
import time
import anyio
import pytest
from pydantic import BaseModel, Field
from pytest_httpserver import HTTPServer
from browser_use.agent.views import ActionResult
from browser_use.browser import BrowserSession
from browser_use.browser.profile import BrowserProfile
from browser_use.filesystem.file_system import FileSystem
from browser_use.tools.service import Tools
@pytest.fixture(scope='session')
def http_server():
"""Create and provide a test HTTP server that serves static content."""
server = HTTPServer()
server.start()
# Add routes for common test pages
server.expect_request('/').respond_with_data(
'<html><head><title>Test Home Page</title></head><body><h1>Test Home Page</h1><p>Welcome to the test site</p></body></html>',
content_type='text/html',
)
server.expect_request('/page1').respond_with_data(
'<html><head><title>Test Page 1</title></head><body><h1>Test Page 1</h1><p>This is test page 1</p></body></html>',
content_type='text/html',
)
server.expect_request('/page2').respond_with_data(
'<html><head><title>Test Page 2</title></head><body><h1>Test Page 2</h1><p>This is test page 2</p></body></html>',
content_type='text/html',
)
server.expect_request('/search').respond_with_data(
"""
<html>
<head><title>Search Results</title></head>
<body>
<h1>Search Results</h1>
<div class="results">
<div class="result">Result 1</div>
<div class="result">Result 2</div>
<div class="result">Result 3</div>
</div>
</body>
</html>
""",
content_type='text/html',
)
yield server
server.stop()
@pytest.fixture(scope='session')
def base_url(http_server):
"""Return the base URL for the test HTTP server."""
return f'http://{http_server.host}:{http_server.port}'
@pytest.fixture(scope='module')
async def browser_session():
"""Create and provide a Browser instance with security disabled."""
browser_session = BrowserSession(
browser_profile=BrowserProfile(
headless=True,
user_data_dir=None,
keep_alive=True,
)
)
await browser_session.start()
yield browser_session
await browser_session.kill()
@pytest.fixture(scope='function')
def tools():
"""Create and provide a Tools instance."""
return Tools()
class TestToolsIntegration:
"""Integration tests for Tools using actual browser instances."""
async def test_registry_actions(self, tools, browser_session):
"""Test that the registry contains the expected default actions."""
# Check that common actions are registered
common_actions = [
'navigate',
'search',
'click',
'input',
'scroll',
'go_back',
'switch',
'close',
'wait',
]
for action in common_actions:
assert action in tools.registry.registry.actions
assert tools.registry.registry.actions[action].function is not None
assert tools.registry.registry.actions[action].description is not None
async def test_custom_action_registration(self, tools, browser_session, base_url):
"""Test registering a custom action and executing it."""
# Define a custom action
class CustomParams(BaseModel):
text: str
@tools.action('Test custom action', param_model=CustomParams)
async def custom_action(params: CustomParams, browser_session):
current_url = await browser_session.get_current_page_url()
return ActionResult(extracted_content=f'Custom action executed with: {params.text} on {current_url}')
# Navigate to a page first
await tools.navigate(url=f'{base_url}/page1', new_tab=False, browser_session=browser_session)
# Execute the custom action directly
result = await tools.custom_action(text='test_value', browser_session=browser_session)
# Verify the result
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'Custom action executed with: test_value on' in result.extracted_content
assert f'{base_url}/page1' in result.extracted_content
async def test_wait_action(self, tools, browser_session):
"""Test that the wait action correctly waits for the specified duration."""
# verify that it's in the default action set
wait_action = None
for action_name, action in tools.registry.registry.actions.items():
if 'wait' in action_name.lower() and 'seconds' in str(action.param_model.model_fields):
wait_action = action
break
assert wait_action is not None, 'Could not find wait action in tools'
# Check that it has seconds parameter with default
assert 'seconds' in wait_action.param_model.model_fields
schema = wait_action.param_model.model_json_schema()
assert schema['properties']['seconds']['default'] == 3
# Record start time
start_time = time.time()
# Execute wait action
result = await tools.wait(seconds=3, browser_session=browser_session)
# Record end time
end_time = time.time()
# Verify the result
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'Waited for' in result.extracted_content or 'Waiting for' in result.extracted_content
# Verify that approximately 1 second has passed (allowing some margin)
assert end_time - start_time <= 2.5 # We wait 3-1 seconds for LLM call
# longer wait
# Record start time
start_time = time.time()
# Execute wait action
result = await tools.wait(seconds=5, browser_session=browser_session)
# Record end time
end_time = time.time()
# Verify the result
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'Waited for' in result.extracted_content or 'Waiting for' in result.extracted_content
assert 3.5 <= end_time - start_time <= 4.5 # We wait 5-1 seconds for LLM call
async def test_go_back_action(self, tools, browser_session, base_url):
"""Test that go_back action navigates to the previous page."""
# Navigate to first page
await tools.navigate(url=f'{base_url}/page1', new_tab=False, browser_session=browser_session)
# Store the first page URL
first_url = await browser_session.get_current_page_url()
print(f'First page URL: {first_url}')
# Navigate to second page
await tools.navigate(url=f'{base_url}/page2', new_tab=False, browser_session=browser_session)
# Verify we're on the second page
second_url = await browser_session.get_current_page_url()
print(f'Second page URL: {second_url}')
assert f'{base_url}/page2' in second_url
# Execute go back action
result = await tools.go_back(browser_session=browser_session)
# Verify the result
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'Navigated back' in result.extracted_content
# Add another delay to allow the navigation to complete
await asyncio.sleep(1)
# Verify we're back on a different page than before
final_url = await browser_session.get_current_page_url()
print(f'Final page URL after going back: {final_url}')
# Try to verify we're back on the first page, but don't fail the test if not
assert f'{base_url}/page1' in final_url, f'Expected to return to page1 but got {final_url}'
async def test_navigation_chain(self, tools, browser_session, base_url):
"""Test navigating through multiple pages and back through history."""
# Set up a chain of navigation: Home -> Page1 -> Page2
urls = [f'{base_url}/', f'{base_url}/page1', f'{base_url}/page2']
# Navigate to each page in sequence
for url in urls:
await tools.navigate(url=url, new_tab=False, browser_session=browser_session)
# Verify current page
current_url = await browser_session.get_current_page_url()
assert url in current_url
# Go back twice and verify each step
for expected_url in reversed(urls[:-1]):
await tools.go_back(browser_session=browser_session)
await asyncio.sleep(1) # Wait for navigation to complete
current_url = await browser_session.get_current_page_url()
assert expected_url in current_url
async def test_excluded_actions(self, browser_session):
"""Test that excluded actions are not registered."""
# Create tools with excluded actions
excluded_tools = Tools(exclude_actions=['search', 'scroll'])
# Verify excluded actions are not in the registry
assert 'search' not in excluded_tools.registry.registry.actions
assert 'scroll' not in excluded_tools.registry.registry.actions
# But other actions are still there
assert 'navigate' in excluded_tools.registry.registry.actions
assert 'click' in excluded_tools.registry.registry.actions
async def test_search_action(self, tools, browser_session, base_url):
"""Test the search action."""
await browser_session.get_current_page_url()
# Execute search action - it will actually navigate to our search results page
result = await tools.search(query='Python web automation', browser_session=browser_session)
# Verify the result
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert 'Searched' in result.extracted_content and 'Python web automation' in result.extracted_content
# For our test purposes, we just verify we're on some URL
current_url = await browser_session.get_current_page_url()
assert current_url is not None and 'Python' in current_url
async def test_done_action(self, tools, browser_session, base_url):
"""Test that DoneAction completes a task and reports success or failure."""
# Create a temporary directory for the file system
with tempfile.TemporaryDirectory() as temp_dir:
file_system = FileSystem(temp_dir)
# First navigate to a page
await tools.navigate(url=f'{base_url}/page1', new_tab=False, browser_session=browser_session)
success_done_message = 'Successfully completed task'
# Execute done action with file_system
result = await tools.done(
text=success_done_message, success=True, browser_session=browser_session, file_system=file_system
)
# Verify the result
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert success_done_message in result.extracted_content
assert result.success is True
assert result.is_done is True
assert result.error is None
failed_done_message = 'Failed to complete task'
# Execute failed done action with file_system
result = await tools.done(
text=failed_done_message, success=False, browser_session=browser_session, file_system=file_system
)
# Verify the result
assert isinstance(result, ActionResult)
assert result.extracted_content is not None
assert failed_done_message in result.extracted_content
assert result.success is False
assert result.is_done is True
assert result.error is None
async def test_get_dropdown_options(self, tools, browser_session, base_url, http_server):
"""Test that get_dropdown_options correctly retrieves options from a dropdown."""
# Add route for dropdown test page
http_server.expect_request('/dropdown1').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head>
<title>Dropdown Test</title>
</head>
<body>
<h1>Dropdown Test</h1>
<select id="test-dropdown" name="test-dropdown">
<option value="">Please select</option>
<option value="option1">First Option</option>
<option value="option2">Second Option</option>
<option value="option3">Third Option</option>
</select>
</body>
</html>
""",
content_type='text/html',
)
# Navigate to the dropdown test page
await tools.navigate(url=f'{base_url}/dropdown1', new_tab=False, browser_session=browser_session)
# Wait for the page to load using CDP
cdp_session = await browser_session.get_or_create_cdp_session()
assert cdp_session is not None, 'CDP session not initialized'
# Wait for page load by checking document ready state
await asyncio.sleep(0.5) # Brief wait for navigation to start
ready_state = await cdp_session.cdp_client.send.Runtime.evaluate(
params={'expression': 'document.readyState'}, session_id=cdp_session.session_id
)
# If not complete, wait a bit more
if ready_state.get('result', {}).get('value') != 'complete':
await asyncio.sleep(1.0)
# Initialize the DOM state to populate the selector map
await browser_session.get_browser_state_summary()
# Get the selector map
selector_map = await browser_session.get_selector_map()
# Find the dropdown element in the selector map
dropdown_index = None
for idx, element in selector_map.items():
if element.tag_name.lower() == 'select':
dropdown_index = idx
break
assert dropdown_index is not None, (
f'Could not find select element in selector map. Available elements: {[f"{idx}: {element.tag_name}" for idx, element in selector_map.items()]}'
)
# Execute the action with the dropdown index
result = await tools.dropdown_options(index=dropdown_index, browser_session=browser_session)
expected_options = [
{'index': 0, 'text': 'Please select', 'value': ''},
{'index': 1, 'text': 'First Option', 'value': 'option1'},
{'index': 2, 'text': 'Second Option', 'value': 'option2'},
{'index': 3, 'text': 'Third Option', 'value': 'option3'},
]
# Verify the result structure
assert isinstance(result, ActionResult)
# Core logic validation: Verify all options are returned
assert result.extracted_content is not None
for option in expected_options[1:]: # Skip the placeholder option
assert option['text'] in result.extracted_content, f"Option '{option['text']}' not found in result content"
# Verify the instruction for using the text in select_dropdown is included
assert 'Use the exact text or value string' in result.extracted_content and 'select_dropdown' in result.extracted_content
# Verify the actual dropdown options in the DOM using CDP
dropdown_options_result = await cdp_session.cdp_client.send.Runtime.evaluate(
params={
'expression': """
JSON.stringify((() => {
const select = document.getElementById('test-dropdown');
return Array.from(select.options).map(opt => ({
text: opt.text,
value: opt.value
}));
})())
""",
'returnByValue': True,
},
session_id=cdp_session.session_id,
)
dropdown_options_json = dropdown_options_result.get('result', {}).get('value', '[]')
import json
dropdown_options = json.loads(dropdown_options_json) if isinstance(dropdown_options_json, str) else dropdown_options_json
# Verify the dropdown has the expected options
assert len(dropdown_options) == len(expected_options), (
f'Expected {len(expected_options)} options, got {len(dropdown_options)}'
)
for i, expected in enumerate(expected_options):
actual = dropdown_options[i]
assert actual['text'] == expected['text'], (
f"Option at index {i} has wrong text: expected '{expected['text']}', got '{actual['text']}'"
)
assert actual['value'] == expected['value'], (
f"Option at index {i} has wrong value: expected '{expected['value']}', got '{actual['value']}'"
)
async def test_select_dropdown_option(self, tools, browser_session, base_url, http_server):
"""Test that select_dropdown_option correctly selects an option from a dropdown."""
# Add route for dropdown test page
http_server.expect_request('/dropdown2').respond_with_data(
"""
<!DOCTYPE html>
<html>
<head>
<title>Dropdown Test</title>
</head>
<body>
<h1>Dropdown Test</h1>
<select id="test-dropdown" name="test-dropdown">
<option value="">Please select</option>
<option value="option1">First Option</option>
<option value="option2">Second Option</option>
<option value="option3">Third Option</option>
</select>
</body>
</html>
""",
content_type='text/html',
)
# Navigate to the dropdown test page
await tools.navigate(url=f'{base_url}/dropdown2', new_tab=False, browser_session=browser_session)
# Wait for the page to load using CDP
cdp_session = await browser_session.get_or_create_cdp_session()
assert cdp_session is not None, 'CDP session not initialized'
# Wait for page load by checking document ready state
await asyncio.sleep(0.5) # Brief wait for navigation to start
ready_state = await cdp_session.cdp_client.send.Runtime.evaluate(
params={'expression': 'document.readyState'}, session_id=cdp_session.session_id
)
# If not complete, wait a bit more
if ready_state.get('result', {}).get('value') != 'complete':
await asyncio.sleep(1.0)
# populate the selector map with highlight indices
await browser_session.get_browser_state_summary()
# Now get the selector map which should contain our dropdown
selector_map = await browser_session.get_selector_map()
# Find the dropdown element in the selector map
dropdown_index = None
for idx, element in selector_map.items():
if element.tag_name.lower() == 'select':
dropdown_index = idx
break
assert dropdown_index is not None, (
f'Could not find select element in selector map. Available elements: {[f"{idx}: {element.tag_name}" for idx, element in selector_map.items()]}'
)
# Execute the action with the dropdown index
result = await tools.select_dropdown(index=dropdown_index, text='Second Option', browser_session=browser_session)
# Verify the result structure
assert isinstance(result, ActionResult)
# Core logic validation: Verify selection was successful
assert result.extracted_content is not None
assert 'selected option' in result.extracted_content.lower()
assert 'Second Option' in result.extracted_content
# Verify the actual dropdown selection was made by checking the DOM using CDP
selected_value_result = await cdp_session.cdp_client.send.Runtime.evaluate(
params={'expression': "document.getElementById('test-dropdown').value"}, session_id=cdp_session.session_id
)
selected_value = selected_value_result.get('result', {}).get('value')
assert selected_value == 'option2' # Second Option has value "option2"
class TestStructuredOutputDoneWithFiles:
"""Tests for file handling in structured output done action."""
async def test_structured_output_done_without_files(self, browser_session, base_url):
"""Structured output done action works without files (backward compat)."""
class MyOutput(BaseModel):
answer: str = Field(description='The answer')
tools = Tools(output_model=MyOutput)
with tempfile.TemporaryDirectory() as temp_dir:
file_system = FileSystem(temp_dir)
result = await tools.done(
data={'answer': 'hello'},
success=True,
browser_session=browser_session,
file_system=file_system,
)
assert isinstance(result, ActionResult)
assert result.is_done is True
assert result.success is True
assert result.extracted_content is not None
output = json.loads(result.extracted_content)
assert output == {'answer': 'hello'}
assert result.attachments == []
async def test_structured_output_done_with_files_to_display(self, browser_session, base_url):
"""Structured output done action resolves files_to_display into attachments."""
class MyOutput(BaseModel):
summary: str
tools = Tools(output_model=MyOutput)
with tempfile.TemporaryDirectory() as temp_dir:
file_system = FileSystem(temp_dir)
await file_system.write_file('report.txt', 'some report content')
result = await tools.done(
data={'summary': 'done'},
success=True,
files_to_display=['report.txt'],
browser_session=browser_session,
file_system=file_system,
)
assert isinstance(result, ActionResult)
assert result.is_done is True
assert result.success is True
assert result.extracted_content is not None
output = json.loads(result.extracted_content)
assert output == {'summary': 'done'}
assert result.attachments is not None
assert len(result.attachments) == 1
assert result.attachments[0].endswith('report.txt')
async def test_structured_output_done_auto_attaches_downloads(self, browser_session, base_url):
"""Session downloads are auto-attached even without files_to_display."""
class MyOutput(BaseModel):
url: str
tools = Tools(output_model=MyOutput)
with tempfile.TemporaryDirectory() as temp_dir:
file_system = FileSystem(temp_dir)
# Simulate a CDP-tracked browser download
fake_download = os.path.join(temp_dir, 'tax-bill.pdf')
await anyio.Path(fake_download).write_bytes(b'%PDF-1.4 fake pdf content')
saved_downloads = browser_session._downloaded_files.copy()
browser_session._downloaded_files.append(fake_download)
try:
result = await tools.done(
data={'url': f'{base_url}/bill.pdf'},
success=True,
browser_session=browser_session,
file_system=file_system,
)
assert isinstance(result, ActionResult)
assert result.is_done is True
assert result.extracted_content is not None
output = json.loads(result.extracted_content)
assert output == {'url': f'{base_url}/bill.pdf'}
# The download should be auto-attached
assert result.attachments is not None
assert len(result.attachments) == 1
assert result.attachments[0] == fake_download
finally:
browser_session._downloaded_files = saved_downloads
async def test_structured_output_done_deduplicates_attachments(self, browser_session):
"""Downloads already covered by files_to_display are not duplicated."""
class MyOutput(BaseModel):
status: str
tools = Tools(output_model=MyOutput)
with tempfile.TemporaryDirectory() as temp_dir:
file_system = FileSystem(temp_dir)
await file_system.write_file('report.txt', 'content here')
# The same file appears in both files_to_display and session downloads
fs_path = str(file_system.get_dir() / 'report.txt')
saved_downloads = browser_session._downloaded_files.copy()
browser_session._downloaded_files.append(fs_path)
try:
result = await tools.done(
data={'status': 'ok'},
success=True,
files_to_display=['report.txt'],
browser_session=browser_session,
file_system=file_system,
)
assert isinstance(result, ActionResult)
# Should have exactly 1 attachment, not 2
assert result.attachments is not None
assert len(result.attachments) == 1
assert result.attachments[0] == fs_path
finally:
browser_session._downloaded_files = saved_downloads
async def test_structured_output_done_nonexistent_file_ignored(self, browser_session):
"""Files that don't exist in FileSystem are not included via files_to_display."""
class MyOutput(BaseModel):
value: int
tools = Tools(output_model=MyOutput)
with tempfile.TemporaryDirectory() as temp_dir:
file_system = FileSystem(temp_dir)
result = await tools.done(
data={'value': 42},
success=True,
files_to_display=['nonexistent.txt'],
browser_session=browser_session,
file_system=file_system,
)
assert isinstance(result, ActionResult)
assert result.is_done is True
assert result.extracted_content is not None
output = json.loads(result.extracted_content)
assert output == {'value': 42}
# nonexistent file should not appear in attachments
assert result.attachments == []
async def test_structured_output_schema_hides_internal_fields(self):
"""The JSON schema for StructuredOutputAction hides success and files_to_display."""
from browser_use.tools.views import StructuredOutputAction
class MyOutput(BaseModel):
name: str
schema = StructuredOutputAction[MyOutput].model_json_schema()
top_level_props = schema.get('properties', {})
assert 'success' not in top_level_props
assert 'files_to_display' not in top_level_props
# data should still be present
assert 'data' in top_level_props
+408
View File
@@ -0,0 +1,408 @@
"""Unit tests for variable detection in agent history"""
from browser_use.agent.variable_detector import (
_detect_from_attributes,
_detect_from_value_pattern,
_detect_variable_type,
_ensure_unique_name,
detect_variables_in_history,
)
from browser_use.agent.views import DetectedVariable
from browser_use.dom.views import DOMInteractedElement, NodeType
def create_test_element(attributes: dict[str, str] | None = None) -> DOMInteractedElement:
"""Helper to create a DOMInteractedElement for testing"""
return DOMInteractedElement(
node_id=1,
backend_node_id=1,
frame_id='frame1',
node_type=NodeType.ELEMENT_NODE,
node_value='',
node_name='input',
attributes=attributes or {},
bounds=None,
x_path='//*[@id="test"]',
element_hash=12345,
)
def create_mock_history(actions_with_elements: list[tuple[dict, DOMInteractedElement | None]]):
"""Helper to create mock history for testing"""
from types import SimpleNamespace
history_items = []
for action_dict, element in actions_with_elements:
mock_action = SimpleNamespace(**action_dict)
mock_output = SimpleNamespace(action=[mock_action])
mock_state = SimpleNamespace(interacted_element=[element] if element else None)
mock_history_item = SimpleNamespace(model_output=mock_output, state=mock_state)
history_items.append(mock_history_item)
return SimpleNamespace(history=history_items)
def test_detect_email_from_attributes():
"""Test email detection via type='email' attribute"""
attributes = {'type': 'email', 'id': 'email-input'}
result = _detect_from_attributes(attributes)
assert result is not None
var_name, var_format = result
assert var_name == 'email'
assert var_format == 'email'
def test_detect_email_from_pattern():
"""Test email detection via pattern matching"""
result = _detect_from_value_pattern('test@example.com')
assert result is not None
var_name, var_format = result
assert var_name == 'email'
assert var_format == 'email'
def test_detect_phone_from_attributes():
"""Test phone detection via type='tel' attribute"""
attributes = {'type': 'tel', 'name': 'phone'}
result = _detect_from_attributes(attributes)
assert result is not None
var_name, var_format = result
assert var_name == 'phone'
assert var_format == 'phone'
def test_detect_phone_from_pattern():
"""Test phone detection via pattern matching"""
result = _detect_from_value_pattern('+1 (555) 123-4567')
assert result is not None
var_name, var_format = result
assert var_name == 'phone'
assert var_format == 'phone'
def test_detect_date_from_attributes():
"""Test date detection via type='date' attribute"""
attributes = {'type': 'date', 'id': 'dob'}
result = _detect_from_attributes(attributes)
assert result is not None
var_name, var_format = result
assert var_name == 'date'
assert var_format == 'date'
def test_detect_date_from_pattern():
"""Test date detection via YYYY-MM-DD pattern"""
result = _detect_from_value_pattern('1990-01-01')
assert result is not None
var_name, var_format = result
assert var_name == 'date'
assert var_format == 'date'
def test_detect_first_name_from_attributes():
"""Test first name detection from element attributes"""
attributes = {'name': 'first_name', 'placeholder': 'Enter your first name'}
result = _detect_from_attributes(attributes)
assert result is not None
var_name, var_format = result
assert var_name == 'first_name'
assert var_format is None
def test_detect_first_name_from_pattern():
"""Test first name detection from pattern (single capitalized word)"""
result = _detect_from_value_pattern('John')
assert result is not None
var_name, var_format = result
assert var_name == 'first_name'
assert var_format is None
def test_detect_full_name_from_pattern():
"""Test full name detection from pattern (two capitalized words)"""
result = _detect_from_value_pattern('John Doe')
assert result is not None
var_name, var_format = result
assert var_name == 'full_name'
assert var_format is None
def test_detect_address_from_attributes():
"""Test address detection from element attributes"""
attributes = {'name': 'street_address', 'id': 'address-input'}
result = _detect_from_attributes(attributes)
assert result is not None
var_name, var_format = result
assert var_name == 'address'
assert var_format is None
def test_detect_billing_address_from_attributes():
"""Test billing address detection from element attributes"""
attributes = {'name': 'billing_address', 'placeholder': 'Billing street address'}
result = _detect_from_attributes(attributes)
assert result is not None
var_name, var_format = result
assert var_name == 'billing_address'
assert var_format is None
def test_detect_comment_from_attributes():
"""Test comment detection from element attributes"""
attributes = {'name': 'comment', 'placeholder': 'Enter your comment'}
result = _detect_from_attributes(attributes)
assert result is not None
var_name, var_format = result
assert var_name == 'comment'
assert var_format is None
def test_detect_city_from_attributes():
"""Test city detection from element attributes"""
attributes = {'name': 'city', 'id': 'city-input'}
result = _detect_from_attributes(attributes)
assert result is not None
var_name, var_format = result
assert var_name == 'city'
assert var_format is None
def test_detect_state_from_attributes():
"""Test state detection from element attributes"""
attributes = {'name': 'state', 'aria-label': 'State or Province'}
result = _detect_from_attributes(attributes)
assert result is not None
var_name, var_format = result
assert var_name == 'state'
assert var_format is None
def test_detect_country_from_attributes():
"""Test country detection from element attributes"""
attributes = {'name': 'country', 'id': 'country-select'}
result = _detect_from_attributes(attributes)
assert result is not None
var_name, var_format = result
assert var_name == 'country'
assert var_format is None
def test_detect_zip_code_from_attributes():
"""Test zip code detection from element attributes"""
attributes = {'name': 'zip_code', 'placeholder': 'Zip or postal code'}
result = _detect_from_attributes(attributes)
assert result is not None
var_name, var_format = result
assert var_name == 'zip_code'
assert var_format == 'postal_code'
def test_detect_company_from_attributes():
"""Test company detection from element attributes"""
attributes = {'name': 'company', 'id': 'company-input'}
result = _detect_from_attributes(attributes)
assert result is not None
var_name, var_format = result
assert var_name == 'company'
assert var_format is None
def test_detect_number_from_pattern():
"""Test number detection from pattern (pure digits)"""
result = _detect_from_value_pattern('12345')
assert result is not None
var_name, var_format = result
assert var_name == 'number'
assert var_format == 'number'
def test_no_detection_for_random_text():
"""Test that random text is not detected as a variable"""
result = _detect_from_value_pattern('some random text that is not a variable')
assert result is None
def test_no_detection_for_short_text():
"""Test that very short text is not detected"""
result = _detect_from_value_pattern('a')
assert result is None
def test_element_attributes_take_priority_over_pattern():
"""Test that element attributes are checked before pattern matching"""
# A value that could match pattern (capitalized name)
value = 'Test'
# Element with explicit type="email"
element = create_test_element(attributes={'type': 'email', 'id': 'email-input'})
result = _detect_variable_type(value, element)
assert result is not None
var_name, var_format = result
# Should detect as email (from attributes), not first_name (from pattern)
assert var_name == 'email'
assert var_format == 'email'
def test_pattern_matching_used_when_no_element():
"""Test that pattern matching is used when element context is missing"""
value = 'test@example.com'
result = _detect_variable_type(value, element=None)
assert result is not None
var_name, var_format = result
assert var_name == 'email'
assert var_format == 'email'
def test_ensure_unique_name_no_conflict():
"""Test unique name generation with no conflicts"""
existing = {}
result = _ensure_unique_name('email', existing)
assert result == 'email'
def test_ensure_unique_name_with_conflict():
"""Test unique name generation with conflicts"""
existing = {
'email': DetectedVariable(name='email', original_value='test1@example.com'),
}
result = _ensure_unique_name('email', existing)
assert result == 'email_2'
def test_ensure_unique_name_with_multiple_conflicts():
"""Test unique name generation with multiple conflicts"""
existing = {
'email': DetectedVariable(name='email', original_value='test1@example.com'),
'email_2': DetectedVariable(name='email_2', original_value='test2@example.com'),
}
result = _ensure_unique_name('email', existing)
assert result == 'email_3'
def test_detect_variables_in_empty_history():
"""Test variable detection in empty history"""
from types import SimpleNamespace
history = SimpleNamespace(history=[])
result = detect_variables_in_history(history) # type: ignore[arg-type]
assert result == {}
def test_detect_variables_in_history_with_input_action():
"""Test variable detection in history with input action"""
# Use mock objects to avoid Pydantic validation issues
from types import SimpleNamespace
# Create mock history structure
element = create_test_element(attributes={'type': 'email', 'id': 'email-input'})
mock_action = SimpleNamespace(**{'input': {'index': 1, 'text': 'test@example.com'}})
mock_output = SimpleNamespace(action=[mock_action])
mock_state = SimpleNamespace(interacted_element=[element])
mock_history_item = SimpleNamespace(model_output=mock_output, state=mock_state)
mock_history = SimpleNamespace(history=[mock_history_item])
result = detect_variables_in_history(mock_history) # type: ignore[arg-type]
assert len(result) == 1
assert 'email' in result
assert result['email'].original_value == 'test@example.com'
assert result['email'].format == 'email'
def test_detect_variables_skips_duplicate_values():
"""Test that duplicate values are only detected once"""
# Create history with same value entered twice
element = create_test_element(attributes={'type': 'email'})
history = create_mock_history(
[
({'input': {'index': 1, 'text': 'test@example.com'}}, element),
({'input': {'index': 2, 'text': 'test@example.com'}}, element),
]
)
result = detect_variables_in_history(history) # type: ignore[arg-type]
# Should only detect one variable, not two
assert len(result) == 1
assert 'email' in result
def test_detect_variables_handles_missing_state():
"""Test that detection works when state is missing"""
from types import SimpleNamespace
# Create history with None state
mock_action = SimpleNamespace(**{'input': {'index': 1, 'text': 'test@example.com'}})
mock_output = SimpleNamespace(action=[mock_action])
mock_history_item = SimpleNamespace(model_output=mock_output, state=None)
history = SimpleNamespace(history=[mock_history_item])
result = detect_variables_in_history(history) # type: ignore[arg-type]
# Should still detect via pattern matching
assert len(result) == 1
assert 'email' in result
assert result['email'].original_value == 'test@example.com'
def test_detect_variables_handles_missing_interacted_element():
"""Test that detection works when interacted_element is missing"""
# Use None as element to test when interacted_element is None
history = create_mock_history(
[
({'input': {'index': 1, 'text': 'test@example.com'}}, None),
]
)
result = detect_variables_in_history(history) # type: ignore[arg-type]
# Should still detect via pattern matching
assert len(result) == 1
assert 'email' in result
def test_detect_variables_multiple_types():
"""Test detection of multiple variable types in one history"""
history = create_mock_history(
[
({'input': {'index': 1, 'text': 'test@example.com'}}, create_test_element(attributes={'type': 'email'})),
({'input': {'index': 2, 'text': 'John'}}, create_test_element(attributes={'name': 'first_name'})),
({'input': {'index': 3, 'text': '1990-01-01'}}, create_test_element(attributes={'type': 'date'})),
]
)
result = detect_variables_in_history(history) # type: ignore[arg-type]
assert len(result) == 3
assert 'email' in result
assert 'first_name' in result
assert 'date' in result
assert result['email'].original_value == 'test@example.com'
assert result['first_name'].original_value == 'John'
assert result['date'].original_value == '1990-01-01'
+272
View File
@@ -0,0 +1,272 @@
"""Unit tests for variable substitution in agent history"""
from types import SimpleNamespace
from browser_use.agent.service import Agent
from browser_use.dom.views import DOMInteractedElement, NodeType
def create_test_element(attributes: dict[str, str] | None = None) -> DOMInteractedElement:
"""Helper to create a DOMInteractedElement for testing"""
return DOMInteractedElement(
node_id=1,
backend_node_id=1,
frame_id='frame1',
node_type=NodeType.ELEMENT_NODE,
node_value='',
node_name='input',
attributes=attributes or {},
bounds=None,
x_path='//*[@id="test"]',
element_hash=12345,
)
def create_mock_history(actions_with_elements: list[tuple[dict, DOMInteractedElement | None]]):
"""Helper to create mock history for testing"""
history_items = []
for action_dict, element in actions_with_elements:
mock_action = SimpleNamespace(**action_dict)
mock_output = SimpleNamespace(action=[mock_action])
mock_state = SimpleNamespace(interacted_element=[element] if element else None)
mock_history_item = SimpleNamespace(model_output=mock_output, state=mock_state)
history_items.append(mock_history_item)
return SimpleNamespace(history=history_items)
def test_substitute_single_variable(mock_llm):
"""Test substitution of a single variable"""
agent = Agent(task='test', llm=mock_llm)
# Create mock history with email
element = create_test_element(attributes={'type': 'email'})
history = create_mock_history(
[
({'input': {'index': 1, 'text': 'old@example.com'}}, element),
]
)
# Substitute the email
modified_history = agent._substitute_variables_in_history(
history, # type: ignore[arg-type]
{'email': 'new@example.com'},
)
# Check that the value was substituted
action = modified_history.history[0].model_output.action[0] # type: ignore[attr-defined]
action_dict = vars(action)
assert action_dict['input']['text'] == 'new@example.com'
def test_substitute_multiple_variables(mock_llm):
"""Test substitution of multiple variables"""
agent = Agent(task='test', llm=mock_llm)
# Create mock history with email and name
history = create_mock_history(
[
({'input': {'index': 1, 'text': 'old@example.com'}}, create_test_element(attributes={'type': 'email'})),
({'input': {'index': 2, 'text': 'John'}}, create_test_element(attributes={'name': 'first_name'})),
({'input': {'index': 3, 'text': '1990-01-01'}}, create_test_element(attributes={'type': 'date'})),
]
)
# Substitute all variables
modified_history = agent._substitute_variables_in_history(
history, # type: ignore[arg-type]
{
'email': 'new@example.com',
'first_name': 'Jane',
'date': '1995-05-15',
},
)
# Check that all values were substituted
action1 = modified_history.history[0].model_output.action[0] # type: ignore[attr-defined]
action2 = modified_history.history[1].model_output.action[0] # type: ignore[attr-defined]
action3 = modified_history.history[2].model_output.action[0] # type: ignore[attr-defined]
assert vars(action1)['input']['text'] == 'new@example.com'
assert vars(action2)['input']['text'] == 'Jane'
assert vars(action3)['input']['text'] == '1995-05-15'
def test_substitute_partial_variables(mock_llm):
"""Test substitution of only some variables"""
agent = Agent(task='test', llm=mock_llm)
# Create mock history with email and name
history = create_mock_history(
[
({'input': {'index': 1, 'text': 'old@example.com'}}, create_test_element(attributes={'type': 'email'})),
({'input': {'index': 2, 'text': 'John'}}, create_test_element(attributes={'name': 'first_name'})),
]
)
# Substitute only email
modified_history = agent._substitute_variables_in_history(
history, # type: ignore[arg-type]
{'email': 'new@example.com'},
)
# Check that only email was substituted
action1 = modified_history.history[0].model_output.action[0] # type: ignore[attr-defined]
action2 = modified_history.history[1].model_output.action[0] # type: ignore[attr-defined]
assert vars(action1)['input']['text'] == 'new@example.com'
assert vars(action2)['input']['text'] == 'John' # Unchanged
def test_substitute_nonexistent_variable(mock_llm):
"""Test that substituting a nonexistent variable doesn't break things"""
agent = Agent(task='test', llm=mock_llm)
# Create mock history with email
element = create_test_element(attributes={'type': 'email'})
history = create_mock_history(
[
({'input': {'index': 1, 'text': 'old@example.com'}}, element),
]
)
# Try to substitute a variable that doesn't exist
modified_history = agent._substitute_variables_in_history(
history, # type: ignore[arg-type]
{'nonexistent_var': 'some_value'},
)
# Check that nothing changed
action = modified_history.history[0].model_output.action[0] # type: ignore[attr-defined]
action_dict = vars(action)
assert action_dict['input']['text'] == 'old@example.com'
def test_substitute_in_nested_dict(mock_llm):
"""Test substitution in nested dictionary structures"""
agent = Agent(task='test', llm=mock_llm)
# Create a more complex action with nested structure
complex_action = {
'search_google': {
'query': 'test@example.com',
'metadata': {'user': 'test@example.com'},
}
}
element = create_test_element(attributes={'type': 'email'})
history = create_mock_history([(complex_action, element)])
# Substitute the email
modified_history = agent._substitute_variables_in_history(
history, # type: ignore[arg-type]
{'email': 'new@example.com'},
)
# Check that values in nested structures were substituted
action = modified_history.history[0].model_output.action[0] # type: ignore[attr-defined]
action_dict = vars(action)
assert action_dict['search_google']['query'] == 'new@example.com'
assert action_dict['search_google']['metadata']['user'] == 'new@example.com'
def test_substitute_in_list(mock_llm):
"""Test substitution in list structures"""
agent = Agent(task='test', llm=mock_llm)
# Create history with an input action first (so email is detected)
# Then an action with a list containing the same email
history = create_mock_history(
[
({'input': {'index': 1, 'text': 'test@example.com'}}, create_test_element(attributes={'type': 'email'})),
({'some_action': {'items': ['test@example.com', 'other_value', 'test@example.com']}}, None),
]
)
# Substitute the email
modified_history = agent._substitute_variables_in_history(
history, # type: ignore[arg-type]
{'email': 'new@example.com'},
)
# Check that values in the first action were substituted
action1 = modified_history.history[0].model_output.action[0] # type: ignore[attr-defined]
assert vars(action1)['input']['text'] == 'new@example.com'
# Check that values in lists were also substituted
action2 = modified_history.history[1].model_output.action[0] # type: ignore[attr-defined]
action_dict = vars(action2)
assert action_dict['some_action']['items'] == ['new@example.com', 'other_value', 'new@example.com']
def test_substitute_preserves_original_history(mock_llm):
"""Test that substitution doesn't modify the original history"""
agent = Agent(task='test', llm=mock_llm)
# Create mock history
element = create_test_element(attributes={'type': 'email'})
history = create_mock_history(
[
({'input': {'index': 1, 'text': 'old@example.com'}}, element),
]
)
# Get original value
original_action = history.history[0].model_output.action[0]
original_value = vars(original_action)['input']['text']
# Substitute
agent._substitute_variables_in_history(history, {'email': 'new@example.com'}) # type: ignore[arg-type]
# Check that original history is unchanged
current_value = vars(original_action)['input']['text']
assert current_value == original_value
assert current_value == 'old@example.com'
def test_substitute_empty_variables(mock_llm):
"""Test substitution with empty variables dict"""
agent = Agent(task='test', llm=mock_llm)
# Create mock history
element = create_test_element(attributes={'type': 'email'})
history = create_mock_history(
[
({'input': {'index': 1, 'text': 'old@example.com'}}, element),
]
)
# Substitute with empty dict
modified_history = agent._substitute_variables_in_history(history, {}) # type: ignore[arg-type]
# Check that nothing changed
action = modified_history.history[0].model_output.action[0] # type: ignore[attr-defined]
action_dict = vars(action)
assert action_dict['input']['text'] == 'old@example.com'
def test_substitute_same_value_multiple_times(mock_llm):
"""Test that the same value is substituted across multiple actions"""
agent = Agent(task='test', llm=mock_llm)
# Create history where same email appears twice
element = create_test_element(attributes={'type': 'email'})
history = create_mock_history(
[
({'input': {'index': 1, 'text': 'old@example.com'}}, element),
({'input': {'index': 2, 'text': 'old@example.com'}}, element),
]
)
# Substitute the email
modified_history = agent._substitute_variables_in_history(
history, # type: ignore[arg-type]
{'email': 'new@example.com'},
)
# Check that both occurrences were substituted
action1 = modified_history.history[0].model_output.action[0] # type: ignore[attr-defined]
action2 = modified_history.history[1].model_output.action[0] # type: ignore[attr-defined]
assert vars(action1)['input']['text'] == 'new@example.com'
assert vars(action2)['input']['text'] == 'new@example.com'
+299
View File
@@ -0,0 +1,299 @@
"""
Debug test for iframe scrolling issue where DOM tree only shows top elements after scrolling.
This test verifies that after scrolling inside an iframe, the selector_map correctly
contains lower input elements like City, State, Zip Code, etc.
"""
import asyncio
import sys
from pathlib import Path
# Add parent directory to path to import browser_use modules
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
from browser_use.agent.service import Agent
from browser_use.agent.views import ActionModel
from browser_use.browser import BrowserProfile, BrowserSession
from browser_use.browser.events import BrowserStateRequestEvent
# Import the mock LLM helper from conftest
from tests.ci.conftest import create_mock_llm
async def debug_iframe_scrolling():
"""Debug iframe scrolling and DOM visibility issue."""
print('Starting iframe scrolling debug test...')
# Create the sequence of actions for the mock LLM
# We need to format these as the LLM would return them
actions = [
# First action: Navigate to the test URL
"""
{
"thinking": "Navigating to the iframe test page",
"evaluation_previous_goal": null,
"memory": "Starting test",
"next_goal": "Navigate to the iframe test page",
"action": [
{
"navigate": {
"url": "https://browser-use.github.io/stress-tests/challenges/iframe-inception-level1.html",
"new_tab": false
}
}
]
}
""",
# Second action: Input text in the first name field (to verify we can interact)
"""
{
"thinking": "Inputting text in the first name field to test interaction",
"evaluation_previous_goal": "Successfully navigated to the page",
"memory": "Page loaded with nested iframes",
"next_goal": "Type text in the first name field",
"action": [
{
"input_text": {
"index": 1,
"text": "TestName"
}
}
]
}
""",
# Third action: Scroll the iframe (element_index=2 should be the iframe)
"""
{
"thinking": "Scrolling inside the iframe to reveal lower form elements",
"evaluation_previous_goal": "Successfully typed in first name field",
"memory": "Typed TestName in first field",
"next_goal": "Scroll inside the innermost iframe to see more form fields",
"action": [
{
"scroll": {
"down": true,
"num_pages": 1.0,
"index": 2
}
}
]
}
""",
# Fourth action: Done
"""
{
"thinking": "Completed scrolling, ready to inspect DOM",
"evaluation_previous_goal": "Successfully scrolled inside iframe",
"memory": "Scrolled to reveal lower form fields",
"next_goal": "Task completed",
"action": [
{
"done": {
"text": "Scrolling completed",
"success": true
}
}
]
}
""",
]
# Create mock LLM with our action sequence
mock_llm = create_mock_llm(actions=actions)
# Create browser session with headless=False so we can see what's happening
browser_session = BrowserSession(
browser_profile=BrowserProfile(
headless=False, # Set to False to see the browser
user_data_dir=None, # Use temporary directory
keep_alive=True,
enable_default_extensions=True,
cross_origin_iframes=True, # Enable cross-origin iframe support
)
)
try:
# Start the browser session
await browser_session.start()
print('Browser session started')
# Create an agent with the mock LLM
agent = Agent(
task='Navigate to the iframe test page and scroll inside the iframe',
llm=mock_llm,
browser_session=browser_session,
)
# Helper function to capture and analyze DOM state
async def capture_dom_state(label: str) -> dict:
"""Capture DOM state and return analysis"""
print(f'\n📸 Capturing DOM state: {label}')
state_event = browser_session.event_bus.dispatch(
BrowserStateRequestEvent(include_dom=True, include_screenshot=False, include_recent_events=False)
)
browser_state = await state_event.event_result()
if browser_state and browser_state.dom_state and browser_state.dom_state.selector_map:
selector_map = browser_state.dom_state.selector_map
element_count = len(selector_map)
# Check for specific elements
found_elements = {}
expected_checks = [
('First Name', ['firstName', 'first name']),
('Last Name', ['lastName', 'last name']),
('Email', ['email']),
('City', ['city']),
('State', ['state']),
('Zip', ['zip', 'zipCode']),
]
for name, keywords in expected_checks:
for index, element in selector_map.items():
element_str = str(element).lower()
if any(kw.lower() in element_str for kw in keywords):
found_elements[name] = True
break
return {
'label': label,
'total_elements': element_count,
'found_elements': found_elements,
'selector_map': selector_map,
}
return {'label': label, 'error': 'No DOM state available'}
# Capture initial state before any actions
print('\n' + '=' * 80)
print('PHASE 1: INITIAL PAGE LOAD')
print('=' * 80)
# Navigate to the page first
from browser_use.tools.service import Tools
tools = Tools()
# Create the action model for navigation
goto_action = ActionModel.model_validate_json(actions[0])
await tools.act(goto_action, browser_session)
await asyncio.sleep(2) # Wait for page to fully load
initial_state = await capture_dom_state('INITIAL (after page load)')
# Now run the rest of the actions via the agent
print('\n' + '=' * 80)
print('PHASE 2: EXECUTING ACTIONS')
print('=' * 80)
# Create new agent with remaining actions
remaining_actions = actions[1:] # Skip the navigation we already did
mock_llm_remaining = create_mock_llm(actions=remaining_actions)
agent = Agent(
task='Input text and scroll inside the iframe',
llm=mock_llm_remaining,
browser_session=browser_session,
)
# Hook into agent actions to capture state after each one
states = []
original_act = tools.act
async def wrapped_act(action, session):
result = await original_act(action, session)
# Capture state after each action
action_type = 'unknown'
if hasattr(action, 'input_text') and action.input_text:
action_type = 'input_text'
await asyncio.sleep(1) # Give time for DOM to update
state = await capture_dom_state('AFTER INPUT_TEXT')
states.append(state)
elif hasattr(action, 'scroll') and action.scroll:
action_type = 'scroll'
await asyncio.sleep(2) # Give more time after scroll
state = await capture_dom_state('AFTER SCROLL')
states.append(state)
return result
tools.act = wrapped_act
# Run the agent with remaining actions
result = await agent.run()
print(f'\nAgent completed with result: {result}')
# Analyze all captured states
print('\n' + '=' * 80)
print('PHASE 3: ANALYSIS OF DOM STATES')
print('=' * 80)
all_states = [initial_state] + states
for state in all_states:
if 'error' in state:
print(f'\n{state["label"]}: {state["error"]}')
else:
print(f'\n📊 {state["label"]}:')
print(f' Total elements: {state["total_elements"]}')
print(' Found elements:')
for elem_name, found in state['found_elements'].items():
status = '' if found else ''
print(f' {status} {elem_name}')
# Compare states
print('\n' + '=' * 80)
print('COMPARISON SUMMARY')
print('=' * 80)
if len(all_states) >= 3:
initial = all_states[0]
after_input = all_states[1] if len(all_states) > 1 else None
after_scroll = all_states[2] if len(all_states) > 2 else None
print('\nElement count changes:')
print(f' Initial: {initial.get("total_elements", 0)} elements')
if after_input:
print(f' After input_text: {after_input.get("total_elements", 0)} elements')
if after_scroll:
print(f' After scroll: {after_scroll.get("total_elements", 0)} elements')
# Check if lower form fields appear after scroll
if after_scroll and 'found_elements' in after_scroll:
lower_fields = ['City', 'State', 'Zip']
missing_fields = [f for f in lower_fields if not after_scroll['found_elements'].get(f, False)]
if missing_fields:
print('\n⚠️ BUG CONFIRMED: Lower form fields missing after scroll:')
for field in missing_fields:
print(f'{field}')
print('\nThis confirms that scrolling inside iframes does not update the DOM tree properly.')
else:
print('\n✅ SUCCESS: All lower form fields are visible after scrolling!')
# Show first few elements from final state for debugging
if states and 'selector_map' in states[-1]:
print('\n' + '=' * 80)
print('DEBUG: First 5 elements in final selector_map')
print('=' * 80)
final_map = states[-1]['selector_map']
for i, (index, element) in enumerate(list(final_map.items())[:5]):
elem_preview = str(element)[:150]
print(f'\n [{index}]: {elem_preview}...')
# Keep browser open for manual inspection if needed
print('\n' + '=' * 80)
print('Test complete. Browser will remain open for 10 seconds for inspection...')
print('=' * 80)
await asyncio.sleep(10)
finally:
# Clean up
print('\nCleaning up...')
await browser_session.kill()
await browser_session.event_bus.stop(clear=True, timeout=5)
print('Browser session closed')
if __name__ == '__main__':
# Run the debug test
asyncio.run(debug_iframe_scrolling())
+260
View File
@@ -0,0 +1,260 @@
#!/usr/bin/env python3
"""Test frame hierarchy for any URL passed as argument."""
import asyncio
import sys
from browser_use.browser import BrowserSession
from browser_use.browser.events import BrowserStartEvent
from browser_use.browser.profile import BrowserProfile
async def analyze_frame_hierarchy(url):
"""Analyze and display complete frame hierarchy for a URL."""
profile = BrowserProfile(headless=True, user_data_dir=None)
session = BrowserSession(browser_profile=profile)
try:
print('🚀 Starting browser...')
await session.on_BrowserStartEvent(BrowserStartEvent())
print(f'📍 Navigating to: {url}')
await session._cdp_navigate(url)
await asyncio.sleep(3)
print('\n' + '=' * 80)
print('FRAME HIERARCHY ANALYSIS')
print('=' * 80)
# Get all targets from SessionManager
all_targets = session.session_manager.get_all_targets()
# Separate by type
page_targets = [target for target in all_targets.values() if target.target_type == 'page']
iframe_targets = [target for target in all_targets.values() if target.target_type == 'iframe']
print('\n📊 Target Summary:')
print(f' Total targets: {len(all_targets)}')
print(f' Page targets: {len(page_targets)}')
print(f' Iframe targets (OOPIFs): {len(iframe_targets)}')
# Show all targets
print('\n📋 All Targets:')
for i, (target_id, target) in enumerate(all_targets.items()):
if target.target_type in ['page', 'iframe']:
print(f'\n [{i + 1}] Type: {target.target_type}')
print(f' URL: {target.url}')
print(f' Target ID: {target.target_id[:30]}...')
# Check if target has active sessions using the public API
try:
cdp_session = await session.get_or_create_cdp_session(target.target_id, focus=False)
has_session = cdp_session is not None
except Exception:
has_session = False
print(f' Has Session: {has_session}')
# Get main page frame tree
main_target = next((t for t in page_targets if url in t.url), page_targets[0] if page_targets else None)
if main_target:
print('\n📐 Main Page Frame Tree:')
print(f' Target: {main_target.url}')
print(f' Target ID: {main_target.target_id[:30]}...')
s = await session.cdp_client.send.Target.attachToTarget(params={'targetId': main_target.target_id, 'flatten': True})
sid = s['sessionId']
try:
await session.cdp_client.send.Page.enable(session_id=sid)
tree = await session.cdp_client.send.Page.getFrameTree(session_id=sid)
print('\n Frame Tree Structure:')
def print_tree(node, indent=0, parent_id=None):
frame = node['frame']
frame_id = frame.get('id', 'unknown')
frame_url = frame.get('url', 'none')
prefix = ' ' * indent + ('└─ ' if indent > 0 else '')
print(f'{prefix}Frame: {frame_url}')
print(f'{" " * (indent + 1)}ID: {frame_id[:30]}...')
if parent_id:
print(f'{" " * (indent + 1)}Parent: {parent_id[:30]}...')
# Check cross-origin status
cross_origin = frame.get('crossOriginIsolatedContextType', 'unknown')
if cross_origin != 'NotIsolated':
print(f'{" " * (indent + 1)}⚠️ Cross-Origin: {cross_origin}')
# Process children
for child in node.get('childFrames', []):
print_tree(child, indent + 1, frame_id)
print_tree(tree['frameTree'])
finally:
await session.cdp_client.send.Target.detachFromTarget(params={'sessionId': sid})
# Show iframe target trees
if iframe_targets:
print('\n🔸 OOPIF Target Frame Trees:')
for iframe_target in iframe_targets:
print(f'\n OOPIF Target: {iframe_target.url}')
print(f' Target ID: {iframe_target.target_id[:30]}...')
s = await session.cdp_client.send.Target.attachToTarget(
params={'targetId': iframe_target.target_id, 'flatten': True}
)
sid = s['sessionId']
try:
await session.cdp_client.send.Page.enable(session_id=sid)
tree = await session.cdp_client.send.Page.getFrameTree(session_id=sid)
frame = tree['frameTree']['frame']
print(f' Frame ID: {frame.get("id", "unknown")[:30]}...')
print(f' Frame URL: {frame.get("url", "none")}')
print(' ⚠️ This frame runs in a separate process (OOPIF)')
except Exception as e:
print(f' Error: {e}')
finally:
await session.cdp_client.send.Target.detachFromTarget(params={'sessionId': sid})
# Now show unified view from get_all_frames
print('\n' + '=' * 80)
print('UNIFIED FRAME HIERARCHY (get_all_frames method)')
print('=' * 80)
all_frames, target_sessions = await session.get_all_frames()
# Clean up sessions
for tid, sess_id in target_sessions.items():
try:
await session.cdp_client.send.Target.detachFromTarget(params={'sessionId': sess_id})
except Exception:
pass
print('\n📊 Frame Statistics:')
print(f' Total frames discovered: {len(all_frames)}')
# Separate root and child frames
root_frames = []
child_frames = []
for frame_id, frame_info in all_frames.items():
if not frame_info.get('parentFrameId'):
root_frames.append((frame_id, frame_info))
else:
child_frames.append((frame_id, frame_info))
print(f' Root frames: {len(root_frames)}')
print(f' Child frames: {len(child_frames)}')
# Display all frames with details
print('\n📋 All Frames:')
for i, (frame_id, frame_info) in enumerate(all_frames.items()):
url = frame_info.get('url', 'none')
parent = frame_info.get('parentFrameId')
target_id = frame_info.get('frameTargetId', 'unknown')
is_cross = frame_info.get('isCrossOrigin', False)
print(f'\n [{i + 1}] Frame URL: {url}')
print(f' Frame ID: {frame_id[:30]}...')
print(f' Parent Frame ID: {parent[:30] + "..." if parent else "None (ROOT)"}')
print(f' Target ID: {target_id[:30]}...')
print(f' Cross-Origin: {is_cross}')
# Highlight problems
if not parent and 'v0-simple-landing' in url:
print(' ❌ PROBLEM: Cross-origin frame incorrectly marked as root!')
elif not parent and url != 'about:blank' and url not in ['chrome://newtab/', 'about:blank']:
# Check if this should be the main frame
if any(url in t.url for t in page_targets):
print(' ✅ Correctly identified as root frame')
if is_cross:
print(' 🔸 This is a cross-origin frame (OOPIF)')
# Show parent-child relationships
print('\n🌳 Frame Relationships:')
# Build a tree structure
def print_frame_tree(frame_id, frame_info, indent=0, visited=None):
if visited is None:
visited = set()
if frame_id in visited:
return
visited.add(frame_id)
url = frame_info.get('url', 'none')
prefix = ' ' * indent + ('└─ ' if indent > 0 else '')
print(f'{prefix}{url[:60]}...')
print(f'{" " * (indent + 1)}[{frame_id[:20]}...]')
# Find children
for child_id, child_info in all_frames.items():
if child_info.get('parentFrameId') == frame_id:
print_frame_tree(child_id, child_info, indent + 1, visited)
# Print trees starting from roots
for frame_id, frame_info in root_frames:
print('\n Tree starting from root:')
print_frame_tree(frame_id, frame_info)
print('\n' + '=' * 80)
print('✅ Analysis complete!')
print('=' * 80)
except Exception as e:
print(f'❌ Error: {e}')
import traceback
traceback.print_exc()
finally:
# Stop the CDP client first before killing the browser
print('\n🛑 Shutting down...')
# Close CDP connection first while browser is still alive
if session._cdp_client_root:
try:
await session._cdp_client_root.stop()
except Exception:
pass # Ignore errors if already disconnected
# Then stop the browser process
from browser_use.browser.events import BrowserStopEvent
stop_event = session.event_bus.dispatch(BrowserStopEvent())
try:
await asyncio.wait_for(stop_event, timeout=2.0)
except TimeoutError:
print('⚠️ Browser stop timed out')
def main():
if len(sys.argv) != 2:
print('Usage: python test_frame_hierarchy.py <URL>')
print('\nExample URLs to test:')
print(' https://v0-website-with-clickable-elements.vercel.app/nested-iframe')
print(' https://v0-website-with-clickable-elements.vercel.app/cross-origin')
print(' https://v0-website-with-clickable-elements.vercel.app/shadow-dom')
sys.exit(1)
url = sys.argv[1]
asyncio.run(analyze_frame_hierarchy(url))
# Ensure clean exit
print('✅ Script completed')
sys.exit(0)
if __name__ == '__main__':
main()