4cd2d4af2b
Test Browser Use CLI Install / uv pip install (macos-latest) (push) Waiting to run
Test Browser Use CLI Install / uv pip install (windows-latest) (push) Waiting to run
Test Browser Use CLI Install / uv pip install (ubuntu-latest) (push) Failing after 1s
Test Browser Use CLI Install / uvx browser-use from local wheel (push) Failing after 1s
Test Browser Use CLI Install / uvx browser-use[cli] from PyPI (push) Failing after 1s
package / pip-install-on-macos-latest-py-3.11 (push) Has been skipped
package / pip-install-on-macos-latest-py-3.13 (push) Has been skipped
package / pip-install-on-ubuntu-latest-py-3.11 (push) Has been skipped
package / pip-install-on-windows-latest-py-3.13 (push) Has been skipped
cloud_evals / trigger_cloud_eval_image_build (push) Failing after 1s
docker / build_publish_image (push) Failing after 1s
Test Browser Use CLI Install / browser-use skill sync (push) Failing after 1s
lint / code-style (push) Failing after 0s
lint / type-checker (push) Failing after 1s
package / pip-build (push) Failing after 1s
lint / syntax-errors (push) Failing after 3s
package / pip-install-on-ubuntu-latest-py-3.13 (push) Has been skipped
package / pip-install-on-windows-latest-py-3.11 (push) Has been skipped
test / ${{ matrix.test_filename }} (push) Has been skipped
test / evaluate-tasks (push) Has been skipped
test / setup-chromium (push) Failing after 2s
test / find_tests (push) Failing after 2s
30 KiB
30 KiB
Rust Agent Integration Feature/Proof Ledger
This branch keeps the Python Agent unchanged unless callers explicitly import
from browser_use.beta import Agent.
Current Features
- Rust SDK server execution path
browser_use.beta.Agentnow runs normal tasks through the terminal SDK server (browser-use-terminal sdk-server --transport stdio) using the normalizedagent.run_taskrequest/response protocol.- Follow-up tasks reuse the returned SDK
agent_id,session_id, andbrowser_idthroughagent.run, so the Python-facing interface can keep one Rust-owned session. - Browser-use-style options are mapped into SDK params, including model/provider, CDP URL/headers, viewport, user agent, storage state, downloads path, structured output schema, max steps, vision, cost calculation, and action limits.
- The returned normalized event history is reconstructed into Browser Use-compatible
AgentHistoryList, callbacks, usage, telemetry, Laminar replay, downloads, and final result handling. - The SDK stdout reader accepts large JSON-RPC response lines by reading stdout in chunks and splitting newline-delimited JSON-RPC manually. This avoids asyncio's separator-limit failure when normalized history includes screenshots, tool output, and observability data in a single response line.
- SDK server
agent.event/agent.projected_eventnotifications are retained and surfaced as concise in-flight progress logs, so GitHub-runner evals can show where a Rust-backed run is spending time before the final history response arrives. - Terminal SDK
agent.runnow mirrors the live executor by passing the latest durable task/follow-up input intoRuntimeHandle::run_agentasinitial_input, so SDKagent.run_taskenters the runtime-owned loop withagent.input.acceptedandagent.input.consumedevents instead of stalling after browser creation. - Browser-use
llm_timeout/SDKllm.timeoutnow reaches the terminal Rust model stream path as both a response-open timeout and a stream-idle timeout, so a provider request that never returns response headers, or a stream that opens and then sends no SSE bytes, becomes a retryable transport error instead of holding a GitHub eval runner indefinitely aftermodel.turn.request. - Terminal SDK
agent.run_tasknow drives the runtime on a multi-thread Tokio runtime, matching the live model transport bridge'sblock_in_placeusage. This prevents SDK evals from stalling aftermodel.turn.requestbefore the configured response-open/stream-idle timeout can make progress. - Running
browser_scriptcalls now preserve therun_idand observe instruction in Rust event persistence, replay reconstruction, and PythonAgentHistoryListreconstruction. This prevents eval traces from showing empty browser tool results while a script is still active and keeps the next model turn on the intended observe/cancel path instead of repeatedly navigating or reconnecting. - Simple initial navigation actions on CDP-backed Rust runs are now left for
the Rust SDK by default, so the first navigation is a model-visible
browser_scriptaction with terminal page-state output instead of a hidden Python-side BrowserSession preload. The old direct CDP preload remains available behindBROWSER_USE_RUST_DIRECT_INITIAL_NAVIGATION=1. - Structured
browser_scriptlifecycle events now preserve outputs, summaries, images, and browser state through terminal event persistence and Python history reconstruction. This prevents the first post-navigation page probe from appearing blank in eval/Laminar history when the script emitted onlyemit_output(...)or screenshots and no stdout text. - Printed
browser_scriptpage probes are also used to reconstruct browser state when they containpage_info()dictionaries,list_tabs()rows, or a bare URL. This keepsPage Statealigned with visible tool output even when the script usedprint(info)instead ofemit_output(info, ...). - Terminal
list_tabs()hides the internalStarting agent ...about:blank placeholder tab. This prevents the model from mistaking the startup tab for a user-relevant target and spending turns on unnecessary reconnect/reattach recovery after a successful navigation. - Terminal Rust runs only advertise sub-agent tools when the run has a
configured child-agent runner, and SDK JSON-RPC runs now attach the same
runtime-backed child runner as CLI runs. This enables terminal's advanced
sub-agent system in
browser_use.beta.Agentevals without exposing spawn tools on unsupported run surfaces. - Eval GitHub runners now refresh Convex progress during long
run_agentandevaluatestages, so slow Rust or Agent SDK judge calls stay visible instead of looking stale while they are still legitimately running. - Terminal provider conversion now downsamples oversized data-URL screenshots before Anthropic requests. This keeps visual context in the model input while avoiding Claude's many-image 2000px dimension rejection, which was causing completed eval tasks to show a one-step HTTP 400 error and an empty final result.
- Terminal direct Anthropic Messages serialization now also downsamples
oversized inline
ContentPart::Mediabase64 screenshots before request construction. This covers Rust live-model transports that bypass provider JSON normalization and previously still hit the same 2000px many-image rejection. - Terminal navigation helpers now wait briefly for page readiness and return
navigation_readypluspage_infoin the samebrowser_scriptresult. This gives the next model turn concrete evidence that navigation landed, instead of a bare "navigation sent" placeholder that can lead to repeated navigate/status/recover loops on Cloud Browser CDP sessions.
- Terminal
browser_script observenow waits through the requested observe window instead of returning on the first partial stream event. If a navigation or extraction script emits partial page events and then finishes shortly after, the model receives the final result in the same tool call rather than spending extra LLM turns polling the samerun_id. - Terminal
browser_scriptstart now auto-collects a previous active run that has already finished or timed out. If the model starts another browser action immediately after navigation completed in the background, the tool returns the completed navigation/page result in that same call instead of forcing a separate observe/status/recover turn. - Terminal remote-CDP attach now reuses an existing ordinary blank page target
before creating a new
about:blanktab. Browser Use Cloud sessions therefore begin with one stable controlled tab instead of splitting Python setup and Rust execution across separate blank targets. - Eval payloads preserve Rust/browser-use usage fields and add dashboard
aliases (
input_tokens,output_tokens, cached/cache-creation tokens, andcost_usd) before saving. This keeps cost/token displays working for Rust histories without changing the canonical browser-use usage structure. - Browser-use Rust CDP initial navigation now waits for a concrete post-navigation browser state summary and passes the observed current URL/title into the Rust task context. Start-URL tasks should begin by inspecting or extracting from the already-loaded page instead of spending early turns on repeated navigation/status recovery.
- Browser-use Rust direct CDP pre-navigation now only records an initial
navigation as completed when the observed browser state matches the
requested URL. If the Cloud Browser remains on
about:blankor another mismatched target, the original navigation stays in the Rust task context instead of giving the model a false "already loaded" state. - Eval GitHub runners now preserve visible output for interrupted Rust runs: if an agent is cancelled or errors after collecting history, formatting synthesizes a partial final answer from the latest memory/tool evidence; if no history exists, the saved dashboard payload still includes a non-empty failure response with the stage error.
- Rust
token_countusage reconstruction now treats summed per-turnlast_token_usageas the billed usage source when it exceeds the latest cumulative context counters. Dashboard usage therefore reflects the whole agent run instead of only the latest context-sized prompt after recompute/compaction paths. - Rust cost-enabled usage reconstruction now reports token totals from the same summed per-turn usage entries used to calculate costs. This keeps dashboard token counts, cached/cache-creation buckets, and cost fields on one basis instead of showing summed cost beside latest-context token counters.
- Rust cost-enabled usage reconstruction now treats
token_countas a fallback billing source when providermodel.usageevents are absent. Mixed streams therefore do not double-count usage by pricing both provider usage and context occupancy counters. - Terminal SDK JSON-RPC responses are now bounded before they are written to stdio. Oversized final histories compact durable event payloads while preserving final output, success state, errors, usage, files, and recent events, so a completed run cannot lose its final answer because the Python wrapper hits its newline-delimited frame limit.
- Browser-use Rust SDK runs now prefer live
agent.eventnotifications when the final SDK response is missing, truncated, or smaller than the already observed stream. If a run is cancelled or the final transport fails aftersession.done, Python reconstructsAgentHistoryList, usage, and final output from the notification stream instead of returning an empty result. - Browser-use Rust SDK notification recovery now accepts both top-level
event_typerecords and nested SDK event envelopes. Runs that visibly emitsession.donein GitHub runner progress logs therefore reconstruct the final answer even if the final JSON-RPC history response is empty or compacted differently. - Browser-use Rust SDK notification recovery now normalizes both
agent.eventandagent.projected_eventnotifications into the retained event history. Runs whose progress logs show projectedsession.doneoragent.completedevents therefore no longer fall back to "Rust terminal session did not produce a final result." - Browser-use Rust SDK history reconstruction now prefers retained live
notifications when the final JSON-RPC response history is present but lacks
a final result. Stale response-side compaction errors no longer override a
session.doneevent that Python already observed from the stream. - Browser-use Rust SDK billing reconstruction now falls back to the SDK
response's aggregate
history.usagewhen compacted histories omit usage events. This keeps dashboard usage nonzero even when the event payload is compacted before it reaches Python. - Terminal provider overload errors are treated as retryable transient
capacity failures. This prevents a single Claude/OpenAI
server overloadedresponse from becoming an immediate no-output eval failure. - Terminal
browser_script observenow defaults to a coarse 30 second wait, clamps too-small observe requests up to that window, and allows waits up to 120 seconds. Long navigation/extraction scripts therefore spend fewer model turns polling the samerun_idand are less likely to hit the step limit before finalizing partial evidence. - Terminal now includes a locally executed DuckDuckGo Lite
searchtool from the terminal web-search merge. URL-finding and general web-search tasks can discover candidate pages without spending browser-navigation turns on search engine result pages. - Terminal SDK histories now return child-agent events separately from parent
events and expose a combined
usage_eventsstream. Browser-use prices and traces against that combined stream, so sub-agent model calls contribute to run token/cost totals without letting childsession.doneevents override the parent final answer. browser_use.beta.Agentno longer treats_run_processmonkeypatches as an alternate runtime. Productionrun()andfollow_up()now require the Rust SDK server path, so the legacy terminal CLI adapter cannot silently replace the new server protocol while the normal Pythonbrowser_use.Agentremains unchanged.- Terminal SDK browser requests now preserve and forward additional
Browser Use-style browser settings: proxy country, local profile label,
allowed/blocked domains, window size, and state directory. For cloud
browser runs, proxy country is passed to
browser remote start, so Rust SDK sessions can use Browser Use Cloud browser proxies through the same Python-facingbrowser_profile/browser_sessionoptions. - Browser-use now deduplicates SDK response/projected events before
reconstructing history and usage. Duplicate
token_countandsession.donerecords no longer inflate eval history, dashboard usage, or telemetry payloads when the terminal server returns both live and projected event streams. - Browser-use no longer sends a Rust SDK
tool_allowlistoverride. The terminal SDK server now owns its tool registry forbrowser_use.beta.Agentruns, so local search, web/search helpers, and v2 sub-agent controls are available in evals when the Rust core registers them. - Reconstructed browser history ignores internal browser-control endpoint
URLs such as
http://127.0.0.1:<port>while still preserving genuine local page URLs. Eval traces therefore show user-visible pages instead of CDP helper endpoints. - The Rust Agent default LLM resolution follows the normal browser-use path:
explicit
DEFAULT_LLMwins, otherwiseChatBrowserUse()is used. Real eval runs still pass Claude Sonnet 4.6 explicitly through the SDK request. - Terminal
http_get_many(...)andbrowser_fetch(..., return_error=True)now return recoverable error records that are both dict-compatible and response-compatible (.status_code,.status,.headers,.text,.content,.url). General browser-script code can inspect a failed helper response without crashing the whole step withAttributeError: 'dict' object has no attribute 'status_code'. - Terminal
js(...)inbrowser_scriptnow tolerates common anonymous function snippets and async function-IIFE snippets emitted by agents. This avoids repeated generic JavaScript syntax failures such asFunction statements require a function nameandawait is only valid in async functionswithout adding task- or domain-specific behavior. - Terminal SDK
agent.createnow persists the initial task input only through the runtime observed-event path. SDK LLM prompts and Laminar spans therefore contain the task once instead of duplicating the same initial user message, which improves trace fidelity and reduces avoidable prompt/cache churn.
Current Proof
- terminal
cargo check -p browser-use-cli - terminal
cargo test -p browser-use-cli sdk_ -- --nocapture - terminal
cargo test -p browser-use-cli sdk_run_runtime_supports_model_transport_blocking_bridge -- --nocapture - terminal
cargo test -p browser-use-agent running_browser_script -- --nocapture - terminal
cargo test -p browser-use-agent runtime_browser_backend_records_script_lifecycle -- --nocapture - terminal
cargo test -p browser-use-browser browser_script_list_tabs_hides_agent_startup_placeholder -- --nocapture - terminal
cargo test -p browser-use-agent subagent_tools -- --nocapture - terminal
cargo test -p browser-use-agent spawn_agent_agent_type_guidance_discourages_default_override -- --nocapture - terminal
cargo test -p browser-use-providers anthropic_messages_downsamples_oversized_tool_images -- --nocapture - terminal
cargo test -p browser-use-providers anthropic_messages -- --nocapture - terminal
cargo test -p browser-use-llm build_body_downsamples_oversized_inline_media_for_anthropic -- --nocapture - terminal
cargo test -p browser-use-llm anthropic_messages -- --nocapture - terminal
cargo test -p browser-use-browser browser_script_navigation_helpers_wait_for_page_state -- --nocapture - terminal
cargo test -p browser-use-browser browser_script_start_observe_finishes_slow_scripts -- --test-threads=1 --nocapture - terminal
cargo test -p browser-use-browser browser_script_observe_waits_for_completion_after_partial_output --lib - terminal
cargo test -p browser-use-browser browser_script_observe --lib - terminal
cargo test -p browser-use-browser browser_script_start_observe_finishes_slow_scripts --lib - terminal
cargo test -p browser-use-browser browser_script_start_ -- --nocapture - terminal
cargo test -p browser-use-browser browser_script_observe_is_idempotent_after_completion -- --nocapture - terminal
cargo test -p browser-use-browser remote_cdp_attach_reuses_existing_blank_page_before_creating_target -- --nocapture - terminal
cargo test -p browser-use-cli sdk_json_rpc_agent_run_task_executes_fake_backend_with_normalized_history -- --nocapture - terminal
cargo test -p browser-use-llm stream_ -- --nocapture - terminal
cargo test -p browser-use-cli sdk_provider_run_config_maps_browser_use_options_to_rust_core -- --nocapture - browser-use
uv run python -m py_compile browser_use/beta/service.py - browser-use
uv run pytest tests/ci/test_beta_agent.py -k 'browser_script_lifecycle_outputs_as_result or initial_actions_pre_navigate_existing_cdp_session or run_hands_off_completed_initial_navigation_as_context' -q - browser-use
uv run pytest tests/ci/test_beta_agent.py -k 'printed_browser_script_page_info_as_state or browser_script_lifecycle_outputs_as_result or initial_actions_pre_navigate_existing_cdp_session or run_hands_off_completed_initial_navigation_as_context' -q - browser-use
uv run pytest tests/ci/test_beta_agent.py::test_rust_history_surfaces_running_browser_script_observe_instruction -q - browser-use
uv run pytest tests/ci/test_beta_agent.py -k 'initial_actions_pre_navigate_existing_cdp_session or run_executes_initial_actions_before_sdk or run_hands_off_completed_initial_navigation_as_context' -q - browser-use
uv run pytest tests/ci/test_beta_agent.py - browser-use
uv run pytest tests/ci/test_beta_agent.py -k 'sdk_client_reads_large_json_rpc_lines or sdk_and_reuses_session or translates_browser_use_args_to_terminal' - browser-use
uv run pytest tests/ci/test_beta_agent.py -k 'sdk_client_queues_agent_notifications_before_response or sdk_client_reads_large_json_rpc_lines or sdk_and_reuses_session or translates_browser_use_args_to_terminal' - browser-use
uv run pytest tests/ci/test_beta_agent.py -k 'rust_sdk_client_reads_large_json_rpc_lines or rust_sdk_client_queues_agent_notifications_before_response' -q - browser-use
uv run pytest tests/ci/test_beta_agent.py::test_rust_sdk_client_reads_large_json_rpc_lines tests/ci/test_beta_agent.py::test_beta_agent_run_leaves_initial_navigation_for_sdk_by_default tests/ci/test_beta_agent.py::test_beta_agent_initial_actions_can_pre_navigate_existing_cdp_session tests/ci/test_beta_agent.py::test_beta_agent_translates_browser_use_args_to_terminal -q - browser-use
PYTHONPATH=. uv run pytest tests/ci/test_beta_agent.py -q -k 'pre_navigates_cdp_session_before_sdk_by_default or initial_actions_can_pre_navigate_existing_cdp_session or direct_initial_navigation_defaults_on_for_cdp or direct_initial_navigation_can_be_disabled' - browser-use
uv run pytest -q tests/ci/test_beta_agent.py -k "pre_navigates_cdp_session_before_sdk_by_default or keeps_initial_navigation_when_direct_state_mismatches or initial_actions_can_pre_navigate_existing_cdp_session or direct_initial_navigation_defaults_on_for_cdp" - browser-use
uv run pytest -q tests/ci/test_beta_agent.py -k "terminal_token_count_usage or sums_token_count_last_usage_when_latest_total_underreports or terminal_usage_prices_token_count_events or terminal_usage_sums_token_count_cache_creation" - browser-use
uv run pytest -q tests/ci/test_beta_agent.py -k "terminal_nested_model_usage or token_count_does_not_shrink_model_usage_totals or terminal_usage_prices_token_count_events or terminal_usage_prices_anthropic_raw_cache_reads or terminal_usage_sums_token_count_cache_creation or priced_summary_sums_cache_read_tokens or mixed_events_do_not_shrink_totals or priced_usage_prefers_model_usage_over_token_count or sums_token_count_last_usage_when_latest_total_underreports" - browser-use
python -m py_compile browser_use/beta/service.py - terminal
cargo test -p browser-use-cli sdk_transport -- --nocapture - terminal
cargo test -p browser-use-providers server_overloaded -- --nocapture - terminal
cargo test -p browser-use-agent observe_timeout -- --nocapture - terminal
cargo test -p browser-use-agent observe_routes_to_observe_script -- --nocapture - terminal
cargo test -p browser-use-cli sdk_ -- --nocapture - terminal
cargo test -p browser-use-agent subagent_tools_are_registered_in_the_dispatcher -- --nocapture - terminal
cargo test -p browser-use-cli sdk_run_attaches_child_agent_runner_to_provider_config -- --nocapture - terminal
cargo test -p browser-use-agent search -- --nocapture - terminal
cargo test -p browser-use-agent dispatcher -- --nocapture - terminal
cargo test -p browser-use-cli sdk_json_rpc_agent_run_returns_child_usage_events_separately -- --nocapture - browser-use
uv run pytest tests/ci/test_beta_agent.py::test_beta_agent_prices_sdk_child_usage_events_without_overriding_parent_result -q - browser-use
uv run pytest tests/ci/test_beta_agent.py::test_beta_agent_runs_through_sdk_and_reuses_session_for_followup tests/ci/test_beta_agent.py::test_beta_agent_recovers_final_result_from_sdk_notifications_after_transport_error tests/ci/test_beta_agent.py::test_beta_agent_preserves_sdk_notification_history_on_cancel -q - browser-use
uv run pytest tests/ci/test_beta_agent.py::test_beta_agent_recovers_final_result_from_sdk_notifications_after_transport_error tests/ci/test_beta_agent.py::test_beta_agent_recovers_nested_sdk_notification_events tests/ci/test_beta_agent.py::test_beta_agent_prices_sdk_child_usage_events_without_overriding_parent_result -q - browser-use
uv run pytest tests/ci/test_beta_agent.py::test_beta_agent_recovers_nested_sdk_notification_events tests/ci/test_beta_agent.py::test_beta_agent_prefers_notification_final_when_response_history_lacks_result tests/ci/test_beta_agent.py::test_beta_agent_uses_sdk_history_usage_when_events_do_not_include_usage tests/ci/test_beta_agent.py::test_beta_agent_prices_sdk_child_usage_events_without_overriding_parent_result -q - browser-use
uv run pytest tests/ci/test_beta_agent.py::test_beta_agent_recovers_final_result_from_sdk_notifications_after_transport_error tests/ci/test_beta_agent.py::test_beta_agent_recovers_nested_sdk_notification_events tests/ci/test_beta_agent.py::test_beta_agent_recovers_projected_sdk_final_events tests/ci/test_beta_agent.py::test_beta_agent_prefers_notification_final_when_response_history_lacks_result tests/ci/test_beta_agent.py::test_beta_agent_uses_sdk_history_usage_when_events_do_not_include_usage tests/ci/test_beta_agent.py::test_beta_agent_prices_sdk_child_usage_events_without_overriding_parent_result -q - browser-use
uv run pytest tests/ci/test_beta_agent.py::test_beta_agent_runs_through_sdk_and_reuses_session_for_followup -q - terminal
cargo fmt --check - terminal
cargo test -p browser-use-cli sdk_json_rpc_browser_create_preserves_browser_use_settings -- --nocapture - terminal
cargo test -p browser-use-cli sdk_provider_run_config_maps_browser_use_options_to_rust_core -- --nocapture - terminal
cargo test -p browser-use-agent stored_cloud_profile_uses_sdk_proxy_country_env_when_connecting -- --nocapture - browser-use
uv run pytest tests/ci/test_beta_agent.py::test_beta_agent_sdk_browser_payload_includes_profile_domains_window_and_proxy -q - browser-use
uv run python -m py_compile browser_use/beta/service.py - browser-use
uv run pytest tests/ci/test_beta_agent.py::test_rust_sdk_event_dedupe_removes_projected_usage_duplicates tests/ci/test_beta_agent.py::test_rust_history_ignores_internal_browser_connection_url tests/ci/test_beta_agent.py::test_beta_agent_default_llm_matches_browser_use_default tests/ci/test_beta_agent.py::test_beta_agent_default_llm_respects_default_llm_env tests/ci/test_beta_agent.py::test_beta_agent_exposes_logging_helper_methods tests/ci/test_beta_agent.py::test_beta_agent_telemetry_filters_empty_reconstructed_urls -q - browser-use
uv run pytest tests/ci/test_beta_agent.py::test_beta_agent_translates_browser_use_args_to_terminal tests/ci/test_beta_agent.py::test_beta_agent_sdk_params_leave_terminal_tools_unrestricted -q - browser-use
uv run pytest tests/ci/test_beta_agent.py -q - browser-use
uv run ruff check browser_use/beta/service.py tests/ci/test_beta_agent.py tests/ci/models/test_llm_model_factory.py - terminal
cargo test -p browser-use-browser browser_script_http_get_many_preserves_order_and_errors -- --nocapture - terminal
cargo test -p browser-use-browser browser_script_browser_fetch_single_returns_structured_errors_by_default -- --nocapture - terminal
cargo test -p browser-use-browser browser_script_js_accepts_anonymous_function_snippets -- --nocapture - terminal
cargo test -p browser-use-browser browser_script_js_asyncifies_parenthesized_function_iife_with_await -- --nocapture - terminal
cargo fmt --check - terminal
cargo test -p browser-use-cli sdk_run_attaches_child_agent_runner_to_provider_config -- --nocapture - terminal
cargo test -p browser-use-agent subagent_tools_are_registered_in_the_dispatcher -- --nocapture - terminal
cargo test -p browser-use-cli sdk_json_rpc_agent_run_executes_fake_backend -- --nocapture - terminal
cargo test -p browser-use-cli sdk_json_rpc_agent_run_task_executes_fake_backend_with_normalized_history -- --nocapture - evaluations-internal
uv run python -m py_compile eval/service.py - evaluations-internal
python -m py_compile eval/task_types.py - evaluations-internal
PYTHONPATH=. uv run pytest tests/test_service_cli.py -q -k 'usage_aliases or trims_oversized_history_fields or rust_eval_uses_adapter_initial_navigation_default or rust_eval_preserves_explicit_direct_initial_navigation_override' - evaluations-internal
PYTHONPATH=. uv run pytest -q tests/test_service_cli.py -k "synthesizes_partial_result_on_timeout or synthesizes_partial_result_without_timeout_marker or server_payload_includes_failure_final_response_without_history" - evaluations-internal
PYTHONPATH="$PWD" uv run pytest tests/test_service_cli.py::test_progress_updates_tolerate_transient_failures_by_default tests/test_service_cli.py::test_run_stage_with_progress_heartbeat_refreshes_active_stage -q - browser-use process-backed smoke with
BROWSER_USE_TERMINAL_BINARY=/home/exedev/Developer/terminal/target/debug/browser-use-terminal, provingAgent.run()calls the real SDK server andAgent.follow_up()reuses the same SDK session. - real_v8 5-task eval smoke
kh721gr6v248emmdw9kn4mf645882qdkon browser-usead74b9f23da5c3ec1773b57dce856df9467777c5and terminalb59372cc03b574e1bb82d9ad814ebb6c2d79bd1c: 5/5 completed, scores 80/90/100/100/100 with Agent SDK judge and Browser Use Cloud CDP browser. - real_v8 50-task eval
kh7749jfyd5x54n5wzt4cqezmh883tgpon the same browser-use SHA and terminalb59372cc03b574e1bb82d9ad814ebb6c2d79bd1ccompleted all 50 task rows but scored below target; repeated low-output rows exposed the terminal fetch error-record compatibility issue fixed by terminal5382d8b. - real_v8 50-task eval
kh7530fhjr52h81b0fvx7fhem5882ty1on browser-use7bcf9754f103a1bb6c2e6d031940a162bb4adfbeand terminal5382d8b7ccc72c102fbeb2b68940177e5371d753was still running when inspected, with 38/50 rows saved, no empty final responses in fetched full histories, no access-denied count, and repeated recoverable browser-script JS syntax errors that motivated terminalaa3f3ea. - final real_v8 50-task eval
kh7br0crtahkq408f9dw41z901883qy5was dispatched on browser-use7bcf9754f103a1bb6c2e6d031940a162bb4adfbeand terminalaa3f3ea78d45564ea0e5f5443e4f13145e5ca9a5with Browser Use Cloud CDP browser and Agent SDK judge. It completed all 50 rows with Agent SDK judging, averaged 78.5, saved partial final responses for the six timeboxed zero-score rows, and is tagged aseval/kh7br-real-v8-50-78p50in both browser-use and terminal. - real_v8 50-task eval
kh7880wm0ffgsyqkwzfwn9hc7s882mffwas dispatched on browser-use5c40474473a61651f25cd2d084aa1fc278c5c714and terminal640e052ca5f8e8654069a414814ac2f061861ce2with Browser Use Cloud CDP browser, no--proxylessflag, a 30 minute task/agent timebox, and Agent SDK judge. GitHub runner logs show--browser browser-use-cloud, Browser Use Cloud session creation, and Rustbrowser_mode=remote-cdp. When inspected, 49 scored rows averaged 76.29 while one placeholder row was rerunning; low rows mostly had partial final responses after 30 minute cancellations rather than missing branch/CDP/judge/Laminar infrastructure. - Laminar trace inspection for
kh7880wm0ffgsyqkwzfwn9hc7s882mffconfirmed main-agentrust_core.llmspans, browser tool spans, usage attributes, and a duplicated initial task user message. Terminal06627d9fixes the duplicatedsession.inputpersistence and the focused SDK JSON-RPC tests above prove the initial task input is now stored exactly once for bothagent.runandagent.run_task.
Known Transitional Debt
- The production Rust path no longer uses
_run_process/_load_events, legacy process-backed SDK adapter code, or direct CLIrun-*command construction. The production wrapper now goes through the SDK server protocol.