wehub-resource-sync
ec2b666284
Continuous Integration / Pre-commit Linter (push) Has been cancelled
Continuous Integration / Mypy Check (Python 3.10) (push) Has been cancelled
Continuous Integration / Mypy Check (Python 3.11) (push) Has been cancelled
Continuous Integration / Mypy Check (Python 3.12) (push) Has been cancelled
Continuous Integration / Mypy Check (Python 3.13) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.10) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.11) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.12) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.13) (push) Has been cancelled
Continuous Integration / Unit Tests (Python 3.14) (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.10) (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.11) (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.12) (push) Has been cancelled
Copybara PR Handler / close-imported-pr (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.13) (push) Has been cancelled
Continuous Integration / A2A v0.3 Tests (Python 3.14) (push) Has been cancelled
68 lines
2.2 KiB
Markdown
68 lines
2.2 KiB
Markdown
# Instructions
|
|
|
|
## Run Evaluation
|
|
|
|
1. Set environment variables in your terminal:
|
|
|
|
```shell
|
|
export GOOGLE_GENAI_USE_ENTERPRISE=FALSE
|
|
export GOOGLE_API_KEY=<your_api_key>
|
|
export GOOGLE_CLOUD_PROJECT=<your_bigquery_project>
|
|
```
|
|
1. Change to the current directory:
|
|
|
|
```shell
|
|
cd tests/integration/fixture/bigquery_agent/
|
|
```
|
|
1. Customize the evaluation dataset to the environment `GOOGLE_CLOUD_PROJECT`
|
|
by replacing the placeholder to the real project set in your environment:
|
|
|
|
```shell
|
|
sed -e "s:\${GOOGLE_CLOUD_PROJECT}:${GOOGLE_CLOUD_PROJECT}:g" simple.test.json -i
|
|
```
|
|
1. Run the following command as per https://google.github.io/adk-docs/evaluate/#3-adk-eval-run-evaluations-via-the-cli:
|
|
|
|
```shell
|
|
adk eval . simple.test.json --config_file_path=test_config.json
|
|
```
|
|
|
|
If it fails, re-run with `--print_detailed_results` flag to see more details
|
|
on turn-by-turn evaluation.
|
|
|
|
## Generate Evaluation dataset
|
|
|
|
1. Set environment variables in your terminal:
|
|
|
|
```shell
|
|
export GOOGLE_GENAI_USE_ENTERPRISE=FALSE
|
|
export GOOGLE_API_KEY=<your_api_key>
|
|
export GOOGLE_CLOUD_PROJECT=<your_bigquery_project>
|
|
```
|
|
1. Set up google [application default credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc)
|
|
on your machine.
|
|
|
|
```shell
|
|
gcloud auth application-default login
|
|
```
|
|
1. Change to the directory containing agent folder:
|
|
|
|
```shell
|
|
cd tests/integration/fixture/
|
|
```
|
|
1. Run the following command to start the ADK web app:
|
|
|
|
```shell
|
|
adk web
|
|
```
|
|
1. Open the ADK web UI in your browser http://127.0.0.1:8000/dev-ui/?app=bigquery_agent.
|
|
1. Create an evaluation dataset by following [these steps](https://google.github.io/adk-docs/evaluate/#1-adk-web-run-evaluations-via-the-web-ui).
|
|
This would generate file `bigquery_agent/simple.evalset.json`.
|
|
1. Note that this evaluation data would be tied to the agent interaction in the
|
|
`GOOGLE_CLOUD_PROJECT` set in your environment. To normalize it by replacing
|
|
the real project set in your environment to a placeholder, let's run the
|
|
following command:
|
|
|
|
```shell
|
|
sed -e "s:${GOOGLE_CLOUD_PROJECT}:\${GOOGLE_CLOUD_PROJECT}:g" bigquery_agent/simple.evalset.json > bigquery_agent/simple.test.json
|
|
```
|