chore: import upstream snapshot with attribution

examples/google-live/README.md

@@ -0,0 +1,137 @@
+# google-live (Google Live API with Gemini)
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example google-live
+cd google-live
+```
+
+This example demonstrates how to use promptfoo with Google's WebSocket-based Live API, which enables low-latency bidirectional interactions with Gemini models. The example includes four different configurations:
+
+1. Basic query demonstration
+2. Multiline conversation demonstration
+3. Function calling and Google Search demonstration
+4. Stateful API with Python backend
+
+## Prerequisites
+
+- promptfoo CLI installed (`npm install -g promptfoo` or `brew install promptfoo`)
+- Google AI Studio API key set as `GOOGLE_API_KEY`
+- For the stateful API example: Python 3 with Flask installed (`pip install flask`)
+
+You can obtain a Google AI Studio API key from the [Google AI Studio website](https://ai.google.dev/).
+
+## Running the Examples
+
+You can initialize this example with:
+
+```bash
+npx promptfoo@latest init --example google-live
+```
+
+This will create a directory with all necessary configuration files. After running any evaluation, you can view the results by running:
+
+```bash
+promptfoo view
+```
+
+### Basic Query Example
+
+The basic configuration in `promptfooconfig.yaml` demonstrates a simple query to the Gemini model:
+
+```bash
+promptfoo eval -c promptfooconfig.yaml -j 3
+```
+
+> Note: Rate limits of 3 concurrent sessions per API key apply, which is why we use `-j 3` to limit concurrency.
+
+### Multiline Conversation Example
+
+The multiline configuration in `promptfooconfig.multiline.yaml` demonstrates a multi-turn conversation:
+
+```bash
+promptfoo eval -c promptfooconfig.multiline.yaml -j 3
+```
+
+### Function Calling and Tools Example
+
+The tools configuration in `promptfooconfig.tools.yaml` demonstrates function calling (where the model can invoke defined functions) and built-in Google Search:
+
+```bash
+promptfoo eval -c promptfooconfig.tools.yaml -j 3
+```
+
+### Stateful API Example
+
+This example runs a local Python API that maintains state between function calls:
+
+```bash
+promptfoo eval -c promptfooconfig.statefulapi.yaml -j 3
+```
+
+**Setup:**
+
+- Requires Python 3 with Flask (`pip install flask`)
+- Uses `python3` command by default
+- Custom Python path: Set `pythonExecutable` in config or use `PROMPTFOO_PYTHON` environment variable
+- Runs on port 8765 (configured in both the Python file and YAML config)
+
+If you encounter errors, check that:
+
+- Flask is properly installed
+- Port 8765 is available (not in use by another application)
+- Python 3 is in your PATH or correctly configured
+
+## What This Example Demonstrates
+
+### 1. Basic Query
+
+- Simple interaction with the Live API
+- Basic assertion testing on model responses
+
+### 2. Multiline Conversation
+
+- Multi-turn conversations with the model
+- Using YAML files to structure conversations
+- Testing how the model maintains context across turns
+
+### 3. Function Calling and Tools
+
+- Custom function declarations (weather and stock price)
+- Built-in Google Search integration
+- Assertions on function calling behavior
+
+### 4. Stateful API Integration
+
+- Promptfoo spawns a local Python API that maintains state between calls
+- Demonstrates how to integrate an external service with LLM function calling
+- Shows how to test stateful interactions with assertions
+
+## Configuration Details
+
+### Provider Configuration
+
+All examples use the `google:live:gemini-3.1-flash-live-preview` model with various configurations:
+
+```yaml
+providers:
+  # Using Gemini 3.1 Flash Live Preview
+  - id: 'google:live:gemini-3.1-flash-live-preview'
+    config:
+      generationConfig:
+        response_modalities: ['audio']
+        outputAudioTranscription: {}
+      timeoutMs: 10000
+```
+
+### Tools Configuration
+
+The function calling examples use JSON configuration files that define:
+
+1. Custom function declarations
+2. The built-in Google Search capability
+
+For the stateful API, `counter_api.py` implements a simple counter service with endpoints for adding to and retrieving a count value.
+
+For more information about the Google Live API and other Google AI models, see the [Google AI documentation](/docs/providers/google).

examples/google-live/counter_api.py

@@ -0,0 +1,37 @@
+import flask
+
+app = flask.Flask(__name__)
+
+counter = 3
+
+
+@app.route("/add_one", methods=["POST"])
+def add_one():
+    global counter
+    counter += 1
+    return flask.jsonify({})
+
+
+@app.route("/get_count", methods=["GET"])
+def get_count():
+    return flask.jsonify({"counter": counter})
+
+
+# If you want to assert on the final state the app must include this method
+@app.route("/get_state", methods=["GET"])
+def get_state():
+    return flask.jsonify({"counter": counter})
+
+
+# If you want the app to be cleaned up it must include this method
+@app.route("/shutdown", methods=["POST"])
+def shutdown():
+    func = flask.request.environ.get("werkzeug.server.shutdown")
+    if func is None:
+        return flask.jsonify({"message": "Error: not running with the Werkzeug Server"})
+    func()
+    return flask.jsonify({"message": "Server shutting down..."})
+
+
+if __name__ == "__main__":
+    app.run(port=8765)

examples/google-live/prompt.yaml

@@ -0,0 +1,5 @@
+- role: user
+  content: '{{ first_message}}'
+
+- role: user
+  content: '{{ second_message }}'

examples/google-live/promptfooconfig.callback.js

@@ -0,0 +1,71 @@
+module.exports = /** @type {import('promptfoo').TestSuiteConfig} */ ({
+  description: 'Function calling and callback execution demonstration',
+  prompts: [
+    'Please add the following numbers together: {{a}} and {{b}}',
+    'What is the sum of {{a}} and {{b}}?',
+  ],
+  providers: [
+    {
+      // Using Gemini 3.1 Flash Live Preview
+      id: 'google:live:gemini-3.1-flash-live-preview',
+      config: {
+        generationConfig: {
+          response_modalities: ['audio'],
+          outputAudioTranscription: {},
+        },
+        timeoutMs: 10000,
+        tools: [
+          {
+            functionDeclarations: [
+              {
+                name: 'addNumbers',
+                description: 'Add two numbers together',
+                parameters: {
+                  type: 'object',
+                  properties: {
+                    a: { type: 'number' },
+                    b: { type: 'number' },
+                  },
+                  required: ['a', 'b'],
+                },
+              },
+            ],
+          },
+        ],
+        functionToolCallbacks: {
+          addNumbers: (parametersJsonString) => {
+            const { a, b } = JSON.parse(parametersJsonString);
+            return { sum: a + b };
+          },
+        },
+      },
+    },
+  ],
+  tests: [
+    {
+      vars: { a: 5, b: 6 },
+      assert: [
+        {
+          type: 'is-valid-function-call',
+        },
+        {
+          type: 'contains',
+          value: '11',
+          transform: 'output.text',
+        },
+      ],
+    },
+    {
+      vars: { a: 10, b: 20 },
+      assert: [
+        {
+          type: 'is-valid-function-call',
+        },
+        {
+          type: 'javascript',
+          value: "output.text.includes('30')",
+        },
+      ],
+    },
+  ],
+});

examples/google-live/promptfooconfig.multiline.yaml

@@ -0,0 +1,34 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+
+# Learn more about building a configuration: https://promptfoo.dev/docs/configuration/guide
+description: 'Multiline query demonstration'
+prompts:
+  - file://prompt.yaml
+providers:
+  # Using Gemini 3.1 Flash Live Preview
+  - id: 'google:live:gemini-3.1-flash-live-preview'
+    config:
+      generationConfig:
+        response_modalities: ['audio']
+        outputAudioTranscription: {}
+      timeoutMs: 10000
+tests:
+  - vars:
+      first_message: why is the the sea salty?
+      second_message: can you give a more brief explanation
+    assert:
+      - type: icontains
+        value: salt
+        transform: output.text
+      - type: icontains-any
+        value:
+          - weathering
+          - erosion
+          - rocks
+          - minerals
+          - rivers
+          - rain
+        transform: output.text
+  - vars:
+      first_message: hey
+      second_message: tell me about hawaii

examples/google-live/promptfooconfig.statefulapi.yaml

@@ -0,0 +1,28 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Example for mocking a stateful python api'
+
+prompts: '{{query}}'
+
+providers:
+  # Using Gemini 3.1 Flash Live Preview
+  - id: 'google:live:gemini-3.1-flash-live-preview'
+    config:
+      tools: file://tools.json
+      functionToolStatefulApi:
+        file: file://examples/google-live/counter_api.py
+        url: http://127.0.0.1:8765
+        # Optionally specify a custom Python executable path
+        # pythonExecutable: '/usr/local/bin/python3'
+      generationConfig:
+        response_modalities: ['audio']
+        outputAudioTranscription: {}
+      timeoutMs: 10000
+
+tests:
+  - vars:
+      query: Add to the counter until it reaches 5
+    assert:
+      - type: is-valid-function-call
+      - type: equals
+        value: { 'counter': 5 }
+        transform: output.statefulApiState

examples/google-live/promptfooconfig.tools.yaml

@@ -0,0 +1,47 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Function calling demonstration'
+
+prompts: '{{query}}'
+
+providers:
+  # Using Gemini 3.1 Flash Live Preview
+  - id: 'google:live:gemini-3.1-flash-live-preview'
+    config:
+      tools: file://tools.json
+      generationConfig:
+        response_modalities: ['audio']
+        outputAudioTranscription: {}
+      timeoutMs: 10000
+
+tests:
+  # Example of non-built-in tool that the user must execute
+  - vars:
+      query: What is the weather in {{location}}?
+      location: San Francisco
+    assert:
+      - type: is-valid-function-call
+      - type: equals
+        value: get_weather
+        transform: output.toolCall.functionCalls[0].name
+      - type: similar
+        value: '{{location}}'
+        threshold: 0.9
+        transform: output.toolCall.functionCalls[0].args.city
+  # Example of built-in google search tool that is executed by live
+  - vars:
+      query: google search for why the sea is salty
+    assert:
+      - type: icontains
+        value: salt
+        transform: output.text
+      - type: icontains-any
+        value:
+          - 'weathering'
+          - 'rainwater'
+          - 'erosion'
+          - 'eroded'
+          - 'rocks'
+          - 'minerals'
+          - 'rivers'
+          - 'rain'
+        transform: output.text

examples/google-live/promptfooconfig.yaml

@@ -0,0 +1,39 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+
+# Learn more about building a configuration: https://promptfoo.dev/docs/configuration/guide
+
+description: 'Basic query demonstration'
+
+prompts:
+  - Why is {{concept}}?
+
+providers:
+  # Using Gemini 3.1 Flash Live Preview
+  - id: 'google:live:gemini-3.1-flash-live-preview'
+    config:
+      generationConfig:
+        response_modalities: ['audio']
+        outputAudioTranscription: {}
+      timeoutMs: 10000
+
+defaultTest:
+  assert:
+    - type: icontains
+      value: salt
+      transform: output.text
+    - type: icontains-any
+      value:
+        - weathering
+        - erosion
+        - erodes
+        - eroded
+        - runoff
+        - rocks
+        - minerals
+        - rivers
+        - rain
+      transform: output.text
+
+tests:
+  - vars:
+      concept: the sea salty

examples/google-live/tools.json

@@ -0,0 +1,54 @@
+[
+  {
+    "functionDeclarations": [
+      {
+        "name": "get_weather",
+        "description": "Get current weather information for a city",
+        "parameters": {
+          "type": "OBJECT",
+          "properties": {
+            "city": {
+              "type": "STRING",
+              "description": "The name of the city to get weather for"
+            }
+          },
+          "required": ["city"]
+        }
+      },
+      {
+        "name": "get_stock_price",
+        "description": "Get current stock price and related information for a given stock symbol",
+        "parameters": {
+          "type": "OBJECT",
+          "properties": {
+            "symbol": {
+              "type": "STRING",
+              "description": "The stock symbol to get price information for (e.g., AAPL, GOOGL)"
+            }
+          },
+          "required": ["symbol"]
+        }
+      },
+      {
+        "name": "add_one",
+        "description": "add one to counter"
+      },
+      {
+        "name": "get_count",
+        "description": "return the current value of the counter",
+        "response": {
+          "type": "OBJECT",
+          "properties": {
+            "counter": {
+              "type": "INTEGER",
+              "description": "value of counter"
+            }
+          }
+        }
+      }
+    ]
+  },
+  {
+    "googleSearch": {}
+  }
+]