chore: import upstream snapshot with attribution

examples/google-vertex/README.md

@@ -0,0 +1,93 @@
+# google-vertex (Google Vertex AI Examples)
+
+Example configurations for testing Google Vertex AI models with promptfoo.
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example google-vertex
+cd google-vertex
+```
+
+## Purpose
+
+- Test Vertex AI's Gemini, Claude, and Llama models
+- Configure model-specific features and search grounding
+- Compare performance across different tasks
+
+## Prerequisites
+
+- Google Cloud account with Vertex AI API enabled
+- API credentials
+- Node.js ^20.20.0 or >=22.22.0 (Node.js 20 support ends July 30, 2026; Node.js 24 LTS recommended)
+
+## Environment Variables
+
+- `VERTEX_PROJECT_ID` - Your Google Cloud project ID
+- `GOOGLE_APPLICATION_CREDENTIALS` - Path to service account credentials (optional)
+
+## Setup
+
+1. Install dependencies:
+
+   ```sh
+   npm install google-auth-library
+   ```
+
+2. Configure authentication:
+
+   ```sh
+   # User account (development)
+   gcloud auth application-default login
+
+   # Or service account
+   export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json
+   ```
+
+3. Set your project ID:
+   ```sh
+   export VERTEX_PROJECT_ID=your-project-id
+   ```
+
+## Configurations
+
+This example includes:
+
+- `promptfooconfig.gemini.yaml`: Gemini models with function calling, system instructions, and safety settings
+- `promptfooconfig.claude.yaml`: Claude models for technical writing and code analysis
+- `promptfooconfig.llama.yaml`: Llama models with safety features and region configuration
+- `promptfooconfig.search.yaml`: Search grounding for real-time information
+- `promptfooconfig.response-schema.yaml`: Response schemas with structured output
+
+## Running Examples
+
+```sh
+# Basic example
+promptfoo eval -c promptfooconfig.yaml
+
+# Model-specific examples
+promptfoo eval -c promptfooconfig.gemini.yaml
+promptfoo eval -c promptfooconfig.claude.yaml
+promptfoo eval -c promptfooconfig.llama.yaml
+
+# Search grounding tool and image understanding
+promptfoo eval -c promptfooconfig.search.yaml
+promptfoo eval -c promptfooconfig.image.yaml
+
+# Structured output with response schemas
+promptfoo eval -c promptfooconfig.response-schema.yaml
+
+# View results
+promptfoo view
+```
+
+## Expected Results
+
+Each configuration demonstrates different model capabilities, from function calling and tool use to safety features and real-time information retrieval.
+
+## Learn More
+
+- [Vertex AI Provider Documentation](https://www.promptfoo.dev/docs/providers/vertex/)
+- [Google Cloud Vertex AI Documentation](https://cloud.google.com/vertex-ai/docs)
+- [Google documentation on Grounding with Google Search](https://ai.google.dev/docs/gemini_api/grounding)
+- [Google documentation on Image Understanding](https://ai.google.dev/gemini-api/docs/image-understanding#inline-image)

examples/google-vertex/promptfooconfig.claude.yaml

@@ -0,0 +1,48 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Vertex Claude Agent SDK Analysis'
+
+prompts:
+  - 'Analyze this code and list any potential issues: ```{{code}}```'
+
+providers:
+  - id: vertex:claude-opus-4-6
+    label: Claude 4.6 Opus
+    config:
+      region: us-east5
+  - id: vertex:claude-sonnet-4-6
+    label: Claude Sonnet 4.6
+    config:
+      region: us-east5
+  - id: vertex:claude-haiku-4-5@20251001
+    label: Claude 4.5 Haiku
+    config:
+      region: us-east5
+  - id: vertex:claude-opus-4-1@20250805
+    label: Claude 4.1 Opus
+    config:
+      region: us-east5
+  - id: vertex:claude-opus-4@20250514
+    label: Claude 4 Opus
+    config:
+      region: us-east5
+tests:
+  - vars:
+      code: |
+        def calculate_average(numbers):
+            total = 0
+            for num in numbers:
+                total += num
+            return total / len(numbers)
+  - vars:
+      code: |
+        def fibonacci(n):
+            if n <= 1:
+                return n
+            return fibonacci(n-1) + fibonacci(n-2)
+  - vars:
+      code: |
+        def search_list(items, target):
+            for i in range(len(items)):
+                if items[i] == target:
+                    return i
+            return -1

examples/google-vertex/promptfooconfig.gemini.yaml

@@ -0,0 +1,278 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+
+description: 'Evaluation of Gemini 2.5 models with function calling capabilities'
+
+prompts:
+  - label: 'Direct Question'
+    raw: |
+      What's the weather like in {{city}} and what are some good restaurants there?
+
+  - label: 'Detailed Query'
+    raw: |
+      I'm planning to visit {{city}}. Could you help me understand:
+      1. The current weather conditions
+      2. Some recommended places to eat, focusing on {{cuisine}} restaurants
+
+providers:
+  - id: vertex:gemini-2.5-pro
+    config:
+      generationConfig:
+        temperature: 0.7
+        maxOutputTokens: 1024
+      toolConfig:
+        functionCallingConfig:
+          mode: 'AUTO'
+      tools:
+        - functionDeclarations:
+            - name: 'get_weather'
+              description: 'Get weather information for a location'
+              parameters:
+                type: 'OBJECT'
+                properties:
+                  location:
+                    type: 'STRING'
+                    description: 'City name'
+                required: ['location']
+            - name: 'search_places'
+              description: 'Search for places of interest'
+              parameters:
+                type: 'OBJECT'
+                properties:
+                  query:
+                    type: 'STRING'
+                    description: 'Search query'
+                  type:
+                    type: 'STRING'
+                    description: 'Type of place'
+                    enum: ['restaurant', 'hotel', 'attraction']
+                required: ['query']
+
+  - id: vertex:gemini-3.5-flash
+    config:
+      region: global # Gemini 3.5 Flash is served on Vertex AI's global endpoint
+      apiHost: aiplatform.googleapis.com
+      generationConfig:
+        temperature: 0.7
+        maxOutputTokens: 2048
+      toolConfig:
+        functionCallingConfig:
+          mode: 'AUTO'
+      tools:
+        - functionDeclarations:
+            - name: 'get_weather'
+              description: 'Get weather information for a location'
+              parameters:
+                type: 'OBJECT'
+                properties:
+                  location:
+                    type: 'STRING'
+                    description: 'City name'
+                required: ['location']
+            - name: 'search_places'
+              description: 'Search for places of interest'
+              parameters:
+                type: 'OBJECT'
+                properties:
+                  query:
+                    type: 'STRING'
+                    description: 'Search query'
+                  type:
+                    type: 'STRING'
+                    description: 'Type of place'
+                    enum: ['restaurant', 'hotel', 'attraction']
+                required: ['query']
+
+  - id: vertex:gemini-2.5-flash
+    config:
+      generationConfig:
+        temperature: 0.7
+        maxOutputTokens: 2048
+      toolConfig:
+        functionCallingConfig:
+          mode: 'AUTO'
+      tools:
+        - functionDeclarations:
+            - name: 'get_weather'
+              description: 'Get weather information for a location'
+              parameters:
+                type: 'OBJECT'
+                properties:
+                  location:
+                    type: 'STRING'
+                    description: 'City name'
+                required: ['location']
+            - name: 'search_places'
+              description: 'Search for places of interest'
+              parameters:
+                type: 'OBJECT'
+                properties:
+                  query:
+                    type: 'STRING'
+                    description: 'Search query'
+                  type:
+                    type: 'STRING'
+                    description: 'Type of place'
+                    enum: ['restaurant', 'hotel', 'attraction']
+                required: ['query']
+
+  - id: vertex:gemini-2.5-flash-lite
+    config:
+      generationConfig:
+        temperature: 0
+        maxOutputTokens: 1024
+      toolConfig:
+        functionCallingConfig:
+          mode: 'AUTO'
+      tools:
+        - functionDeclarations:
+            - name: 'get_weather'
+              description: 'Get weather information for a location'
+              parameters:
+                type: 'OBJECT'
+                properties:
+                  location:
+                    type: 'STRING'
+                    description: 'City name'
+                required: ['location']
+            - name: 'search_places'
+              description: 'Search for places of interest'
+              parameters:
+                type: 'OBJECT'
+                properties:
+                  query:
+                    type: 'STRING'
+                    description: 'Search query'
+                  type:
+                    type: 'STRING'
+                    description: 'Type of place'
+                    enum: ['restaurant', 'hotel', 'attraction']
+                required: ['query']
+
+tests:
+  - description: Test with major city
+    vars:
+      city: 'San Francisco'
+      cuisine: 'Italian'
+    assert:
+      - type: contains-json
+        value: |
+          {
+            "type": "object",
+            "required": ["functionCall"],
+            "properties": {
+              "functionCall": {
+                "type": "object",
+                "required": ["name", "args"],
+                "properties": {
+                  "name": {
+                    "type": "string",
+                    "enum": ["get_weather"]
+                  },
+                  "args": {
+                    "type": "object",
+                    "required": ["location"],
+                    "properties": {
+                      "location": {
+                        "type": "string"
+                      }
+                    }
+                  }
+                }
+              }
+            }
+          }
+      - type: contains-json
+        value: |
+          {
+            "type": "object",
+            "required": ["functionCall"],
+            "properties": {
+              "functionCall": {
+                "type": "object",
+                "required": ["name", "args"],
+                "properties": {
+                  "name": {
+                    "type": "string",
+                    "enum": ["search_places"]
+                  },
+                  "args": {
+                    "type": "object",
+                    "required": ["query", "type"],
+                    "properties": {
+                      "query": {
+                        "type": "string"
+                      },
+                      "type": {
+                        "type": "string",
+                        "enum": ["restaurant"]
+                      }
+                    }
+                  }
+                }
+              }
+            }
+          }
+
+  - description: Test with smaller city
+    vars:
+      city: 'Boulder, Colorado'
+      cuisine: 'Mexican'
+    assert:
+      - type: contains-json
+        value: |
+          {
+            "type": "object",
+            "required": ["functionCall"],
+            "properties": {
+              "functionCall": {
+                "type": "object",
+                "required": ["name", "args"],
+                "properties": {
+                  "name": {
+                    "type": "string",
+                    "enum": ["get_weather"]
+                  },
+                  "args": {
+                    "type": "object",
+                    "required": ["location"],
+                    "properties": {
+                      "location": {
+                        "type": "string"
+                      }
+                    }
+                  }
+                }
+              }
+            }
+          }
+      - type: contains-json
+        value: |
+          {
+            "type": "object",
+            "required": ["functionCall"],
+            "properties": {
+              "functionCall": {
+                "type": "object",
+                "required": ["name", "args"],
+                "properties": {
+                  "name": {
+                    "type": "string",
+                    "enum": ["search_places"]
+                  },
+                  "args": {
+                    "type": "object",
+                    "required": ["query", "type"],
+                    "properties": {
+                      "query": {
+                        "type": "string"
+                      },
+                      "type": {
+                        "type": "string",
+                        "enum": ["restaurant"]
+                      }
+                    }
+                  }
+                }
+              }
+            }
+          }

examples/google-vertex/promptfooconfig.image.yaml

@@ -0,0 +1,17 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+
+description: Image captioning using Gemini
+
+prompts: |
+  {{imageFile}}
+  Describe what you see.
+
+providers:
+  - id: vertex:gemini-2.5-flash
+
+tests:
+  - vars:
+      imageFile: file://assets/red-panda.jpg
+    assert:
+      - type: contains
+        value: red panda

examples/google-vertex/promptfooconfig.llama.yaml

@@ -0,0 +1,64 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+
+description: 'Evaluation of Llama vision models for e-commerce product analysis'
+
+prompts:
+  - label: 'Product Description'
+    raw: |
+      Analyze this product image and provide a detailed description suitable for an e-commerce listing:
+      [Image]({{image_url}})
+
+  - label: 'Quality Check'
+    raw: |
+      Review this product image and identify any issues that would make it unsuitable for an e-commerce listing:
+      1. Image quality (blur, lighting, resolution)
+      2. Background and presentation
+      3. Product visibility and completeness
+      [Image]({{image_url}})
+
+providers:
+  - id: vertex:llama-3.2-90b-vision-instruct-maas
+    config:
+      generationConfig:
+        temperature: 0.7
+        maxOutputTokens: 1024
+      context: 'You are a professional e-commerce product photographer and quality assurance specialist.'
+      extra_body:
+        google:
+          model_safety_settings:
+            enabled: true
+            llama_guard_settings: {}
+
+tests:
+  - description: Test with good product image
+    vars:
+      image_url: 'https://storage.googleapis.com/vertex-ai-samples/product-images/coffee-mug-white.jpg'
+    assert:
+      - type: contains-all
+        value:
+          - 'resolution'
+          - 'lighting'
+          - 'background'
+      - type: contains-any
+        value:
+          - 'high quality'
+          - 'clear'
+          - 'well-lit'
+      - type: not-contains-any
+        value:
+          - 'blurry'
+          - 'poor quality'
+          - 'unclear'
+
+  - description: Test with poor quality image
+    vars:
+      image_url: 'https://storage.googleapis.com/vertex-ai-samples/product-images/coffee-mug-blurry.jpg'
+    assert:
+      - type: contains-any
+        value:
+          - 'blurry'
+          - 'poor lighting'
+          - 'low resolution'
+          - 'unclear'
+      - type: contains
+        value: 'improvement'

examples/google-vertex/promptfooconfig.response-schema.yaml

@@ -0,0 +1,32 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: 'Test Vertex AI responseSchema with file:// protocol'
+
+prompts:
+  - 'Write a tweet about {{topic}}'
+
+providers:
+  - id: vertex:gemini-2.5-flash
+    config:
+      region: us-central1
+      responseSchema: file://response-schema.json
+
+tests:
+  - vars:
+      topic: bananas
+    assert:
+      - type: icontains
+        value: 'banana'
+        transform: JSON.parse(output).tweet
+
+  - vars:
+      topic: artificial intelligence
+    assert:
+      - type: contains
+        value: 'AI'
+        transform: JSON.parse(output).tweet
+
+  - vars:
+      topic: climate change
+    assert:
+      - type: is-json
+        # This verifies the output is valid JSON that matches our schema

examples/google-vertex/promptfooconfig.search.yaml

@@ -0,0 +1,62 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+
+description: Google Vertex AI Search grounding demonstration
+
+prompts:
+  - '{{query}}'
+
+providers:
+  - id: vertex:gemini-2.5-flash
+    label: gemini-2.5-flash-search
+    config:
+      temperature: 0.2
+      maxOutputTokens: 1024
+      tools:
+        - googleSearch: {}
+  - id: vertex:gemini-2.5-flash
+    label: gemini-2.5-flash-search-thinking
+    config:
+      temperature: 0.2
+      maxOutputTokens: 1024
+      generationConfig:
+        thinkingConfig:
+          thinkingBudget: 1024
+      tools:
+        - googleSearch: {}
+
+tests:
+  # Test current events
+  - vars:
+      query: What is the current Google stock price?
+    assert:
+      - type: contains
+        value: stock
+      - type: contains
+        value: '$'
+
+  # Test a technical question
+  - vars:
+      query: What are the latest features in Python 3.13?
+    assert:
+      - type: contains
+        value: Python
+      - type: contains
+        value: feature
+
+  # Test a factual question
+  - vars:
+      query: Who won the latest Formula 1 Grand Prix?
+    assert:
+      - type: contains
+        value: Formula 1
+      - type: contains
+        value: won
+
+  # Test with weather information
+  - vars:
+      query: What is the current weather in Tokyo?
+    assert:
+      - type: contains
+        value: Tokyo
+      - type: contains
+        value: weather

examples/google-vertex/promptfooconfig.yaml

@@ -0,0 +1,108 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Google Vertex AI model evaluation and testing
+
+# This configuration runs each prompt through a series of example inputs and checks if they meet requirements.
+
+prompts:
+  - "You are a helpful assistant. Reply with a concise answer to this inquiry: '{{question}}'"
+
+providers:
+  - id: 'vertex:gemini-2.5-flash'
+    label: gemini-2.5-flash-cow
+    config:
+      systemInstruction:
+        parts:
+          - text: Always talk like a cow. Mooo!
+  - id: 'vertex:gemini-2.5-flash'
+    label: gemini-2.5-flash-sheep
+    config:
+      systemInstruction:
+        parts:
+          - text: Always talk like a sheep. Baaaah!
+
+defaultTest:
+  options:
+    provider:
+      # Text model used by model-graded assertions such as llm-rubric and factuality
+      text: vertex:gemini-2.5-pro
+      # Embedding model used by similar/answer-relevance assertions
+      embedding: vertex:embedding:gemini-embedding-001
+
+tests:
+  - vars:
+      question: 'Solve 3*x^3-5*x=1'
+  - vars:
+      question: What's the weather in New York?
+    assert:
+      - type: similar
+        value: 75 and sunny
+  - vars:
+      question: Who won the latest football match between the Giants and 49ers?
+  - vars:
+      question: "Which magazine was started first Arthur's Magazine or First for Women?"
+  - vars:
+      question: 'The Oberoi family is part of a hotel company that has a head office in what city?'
+  - vars:
+      question: 'Musician and satirist Allie Goertz wrote a song about the "The Simpsons" character Milhouse, who Matt Groening named after who?'
+  - vars:
+      question: "What nationality was James Henry Miller's wife?"
+  - vars:
+      question: 'Cadmium Chloride is slightly soluble in this chemical, it is also called what?'
+  - vars:
+      question: 'Which tennis player won more Grand Slam titles, Henri Leconte or Jonathan Stark?'
+  - vars:
+      question: "Which genus of moth in the world's seventh-largest country contains only one species?"
+  - vars:
+      question: 'Who was once considered the best kick boxer in the world, however he has been involved in a number of controversies relating to his "unsportsmanlike conducts" in the sport and crimes of violence outside of the ring.'
+  - vars:
+      question: 'The Dutch-Belgian television series that "House of Anubis" was based on first aired in what year?'
+  - vars:
+      question: 'What is the length of the track where the 2013 Liqui Moly Bathurst 12 Hour was staged?'
+  - vars:
+      question: 'Fast Cars, Danger, Fire and Knives includes guest appearances from which hip hop record executive?'
+  - vars:
+      question: 'Gunmen from Laredo starred which narrator of "Frontier"?'
+  - vars:
+      question: 'Where did the form of music played by Die Rhöner Säuwäntzt originate?'
+  - vars:
+      question: 'In which American football game was Malcolm Smith named Most Valuable player?'
+  - vars:
+      question: 'What U.S Highway gives access to Zilpo Road, and is also known as Midland Trail?'
+  - vars:
+      question: 'The 1988 American comedy film, The Great Outdoors, starred a four-time Academy Award nominee, who received a star on the Hollywood Walk of Fame in what year?'
+  - vars:
+      question: 'What are the names of the current members of  American heavy metal band who wrote the music for  Hurt Locker The Musical?'
+  - vars:
+      question: 'Human Error" is the season finale of the third season of a tv show that aired on what network?'
+  - vars:
+      question: 'Dua Lipa, an English singer, songwriter and model, the album spawned the number-one single "New Rules" is a song by English singer Dua Lipa from her eponymous debut studio album, released in what year?'
+  - vars:
+      question: 'American politician Joe Heck ran unsuccessfully against Democrat Catherine Cortez Masto, a woman who previously served as the 32nd Attorney General of where?'
+  - vars:
+      question: 'Which state does the drug stores, of which the CEO is Warren Bryant, are located?'
+  - vars:
+      question: 'Which  American politician did Donahue replaced '
+  - vars:
+      question: 'Which band was founded first, Hole, the rock band that Courtney Love was a frontwoman of, or The Wolfhounds?'
+  - vars:
+      question: 'How old is the female main protagonist of Catching Fire?'
+  - vars:
+      question: 'Chang Ucchin was born in korea during a time that ended with the conclusion of what?'
+  - vars:
+      question: 'Who is the director of the 2003 film which has scenes in it filmed at the Quality Cafe in Los Angeles?'
+  - vars:
+      question: "Which actress played the part of fictitious character Kimberly Ann Hart, in the franchise built around a live action superhero television series taking much of its footage from the Japanese tokusatsu 'Super Sentai'?"
+  - vars:
+      question: 'Who was born first, Pablo Trapero or Aleksander Ford?'
+  - vars:
+      question: "Are Jane and First for Women both women's magazines?"
+  - vars:
+      question: 'What profession does Nicholas Ray and Elia Kazan have in common?'
+  - vars:
+      question: 'Where is the company that purchased Aixam based in?'
+  - vars:
+      question: 'Which documentary is about Finnish rock groups, Adam Clayton Powell or The Saimaa Gesture?'
+  - vars:
+      question: 'Who was inducted into the Rock and Roll Hall of Fame, David Lee Roth or Cia Berg?'
+  - vars:
+      question: "Zimbabwe's Guwe Secondary School has a sister school in what New York county?"

examples/google-vertex/response-schema.json

@@ -0,0 +1,10 @@
+{
+  "type": "object",
+  "properties": {
+    "tweet": {
+      "description": "The tweet content",
+      "type": "string"
+    }
+  },
+  "required": ["tweet"]
+}