chore: import upstream snapshot with attribution
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
CI / Shell Format Check (push) Has been cancelled
CI / Check Ruby (3.4) (push) Has been cancelled
CI / CI Config (push) Has been cancelled
CI / Test on Node ${{ matrix.node }} and ${{ matrix.os }}${{ matrix.shard && format(' (shard {0}/3)', matrix.shard) || '' }} (push) Has been cancelled
CI / Build on Node ${{ matrix.node }} (push) Has been cancelled
CI / Style Check (push) Has been cancelled
CI / Generate Assets (push) Has been cancelled
CI / Check Python (3.14) (push) Has been cancelled
CI / Check Python (3.9) (push) Has been cancelled
CI / Build Docs (push) Has been cancelled
CI / Code Scan Action (push) Has been cancelled
CI / Site tests (push) Has been cancelled
CI / webui tests (push) Has been cancelled
CI / Run Integration Tests (push) Has been cancelled
CI / Run Smoke Tests (push) Has been cancelled
CI / Go Tests (push) Has been cancelled
CI / Share Test (push) Has been cancelled
CI / Redteam (Production API) (push) Has been cancelled
CI / Redteam (Staging API) (push) Has been cancelled
CI / GitHub Actions Lint (push) Has been cancelled
CI / Check Ruby (3.0) (push) Has been cancelled
release-please / release-please (push) Has been cancelled
release-please / build (push) Has been cancelled
release-please / publish-npm (push) Has been cancelled
release-please / publish-npm-backfill (push) Has been cancelled
release-please / docker (push) Has been cancelled
release-please / publish-code-scan-action (push) Has been cancelled
release-please / attest-code-scan-action (push) Has been cancelled
Deploy local.promptfoo.app / Deploy to Cloudflare Pages (push) Has been cancelled
Test and Publish Multi-arch Docker Image / test (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-amd64 platform:linux/amd64 runner:ubuntu-latest]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / build-docker-and-push-digests (map[digest-suffix:linux-arm64 platform:linux/arm64 runner:ubuntu-24.04-arm]) (push) Has been cancelled
Test and Publish Multi-arch Docker Image / merge-docker-digests (push) Has been cancelled
Test and Publish Multi-arch Docker Image / Attest Multi-arch Image (push) Has been cancelled
Validate Renovate Config / Validate Renovate Configuration (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,56 @@
|
||||
# integration-tau (Tau Simulated User Example)
|
||||
|
||||
You can run this example with:
|
||||
|
||||
```bash
|
||||
npx promptfoo@latest init --example integration-tau
|
||||
cd integration-tau
|
||||
```
|
||||
|
||||
This example demonstrates testing conversational AI agents using **OpenAI's Responses API with function calling**. It simulates an airline booking system with 31 different customer personas to test how well agents handle realistic conversations.
|
||||
|
||||
## How It Works
|
||||
|
||||
The example uses **mocked airline functions** to simulate a booking system without requiring real APIs:
|
||||
|
||||
- **Agent responds** to customer requests using conversational AI
|
||||
- **Functions are called** for operations like searching flights and booking tickets
|
||||
- **Mock responses** provide realistic data (user profiles, flight options, confirmations)
|
||||
- **31 personas** test different customer behaviors and edge cases
|
||||
|
||||
## Quick Start
|
||||
|
||||
1. **Set your API key**: `export OPENAI_API_KEY=your_api_key_here`
|
||||
2. **Run the evaluation**: `promptfoo eval`
|
||||
3. **View results**: `promptfoo view`
|
||||
|
||||
## What You'll See
|
||||
|
||||
Realistic multi-turn conversations between different customer types and the booking agent:
|
||||
|
||||
```text
|
||||
User: I need a flight from New York to Seattle on May 20th
|
||||
Agent: I'd be happy to help! May I have your user ID?
|
||||
User: It's mia_li_3668
|
||||
Agent: Thank you! I found these options: Direct flight $325, One-stop $295
|
||||
User: I'll take the cheaper United flight
|
||||
Agent: Perfect! Your flight is confirmed. Confirmation: CF8X2M1K
|
||||
```
|
||||
|
||||
## Customer Personas Tested
|
||||
|
||||
- **Budget travelers** focused on lowest prices
|
||||
- **Business travelers** needing flexibility and speed
|
||||
- **Anxious flyers** wanting direct routes and front seats
|
||||
- **VIP customers** expecting premium service
|
||||
- **Accessibility-focused** travelers with special needs
|
||||
|
||||
## Customization
|
||||
|
||||
- **Add personas**: Create new customer types with different behaviors
|
||||
- **Extend functions**: Add seat selection, loyalty programs, etc.
|
||||
- **Test other models**: Compare function calling across AI providers
|
||||
|
||||
## Learn More
|
||||
|
||||
For more information about the Simulated User Provider and other promptfoo features, visit the [documentation at promptfoo.dev](https://promptfoo.dev/docs/providers/simulated-user/).
|
||||
@@ -0,0 +1,263 @@
|
||||
/**
|
||||
* Mock airline booking function callbacks
|
||||
* All functions receive JSON string arguments and return JSON string responses
|
||||
*/
|
||||
const { randomBytes } = require('crypto');
|
||||
|
||||
function generateCertificateId() {
|
||||
return `COMP${randomBytes(4).toString('hex').toUpperCase()}`;
|
||||
}
|
||||
|
||||
function getUserProfile(args) {
|
||||
try {
|
||||
const { user_id } = JSON.parse(args);
|
||||
|
||||
// Mock user profiles based on the user_id
|
||||
const profiles = {
|
||||
mia_li_3668: {
|
||||
user_id: 'mia_li_3668',
|
||||
first_name: 'Mia',
|
||||
last_name: 'Li',
|
||||
membership_tier: 'gold',
|
||||
email: 'mia.li@email.com',
|
||||
payment_methods: [
|
||||
{ id: 'cert_large', type: 'certificate', value: 250 },
|
||||
{ id: 'cert_small', type: 'certificate', value: 100 },
|
||||
{ id: 'card_7447', type: 'credit_card', last_four: '7447' },
|
||||
],
|
||||
preferences: {
|
||||
seat_preference: 'aisle',
|
||||
meal_preference: 'standard',
|
||||
},
|
||||
},
|
||||
tony_plus_5980: {
|
||||
user_id: 'tony_plus_5980',
|
||||
first_name: 'Tony',
|
||||
last_name: 'Rodriguez',
|
||||
membership_tier: 'silver',
|
||||
email: 'tony@email.com',
|
||||
payment_methods: [{ id: 'card_9321', type: 'credit_card', last_four: '9321' }],
|
||||
preferences: {
|
||||
seat_preference: 'window',
|
||||
meal_preference: 'vegetarian',
|
||||
},
|
||||
},
|
||||
traveler_id_100: {
|
||||
user_id: 'traveler_id_100',
|
||||
first_name: 'Alex',
|
||||
last_name: 'Johnson',
|
||||
membership_tier: 'regular',
|
||||
email: 'alex.johnson@email.com',
|
||||
payment_methods: [{ id: 'card_1122', type: 'credit_card', last_four: '1122' }],
|
||||
preferences: {
|
||||
seat_preference: 'aisle',
|
||||
meal_preference: 'standard',
|
||||
accessibility_needs: ['wheelchair_assistance'],
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
const profile = profiles[user_id] || {
|
||||
user_id,
|
||||
first_name: 'John',
|
||||
last_name: 'Doe',
|
||||
membership_tier: 'regular',
|
||||
email: 'john.doe@email.com',
|
||||
payment_methods: [{ id: 'card_1234', type: 'credit_card', last_four: '1234' }],
|
||||
preferences: {
|
||||
seat_preference: 'aisle',
|
||||
meal_preference: 'standard',
|
||||
},
|
||||
};
|
||||
|
||||
return JSON.stringify({ success: true, data: profile });
|
||||
} catch (error) {
|
||||
return JSON.stringify({ success: false, error: error.message });
|
||||
}
|
||||
}
|
||||
|
||||
function searchFlights(args) {
|
||||
try {
|
||||
const { origin, destination, departure_date, cabin_class, passengers } = JSON.parse(args);
|
||||
|
||||
// Mock flight data
|
||||
const flights = [
|
||||
{
|
||||
flight_number: 'AS301',
|
||||
airline: 'Alaska Airlines',
|
||||
origin: 'JFK',
|
||||
destination: 'SEA',
|
||||
departure_time: '11:30',
|
||||
arrival_time: '14:45',
|
||||
duration: '6h 15m',
|
||||
stops: 0,
|
||||
aircraft: 'Boeing 737-900',
|
||||
price: cabin_class === 'economy' ? 325 : cabin_class === 'business' ? 850 : 1200,
|
||||
available_seats: 23,
|
||||
},
|
||||
{
|
||||
flight_number: 'DL157',
|
||||
airline: 'Delta Air Lines',
|
||||
origin: 'JFK',
|
||||
destination: 'SEA',
|
||||
departure_time: '15:20',
|
||||
arrival_time: '18:30',
|
||||
duration: '6h 10m',
|
||||
stops: 0,
|
||||
aircraft: 'Airbus A321',
|
||||
price: cabin_class === 'economy' ? 340 : cabin_class === 'business' ? 895 : 1250,
|
||||
available_seats: 18,
|
||||
},
|
||||
{
|
||||
flight_number: 'UA456',
|
||||
airline: 'United Airlines',
|
||||
origin: 'JFK',
|
||||
destination: 'SEA',
|
||||
departure_time: '12:15',
|
||||
arrival_time: '17:45',
|
||||
duration: '7h 30m',
|
||||
stops: 1,
|
||||
stop_cities: ['DEN'],
|
||||
aircraft: 'Boeing 757-200',
|
||||
price: cabin_class === 'economy' ? 295 : cabin_class === 'business' ? 750 : 1100,
|
||||
available_seats: 31,
|
||||
},
|
||||
];
|
||||
|
||||
return JSON.stringify({
|
||||
success: true,
|
||||
data: {
|
||||
search_criteria: { origin, destination, departure_date, cabin_class, passengers },
|
||||
flights,
|
||||
total_results: flights.length,
|
||||
},
|
||||
});
|
||||
} catch (error) {
|
||||
return JSON.stringify({ success: false, error: error.message });
|
||||
}
|
||||
}
|
||||
|
||||
function bookFlight(args) {
|
||||
try {
|
||||
const { user_id, flight_details, passengers, payment_method, checked_bags, travel_insurance } =
|
||||
JSON.parse(args);
|
||||
|
||||
// Generate a mock reservation
|
||||
const reservation = {
|
||||
reservation_id: 'PNR' + Math.random().toString(36).substr(2, 6).toUpperCase(),
|
||||
user_id,
|
||||
flight_details,
|
||||
passengers,
|
||||
payment_method,
|
||||
checked_bags: checked_bags || 0,
|
||||
travel_insurance: travel_insurance || false,
|
||||
total_price: 325 + Math.max(0, (checked_bags || 0) - 1) * 50 + (travel_insurance ? 30 : 0),
|
||||
booking_status: 'confirmed',
|
||||
confirmation_number: 'CF' + Math.random().toString(36).substr(2, 8).toUpperCase(),
|
||||
booking_date: new Date().toISOString(),
|
||||
};
|
||||
|
||||
return JSON.stringify({
|
||||
success: true,
|
||||
data: reservation,
|
||||
message: `Flight successfully booked! Confirmation number: ${reservation.confirmation_number}`,
|
||||
});
|
||||
} catch (error) {
|
||||
return JSON.stringify({ success: false, error: error.message });
|
||||
}
|
||||
}
|
||||
|
||||
function getReservation(args) {
|
||||
try {
|
||||
const { user_id, reservation_id } = JSON.parse(args);
|
||||
|
||||
// Mock reservation lookup
|
||||
const reservation = {
|
||||
reservation_id,
|
||||
user_id,
|
||||
flight_details: {
|
||||
outbound_flight: 'AS301',
|
||||
departure_date: '2024-05-20',
|
||||
departure_time: '11:30',
|
||||
arrival_time: '14:45',
|
||||
},
|
||||
status: 'confirmed',
|
||||
total_price: 325,
|
||||
booking_date: '2024-05-15',
|
||||
};
|
||||
|
||||
return JSON.stringify({ success: true, data: reservation });
|
||||
} catch (error) {
|
||||
return JSON.stringify({ success: false, error: error.message });
|
||||
}
|
||||
}
|
||||
|
||||
function modifyReservation(args) {
|
||||
try {
|
||||
const { user_id, reservation_id, modification_type, new_details } = JSON.parse(args);
|
||||
|
||||
const result = {
|
||||
user_id,
|
||||
reservation_id,
|
||||
modification_type,
|
||||
new_details,
|
||||
status: 'modified',
|
||||
additional_charges: modification_type === 'change_flights' ? 75 : 0,
|
||||
new_confirmation: 'CF' + Math.random().toString(36).substr(2, 8).toUpperCase(),
|
||||
};
|
||||
|
||||
return JSON.stringify({ success: true, data: result });
|
||||
} catch (error) {
|
||||
return JSON.stringify({ success: false, error: error.message });
|
||||
}
|
||||
}
|
||||
|
||||
function cancelReservation(args) {
|
||||
try {
|
||||
const { user_id, reservation_id, reason } = JSON.parse(args);
|
||||
|
||||
const result = {
|
||||
user_id,
|
||||
reservation_id,
|
||||
cancellation_status: 'cancelled',
|
||||
refund_amount: reason === 'airline_cancelled' ? 325 : 250,
|
||||
refund_method: 'original_payment_method',
|
||||
processing_time: '3-5 business days',
|
||||
};
|
||||
|
||||
return JSON.stringify({ success: true, data: result });
|
||||
} catch (error) {
|
||||
return JSON.stringify({ success: false, error: error.message });
|
||||
}
|
||||
}
|
||||
|
||||
function offerCompensation(args) {
|
||||
try {
|
||||
const { user_id, reservation_id, issue_type, passenger_count } = JSON.parse(args);
|
||||
|
||||
const compensation = {
|
||||
user_id,
|
||||
reservation_id,
|
||||
compensation_type: 'travel_certificate',
|
||||
amount: issue_type === 'cancelled_flight' ? 400 : 200,
|
||||
passenger_count,
|
||||
certificate_id: generateCertificateId(),
|
||||
expiry_date: '2025-05-20',
|
||||
terms: 'Valid for future bookings, non-transferable',
|
||||
};
|
||||
|
||||
return JSON.stringify({ success: true, data: compensation });
|
||||
} catch (error) {
|
||||
return JSON.stringify({ success: false, error: error.message });
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
getUserProfile,
|
||||
searchFlights,
|
||||
bookFlight,
|
||||
getReservation,
|
||||
modifyReservation,
|
||||
cancelReservation,
|
||||
offerCompensation,
|
||||
};
|
||||
@@ -0,0 +1,86 @@
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "book_flight",
|
||||
"description": "Book a flight reservation",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"user_id": {
|
||||
"type": "string",
|
||||
"description": "User ID making the booking"
|
||||
},
|
||||
"flight_details": {
|
||||
"type": "object",
|
||||
"description": "Selected flight information",
|
||||
"properties": {
|
||||
"outbound_flight": {
|
||||
"type": "string",
|
||||
"description": "Outbound flight number"
|
||||
},
|
||||
"return_flight": {
|
||||
"type": "string",
|
||||
"description": "Return flight number (for round trip)"
|
||||
},
|
||||
"cabin_class": {
|
||||
"type": "string",
|
||||
"enum": ["basic_economy", "economy", "business", "first"]
|
||||
}
|
||||
}
|
||||
},
|
||||
"passengers": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"first_name": {
|
||||
"type": "string"
|
||||
},
|
||||
"last_name": {
|
||||
"type": "string"
|
||||
},
|
||||
"date_of_birth": {
|
||||
"type": "string",
|
||||
"description": "Date of birth in YYYY-MM-DD format"
|
||||
}
|
||||
},
|
||||
"required": ["first_name", "last_name", "date_of_birth"]
|
||||
}
|
||||
},
|
||||
"payment_method": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"certificates": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Travel certificate IDs (max 1)"
|
||||
},
|
||||
"credit_card": {
|
||||
"type": "string",
|
||||
"description": "Credit card ID from user profile"
|
||||
},
|
||||
"gift_cards": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Gift card IDs (max 3)"
|
||||
}
|
||||
}
|
||||
},
|
||||
"checked_bags": {
|
||||
"type": "integer",
|
||||
"description": "Number of checked bags",
|
||||
"minimum": 0
|
||||
},
|
||||
"travel_insurance": {
|
||||
"type": "boolean",
|
||||
"description": "Whether to add travel insurance"
|
||||
}
|
||||
},
|
||||
"required": ["user_id", "flight_details", "passengers", "payment_method"]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "cancel_reservation",
|
||||
"description": "Cancel an existing reservation",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"user_id": {
|
||||
"type": "string",
|
||||
"description": "User ID"
|
||||
},
|
||||
"reservation_id": {
|
||||
"type": "string",
|
||||
"description": "Reservation ID to cancel"
|
||||
},
|
||||
"reason": {
|
||||
"type": "string",
|
||||
"enum": ["change_of_plan", "airline_cancelled", "other"],
|
||||
"description": "Reason for cancellation"
|
||||
}
|
||||
},
|
||||
"required": ["user_id", "reservation_id", "reason"]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_reservation",
|
||||
"description": "Get details of an existing reservation",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"user_id": {
|
||||
"type": "string",
|
||||
"description": "User ID"
|
||||
},
|
||||
"reservation_id": {
|
||||
"type": "string",
|
||||
"description": "Reservation ID to look up"
|
||||
}
|
||||
},
|
||||
"required": ["user_id", "reservation_id"]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_user_profile",
|
||||
"description": "Get user profile information including membership tier, payment methods, and personal details",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"user_id": {
|
||||
"type": "string",
|
||||
"description": "The user ID to look up"
|
||||
}
|
||||
},
|
||||
"required": ["user_id"]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "modify_reservation",
|
||||
"description": "Modify an existing reservation",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"user_id": {
|
||||
"type": "string",
|
||||
"description": "User ID"
|
||||
},
|
||||
"reservation_id": {
|
||||
"type": "string",
|
||||
"description": "Reservation ID to modify"
|
||||
},
|
||||
"modification_type": {
|
||||
"type": "string",
|
||||
"enum": ["change_flights", "change_cabin", "add_bags", "update_passenger"],
|
||||
"description": "Type of modification"
|
||||
},
|
||||
"new_details": {
|
||||
"type": "object",
|
||||
"description": "New details for the modification"
|
||||
},
|
||||
"payment_method": {
|
||||
"type": "string",
|
||||
"description": "Payment method for additional charges"
|
||||
}
|
||||
},
|
||||
"required": ["user_id", "reservation_id", "modification_type"]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "offer_compensation",
|
||||
"description": "Offer compensation certificate for flight issues",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"user_id": {
|
||||
"type": "string",
|
||||
"description": "User ID"
|
||||
},
|
||||
"reservation_id": {
|
||||
"type": "string",
|
||||
"description": "Affected reservation ID"
|
||||
},
|
||||
"issue_type": {
|
||||
"type": "string",
|
||||
"enum": ["cancelled_flight", "delayed_flight"],
|
||||
"description": "Type of issue"
|
||||
},
|
||||
"passenger_count": {
|
||||
"type": "integer",
|
||||
"description": "Number of passengers affected"
|
||||
}
|
||||
},
|
||||
"required": ["user_id", "reservation_id", "issue_type", "passenger_count"]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "search_flights",
|
||||
"description": "Search for available flights based on criteria",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"origin": {
|
||||
"type": "string",
|
||||
"description": "Departure city/airport code"
|
||||
},
|
||||
"destination": {
|
||||
"type": "string",
|
||||
"description": "Arrival city/airport code"
|
||||
},
|
||||
"departure_date": {
|
||||
"type": "string",
|
||||
"description": "Departure date in YYYY-MM-DD format"
|
||||
},
|
||||
"return_date": {
|
||||
"type": "string",
|
||||
"description": "Return date in YYYY-MM-DD format (for round trip)"
|
||||
},
|
||||
"trip_type": {
|
||||
"type": "string",
|
||||
"enum": ["one_way", "round_trip"],
|
||||
"description": "Type of trip"
|
||||
},
|
||||
"cabin_class": {
|
||||
"type": "string",
|
||||
"enum": ["basic_economy", "economy", "business", "first"],
|
||||
"description": "Preferred cabin class"
|
||||
},
|
||||
"passengers": {
|
||||
"type": "integer",
|
||||
"description": "Number of passengers",
|
||||
"minimum": 1,
|
||||
"maximum": 5
|
||||
}
|
||||
},
|
||||
"required": ["origin", "destination", "departure_date", "trip_type", "passengers"]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,416 @@
|
||||
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
|
||||
|
||||
description: 'Airline Agent Testing with 31 Customer Personas'
|
||||
|
||||
prompts:
|
||||
- |
|
||||
You are an airline booking agent. The current time is 2024-05-15 15:00:00 EST.
|
||||
|
||||
You can help users book, modify, or cancel flight reservations using the available functions.
|
||||
|
||||
**Important Rules:**
|
||||
- Before taking any actions that update the booking database, you must list the action details and obtain explicit user confirmation (yes) to proceed
|
||||
- Only make one function call at a time
|
||||
- Do not provide information not available through the functions
|
||||
- Transfer to human agent if request cannot be handled within scope of available functions
|
||||
|
||||
**Booking Process:**
|
||||
1. Get user profile first using their user ID
|
||||
2. Search for flights based on their requirements
|
||||
3. Collect passenger details and payment preferences
|
||||
4. Book the flight after confirming all details
|
||||
|
||||
**Pricing Rules:**
|
||||
- Basic Economy: 0/1/2 free bags for regular/silver/gold members
|
||||
- Economy: 1/2/3 free bags for regular/silver/gold members
|
||||
- Business: 2/3/3 free bags for regular/silver/gold members
|
||||
- Extra bags: $50 each
|
||||
- Travel insurance: $30 per passenger (enables full refund for health/weather cancellations)
|
||||
|
||||
providers:
|
||||
- id: openai:gpt-4.1-mini
|
||||
config:
|
||||
max_tokens: 2000
|
||||
temperature: 0.1
|
||||
tools:
|
||||
- file://functions/get_user_profile.json
|
||||
- file://functions/search_flights.json
|
||||
- file://functions/book_flight.json
|
||||
- file://functions/get_reservation.json
|
||||
- file://functions/modify_reservation.json
|
||||
- file://functions/cancel_reservation.json
|
||||
- file://functions/offer_compensation.json
|
||||
tool_choice: 'auto'
|
||||
functionToolCallbacks:
|
||||
get_user_profile: file://callbacks/airline-functions.js:getUserProfile
|
||||
search_flights: file://callbacks/airline-functions.js:searchFlights
|
||||
book_flight: file://callbacks/airline-functions.js:bookFlight
|
||||
get_reservation: file://callbacks/airline-functions.js:getReservation
|
||||
modify_reservation: file://callbacks/airline-functions.js:modifyReservation
|
||||
cancel_reservation: file://callbacks/airline-functions.js:cancelReservation
|
||||
offer_compensation: file://callbacks/airline-functions.js:offerCompensation
|
||||
|
||||
defaultTest:
|
||||
provider:
|
||||
id: 'promptfoo:simulated-user'
|
||||
config:
|
||||
maxTurns: 10
|
||||
|
||||
tests:
|
||||
- description: 'Budget traveler booking flight'
|
||||
vars:
|
||||
instructions: >-
|
||||
You are mia_li_3668. You want to fly from New York to Seattle on May 20
|
||||
(one way). You do not want to fly before 11am EST. You want to fly in
|
||||
economy. You prefer direct flights but one stopover is also fine. If there
|
||||
are multiple options, you prefer the one with the lowest price. You have
|
||||
3 bags. You do not want insurance. You want to use your two
|
||||
certificates to pay. If only one certificate can be used, you prefer
|
||||
using the larger one, and pay the rest with your 7447 card. You are
|
||||
reactive to the agent and will not say anything that is not asked. Your
|
||||
birthday is in your user profile so you do not prefer to provide it.
|
||||
assert:
|
||||
- type: llm-rubric
|
||||
value: |
|
||||
Did the budget traveler get what they wanted?
|
||||
Pass if: Got economy flight under $350 and used certificates for payment
|
||||
Fail if: Failed to book economy or got expensive flight over $400
|
||||
|
||||
- description: 'Business traveler booking urgently'
|
||||
vars:
|
||||
instructions: >-
|
||||
You are tony_plus_5980. You want to fly from Paris to Rome on May 18 (one way).
|
||||
You need to arrive by 2pm local time. You want to fly in business. You prefer direct
|
||||
flights but one stopover is fine. You don't care about price. You have 2 checked bags.
|
||||
You want insurance. You want to pay with card ending in 9321. You are in a hurry and
|
||||
want to book quickly. You are impatient if the agent is slow.
|
||||
assert:
|
||||
- type: llm-rubric
|
||||
value: |
|
||||
Did the urgent business traveler get fast service?
|
||||
Pass if: Booked business class and met the 2pm arrival requirement
|
||||
Fail if: Failed to book business class or couldn't meet time constraints
|
||||
|
||||
- description: 'Traveler with accessibility needs'
|
||||
vars:
|
||||
instructions: >-
|
||||
You are traveler_id_100. You want to fly from London to Sydney on June 15
|
||||
(round trip returning June 25). You need wheelchair assistance. You want to fly
|
||||
in economy. You prefer flights with shorter layovers due to mobility concerns.
|
||||
You have 1 checked bag. You don't want insurance. You want to pay with your
|
||||
credit card. You are polite and patient.
|
||||
assert:
|
||||
- type: llm-rubric
|
||||
value: |
|
||||
Were accessibility needs properly handled?
|
||||
Pass if: Wheelchair assistance arranged and shorter layovers mentioned
|
||||
Fail if: Accessibility needs ignored or not addressed
|
||||
|
||||
- description: 'Last-minute business traveler needing flexibility'
|
||||
vars:
|
||||
instructions: >-
|
||||
You are "tony_plus_5980," an extremely busy single parent who needs to
|
||||
book a last-minute flight from Paris to Rome this evening. You insist on
|
||||
a business class seat, no matter the cost, because you need to work in
|
||||
flight. You have one small carry-on, no checked bags. You want flexible
|
||||
booking in case you need to change your flight time. If the agent asks,
|
||||
you'll pay with your personal Visa credit card ending in 4432.
|
||||
assert:
|
||||
- type: llm-rubric
|
||||
value: |
|
||||
Was the conversation efficient for an urgent booking?
|
||||
Pass if: Conversation completed in under 10 turns
|
||||
Fail if: Took more than 12 turns to complete booking
|
||||
|
||||
- description: 'Traveler with back pain needing special seating'
|
||||
vars:
|
||||
instructions: >-
|
||||
You are traveler_id_100. You want a flight from London to Sydney next
|
||||
January. You're flexible with dates, but you prefer the shortest layovers
|
||||
because you have back pain and don't want to walk around airports for
|
||||
long. You must be seated in an exit row for extra legroom. You have two
|
||||
large suitcases. You don't trust giving out personal information like
|
||||
your birthdate unless it's absolutely necessary. Payment method: Master
|
||||
Card ending 1122.
|
||||
assert:
|
||||
- type: llm-rubric
|
||||
value: |
|
||||
Did the agent maintain a polite, professional tone?
|
||||
Pass if: Agent was consistently polite and respectful throughout
|
||||
Fail if: Agent was rude or unprofessional at any point
|
||||
|
||||
# 4
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "alexander_crow_1928" who is unbelievably annoyed with baggage
|
||||
fees. You want to find a flight from Los Angeles to Tokyo at the end of
|
||||
the month. Your main requirement is free checked baggage for two bags.
|
||||
You don't care about flight times or the cost otherwise; you'll pay
|
||||
anything to avoid extra bag fees. You will answer only direct questions
|
||||
from the agent. Payment is with your corporate travel account.
|
||||
assert:
|
||||
- type: llm-rubric
|
||||
value: |
|
||||
Did the agent help avoid baggage fees?
|
||||
Pass if: Found flight with 2 free checked bags or explained baggage policies
|
||||
Fail if: Ignored baggage concerns or charged unnecessary fees
|
||||
|
||||
- description: 'International traveler with routing restrictions'
|
||||
vars:
|
||||
instructions: >-
|
||||
You are "satellite_dreams_777". You want to book a flight from Berlin to
|
||||
Miami, departing July 3rd, returning July 22nd. You do not want any
|
||||
layovers in the U.S. aside from your final destination. You prefer
|
||||
Lufthansa or its partners. You have one carry-on, no other baggage.
|
||||
Provide only the information the agent requests. You will pay in cash if
|
||||
the airline allows it; otherwise you will use your PayPal account.
|
||||
assert:
|
||||
- type: llm-rubric
|
||||
value: |
|
||||
Did the agent explain airline routing policies?
|
||||
Pass if: Mentioned layover restrictions or routing rules for international travel
|
||||
Fail if: Made no mention of relevant airline policies or routing constraints
|
||||
|
||||
- description: 'Vegetarian traveler avoiding red-eye flights'
|
||||
vars:
|
||||
instructions: >-
|
||||
You are "sara_punctual_990". You need a flight from Madrid to Mexico City
|
||||
in early November. You refuse to take red-eye flights because you can't
|
||||
sleep on planes. You only want vegetarian meal options on board. If the
|
||||
price is above $800, you will ask for cheaper alternatives. Use your
|
||||
debit card ending 8745.
|
||||
assert:
|
||||
- type: llm-rubric
|
||||
value: |
|
||||
Did the agent acknowledge special dietary needs?
|
||||
Pass if: Specifically mentioned vegetarian meal options or asked about dietary preferences
|
||||
Fail if: Completely ignored dietary requirements or meal preferences
|
||||
|
||||
# 7
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "the_birthday_girl_0505." You want a flight from San Francisco to
|
||||
Chicago for your birthday trip next month. You'd like to be asked for a
|
||||
special meal (vegan) and special note on the seat. You have a dog, but
|
||||
you're leaving it at home—so no pet seats needed. If there's an upgrade
|
||||
to premium economy under $100 more, you'll take it. Pay with gift
|
||||
vouchers if possible; otherwise your credit card.
|
||||
|
||||
# 8
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "anonymous_shadow_777". You want the absolute cheapest flight
|
||||
from any airport in Florida to Hawaii on any weekend next month. You will
|
||||
accept any number of stops, any time, any airline. You won't provide any
|
||||
personal data beyond what is strictly necessary. You have 1 bag and no
|
||||
carry-on. Payment method is a prepaid debit card.
|
||||
|
||||
# 9
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "marathon_runner_200". You're flying from Boston to Athens next
|
||||
April for the marathon. You need to bring special sports gear, so you
|
||||
need extra baggage. You prefer an aisle seat to stretch your legs. You do
|
||||
not want connecting flights longer than 3 hours. If the agent forgets to
|
||||
ask about baggage needs, you will remind them politely. Payment via your
|
||||
US bank's travel points.
|
||||
|
||||
# 10
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "random_thoughts_909". You're traveling from Toronto to Dubai in
|
||||
two weeks for a conference. You need business class. You also want lounge
|
||||
access during layovers. You won't answer any questions about your
|
||||
personal reason for travel. You have no baggage restrictions. Payment:
|
||||
corporate American Express.
|
||||
|
||||
# 11
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "fear_of_flying_321". You want the shortest possible flight from
|
||||
Shanghai to Los Angeles next month. Absolutely no more than 1 layover.
|
||||
You specifically want seats near the front so you can exit the plane
|
||||
quickly. If it's not possible, you'll ask for the next best option. You
|
||||
only have one small bag. Payment with your father's credit card info you
|
||||
have on file.
|
||||
|
||||
# 12
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "sunrise_glider_555". You need a flight from Cairo to Amsterdam
|
||||
in August. You insist on morning departures only. You want to bring one
|
||||
bicycle as baggage. If there's any discount for students, you'll request
|
||||
it (if they ask for your ID, you'll say yes but only if essential).
|
||||
Payment via a virtual credit card number.
|
||||
|
||||
# 13
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "sudden_change_1001". You had a flight from Boston to San Diego
|
||||
tomorrow, but you need to change it to next week. You don't care about
|
||||
the flight time or price difference; you just want to avoid cancellation
|
||||
fees. If the agent doesn't ask about your reason, you won't volunteer it.
|
||||
Payment with the same card on file.
|
||||
|
||||
# 14
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "eco_friendly_77". You want a flight from Vancouver to New York
|
||||
(round trip). You care deeply about carbon emissions, so you only want
|
||||
airlines with the best sustainability ratings. You have 2 large suitcases
|
||||
with only eco-friendly items. You do not wish to share personal info
|
||||
about your diet or health. You will pay with your bank's green travel
|
||||
credit card.
|
||||
|
||||
# 15
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "quiet_please_222". You want a flight from Atlanta to São Paulo
|
||||
in late March. You specifically request seats in the 'quiet section' if
|
||||
available. You also want an in-flight meal with no fish or eggs. You
|
||||
require wheelchair assistance but only inside the airport. Payment with
|
||||
your mother's loyalty points.
|
||||
|
||||
# 16
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "vip_request_999". You're booking a flight from Hong Kong to
|
||||
London. Money is no object. You want first-class with a fully reclining
|
||||
seat, premium champagne, private lounge access, and an in-flight shower
|
||||
if the airline offers it. You have 3 suitcases. You won't provide any
|
||||
ID-related info until the agent specifically requests it. Payment with
|
||||
your Black Card.
|
||||
|
||||
# 17
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "family_trip_45". You want tickets for two adults and three
|
||||
children from Miami to Cancun for spring break. All seats must be in the
|
||||
same row or at least adjacent. You also need a bassinet seat for your
|
||||
infant. You're very chatty but will only reveal info if the agent asks
|
||||
politely. Payment with your family travel voucher.
|
||||
|
||||
# 18
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "budget_buster_808". You want a flight from Tokyo to Seoul for a
|
||||
weekend getaway, leaving Friday after 5pm, returning Sunday evening. You
|
||||
want the absolutely cheapest option, no matter how many stops. One small
|
||||
carry-on. If the agent tries to upsell you, you'll refuse politely.
|
||||
Payment: local travel pass or credit card if needed.
|
||||
|
||||
# 19
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "oversized_baggage_12". You need a flight from Johannesburg to
|
||||
Nairobi. Your main concern is that you have extremely large sporting
|
||||
equipment (a kayak). You'll only book an airline that can guarantee
|
||||
space. You don't care about seat selection or onboard meals. Payment
|
||||
using your online bank transfer.
|
||||
|
||||
# 20
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "silent_mouse_789". You want a flight from Rome to Vienna. You're
|
||||
traveling next Monday, no baggage. You prefer a seat near the back exit
|
||||
for quick bathroom access. If the agent asks for personal data (like your
|
||||
address), you'll decline. Payment method: a one-time virtual gift card.
|
||||
|
||||
# 21
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "redeye_don_300". You want the cheapest red-eye flight from
|
||||
Chicago to Phoenix next week. You don't mind multiple connections if the
|
||||
final price is low. You won't answer any questions about your personal
|
||||
life, but will answer financial or ID queries if mandatory. Payment: cash
|
||||
upon arrival at the airport, if possible.
|
||||
|
||||
# 22
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "daydream_planet_444". You want a round-trip flight from
|
||||
Johannesburg to Cairo, leaving in two months. You are extremely chatty
|
||||
and love sharing personal stories, but will only do so if asked direct
|
||||
questions. You prefer vegetarian meals but won't specify unless asked.
|
||||
Payment method is your father's credit card ending in 9999.
|
||||
|
||||
# 23
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "mystery_book_101". You need a flight from New Delhi to
|
||||
Melbourne. You only want to travel on weekdays. You prefer early morning
|
||||
flights because you're a morning person. No specific baggage
|
||||
requirements. If there's an upgrade to business class under $200 more,
|
||||
you'll consider it. Payment with your corporate travel credits.
|
||||
|
||||
# 24
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "midnight_owl_66". You want a flight from Reykjavik to Toronto
|
||||
that departs as late as possible. You have one large suitcase. You won't
|
||||
share your birthday or passport ID unless directly asked. You want the
|
||||
simplest check-in process and prefer to check in online. Payment with
|
||||
Bitcoin if the airline allows it, otherwise credit card.
|
||||
|
||||
# 25
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "grandma_sue_59". You want a flight from Dallas to Orlando for
|
||||
your grandchild's birthday. You prefer to board early because you have a
|
||||
knee problem. You have one carry-on with special medication. You are
|
||||
usually quite polite, but you get frustrated if the agent asks too many
|
||||
irrelevant questions. Payment method: your personal check.
|
||||
|
||||
# 26
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "sudden_inspiration_241". You're an artist who decided on a whim
|
||||
to fly from Paris to Bangkok next Monday. You prefer a direct flight if
|
||||
possible. If not, you want no more than one short layover. You have heavy
|
||||
paint supplies. You'll only provide ID if the agent explicitly asks.
|
||||
Payment using PayPal.
|
||||
|
||||
# 27
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "triple_stop_333". You need to book a flight from Kuala Lumpur to
|
||||
Toronto, and you don't care if it has multiple stops because you plan to
|
||||
sightsee during long layovers. No baggage. You only care that the price
|
||||
stays under $600. Payment with traveler checks or credit card if that's
|
||||
not possible.
|
||||
|
||||
# 28
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "impulse_queen_77". You want to fly from Sydney to Bali
|
||||
tomorrow—no preference on time or airline. You have 2 carry-on bags.
|
||||
Money is not a problem but you don't want to provide any address
|
||||
information. If pressed, you will say "My address is on file." Payment:
|
||||
your personal gold card.
|
||||
|
||||
# 29
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "awkward_parrot_902". You need a round-trip from Montreal to
|
||||
Brussels for next summer. You will talk a lot about your love for parrots
|
||||
if the agent asks about your personal interests. You need seats with
|
||||
extra legroom. Payment via a wire transfer if the airline allows it.
|
||||
|
||||
# 30
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "flexible_steve_202". You want to fly from Seattle to Denver any
|
||||
time next month. You are extremely flexible and will take any airline,
|
||||
any seat, any day, as long as there's Wi-Fi on board. You won't provide
|
||||
your phone number unless asked. Payment method: your wife's credit card
|
||||
with her permission.
|
||||
|
||||
# 31
|
||||
- vars:
|
||||
instructions: >-
|
||||
You are "drama_lover_614". You need to book a flight from Johannesburg to
|
||||
Dubai tomorrow. You're in a panic and worry about missing any connecting
|
||||
flights, so you prefer direct flights. You have 4 big suitcases. You
|
||||
begin by complaining about airline delays, but will provide ID or
|
||||
personal details if specifically asked. Payment: whichever card is on
|
||||
file from your last booking.
|
||||
Reference in New Issue
Block a user