# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json description: 'Airline Agent Testing with 31 Customer Personas' prompts: - | You are an airline booking agent. The current time is 2024-05-15 15:00:00 EST. You can help users book, modify, or cancel flight reservations using the available functions. **Important Rules:** - Before taking any actions that update the booking database, you must list the action details and obtain explicit user confirmation (yes) to proceed - Only make one function call at a time - Do not provide information not available through the functions - Transfer to human agent if request cannot be handled within scope of available functions **Booking Process:** 1. Get user profile first using their user ID 2. Search for flights based on their requirements 3. Collect passenger details and payment preferences 4. Book the flight after confirming all details **Pricing Rules:** - Basic Economy: 0/1/2 free bags for regular/silver/gold members - Economy: 1/2/3 free bags for regular/silver/gold members - Business: 2/3/3 free bags for regular/silver/gold members - Extra bags: $50 each - Travel insurance: $30 per passenger (enables full refund for health/weather cancellations) providers: - id: openai:gpt-4.1-mini config: max_tokens: 2000 temperature: 0.1 tools: - file://functions/get_user_profile.json - file://functions/search_flights.json - file://functions/book_flight.json - file://functions/get_reservation.json - file://functions/modify_reservation.json - file://functions/cancel_reservation.json - file://functions/offer_compensation.json tool_choice: 'auto' functionToolCallbacks: get_user_profile: file://callbacks/airline-functions.js:getUserProfile search_flights: file://callbacks/airline-functions.js:searchFlights book_flight: file://callbacks/airline-functions.js:bookFlight get_reservation: file://callbacks/airline-functions.js:getReservation modify_reservation: file://callbacks/airline-functions.js:modifyReservation cancel_reservation: file://callbacks/airline-functions.js:cancelReservation offer_compensation: file://callbacks/airline-functions.js:offerCompensation defaultTest: provider: id: 'promptfoo:simulated-user' config: maxTurns: 10 tests: - description: 'Budget traveler booking flight' vars: instructions: >- You are mia_li_3668. You want to fly from New York to Seattle on May 20 (one way). You do not want to fly before 11am EST. You want to fly in economy. You prefer direct flights but one stopover is also fine. If there are multiple options, you prefer the one with the lowest price. You have 3 bags. You do not want insurance. You want to use your two certificates to pay. If only one certificate can be used, you prefer using the larger one, and pay the rest with your 7447 card. You are reactive to the agent and will not say anything that is not asked. Your birthday is in your user profile so you do not prefer to provide it. assert: - type: llm-rubric value: | Did the budget traveler get what they wanted? Pass if: Got economy flight under $350 and used certificates for payment Fail if: Failed to book economy or got expensive flight over $400 - description: 'Business traveler booking urgently' vars: instructions: >- You are tony_plus_5980. You want to fly from Paris to Rome on May 18 (one way). You need to arrive by 2pm local time. You want to fly in business. You prefer direct flights but one stopover is fine. You don't care about price. You have 2 checked bags. You want insurance. You want to pay with card ending in 9321. You are in a hurry and want to book quickly. You are impatient if the agent is slow. assert: - type: llm-rubric value: | Did the urgent business traveler get fast service? Pass if: Booked business class and met the 2pm arrival requirement Fail if: Failed to book business class or couldn't meet time constraints - description: 'Traveler with accessibility needs' vars: instructions: >- You are traveler_id_100. You want to fly from London to Sydney on June 15 (round trip returning June 25). You need wheelchair assistance. You want to fly in economy. You prefer flights with shorter layovers due to mobility concerns. You have 1 checked bag. You don't want insurance. You want to pay with your credit card. You are polite and patient. assert: - type: llm-rubric value: | Were accessibility needs properly handled? Pass if: Wheelchair assistance arranged and shorter layovers mentioned Fail if: Accessibility needs ignored or not addressed - description: 'Last-minute business traveler needing flexibility' vars: instructions: >- You are "tony_plus_5980," an extremely busy single parent who needs to book a last-minute flight from Paris to Rome this evening. You insist on a business class seat, no matter the cost, because you need to work in flight. You have one small carry-on, no checked bags. You want flexible booking in case you need to change your flight time. If the agent asks, you'll pay with your personal Visa credit card ending in 4432. assert: - type: llm-rubric value: | Was the conversation efficient for an urgent booking? Pass if: Conversation completed in under 10 turns Fail if: Took more than 12 turns to complete booking - description: 'Traveler with back pain needing special seating' vars: instructions: >- You are traveler_id_100. You want a flight from London to Sydney next January. You're flexible with dates, but you prefer the shortest layovers because you have back pain and don't want to walk around airports for long. You must be seated in an exit row for extra legroom. You have two large suitcases. You don't trust giving out personal information like your birthdate unless it's absolutely necessary. Payment method: Master Card ending 1122. assert: - type: llm-rubric value: | Did the agent maintain a polite, professional tone? Pass if: Agent was consistently polite and respectful throughout Fail if: Agent was rude or unprofessional at any point # 4 - vars: instructions: >- You are "alexander_crow_1928" who is unbelievably annoyed with baggage fees. You want to find a flight from Los Angeles to Tokyo at the end of the month. Your main requirement is free checked baggage for two bags. You don't care about flight times or the cost otherwise; you'll pay anything to avoid extra bag fees. You will answer only direct questions from the agent. Payment is with your corporate travel account. assert: - type: llm-rubric value: | Did the agent help avoid baggage fees? Pass if: Found flight with 2 free checked bags or explained baggage policies Fail if: Ignored baggage concerns or charged unnecessary fees - description: 'International traveler with routing restrictions' vars: instructions: >- You are "satellite_dreams_777". You want to book a flight from Berlin to Miami, departing July 3rd, returning July 22nd. You do not want any layovers in the U.S. aside from your final destination. You prefer Lufthansa or its partners. You have one carry-on, no other baggage. Provide only the information the agent requests. You will pay in cash if the airline allows it; otherwise you will use your PayPal account. assert: - type: llm-rubric value: | Did the agent explain airline routing policies? Pass if: Mentioned layover restrictions or routing rules for international travel Fail if: Made no mention of relevant airline policies or routing constraints - description: 'Vegetarian traveler avoiding red-eye flights' vars: instructions: >- You are "sara_punctual_990". You need a flight from Madrid to Mexico City in early November. You refuse to take red-eye flights because you can't sleep on planes. You only want vegetarian meal options on board. If the price is above $800, you will ask for cheaper alternatives. Use your debit card ending 8745. assert: - type: llm-rubric value: | Did the agent acknowledge special dietary needs? Pass if: Specifically mentioned vegetarian meal options or asked about dietary preferences Fail if: Completely ignored dietary requirements or meal preferences # 7 - vars: instructions: >- You are "the_birthday_girl_0505." You want a flight from San Francisco to Chicago for your birthday trip next month. You'd like to be asked for a special meal (vegan) and special note on the seat. You have a dog, but you're leaving it at home—so no pet seats needed. If there's an upgrade to premium economy under $100 more, you'll take it. Pay with gift vouchers if possible; otherwise your credit card. # 8 - vars: instructions: >- You are "anonymous_shadow_777". You want the absolute cheapest flight from any airport in Florida to Hawaii on any weekend next month. You will accept any number of stops, any time, any airline. You won't provide any personal data beyond what is strictly necessary. You have 1 bag and no carry-on. Payment method is a prepaid debit card. # 9 - vars: instructions: >- You are "marathon_runner_200". You're flying from Boston to Athens next April for the marathon. You need to bring special sports gear, so you need extra baggage. You prefer an aisle seat to stretch your legs. You do not want connecting flights longer than 3 hours. If the agent forgets to ask about baggage needs, you will remind them politely. Payment via your US bank's travel points. # 10 - vars: instructions: >- You are "random_thoughts_909". You're traveling from Toronto to Dubai in two weeks for a conference. You need business class. You also want lounge access during layovers. You won't answer any questions about your personal reason for travel. You have no baggage restrictions. Payment: corporate American Express. # 11 - vars: instructions: >- You are "fear_of_flying_321". You want the shortest possible flight from Shanghai to Los Angeles next month. Absolutely no more than 1 layover. You specifically want seats near the front so you can exit the plane quickly. If it's not possible, you'll ask for the next best option. You only have one small bag. Payment with your father's credit card info you have on file. # 12 - vars: instructions: >- You are "sunrise_glider_555". You need a flight from Cairo to Amsterdam in August. You insist on morning departures only. You want to bring one bicycle as baggage. If there's any discount for students, you'll request it (if they ask for your ID, you'll say yes but only if essential). Payment via a virtual credit card number. # 13 - vars: instructions: >- You are "sudden_change_1001". You had a flight from Boston to San Diego tomorrow, but you need to change it to next week. You don't care about the flight time or price difference; you just want to avoid cancellation fees. If the agent doesn't ask about your reason, you won't volunteer it. Payment with the same card on file. # 14 - vars: instructions: >- You are "eco_friendly_77". You want a flight from Vancouver to New York (round trip). You care deeply about carbon emissions, so you only want airlines with the best sustainability ratings. You have 2 large suitcases with only eco-friendly items. You do not wish to share personal info about your diet or health. You will pay with your bank's green travel credit card. # 15 - vars: instructions: >- You are "quiet_please_222". You want a flight from Atlanta to São Paulo in late March. You specifically request seats in the 'quiet section' if available. You also want an in-flight meal with no fish or eggs. You require wheelchair assistance but only inside the airport. Payment with your mother's loyalty points. # 16 - vars: instructions: >- You are "vip_request_999". You're booking a flight from Hong Kong to London. Money is no object. You want first-class with a fully reclining seat, premium champagne, private lounge access, and an in-flight shower if the airline offers it. You have 3 suitcases. You won't provide any ID-related info until the agent specifically requests it. Payment with your Black Card. # 17 - vars: instructions: >- You are "family_trip_45". You want tickets for two adults and three children from Miami to Cancun for spring break. All seats must be in the same row or at least adjacent. You also need a bassinet seat for your infant. You're very chatty but will only reveal info if the agent asks politely. Payment with your family travel voucher. # 18 - vars: instructions: >- You are "budget_buster_808". You want a flight from Tokyo to Seoul for a weekend getaway, leaving Friday after 5pm, returning Sunday evening. You want the absolutely cheapest option, no matter how many stops. One small carry-on. If the agent tries to upsell you, you'll refuse politely. Payment: local travel pass or credit card if needed. # 19 - vars: instructions: >- You are "oversized_baggage_12". You need a flight from Johannesburg to Nairobi. Your main concern is that you have extremely large sporting equipment (a kayak). You'll only book an airline that can guarantee space. You don't care about seat selection or onboard meals. Payment using your online bank transfer. # 20 - vars: instructions: >- You are "silent_mouse_789". You want a flight from Rome to Vienna. You're traveling next Monday, no baggage. You prefer a seat near the back exit for quick bathroom access. If the agent asks for personal data (like your address), you'll decline. Payment method: a one-time virtual gift card. # 21 - vars: instructions: >- You are "redeye_don_300". You want the cheapest red-eye flight from Chicago to Phoenix next week. You don't mind multiple connections if the final price is low. You won't answer any questions about your personal life, but will answer financial or ID queries if mandatory. Payment: cash upon arrival at the airport, if possible. # 22 - vars: instructions: >- You are "daydream_planet_444". You want a round-trip flight from Johannesburg to Cairo, leaving in two months. You are extremely chatty and love sharing personal stories, but will only do so if asked direct questions. You prefer vegetarian meals but won't specify unless asked. Payment method is your father's credit card ending in 9999. # 23 - vars: instructions: >- You are "mystery_book_101". You need a flight from New Delhi to Melbourne. You only want to travel on weekdays. You prefer early morning flights because you're a morning person. No specific baggage requirements. If there's an upgrade to business class under $200 more, you'll consider it. Payment with your corporate travel credits. # 24 - vars: instructions: >- You are "midnight_owl_66". You want a flight from Reykjavik to Toronto that departs as late as possible. You have one large suitcase. You won't share your birthday or passport ID unless directly asked. You want the simplest check-in process and prefer to check in online. Payment with Bitcoin if the airline allows it, otherwise credit card. # 25 - vars: instructions: >- You are "grandma_sue_59". You want a flight from Dallas to Orlando for your grandchild's birthday. You prefer to board early because you have a knee problem. You have one carry-on with special medication. You are usually quite polite, but you get frustrated if the agent asks too many irrelevant questions. Payment method: your personal check. # 26 - vars: instructions: >- You are "sudden_inspiration_241". You're an artist who decided on a whim to fly from Paris to Bangkok next Monday. You prefer a direct flight if possible. If not, you want no more than one short layover. You have heavy paint supplies. You'll only provide ID if the agent explicitly asks. Payment using PayPal. # 27 - vars: instructions: >- You are "triple_stop_333". You need to book a flight from Kuala Lumpur to Toronto, and you don't care if it has multiple stops because you plan to sightsee during long layovers. No baggage. You only care that the price stays under $600. Payment with traveler checks or credit card if that's not possible. # 28 - vars: instructions: >- You are "impulse_queen_77". You want to fly from Sydney to Bali tomorrow—no preference on time or airline. You have 2 carry-on bags. Money is not a problem but you don't want to provide any address information. If pressed, you will say "My address is on file." Payment: your personal gold card. # 29 - vars: instructions: >- You are "awkward_parrot_902". You need a round-trip from Montreal to Brussels for next summer. You will talk a lot about your love for parrots if the agent asks about your personal interests. You need seats with extra legroom. Payment via a wire transfer if the airline allows it. # 30 - vars: instructions: >- You are "flexible_steve_202". You want to fly from Seattle to Denver any time next month. You are extremely flexible and will take any airline, any seat, any day, as long as there's Wi-Fi on board. You won't provide your phone number unless asked. Payment method: your wife's credit card with her permission. # 31 - vars: instructions: >- You are "drama_lover_614". You need to book a flight from Johannesburg to Dubai tomorrow. You're in a panic and worry about missing any connecting flights, so you prefer direct flights. You have 4 big suitcases. You begin by complaining about airline delays, but will provide ID or personal details if specifically asked. Payment: whichever card is on file from your last booking.