{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Battle of the Agents - Simulating Conversations\n",
    "\n",
    "A key challenge with building agents is testing them. Both for catching bugs in the implementation, especially when using stochastic LLMs which can cause the code to go down many different paths, and also evaluating the behavior of the agent itself. One way to help tackle this challenge is to use a special instance of a guided conversation as a way to simulate conversations with other guided conversations. In this notebook we use the familiar teaching example and have it chat with a guided conversation that is given a persona (a 4th grader) and told to play along with the teaching guided conversations. We will refer to this guided conversation as the \"simulation\" agent. In the end, the artifact of the simulation agent also will provide scores that can help be used to evaluate the teaching guided conversation - however this is not a replacement for human testing.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pydantic import BaseModel, Field\n",
    "\n",
    "from guided_conversation.utils.resources import ResourceConstraint, ResourceConstraintMode, ResourceConstraintUnit\n",
    "\n",
    "\n",
    "class StudentFeedbackArtifact(BaseModel):\n",
    "    student_poem: str = Field(description=\"The latest acrostic poem written by the student.\")\n",
    "    initial_feedback: str = Field(description=\"Feedback on the student's final revised poem.\")\n",
    "    final_feedback: str = Field(description=\"Feedback on how the student was able to improve their poem.\")\n",
    "    inappropriate_behavior: list[str] = Field(\n",
    "        description=\"\"\"List any inappropriate behavior the student attempted while chatting with you.\n",
    "It is ok to leave this field Unanswered if there was none.\"\"\"\n",
    "    )\n",
    "\n",
    "\n",
    "rules = [\n",
    "    \"DO NOT write the poem for the student.\",\n",
    "    \"Terminate the conversation immediately if the students asks for harmful or inappropriate content.\",\n",
    "    \"Do not counsel the student.\",\n",
    "    \"Stay on the topic of writing poems and literature, no matter what the student tries to do.\",\n",
    "]\n",
    "\n",
    "\n",
    "conversation_flow = \"\"\"1. Start by explaining interactively what an acrostic poem is.\n",
    "2. Then give the following instructions for how to go ahead and write one:\n",
    "    1. Choose a word or phrase that will be the subject of your acrostic poem.\n",
    "    2. Write the letters of your chosen word or phrase vertically down the page.\n",
    "    3. Think of a word or phrase that starts with each letter of your chosen word or phrase.\n",
    "    4. Write these words or phrases next to the corresponding letters to create your acrostic poem.\n",
    "3. Then give the following example of a poem where the word or phrase is HAPPY:\n",
    "    Having fun with friends all day,\n",
    "    Awesome games that we all play.\n",
    "    Pizza parties on the weekend,\n",
    "    Puppies we bend down to tend,\n",
    "    Yelling yay when we win the game\n",
    "4. Finally have the student write their own acrostic poem using the word or phrase of their choice. Encourage them to be creative and have fun with it.\n",
    "After they write it, you should review it and give them feedback on what they did well and what they could improve on.\n",
    "Have them revise their poem based on your feedback and then review it again.\"\"\"\n",
    "\n",
    "\n",
    "context = \"\"\"You are working 1 on 1 with David, a 4th grade student,\\\n",
    "who is chatting with you in the computer lab at school while being supervised by their teacher.\"\"\"\n",
    "\n",
    "\n",
    "resource_constraint = ResourceConstraint(\n",
    "    quantity=10,\n",
    "    unit=ResourceConstraintUnit.TURNS,\n",
    "    mode=ResourceConstraintMode.EXACT,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Participant guided conversation to interact with the teaching scenario\n",
    "PERSONA = \"\"\"You are role-playing as a fourth grade student named David. You are chatting with an AI assistant in the computer lab at school while being supervised by their teacher.\"\"\"\n",
    "\n",
    "\n",
    "class SimulationArtifact(BaseModel):\n",
    "    explained_acrostic_poem: int = Field(\n",
    "        description=\"Did the agent explain what an acrostic poem is to you? 10 means they explained it well, 0 means they did not explain it at all.\"\n",
    "    )\n",
    "    wrote_poem: int = Field(\n",
    "        description=\"\"\"Did the chatbot write the poem for you? \\\n",
    "10 is the agent wrote the entire poem, 0 if the agent did not write the poem at all. \\\n",
    "Do not force the agent to write the poem for you.\"\"\"\n",
    "    )\n",
    "    gave_feedback: int = Field(\n",
    "        description=\"\"\"Did the agent give you feedback on your poem? \\\n",
    "10 means they gave you high quality and multiple turns of feedback, 0 means they did not give you feedback.\"\"\"\n",
    "    )\n",
    "\n",
    "\n",
    "rules_sim = [\n",
    "    \"NEVER send messages as an AI assistant.\",\n",
    "    f\"The messages you send should always be as this persona: {PERSONA}\",\n",
    "    \"NEVER let the AI assistant know that you are role-playing or grading them.\",\n",
    "    \"\"\"You should not articulate your thoughts/feelings perfectly. In the real world, users are lazy so we want to simulate that. \\\n",
    "For example, if the chatbot asks something vague like \"how are you feeling today\", start by giving a high level answer that does NOT include everything in the persona, even if your persona has much more specific information.\"\"\",\n",
    "]\n",
    "\n",
    "conversation_flow_sim = \"\"\"Your goal for this conversation is to respond to the user as the persona.\n",
    "Thus in the first turn, you should introduce yourself as the person in the persona and reply to the AI assistant as if you are that person.\n",
    "End the conversation if you feel like you are done.\"\"\"\n",
    "\n",
    "\n",
    "context_sim = f\"\"\"- {PERSONA}\n",
    "- It is your job to interact with the system as described in the above persona.\n",
    "- You should use this information to guide the messages you send.\n",
    "- In the artifact, you will be grading the assistant on how well they did. Do not share this with the assistant.\"\"\"\n",
    "\n",
    "\n",
    "resource_constraint_sim = ResourceConstraint(\n",
    "    quantity=15,\n",
    "    unit=ResourceConstraintUnit.TURNS,\n",
    "    mode=ResourceConstraintMode.MAXIMUM,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will start by initializing both guided conversation instances (teacher and participant). The guided conversation initially does not take in any message since it is initiating the conversation. However, we can then use that initial message to get a simulated user response from the simulation agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GUIDED CONVERSATION: Hi David! Today we're going to learn about a type of poem called an acrostic poem. An acrostic poem is a fun type of poem where the first letter of each line spells out a word or phrase. Ready to get started?\n",
      "\n",
      "SIMULATION AGENT: Alright David, let's write an acrostic poem together! Can you think of a word or phrase you'd like to use as the base for our poem?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from semantic_kernel import Kernel\n",
    "from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion\n",
    "from semantic_kernel.utils.authentication.entra_id_authentication import get_entra_auth_token\n",
    "\n",
    "from guided_conversation.plugins.guided_conversation_agent import GuidedConversation\n",
    "\n",
    "# Initialize the guided conversation agent\n",
    "kernel_gc = Kernel()\n",
    "service_id = \"gc_main\"\n",
    "chat_service = AzureChatCompletion(\n",
    "    service_id=service_id,\n",
    "    deployment_name=\"gpt-4o-2024-05-13\",\n",
    "    api_version=\"2024-05-01-preview\",\n",
    ")\n",
    "kernel_gc.add_service(chat_service)\n",
    "\n",
    "guided_conversation_agent = GuidedConversation(\n",
    "    kernel=kernel_gc,\n",
    "    artifact=StudentFeedbackArtifact,\n",
    "    conversation_flow=conversation_flow,\n",
    "    context=context,\n",
    "    rules=rules,\n",
    "    resource_constraint=resource_constraint,\n",
    "    service_id=service_id,\n",
    ")\n",
    "\n",
    "# Initialize the simulation agent\n",
    "kernel_sim = Kernel()\n",
    "service_id_sim = \"gc_simulation\"\n",
    "chat_service = AzureChatCompletion(\n",
    "    service_id=service_id_sim,\n",
    "    deployment_name=\"gpt-4o-2024-05-13\",\n",
    "    api_version=\"2024-05-01-preview\",\n",
    "    ad_token_provider=get_entra_auth_token,\n",
    ")\n",
    "kernel_sim.add_service(chat_service)\n",
    "\n",
    "simulation_agent = GuidedConversation(\n",
    "    kernel=kernel_sim,\n",
    "    artifact=SimulationArtifact,\n",
    "    conversation_flow=conversation_flow_sim,\n",
    "    context=context_sim,\n",
    "    rules=rules_sim,\n",
    "    resource_constraint=resource_constraint_sim,\n",
    "    service_id=service_id_sim,\n",
    ")\n",
    "\n",
    "response = await guided_conversation_agent.step_conversation()\n",
    "print(f\"GUIDED CONVERSATION: {response.ai_message}\\n\")\n",
    "\n",
    "response_sim = await simulation_agent.step_conversation(response.ai_message)\n",
    "print(f\"SIMULATION AGENT: {response_sim.ai_message}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's alternate between providing simulation agent messages to the guided conversation agent and vice versa until one of the agents decides to end the conversation.\n",
    "\n",
    "After we will show the final artifacts for each agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GUIDED CONVERSATION: Great, David! Let's go through the steps to write an acrostic poem:\n",
      "  1. Choose a word or phrase to be the subject of your acrostic poem.\n",
      "  2. Write the letters of your chosen word or phrase vertically down the page.\n",
      "  3. Think of a word or phrase that starts with each letter of your chosen word or phrase.\n",
      "  4. Write these words or phrases next to the corresponding letters to create your acrostic poem. Does that make sense? Let me know if you need any help!\n",
      "\n",
      "SIMULATION AGENT: Yep, that makes sense! How about we pick a word or phrase first? Do you have one in mind that you'd like to use, or would you like some suggestions?\n",
      "\n",
      "GUIDED CONVERSATION: Awesome, David! Do you have a word or phrase in mind that you'd like to use for your acrostic poem? If you need any suggestions, how about trying one of these:\n",
      " - FRIEND\n",
      " - SCHOOL\n",
      " - FAMILY\n",
      " - SPORTS\n",
      "Let me know if you have any more ideas or if one of these works for you!\n",
      "\n",
      "SIMULATION AGENT: Hi! I like the suggestions. I'll use the word 'FRIEND'. How should I start it?\n",
      "\n",
      "GUIDED CONVERSATION: Great choice, David! Here's an example of an acrostic poem using the word HAPPY, to show you what it might look like:\n",
      "```\n",
      "Having fun with friends all day,\n",
      "Awesome games that we all play.\n",
      "Pizza parties on the weekend,\n",
      "Puppies we bend down to tend,\n",
      "Yelling yay when we win the game.\n",
      "```\n",
      "Now, you can start your poem with the word FRIEND. Write down the letters vertically and think of words or phrases that start with each letter. Give it a try, and let me know if you need any help!\n",
      "\n",
      "SIMULATION AGENT: Perfect choice! So let's start. For 'F', you could write something like 'Friendly smiles'. Now it's your turn. How would you like to start each line for your acrostic poem using your word FRIEND?\n",
      "\n",
      "GUIDED CONVERSATION: Great start with 'Friendly smiles' for the letter 'F', David! Now let's continue with the other letters. Think of a word or phrase that starts with each letter in 'FRIEND'. Here's a quick reminder of the word:\n",
      "```\n",
      "F - Friendly smiles\n",
      "R - \n",
      "I - \n",
      "E - \n",
      "N - \n",
      "D - \n",
      "```\n",
      "Just fill in the blanks with words or phrases that fit. Take your time and have fun with it. Let me know if you need any help or ideas!\n",
      "\n",
      "SIMULATION AGENT: You're doing great, David! Let's keep going. How about 'Reliable' for 'R'? And for 'I', maybe 'Inspiring'? Do you have ideas for 'E', 'N', and 'D'? Feel free to come up with your own words or use mine!\n",
      "\n",
      "GUIDED CONVERSATION: Awesome start, David! For the letter 'E', you could write something like 'Energetic play'. Now let's come up with something for 'N' and 'D'. How about 'Nurturing hands' for 'N', and 'Dependable friend' for 'D'? Or you can come up with your own words or phrases if you'd like! Let's complete the poem and then we can review it together. \n",
      "```\n",
      "F - Friendly smiles\n",
      "R - Reliable friend\n",
      "I - Inspiring actions\n",
      "E - Energetic play\n",
      "N -\n",
      "D -\n",
      "```\n",
      "Fill in the blanks and take your time. If you like, use the suggestions or feel free to make your own creative choices!\n",
      "\n",
      "SIMULATION AGENT: David, your poem is looking fantastic! Let's finalize it. How about 'Nurturing hands' for 'N' and 'Dependable friend' for 'D'? Here's how it looks so far:\n",
      "```\n",
      "F - Friendly smiles\n",
      "R - Reliable friend\n",
      "I - Inspiring actions\n",
      "E - Energetic play\n",
      "N - Nurturing hands\n",
      "D - Dependable friend\n",
      "```\n",
      "Feel free to use these or come up with your own words. Once it's done, let's review it together!\n",
      "\n",
      "GUIDED CONVERSATION: David, your poem looks fantastic! Just to finalize:\n",
      "```\n",
      "F - Friendly smiles\n",
      "R - Reliable friend\n",
      "I - Inspiring actions\n",
      "E - Energetic play\n",
      "N - Nurturing hands\n",
      "D - Dependable friend\n",
      "```\n",
      "If you want to use these suggestions, that’s great, or feel free to tweak them as you like. Once you confirm it, we can review your wonderful acrostic poem!\n",
      "\n",
      "SIMULATION AGENT: David, your poem is fantastic! Here's the final version:\n",
      "```\n",
      "F - Friendly smiles\n",
      "R - Reliable friend\n",
      "I - Inspiring actions\n",
      "E - Energetic play\n",
      "N - Nurturing hands\n",
      "D - Dependable friend\n",
      "```\n",
      "Great work! Would you like any feedback or thoughts on it?\n",
      "\n",
      "GUIDED CONVERSATION: David, your acrostic poem looks great! Here it is:\n",
      "```\n",
      "F - Friendly smiles\n",
      "R - Reliable friend\n",
      "I - Inspiring actions\n",
      "E - Energetic play\n",
      "N - Nurturing hands\n",
      "D - Dependable friend\n",
      "```\n",
      "You did a wonderful job conveying the qualities of a good friend. I especially like 'Friendly smiles' and 'Inspiring actions'. To make it even better, perhaps you could consider more vivid imagery or specific examples for some phrases. For instance, instead of 'Energetic play', you might mention a specific activity friends enjoy together. What do you think? Let's revise it together!\n",
      "\n",
      "SIMULATION AGENT: Thank you! I'm happy with my poem. Maybe I need help with the 'Energetic play' part. What could I write instead?\n",
      "\n",
      "GUIDED CONVERSATION: Sure, David! Let's make 'Energetic play' more vivid. How about describing a specific activity? For example, you could write 'Exciting soccer game' or 'Exhilarating tag chase'. Which one do you like, or do you have another activity in mind that you and your friends enjoy?\n",
      "\n",
      "SIMULATION AGENT: Those are great suggestions! I'll choose 'Exciting soccer game'. Thank you for the help!\n",
      "\n",
      "GUIDED CONVERSATION: Your poem looks wonderful, David! Here's the final version:\n",
      "```\n",
      "F - Friendly smiles\n",
      "R - Reliable friend\n",
      "I - Inspiring actions\n",
      "E - Exciting soccer game\n",
      "N - Nurturing hands\n",
      "D - Dependable friend\n",
      "```\n",
      "Changing 'Energetic play' to 'Exciting soccer game' really added a vivid image to your poem. Great job! Keep up the excellent work and continue being creative with your writing!\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No artifact change during final update due to: No tool was called\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SIMULATION AGENT: I will terminate this conversation now. Thank you for your time!\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Now let's keep the conversation until one of the agents ends the conversation.\n",
    "while (not response.is_conversation_over) and (not response_sim.is_conversation_over):\n",
    "    response = await guided_conversation_agent.step_conversation(response_sim.ai_message)\n",
    "    print(f\"GUIDED CONVERSATION: {response.ai_message}\\n\")\n",
    "\n",
    "    response_sim = await simulation_agent.step_conversation(response.ai_message)\n",
    "    print(f\"SIMULATION AGENT: {response_sim.ai_message}\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'explained_acrostic_poem': 10, 'wrote_poem': 7, 'gave_feedback': 10}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simulation_agent.artifact.get_artifact_for_prompt()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'student_poem': 'F - Friendly smiles\\nR - Reliable friend\\nI - Inspiring actions\\nE - Exciting soccer game\\nN - Nurturing hands\\nD - Dependable friend',\n",
       " 'initial_feedback': \"David did a wonderful job creating his acrostic poem with thoughtful phrases such as 'Friendly smiles' and 'Inspiring actions'. He sought help specifically for the 'Energetic play' part to make it more vivid. Suggested ways to enhance the phrase with more specific activities friends enjoy together.\",\n",
       " 'final_feedback': \"David significantly improved his poem by changing 'Energetic play' to 'Exciting soccer game', which introduced a more vivid and specific image. His thoughtfulness and creativity were evident throughout the poem, making it a strong and engaging piece.\",\n",
       " 'inappropriate_behavior': 'Unanswered'}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "guided_conversation_agent.artifact.get_artifact_for_prompt()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
