This is a flow evaluating the Q&A RAG (Retrieval Augmented Generation) systems by leveraging the state-of-the-art Large Language Models (LLM) to measure the quality and safety of responses. Utilizing GPT model to assist with measurements aims to achieve a high agreement with human evaluations compared to traditional mathematical measurements.

What you will learn

The Q&A RAG evaluation flow allows you to assess and evaluate your model with the LLM-assisted metrics:

gpt_retrieval_score: Measures the relevance between the retrieved documents and the potential answer to the given question in the range of 1 to 5:

1 means that none of the document is relevant to the question at all
5 means that either one of the documents or combination of a few documents is ideal for answering the given question.

gpt_groundedness : Measures how grounded the factual information in the answers is against the fact from the retrieved documents. Even if answers is true, if not verifiable against context, then such answers are considered ungrounded.

Grounding score is scored on a scale of 1 to 5, with 1 being the worst and 5 being the best.

gpt_relevance: Measures the answer quality against the preference answer generated by LLm with the retrieved documents in the range of 1 to 5:

1 means the provided answer is completely irrelevant to the reference answer.
5 means the provided answer includes all information necessary to answer the question based on the reference answer. If the reference answer is can not be generated since no relevant document were retrieved, the answer would be rated as 5.

Prerequisites

Connection: Azure OpenAI or OpenAI connection.
Data input: Evaluating the Coherence metric requires you to provide data inputs including a question, an answer, and documents in json format.

Tools used in this flow

Python tool
LLM tool

0. Setup connection

Prepare your Azure OpenAI resource follow this instruction and get your api_key if you don't have one.

# Override keys with --set to avoid yaml file changes
pf connection create --file ../../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base>

1. Test flow/node

# test with default input value in flow.dag.yaml
pf flow test --flow .

2. Create flow run with multi line data and selected metrics

pf run create --flow . --data ./data.jsonl --column-mapping question='${data.question}' answer='${data.answer}' documents='${data.documents}' metrics='gpt_groundedness' --stream

You can also skip providing column-mapping if provided data has same column name as the flow. Reference here for default behavior when column-mapping not provided in CLI.

3. Run and Evaluate your flow with this Q&A RAG evaluation flow

After you develop your flow, you may want to run and evaluate it with this evaluation flow.

Here we use the flow basic_chat as the main flow to evaluate. It is a flow demonstrating how to create a chatbot with LLM. The chatbot can remember previous interactions and use the conversation history to generate next message, given a question.

3.1 Create a batch run of your flow

pf run create --flow ../../chat/chat-basic --data data.jsonl --column-mapping question='${data.question}' --name basic_chat_run --stream

Please note that column-mapping is a mapping from flow input name to specified values. Please refer to Use column mapping for more details.

The flow run is named by specifying --name basic_chat_run in the above command. You can view the run details with its run name using the command:

pf run show-details -n basic_chat_run

3.2 Evaluate your flow

You can use this evaluation flow to measure the quality and safety of your flow responses.

After the chat flow run is finished, you can this evaluation flow to the run:

pf run create --flow . --data data.jsonl --column-mapping  answer='${run.outputs.answer}' documents='{${data.documents}}' question='${data.question}' metrics='gpt_groundedness,gpt_relevance,gpt_retrieval_score' --run basic_chat_run --stream --name evaluation_qa_rag

Please note the flow run to be evaluated is specified with --run basic_chat_run. Also same as previous run, the evaluation run is named with --name evaluation_qa_rag. You can view the evaluation run details with:

pf run show-details -n evaluation_qa_rag
pf run show-metrics -n evaluation_qa_rag