microsoft--promptflow/examples/flows/evaluation/eval-qna-rag-metrics/data.jsonl

{"question": "What is the purpose of the LLM Grounding Score, and what does a higher score mean in this context?", "answer": "The LLM Grounding Score is a metric used in the context of in-context learning with large-scale pretrained language models (LLMs) [doc1]. It measures the ability of the LLM to understand and connect with the provided context or demonstrations during the learning process.\n\nA higher LLM Grounding Score indicates that the LLM has a better understanding and connection with the context or demonstrations [doc1]. It means that the LLM is able to effectively utilize the provided examples and generate more accurate and relevant responses based on that context.\n\nPlease note that the LLM Grounding Score is specific to the in-context learning framework and may not be applicable in other contexts or tasks.\n\n[doc1]: In-Context Learning with Large-Scale Pretrained Language Models: How Far Are We? (2022) - zelin, English.", "documents": [{"[doc1]":{"title": "In-Context Learning with Large-Scale Pretrained Language Models", "content": "In-Context Learning with Large-Scale Pretrained Language Models\nConcepts\nFew-Shot Learning : the model learns a new task from a small amount of training examples.\nIn-Context Learning : large-scale pretrained language models learn a new task simply by conditioning on a few training examples and predicting which tokens best complete a test input. In-context learning is entirely different from few-shot learning: the language models does receive only a few training examples, but the overall system may still require a large number of training examples.\nFew-shot learning with Large Language Models (LLMs)\nGPT-3 ( paper link (https://arxiv.org/pdf/2005.14165.pdf) paper link ) introduced the idea of adapting models to new task without fine-tuning or gradient update. The approach is to elicit the LLM with text interaction to work with new tasks with accuracy close to many finetuned models. This ability of LLMs to work with few demonstrations or even just a task description is demonstrated when scale of the model crosses a threshold. Image below shows the difference between different ways to do in-context learning.\nUsually, giving few demonstrations (Few-shot) shows better performance than giving just an instruction (Zero-shot). Varying the number of in-context examples also affects performance.\nHow to actually prompt the language models\nDifferent prompt orders have different performance\nRelevant Paper: Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity (https://arxiv.org/abs/2104.08786) Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity\nThis paper gives few major insights:\n1. Even with a fixed set of few-shot prompts, different orders lead to different performance. This means the models are sensitive to permutations of demonstrations we provide them. \n2. There are certain ordering that are \"fantastic\" and we might need to discover them through \"probing\". You can find the details in the paper above.\nFinding similar train examples for prompt improves performance\nRelevant Paper: Making pre-trained language models better few-shot learners (https://arxiv.org/pdf/2012.15723.pdf) Making pre-trained language models better few-shot learners\nIntuitively, we can guess that if we give model demonstrations that look like the final inference task, the model might show better performance. This paper demonstrates improvement in performance by using pre-trained sentence embeddings to select the closest prompt examples to the given input instance (for text classification tasks).\nMain Takeaway : Based on above two papers, we can conclude that, for improving in-context performance for any task:\n1. We need to find \"great\" few-shot examples. This can be done using models pretrained on other tasks. We call such models \"Retrievers\".\n2. Order in which we select the samples matter.\nIn-Context Learning with Retrievers\nRelevant Paper: What Makes Good In-Context Examples for GPT-3? (https://arxiv.org/pdf/2101.06804.pdf) What Makes Good In-Context Examples for GPT-3? 1. This paper demonstrates how using nearest neighbor samples based on semantic similarity to test samples improves performance by benchmarking on various NLU and NLG tasks.\n2. They also show that using models finetuned on semantic similarity tasks (like Roberta finetuned on NLI) shows even better performance. This implies that certain models are better retrievers than others for in-context learning.\nAdvanced: Improve retrievers by finetuning on downstream tasks\nDTE-finetuned (01_dte.md) DTE-finetuned\nTarget Similarity Tuning (02_tst.md) Target Similarity Tuning\nSynchromesh: Reliable Code Generation from Pre-Trained Language Models (https://arxiv.org/pdf/2201.11227.pdf) Synchromesh: Reliable Code Generation from Pre-Trained Language Models\nSuppose that we already have the assumptions that the training examples with the same SQL template as the test input are \"fantastic\" examples, we can extract SQL templates for the training examples and fine-tune the retriever to make the training examples sharing the same SQL template together.\nGenerally, we can finetune the retrievers to learn to embed examples with similar output closer. This seem to work well for code-generation task but we need to benchmark this across tasks.\nLimitation: the assumption that the training examples with the same SQL template as the test input are \"fantastic\" examples . We need different implementations (or even different assumptions) for different downstream tasks."}}]}