meta-llama--llama-cookbook/end-to-end-use-cases/NotebookLlama/Step-1 PDF-Pre-Processing-Logic.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4f67a6a6",
   "metadata": {},
   "source": [
    "## Notebook 1: PDF Pre-processing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f68aee84-04e3-4cbc-be78-6de9e06e704f",
   "metadata": {},
   "source": [
    "In the series, we will be going from a PDF to Podcast using all open models. \n",
    "\n",
    "The first step in getting to the podcast is finding a script, right now our logic is:\n",
    "- Use any PDF on any topic\n",
    "- Prompt `Llama-3.2-1B-Instruct` model to process it into a text file\n",
    "- Re-write this into a podcast transcript in next notebook.\n",
    "\n",
    "In this notebook, we will upload a PDF and save it into a `.txt` file using the `PyPDF2` library, later we will process chunks from the text file using our featherlight model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61cb3584",
   "metadata": {},
   "source": [
    "Most of us shift-enter pass the comments to realise later we need to install libraries. For the few that read the instructions, please remember to do so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "f4fc7aef-3505-482e-a998-790b8b9d48e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install PyPDF2\n",
    "#!pip install rich ipywidgets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b23d509",
   "metadata": {},
   "source": [
    "Assuming you have a PDF uploaded on the same machine, please set the path for the file. \n",
    "\n",
    "Also, if you want to flex your GPU-please switch to a bigger model although the featherlight models work perfectly for this task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "60d0061b-8b8c-4353-850f-f19466a0ae2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "pdf_path = './resources/2402.13116v4.pdf'\n",
    "DEFAULT_MODEL = \"meta-llama/Llama-3.2-1B-Instruct\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "21029232-ac5f-42ca-b26b-baad5b2f49b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import PyPDF2\n",
    "from typing import Optional\n",
    "import os\n",
    "import torch\n",
    "from accelerate import Accelerator\n",
    "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
    "\n",
    "from tqdm.notebook import tqdm\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "203c22eb",
   "metadata": {},
   "source": [
    "Let's make sure we don't stub our toe by checking if the file exists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "153d9ece-37a4-4fff-a8e8-53f923a2b0a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def validate_pdf(file_path: str) -> bool:\n",
    "    if not os.path.exists(file_path):\n",
    "        print(f\"Error: File not found at path: {file_path}\")\n",
    "        return False\n",
    "    if not file_path.lower().endswith('.pdf'):\n",
    "        print(\"Error: File is not a PDF\")\n",
    "        return False\n",
    "    return True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a362ac3",
   "metadata": {},
   "source": [
    "Convert PDF to a `.txt` file. This would simply read and dump the contents of the file. We set the maximum characters to 100k. \n",
    "\n",
    "For people converting their favorite novels into a podcast, they will have to add extra logic of going outside the Llama models context length which is 128k tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b57c2d64-3d75-4aeb-b4ee-bd1661286b66",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_text_from_pdf(file_path: str, max_chars: int = 100000) -> Optional[str]:\n",
    "    if not validate_pdf(file_path):\n",
    "        return None\n",
    "    \n",
    "    try:\n",
    "        with open(file_path, 'rb') as file:\n",
    "            # Create PDF reader object\n",
    "            pdf_reader = PyPDF2.PdfReader(file)\n",
    "            \n",
    "            # Get total number of pages\n",
    "            num_pages = len(pdf_reader.pages)\n",
    "            print(f\"Processing PDF with {num_pages} pages...\")\n",
    "            \n",
    "            extracted_text = []\n",
    "            total_chars = 0\n",
    "            \n",
    "            # Iterate through all pages\n",
    "            for page_num in range(num_pages):\n",
    "                # Extract text from page\n",
    "                page = pdf_reader.pages[page_num]\n",
    "                text = page.extract_text()\n",
    "                \n",
    "                # Check if adding this page's text would exceed the limit\n",
    "                if total_chars + len(text) > max_chars:\n",
    "                    # Only add text up to the limit\n",
    "                    remaining_chars = max_chars - total_chars\n",
    "                    extracted_text.append(text[:remaining_chars])\n",
    "                    print(f\"Reached {max_chars} character limit at page {page_num + 1}\")\n",
    "                    break\n",
    "                \n",
    "                extracted_text.append(text)\n",
    "                total_chars += len(text)\n",
    "                print(f\"Processed page {page_num + 1}/{num_pages}\")\n",
    "            \n",
    "            final_text = '\\n'.join(extracted_text)\n",
    "            print(f\"\\nExtraction complete! Total characters: {len(final_text)}\")\n",
    "            return final_text\n",
    "            \n",
    "    except PyPDF2.PdfReadError:\n",
    "        print(\"Error: Invalid or corrupted PDF file\")\n",
    "        return None\n",
    "    except Exception as e:\n",
    "        print(f\"An unexpected error occurred: {str(e)}\")\n",
    "        return None\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e023397b",
   "metadata": {},
   "source": [
    "Helper function to grab meta info about our PDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "0984bb1e-d52c-4cec-a131-67a48061fabc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get PDF metadata\n",
    "def get_pdf_metadata(file_path: str) -> Optional[dict]:\n",
    "    if not validate_pdf(file_path):\n",
    "        return None\n",
    "    \n",
    "    try:\n",
    "        with open(file_path, 'rb') as file:\n",
    "            pdf_reader = PyPDF2.PdfReader(file)\n",
    "            metadata = {\n",
    "                'num_pages': len(pdf_reader.pages),\n",
    "                'metadata': pdf_reader.metadata\n",
    "            }\n",
    "            return metadata\n",
    "    except Exception as e:\n",
    "        print(f\"Error extracting metadata: {str(e)}\")\n",
    "        return None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6019affc",
   "metadata": {},
   "source": [
    "Finally, we can run our logic to extract the details from the file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "63848943-79cc-4e21-8396-6eab5df493e0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting metadata...\n",
      "\n",
      "PDF Metadata:\n",
      "Number of pages: 44\n",
      "Document info:\n",
      "/Author: \n",
      "/CreationDate: D:20240311015030Z\n",
      "/Creator: LaTeX with hyperref\n",
      "/Keywords: \n",
      "/ModDate: D:20240311015030Z\n",
      "/PTEX.Fullbanner: This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5\n",
      "/Producer: pdfTeX-1.40.25\n",
      "/Subject: \n",
      "/Title: \n",
      "/Trapped: /False\n",
      "\n",
      "Extracting text...\n",
      "Processing PDF with 44 pages...\n",
      "Processed page 1/44\n",
      "Processed page 2/44\n",
      "Processed page 3/44\n",
      "Processed page 4/44\n",
      "Processed page 5/44\n",
      "Processed page 6/44\n",
      "Processed page 7/44\n",
      "Processed page 8/44\n",
      "Processed page 9/44\n",
      "Processed page 10/44\n",
      "Processed page 11/44\n",
      "Processed page 12/44\n",
      "Processed page 13/44\n",
      "Processed page 14/44\n",
      "Processed page 15/44\n",
      "Processed page 16/44\n",
      "Reached 100000 character limit at page 17\n",
      "\n",
      "Extraction complete! Total characters: 100016\n",
      "\n",
      "Preview of extracted text (first 500 characters):\n",
      "--------------------------------------------------\n",
      "1\n",
      "A Survey on Knowledge Distillation of Large\n",
      "Language Models\n",
      "Xiaohan Xu1, Ming Li2, Chongyang Tao3, Tao Shen4, Reynold Cheng1, Jinyang Li1,\n",
      "Can Xu5, Dacheng Tao6, Tianyi Zhou2\n",
      "1The University of Hong Kong2University of Maryland3Microsoft\n",
      "4University of Technology Sydney5Peking University6The University of Sydney\n",
      "{shawnxxh,chongyangtao,hishentao }@gmail.com {minglii,tianyi }@umd.edu\n",
      "ckcheng@cs.hku.hk jl0725@connect.hku.hk\n",
      "Abstract —In the era of Large Language Models (LLMs), Knowledge Distillati\n",
      "--------------------------------------------------\n",
      "\n",
      "Total characters extracted: 100016\n",
      "\n",
      "Extracted text has been saved to extracted_text.txt\n"
     ]
    }
   ],
   "source": [
    "# Extract metadata first\n",
    "print(\"Extracting metadata...\")\n",
    "metadata = get_pdf_metadata(pdf_path)\n",
    "if metadata:\n",
    "    print(\"\\nPDF Metadata:\")\n",
    "    print(f\"Number of pages: {metadata['num_pages']}\")\n",
    "    print(\"Document info:\")\n",
    "    for key, value in metadata['metadata'].items():\n",
    "        print(f\"{key}: {value}\")\n",
    "\n",
    "# Extract text\n",
    "print(\"\\nExtracting text...\")\n",
    "extracted_text = extract_text_from_pdf(pdf_path)\n",
    "\n",
    "# Display first 500 characters of extracted text as preview\n",
    "if extracted_text:\n",
    "    print(\"\\nPreview of extracted text (first 500 characters):\")\n",
    "    print(\"-\" * 50)\n",
    "    print(extracted_text[:500])\n",
    "    print(\"-\" * 50)\n",
    "    print(f\"\\nTotal characters extracted: {len(extracted_text)}\")\n",
    "\n",
    "# Optional: Save the extracted text to a file\n",
    "if extracted_text:\n",
    "    output_file = 'extracted_text.txt'\n",
    "    with open(output_file, 'w', encoding='utf-8') as f:\n",
    "        f.write(extracted_text)\n",
    "    print(f\"\\nExtracted text has been saved to {output_file}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "946d1f59",
   "metadata": {},
   "source": [
    "### Llama Pre-Processing\n",
    "\n",
    "Now let's proceed to justify our distaste for writing regex and use that as a justification for a LLM instead:\n",
    "\n",
    "At this point, have a text file extracted from a PDF of a paper. Generally PDF extracts can be messy due to characters, formatting, Latex, Tables, etc. \n",
    "\n",
    "One way to handle this would be using regex, instead we can also prompt the feather light Llama models to clean up our text for us. \n",
    "\n",
    "Please try changing the `SYS_PROMPT` below to see what improvements you can make:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "7c0828a5-964d-475e-b5f5-40a04e287725",
   "metadata": {},
   "outputs": [],
   "source": [
    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "\n",
    "SYS_PROMPT = \"\"\"\n",
    "You are a world class text pre-processor, here is the raw data from a PDF, please parse and return it in a way that is crispy and usable to send to a podcast writer.\n",
    "\n",
    "The raw data is messed up with new lines, Latex math and you will see fluff that we can remove completely. Basically take away any details that you think might be useless in a podcast author's transcript.\n",
    "\n",
    "Remember, the podcast could be on any topic whatsoever so the issues listed above are not exhaustive\n",
    "\n",
    "Please be smart with what you remove and be creative ok?\n",
    "\n",
    "Remember DO NOT START SUMMARIZING THIS, YOU ARE ONLY CLEANING UP THE TEXT AND RE-WRITING WHEN NEEDED\n",
    "\n",
    "Be very smart and aggressive with removing details, you will get a running portion of the text and keep returning the processed text.\n",
    "\n",
    "PLEASE DO NOT ADD MARKDOWN FORMATTING, STOP ADDING SPECIAL CHARACTERS THAT MARKDOWN CAPATILISATION ETC LIKES\n",
    "\n",
    "ALWAYS start your response directly with processed text and NO ACKNOWLEDGEMENTS about my questions ok?\n",
    "Here is the text:\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd393fae",
   "metadata": {},
   "source": [
    "Instead of having the model process the entire file at once, as you noticed in the prompt-we will pass chunks of the file. \n",
    "\n",
    "One issue with passing chunks counted by characters is, we lose meaning of words so instead we chunk by words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "24e8a547-9d7c-4e2f-be9e-a3aea09cce76",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_word_bounded_chunks(text, target_chunk_size):\n",
    "    \"\"\"\n",
    "    Split text into chunks at word boundaries close to the target chunk size.\n",
    "    \"\"\"\n",
    "    words = text.split()\n",
    "    chunks = []\n",
    "    current_chunk = []\n",
    "    current_length = 0\n",
    "    \n",
    "    for word in words:\n",
    "        word_length = len(word) + 1  # +1 for the space\n",
    "        if current_length + word_length > target_chunk_size and current_chunk:\n",
    "            # Join the current chunk and add it to chunks\n",
    "            chunks.append(' '.join(current_chunk))\n",
    "            current_chunk = [word]\n",
    "            current_length = word_length\n",
    "        else:\n",
    "            current_chunk.append(word)\n",
    "            current_length += word_length\n",
    "    \n",
    "    # Add the last chunk if it exists\n",
    "    if current_chunk:\n",
    "        chunks.append(' '.join(current_chunk))\n",
    "    \n",
    "    return chunks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d74223f",
   "metadata": {},
   "source": [
    "Let's load in the model and start processing the text chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "d04a4f07-b0b3-45ca-8f41-a433e1abe050",
   "metadata": {},
   "outputs": [],
   "source": [
    "accelerator = Accelerator()\n",
    "model = AutoModelForCausalLM.from_pretrained(\n",
    "    DEFAULT_MODEL,\n",
    "    torch_dtype=torch.bfloat16,\n",
    "    use_safetensors=True,\n",
    "    device_map=device,\n",
    ")\n",
    "tokenizer = AutoTokenizer.from_pretrained(DEFAULT_MODEL, use_safetensors=True)\n",
    "model, tokenizer = accelerator.prepare(model, tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "bbda5241-e890-4402-87dd-514d6761bb9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_chunk(text_chunk, chunk_num):\n",
    "    \"\"\"Process a chunk of text and return both input and output for verification\"\"\"\n",
    "    conversation = [\n",
    "        {\"role\": \"system\", \"content\": SYS_PROMPT},\n",
    "        {\"role\": \"user\", \"content\": text_chunk},\n",
    "    ]\n",
    "    \n",
    "    prompt = tokenizer.apply_chat_template(conversation, tokenize=False)\n",
    "    inputs = tokenizer(prompt, return_tensors=\"pt\").to(device)\n",
    "    \n",
    "    with torch.no_grad():\n",
    "        output = model.generate(\n",
    "            **inputs,\n",
    "            temperature=0.7,\n",
    "            top_p=0.9,\n",
    "            max_new_tokens=512\n",
    "        )\n",
    "    \n",
    "    processed_text = tokenizer.decode(output[0], skip_special_tokens=True)[len(prompt):].strip()\n",
    "    \n",
    "    # Print chunk information for monitoring\n",
    "    #print(f\"\\n{'='*40} Chunk {chunk_num} {'='*40}\")\n",
    "    print(f\"INPUT TEXT:\\n{text_chunk[:500]}...\")  # Show first 500 chars of input\n",
    "    print(f\"\\nPROCESSED TEXT:\\n{processed_text[:500]}...\")  # Show first 500 chars of output\n",
    "    print(f\"{'='*90}\\n\")\n",
    "    \n",
    "    return processed_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "a0183c47-339d-4041-ae83-77fc34931075",
   "metadata": {},
   "outputs": [],
   "source": [
    "INPUT_FILE = \"./resources/extracted_text.txt\"  # Replace with your file path\n",
    "CHUNK_SIZE = 1000  # Adjust chunk size if needed\n",
    "\n",
    "chunks = create_word_bounded_chunks(text, CHUNK_SIZE)\n",
    "num_chunks = len(chunks)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "bb36814f-9310-4734-bf54-e16a5032339e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "101"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "447188d3-ebf0-42d5-940e-4d7e0d9dbf32",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read the file\n",
    "with open(INPUT_FILE, 'r', encoding='utf-8') as file:\n",
    "    text = file.read()\n",
    "\n",
    "# Calculate number of chunks\n",
    "num_chunks = (len(text) + CHUNK_SIZE - 1) // CHUNK_SIZE\n",
    "\n",
    "# Cell 6: Process the file with ordered output\n",
    "# Create output file name\n",
    "output_file = f\"clean_{os.path.basename(INPUT_FILE)}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "7917dfdd-b3af-44fc-a8c0-2760ace9363e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b767f45b5e514e7db936cef825af6fce",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Processing chunks:   0%|          | 0/101 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n",
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "1 A Survey on Knowledge Distillation of Large Language Models Xiaohan Xu1, Ming Li2, Chongyang Tao3, Tao Shen4, Reynold Cheng1, Jinyang Li1, Can Xu5, Dacheng Tao6, Tianyi Zhou2 1The University of Hong Kong2University of Maryland3Microsoft 4University of Technology Sydney5Peking University6The University of Sydney {shawnxxh,chongyangtao,hishentao }@gmail.com {minglii,tianyi }@umd.edu ckcheng@cs.hku.hk jl0725@connect.hku.hk Abstract —In the era of Large Language Models (LLMs), Knowledge Distillati...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "===============\n",
      "\n",
      "Knowledge Distillation is a methodology that transfers advanced capabilities from leading proprietary Large Language Models (LLMs) to their open-source counterparts, such as LLaMA and Mistral. This paper presents a comprehensive survey of KD's role in imparting advanced knowledge.\n",
      "\n",
      "Abstract —In the era of Large Language Models, Knowledge Distillation emerges as a pivotal methodology for transferring advanced capabilities from proprietary LLMs to open-source counterparts, facilit...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "advanced knowledge to smaller models and its utility in model compression and self- improvement. Our survey is meticulously structured around three foundational pillars: algorithm ,skill, and verticalization – providing a comprehensive examination of KD mechanisms, the enhancement of specific cognitive abilities, and their practical implications across diverse fields. Crucially, the survey navigates the intricate interplay between data augmentation (DA) and KD, illustrating how DA emerges as a p...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "xamined through a meticulous survey that delves into the foundational pillars of algorithm, skill, and verticalization, which form the backbone of knowledge distillation and deep learning models. The survey provides a comprehensive examination of key mechanisms within the knowledge distillation framework, specifically focusing on the enhancement of cognitive abilities and their practical implications across various fields, with a particular emphasis on the interplay between data augmentation (DA...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "distillation and proposing future research directions. By bridging the gap between proprietary and open-source LLMs, this survey underscores the potential for more accessible, efficient, and powerful AI solutions. Most importantly, we firmly advocate for compliance with the legal terms that regulate the use of LLMs, ensuring ethical and lawful application of KD of LLMs. An associated Github repository is available at https://github.com/Tebmer/Awesome-Knowledge-Distillation-of-LLMs. Index Terms —...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "en-source LLMs, this survey highlights the potential for more accessible, efficient, and powerful AI solutions.\n",
      "\n",
      "Most importantly, we advocate for compliance with legal terms that regulate the use of LLMs, ensuring ethical and lawful application of knowledge distillation.\n",
      "\n",
      "An associated Github repository is available at https://github.com/Tebmer/Awesome-Knowledge-Distillation-of-LLMs. Index Terms - Large language models, knowledge distillation, data augmentation, skill distillation, supervised f...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "complexity, have un- locked new realms of possibility, from generating human- like text to offering sophisticated problem-solving capa- bilities. The core significance of these LLMs lies in their emergent abilities (Wei et al., 2022a,b; Xu et al., 2024a), a phenomenon where the models display capabilities beyond their explicit training objectives, enabling them to tackle a diverse array of tasks with remarkable proficiency. Their deep understanding of context, nuance, and the intrica- cies of hu...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "sophisticated problem-solving capabilities, the core significance of these large language models (LLMs) lies in their emergent abilities, enabling them to tackle a diverse array of tasks with remarkable proficiency....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "applications, promising to revolutionize industries, augment human creativity, and redefine our interaction with technology. Despite the remarkable capabilities of proprietary LLMs like GPT-4 and Gemini, they are not without their shortcom- ings, particularly when viewed in light of the advantages offered by open-source models. A significant drawback is their limited accessibility and higher cost (OpenAI et al., 2023). These proprietary models often come with substantial usage fees and restricte...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "their remarkable capabilities, have some notable limitations, particularly when considering the advantages offered by open-source models, such as GPT-4 and Gemini. These models are often expensive, with substantial usage fees and restricted access, making them inaccessible to individuals and smaller organizations....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "applica- tions. The constraints of accessibility, cost, and adaptability thus present significant challenges in leveraging the full potential of proprietary LLMs. In contrast to proprietary LLMs, open-source modelsarXiv:2402.13116v3 [cs.CL] 8 Mar 2024 2 like LLaMA (Touvron et al., 2023) and Mistral (Jiang et al., 2023a) bring several notable advantages. One of the primary benefits of open-source models is their accessibility and adaptability. Without the constraints of licensing fees or restrict...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "ng restrictions and costs. In contrast, open-source LLMs like LLaMA and Mistral bring several advantages. Accessibility and adaptability are key benefits, as they are more readily available to a broader range of users, including researchers and organizations....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "of drawbacks, primarily stemming from their relatively limited scale and resources compared to their proprietary counterparts. One of the most significant limitations is the smaller model scale, which often results in lower per- formance on real-world tasks with a bunch of instruc- tions (Zheng et al., 2023a). These models, with fewer pa- rameters, may struggle to capture the depth and breadth of knowledge embodied in larger models like GPT-4. Ad- ditionally, the pre-training investment in these...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "ts. One of the most significant limitations is the smaller model scale, resulting in lower performance on real-world tasks with multiple instructions (Zheng et al., 2023a). Models with fewer parameters struggle to capture the depth and breadth of knowledge embodied in larger models like GPT-4. Additionally, the pre-training investment in these open-source models is typically less substantial. This reduced investment can lead to a narrower range of pre-training data, potentially limiting their un...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "effectiveness in specialized applications. This limitation becomes particularly evident when these models are compared to the highly fine-tuned proprietary LLMs, which are often tailored to excel in a wide array of complex scenarios (OpenAI et al., 2023). Primarily, recognizing the disparities between propri- etary and open-source LLMs, KD techniques have surged as a means to bridge the performance gap between these models (Gou et al., 2021; Gupta and Agrawal, 2022). Knowl- edge distillation, in...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "ary models becomes apparent when compared to highly fine-tuned proprietary LLMs. Primarily, the disparity between proprietary and open-source LLMs becomes evident, with proprietary models excelling in complex scenarios, while open-source models excel in a wide range of scenarios. Knowledge distillation, a technique that leverages the advanced capabilities of proprietary models, is used to enhance the competencies of open-source models. This process is similar to transferring the performance of a...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "augmentation (DA) (Feng et al., 2021) has emerged as a prevalent paradigm to achieve knowledge distillation of LLMs, where a small seed of knowledge is used to prompt the LLM to generate more data with respect to a specific skill or domain (Taori et al., 2023). Secondly, KD still retains its fundamental role in compressing LLMs, making them more efficient without significant loss in performance. (Gu et al., 2024; Agarwal et al., 2024). More recently, the strategy of employing open-source LLMs as...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "tillation of LLMs, where a small seed of knowledge is used to prompt the LLM to generate more data with respect to a specific skill or domain (Taori et al., 2023). Furthermore, KD retains its fundamental role in compressing LLMs, making them more efficient without significant loss in performance....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "trend of self-improvement via self-generated knowledge. A key aspect of the knowledge distillation is the en- hancement of skills such as advanced context following (e.g., in-context learning (Huang et al., 2022a) and in- struction following (Taori et al., 2023)), improved align- ment with user intents (e.g., human values/principles (Cui et al., 2023a), and thinking patterns like chain-of-thought (CoT) (Mukherjee et al., 2023)), and NLP task specialization (e.g., semantic understanding (Ding et ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "advanced context following and instruction following**\n",
      "\n",
      "**key aspects of knowledge distillation**\n",
      "\n",
      "* **contextual understanding**: in-context learning and instruction following\n",
      "* **alignment with user intents**: human values/principles and thinking patterns like chain-of-thought\n",
      "* **NLP task specialization**: semantic understanding and code generation\n",
      "\n",
      "**critical skills for various applications**\n",
      "\n",
      "* **healthcare**: accuracy and contextual knowledge\n",
      "* **law**: contextual knowledge and precision\n",
      "*...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "performance by learning from the proprietary models that have been extensively trained and fine-tuned in these areas. The benefits of knowledge distillation in the era of LLMs are multifaceted and transformative (Gu et al., 2024). Through a suite of distillation techniques, the gap between proprietary and open-source models is significantly nar- rowed (Chiang et al., 2023; Xu et al., 2023a) and even filled (Zhao et al., 2023a). This process not only streamlines computational requirements but als...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "ned in the era of LLMs, the benefits of knowledge distillation in the era of LLMs are multifaceted and transformative. Through a suite of distillation techniques, the gap between proprietary and open-source models narrows and is filled. This process streamlines computational requirements and enhances environmental sustainability of AI operations, as open-source models become more proficient with lower overhead....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "catalyzing innovation and growth across various industries and research domains. The escalating need for a comprehensive survey on the knowledge distillation of LLMs stems from the rapidly evolving landscape of AI (OpenAI et al., 2023; Team et al., 2023) and the increasing complexity of these models. As AI continues to penetrate various sectors, the ability to effi- ciently and effectively distill knowledge from proprietary LLMs to open-source ones becomes not just a technical aspiration but a p...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "ch domains. The escalating need for a comprehensive survey on the knowledge distillation of LLMs stems from the rapidly evolving landscape of AI and the increasing complexity of these models. The ability to efficiently and effectively distill knowledge from proprietary LLMs to open-source ones becomes a practical necessity. This is driven by the need to bridge the knowledge gap between the proprietary and open-source LLMs.\n",
      "\n",
      "This need is driven by the 3 models mentioned, including Student, Vicuna...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "SupervisedFine-tuningX,Y preferenceRankOptimizationy,1y,2y3y1y2y3≻≻rank…… DataCuration X,YrawdatasynthesizefeedbackFeedback input outputSelf-Knowledge outputinputinput YlabelLabelingExpansion X,YdemonstrationsexpandFeature featureinput,outputextractSec.4Sec.5 Sec.3.1Sec.3.2 Fig. 2: An overview of this survey on knowledge distillation of large language models. Note that ‘Section’ is abbreviated as ‘Sec.’ in this figure. RM S(·)denotes the student reward model. the growing demand for more accessib...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "synthesizefeedbackFeedback input outputSelf-Knowledge outputinputinput YlabelLabelingExpansion X,Y demonstrationsexpandFeature featureinput,outputextractSec.4Sec.5 Sec.3.1Sec.3.2 Fig. 2: An overview of this survey on knowledge distillation of large language models...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "gaps in current techniques and proposing direc- tions for future research. Survey Organization. The remainder of this survey is orga- nized into several comprehensive sections, each designed to offer a deep dive into the multifaceted aspects of knowledge distillation within the realm ofLLMs. Following this intro- duction, §2 provides a foundational overview of knowledge distillation, comparing traditional techniques with those emerging in the era of LLMs and highlighting the role of data augment...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "es emerging, but there is still much to be learned from the era of Large Language Models (LLMs). In this section, we provide a foundational overview of knowledge distillation, highlighting the role of data augmentation (DA) in this context.\n",
      "\n",
      "Traditional techniques, such as supervised fine-tuning, have shown promise in distilling knowledge from LLMs. However, the increasing complexity of these models requires careful consideration of the trade-offs between accuracy and computational resources. To...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "includes discus- sions on natural language understanding (NLU), genera- tion (NLG), information retrieval, recommendation systems, and the evaluation of text generation. In §5, we ventureinto domain-specific vertical distillation, showcasing how knowledge distillation techniques are applied within spe- cialized fields such as law, healthcare, finance, and science, illustrating the practical implications and transformative impact of these approaches. The survey suggests open problems in §6, ident...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "mmendation systems, and the evaluation of text generation. In §5, we delve into domain-specific vertical distillation, demonstrating how knowledge distillation techniques are applied in specialized fields such as law, healthcare, finance, and science, highlighting their practical implications and transformative impact. The survey reveals open problems in §6, highlighting current challenges and gaps in knowledge distillation research that present opportunities for future work....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "process of transferring knowledge from a large, complex model (teacher) to a smaller, more efficient model (student) (Gou et al., 2021). This technique is pivotal in mitigating the challenges posed by the computational demands and resource constraints of deploying large-scale models in practical applications. Historically, knowledge distillation techniques, prior to the era of LLMs, primarily concentrated on transferring knowledge from complex, often cumbersome neural net- works to more compact ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "large, complex model to a smaller, more efficient model, mitigating the challenges of computational demands and resource constraints in deploying large-scale models in practical applications. This process, prior to the era of Large Language Models (LLMs), focused on compacting complex neural networks for deployment in resource-constrained environments, such as mobile devices or edge computing platforms, where computational efficiency was paramount....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "Mammoth (Yue et al., 2023a), Mixed Distill (Chenglin et al., 2023) ExpansionSelf-Instruct (Wang et al., 2022a), Alpaca (Taori et al., 2023), Code Alpaca (Chaudhary, 2023) Self-Align (Sun et al., 2024b), WizardLM (Xu et al., 2023a), WizardCoder (Luo et al., 2023a), WizardMath (Luo et al., 2023b), AugGPT (Dai et al., 2023a), TDG (He et al., 2023b) CurationUltraChat (Ding et al., 2023b), Phi-1 (Gunasekar et al., 2023), Phi-1.5 (Li et al., 2023a), Phi-2 (Mar, 2023), Magicoder (Wei et al., 2023), Wav...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "al., 2022a), Alpaca (Taori et al., 2023), Code Alpaca (Chaudhary, 2023) Self-Align (Sun et al., 2024b), WizardLM (Xu et al., 2023a), WizardCoder (Luo et al., 2023a), WizardMath (Luo et al., 2023b), AugGPT (Dai et al., 2023a), TDG (He et al., 2023b), CurationUltraChat (Ding et al., 2023b), Phi-1 (Gunasekar et al., 2023), Phi-1.5 (Li et al., 2023a), Phi-2 (Mar, 2023), Magicoder (Wei et al., 2023), WaveCoder (Yu et al., 2024), ZeroGen (Ye et al., 2022), InPars (Bonifacio et al., 2022)...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "(Chen et al., 2023a), GKD (Agarwal et al., 2024) Self-KnowledgeSelf-Instruct (Wang et al., 2022a), Self-Align (Sun et al., 2024b), RLCD (Yang et al., 2024a), ImpDistill (Jung et al., 2023), LMSI (Huang et al., 2023a), ReST (Gulcehre et al., 2023), Self-Rewarding (Yuan et al., 2024a), Baize (Xu et al., 2023b), STaR (Zelikman et al., 2022) DistillationSupervised Fine-TuningAlpaca (Taori et al., 2023), Vicuna (Chiang et al., 2023), WizardLM (Xu et al., 2023a), Self-Instruct (Wang et al., 2022a), Ba...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Self-Align (Sun et al., 2024b), RLCD (Yang et al., 2024a), ImpDistill (Jung et al., 2023), LMSI (Huang et al., 2023a), ReST (Gulcehre et al., 2023), Self-Rewarding (Yuan et al., 2024a), Baize (Xu et al., 2023b), STaR (Zelikman et al., 2022) DistillationSupervised Fine-TuningAlpaca (Taori et al., 2023), Vicuna (Chiang et al., 2023), WizardLM (Xu et al., 2023a), Self-Instruct (Wang et al., 2022a), Baize (Xu et al., 2023b), STaR (Zelikman et al., 2022), Divergence and SimilarityDistilGPT (Sanh et a...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "al., 2023), CycleAlign (Hong et al., 2023), Skill DistillationContext FollowingInstruction FollowingSelf-Instruct (Wang et al., 2022a), Alpaca (Taori et al., 2023), Vicuna (Chiang et al., 2023), WizardLM (Xu et al., 2023a), Orca (Mukherjee et al., 2023), Orca 2 (Mitra et al., 2023), WizardMath (Luo et al., 2023b), Llama-GPT4 (Peng et al., 2023a), Multi-turn DialogueVicuna (Chiang et al., 2023), Baize (Xu et al., 2023b), UltraLLaMA (Ding et al., 2023b), CAMEL (Li et al., 2023b), OpenChat (Wang et...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "ollowingInstruction FollowingSelf-Instruct Wang et al., 2022a, Alpaca Taori et al., 2023, Vicuna Chiang et al., 2023, WizardLM Xu et al., 2023a, Orca Mukherjee et al., 2023, Orca2 Mitra et al., 2023, WizardMath Luo et al., 2023b, Llama-GPT4 Peng et al., 2023a, Multi-turn Dialogue Chiang et al., 2023, Baize Xu et al., 2023b, UltraLLaMA Ding et al., 2023b, CAMEL Li et al., 2023b, OpenChat Wang et al., 2023c, Zephyr Tunstall et al., 2023, RAG Kang et al., 2023a, SAIL Luo et al., 2023c, Self-RAG Asa...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "(Lee et al., 2023a), Zephy (Tunstall et al., 2023), UltraFeedback (Cui et al., 2023a), ValueCAI (Bai et al., 2022a), Align Honesty (Yang et al., 2023a), SANDBOX (Liu et al., 2023b), Self-Align (Sun et al., 2024b), UltraFeedback (Cui et al., 2023a), RLCD (Yang et al., 2024a) AgentTool UsingToolformer (Schick et al., 2023), Graph-ToolFormer (Zhang, 2023), Gorilla (Patil et al., 2023), ToolAlpaca (Tang et al., 2023a), ToolLLM (Qin et al., 2023a), CRAFT (Yuan et al., 2023a), Confucius (Gao et al., 2...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "i et al., 2022a), Align Honesty (Yang et al., 2023a), SANDBOX (Liu et al., 2023b), Self-Align (Sun et al., 2024b), UltraFeedback (Cui et al., 2023a), RLCD (Yang et al., 2024a), AgentToolformer (Schick et al., 2023), Graph-ToolFormer (Zhang, 2023), Gorilla (Patil et al., 2023), ToolAlpaca (Tang et al., 2023a), ToolLLM (Qin et al., 2023a), CRAFT (Yuan et al., 2023a), Confucius (Gao et al., 2023b), MLLM-Tool (Wang et al., 2024), α-UMi (Shen et al., 2024), PlanningFireAct (Chen et al., 2023b), Agent...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "2022), NLGInheritSumm (Xu et al., 2023c), RECOMP (Xu et al., 2024b), MaRio (Ramnath et al., 2023), ID (Jung et al., 2023), GPT-3 Labeling (Wang et al., 2021b), BioGPT (Guo et al., 2023a), ChatGPT NMT (Yang and Nicolai, 2023), Information RetrievalQUILL (Srinivasan et al., 2022), Promptgator (Dai et al., 2023b), InPars (Bonifacio et al., 2022), AugTriever (Meng et al., 2023), (Sun et al., 2023a), RankVicuna (Pradeep et al., 2023a), RankZephyr (Pradeep et al., 2023b), ExaRanker (Ferraretto et al.,...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "al., 2023 GPT-3 Labeling Wang et al., 2021b BioGPT Guo et al., 2023a ChatGPT NMT Yang and Nicolai, 2023 Information RetrievalQUILL Srinivasan et al., 2022 Promptgator Dai et al., 2023b InPars Bonifacio et al., 2022 AugTriever Meng et al., 2023 Sun et al., 2023a RankVicuna Pradeep et al., 2023a RankZephyr Pradeep et al., 2023b ExaRanker Ferraretto et al., 2023 Recommendation NDR Mysore et al., 2023 InstrcutRec Zhang et al., 2023b ONCE Liu et al., 2023c Text Generation Evaluation PandaLM Wang et a...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "al., 2024), Code Clean (Jain et al., 2023), Multi-ModalityLLaVA (Liu et al., 2023e), SVIT (Zhao et al., 2023b), LVIS-Instruct4V (Wang et al., 2023e), Shikra (Chen et al., 2023c), LSKD (Park et al., 2023), DetGPT (Pi et al., 2023; Zhao et al., 2023c), LRV (Liu et al., 2023f), NExT-GPT (Wu et al., 2023b), Valley (Luo et al., 2023d), ILuvUI (Jiang et al., 2023d), StableLLaVA (Li et al., 2023c), PointLLM (Xu et al., 2023e), Verticalization DistillationLaw (Huang et al., 2023b; Cui et al., 2023b); Me...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "et al., 2023e), SVIT (Zhao et al., 2023b), LVIS-Instruct4V (Wang et al., 2023e), Shikra (Chen et al., 2023c), LSKD (Park et al., 2023), DetGPT (Pi et al., 2023; Zhao et al., 2023c), LRV (Liu et al., 2023f), NExT-GPT (Wu et al., 2023b), Valley (Luo et al., 2023d), ILuvUI (Jiang et al., 2023d), StableLLaVA (Li et al., 2023c), PointLLM (Xu et al., 2023e), Verticalization DistillationLaw (Huang et al., 2023b; Cui et al., 2023b); Medical & Healthcare (Zhang et al., 2023c; Chen et al., 2023d); Finance...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "earlier methods involved training a smaller student network to mimic the output of a larger teacher network, often through techniques like soft target training, where the student learns from the softened softmax output of the teacher. Please refer to the survey (Gou et al., 2021) for more details on general knowledge distillation techniques in AI and DL. In contrast, the advent of LLMs has revolutionized the knowledge distillation landscape. The current era of knowledge distillation in LLMs shif...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "r network, often through techniques like soft target training, where the student learns from the softened softmax output of the teacher.\n",
      "\n",
      "The distillation of knowledge from larger models to smaller ones is a technique used to improve the performance of AI models. In this context, distillation refers to the process of distilling the knowledge from a larger model into a smaller model, allowing it to learn from the teacher model's output.\n",
      "\n",
      "The current era of knowledge distillation in large language...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "replicate the output behavior of the teacher model or reduce the model size , the current focus in LLM-based knowledge distillation is to extract and transfer the rich, nuanced understanding that these models have developed. The key to this modern approach lies in heuristic and carefully designed prompts, which are used to elicit specific knowledge (Ding et al., 2023b) or capabilities (Chaudhary, 2023) from the LLMs. These prompts are crafted to tap into the LLM’s understanding and capabilities ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "size, the current focus in llm-based knowledge distillation is to extract and transfer the rich, nuanced understanding that these models have developed the key to this modern approach lies in carefully designed prompts that elicit specific knowledge or capabilities from the llms, tapping into their understanding and capabilities in various domains ranging from natural language understanding to more complex cognitive tasks like reasoning and problem-solving...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "of LLMs, where the models exhibit capabilities beyond their explicit training objectives. Furthermore, this era of knowledge distillation also em- phasizes the transfer of more abstract qualities such as reasoning patterns (Mitra et al., 2023), preference align- ment (Cui et al., 2023a), and value alignment (Sun et al., 2024b). This is in stark contrast to the earlier focus on output replication (Taori et al., 2023), indicating a shift towards a more holistic and comprehensive transfer of cognit...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "explicit training objectives. This era of knowledge distillation also emphasizes the transfer of abstract qualities such as reasoning patterns and preference alignment. This is in stark contrast to the earlier focus on output replication, indicating a shift towards a more holistic and comprehensive transfer of cognitive capabilities. The current techniques involve not just the replication of outputs, but also the emulation of thought processes and decision-making patterns of the teacher model. T...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "LLMs, Data Augmentation (DA) (Wang et al., 2022a; Ye et al., 2022) emerges as a critical paradigm integral to the process of knowledge distillation. Unlike traditional DA techniques such as paraphrasing (Gangal et al., 2022) orback-translation (Longpre et al., 2019), which primarily aim at expanding the training dataset in a somewhat mechanical manner. DA within the context of LLMs focuses on the generation of novel, context-rich training data tailored to specific domains and skills. This innova...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "llation, Unlike traditional techniques such as paraphrasing, or back-translation, which primarily aim at expanding the training dataset in a somewhat mechanical manner. DA within the context of LLMs focuses on the generation of novel, context-rich training data tailored to specific domains and skills. This innovation is driven by the unique capabilities of LLMs to generate coherent, diverse, and intricate data samples that closely mimic the nuanced understanding and cognitive abilities of human ...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "as a potent mechanism for bridging the knowl- edge and capability gap between proprietary and open- source models. Through DA, LLMs are prompted to create targeted, high-quality datasets that are not merely larger in volume but are also rich in diversity and specificity. This approach enables the distillation process to be more effec- tive, ensuring that the distilled models not only replicate the teacher model’s output behavior but also embody its deep-seated understanding and cognitive strateg...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "ource models, through Deep Learning Models (LLMs) are prompted to create targeted, high-quality datasets that are not merely larger in volume but also rich in diversity and specificity. This approach enables the distillation process to be more effective, ensuring that the distilled models replicate the teacher model's output behavior and embody its deep-seated understanding and cognitive strategies. The significance and necessity of Data Augmentation (DA) for achieving Knowledge Domains (KD) in ...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "pivotal shift towards a more efficient, sustainable, and accessible approach to harnessing the power of LLMs. It empowers open-source models with the ability to approximate the contextual adeptness, ethical alignment, and deep semantic insights characteristic of their proprietary counterparts, thereby democratizing access to advanced AI capabilities and fostering innovation across a broader spectrum of applications and users. 2.3 Survey Scope Building on the discussions introduced earlier, this ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "er of LLMs empowers open-source models with the ability to approximate the contextual adeptness, ethical alignment, and deep semantic insights characteristic of their proprietary counterparts thereby democratizing access to advanced AI capabilities and fostering innovation across a broader spectrum of applications and users 2 3 Survey Scope Building on the discussions introduced earlier this survey aims to comprehensively explore the landscape of knowledge distillation within the context of LLMs...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "distillation. KD Algorithms. This segment focuses on the technical foundations and methodologies of knowledge distillation. It includes an in-depth exploration of the processes involved in constructing knowledge from teacher models (e.g., pro- prietary LLMs) and integrating this knowledge into student models (e.g., open-source LLMs). Under the umbrella of ‘knowledge ’, we delve into strategies such as labeling (Hsieh et al., 2023), expansion (Taori et al., 2023), curation (Gu- nasekar et al., 20...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "undations and methodologies of knowledge distillation. It includes an in-depth exploration of processes involved in constructing knowledge from teacher models (e.g., proprietary LLMs) and integrating this knowledge into student models (e.g., open-source LLMs). Under the umbrella of 'knowledge', we delve into strategies such as labeling, expansion, curation, feature understanding, and feedback mechanisms. The exploration seeks to uncover the various ways in which knowledge can be identified, expa...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "et al., 2023a), and rank optimization strategies (Tunstall et al., 2023). This analysis aims to illuminate how these algorithms facilitate the trans- fer of knowledge, ensuring that open-source models can replicate and, in some cases, surpass the capabilities of their proprietary counterparts. Skill Distillation. This facet examines the specific compe- tencies and capabilities enhanced through KD. It encom- passes detailed discussions on context following (Taori et al., 2023; Luo et al., 2023c),...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "ow algorithms enable knowledge transfer, allowing open-source models to replicate and sometimes surpass proprietary capabilities. Skill Distillation examines specific competencies and capabilities enhanced through Knowledge Distillation. Contextual discussions follow (Taori et al., 2023; Luo et al., 2023c), including instruction following and retrieval-augmented generation (RAG) capabilities. Alignment research investigates thinking patterns, persona/preference modeling, and value alignment. The...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "lan- guage generation (NLG), information retrieval, recommen- dation systems, text generation evaluation, and code gen- eration. Finally, the survey addresses multi-modality (Liu et al., 2023e; Zhao et al., 2023b), exploring how KD enhances LLMs’ ability to interpret and integrate multiple forms of input, enriching their utility and applicability across various contexts. Verticalization Distillation. This section assesses the ap- plication of KD across diverse vertical domains, offering insights...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "tion, and Code Generation**\n",
      "\n",
      "Finally, the survey explores how Knowledge Distillation (KD) enhances Large Language Models (LLMs) in interpreting and integrating multiple forms of input, enriching their utility and applicability across various contexts. Verticalization Distillation\n",
      "This section examines the application of KD across diverse domains, providing insights into how distilled LLMs can be tailored for specialized fields such as Law, Medical & Healthcare (Wang et al., 2023a), Finance (Zhan...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "meet the nuanced demands of different industries, thus contributing to the broader AI and ML ecosystem. By navigating through these facets, this survey en- deavors to provide an extensive and nuanced analysis of knowledge distillation in the era of LLMs. It serves as a guide for researchers, practitioners, and enthusiasts in the field, shedding light on current methodologies, challenges, and opportunities for innovation in this rapidly evolving domain. Declaration. This survey represents our ear...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "stem. by navigating through these facets, this survey endeavors to provide an extensive and nuanced analysis of knowledge distillation in the era of LLMs. it serves as a guide for researchers, practitioners, and enthusiasts in the field, shedding light on current methodologies, challenges, and opportunities for innovation in this rapidly evolving domain....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "foundational paradigms of knowledge dis- tillation, highlighting key methodologies and their impacts across a range of applications. 2.4 Distillation Pipeline in LLM Era SeedKnowledgeSkill/Domain TeacherLLMKnowledgeElicitationStudentModelDistillationAlgorithmsteer driveGeneratedKnowledgeLearningObjectivetrain Fig. 4: An illustration of a general pipeline to distill knowl- edge from a large language model to a student model. The general distillation pipeline of LLMs is a structured and methodical...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "across a range of applications.\n",
      "\n",
      "Distillation Pipeline in LLM Era\n",
      "---------------------------\n",
      "\n",
      "The Distillation Pipeline is a structured and methodical process aimed at transferring knowledge from a sophisticated teacher model to a less complex student model. This pipeline is integral for leveraging the advanced capabilities of models like GPT-4 or Gemini in more accessible and efficient open-source counterparts.\n",
      "\n",
      "Stages of Distillation Pipeline\n",
      "-----------------------------\n",
      "\n",
      "1.  **Knowledge Eli...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "seen in Figure 2. I. Target Skill or Domain Steering Teacher LLM. The first stage involves directing the teacher LLM towards a specific target skill or domain. This is achieved through care- fully crafted instructions or templates that guide the LLM’s focus. These instructions are designed to elicit responses that demonstrate the LLM’s proficiency in a particular area, be it a specialized domain like healthcare or law, or a skill such as reasoning or language understanding. The objective here is...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "ards a specific target skill or domain This is achieved through carefully crafted instructions or templates that guide the LLM's focus These instructions are designed to elicit responses that demonstrate the LLM's proficiency in a particular area be it a specialized domain like healthcare or law or a skill such as reasoning or language understanding The objective here is to utilize the teacher LLM's extensive training and nuanced capabilities to generate outputs that are rich in the specific kno...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "to generate more elaborate and detailed outputs based on this initial infor- mation. The seed knowledge is crucial as it provides a foundation upon which the teacher model can build and expand, thereby creating more comprehensive and in-depth knowledge examples. III. Generation of Distillation Knowledge. In response to the seed knowledge and steering instructions, the teacher LLM generates knowledge examples. These examples are predominantly in the form of question-and-answer (QA) dialogues or n...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "his initial information the teacher model generates knowledge examples predominantly in the form of question-and-answer dialogues or narrative explanations aligning with the natural language processing understanding capabilities of the 7 LLM these examples are typically in the form of explanations or narratives addressing various topics thereby creating more comprehensive and in-depth knowledge examples the generated knowledge examples constitute the core of the distillation knowledge encapsulat...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "Specific Learn- ing Objective. The final stage involves the utilization of the generated knowledge examples to train the student model. This training is guided by a loss function that aligns with the learning objectives. The loss function quantifies the student model’s performance in replicating or adapting the knowledge from the teacher model. By minimizing this loss, the student model learns to emulate the target skills or domain knowledge of the teacher, thereby acquiring similar capabilities...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "knowledge examples to train the student model. This training is guided by a loss function that aligns with the learning objectives. The loss function quantifies the student model's performance in replicating or adapting the knowledge from the teacher model. By minimizing this loss, the student model learns to emulate the target skills or domain knowledge of the teacher, thereby acquiring similar capabilities....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "domain to steer the LLM and elicit knowledge, s∼ S denotes an example of the seed knowledge, upon which the LLM can explore to generate novel knowledge, Parse( o, s)stands for to parse the distillation example ( e.g., (x, y)) from the teacher LLM’s output o(plus the input sin some cases), andpTrepresents the teacher LLM with parameters θT. Given the datasets D(kd) Ibuilt for distillation, we then define a learning objective as L=X ILI(D(kd) I;θS), (2) whereP Idenotes there could be multiple task...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "which the LLM can explore to generate novel knowledge, Parse( o, s)stands for to parse the distillation example ( e.g., (x, y)) from the teacher LLM’s output o(plus the input sin some cases), andpTrepresents the teacher LLM with parameters θT. Given the datasets D(kd) Ibuilt for distillation, we then define a learning objective as L=X ILI(D(kd) I;θS), (2) where P Idenotes there could be multiple tasks or skills being distilled into one student model, LI(·;·)stands for a specific learning objecti...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "it is categorized into two principal steps: ‘Knowledge,’ focusing on eliciting knowledge from teacher LLMs (Eq.1), and ‘Distillation,’ centered on injecting this knowledge into student models (Eq.2). We will elaborate on these two processes in the subsequent sections. 3.1 Knowledge This section focuses on the approaches to elicit knowledge from teacher LLMs. According to the manners to acquire knowledge, we divided them into Labeling ,Expansion ,DataCuration ,Feature ,Feedback , and Self-Knowled...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "dataset and feeding it into LLMs to obtain the desired generations. Moreover, the generation of yis controllable through the predefined Iandc. This process can be formulated as follows: D(lab)={x, y|x∼ X, y∼pT(y|I⊕c⊕x)}. (3) Input xcould be sourced from existing NLP task datasets, which serve as typical reservoirs for distillation efforts. Numerous works have sought to harness the capa- bilities of powerful LLMs as teachers for annotating dataset samples across a range of tasks. For instance, ef...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "is process can be formulated as follows: D(lab)={x, y|x∼ X, y∼pT(y|I⊕c⊕x)}....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "al., 2023; Li et al., 2022; Ho et al., 2023; Magister et al., 2023; Fu et al., 2023; Ramnath et al., 2023; Li et al., 2023d; Liu et al., 2023g), among others. Rather than concentrating on specific tasks, many current works focus on labeling outputs based on instructions, thereby teaching student models to solve tasks in a more flexible way by following in- structions. Collections of various NLP tasks, complemented by instructional templates, serve as valuable input sources forx. For instance, FL...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "works concentrate on labeling outputs based on instructions, teaching student models to solve tasks in a more flexible way by following instructions. Collections of various NLP tasks, complemented by instructional templates, serve as valuable input sources for training models. For instance, FLAN-v2 collections provide extensive publicly available sets of tasks with labeled responses from teacher LLMs, built from predefined templates that lack diversity and may have gaps between human queries....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "powerful LLMs, like ShareGPT. Additionally, Xu et al. (2023b) and Anand et al. (2023) label the real questions sampled from forums like Quora and Stack Overflow. Moreover, the process of labeling could be guided by instructions Ior demonstrations c. A commonly used in- struction type for guiding labeling is chain-of-thought (CoT) prompt (Hsieh et al., 2023; Fu et al., 2023; Magister et al., 2023). Mukherjee et al. (2023) add multiple system messages (e.g. “You must generate a detailed and long a...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "023b) and Anand et al. (2023) label the real questions sampled from forums like Quora and Stack Overflow. Moreover, the process of labeling could be guided by instructions or demonstrations. A commonly used instruction type for guiding labeling is the chain-of-thought (CoT) prompt. Mukherjee et al. (2023) add multiple system messages (e.g. “You must generate a detailed and long answer.” or “explain like I’m five, think step-by-step”) to elicit rich signals. Yue et al. (2023a) and Chenglin et al....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "Generate≻≻𝑦\" 𝑦! 𝑦# 𝑥 𝑥& CorrectExpand𝑐 Fig. 5: An illustration of different knowledge elicitation methods from teacher LLMs. Labeling : The teacher generates the output from the input; Expansion : The teacher generates samples similar to the given demonstrations through in- context learning; Data Curation : The teacher synthesizes data according to meta-information, such as a topic or an entity; Feature : Feed the data into the teacher and extract its internal knowledge, such as logits and featu...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "utput from input; Teacher generates samples similar to given demonstrations through in-context learning; Data is curated according to meta-information such as topic or entity; Data is fed into the teacher to extract knowledge such as logits and features; Teacher provides feedback on student's output such as preferences, corrections, and expansions of challenging samples; Student generates outputs which is then filtered for high-quality or evaluated by student itself\"...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "Overflow. 3.1.2 Expansion While the labeling approach is simple and effective, it faces certain limitations. Primarily, it is constrained by the scale and variety of the input data. In real-world applications, especially those involving user conversations, there are also concerns regarding the privacy of the data involved. To address these limitations, various expansion methods have been proposed (Wang et al., 2022a; Taori et al., 2023; Chaud- hary, 2023; Si et al., 2023; Ji et al., 2023a; Luo e...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "s constrained by the scale and variety of the input data. In real-world applications, especially those involving user conversations, there are concerns regarding the privacy of the data involved. Various expansion methods have been proposed to address these limitations. These methods take the demonstrations as seed knowledge and aim to expand a large scale and diverse data by in-context learning....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "the existing dataset, in the expansion approach, both x andyare generated by teacher LLMs. This process can be formulated as follows: D(exp)={(x, y)|x∼pT(x|I⊕c), y∼pT(y|I⊕x)}.(4) In this formulation, xand yrepresent the new input- output pairs generated by the teacher LLM. The input x is generated based on a set of input-output demonstrations c. The output yis then generated in response to the new input xunder the guidance of an instruction I. Note thatthe demonstrations could be predefined or d...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "subsequent expansion iterations. Subsequently, Taori et al. (2023) applies this ex- pansion method to a more powerful teacher LLM, text- davinci-003, to distill 52K high-quality data. To improve the diversity and coverage during expansion, Wu et al. (2023c) and (Sun et al., 2024b) prompt the teacher LLM to generate instructions corresponding to some specific topics. Xu et al. (2023a) propose an Evol-Instruct method to ex- pand the instructions from two dimensions: difficulty (e.g. rewriting the ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "to a more powerful teacher LLM, text- davinci-003, to distill 52K high-quality data. To improve the diversity and coverage during expansion, Wu et al. (2023c) and (Sun et al., 2024b) prompt the teacher LLM to generate instructions corresponding to some specific topics. Xu et al. (2023a) propose an Evol-Instruct method to expand the instructions from two dimensions: difficulty (e.g. rewriting the question to be more complex) and diversity (e.g. generating more long-tailed instructions). This Evol...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "multi- ple conceptually similar, but semantically varied, samples to improve classification performance. Similarly, TDG (He et al., 2023b) proposes the Targeted Data Generation (TDG) framework, which automatically identifies challenging sub- groups within data and generates new samples for these subgroups using LLMs through in-context learning. In summary, the expansion method leverages the in- 9 context learning strengths of LLMs to produce more var- ied and extensive datasets with both inputs ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "TDG framework leverages LLMs' strengths in in-context learning to generate varied and extensive datasets, but quality and diversity rely heavily on teacher LLMs and initial seed demonstrations, leading to bias and homogeneity issues...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "data. 3.1.3 Data Curation The pursuit of high-quality and scalable data generation in knowledge distillation from LLMs has led to the emergence of the Data Curation approach. This method arises in re- sponse to the limitations observed in both the Labeling and Expansion approaches. These methods often yield data of variable quality and face constraints in quantity. In Labeling, the seed knowledge is sourced from task datasets, leading to potential noise and dirty data. Meanwhile, in Expansion, t...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "ed to the emergence of the Data Curation approach. This method arises in response to the limitations observed in both the Labeling and Expansion approaches. These methods often yield data of variable quality and face constraints in quantity.\n",
      "\n",
      "In Labeling, the seed knowledge is sourced from task datasets, leading to potential noise and dirty data. Meanwhile, in Expansion, the input data is derived from seed demonstrations, which can result in homogeneous data when generated in large quantities. T...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "approach to synthesize data from scratch. Numerous diverse meta- information, such as topics or knowledge points, could be incorporated into this process to generate controllable x andy. Thus, this process can be meticulously controlled to yield datasets that are not only large in scale but also of high quality. The formulation for Data Curation can be represented as: D(cur)={(x, y)|x∼pT(x|I⊕m), y∼pT(y|I⊕x)}.(5) In this formulation, mrepresents the diverse meta- information used to guide the syn...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "edge points, could be incorporated into this process to generate controllable output. Thus, this process can be meticulously controlled to yield datasets that are not only large in scale but also of high quality. The formulation for Data Curation can be represented as: D(cur)={(x, y)|x∼pT(x|I⊕m), y∼pT(y|I⊕x)}. In this formulation, mrepresents the diverse meta-information used to guide the synthesis of x, and Iis the instruction guiding teacher LLMs to generate xory. Different studies primarily v...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "the World , they explore 30 meta-topics like ”Technology” and ”Food and Drink.” the teacher LLMs then use this meta-information to distill a broad array of instructions and conversations, achieving a substantial scale of 1.5 million instances. UltraChat stands out with its lexical and topical diversity. The UltraLLaMA model, fine- tuned on this data, consistently surpasses other open-source models. Another notable series, phi(Gunasekar et al., 2023; Li et al., 2023a; Mar, 2023), focuses on disti...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "ion to distill a broad array of instructions and conversations, resulting in a substantial scale of 1.5 million instances. UltraChat stands out with its lexical and topical diversity, fine-tuned on this data to consistently surpass other open-source models....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "tokens of Python exercises with solutions. Remarkably, thephi-1 model, despite its smaller size, outperforms nearly all open-source models on coding benchmarks like Hu- manEval and MBPP while being 10 times smaller in model size and 100 times smaller in dataset size. MFTCoder (Liu et al., 2023d) utilizes hundreds of Python knowledge points as meta-information to create a CodeExercise Dataset. In contrast, Magicoder (Wei et al., 2023) and WaveCoder (Yu et al., 2024) get raw code collections from ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "el outperforms nearly all open-source models on coding benchmarks like HumanEval and MBPP while being 10 times smaller in model size and 100 times smaller in dataset size. MFTCoder (Liu et al., 2023) utilizes hundreds of Python knowledge points as meta-information to create a CodeExercise Dataset. In contrast, Magicoder (Wei et al., 2023) and WaveCoder (Yu et al., 2024) generate instructional data from open-source code collections using this as meta-information for data augmentation. In the cont...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "et al., 2022; Meng et al., 2023). In conclusion, Data Curation through teacher LLMs has emerged as a promising technique for synthesizing datasets that are not only high-quality and diverse but also large in scale. The success of models like phi-1 in specialized domains underscores the efficacy of this method. The ability to create synthetic datasets will become a crucial technical skill and a key area of focus in AI (Li et al., 2023a). 3.1.4 Feature The previously discussed knowledge elicitatio...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "a promising technique for synthesizing datasets that are not only high-quality and diverse but also large in scale. The success of models like phi-1 in specialized domains underscores the efficacy of this method. The ability to create synthetic datasets will become a crucial technical skill and a key area of focus in AI....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "with fewer than 1 billion parameters (cf. Gou et al. (2021) for detail). However, recent research has begun to explore white-box distillation in the context of generative LLMs (Timiryasov and Tastet, 2023; Liang et al., 2023a; Gu et al., 2024; Agarwal et al., 2024; Liu et al., 2023a; Wen et al., 2023; Wan et al., 2024a; Zhao and Zhu, 2023; Qin et al., 2023b; Boizard et al., 2024; Zhong et al., 2024). The typical method for acquiring this feature knowledge involves teacher LLMs annotating the out...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "to explore white-box distillation in the context of generative LLMs (Timiryasov and Tastet, 2023; Liang et al., 2023a; Gu et al., 2024; Agarwal et al., 2024; Liu et al., 2023a; Wen et al., 2023; Wan et al., 2024a; Zhao and Zhu, 2023; Qin et al., 2023b; Boizard et al., 2024; Zhong et al., 2024). typically involves teacher LLMs annotating the output sequence y with its internal representations. these annotations are then distilled into the student model using methods such as kullback-leibler diver...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "(such as output distri- bution) from the teacher LLM. 10 The most straightforward method to elicit feature knowl- edge of teacher is to label a fixed dataset of sequences with token-level probability distributions (Sanh et al., 2019; Wen et al., 2023). To leverage the rich semantic and syntactic knowledge in intermediate layers of the teacher model, TED (Liang et al., 2023a) designs task-aware layer-wise distillation. They align the student’s hidden representations with those of the teacher at e...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "d to elicit feature knowledge of teacher is to label a fixed dataset of sequences with token-level probability distributions. TED (Liang et al., 2023a) designs task-aware layer-wise distillation. They align the student's hidden representations with those of the teacher at each layer, selectively extracting knowledge pertinent to the target task. Gu et al. (2024) and Agarwal et al. (2024) introduce a novel approach where the student model generates sequences, termed'self-generated sequences'. The...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "distilling feature knowledge from teacher LLMs have been proposed (Tao et al., 2022a; Liu et al., 2023a; Kim et al., 2023b). These methods aim to preserve the original output distribution when quantizing the LLMs, ensuring minimal loss of performance. Additionally, feature knowledge could serve as a potent source for multi-teacher knowledge distil- lation. Timiryasov and Tastet (2023) leverages an ensemble of GPT-2 and LLaMA as teacher models to extract output distributions. Similarly, FuseLLM (...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "n when quantizing LLMs, ensuring minimal loss of performance. Additionally, feature knowledge could serve as a potent source for multi-teacher knowledge distillation....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "knowledge from teacher LLMs, such as output distributions and intermediate layer features, white- box approaches enable a more nuanced transfer of informa- tion. While showing promise, especially in smaller models, its application is not suitable for black-box LLMs where internal parameters are inaccessible. Furthermore, student models distilled from white-box LLMs may underperform compared to their black-box counterparts, as the black-box teacher LLMs (e.g. GPT-4) tend to be more powerful. 3.1....\n",
      "\n",
      "PROCESSED TEXT:\n",
      "ox approaches enable a more nuanced transfer of information. While showing promise, especially in smaller models, its application is not suitable for black-box LLMs where internal parameters are inaccessible. Furthermore, student models distilled from white-box LLMs may underperform compared to their black-box counterparts, as black-box teacher LLMs tend to be more powerful. 3.1.5 Feedback Most previous works focus on one-way knowledge transfer from the teacher to the student for imitation, with...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "through Reinforcement Learning from AI Feedback (RLAIF) (Bai et al., 2022a). Here is a generalized formulation for eliciting feedback knowledge: D(fb)={(x, y, ϕ fb(x, y;θT))|x∼ X, y∼pS(y|x)}, (7) where ydenotes the output generated by the student model in response to x, and ϕfb(·;θT))represents providing feedback from teacher LLMs. This operation evaluates thestudent’s output ygiven the input x, by offering assess- ment, corrective information, or other forms of guidance. This feedback knowledge...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "2022a). This generalized formulation for eliciting feedback knowledge involves the following steps: \n",
      "\n",
      "1. D(fb)={(x, y, ϕ fb(x, y;θT))|x∼ X, y∼pS(y|x)}, where ydenotes the output generated by the student model in response to x, and ϕfb(·;θT))represents providing feedback from teacher LLMs. This operation evaluates the student’s output ygiven the input x, by offering assessment, corrective information, or other forms of guidance. This feedback knowledge enables the student to refine its responses ...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "2023; Lee et al., 2023a). Preference, as previously discussed, represents a notable form of feedback knowledge from teacher models. Various knowledge of preferences could be distilled from teachers by prompting it with specific criteria. Bai et al. (2022a) in- troduce RLAIF for distilling harmlessness preferences from LLMs. This involves using an SFT-trained LLM to generate response pairs for each prompt, then ranking them for harmlessness to create a preference dataset. This dataset is distille...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "g proposed to distill them from teacher models. One notable approach is the use of RLAIF, which involves generating response pairs for each prompt and ranking them for harmlessness to create a preference dataset. This dataset is then used to train a more harmless LLM policy, such as Wizard- Math (Luo et al., 2023b), which focuses on mathematical reasoning. To further improve the quality of distilled preference data, researchers have developed the UltraFeedback dataset, a large-scale collection o...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "various instructions and models to produce comparative data. Then, GPT-4 is used to score candidates from various aspects of preference, including instruction-following, truthfulness, honesty and helpfulness. Beyond merely assessing student generations, teachers can also furnish extensive feedback on instances where students underperform. In Lion (Jiang et al., 2023b), teacher model pinpoints instructions that pose challenges to the student model, generating new, more difficult instructions aime...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "s from various aspects of preference, including instruction-following, truthfulness, honesty and helpfulness. Beyond merely assessing student generations, teachers can also furnish extensive feedback on instances where students underperform. In Lion (Jiang et al., 2023b), teacher model pinpoints instructions that pose challenges to the student model, generating new, more difficult instructions aimed at bolstering the student’s abilities. PERsD (Chen et al., 2023a) showcases a method where teache...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "teacher model’s distribution over the student’s generations can itself act as a form of feedback. MiniLLM (Gu et al., 2024) and GKD (Agarwal et al., 2024) present an innovative strategy wherein the student model initially generates sequences, followed by teacher model producing an output distribution as feedback. This method leverages the teacher’s insight to directly inform and refine the student model’s learning process. 3.1.6 Self-Knowledge The knowledge could also be elicited from the studen...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "iniLLM and GKD present an innovative strategy wherein the student model generates sequences, followed by the teacher model producing an output distribution as feedback. This method leverages the teacher’s insight to directly inform and refine the student model’s learning process. 3.1.6 Self-Knowledge The knowledge can be elicited from the student itself, which we refer to as Self-Knowledge. In this setting, the same model acts both as the teacher and the student, iteratively improving itself by ...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "self-knowledge could be formulated as: D(sk)={(x, y, ϕ sk(x, y))|x∼ S, y∼pS(y|I⊕x)},(8) where ϕsk(·)is a generalized function that represents an additional process to the self-generated outputs y, which could include but is not limited to filtering, rewarding, or any other mechanisms for enhancing or evaluating y. It could be governed by external tools or the student itself θS. Recent research in this area has proposed various innovative methodologies to elicit self-knowledge, demonstrating its ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "at represents an additional process to the self-generated outputs y, which could include but is not limited to filtering, rewarding, or any other mechanisms for enhancing or evaluating y....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "which utilizes GPT-3 for data augmentation through the Expansion approach, gen- erating additional data samples to enhance the dataset. This enriched dataset subsequently fine-tunes the original model. Other methods aim to elicit targeted knowledge from student models by modifying prompts, and leveraging these data for further refinement. In Self-Align (Sun et al., 2024b), they find that models fine-tuned by Self-Instruct data tend to generate short or indirect responses. They prompt this model ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "es to enhance the model's capabilities. This process fine-tunes the original model, allowing it to produce more accurate and detailed responses. Other methods aim to elicit targeted knowledge from student models by modifying prompts and leveraging the data for further refinement....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "reinforcement learning. Several other approaches employ filtering methods to refine self-generated data. For exam- ple, Impossible Distillation (Jung et al., 2023) targets sen- tence summarization tasks, implementing filters based on entailment, length, and diversity to screen self-generated summaries. LMSI (Huang et al., 2023a) generates multiple CoT reasoning paths and answers for each question, and then retains only those paths that lead to the most consistent answer. Note that refined self-k...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "f-generated data. For instance, Impossible Distillation targets sentence summarization tasks, implementing filters based on entailment, length, and diversity to screen self-generated summaries. LMSI generates multiple CoT reasoning paths and answers for each question, and retains only those paths that lead to the most consistent answer. This process enables refined self-knowledge to be iteratively acquired as the student model improves further enhancing its capabilities....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "and filtered using a scoring function. Subsequently, the lan- guage model undergoes fine-tuning on this curated dataset,employing an offline RL objective. Self-Play (Chen et al., 2024a) introduces a framework resembling iterative DPO, where the language model is fine-tuned to differentiate the self-generated responses from the human-annotated data. These self-generated responses could be seen as “negative knowledge” to promote the student to better align with the target distribution. Self-Reward...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "this curated dataset, employing an offline RL objective. Self-Play (Chen et al., 2024a) introduces a framework resembling iterative DPO, where the language model is fine-tuned to differentiate the self-generated responses from the human-annotated data. These self-generated responses could be seen as “negative knowledge” to promote the student to better align with the target distribution. Self-Rewarding (Yuan et al., 2024a) explores a novel and promising approach by utilizing the language model i...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "range of distillation tech- niques, from the strategies that enhance imitation by Su- pervised Fine-Tuning ,Divergence and Similarity , to advanced methods like Reinforcement Learning and Rank Optimization , as shown in Figure 3. 3.2.1 Supervised Fine-Tuning Supervised Fine-Tuning (SFT), or called Sequence-Level KD (SeqKD) (Kim and Rush, 2016), is the simplest and one of the most effective methods for distilling powerful black-box LLMs. SFT finetunes student model by maximizing the like- lihood ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "ivergence and similarity, to advanced methods like reinforcement learning and rank optimization, as shown in Figure 3.2.1. Supervised fine-tuning, or sequence-level knowledge distillation (SeqKD), is a simple yet effective method for distilling powerful black-box LLMs....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "LLMs (Taori et al., 2023; Chiang et al., 2023; Wu et al., 2023c; Xu et al., 2023a; Luo et al., 2023b). Additionally, SFT has been ex- plored in many self-distillation works (Wang et al., 2022a; Huang et al., 2023c; Xu et al., 2023b; Zelikman et al., 2022). Due to the large number of KD works applying SFT, we only list representative ones here. More detailed works can be found in §4. 3.2.2 Divergence and Similarity This section mainly concentrates on algorithms designed for distilling feature kno...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "works....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "log2p(t) p(t)+q(t)+Pq(t) log2q(t) p(t)+q(t)\u0011 TABLE 1: Functional forms of Dfor various divergence types. p: reference Similarity Function LF Expression L2-Norm Distance ∥ΦT(fT(x, y))−ΦS(fS(x, y))∥2 L1-Norm Distance ∥ΦT(fT(x, y))−ΦS(fS(x, y))∥1 Cross-Entropy Loss −PΦT(fT(x, y)) log(Φ S(fS(x, y))) Maximum Mean Discrepancy MMD (ΦT(fT(x, y)),ΦS(fS(x, y))) TABLE 2: Summary of similarity functions in knowledge distillation. and student models, represented by a general divergence function D: LDiv= E x∼...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "types\n",
      "p: reference Similarity Function L2-Norm Distance ∥ΦT(fT(x, y))−ΦS(fS(x, y))∥2 L1-Norm Distance ∥ΦT(fT(x, y))−ΦS(fS(x, y))∥1 Cross-Entropy Loss −PΦT(fT(x, y)) log(Φ S(fS(x, y))) Maximum Mean Discrepancy MMD (ΦT(fT(x, y)),ΦS(fS(x, y)))...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "modes of pT. However, when a student model is unable to learn all modes of a highly complex teacher, the re- sultant “mode-covering” behavior might cause the student to assign probability mass to tokens with low probability under the teacher’s distribution (cf. Figure 6 blue curve). This mode-covering phenomenon can potentially lead to hallucinations and low-quality generations. Alternatively, mode-seeking divergences like reverse KL prioritize tokens where the teacher assigns high probabilities...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "probability mass to tokens with low probability under the teacher's distribution. This can result in hallucinations and low-quality generations. \n",
      "\n",
      "mode-seeking divergences, such as reverse KL, prioritize tokens with high probabilities, mitigating the risk of low-quality outputs. However, they often come at the cost of reduced diversity. Gu et al. (2024) use policy gradient methods to optimize for this approach, while Agarwal et al. (2024) and Sason and Verd´u (2016) assess the efficacy of differ...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "distillation, finding the optimal divergence to be task-dependent. For instance, forward KL divergence is more suitable for tasks like Machine Translation, where the output has fewer modes or variations, while reverse KL divergence is preferable for tasks like dialogue generation and instruction tuning, which involve multiple modes and a wider range of potential responses. Thus, the nature of the task significantly influences the selection of the divergence function for optimal performance. Simi...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "nce is more suitable for tasks like machine translation, where the output has fewer modes or variations, while reverse KL divergence is preferable for tasks like dialogue generation and instruction tuning, which involve multiple modes and a wider range of potential responses. Thus, the nature of the task significantly influences the selection of the divergence function for optimal performance. Similarity-based methods in knowledge distillation aim to align the hidden states or features of the st...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "“mode-seeking” behavior. model with those of the teacher. These methods use various similarity metrics to measure and optimize the congruence of internal representations between the two models. The objective is to ensure that the student model not only produces similar outputs to the teacher but also processes information in a comparable manner. The formulation for a similarity-based objective might look like this: LSim= E x∼X,y∼Y[LF(ΦT(fT(x, y)),ΦS(fS(x, y)))],(11) where fT(x, y)andfS(x, y)are ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "ntations...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "task-aware filters. These filters are designed to selectively capture the most pertinent informa- tion for a specific task from the teacher model. The key objective is to minimize the discrepancy between the filtered representations in both teacher and student models. While similarity-based approaches are common in encoder-based LMs (Sun et al., 2019, 2020; Jiao et al., 2020; Hou et al., 2020; Zuo et al., 2022; Liang et al., 2021), their application in LLM knowledge distillation is not as widesp...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "on for a specific task from the teacher model. The key objective is to minimize the discrepancy between the filtered representations in both teacher and student models. While similarity-based approaches are common in encoder-based LMs, their application in LLM knowledge distillation is not as widespread. However, considering their effectiveness, we anticipate an increase in research exploring these methods for LLM distillation in the near future.\n",
      "\n",
      "3.2.3 Reinforcement Learning This section explor...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "2024b; Ma et al., 2023a; Pang et al., 2023; Du et al., 2023a). The RL-based distillation process typically involves two main stages: 13 Distilled Reward Model Training. The first stage involves training a reward model rϕusing the feedback data D(fd) generated by teacher LLMs. Preference data, as one of the typical feedback, is employed to train the student reward model (Bai et al., 2022a; Cui et al., 2023a; Lee et al., 2023a; Kim et al., 2023a). They usually consist of input-output pairs (x, yw,...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "3 Distilled Reward Model Training. First stage involves training a reward model ϕ using feedback data D(fd) generated by teacher LLMs. Preference data, one of typical feedback, is used to train the student reward model. This typically consists of input-output pairs (x, yw, yl). Here, ywandyl represent \"winning\" and \"losing\" outputs relative to the teacher's preferences. Loss function for the reward model is defined as: LRM(rϕ,D(fd)) = - E (x,yw,yl) ∼D(fd)[logσ(rϕ(x, yw) - rϕ(x, yl))]...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "model. It is trained on an erroneous solution rewriting data distilled from a teacher LLM. This distilled reward model can pro- duce token-level rewards for RL training. Reinforcement Learning Optimization. In the second stage, the student model, represented by a policy πθ, is optimized to maximize the expected reward as per the trained reward model. Simultaneously, it minimizes the divergence from a reference policy πref, typically the initial policy of the student model trained by SFT, control...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "reward model can pro- duce token-level rewards for RL training. Reinforcement Learning Optimization. In the second stage, the student model, represented by a policy πθ, is optimized to maximize the expected reward as per the trained reward model. Simultaneously, it minimizes the divergence from a reference policy πref, typically the initial policy of the student model trained by SFT, controlled by a factor β. The RL objective is given by: max πθE x∼X,y∼πθ(y|x)[rϕ(x, y)]−βDKL[πθ(y|x)∥πref(y|x)] (...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "reward model to directly assign rewards during RL, circumventing the need for training a reward model (Lee et al., 2023a; Kwon et al., 2023). While this approach may exhibit superior performance, it comes at a higher computational cost compared to employing a smaller distilled reward model. 3.2.4 Ranking Optimization Ranking optimization presents a stable and computationally efficient alternative to RL for injecting preference feedback into language models (Rafailov et al., 2023; Song et al., 20...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "del\n",
      "\n",
      "While this approach may exhibit superior performance, it comes at a higher computational cost compared to employing a smaller distilled reward model\n",
      "\n",
      "Ranking optimization presents a stable and computationally efficient alternative to RL for injecting preference feedback into language models\n",
      "\n",
      "This method, diverging from traditional RL approaches, directly incorporates ranking information into language models from a fixed preference dataset during fine-tuning\n",
      "\n",
      "Intuitively, it directly updates...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "ranking optimization todistill teacher’s preferences into student models (Tunstall et al., 2023; Hong et al., 2023; Yuan et al., 2024a). Zephyr (Tunstall et al., 2023) utilizes Direct Preference Optimization (DPO) (Rafailov et al., 2023) to distill the preference alignment in teacher LLMs. DPO streamlines the objective of reinforcement learning (as in Eq. 13), which involves reward maximization with a KL-divergence constraint, into a single-stage policy training. Specifically, DPO’s training goa...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "ng et al., 2023; Yuan et al., 2024a). Zephyr (Tunstall et al., 2023) utilizes Direct Preference Optimization (DPO) (Rafailov et al., 2023) to distill the preference alignment in teacher LLMs. DPO streamlines the objective of reinforcement learning (as in Eq. 13), which involves reward maximization with a KL-divergence constraint, into a single-stage policy training. Specifically, DPO’s training goal is to maximize the following expectation: E (x,yw,yl)∼D fd logπθ(yw|x) πref(yw|x)−βlogπθ(yl|x) πr...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "LRRHF =X ri<rjmax(0 , pi−pj), (15) where riandrjare the reward scores assigned by the teacher LLM for responses yiandyj, respectively, and pi,pj are their corresponding conditional log probabilities under the policy πθ. This approach emphasizes direct comparison and ranking of responses based on the teacher’s preferences. PRO (Song et al., 2023a) expands the concept of pairwise comparison to handle preference rankings of any length. For a given instruction xand a sequence of responses ordered by...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "eward scores assigned by the teacher LLM for responses yiandyj, and pi,pj are their corresponding conditional log probabilities under the policy πθ. This approach emphasizes direct comparison and ranking of responses based on the teacher's preferences. PRO expands the concept of pairwise comparison to handle preference rankings of any length. For a given instruction x and a sequence of responses ordered by teacher preference as y1≻y2≻...≻yn, the RPO training objective is: LPRO=−n−1X k=1logexp (p...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "eliciting knowledge and distillation algorithms, we shift our focus to how these techniques facilitate the distillation of specific skills in LLMs. Our exploration will encompass a diverse range of skills exhibited by LLMs, including Context Following ,Alignment ,Agent ,NLP Task Specializa- tion and Multi-Modality .Context Following focuses on the student’s ability to comprehend and respond effectively to input information. Alignment delves into the student’s capability to align its output with ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "l role in enhancing the capabilities of Large Language Models (LLMs). A diverse range of skills are exhibited by LLMs, including Context Following, Alignment, Agent, and NLP Task Specialization. Context Following focuses on the student's ability to effectively comprehend and respond to input information. Alignment involves aligning the model's output with the teacher's responses. Agent emphasizes the autonomous nature of language models, highlighting their ability to operate independently. NLP T...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "175 human-curated tasks GPT3 LLaMA Expansion + Self-Knowledge SFT LaMini-LM (Wu et al., 2023c) IF3.5K Wikipedia Categories + Mixed DatasetChatGPT Various Models Expansion SFT WizardLM (Xu et al., 2023a) IF Alpaca Data ChatGPT LLaMA Expansion SFT Lion (Jiang et al., 2023b) IF Alpaca Cata ChatGPT LLaMA Labeling + Expansion + Feedback - BabyLlama (Timiryasov and Tastet, 2023) IF 10M-word BabyLM dataset GPT-2 + small LLaMA 58M-parameter LLaMA Feature D&S MiniLLM (Gu et al., 2024) IF Dolly Dataset GP...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "Labeling SFT Selective Reflection-Tuning (Li et al., 2024d) IF Alpaca/WizardLM Dataset ChatGPT LLaMA Labeling SFT Vicuna (Chiang et al., 2023) IF/MD Human Conversation ChatGPT + GPT4 LLaMA Labeling SFT Koala (Geng et al., 2023) IF/MD Human Conversation ChatGPT LLaMA Labeling SFT Baize (Xu et al., 2023b) IF/MD Quora + Stack Overflow ChatGPT LLaMA Expansion + Self-Knowledge SFT UltraChat (Ding et al., 2023b) IF/MD Wikidata + Text Material + C4 ChatGPT LLaMA Curation SFT Orca (Mukherjee et al., 202...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "ng SFT Vicuna (Chiang et al., 2023) IF/MD Human Conversation ChatGPT + GPT4 LLaMA Labeling SFT Koala (Geng et al., 2023) IF/MD Human Conversation ChatGPT LLaMA Labeling SFT Baize (Xu et al., 2023b) IF/MD Quora + Stack Overflow ChatGPT LLaMA Expansion + Self-Knowledge SFT UltraChat (Ding et al., 2023b) IF/MD Wikidata + Text Material + C4 ChatGPT LLaMA Curation SFT Orca (Mukherjee et al., 2023) IF/TP FLAN-v2 ChatGPT + GPT4 LLaMA Labeling SFT Orca2 (Mitra et al., 2023) IF/TP FLAN-v2 + Few-Shot/Math...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "Labeling SFT Phi-1 (Gunasekar et al., 2023) IF/Code - GPT3.5 phi-1 Curation SFT Phi-1.5 (Li et al., 2023a) IF/Code 20k Topics from Web GPT3.5 phi-1 Curation + Labeling SFT SAIL (Luo et al., 2023c) IF/RAG Alpaca Data + Web Content GPT4 LLaMA Label SFT KARD (Kang et al., 2023b) IF/RAG MedQAUSMLE ChatGPT T5 + OPT Label SFT + D&S Self-RAG (Asai et al., 2023) IF/RAG Open-Instruct GPT4 LLaMA Labeling SFT Alignment OpenChat (Wang et al., 2023c) IF/Preference Human Conversation ChatGPT + GPT4 LLaMA Labe...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "IF/Code 20k Topics from Web GPT3.5 phi-1 Curation + Labeling SFT SAIL (Luo et al., 2023c) IF/RAG Alpaca Data + Web Content GPT4 LLaMA Label SFT KARD (Kang et al., 2023b) IF/RAG MedQAUSMLE ChatGPT T5 + OPT Label SFT + D&S Self-RAG (Asai et al., 2023) IF/RAG Open-Instruct GPT4 LLaMA Labeling SFT Alignment OpenChat (Wang et al., 2023c) IF/Preference Human Conversation ChatGPT + GPT4 LLaMA Labeling SFT + RL Zephyr (Tunstall et al., 2023) IF/Preference Mixed Datasets GPT4 Mistral Labeling + Feedback ...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "ILF (Scheurer et al., 2023) Preference Task-specific Datasets GPT3 + FeedME GPT3 Labeling RL ULTRAFEEDBACK (Cui et al., 2023a) Preference Mixed Datasets GPT4 LLaMA Labeling RL Constitutional AI (Bai et al., 2022a) Preference/Value Human-written Prompts Self-defined Student Model Self-defined Model Labeling + Expansion + Feedback SFT + RL SANDBOX (Liu et al., 2023b) Value Simulationtext-davinci-002/-003 + GPT4 + ChatGPTLLaMA Data Curation SFT + RL Agent Toolformer (Schick et al., 2023) Tool CCNet...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "FEEDBACK (Cui et al., 2023a) Preference Mixed Datasets GPT4 LLaMA Labeling RL Constitutional AI (Bai et al., 2022a) Preference/Value Human-written Prompts Self-defined Student Model Self-defined Model Labeling + Expansion + Feedback SFT + RL SANDBOX (Liu et al., 2023b) Value Simulationtext-davinci-002/-003 + GPT4 + ChatGPTLLaMA Data Curation SFT + RL Agent Toolformer (Schick et al., 2023) Tool CCNet GPT-J GPT-J Labeling SFT Graph-ToolFormer (Zhang, 2023) Tool Mixed Graph Dataset ChatGPT GPT-J + ...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "Model Cards GPT4 LLaMA Curation SFT FireAct (Chen et al., 2023b) Planning Mixed QA Dataset GPT4 LLaMA Labeling SFT AgentTuning (Zeng et al., 2023a) Planning 6 Agent Tasks GPT4 + ChatGPT LLaMA Labeling + Expansion SFT Lumos (Yin et al., 2023a) Planning Mixed Interactive Tasks GPT4 LLaMA Labeling SFT AUTOACT (Qiao et al., 2024) Planning Mixed QA Tasks LLaMA LLaMA Labeling SFT NLP Task Specialization AugGPT (Dai et al., 2023a) NLU Amazon/Symptoms/PubMed20k Dataset ChatGPT BERT Label SFT TDG (He et ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "LLaMA Labeling SFT AgentTuning (Zeng et al., 2023a) Planning 6 Agent Tasks GPT4 + ChatGPT LLaMA Labeling + Expansion SFT Lumos (Yin et al., 2023a) Planning Mixed Interactive Tasks GPT4 LLaMA Labeling SFT AUTOACT (Qiao et al., 2024) Planning Mixed QA Tasks LLaMA LLaMA Labeling SFT NLP Task Specialization AugGPT (Dai et al., 2023a) NLU Amazon/Symptoms/PubMed20k Dataset ChatGPT BERT Label SFT TDG (He et al., 2023b) NLU SST + QQP + MNLI GPT3 BERT Expansion SFT SunGen (Gao et al., 2023a) NLU Text Cla...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "et al., 2024) NLG/NLU/IF XSum+WMT14 en-de+GSM8K+FLAN2021 T5-XL T5 Feature + Feedback D&S + RL QUILL (Srinivasan et al., 2022) IR IR Datasets T5 4-layer Transformer Internal Knowledge D&S RankVicuna (Pradeep et al., 2023a) IR IR Datasets ChatGPT LLaMA Labeling SFT RankZephyr (Pradeep et al., 2023b) IR IR Datasets ChatGPT + GPT4 Mistral Labeling SFT NDR (Mysore et al., 2023) Recommendation Recommendation Datasets GPT3 MPnet-110M Labeling SFT InstrcutRec (Zhang et al., 2023b) Recommendation 39 inst...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "et al., 2022) IR IR Datasets T5 4-layer Transformer Internal Knowledge D&S RankVicuna (Pradeep et al., 2023a) IR IR Datasets ChatGPT LLaMA Labeling SFT RankZephyr (Pradeep et al., 2023b) IR IR Datasets ChatGPT + GPT4 Mistral Labeling SFT NDR (Mysore et al., 2023) Recommendation Recommendation Datasets GPT3 MPnet-110M Labeling SFT InstructRec (Zhang et al., 2023b) Recommendation 39 instruction templates ChatGPT Flan-T5 Expansion + Self-Knowledge SFT ONCE (Liu et al., 2023c) Recommendation Recomme...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "(Yue et al., 2023a) Math/TP Mixed Math Dataset GPT4 LLaMA Labeling SFT Mixed Distill (Chenglin et al., 2023) Math/TP SVAMP + GSM8K + ASDIV + StrategyQA ChatGPT LLaMa Labeling SFT WizardCoder (Luo et al., 2023a) Code Code Alpaca Data ChatGPT StarCoder Expansion SFT Magicoder (Wei et al., 2023) Code Existing Source Codes ChatGPT LLaMa Curation SFT WaveCoder (Yu et al., 2024) Code Existing Source Codes GPT4 LLaMa Curation SFT Code Alpaca (Chaudhary, 2023) Code Code Instructions ChatGPT LLaMA Expans...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "VAMP + GSM8K + ASDIV + StrategyQA ChatGPT LLaMa Labeling SFT WizardCoder (Luo et al., 2023a) Code Code Alpaca Data ChatGPT StarCoder Expansion SFT Magicoder (Wei et al., 2023) Code Existing Source Codes ChatGPT LLaMa Curation SFT WaveCoder (Yu et al., 2024) Code Existing Source Codes GPT4 LLaMa Curation SFT Code Alpaca (Chaudhary, 2023) Code Code Instructions ChatGPT LLaMA Expansion + Self-Knowledge SFT Code Llama (Rozi `ere et al., 2023) Code Human-written Instructions LLaMA LLaMA Expansion + S...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "Vision-Language LAION GPT4 LLaMA Labeling SFT Macaw-LLM (Lyu et al., 2023) Multiple Modalities Image/Video with Caption ChatGPT LLaMA Labeling SFT MIMIC-IT (Li et al., 2023f) Multiple Modalities Image/Video Dataset ChatGPT LLaMA Labeling SFT ChatBridge (Zhao et al., 2023d) Multiple Modalities Task-Specific/Multimodal-Chat Data GPT4 + ChatGPT LLaMA Labeling SFT TABLE 3: A summary of skill distillation works. IF: Instruction Following, MD: Multi-turn Dialoue, TP: Think Pattern, RAG: Retrieval-Augm...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "/Video with Caption \n",
      "ChatGPT LLaMA Labeling SFT MIMIC-IT (Li et al., 2023f) Multiple Modalities Image/Video Dataset \n",
      "ChatGPT LLaMA Labeling SFT ChatBridge (Zhao et al., 2023d) Multiple Modalities Task-Specific/Multimodal-Chat \n",
      "Data GPT4 + ChatGPT LLaMA Labeling SFT TABLE 3: A summary of skill distillation works. \n",
      "IF Instruction Following, MD Multi-turn Dialogue, TP Think Pattern, RAG Retrieval-Augmented Generation, NLU Natural Language Understanding, NLG Natural Language Generation, IR Informati...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "method, and training objectives.4.1 Context Following This part concentrates on the distillation of context follow- ing skills from LLMs. This process involves transferring the ability of LLMs to handle a variety of complex contexts — such as few-shot demonstrations, intricate instructions, dia- logue history, and retrieval-augmented information — into smaller models. Many research efforts in this domain aim to imbue smaller models with these sophisticated, context- 15 following capabilities. Ou...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "s part focuses on distilling context following skills from LLMs. This process involves transferring the ability of LLMs to handle various complex contexts, such as few-shot demonstrations, intricate instructions, dialogue history, and retrieval-augmented information, into smaller models. Many research efforts in this domain aim to imbue smaller models with these sophisticated, context-based capabilities. Our discussion will dissect this aspect of skill distillation, categorizing it based on diff...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "acquiring this skill involves construct- ing instruction-like prompt-response pairs and employing Supervised Fine Tuning (SFT) for model training. Data for this purpose can be manually curated by human experts or transformed from existing NLP tasks into instructional formats with templates, such as prefacing machine transla- tion data with ”Translate this sentence to Spanish:” . However, these approaches have limitations. Manual data creation is labor-intensive, while template-based transformati...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "ervised Fine Tuning (SFT) for model training. Data for this purpose can be manually curated by human experts or transformed from existing NLP tasks into instructional formats with templates, such as prefacing machine translation data with \"Translate this sentence to Spanish:\". However, these approaches have limitations. Manual data creation is labor-intensive, while template-based transformation lacks diversity in instructions and may not align well with natural human input. LLMs like GPT-4 offe...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "Mukherjee et al., 2023; Mitra et al., 2023; Luo et al., 2023b; Peng et al., 2023a). Basic Instructions. Self-Instruct (Wang et al., 2022a) lever- ages the in-context learning capability of GPT-3 to expand a seed pool of 175 tasks to 52K task-agnostic instructions, ensuring a broad spectrum of general instructions. Addi- tionally, a filtering and post-processing stage is introduced to eliminate redundant or similar instructions. Notably, through training with this enriched dataset, GPT-3 acquires...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "re learned in context to expand a large pool of general instructions, ensuring a broad spectrum of capabilities. Additional steps are added to filter out redundant or similar instructions, enabling GPT-3 to follow instructions accurately....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "introduce a technique known asTopic-Guided Instruction Generation . This method involves gathering 3.5K common topics from Wikipedia to serve as guidance during the generation process. Complex Instructions. Some works promote students to solve more complex instructions (Xu et al., 2023a; Luo et al., 2023b,a; Guo et al., 2023c). According to Xu et al. (2023a), in- struction datasets derived from human-written seeds often exhibit low to moderate complexity. To enhance the com- plex instruction-fol...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "common topics from Wikipedia to serve as guidance during the generation process. This technique promotes students to solve more complex instructions by utilizing a large dataset of human-written seeds. Instructions derived from human-written seeds often exhibit low to moderate complexity, making them suitable for smaller models. To enhance the complexity of these models, WizardLM introduces Evol-Instruct, a method that transforms instructions into more complex forms through a multi-step evolutio...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "dataset. In the high-difficulty section of test instructions, WizardLM even outperformed ChatGPT, achieving a win rate 7.9% higher than ChatGPT. Zhao et al. (2023e) further conduct preliminary studies revealing the effectiveness of increasing instruction complexity. Instruction Fusion (Guo et al., 2023c) further uses teacher LLMs to increase the complexity by fusing two distinct evolved instructions. Furthermore, this concept of “evolving” instructions has been extended to distill specific skill...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "a win rate 7.9% higher than ChatGPT. Zhao et al. (2023) further conduct preliminary studies revealing the effectiveness of increasing instruction complexity. Instruction Fusion (Guo et al., 2023) further uses teacher LLMs to increase the complexity by fusing two distinct evolved instructions. Furthermore, this concept of “evolving” instructions has been extended to distill specific skills such as coding (Luo et al., 2023a) and mathematics (Luo et al., 2023b). Human Instructions. In contrast to w...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "forum for users to share their interactions with Chat- GPT. It’s important to note, however, that models trained on such natural conversations might mimic the style but may not fully capture the reasoning process of the original teacher (Gudibande et al., 2023; Mukherjee et al., 2023). System Instructions. To encourage student models to learn the reasoning process, Orca and Orca 2 (Mukherjee et al., 2023; Mitra et al., 2023) enhance the prompt, response data pairs by introducing a system message...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "like us are important. However, models trained on these conversations may not fully capture the teacher's reasoning process. To improve this, we introduce a system message, which prompts the model to explain the reasoning process in simple terms. This helps models like GPT-4 and Orca 2 (Mukherjee et al., 2023; Mitra et al., 2023) to learn the process by tracing the teacher's steps and identifying the most effective strategy for each task. This approach enhances the model's ability to follow inst...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "High-Quality Instructions. As demonstrated in Zhou et al. (2023a) and (Li et al., 2024f), the data quality is crucial for instruction following training. UltraChat (Ding et al., 2023b) distills large-scale data with high-quality and di- verse instructions from teacher LLMs by various meta- information. The UltraLLaMA model, fine-tuned on this data, consistently surpasses other open-source models. The Phi series models (Gunasekar et al., 2023; Li et al., 2023a; Mar, 2023) prioritize data quality ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "-scale data with high-quality and diverse instructions from teacher LLMs by various metadata. The UltraLLaMA model, fine-tuned on this data, consistently surpasses other open-source models. Phi series models prioritize data quality and employ synthetic methods to generate data of \"textbook quality\" to enhance the learning experience for smaller models. Notably, Phi exhibits the ability to follow instructions effectively, even without specific instruction fine-tuning. Phi-2, with 2.7 billion para...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "of existing instruction data, including both the improvement of instruction and corresponding response. SelFee (Ye et al., 2023) utilizes the ChatGPT to iter- atively improve the quality of responses. ExpertLLaMA (Xu et al., 2023f) improves the quality of responses by augment- 16 ing vanilla instructions with specialized Expert Identity descriptions. Reflection-Tuning (Li et al., 2023e) improves both the instruction and response sequentially by reflecting on specific criteria. DEITA (Liu et al.,...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "ertLLaMA (Xu et al., 2023f) improves the quality of responses by augmenting vanilla instructions with specialized Expert Identity descriptions. Reflection-Tuning (Li et al., 2023e) improves both the instruction and response sequentially by reflecting on specific criteria. DEITA (Liu et al., 2023h) proposes to enhance and score instructions in three directions including complexity, quality, and diversity to get high-quality distillation data. MUFFIN (Lou et al., 2023) proposes to scale the instru...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "data learn from. In summary, distilling instruction data from teachers presents a promising avenue for training cheap and re- producible instruction-following language models. Cur- rent small models have made strides in enhancing var- ious aspects of instruction-following ability, like diver- sity, complexity and explanation. However, student mod- els trained on instruction data expanded by ChatGPT of- ten mimic ChatGPT’s style without replicating its factual accuracy (Gudibande et al., 2023). A...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "mising avenue for training language models. Current models have made strides in enhancing diversity, complexity, and explanation. However, student models trained on instruction data often mimic ChatGPT's style without replicating its factual accuracy....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "and maintain context through ongoing interactions. This skill is vital for models to engage meaningfully in human-like conversations and respond coherently over suc- cessive dialogue turns. Some works have been dedicated to train to small chat models by distilling multi-turn knowl- edge from teacher LLMs (Chiang et al., 2023; Xu et al., 2023b; Ding et al., 2023b; Li et al., 2023b; Wang et al., 2023c; Tunstall et al., 2023). ShareGPT serves as a platform for users to share their conversations wit...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "onversations and respond coherently over successive dialogue turns.\n",
      "\n",
      "Some works have focused on training small chat models to distill multi-turn knowledge from teacher LLMs (Chiang et al., 2023; Xu et al., 2023b; Ding et al., 2023b; Li et al., 2023b; Wang et al., 2023c; Tunstall et al., 2023). This platform serves as a repository of conversations for users to share their interactions, offering a vast array of multi-turn dialogues readily available....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "conducted by Wang et al. (2023c), GPT-3.5 and GPT-4 are employed to generate mixed responses using ShareGPT data. They assign higher rewards to responses generated by GPT-4, aiming to incentivize student models to produce high-quality responses. Addi- tionally, Ye et al. (2023) enhance the quality of multi-turn data from ShareGPT by generating self-feedback on model responses and iteratively refining the responses based on the received feedback. 3. MT-Bench: a multi-turn question set, where the ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "ShareGPT data. They assign higher rewards to responses generated by GPT-4, aiming to incentivize student models to produce high-quality responses. Addition-ally, Ye et al. (2023) enhance the quality of multi-turn data from ShareGPT by generating self-feedback on model responses and iteratively refining the responses based on the received feedback. 3. MT-Bench: a multi-turn question set, where the generations of models are evaluated by LLM....\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "Subsequently, they employ parameter- efficient tuning to train a chat model named Baize. Ding et al. (2023b) first construct a significantly larger dataset called UltraChat, comprising 1.5 million high-quality multi- turn dialogues. They achieve this by distilling instructions and dialogues from ChatGPT. Notably, UltraChat encom- passes a wide range of topics and instructions. Building upon the UltraChat dataset, they fine-tune a LLaMA model, resulting in the creation of a powerful chat model kn...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "l. (2023b) first construct a significantly larger dataset called UltraChat, comprising 1.5 million high-quality multi-turn dialogues. They achieve this by distilling instructions and dialogues from ChatGPT. Notably, UltraChat passes a wide range of topics and instructions. Building upon the UltraChat dataset, they fine-tune a LLaMA model, resulting in the creation of a powerful chat model known as UltraLLaMA. UltraLLaMA consistently outperforms other open-source chat models, including Vicuna and...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "inaccuracies due to their sole reliance on the parametric knowledge. Retrieval-Augmented Generation (RAG) is a promising technique to decrease this issue. Handling the augmented context of retrieved information is also a non- trivial skill of LLMs. Several approaches to distill RAG capabilities have been proposed (Kang et al., 2023a; Luo et al., 2023c; Asai et al., 2023). SAIL (Luo et al., 2023c) starts by retrieving search results for each training case using search APIs, creating search- augme...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "is a promising technique to decrease this issue Handling the augmented context of retrieved information is also a non-trivial skill of LLMs Several approaches to distill Retrieval-Augmented Generation capabilities have been proposed SAIL starts by retrieving search results for each training case using search APIs creating search-augmented instructions including both instruction and grounding information To encourage the language model to prioritize informative retrieval results input each retrie...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "becomes proficient at de- noising search results and generating accurate responses. KARD (Kang et al., 2023b) distills rationales rfrom the teacher LLM in response to questions x. These rationales are then utilized to train two models: a student LM and a Reranker. For training the student LM, the rationales serve as a means to retrieve relevant knowledge d, and the student LM is subsequently fine-tuned using the rationales along- side questions and knowledge. However, during inference, only ques...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "n utilizes these rationales to train two models: a student LM and a Reranker. For training the student LM, rationales serve as a means to retrieve relevant knowledge. The student LM is fine-tuned alongside questions and knowledge. During inference, only questions are available, so the Reranker is trained to mimic how the retriever scores passages with the rationale by minimizing KL divergence between Retriever (d|r) and Reranker (d|x). This integration of a fixed number of passages in language m...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "adaptive ability from teacher LLMs into a small critic model. This critic model determines whether retrieval is necessary and evaluates the quality of the retrieved results by generating ‘reflection to- kens.’ For instance, Self-Rag initiates the retrieval operation when generating the reflection token Retrieve . To distill this critic data, GPT-4 is prompted to assess the need for retrieval using few-shot demonstrations I, the task input x, and output yto predict a reflection token ras follows:...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "retrieval is necessary and evaluates the quality of the retrieved results by generating'reflection to- kens.' For instance, Self-Rag initiates the retrieval operation when generating the reflection token. To distill this critic data, GPT-4 is prompted to assess the need for retrieval using few-shot demonstrations I, the task input x, and output y to predict a reflection token ras follows: p(r|I, x, y). This approach has been shown to improve the performance of small critic models. 4.2.1 Thinking...\n",
      "==========================================================================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:None for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "the pure responses but some novel thinking patterns (Ye et al., 2023; Mukherjee et al., 2023; Mitra et al., 2023; Wang et al., 2023d; Cheng et al., 2023; Zhang et al., 2023a). Motivated by the effectiveness of LLMs in generat- ing their own feedback without relying on external mod- els (Schick et al., 2022; Madaan et al., 2023; Saunders et al., 2022), SelFee (Ye et al., 2023) proposes to train a model that has been fine-tuned to continuously revise its own answer until it provides a high-quality...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "utputs, thereby achieving a better understanding of the model's decision-making process....\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "reasoning steps, including explanation traces, step-by-step thought processes, and other complex instructions, from the teacher model, rather than just the vanilla styles. Extensive experiments verify the effectiveness of distilling with this thinking pattern. The following Orca2 (Mitra et al., 2023) further presents to eq...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "and other complex instructions:\n",
      "\n",
      "1. **Text Preprocessing**: The Orca2 model was used to analyze the given text, and the resulting output was then filtered to remove unnecessary characters, including new lines, LaTeX math, and irrelevant information.\n",
      "\n",
      "2. **Fluff Removal**: The text was then subjected to a rigorous removal of fluff, eliminating any extraneous details that could be considered irrelevant to the podcast topic.\n",
      "\n",
      "3. **Sentence Clustering**: The remaining text was then grouped into sent...\n",
      "==========================================================================================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open(output_file, 'w', encoding='utf-8') as out_file:\n",
    "    for chunk_num, chunk in enumerate(tqdm(chunks, desc=\"Processing chunks\")):\n",
    "        # Process chunk and append to complete text\n",
    "        processed_chunk = process_chunk(chunk, chunk_num)\n",
    "        processed_text += processed_chunk + \"\\n\"\n",
    "        \n",
    "        # Write chunk immediately to file\n",
    "        out_file.write(processed_chunk + \"\\n\")\n",
    "        out_file.flush()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31cffe8d",
   "metadata": {},
   "source": [
    "Let's print out the final processed versions to make sure things look good"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "89ef51a7-f13f-49a4-8f73-9ac8ce75319d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Processing complete!\n",
      "Input file: ./extracted_text.txt\n",
      "Output file: clean_extracted_text.txt\n",
      "Total chunks processed: 101\n",
      "\n",
      "Preview of final processed text:\n",
      "\n",
      "BEGINNING:\n",
      "Tao Shen4, Reynold Cheng1, Jinyang Li1,\n",
      "Can Xu5, Dacheng Tao6, Tianyi Zhou2\n",
      "1The University of Hong Kong2University of Maryland3Microsoft\n",
      "4University of Technology Sydney5Peking University6The University of Sydney\n",
      "{shawnxxh,chongyangtao,hishentao }@gmail.com {minglii,tianyi }@umd.edu\n",
      "ckcheng@cs.hku.hk\n",
      "ulously structured around three foundational pillars: algorithm, skill, and verticalization – providing a comprehensive examination of knowledge distillation mechanisms, the enhancement of specific cognitive abilities, and their practical implications across diverse fields. Crucially, the survey navigates the intricate interplay between data augmentation (DA) and knowledge distillation, illustrating how DA emerges as a powerful paradigm within the knowledge distillation framework to bolster large language models' performance. By leveraging DA to generate context-rich, skill-specific training data, knowledge distillation transcends traditional boundaries, enabling open-source models to app\n",
      "\n",
      "...\n",
      "\n",
      "END:\n",
      "ess was extensively tested on a variety of texts, including academic papers, news articles, and technical documents. The results showed a significant reduction in the size and complexity of the output, while maintaining the essential information and meaning of the original text.\n",
      "\n",
      "**Example**\n",
      "\n",
      " Original Text\n",
      "```\n",
      "reasoning steps, including explanation traces, step-by-step thought processes, and other complex instructions, from the teacher model, rather than just the vanilla styles. Extensive experiments verify the effectiveness of distilling with this thinking pattern. The following Orca2 (Mitra et al., 2023) further presents to eq\n",
      "```\n",
      "Processed Text\n",
      "```\n",
      "```\n",
      "reasoning steps, including explanation traces, step-by-step thought processes, and other complex instructions, from the teacher model, rather than just the vanilla styles. Extensive experiments verify the effectiveness of distilling with this thinking pattern. The following Orca2 (Mitra et al., 2023) further presents to equation\n",
      "```\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(f\"\\nProcessing complete!\")\n",
    "print(f\"Input file: {INPUT_FILE}\")\n",
    "print(f\"Output file: {output_file}\")\n",
    "print(f\"Total chunks processed: {num_chunks}\")\n",
    "\n",
    "# Preview the beginning and end of the complete processed text\n",
    "print(\"\\nPreview of final processed text:\")\n",
    "print(\"\\nBEGINNING:\")\n",
    "print(processed_text[:1000])\n",
    "print(\"\\n...\\n\\nEND:\")\n",
    "print(processed_text[-1000:])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d996ac5",
   "metadata": {},
   "source": [
    "### Next Notebook: Transcript Writer\n",
    "\n",
    "Now that we have the pre-processed text ready, we can move to converting into a transcript in the next notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b16ae0e-04cf-4eb9-a369-dee1728b89ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "#fin"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}