wehub-resource-sync
230 lines
8.2 KiB
Plaintext
230 lines
8.2 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Voice Transcription with CPU Friendly AI Models Example (Greatest Speeches of 20th Century)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Introduction"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"If we were given an audio file, is there any way we could identify the time stamps where specific words were said? Is there any way we could extract all the key words mentioned about a topic?\n",
|
|
"\n",
|
|
"With AI 🤖, we can do all of this and much more! The key lies in being able to parse audio into text, allowing us to then harness the natural language processing capabilities of language models to perform sophisticated analyses and inferences on our data.\n",
|
|
"\n",
|
|
"Regardless of who you are, such an approach to audio transcription and analysis will augment how you interact with and extract knowledge from audio files.\n",
|
|
"\n",
|
|
"Let's see how we can do this with llmware, using the Whisper model and the SLIM Extract Tool."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### For Google Colab Users"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"If you are using Colab for free, we highly recommend you activate the T4 GPU hardware accelerator. Our models are designed to run with at least 16GB of RAM, activating T4 will grant the notebook 16GB of GDDR6 RAM as apposed to the ~13GB Colab gives automatically.\n",
|
|
"\n",
|
|
"To activate T4:\n",
|
|
"1. click on the \"Runtime\" tab\n",
|
|
"2. click on \"Change runtime type\"\n",
|
|
"3. select T4 GPU under Hardware Accelerator\n",
|
|
"\n",
|
|
"NOTE: there is a weekly usage limit on using T4 for free"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Installing and importing dependencies"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install llmware"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"\n",
|
|
"from llmware.parsers import Parser\n",
|
|
"from llmware.setup import Setup\n",
|
|
"from llmware.util import Utilities\n",
|
|
"from llmware.models import ModelCatalog"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Worker function"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Our main worker function will do the following:\n",
|
|
"1. Load in sample audio files with the `Setup` class\n",
|
|
"2. Parse the audio into text by calling the WhisperCPP model via the `Parser` class\n",
|
|
"3. Perform a text search for the word \"president\" (these specific text chunks are used in the next step)\n",
|
|
"4. Use the SLIM Extract Tool to extract the president names from these chunks\n",
|
|
"5. Check if the extracted name matches someone in a specified list (`various_american_presidents`)\n",
|
|
" - If there is a match, append information about the location of the extracted name and the time stamp in the audio to a `final_list`\n",
|
|
"6. Output the content of the `final_list`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def greatest_speeches_example():\n",
|
|
"\n",
|
|
" # four sample file sets - \"famous_quotes\" | \"greatest_speeches\" | \"youtube_demos\" | \"earnings_calls\"\n",
|
|
" voice_sample_files = Setup().load_voice_sample_files(small_only=False)\n",
|
|
"\n",
|
|
" input_folder = os.path.join(voice_sample_files, \"greatest_speeches\")\n",
|
|
"\n",
|
|
" print(\"\\nStep 1 - converting, parsing and text chunking ~56 WAV files\")\n",
|
|
"\n",
|
|
" parser_output = Parser(chunk_size=400, max_chunk_size=600).parse_voice(input_folder,\n",
|
|
" write_to_db=False,\n",
|
|
" copy_to_library=False,\n",
|
|
" remove_segment_markers=True,\n",
|
|
" chunk_by_segment=True,\n",
|
|
" real_time_progress=False)\n",
|
|
"\n",
|
|
" print(\"\\nStep 2- look at the text chunks with all of the metadata\")\n",
|
|
"\n",
|
|
" for i, entries in enumerate(parser_output):\n",
|
|
" print(\"all parsed blocks: \", i, entries)\n",
|
|
"\n",
|
|
" print(\"\\nStep 3- run an inline text search for 'president'\")\n",
|
|
"\n",
|
|
" results = Utilities().fast_search_dicts(\"president\", parser_output)\n",
|
|
"\n",
|
|
" for i, res in enumerate(results):\n",
|
|
" print(\"search results: \", i, res)\n",
|
|
"\n",
|
|
" print(\"\\nStep 4- use LLM to review each search result - and if specific U.S. presidents found, then display the source\")\n",
|
|
"\n",
|
|
" extract_model = ModelCatalog().load_model(\"slim-extract-tool\", sample=False, temperature=0.0, max_output=200)\n",
|
|
"\n",
|
|
" final_list = []\n",
|
|
"\n",
|
|
" for i, res in enumerate(results):\n",
|
|
"\n",
|
|
" response = extract_model.function_call(res[\"text\"], params=[\"president name\"])\n",
|
|
"\n",
|
|
" various_american_presidents = [\"kennedy\", \"carter\", \"nixon\", \"reagan\", \"clinton\", \"obama\"]\n",
|
|
"\n",
|
|
" extracted_name = \"\"\n",
|
|
" if \"president_name\" in response[\"llm_response\"]:\n",
|
|
" if len(response[\"llm_response\"][\"president_name\"]) > 0:\n",
|
|
" extracted_name = response[\"llm_response\"][\"president_name\"][0].lower()\n",
|
|
" else:\n",
|
|
" print(\"\\nupdate: skipping result - no president name found - \", response[\"llm_response\"], res[\"text\"])\n",
|
|
"\n",
|
|
" for president in various_american_presidents:\n",
|
|
" if president in extracted_name:\n",
|
|
" print(\"\\nextracted american president text: \", i, extracted_name)\n",
|
|
" print(\"file source: \", res[\"file_source\"])\n",
|
|
" print(\"time stamp: \", res[\"coords_x\"], res[\"coords_y\"], res[\"coords_cx\"], res[\"coords_cy\"])\n",
|
|
" print(\"text: \", i, res[\"text\"])\n",
|
|
" final_list.append({\"key\": president, \"source\": res[\"file_source\"], \"time_start\": res[\"coords_x\"],\n",
|
|
" \"text\": res[\"text\"]})\n",
|
|
"\n",
|
|
" print(\"\\nStep 5 - final results\")\n",
|
|
" for i, f in enumerate(final_list):\n",
|
|
" print(\"final results: \", i, f)\n",
|
|
"\n",
|
|
" return final_list"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Main block"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Here, we call the worker function."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"if __name__ == \"__main__\":\n",
|
|
"\n",
|
|
" greatest_speeches_example()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"The output, `final_list`, is a Python dictionary with information about the instances where a president name mentioned in the audio matches one of our specified presidents in the `various_american_presidents_list`. It has the following keys:\n",
|
|
"- key: the name of the president identified\n",
|
|
"- source: the audio file this was found in\n",
|
|
"- time_start: the time stamp in seconds where the president was mentioned\n",
|
|
"- text: which contains the text chunk the name was found in."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.0"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|