wehub-resource-sync
1402 lines
78 KiB
Plaintext
1402 lines
78 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "f34f9285",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Build your own NotebookLM\n",
|
||
"\n",
|
||
"This notebook walks you through all the core functionalities of our NotebookLM implementation, from document processing and content extraction to memory integration and podcast generation. Each section includes both explanations and code examples to demonstrate how the system works under the hood."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "01c5a6a0",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Table of Contents\n",
|
||
"1. [Document Processing](#1-document-processing)\n",
|
||
"2. [Audio Transcription](#2-audio-transcription)\n",
|
||
"3. [Web Scraping](#3-web-scraping)\n",
|
||
"4. [Embedding Generation](#4-embedding-generation)\n",
|
||
"5. [Vector Database Storage](#5-vector-database-storage)\n",
|
||
"6. [Memory Layer Integration](#6-memory-layer-integration)\n",
|
||
"7. [RAG Query Processing](#7-rag-query-processing)\n",
|
||
"8. [Memory Storage for Conversations](#8-memory-storage-for-conversations)\n",
|
||
"9. [Podcast Script Generation](#9-podcast-script-generation)\n",
|
||
"10. [Text-to-Speech Audio Generation](#10-text-to-speech-audio-generation)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"id": "6d6915c5",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"True"
|
||
]
|
||
},
|
||
"execution_count": 1,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# Necessary imports\n",
|
||
"import os\n",
|
||
"import datetime\n",
|
||
"import tempfile\n",
|
||
"from pathlib import Path\n",
|
||
"from dotenv import load_dotenv\n",
|
||
"\n",
|
||
"load_dotenv()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b2dab11e",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 1. Document Processing\n",
|
||
"\n",
|
||
"The document processing module handles various file formats (PDF, TXT, Markdown) and converts them into structured chunks with metadata for citations and retrieval."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "fa0581a1",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Key features:\n",
|
||
"- **Multi-format support**: PDF, TXT, MD files\n",
|
||
"- **Page tracking**: Maintains page numbers for accurate citations\n",
|
||
"- **Character positions**: Tracks start/end positions for precise references\n",
|
||
"- **Metadata preservation**: Source file, type, and processing timestamp"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"id": "b004783d",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"INFO:src.document_processing.doc_processor:Processing document: raft.pdf\n",
|
||
"INFO:src.document_processing.doc_processor:Processed PDF: 53 chunks from 11 pages\n",
|
||
"INFO:src.document_processing.doc_processor:Processing document: notes.txt\n",
|
||
"INFO:src.document_processing.doc_processor:Processed text file: 5 chunks\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Chunk ID: pdf_0_2c889c2c\n",
|
||
"Source: raft.pdf\n",
|
||
"Page: 1\n",
|
||
"Content: RAFT: Adapting Language Model to Domain Specific RAG\n",
|
||
"Tianjun Zhang Shishir G. Patil Naman Jain Sheng...\n",
|
||
"Citation Info: {'source': 'raft.pdf', 'type': 'pdf', 'chunk_id': 'pdf_0_2c889c2c', 'chunk_index': 0, 'page': 1, 'char_range': '0-964', 'total_pages': 11, 'page_width': 612.0, 'page_height': 792.0, 'processed_at': '2025-10-06T22:11:10.290567'}\n",
|
||
"--------------------------------------------------\n",
|
||
"Chunk ID: pdf_1_62caaf00\n",
|
||
"Source: raft.pdf\n",
|
||
"Page: 1\n",
|
||
"Content: help in answering the question, which we call,\n",
|
||
"distractor documents. RAFT accomplishes this\n",
|
||
"by citin...\n",
|
||
"Citation Info: {'source': 'raft.pdf', 'type': 'pdf', 'chunk_id': 'pdf_1_62caaf00', 'chunk_index': 1, 'page': 1, 'char_range': '965-1942', 'total_pages': 11, 'page_width': 612.0, 'page_height': 792.0, 'processed_at': '2025-10-06T22:11:10.290567'}\n",
|
||
"--------------------------------------------------\n",
|
||
"Chunk ID: pdf_2_90a18e35\n",
|
||
"Source: raft.pdf\n",
|
||
"Page: 1\n",
|
||
"Content: specific document collections (e.g., legal or medical docu-\n",
|
||
"ments). In these settings, general knowl...\n",
|
||
"Citation Info: {'source': 'raft.pdf', 'type': 'pdf', 'chunk_id': 'pdf_2_90a18e35', 'chunk_index': 2, 'page': 1, 'char_range': '1943-2927', 'total_pages': 11, 'page_width': 612.0, 'page_height': 792.0, 'processed_at': '2025-10-06T22:11:10.290567'}\n",
|
||
"--------------------------------------------------\n",
|
||
"Processed 5 chunks from text file\n",
|
||
"Citation Info: {'source': 'notes.txt', 'type': 'txt', 'chunk_id': 'txt_0_29d5d73d', 'chunk_index': 0, 'char_range': '0-874', 'file_size': 4441, 'encoding': 'utf-8', 'processed_at': '2025-10-06T22:11:10.331573'}\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"from src.document_processing.doc_processor import DocumentProcessor, DocumentChunk\n",
|
||
"\n",
|
||
"# Initialize the document processor\n",
|
||
"doc_processor = DocumentProcessor()\n",
|
||
"\n",
|
||
"# Process a PDF document\n",
|
||
"pdf_path = \"data/raft.pdf\"\n",
|
||
"doc_chunks = doc_processor.process_document(pdf_path)\n",
|
||
"\n",
|
||
"# Each chunk contains rich metadata for citations\n",
|
||
"for chunk in doc_chunks[:3]:\n",
|
||
" print(f\"Chunk ID: {chunk.chunk_id}\")\n",
|
||
" print(f\"Source: {chunk.source_file}\")\n",
|
||
" print(f\"Page: {chunk.page_number}\")\n",
|
||
" print(f\"Content: {chunk.content[:100]}...\")\n",
|
||
" print(f\"Citation Info: {chunk.get_citation_info()}\")\n",
|
||
" print(\"-\" * 50)\n",
|
||
"\n",
|
||
"# Process a text file\n",
|
||
"txt_path = \"data/notes.txt\"\n",
|
||
"text_chunks = doc_processor.process_document(txt_path)\n",
|
||
"print(f\"Processed {len(text_chunks)} chunks from text file\")\n",
|
||
"print(f\"Citation Info: {text_chunks[0].get_citation_info()}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "3eacf9d2",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 2. Audio Transcription\n",
|
||
"\n",
|
||
"Our system supports two types of audio transcription: direct audio files and YouTube videos, both using AssemblyAI for high-accuracy transcription with speaker diarization."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "603840b6",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Audio Processing Features:\n",
|
||
"- **Speaker diarization**: Automatically identifies different speakers\n",
|
||
"- **Timestamp preservation**: Maintains timing information for each utterance\n",
|
||
"- **High accuracy**: Uses AssemblyAI's advanced speech recognition\n",
|
||
"- **YouTube integration**: Audio download with yt-dlp + audio transcription with AssemblyAI"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "e1a0849a",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 2.1 Direct Audio Transcription"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"id": "4f11e442",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"INFO:src.audio_processing.audio_transcriber:AudioTranscriber initialized with AssemblyAI\n",
|
||
"INFO:src.audio_processing.audio_transcriber:Starting transcription for: harvard.wav\n",
|
||
"INFO:httpx:HTTP Request: POST https://api.assemblyai.com/v2/upload \"HTTP/1.1 200 OK\"\n",
|
||
"INFO:httpx:HTTP Request: POST https://api.assemblyai.com/v2/transcript \"HTTP/1.1 200 OK\"\n",
|
||
"INFO:httpx:HTTP Request: GET https://api.assemblyai.com/v2/transcript/7ef032fe-8d9c-4c33-91c5-17ef9671fa7b \"HTTP/1.1 200 OK\"\n",
|
||
"INFO:httpx:HTTP Request: GET https://api.assemblyai.com/v2/transcript/7ef032fe-8d9c-4c33-91c5-17ef9671fa7b \"HTTP/1.1 200 OK\"\n",
|
||
"INFO:src.audio_processing.audio_transcriber:Transcription completed for: harvard.wav\n",
|
||
"INFO:src.audio_processing.audio_transcriber:Created 1 chunks from transcript\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"Speaker: ['Speaker B', 'Speaker A']\n",
|
||
"Time: 1.36s - 17.68s\n",
|
||
"Content: [00:01] Speaker A: The stale smell of old beer lingers.\n",
|
||
"[00:04] Speaker B: It takes heat to bring out the odor.\n",
|
||
"[00:06] Speaker A: A cold dip restores health and zest.\n",
|
||
"[00:09] Speaker B: A salt pickle tastes fine with ham.\n",
|
||
"[00:12] Speaker A: Tacos al pastor are my favorite.\n",
|
||
"[00:14] Speaker B: A zestful food is the hot cross bun.\n",
|
||
"Confidence: 0.958\n",
|
||
"----------------------------------------\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"from src.audio_processing.audio_transcriber import AudioTranscriber\n",
|
||
"\n",
|
||
"# Initialize with AssemblyAI API key\n",
|
||
"# audio_transcriber = AudioTranscriber(\"your_assemblyai_api_key\")\n",
|
||
"audio_transcriber = AudioTranscriber(os.getenv(\"ASSEMBLYAI_API_KEY\"))\n",
|
||
"\n",
|
||
"# Transcribe an audio file with speaker diarization\n",
|
||
"audio_path = \"data/harvard.wav\"\n",
|
||
"audio_chunks = audio_transcriber.transcribe_audio(\n",
|
||
" audio_path,\n",
|
||
" enable_speaker_diarization=True,\n",
|
||
" enable_auto_punctuation=True\n",
|
||
")\n",
|
||
"\n",
|
||
"# Each chunk represents a speaker utterance\n",
|
||
"for chunk in audio_chunks[:5]:\n",
|
||
" print(f\"\\nSpeaker: {chunk.metadata.get('speakers', [])}\")\n",
|
||
" print(f\"Time: {chunk.metadata.get('start_timestamp', 0)/1000:.2f}s - {chunk.metadata.get('end_timestamp', 0)/1000:.2f}s\")\n",
|
||
" print(f\"Content: {chunk.content}\")\n",
|
||
" print(f\"Confidence: {chunk.metadata.get('confidence', 'N/A'):.3f}\")\n",
|
||
" print(\"-\" * 40)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "4a915095",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 2.2 YouTube Video Transcription"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"id": "2a7f937a",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"INFO:src.audio_processing.youtube_transcriber:YouTubeTranscriber initialized\n",
|
||
"INFO:src.audio_processing.youtube_transcriber:Downloading audio from: https://www.youtube.com/watch?v=D26sUZ6DHNQ\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
" \r"
|
||
]
|
||
},
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"INFO:src.audio_processing.youtube_transcriber:Audio downloaded successfully: /var/folders/l7/_rny0yhj7yzcq_bjc7gzw4140000gn/T/youtube_transcriber/D26sUZ6DHNQ.m4a\n",
|
||
"INFO:src.audio_processing.youtube_transcriber:Starting transcription with speaker diarization...\n",
|
||
"INFO:httpx:HTTP Request: POST https://api.assemblyai.com/v2/upload \"HTTP/1.1 200 OK\"\n",
|
||
"INFO:httpx:HTTP Request: POST https://api.assemblyai.com/v2/transcript \"HTTP/1.1 200 OK\"\n",
|
||
"INFO:httpx:HTTP Request: GET https://api.assemblyai.com/v2/transcript/99f074b1-86d2-48ab-a3ef-bd39ca2cee31 \"HTTP/1.1 200 OK\"\n",
|
||
"INFO:httpx:HTTP Request: GET https://api.assemblyai.com/v2/transcript/99f074b1-86d2-48ab-a3ef-bd39ca2cee31 \"HTTP/1.1 200 OK\"\n",
|
||
"INFO:httpx:HTTP Request: GET https://api.assemblyai.com/v2/transcript/99f074b1-86d2-48ab-a3ef-bd39ca2cee31 \"HTTP/1.1 200 OK\"\n",
|
||
"INFO:httpx:HTTP Request: GET https://api.assemblyai.com/v2/transcript/99f074b1-86d2-48ab-a3ef-bd39ca2cee31 \"HTTP/1.1 200 OK\"\n",
|
||
"INFO:httpx:HTTP Request: GET https://api.assemblyai.com/v2/transcript/99f074b1-86d2-48ab-a3ef-bd39ca2cee31 \"HTTP/1.1 200 OK\"\n",
|
||
"INFO:src.audio_processing.youtube_transcriber:Transcription completed: 1 utterances\n",
|
||
"INFO:src.audio_processing.youtube_transcriber:Audio file cleaned up\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Transcribed 1 utterances from YouTube video\n",
|
||
"Video ID: D26sUZ6DHNQ\n",
|
||
"Speaker: A\n",
|
||
"Timestamp: 240s\n",
|
||
"Content: Speaker A: 99% of developers don't get sockets. What actually is a socket? You've probably seen this kind of socket, but I'm referring to a completely different kind of socket. In the software industry or computer science universe, we simply do not give justice to certain concepts like sockets, anonymous pipes, ephemeral ports, file descriptors, etc. These concepts end up being amorphous hand wavy constructs in most developers minds. And there is a whole host of additional concepts and jargon that are not taught in nearly enough detail and that needs to be addressed. So in this video I'm going to explain to you exactly what you need to know about sockets and give you an introductory taste of what sockets are. So the next time someone asks you, hey, could you explain to me what a socket is? How does TCP or UDP leverage sockets, and what layer of the OSI model do sockets operate in? You won't be drawing a blank. We should always start with a concrete answer to the question of what are sockets? At their core, sockets are an abstraction provided by operating systems to enable communication between different processes, either on the same machine or over a network. They act as endpoints in a two way communication channel. So when two machines or two applications need to talk to each other over the Internet or a local network, each side of that communication will create a socket. And a socket is essentially a software construct that wraps a combination of a protocol, so TCP or udp, an IP address and a port number. The operating system uses this combination to route messages appropriately. You can think of it like a telephone. A socket is the phone, and the IP address and port together are like the phone number and extension. The two parties must dial each other correctly in order to establish a connection. But perhaps you've heard of the OSI model, but you're curious, where do sockets fit inside that model? So to understand sockets in context, you should place them in the OSI model, which is a conceptual framework for how network systems communicate. Sockets operate primarily at the Transport layer or layer 4 of the OSI model. The application layer or layer 7, like your web browser or a backend service, calls down to the socket API asking it to send or receive data. The socket then wraps the data into TCP or UDP segments, adds some headers, and sends it to the network layer or layer 3, which deals with the IP routing. Sockets thus provide a clean interface between application logic and the underlying network stack and shields developers from the complexities of routing, packet fragmentation, retransmission, etc. Okay, but before you fully understand sockets, you have to realize that there are TCP and UDP sockets and these are the two main types of sockets. TCP or Transmission Control Protocol Sockets are connection oriented and provide reliable, ordered and error checked data transmission before data is sent. A three way handshake is performed to establish the connection involving the exchange of SYN and ACK packets. This guarantees that both sides are ready and TCP ensures that packets will arrive in order and without duplication, which makes it ideal for applications like web browsing, database access or file transfers. UDP or user Datagram protocol on the other hand is connectionless and unreliable. It simply sends datagrams to the target IP and port without any handshake. There's no guarantee of delivery, ordering or integrity. However, it's much faster and much more lightweight as a result, which makes it ideal for real time applications like video streaming where speed is prioritized over reliability. If you are interested in taking your software engineering skills to the next level, I would encourage you to build projects. I'm not talking about going down the rabbit hole of tutorial hell and building a to do list, calculator or weather app. I'm talking about building complex real world projects beyond the basics. And this is where codecrafters comes in. This platform provides interactive tools to build developer tooling from scratch. There are a number of courses that teach building Git from scratch, building an in memory, Redis database, an HTTP web server, your own Docker, your own DNS server, and many others. I personally love that there is built in support for over 20 different languages. My favorite of course is Golang, but I would highly recommend trying a newer language like Zig as well. I'm excited to announce that I'm partnering with Codecrafters to offer all my viewers 40% off. For more details, you can find a link in the description down below as well as the pinned comment. Okay, but when we talk about the socket lifecycle, we still don't fully understand the server perspective as well as the client perspective. On the server side, sockets follow a specific lifecycle. It starts with the creation of a listening socket which is bound to a specific IP address and port. This socket doesn't engage in communication directly, but instead waits for incoming client connections. Once a client initiates a connection, the server accepts it, creating a new socket instance that's dedicated to that specific client, and the original socket continues listening for new requests. This new socket becomes the channel through which all communication with that specific client occurs, and this enables a server to handle many different clients concurrently, each with its own socket in synchronous setups. This is typically handled with multi threading or multiprocessing, where each client connection is managed by a separate thread or process. This model is straightforward and works well for a small number of connections, but it does not scale efficiently. Each thread consumes memory and incurs context switching overhead. And as the number of concurrent sockets grows, the the performance quickly degrades due to contention and resource exhaustion. In high performance systems like real time APIs, message brokers or multiplayer game servers, this limitation is addressed using techniques like non blocking I O or event driven architectures. These approaches allow a single thread or a small number of threads to manage thousands of open sockets concurrently. This is achieved using system calls such as Select, Poll or even more scalable alternatives like ePoll on Linux or KQ on BSD or macOS, which notify the application only when specific sockets are ready for reading or writing. These event notification mechanisms drastically reduce CPU usage and latency by avoiding idle waiting or redundant polling, making them the foundation of frameworks such as nginx, node, js, and Asyncio. The client creates a socket and initiates a connection to the server's IP and port. If using tcp, this initiates the three way handshake. Once connected, the client can write to and read from the socket, just like working with a file. In fact, on Unix like systems, sockets are treated as file descriptors, meaning they can be used with common system calls like read and write. And if you want a reminder of what a file descriptor is, essentially when you open a file, the OS creates an entry to represent that file and store the information about that opened file. So if there are 100 files open in your OS, then there will be 100 corresponding entries in the OS somewhere in the kernel. And these entries are represented by integers like 100, 101, 102, 103 and this entry number is the file descriptor. So really it's just an integer that uniquely represents an open file for the process. If your process opens 10 files, then your process table will have 10 entries for file descriptors. Very similarly, when you open a network socket, it's also represented by an integer, in this case a socket descriptor. After the communication ends, both the client and the server are expected to close the socket to free up system resources and prevent memory leaks. Any improperly closed sockets can lead to resource exhaustion, especially on busy servers. There's also some interesting quirks around socket internals and states. So for TCP sockets, the system maintains a state machine for each connection, there's Some common states such as listen, syn sent, syn received, established fin weight one, time wait and close wait, and these states represent various phases of the connection setup, maintenance and tear down. For instance, the time wait state ensures that delayed packets from an old connection are not misinterpreted as being part of a new one. And knowledge of these states is critical in diagnosing issues like port exhaustion or socket leaks, which often plague long lived servers or high concurrency applications. But how does the operating system actually deal with port number and socket uniqueness? Every socket is uniquely identified by a five protocol source ip, source port, destination IP and destination port, and this combination allows the OS to distinguish between multiple simultaneous connections even if they are all talking to the same remote server and port. For example, multiple browser tabs connecting to example.com 443 will each have a unique source port assigned by the os, which allows the server and the client to differentiate between those sessions. This mechanism is what enables one computer to have hundreds of open connections to different or even the same remote services. While most socket discussions focus on network communication, for example TCP or UDP over ip, there is another important type known as Unix domain sockets or uds. These are not network sockets. They are used for inter process communication or IPC on the same host. So instead of using an IP address and port, you UDS uses a file path on the file system, e.g. temp app sock, as the address. They are much faster than network sockets since they bypass the network stack entirely and don't require any IP routing or protocol overhead. There's applications that you're probably very familiar with, like PostgreSQL or Redis, that often default to UDS for local client server communication when performance and security are critical. But something that's very important to note is that sockets are inherently insecure, and this is because they transmit raw data on unless explicitly encrypted. Secure communication over sockets is usually achieved via transport layer security, or tls. TLS wraps an existing socket connection and ensures that all data sent over it is encrypted and authenticated. In Python. For example, the SSL module provides functionality to wrap a plain socket in tls, and this is very important for applications transmitting sensitive information like passwords, financial transactions, private user data, etc. Any raw or misconfigured sockets that do not use TLS can become targets for man in the middle attacks or packet sniffing. But how often are sockets actually used in distributed systems and microservices? Well, as you may have guessed, in these modern software architectures, especially in microservices and distributed systems Socket based communication is absolutely everywhere. Restful APIs over HTTP, which uses TCP sockets, and GRPC over HTTP 2 which is also TCP based, are very popular protocols for service to service communication. Load balancers like nginx or Envoy and service meshes like Istio manage socket level routing and traffic shaping in high performance systems. Developers sometimes implement custom binary protocols over RAW TCP or UDP sockets for ultra low latency communication. Technologies like Kafka, Redis and Cassandra all use sockets under the hood to distribute data across nodes. There's a very clear reason why you should at least be able to articulate what a socket is. Sockets are truly the foundation of modern networked computing. Every time you load a webpage, stream a video, send an email, or interact with a mobile app, sockets are involved in transmitting the data across the wire. As a developer, having a deep understanding of how sockets work gives you control over network communication performance and security. It also enables you to build scalable backend systems, troubleshoot complex issues, and optimize communication protocols. Having a solid grasp of sockets is going to be a major asset for you. If you learned something new in this video, please give this video a thumbs up and hit the subscribe button and turn on all notifications if you want to be notified for future updates. As always, thank you very much for watching this video and happy coding.\n",
|
||
"----------------------------------------\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"from src.audio_processing.youtube_transcriber import YouTubeTranscriber\n",
|
||
"\n",
|
||
"# Initialize YouTube transcriber\n",
|
||
"youtube_transcriber = YouTubeTranscriber(os.getenv(\"ASSEMBLYAI_API_KEY\"))\n",
|
||
"\n",
|
||
"# Extract and transcribe YouTube video\n",
|
||
"youtube_url = \"https://www.youtube.com/watch?v=D26sUZ6DHNQ\"\n",
|
||
"youtube_chunks = youtube_transcriber.transcribe_youtube_video(youtube_url)\n",
|
||
"\n",
|
||
"print(f\"Transcribed {len(youtube_chunks)} utterances from YouTube video\")\n",
|
||
"\n",
|
||
"# Show speaker-separated content\n",
|
||
"for chunk in youtube_chunks[:3]:\n",
|
||
" print(f\"Video ID: {chunk.metadata.get('video_id')}\")\n",
|
||
" print(f\"Speaker: {chunk.metadata.get('speaker')}\")\n",
|
||
" print(f\"Timestamp: {chunk.metadata.get('start_time')}s\")\n",
|
||
" print(f\"Content: {chunk.content}\")\n",
|
||
" print(\"-\" * 40)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "034bdd87",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 3. Web Scraping\n",
|
||
"\n",
|
||
"The web scraping module uses Firecrawl to extract clean, structured content from websites while preserving the context needed for citations."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d3982334",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Web Scraping Features:\n",
|
||
"- **Clean extraction**: Removes ads, navigation, and irrelevant content\n",
|
||
"- **Metadata preservation**: Preserves title, URL, and page structure\n",
|
||
"- **Citation-ready**: URL fragments for precise source linking\n",
|
||
"- **Chunking**: Intelligent text segmentation with overlap"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"id": "6fb24af6",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"INFO:src.web_scraping.web_scraper:WebScraper initialized with Firecrawl\n",
|
||
"INFO:src.web_scraping.web_scraper:Scraping URL: https://blog.dailydoseofds.com/p/5-chunking-strategies-for-rag\n",
|
||
"INFO:src.web_scraping.web_scraper:Successfully scraped https://blog.dailydoseofds.com/p/5-chunking-strategies-for-rag: 24 chunks created\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Scraped 24 chunks from https://blog.dailydoseofds.com/p/5-chunking-strategies-for-rag\n",
|
||
"URL: https://blog.dailydoseofds.com/p/5-chunking-strategies-for-rag\n",
|
||
"Title: 5 Chunking Strategies For RAG - by Avi Chawla\n",
|
||
"Chunk: 0\n",
|
||
"Content: [![Daily Dose of Data Science](https://substackcdn.com/image/fetch/$s_!heKx!,w_80,h_80,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic...\n",
|
||
"Citation: [Source: 5 Chunking Strategies For RAG - by Avi Chawla, Type: web]\n",
|
||
"URL Fragment: https://blog.dailydoseofds.com/p/5-chunking-strategies-for-rag#chunk-0\n",
|
||
"--------------------------------------------------\n",
|
||
"URL: https://blog.dailydoseofds.com/p/5-chunking-strategies-for-rag\n",
|
||
"Title: 5 Chunking Strategies For RAG - by Avi Chawla\n",
|
||
"Chunk: 1\n",
|
||
"Content: By subscribing, I agree to Substack's [Terms of Use](https://substack.com/tos), and acknowledge its [Information Collection Notice](https://substack.com/ccpa#personal-data-collected) and [Privacy Poli...\n",
|
||
"Citation: [Source: 5 Chunking Strategies For RAG - by Avi Chawla, Type: web]\n",
|
||
"URL Fragment: https://blog.dailydoseofds.com/p/5-chunking-strategies-for-rag#chunk-1\n",
|
||
"--------------------------------------------------\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"from src.web_scraping.web_scraper import WebScraper\n",
|
||
"\n",
|
||
"# Initialize with Firecrawl API key\n",
|
||
"# web_scraper = WebScraper(\"your_firecrawl_api_key\")\n",
|
||
"web_scraper = WebScraper(os.getenv(\"FIRECRAWL_API_KEY\"))\n",
|
||
"# Scrape a website\n",
|
||
"url = \"https://blog.dailydoseofds.com/p/5-chunking-strategies-for-rag\"\n",
|
||
"web_chunks = web_scraper.scrape_url(url)\n",
|
||
"\n",
|
||
"print(f\"Scraped {len(web_chunks)} chunks from {url}\")\n",
|
||
"\n",
|
||
"# Each chunk contains web-specific metadata\n",
|
||
"for chunk in web_chunks[:2]:\n",
|
||
" print(f\"URL: {chunk.metadata.get('original_url', 'N/A')}\")\n",
|
||
" print(f\"Title: {chunk.metadata.get('title', 'No title')}\")\n",
|
||
" print(f\"Chunk: {chunk.chunk_index}\")\n",
|
||
" print(f\"Content: {chunk.content[:200]}...\")\n",
|
||
" print(f\"Citation: [Source: {chunk.source_file}, Type: {chunk.source_type}]\")\n",
|
||
" print(f\"URL Fragment: {chunk.metadata.get('url_fragment')}\")\n",
|
||
" print(\"-\" * 50)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "f894aa41",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 4. Embedding Generation\n",
|
||
"\n",
|
||
"After processing all modalities (documents, audio, web), we generate semantic embeddings for each chunk to enable similarity search and retrieval."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "88c232eb",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Embedding Features:\n",
|
||
"- **Efficient embedding model**: BGE (BAAI General Embedding) for high-quality embeddings\n",
|
||
"- **Batch processing**: Efficient embedding generation for large document sets\n",
|
||
"- **Query optimization**: Specialized query embedding for better retrieval"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"id": "2b90b3aa",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"INFO:src.embeddings.embedding_generator:Initializing embedding model: BAAI/bge-small-en-v1.5\n",
|
||
"INFO:src.embeddings.embedding_generator:Model initialized successfully. Embedding dimension: 384\n",
|
||
"INFO:src.embeddings.embedding_generator:Generating embeddings for 84 chunks\n",
|
||
"INFO:src.embeddings.embedding_generator:Successfully generated 84 embeddings\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Generated embeddings for 84 chunks\n",
|
||
"Embedding dimension: 384\n",
|
||
"\n",
|
||
"Chunk 1:\n",
|
||
"Source: raft.pdf\n",
|
||
"Type: pdf\n",
|
||
"Embedding shape: (384,)\n",
|
||
"Content preview: RAFT: Adapting Language Model to Domain Specific RAG\n",
|
||
"Tianjun Zhang Shishir G. Patil Naman Jain Sheng...\n",
|
||
"Embedding vector: [-4.3315459e-02 3.0148618e-02 -1.9829737e-02 -1.0716271e-02\n",
|
||
" 2.3525460e-02 -2.3313519e-02 -5.7224091e-02 1.4782891e-02\n",
|
||
" 3.9421041e-02 -3.9844923e-02 1.4756397e-02 -4.6600543e-02\n",
|
||
" 1.9644288e-02 3.7566558e-02 3.8175888e-02 3.7036702e-02\n",
|
||
" 1.2093888e-02 5.1872578e-02 2.7578833e-02 2.7658312e-02\n",
|
||
" 8.9942496e-03 1.5821070e-03 -6.4824166e-04 -1.4120574e-02\n",
|
||
" -5.1528174e-02 6.2688198e-03 -2.5406437e-02 -1.3140347e-02\n",
|
||
" -2.7181443e-02 -2.3928148e-01 4.2123292e-03 7.4709230e-03\n",
|
||
" 1.7114243e-02 2.9168392e-02 1.7402349e-03 -8.1382063e-04\n",
|
||
" -3.5261698e-02 -2.1194108e-02 2.6068753e-02 1.9246899e-02\n",
|
||
" -2.2787806e-04 -7.8815585e-03 3.6228679e-03 -8.4047886e-03\n",
|
||
" 2.2571726e-02 -7.2218925e-02 -1.6292971e-02 -1.4067589e-02\n",
|
||
" -7.2960719e-02 -1.5736625e-02]...\n",
|
||
"\n",
|
||
"Chunk 2:\n",
|
||
"Source: raft.pdf\n",
|
||
"Type: pdf\n",
|
||
"Embedding shape: (384,)\n",
|
||
"Content preview: help in answering the question, which we call,\n",
|
||
"distractor documents. RAFT accomplishes this\n",
|
||
"by citin...\n",
|
||
"Embedding vector: [-0.03433292 0.04435887 0.01171468 -0.01557082 0.04765654 -0.06728294\n",
|
||
" -0.04005064 0.03276387 0.0078253 -0.06978279 0.01636864 -0.01401506\n",
|
||
" 0.01217343 0.05606025 0.01281168 0.05196477 0.00339074 0.07786737\n",
|
||
" 0.01733932 0.05090101 -0.00296357 -0.01108307 -0.03380103 0.01321059\n",
|
||
" -0.02079654 -0.02454631 -0.0545178 -0.03162032 -0.03374785 -0.22466652\n",
|
||
" 0.03771036 -0.00938105 -0.0102919 0.03406698 -0.0303704 0.04388018\n",
|
||
" -0.01889507 -0.02042423 -0.01313746 -0.00578421 0.03233836 -0.00577756\n",
|
||
" -0.01700689 -0.00860317 0.01523839 -0.04675234 0.00740644 -0.02611536\n",
|
||
" -0.03587537 -0.02818969]...\n",
|
||
"\n",
|
||
"Chunk 3:\n",
|
||
"Source: raft.pdf\n",
|
||
"Type: pdf\n",
|
||
"Embedding shape: (384,)\n",
|
||
"Content preview: specific document collections (e.g., legal or medical docu-\n",
|
||
"ments). In these settings, general knowl...\n",
|
||
"Embedding vector: [-0.05584931 0.0563797 -0.01285515 -0.02818985 0.04622287 -0.04452564\n",
|
||
" -0.03662294 -0.01181428 0.00395135 -0.00193093 0.00216131 -0.00606294\n",
|
||
" 0.02714234 0.01282864 0.02052582 0.02499429 0.01288167 0.05086372\n",
|
||
" 0.01412144 0.06231999 0.03577432 0.04619635 0.03391798 0.00560217\n",
|
||
" -0.02519319 -0.025286 -0.04179417 -0.01442641 -0.04187373 -0.20334879\n",
|
||
" -0.00510493 0.00885739 0.01079329 0.06486582 -0.01809932 0.00323202\n",
|
||
" -0.01363084 0.02352248 -0.01583193 0.07276853 0.00179501 -0.01439989\n",
|
||
" -0.03070917 -0.02805725 -0.04357095 -0.01103859 -0.01885511 -0.00690824\n",
|
||
" -0.05669793 -0.0005395 ]...\n",
|
||
"\n",
|
||
"Query embedding shape: (384,)\n",
|
||
"Query embedding vector: [ 0.03947863 0.06425266 -0.02316239 -0.05759297 0.05373036 0.05711348\n",
|
||
" 0.02701169 0.07304345 -0.02567975 -0.00111883 0.01871372 -0.01852725\n",
|
||
" 0.00584721 0.05620776 0.07053941 0.01973931 0.00490819 -0.04243553\n",
|
||
" -0.01807439 0.04312813 0.04797639 -0.01941965 0.00034797 -0.04765672\n",
|
||
" -0.02446769 -0.00271049 -0.03497668 -0.02300256 -0.03990484 -0.09835026\n",
|
||
" -0.09206352 0.02020549 0.05234514 0.06329367 -0.00812482 0.02954237\n",
|
||
" 0.07586715 0.02188373 -0.00722576 -0.00586052 0.05524876 -0.07821137\n",
|
||
" -0.05037387 -0.00670298 -0.01771477 0.01124821 -0.03849299 0.00879745\n",
|
||
" -0.0220702 -0.08071541]...\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"from src.embeddings.embedding_generator import EmbeddingGenerator\n",
|
||
"\n",
|
||
"# Initialize embedding generator with BGE model\n",
|
||
"embedding_generator = EmbeddingGenerator(model_name=\"BAAI/bge-small-en-v1.5\")\n",
|
||
"\n",
|
||
"# Generate embeddings for all processed chunks\n",
|
||
"all_chunks = doc_chunks + text_chunks + audio_chunks + youtube_chunks + web_chunks\n",
|
||
"# all_chunks = doc_chunks + text_chunks + audio_chunks + web_chunks\n",
|
||
"embedded_chunks = embedding_generator.generate_embeddings(all_chunks)\n",
|
||
"\n",
|
||
"print(f\"Generated embeddings for {len(embedded_chunks)} chunks\")\n",
|
||
"print(f\"Embedding dimension: {embedded_chunks[0].embedding.shape[0]}\")\n",
|
||
"\n",
|
||
"# Show embedding information\n",
|
||
"for i, embedded_chunk in enumerate(embedded_chunks[:3]):\n",
|
||
" print(f\"\\nChunk {i+1}:\")\n",
|
||
" print(f\"Source: {embedded_chunk.chunk.source_file}\")\n",
|
||
" print(f\"Type: {embedded_chunk.chunk.source_type}\")\n",
|
||
" print(f\"Embedding shape: {embedded_chunk.embedding.shape}\")\n",
|
||
" print(f\"Content preview: {embedded_chunk.chunk.content[:100]}...\")\n",
|
||
" print(f\"Embedding vector: {embedded_chunk.embedding[:50]}...\")\n",
|
||
"\n",
|
||
"# Generate query embedding for search\n",
|
||
"query = \"What is the main topic of research?\"\n",
|
||
"query_embedding = embedding_generator.generate_query_embedding(query)\n",
|
||
"print(f\"\\nQuery embedding shape: {query_embedding.shape}\")\n",
|
||
"print(f\"Query embedding vector: {query_embedding[:50]}...\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d63f740b",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 5. Vector Database Storage\n",
|
||
"\n",
|
||
"All embedded chunks are stored in Milvus vector database with comprehensive metadata for fast similarity search and precise citations."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "34236bce",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Vector Database Features:\n",
|
||
"- **Milvus Lite**: Embedded vector database for fast deployment\n",
|
||
"- **IVF_FLAT indexing**: Optimized for accuracy and speed\n",
|
||
"- **Rich metadata**: Complete citation information stored with vectors\n",
|
||
"- **Flexible search**: Score thresholding and result limiting\n",
|
||
"- **Precise retrieval**: Get exact chunks by ID for citations"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"id": "f401a0e3",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"/Users/akshay/Eigen/ai-engineering-hub/notebook-lm-clone/.venv/lib/python3.11/site-packages/milvus_lite/__init__.py:15: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.\n",
|
||
" from pkg_resources import DistributionNotFound, get_distribution\n",
|
||
"huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
|
||
"To disable this warning, you can either:\n",
|
||
"\t- Avoid using `tokenizers` before the fork if possible\n",
|
||
"\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
|
||
"INFO:src.vector_database.milvus_vector_db:Milvus client initialized with database: ./milvus_lite.db\n",
|
||
"INFO:src.vector_database.milvus_vector_db:Collection 'notebookLM_collection' created successfully\n",
|
||
"INFO:src.vector_database.milvus_vector_db:Creating IVF_FLAT index with nlist=1024\n",
|
||
"INFO:src.vector_database.milvus_vector_db:Index created successfully\n",
|
||
"INFO:src.vector_database.milvus_vector_db:Inserted 84 embeddings into database\n",
|
||
"INFO:src.vector_database.milvus_vector_db:Search completed: 5 results found\n",
|
||
"INFO:src.vector_database.milvus_vector_db:Attempting to retrieve chunk with ID: txt_2_55a5ca52\n",
|
||
"INFO:src.vector_database.milvus_vector_db:Query returned 1 results\n",
|
||
"INFO:src.vector_database.milvus_vector_db:Successfully retrieved chunk: txt_2_55a5ca52\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Stored 84 embeddings in vector database\n",
|
||
"\n",
|
||
"Found 5 relevant chunks:\n",
|
||
"Score: 0.465\n",
|
||
"Source: notes.txt\n",
|
||
"Type: txt\n",
|
||
"Content: Quantum Circuit: A sequence of quantum gates applied to an initial state of qubits, represented by a...\n",
|
||
"----------------------------------------\n",
|
||
"Score: 0.537\n",
|
||
"Source: notes.txt\n",
|
||
"Type: txt\n",
|
||
"Content: Developing robust Quantum Error Correction (QEC) codes is necessary for building a fault-tolerant, l...\n",
|
||
"----------------------------------------\n",
|
||
"Score: 0.539\n",
|
||
"Source: notes.txt\n",
|
||
"Type: txt\n",
|
||
"Content: Quantum Computing Fundamentals\n",
|
||
"I. Core Concepts\n",
|
||
"Qubit (Quantum Bit): The basic unit of information i...\n",
|
||
"----------------------------------------\n",
|
||
"Score: 0.645\n",
|
||
"Source: notes.txt\n",
|
||
"Type: txt\n",
|
||
"Content: Measurement/Collapse: The act of observing a qubit forces it out of its superposition and into a def...\n",
|
||
"----------------------------------------\n",
|
||
"Score: 0.688\n",
|
||
"Source: notes.txt\n",
|
||
"Type: txt\n",
|
||
"Content: Superconducting Qubits: Qubits built using superconducting circuits (like Josephson junctions) opera...\n",
|
||
"----------------------------------------\n",
|
||
"\n",
|
||
"Retrieved chunk: notes.txt\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"from src.vector_database.milvus_vector_db import MilvusVectorDB\n",
|
||
"\n",
|
||
"# Initialize Milvus vector database\n",
|
||
"vector_db = MilvusVectorDB(\n",
|
||
" db_path=\"./milvus_lite.db\",\n",
|
||
" collection_name=\"notebookLM_collection\"\n",
|
||
")\n",
|
||
"\n",
|
||
"# Create index for fast similarity search\n",
|
||
"vector_db.create_index()\n",
|
||
"\n",
|
||
"# Insert all embedded chunks\n",
|
||
"vector_db.insert_embeddings(embedded_chunks)\n",
|
||
"print(f\"Stored {len(embedded_chunks)} embeddings in vector database\")\n",
|
||
"\n",
|
||
"# Perform similarity search\n",
|
||
"query_embedding = embedding_generator.generate_query_embedding(\n",
|
||
" \"Most prominent quantum computing algorithms\"\n",
|
||
")\n",
|
||
"\n",
|
||
"search_results = vector_db.search(\n",
|
||
" query_embedding,\n",
|
||
" limit=5,\n",
|
||
")\n",
|
||
"\n",
|
||
"print(f\"\\nFound {len(search_results)} relevant chunks:\")\n",
|
||
"for result in search_results:\n",
|
||
" print(f\"Score: {result['score']:.3f}\")\n",
|
||
" print(f\"Source: {result['citation']['source_file']}\")\n",
|
||
" print(f\"Type: {result['citation']['source_type']}\")\n",
|
||
" print(f\"Content: {result['content'][:100]}...\")\n",
|
||
" print(\"-\" * 40)\n",
|
||
"\n",
|
||
"# Retrieve specific chunk by ID\n",
|
||
"chunk_id = search_results[0]['id']\n",
|
||
"chunk_data = vector_db.get_chunk_by_id(chunk_id)\n",
|
||
"print(f\"\\nRetrieved chunk: {chunk_data['source_file']}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "37baa16d",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 6. Memory Layer Integration\n",
|
||
"\n",
|
||
"The memory layer uses Zep to maintain conversation context and user sessions, enabling personalized and contextual interactions."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "10ad20f7",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Memory Features:\n",
|
||
"- **Session management**: Persistent user sessions across interactions\n",
|
||
"- **Context preservation**: Maintains conversation history and context\n",
|
||
"- **Source tracking**: Remembers which documents were referenced\n",
|
||
"- **User personalization**: Adapts responses based on user preferences"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"id": "06d0a1ee",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"INFO:httpx:HTTP Request: GET https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json \"HTTP/1.1 200 OK\"\n",
|
||
"INFO:httpx:HTTP Request: GET https://api.getzep.com/api/v2/users/demo_user \"HTTP/1.1 200 OK\"\n",
|
||
"INFO:src.memory.memory_layer:Using existing user: demo_user\n",
|
||
"INFO:httpx:HTTP Request: DELETE https://api.getzep.com/api/v2/threads/demo_session \"HTTP/1.1 200 OK\"\n",
|
||
"INFO:src.memory.memory_layer:Deleted previous session: demo_session\n",
|
||
"INFO:httpx:HTTP Request: POST https://api.getzep.com/api/v2/threads \"HTTP/1.1 201 Created\"\n",
|
||
"INFO:src.memory.memory_layer:Created new session: demo_session\n",
|
||
"INFO:src.memory.memory_layer:NotebookMemoryLayer initialized for user demo_user, session demo_session\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Memory initialized for user: demo_user\n",
|
||
"Session ID: demo_session\n",
|
||
"Memory layer ready for conversation tracking\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"from src.memory.memory_layer import NotebookMemoryLayer\n",
|
||
"\n",
|
||
"# Initialize memory layer with Zep\n",
|
||
"memory = NotebookMemoryLayer(\n",
|
||
" user_id=\"demo_user\",\n",
|
||
" session_id=\"demo_session\",\n",
|
||
" create_new_session=True\n",
|
||
")\n",
|
||
"\n",
|
||
"# Memory automatically tracks conversations\n",
|
||
"print(f\"Memory initialized for user: {memory.user_id}\")\n",
|
||
"print(f\"Session ID: {memory.session_id}\")\n",
|
||
"\n",
|
||
"# Memory will be used by RAG system to:\n",
|
||
"# 1. Maintain conversation context\n",
|
||
"# 2. Store user preferences\n",
|
||
"# 3. Track document interactions\n",
|
||
"# 4. Enable follow-up questions\n",
|
||
"\n",
|
||
"# Memory stores all conversation context for future reference\n",
|
||
"print(\"Memory layer ready for conversation tracking\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "4da62c6b",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 7. Intelligent Response generation\n",
|
||
"\n",
|
||
"The RAG (Retrieval-Augmented Generation) system combines vector search with LLM generation to provide accurate, cited responses grounded in our source documents."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "6aa9d2e9",
|
||
"metadata": {},
|
||
"source": [
|
||
"### RAG Features:\n",
|
||
"- **Semantic retrieval**: Finds most relevant chunks using vector similarity\n",
|
||
"- **Citation integration**: Automatic inline citations with source tracking\n",
|
||
"- **Context optimization**: Intelligent chunk selection and formatting\n",
|
||
"- **LLM integration**: Uses powerful LLMs for natural response generation\n",
|
||
"- **Source transparency**: Complete traceability of information sources"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"id": "cc9ff5dc",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"INFO:src.generation.rag:RAG Generator initialized with gpt-4o-mini\n",
|
||
"INFO:src.generation.rag:Generating response for: 'What is RAFT and how does it differ from RAG and f...'\n",
|
||
"INFO:src.vector_database.milvus_vector_db:Search completed: 10 results found\n",
|
||
"\u001b[92m22:26:48 - LiteLLM:INFO\u001b[0m: utils.py:3258 - \n",
|
||
"LiteLLM completion() model= gpt-4o-mini; provider = openai\n",
|
||
"INFO:LiteLLM:\n",
|
||
"LiteLLM completion() model= gpt-4o-mini; provider = openai\n",
|
||
"INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
|
||
"\u001b[92m22:26:55 - LiteLLM:INFO\u001b[0m: utils.py:1260 - Wrapper: Completed Call, calling success_handler\n",
|
||
"INFO:LiteLLM:Wrapper: Completed Call, calling success_handler\n",
|
||
"INFO:src.generation.rag:Response generated successfully using 4 sources\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Query: What is RAFT and how does it differ from RAG and fine tuning?\n",
|
||
"Response: RAFT, which stands for \"Adapting Language Model to Domain Specific RAG,\" is a training strategy designed to enhance the performance of language models in answering questions within specific domains, particularly in \"open-book\" settings where the model can reference a set of documents during inference [4]. \n",
|
||
"\n",
|
||
"RAFT differs from the traditional RAG (Retrieval-Augmented Generation) approach in that it adapts large language models (LLMs) to read solutions from a curated set of positive and negative documents, rather than relying solely on the outputs from a retriever that mixes memorization and reading [2]. This adaptation allows RAFT to better process context and extract relevant information, which is crucial for answering questions accurately.\n",
|
||
"\n",
|
||
"In comparison to standard fine-tuning methods, RAFT incorporates both instructional tuning and context comprehension into its training dataset. This dual approach helps preserve and enhance the model's ability to process text effectively, as opposed to merely training on question-answer pairs, which can lead to confusion in context interpretation [3]. For instance, a domain-specific fine-tuned (DSF) model might misidentify a screenwriter by citing a film instead of providing the correct name, while the RAFT model would accurately answer the question [3].\n",
|
||
"\n",
|
||
"Overall, RAFT represents a significant advancement over traditional RAG and fine-tuning methods by emphasizing the importance of context processing and the incorporation of distractor documents during training, which ultimately leads to improved performance in question-answering tasks [4].\n",
|
||
"Sources used: 4\n",
|
||
"\n",
|
||
"[1] raft.pdf\n",
|
||
" Page: 5\n",
|
||
" Type: pdf\n",
|
||
" Relevance: 0.468\n",
|
||
"\n",
|
||
"[2] raft.pdf\n",
|
||
" Page: 3\n",
|
||
" Type: pdf\n",
|
||
" Relevance: 0.478\n",
|
||
"\n",
|
||
"[3] raft.pdf\n",
|
||
" Page: 6\n",
|
||
" Type: pdf\n",
|
||
" Relevance: 0.510\n",
|
||
"\n",
|
||
"[4] raft.pdf\n",
|
||
" Page: 9\n",
|
||
" Type: pdf\n",
|
||
" Relevance: 0.513\n",
|
||
"\n",
|
||
"Citation summary:\n",
|
||
"• raft.pdf (pdf) - Page 5\n",
|
||
"• raft.pdf (pdf) - Page 3\n",
|
||
"• raft.pdf (pdf) - Page 6\n",
|
||
"• raft.pdf (pdf) - Page 9\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"from src.generation.rag import RAGGenerator\n",
|
||
"\n",
|
||
"# Initialize RAG system\n",
|
||
"rag_generator = RAGGenerator(\n",
|
||
" embedding_generator=embedding_generator,\n",
|
||
" vector_db=vector_db,\n",
|
||
" openai_api_key=os.getenv(\"OPENAI_API_KEY\"),\n",
|
||
" model_name=\"gpt-4o-mini\"\n",
|
||
")\n",
|
||
"\n",
|
||
"# Process a user query\n",
|
||
"user_query = \"What is RAFT and how does it differ from RAG and fine tuning?\"\n",
|
||
"\n",
|
||
"# RAG system performs:\n",
|
||
"# 1. Query embedding generation\n",
|
||
"# 2. Vector similarity search\n",
|
||
"# 3. Context formatting with citations\n",
|
||
"# 4. LLM response generation\n",
|
||
"# 5. Citation integration\n",
|
||
"\n",
|
||
"rag_result = rag_generator.generate_response(user_query)\n",
|
||
"\n",
|
||
"print(f\"Query: {rag_result.query}\")\n",
|
||
"print(f\"Response: {rag_result.response}\")\n",
|
||
"print(f\"Sources used: {len(rag_result.sources_used)}\")\n",
|
||
"\n",
|
||
"# Show citation details\n",
|
||
"for i, source in enumerate(rag_result.sources_used, 1):\n",
|
||
" print(f\"\\n[{i}] {source['source_file']}\")\n",
|
||
" if source.get('page_number'):\n",
|
||
" print(f\" Page: {source['page_number']}\")\n",
|
||
" print(f\" Type: {source['source_type']}\")\n",
|
||
" print(f\" Relevance: {source['relevance_score']:.3f}\")\n",
|
||
"\n",
|
||
"# The response includes inline citations like [1], [2], etc.\n",
|
||
"print(f\"\\nCitation summary:\")\n",
|
||
"print(rag_result.get_citation_summary())"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ff058e0f",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 8. Memory Storage for Conversations\n",
|
||
"\n",
|
||
"Each RAG interaction is stored in memory for context preservation and conversation continuity. Here's what a sample conversation turn looks like:\n",
|
||
"\n",
|
||
"```\n",
|
||
"conversation_turn = {\n",
|
||
" \"user_message\": user_query,\n",
|
||
" \"assistant_response\": rag_result.response,\n",
|
||
" \"sources_used\": rag_result.sources_used,\n",
|
||
" \"timestamp\": datetime.now().isoformat(),\n",
|
||
" \"session_metadata\": {\n",
|
||
" \"retrieval_count\": rag_result.retrieval_count,\n",
|
||
" \"generation_tokens\": rag_result.generation_tokens\n",
|
||
" }\n",
|
||
"}\n",
|
||
"```"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ed9f7000",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Memory Storage Features:\n",
|
||
"- **Automatic storage**: Every conversation turn is preserved\n",
|
||
"- **Rich metadata**: Includes sources, timestamps, and generation stats\n",
|
||
"- **Context continuity**: Enables natural follow-up conversations\n",
|
||
"- **Session management**: Clean session boundaries for different topics"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 11,
|
||
"id": "abc58633",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"INFO:httpx:HTTP Request: POST https://api.getzep.com/api/v2/threads/demo_session/messages \"HTTP/1.1 201 Created\"\n",
|
||
"INFO:httpx:HTTP Request: POST https://api.getzep.com/api/v2/threads/demo_session/messages \"HTTP/1.1 201 Created\"\n",
|
||
"INFO:httpx:HTTP Request: POST https://api.getzep.com/api/v2/graph \"HTTP/1.1 202 Accepted\"\n",
|
||
"INFO:src.memory.memory_layer:Saved conversation turn with 4 sources\n",
|
||
"INFO:src.memory.memory_layer:Waiting 10s for Zep indexing...\n",
|
||
"INFO:httpx:HTTP Request: GET https://api.getzep.com/api/v2/threads/demo_session/context \"HTTP/1.1 200 OK\"\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"Conversation Context:\n",
|
||
"\n",
|
||
"FACTS and ENTITIES represent relevant context to the current conversation.\n",
|
||
"\n",
|
||
"# These are the most relevant facts and their valid date ranges\n",
|
||
"# format: FACT (Date range: from - to)\n",
|
||
"<FACTS>\n",
|
||
" - RAFT is a specialized training method that enhances the capabilities of language models in specific domains. (2025-10-06 12:10:00 - present)\n",
|
||
" - RAFT differs from traditional RAG. (2025-10-06 12:01:23 - present)\n",
|
||
" - RAFT has an advantage in processing context more effectively than DSF models. (2025-10-06 12:01:23 - present)\n",
|
||
" - RAFT involves organizing the training dataset in a way that includes distractor documents and ensures some portions lack oracle documents in their context. (2025-10-06 12:01:23 - present)\n",
|
||
" - RAFT differs from standard fine-tuning methods. (2025-10-06 12:01:23 - present)\n",
|
||
"</FACTS>\n",
|
||
"\n",
|
||
"# These are the most relevant entities\n",
|
||
"# Name: ENTITY_NAME\n",
|
||
"# Label: entity_label (if present)\n",
|
||
"# Attributes: (if present)\n",
|
||
"# attr_name: attr_value\n",
|
||
"# Summary: entity summary\n",
|
||
"<ENTITIES>\n",
|
||
" - Name: fine tuning\n",
|
||
" Label: Topic\n",
|
||
" Summary: RAFT differs from standard fine-tuning by incorporating positive and negative documents during training, emphasizing context comprehension, and improving accuracy over domain-specific fine-tuned (DSF) models.\n",
|
||
"</ENTITIES>\n",
|
||
"\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Save the conversation turn to memory\n",
|
||
"memory.save_conversation_turn(rag_result)\n",
|
||
"memory.wait_for_indexing()\n",
|
||
"# Get the conversation context\n",
|
||
"context = memory.get_conversation_context()\n",
|
||
"print(f\"\\nConversation Context:\\n{context}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "944413e3",
|
||
"metadata": {},
|
||
"source": [
|
||
"Memory enables:\n",
|
||
"1. Follow-up questions with context\n",
|
||
"2. Reference to previous discussions\n",
|
||
"3. Personalized responses based on history\n",
|
||
"4. Source preference learning"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 12,
|
||
"id": "44913fed",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"INFO:src.generation.rag:Generating response for: 'Can you elaborate more on the distractor and oracl...'\n",
|
||
"INFO:src.vector_database.milvus_vector_db:Search completed: 10 results found\n",
|
||
"\u001b[92m22:28:47 - LiteLLM:INFO\u001b[0m: utils.py:3258 - \n",
|
||
"LiteLLM completion() model= gpt-4o-mini; provider = openai\n",
|
||
"INFO:LiteLLM:\n",
|
||
"LiteLLM completion() model= gpt-4o-mini; provider = openai\n",
|
||
"INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
|
||
"\u001b[92m22:28:57 - LiteLLM:INFO\u001b[0m: utils.py:1260 - Wrapper: Completed Call, calling success_handler\n",
|
||
"INFO:LiteLLM:Wrapper: Completed Call, calling success_handler\n",
|
||
"INFO:src.generation.rag:Response generated successfully using 4 sources\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"Follow-up response: In the context of the RAFT training method, the use of oracle and distractor documents plays a crucial role in enhancing the performance of language models in specific domains. \n",
|
||
"\n",
|
||
"1. **Oracle Documents**: These are the documents that contain the relevant information needed to answer specific questions. In the RAFT framework, for a certain fraction (P) of the questions in the dataset, the oracle document is retained alongside distractor documents. This means that the model is trained with access to the correct context that directly relates to the questions being asked [1].\n",
|
||
"\n",
|
||
"2. **Distractor Documents**: These are documents that do not contain answer-relevant information but are included in the training process to challenge the model. For the remaining fraction (1 - P) of the questions, only distractor documents are included, without any oracle document. This approach is designed to help the model learn to discern relevant information from irrelevant information, thereby improving its reasoning capabilities [2], [3].\n",
|
||
"\n",
|
||
"3. **Training Setup**: The training setup involves including multiple distractor documents alongside the oracle document during training. This method aims to enhance the model's ability to answer questions accurately by exposing it to both relevant and irrelevant contexts. The findings suggest that including a certain percentage of training data without oracle documents can be beneficial for downstream tasks related to answering questions [3].\n",
|
||
"\n",
|
||
"4. **Generalization and Performance**: The RAFT method has been shown to improve the model's performance across various benchmarks, including PubMed, HotpotQA, and Gorilla datasets. This improvement is attributed to the structured training that incorporates both oracle and distractor documents, allowing the model to better handle domain-specific retrieval-augmented generation (RAG) tasks [2], [4].\n",
|
||
"\n",
|
||
"In summary, the strategic use of oracle and distractor documents in the RAFT training method is designed to enhance the model's ability to process context effectively and improve its performance in answering domain-specific questions.\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"INFO:httpx:HTTP Request: DELETE https://api.getzep.com/api/v2/threads/demo_session \"HTTP/1.1 200 OK\"\n",
|
||
"INFO:httpx:HTTP Request: POST https://api.getzep.com/api/v2/threads \"HTTP/1.1 201 Created\"\n",
|
||
"INFO:src.memory.memory_layer:Session demo_session cleared and recreated\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"Session cleared - fresh conversation context\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Example follow-up query\n",
|
||
"followup_query = \"Can you elaborate more on the distractor and oracle document part?\"\n",
|
||
"\n",
|
||
"# Memory provides context from previous conversation\n",
|
||
"followup_result = rag_generator.generate_response(followup_query + f\"Conversation Context:\\n{context}\")\n",
|
||
"# followup_result = rag_generator.generate_response(followup_query)\n",
|
||
"print(f\"\\nFollow-up response: {followup_result.response}\")\n",
|
||
"\n",
|
||
"# Clear session when needed\n",
|
||
"memory.clear_session()\n",
|
||
"print(\"\\nSession cleared - fresh conversation context\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "556beeee",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 9. Podcast Script Generation\n",
|
||
"\n",
|
||
"Transform your documents into engaging podcast conversations using AI-powered script generation."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "4db30744",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stderr",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"INFO:src.podcast.script_generator:Podcast script generator initialized with gpt-4o-mini\n",
|
||
"INFO:src.web_scraping.web_scraper:Scraping URL: https://thinkingmachines.ai/blog/lora/\n",
|
||
"INFO:src.web_scraping.web_scraper:Successfully scraped https://thinkingmachines.ai/blog/lora/: 46 chunks created\n",
|
||
"INFO:src.podcast.script_generator:Generating podcast script from website: https://thinkingmachines.ai/blog/lora/\n",
|
||
"\u001b[92m22:30:26 - LiteLLM:INFO\u001b[0m: utils.py:3258 - \n",
|
||
"LiteLLM completion() model= gpt-4o-mini; provider = openai\n",
|
||
"INFO:LiteLLM:\n",
|
||
"LiteLLM completion() model= gpt-4o-mini; provider = openai\n",
|
||
"INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
|
||
"\u001b[92m22:30:46 - LiteLLM:INFO\u001b[0m: utils.py:1260 - Wrapper: Completed Call, calling success_handler\n",
|
||
"INFO:LiteLLM:Wrapper: Completed Call, calling success_handler\n",
|
||
"INFO:src.podcast.script_generator:Generated website script with 20 lines\n"
|
||
]
|
||
},
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\n",
|
||
"Generated podcast script:\n",
|
||
"Source: https://thinkingmachines.ai/blog/lora/\n",
|
||
"Total lines: 20\n",
|
||
"\n",
|
||
"1. Speaker 1: Welcome everyone to today's episode of our podcast! I'm thrilled to have you join us as we explore some intriguing insights from a recent document titled 'LoRA Without Regret.'.\n",
|
||
"\n",
|
||
"2. Speaker 2: Thanks for tuning in! This topic is super relevant in the world of AI and machine learning, especially when it comes to fine-tuning large language models. So, what exactly is LoRA?\n",
|
||
"\n",
|
||
"3. Speaker 1: Great question! LoRA stands for Low-Rank Adaptation, and it's a method used to fine-tune large language models more efficiently. Instead of adjusting the entire model, it focuses on a smaller set of parameters, making the process faster and less resource-intensive.\n",
|
||
"\n",
|
||
"4. Speaker 2: Exactly! The document mentions that today's language models can have trillions of parameters. It just doesn’t make sense to use all of that when you only need to make specific updates, right?\n",
|
||
"\n",
|
||
"5. Speaker 1: Absolutely! Using a terabit of weights to represent updates from a gigabit or megabit of training data seems pretty wasteful. LoRA helps by creating a low-dimensional representation of these updates.\n",
|
||
"\n",
|
||
"6. Speaker 2: And one of the significant advantages of using LoRA is multi-tenant serving. This means that with LoRA, a single server can handle multiple model versions simultaneously. That’s a big win for efficiency!\n",
|
||
"\n",
|
||
"7. Speaker 1: Right! Plus, because LoRA has fewer parameters to store, it uses less memory, making it easier and faster to set up or transfer models between machines.\n",
|
||
"\n",
|
||
"8. Speaker 2: Speaking of efficiency, the document also highlights that LoRA can be trained on layouts only slightly larger than what’s used for sampling. This makes the training process much more accessible.\n",
|
||
"\n",
|
||
"9. Speaker 1: Definitely! But it’s not all rainbows and butterflies. The document points out that while LoRA performs well for smaller datasets, it struggles with larger datasets that exceed its capacity.\n",
|
||
"\n",
|
||
"10. Speaker 2: Exactly! In those cases, it can underperform compared to what’s known as Full Fine-Tuning, or FullFT. It's interesting how the effectiveness of LoRA depends on the relationship between model capacity and dataset size.\n",
|
||
"\n",
|
||
"11. Speaker 1: And that brings us to the experiments they conducted. They found that under the right conditions, LoRA can match FullFT's performance. But what are those conditions?\n",
|
||
"\n",
|
||
"12. Speaker 2: They looked at factors like the size of the training dataset and the number of LoRA parameters. They even used various datasets for their experiments to ensure their results were generalizable.\n",
|
||
"\n",
|
||
"13. Speaker 1: Exactly! For small-to-medium instruction-tuning datasets, LoRA performed just as well as FullFT. But when the dataset size exceeded what LoRA could handle, that’s when things started to go downhill.\n",
|
||
"\n",
|
||
"14. Speaker 2: It’s fascinating to see how batch sizes also play a role in LoRA’s performance. They found that LoRA is less tolerant of larger batch sizes compared to FullFT, which is a unique insight.\n",
|
||
"\n",
|
||
"15. Speaker 1: Totally! The document emphasizes that this wasn’t improved even when they adjusted the LoRA rank. That's something to keep in mind for anyone working with these models.\n",
|
||
"\n",
|
||
"16. Speaker 2: And let’s not forget about reinforcement learning! LoRA showed promising results there, performing equally well even with small ranks, which was expected based on some theoretical arguments.\n",
|
||
"\n",
|
||
"17. Speaker 1: Yes, it really opens the door for using efficient fine-tuning methods in various applications, especially in scenarios where memory and computing resources are limited.\n",
|
||
"\n",
|
||
"18. Speaker 2: So, to wrap it up, while LoRA has its limitations, its efficiency and flexibility make it a compelling choice for many applications, particularly in post-training scenarios.\n",
|
||
"\n",
|
||
"19. Speaker 1: Absolutely! It’s all about finding the right balance between performance and resource management. Thanks for joining us today, everyone! We hope you found this discussion insightful.\n",
|
||
"\n",
|
||
"20. Speaker 2: Yes, thanks for listening! If you enjoyed the episode, feel free to share it with friends or leave us a review. Until next time!\n"
|
||
]
|
||
},
|
||
{
|
||
"ename": "FileNotFoundError",
|
||
"evalue": "[Errno 2] No such file or directory: 'outputs/generated_podcast.json'",
|
||
"output_type": "error",
|
||
"traceback": [
|
||
"\u001b[31m---------------------------------------------------------------------------\u001b[39m",
|
||
"\u001b[31mFileNotFoundError\u001b[39m Traceback (most recent call last)",
|
||
"\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[13]\u001b[39m\u001b[32m, line 33\u001b[39m\n\u001b[32m 31\u001b[39m \u001b[38;5;66;03m# Save script as JSON\u001b[39;00m\n\u001b[32m 32\u001b[39m script_json = podcast_script.to_json()\n\u001b[32m---> \u001b[39m\u001b[32m33\u001b[39m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28;43mopen\u001b[39;49m\u001b[43m(\u001b[49m\u001b[33;43m\"\u001b[39;49m\u001b[33;43moutputs/generated_podcast.json\u001b[39;49m\u001b[33;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[33;43m\"\u001b[39;49m\u001b[33;43mw\u001b[39;49m\u001b[33;43m\"\u001b[39;49m\u001b[43m)\u001b[49m \u001b[38;5;28;01mas\u001b[39;00m f:\n\u001b[32m 34\u001b[39m f.write(script_json)\n\u001b[32m 35\u001b[39m \u001b[38;5;28mprint\u001b[39m(\u001b[33m\"\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[33mScript saved to outputs/generated_podcast.json\u001b[39m\u001b[33m\"\u001b[39m)\n",
|
||
"\u001b[36mFile \u001b[39m\u001b[32m~/Eigen/ai-engineering-hub/notebook-lm-clone/.venv/lib/python3.11/site-packages/IPython/core/interactiveshell.py:343\u001b[39m, in \u001b[36m_modified_open\u001b[39m\u001b[34m(file, *args, **kwargs)\u001b[39m\n\u001b[32m 336\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m file \u001b[38;5;129;01min\u001b[39;00m {\u001b[32m0\u001b[39m, \u001b[32m1\u001b[39m, \u001b[32m2\u001b[39m}:\n\u001b[32m 337\u001b[39m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[32m 338\u001b[39m \u001b[33mf\u001b[39m\u001b[33m\"\u001b[39m\u001b[33mIPython won\u001b[39m\u001b[33m'\u001b[39m\u001b[33mt let you open fd=\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mfile\u001b[38;5;132;01m}\u001b[39;00m\u001b[33m by default \u001b[39m\u001b[33m\"\u001b[39m\n\u001b[32m 339\u001b[39m \u001b[33m\"\u001b[39m\u001b[33mas it is likely to crash IPython. If you know what you are doing, \u001b[39m\u001b[33m\"\u001b[39m\n\u001b[32m 340\u001b[39m \u001b[33m\"\u001b[39m\u001b[33myou can use builtins\u001b[39m\u001b[33m'\u001b[39m\u001b[33m open.\u001b[39m\u001b[33m\"\u001b[39m\n\u001b[32m 341\u001b[39m )\n\u001b[32m--> \u001b[39m\u001b[32m343\u001b[39m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mio_open\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfile\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m*\u001b[49m\u001b[43m*\u001b[49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
|
||
"\u001b[31mFileNotFoundError\u001b[39m: [Errno 2] No such file or directory: 'outputs/generated_podcast.json'"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"from src.podcast.script_generator import PodcastScriptGenerator\n",
|
||
"\n",
|
||
"# Initialize script generator\n",
|
||
"script_generator = PodcastScriptGenerator(\n",
|
||
" openai_api_key=os.getenv(\"OPENAI_API_KEY\"),\n",
|
||
" model_name=\"gpt-4o-mini\"\n",
|
||
")\n",
|
||
"\n",
|
||
"# Generate podcast from document content\n",
|
||
"url = \"https://thinkingmachines.ai/blog/lora/\"\n",
|
||
"web_chunks = web_scraper.scrape_url(url)\n",
|
||
"# document_content = \"\\n\\n\".join([chunk.content for chunk in doc_chunks[:10]])\n",
|
||
"\n",
|
||
"podcast_script = script_generator.generate_script_from_website(\n",
|
||
" website_chunks=web_chunks,\n",
|
||
" source_url=url,\n",
|
||
" podcast_style=\"conversational\", # or \"educational\", \"interview\", \"debate\"\n",
|
||
" target_duration=\"10 minutes\"\n",
|
||
")\n",
|
||
"\n",
|
||
"print(f\"\\nGenerated podcast script:\")\n",
|
||
"print(f\"Source: {podcast_script.source_document}\")\n",
|
||
"print(f\"Total lines: {podcast_script.total_lines}\")\n",
|
||
"# print(f\"Estimated duration: {podcast_script.estimated_duration}\")\n",
|
||
"\n",
|
||
"# Display the conversation\n",
|
||
"for i, line_dict in enumerate(podcast_script.script, 1):\n",
|
||
" speaker, dialogue = next(iter(line_dict.items()))\n",
|
||
" print(f\"\\n{i}. {speaker}: {dialogue}\")\n",
|
||
"\n",
|
||
"# Save script as JSON\n",
|
||
"script_json = podcast_script.to_json()\n",
|
||
"\n",
|
||
"if not os.path.exists(\"outputs\"):\n",
|
||
" os.makedirs(\"outputs\")\n",
|
||
"\n",
|
||
"with open(\"outputs/generated_podcast.json\", \"w\") as f:\n",
|
||
" f.write(script_json)\n",
|
||
"print(\"\\nScript saved to outputs/generated_podcast.json\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "0585cdd3",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Podcast Script Features:\n",
|
||
"- **Multiple styles**: Conversational, educational, interview, debate formats\n",
|
||
"- **Duration control**: 5, 10, 15, or 20-minute target lengths\n",
|
||
"- **Natural dialogue**: AI-generated conversations between two speakers\n",
|
||
"- **JSON export**: Structured format for further processing"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "dc8bcb7a",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1134bedb",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 10. Text-to-Speech Audio Generation\n",
|
||
"\n",
|
||
"Convert podcast scripts into high-quality multi-speaker audio using Kokoro TTS."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 39,
|
||
"id": "d2cdcff0",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"\u001b[31mERROR: Ignored the following versions that require a different python version: 0.8.1 Requires-Python >=3.10,<3.13; 0.8.2 Requires-Python >=3.10,<3.13; 0.8.3 Requires-Python >=3.10,<3.13; 0.8.4 Requires-Python >=3.10,<3.13; 0.9.2 Requires-Python >=3.10,<3.13; 0.9.4 Requires-Python >=3.10,<3.13\u001b[0m\u001b[31m\n",
|
||
"\u001b[0m\u001b[31mERROR: Could not find a version that satisfies the requirement kokoro>=0.9.4 (from versions: 0.2.1, 0.2.2, 0.2.3, 0.3.0, 0.3.1, 0.3.2, 0.3.3, 0.3.4, 0.3.5, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.7.6, 0.7.8, 0.7.9, 0.7.11, 0.7.12, 0.7.13, 0.7.14, 0.7.15, 0.7.16)\u001b[0m\u001b[31m\n",
|
||
"\u001b[0m\u001b[31mERROR: No matching distribution found for kokoro>=0.9.4\u001b[0m\u001b[31m\n",
|
||
"\u001b[0m"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"!pip install \"kokoro>=0.9.4\""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 38,
|
||
"id": "b4a7368a",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Kokoro not installed. Install with: pip install kokoro>=0.9.4\n"
|
||
]
|
||
},
|
||
{
|
||
"ename": "ImportError",
|
||
"evalue": "Kokoro TTS not available. Install with: pip install kokoro>=0.9.4 soundfile",
|
||
"output_type": "error",
|
||
"traceback": [
|
||
"\u001b[31m---------------------------------------------------------------------------\u001b[39m",
|
||
"\u001b[31mImportError\u001b[39m Traceback (most recent call last)",
|
||
"\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[38]\u001b[39m\u001b[32m, line 4\u001b[39m\n\u001b[32m 1\u001b[39m \u001b[38;5;28;01mfrom\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34;01msrc\u001b[39;00m\u001b[34;01m.\u001b[39;00m\u001b[34;01mpodcast\u001b[39;00m\u001b[34;01m.\u001b[39;00m\u001b[34;01mtext_to_speech\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;28;01mimport\u001b[39;00m PodcastTTSGenerator\n\u001b[32m 3\u001b[39m \u001b[38;5;66;03m# Initialize TTS generator\u001b[39;00m\n\u001b[32m----> \u001b[39m\u001b[32m4\u001b[39m tts_generator = \u001b[43mPodcastTTSGenerator\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[32m 6\u001b[39m \u001b[38;5;66;03m# Generate audio from podcast script\u001b[39;00m\n\u001b[32m 7\u001b[39m output_dir = \u001b[33m\"\u001b[39m\u001b[33moutputs/\u001b[39m\u001b[33m\"\u001b[39m\n",
|
||
"\u001b[36mFile \u001b[39m\u001b[32m~/notebook-lm/src/podcast/text_to_speech.py:33\u001b[39m, in \u001b[36mPodcastTTSGenerator.__init__\u001b[39m\u001b[34m(self, lang_code, sample_rate)\u001b[39m\n\u001b[32m 31\u001b[39m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34m__init__\u001b[39m(\u001b[38;5;28mself\u001b[39m, lang_code: \u001b[38;5;28mstr\u001b[39m = \u001b[33m'\u001b[39m\u001b[33ma\u001b[39m\u001b[33m'\u001b[39m, sample_rate: \u001b[38;5;28mint\u001b[39m = \u001b[32m24000\u001b[39m):\n\u001b[32m 32\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m KPipeline \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[32m---> \u001b[39m\u001b[32m33\u001b[39m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mImportError\u001b[39;00m(\u001b[33m\"\u001b[39m\u001b[33mKokoro TTS not available. Install with: pip install kokoro>=0.9.4 soundfile\u001b[39m\u001b[33m\"\u001b[39m)\n\u001b[32m 35\u001b[39m \u001b[38;5;28mself\u001b[39m.sample_rate = sample_rate\n\u001b[32m 36\u001b[39m \u001b[38;5;28mself\u001b[39m.pipeline = KPipeline(lang_code=lang_code)\n",
|
||
"\u001b[31mImportError\u001b[39m: Kokoro TTS not available. Install with: pip install kokoro>=0.9.4 soundfile"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"from src.podcast.text_to_speech import PodcastTTSGenerator\n",
|
||
"\n",
|
||
"# Initialize TTS generator\n",
|
||
"tts_generator = PodcastTTSGenerator()\n",
|
||
"\n",
|
||
"# Generate audio from podcast script\n",
|
||
"output_dir = \"outputs/\"\n",
|
||
"\n",
|
||
"audio_files = tts_generator.generate_podcast_audio(\n",
|
||
" podcast_script=podcast_script,\n",
|
||
" output_dir=output_dir,\n",
|
||
" combine_audio=True\n",
|
||
")\n",
|
||
"\n",
|
||
"print(f\"Generated {len(audio_files)} audio files:\")\n",
|
||
"for audio_file in audio_files:\n",
|
||
" file_name = Path(audio_file).name\n",
|
||
" print(f\" - {file_name}\")\n",
|
||
" \n",
|
||
" # Show file info\n",
|
||
" if \"complete_podcast\" in file_name:\n",
|
||
" print(f\" 📻 Complete podcast ready!\")\n",
|
||
" elif \"speaker_1\" in file_name:\n",
|
||
" print(f\" 🎤 Speaker 1 segment\")\n",
|
||
" elif \"speaker_2\" in file_name:\n",
|
||
" print(f\" 🎤 Speaker 2 segment\")\n",
|
||
"\n",
|
||
"print(f\"\\nPodcast generation complete!\")\n",
|
||
"print(f\"Total segments: {podcast_script.total_lines}\")\n",
|
||
"print(f\"Output directory: {output_dir}\")\n",
|
||
"\n",
|
||
"# The complete podcast is ready for playback or distribution\n",
|
||
"complete_podcast = next(f for f in audio_files if \"complete_podcast\" in f)\n",
|
||
"print(f\"Complete podcast: {complete_podcast}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "13635eb1",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Text-to-Speech Features:\n",
|
||
"- **Multi-speaker support**: Distinct voices for Speaker 1 and Speaker 2\n",
|
||
"- **Natural speech**: Kokoro TTS for high-quality, natural-sounding audio\n",
|
||
"- **Segment generation**: Individual files for each dialogue segment\n",
|
||
"- **Audio combining**: Automatic creation of complete podcast file\n",
|
||
"- **Professional quality**: 24kHz sample rate for clear audio"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "0d97377c",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "fb2c977b",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Complete Workflow Integration\n",
|
||
"\n",
|
||
"Here's how all components work together in the main application:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "9985aa3c",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "861ca260",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Summary\n",
|
||
"\n",
|
||
"This NotebookLM implementation provides a complete pipeline from raw content to interactive AI conversations and podcast generation:\n",
|
||
"\n",
|
||
"1. **Multi-modal processing**: Documents, audio, video, and web content\n",
|
||
"2. **Semantic search**: Vector embeddings for intelligent retrieval\n",
|
||
"3. **Cited responses**: Transparent source attribution with interactive citations\n",
|
||
"4. **Memory integration**: Contextual conversations with history\n",
|
||
"5. **Podcast generation**: AI-powered script creation and text-to-speech\n",
|
||
"\n",
|
||
"The system is designed to be modular, scalable, and production-ready, with each component handling specific aspects of the content-to-conversation pipeline."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"id": "7131fc4d",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": []
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": ".venv",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.11.11"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|