wehub/googlecloudplatform--generative-ai
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:30:30 +08:00
commit 914fea506e
2793 changed files with 802106 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
gemini/use-cases/document-processing/doc_parsing_with_llamaindex_and_llamaparse.ipynb
+861
View File
@@ -0,0 +1,861 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "b2c3c554daa1"
},
"source": [
"# Comparing LlamaIndex and LlamaParse for Dense Document Questioning Answering on Vertex AI\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/doc_parsing_with_llamaindex_and_llamaparse.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fuse-cases%2Fdocument-processing%2Fdoc_parsing_with_llamaindex_and_llamaparse.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/use-cases/document-processing/doc_parsing_with_llamaindex_and_llamaparse.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/doc_parsing_with_llamaindex_and_llamaparse.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/doc_parsing_with_llamaindex_and_llamaparse.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/doc_parsing_with_llamaindex_and_llamaparse.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/doc_parsing_with_llamaindex_and_llamaparse.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/doc_parsing_with_llamaindex_and_llamaparse.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/doc_parsing_with_llamaindex_and_llamaparse.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a> "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1bc291e87bd1"
},
"source": [
"| | |\n",
"|-|-|\n",
"| Author(s) | [Noa Ben-Efraim](https://github.com/noabenefraim/) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "648701a7bf44"
},
"source": [
"## Overview\n",
"There are many ways to customize RAG pipelines by choosing how to ingest, parse, chunk, and retrieve your data. This notebook focuses on comparing different document parsing capabilities offered by LlamaIndex.\n",
"\n",
"We will compare document parsing with LlamaIndex and LlamaParse on a 10-Q financial document, which is heavily populated with complex tables.\n",
"\n",
"### Objectives\n",
"This notebook compare using LlamaIndex and LlamaParse for ingesting and indexing a complex document. \n",
"\n",
"You will complete the following tasks:\n",
"1. Ingest and parse document using LlamaIndex SimpleDataReader, LlamaIndex LangchainNodeParser, and LlamaParse Parser using Gemini models.\n",
"2. Index your parsed document in a VectorStore.\n",
"3. Create a a query agent for each parsing technique that can answer questions against the input document.\n",
"4. Compare results across LlamaIndex and LlamaParse.\n",
"\n",
"### LlamaIndex\n",
"LlamaIndex is a foundational data framework for building LLM applications. A few of its main capabilities are:\n",
"\n",
"+ Data Ingestion: Loads your data from various sources (documents, databases, APIs).   \n",
"+ Indexing: Structures your data into efficient formats for LLM retrieval (e.g., vector stores, tree structures).   \n",
"+ Querying: Enables you to ask questions or give instructions to the LLM, referencing your indexed data for answers.   \n",
"+ Integration: Connects with various LLMs, vector databases, and other tools.   \n",
"  \n",
"\n",
"### LlamaParse\n",
"LlamaParse is a tool within the LlamaIndex ecosystem, focused on parsing complex documents:\n",
"\n",
"+ PDFs: Handles PDFs with tables, charts, and other embedded elements that can be challenging for standard parsing.  \n",
"+ Semi-structured Data: Extracts structured information from documents that aren't fully formatted databases.   \n",
"+ Enhanced Retrieval: Works seamlessly with LlamaIndex to improve retrieval accuracy for complex documents."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "87008e44295a"
},
"source": [
"## Getting Started\n",
"\n",
"### Authenticate your notebook environment\n",
"\n",
"This notebook expects the following resources to exists:\n",
"+ Initialized Google Cloud project \n",
"+ Vertex AI API enabled\n",
"+ GCS Bucket and Vertex AI Search Index and Endpoint\n",
"+ A LlamaParse API Key [request a key here](https://docs.cloud.llamaindex.ai/llamacloud/getting_started/api_key)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "516b8d774169"
},
"outputs": [],
"source": [
"PROJECT_ID = \"\" # @param {type:\"string\"}\n",
"REGION = \"\" # @param {type:\"string\"}\n",
"GCS_BUCKET = \"\" # @param {type:\"string\"}\n",
"VS_INDEX_NAME = \"\" # @param {type:\"string\"}\n",
"VS_INDEX_ENDPOINT_NAME = \"\" # @param {type:\"string\"}\n",
"DATA_FOLDER = \"./data\" # @param {type:\"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "205f199d8a53"
},
"source": [
"### Set Google Cloud project information and initialize Vertex AI SDK\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "dd11490e3764"
},
"outputs": [],
"source": [
"PROJECT_ID = \"\" # @param {type:\"string\"}\n",
"LOCATION = \"\" # @param {type:\"string\"}\n",
"\n",
"import vertexai\n",
"\n",
"vertexai.init(project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f818f7127a12"
},
"source": [
"### Setting up the Environment\n",
"Install dependencies"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "c76df03a88f3"
},
"outputs": [],
"source": [
"%pip install google-cloud-aiplatform \\\n",
" llama-index \\\n",
" langchain-community \\\n",
" llama-index-embeddings-vertex \\\n",
" llama-index-llms-vertex \\\n",
" termcolor \\\n",
" llama-index-core -q"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "44d71c554004"
},
"source": [
"Set up imports"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"id": "fb7e65420d45"
},
"outputs": [],
"source": [
"from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
"from llama_index.core import Settings, SimpleDirectoryReader, VectorStoreIndex\n",
"from llama_index.core.base.response.schema import Response\n",
"from llama_index.core.extractors import KeywordExtractor, QuestionsAnsweredExtractor\n",
"from llama_index.core.ingestion import IngestionPipeline\n",
"from llama_index.core.node_parser import LangchainNodeParser\n",
"from llama_index.embeddings.vertex import VertexTextEmbedding\n",
"from llama_index.llms.vertex import Vertex\n",
"from llama_index.vector_stores.vertexaivectorsearch import VertexAIVectorStore\n",
"from llama_parse import LlamaParse\n",
"from termcolor import colored"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8d7a288c44bb"
},
"source": [
"Generate credentials"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"id": "5e5cb3d58df5"
},
"outputs": [],
"source": [
"import google.auth\n",
"import google.auth.transport.requests\n",
"\n",
"# credentials will now have an api token\n",
"credentials = google.auth.default(quota_project_id=\"genai-noabe\")[0]\n",
"request = google.auth.transport.requests.Request()\n",
"credentials.refresh(request)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0e9455150d49"
},
"outputs": [],
"source": [
"embedding_model = VertexTextEmbedding(\"text-embedding-005\", credentials=credentials)\n",
"llm = Vertex(model=\"gemini-2.5-flash\", temperature=0.0, max_tokens=5000)\n",
"\n",
"Settings.embed_model = embedding_model\n",
"Settings.llm = llm"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f5822037c828"
},
"source": [
"Set up LlamaIndex settings to point to Gemini models."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "a6c832d2780c"
},
"outputs": [],
"source": [
"import IPython\n",
"\n",
"app = IPython.Application.instance()\n",
"app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8daf430888fc"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"id": "885a3c84ddac"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e9a64d5bf6c3"
},
"source": [
"### Download sample data\n",
"\n",
"For the remainder of the notebook we will examine Alphabet Inc. 10Q document. A 10Q is a financial document that is dense with tables with financial figures. This document is a great candidate to to investigate document parsing capabilities."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e0c896dfda70"
},
"outputs": [],
"source": [
"!mkdir {DATA_FOLDER}\n",
"!wget \"https://abc.xyz/assets/ae/e9/753110054014b6de4d620a2853f6/goog-10-q-q2-2024.pdf\" -P {DATA_FOLDER}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ece6a448d7e0"
},
"source": [
"## Document Parsing with LlamaIndex\n",
"\n",
"This section will ingest and parse the 10Q using LlamaIndex tools, specifically focusing on SimpleDirectoryReader and LangChainNodeParser.\n",
"\n",
"### Option 1: `SimpleDirectoryReader`\n",
"The SimpleDirectoryReader is the core data ingestion tool in LlamaIndex. It's designed to load data from a variety of sources and convert it into a format suitable for further processing and indexing by LlamaIndex."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7b343b4be5d9"
},
"outputs": [],
"source": [
"reader = SimpleDirectoryReader(\"./data\")\n",
"documents = reader.load_data(show_progress=True)\n",
"print(documents[0])"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "f5e3def62a59"
},
"outputs": [],
"source": [
"# Index the parsed document\n",
"simpledirectory_index = VectorStoreIndex.from_documents(documents)\n",
"\n",
"# Generate a query engine based on the SimpleDataReader\n",
"simple_query_engine = simpledirectory_index.as_query_engine(similarity_top_k=2)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ec35c0de8639"
},
"source": [
"### Option 2: `LangChainNodeParser` with LlamaIndex\n",
"The LangChainNodeParser is a part of LlamaIndex. It is a specialized parser designed to extract structured information from text documents using the power of LangChain.\n",
"\n",
"Key Features:\n",
"+ LangChain Integration: Leverages LangChain's powerful language models and tools to parse text.\n",
"+ Node-Based Output: Converts unstructured text into a structured format based on a defined schema, represented as a hierarchy of nodes. This enables more sophisticated querying and analysis of the extracted information.\n",
"+ Customization: Supports defining custom parsing schemas to match the structure of your specific documents.\n",
"+ Flexibility: Can be used in combination with other LlamaIndex components, such as the SimpleDataReader, to process and index the extracted structured data."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"id": "c94da573044a"
},
"outputs": [],
"source": [
"parser = LangchainNodeParser(RecursiveCharacterTextSplitter())\n",
"langchain_nodes = parser.get_nodes_from_documents(documents)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e8e2f0b882ce"
},
"outputs": [],
"source": [
"# An example node that was generated using the LangChainNodeParser and the associated metadata\n",
"langchain_nodes[0]"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"id": "a88552f52124"
},
"outputs": [],
"source": [
"# Index the document based on the LangChain nodes generated above\n",
"langchainparser_index = VectorStoreIndex(nodes=langchain_nodes)\n",
"\n",
"# Create a query engine based off the LangChainNodeParser\n",
"lg_query_engine = langchainparser_index.as_query_engine(similarity_top_k=2)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "adc97e363bac"
},
"source": [
"## LlamaParse\n",
"\n",
"LlamaParse Parser is a powerful tool for extracting structured data from unstructured or semi-structured text, offering flexibility, customization, and seamless integration within the LlamaIndex framework.It can take an unstructured or semi-structured text document and, using a defined schema, extract structured information from it. This structured output is represented as a nested hierarchy of nodes, facilitating further processing and analysis.\n",
"\n",
"A few key features include:\n",
"\n",
"+ JSON Schema: Leverages the standardized JSON Schema format for more complex schemas.\n",
"+ Prompt Templates: Allows you to craft custom prompts to guide the language model's parsing behavior, offering greater control and adaptability.\n",
"+ LLM Selection: You have the flexibility to choose the specific LLM you want to use for parsing, enabling you to tailor the performance to your specific needs and budget.\n",
"+ Node-Based Output:\n",
" + Structured Representation: The parsed output is organized into a hierarchy of nodes, each representing a piece of extracted information.\n",
" + Nested Structure: Nodes can contain other nodes, allowing for the representation of complex relationships and nested data structures within the document.\n",
" + Metadata: Nodes can also include additional metadata, such as confidence scores or source information, enriching the extracted data.\n",
"+ Integration with LlamaIndex: The structured output from parser() seamlessly integrates with other LlamaIndex components, such as indexing and querying, facilitating efficient retrieval and analysis of the extracted information.\n",
"\n",
"#### Define a Parser\n",
"\n",
"Here we will define a LlamaParse() parser with specific parsing instructions, and ingest the data."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"id": "85ec700afd80"
},
"outputs": [],
"source": [
"parser = LlamaParse(\n",
" parsing_instruction=\"You are a financial analyst working specifically with 10Q documents. Not all pages have titles. Try to reconstruct the dialogue spoken in a cohesive way.\",\n",
" api_key=\"\",\n",
" result_type=\"text\", # \"markdown\" and \"text\" are available\n",
" language=\"en\",\n",
" invalidate_cache=True,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "23fd5739fa74"
},
"source": [
"### Option 1 - LlamaParse with SimpleDirectoryReader\n",
"\n",
"This is the apples to apples comparison with LlamaIndex. We are using the SimpleDirectoryReader with the LlamaParse file extractor, and then loading the data directly from documents to a Vector Store for retrieval."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "688898e8b334"
},
"outputs": [],
"source": [
"import nest_asyncio\n",
"\n",
"nest_asyncio.apply()\n",
"file_extractor = {\".pdf\": parser}\n",
"documents = SimpleDirectoryReader(\n",
" input_files=[\"./data/goog-10-q-q2-2024.pdf\"], file_extractor=file_extractor # type: ignore\n",
").load_data()"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"id": "d215261053fb"
},
"outputs": [],
"source": [
"lp_simple = VectorStoreIndex.from_documents(documents)\n",
"lp_simple_engine = lp_simple.as_query_engine(similarity_top_k=2)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "906057ddf157"
},
"source": [
"### Option 2 - LlamaParse and Vertex AI Vector Search\n",
"\n",
"This approach is a more customized approach by defining the Vector Search mechanism through Vertex AI and extracting metadata that will be embedded and stored in the search index. \n",
"\n",
"Using metadata in Retrieval Augmented Generation (RAG) improves accuracy and context by focusing searches and providing additional information. This leads to efficient filtering, ranking, and personalized responses tailored to user needs and history. Metadata also facilitates handling complex multi-criteria queries, making RAG systems more versatile and effective.\n",
"\n",
"The following section will:\n",
"+ Parse the documents using LlamaParse\n",
"+ Extract metadata from documents returned from LlamaParse\n",
"+ Create metadata embeddings attached to each document\n",
"+ Create index in Vertex AI Vector Store\n",
"+ Query against the Vector Store"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0ab41643425c"
},
"source": [
"#### Parse data using LlamaParse"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "d4d5d47ad69e"
},
"outputs": [],
"source": [
"documents = parser.load_data(\"./data/goog-10-q-q2-2024.pdf\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e5bde318580a"
},
"source": [
"#### Create Metadata from Nodes\n",
"\n",
"Using extractors we will generate meta-data for each node. The metadata is generated using Gemini-Pro and focuses on what questions can this text answer and what key words are meaningful in this section. Each metadata piece will be embedded with Gemini text-embedding model. \n",
"\n",
"Creating metadata can be useful for another lookup criteria during RAG based search."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"id": "ea18fa35d496"
},
"outputs": [],
"source": [
"extractors = [\n",
" QuestionsAnsweredExtractor(questions=3, llm=llm),\n",
" KeywordExtractor(keywords=10, llm=llm),\n",
"]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "fd57de02eb9b"
},
"outputs": [],
"source": [
"# Run metadata transformation pipeline.\n",
"pipeline = IngestionPipeline(\n",
" transformations=extractors, # type: ignore\n",
")\n",
"nodes = await pipeline.arun(documents=documents, in_place=False)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8929ab07f054"
},
"source": [
"Example metadata that was generated:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "b41541f14b77"
},
"outputs": [],
"source": [
"print(nodes[1].metadata)"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"id": "476642ff9305"
},
"outputs": [],
"source": [
"# Generate embeddings for each metadata node\n",
"for node in nodes:\n",
" node_embedding = embedding_model.get_text_embedding(\n",
" node.get_content(metadata_mode=\"all\")\n",
" )\n",
" node.embedding = node_embedding"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6e52150147f8"
},
"source": [
"#### Load Nodes into Predefined Vector Store\n",
"\n",
"This following section required a preexisting Vertex AI Vector Store. Vector stores contain embedding vectors of ingested document chunks.\n",
"\n",
"For information to create a vector store, refer to this link https://docs.llamaindex.ai/en/stable/examples/vector_stores/VertexAIVectorSearchDemo/"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "f8cdc3678a71"
},
"outputs": [],
"source": [
"vector_store = VertexAIVectorStore(\n",
" project_id=PROJECT_ID,\n",
" region=REGION,\n",
" index_id=\"\", # Add in your Vertex AI Vector Search Index ID\n",
" endpoint_id=\"\", # Add in your Vertex AI Vector Search Deployed Index ID\n",
" gcs_bucket_name=GCS_BUCKET,\n",
")\n",
"\n",
"# Only need to run once\n",
"vector_store.add(nodes)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d22ef6b3a7e1"
},
"source": [
"#### Create a search index and search and query the Vector Store"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"id": "c135cf0ee925"
},
"outputs": [],
"source": [
"lp_index = VectorStoreIndex.from_vector_store(vector_store)\n",
"lp_query_engine = lp_index.as_query_engine(similarity_top_k=2)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7064bdd2c434"
},
"source": [
"## Query Comparison between LlamaIndex and LlamaParse\n",
"Below are queries that responses can be found in the 10Q document within complex tables. Let's see how each approach compares."
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"id": "bef0f00ab246"
},
"outputs": [],
"source": [
"queries = [\n",
" \"What are the total cash, cash equivalents, and marketable securities as of Dec 23 2023\",\n",
" \"Total investments with fair value change reflected in other comprehensive income as of Dec 23 2023\",\n",
" \"What is the corporate debt securities unrealized loss as of Dec 31 2023 for 12 months or greater?\",\n",
" \"What is the coupon rate for total outstanding debt\",\n",
" \"Provide the table of share repurchases\",\n",
"]"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"id": "a0c410e45aea"
},
"outputs": [],
"source": [
"def print_output(response: Response):\n",
" print(\"Response:\")\n",
" print(\"-\" * 80)\n",
" print(colored(response.response, color=\"red\"))\n",
" print(\"-\" * 80)\n",
" print(\"Source Documents:\")\n",
" print(\"-\" * 80)\n",
" for source in response.source_nodes:\n",
" print(f\"Sample Text: {source.text[:100]}\")\n",
" print(f\"Relevance score: {source.get_score():.3f}\")\n",
" print(f\"File Name: {source.metadata.get('file_name')}\")\n",
" print(f\"Page #: {source.metadata.get('page_label')}\")\n",
" print(f\"File Path: {source.metadata.get('file_path')}\")\n",
" print(\"-\" * 80)\n",
"\n",
"\n",
"def run_query(query_idx: int):\n",
" query = queries[query_idx]\n",
" print(\"Query: \" + query)\n",
" print(colored(\"LlamaIndex SimpleDirectoryReader response....\\n\", color=\"blue\"))\n",
" print_output(simple_query_engine.query(query))\n",
"\n",
" print(\n",
" colored(\n",
" \"LlamaIndex LangChainNodeParser on LlamaIndex response....\\n\", color=\"blue\"\n",
" )\n",
" )\n",
" print_output(lg_query_engine.query(query))\n",
"\n",
" print(colored(\"LlamaParse Simple response....\\n\", color=\"blue\"))\n",
" print_output(lp_simple_engine.query(query))\n",
"\n",
" print(colored(\"LlamaParse on Vertex AI response....\\n\", color=\"blue\"))\n",
" print_output(lp_query_engine.query(query))\n",
" print(\"###################################################\\n\\n\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "3a90e4c451cf"
},
"outputs": [],
"source": [
"run_query(query_idx=0)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8095a2092eab"
},
"outputs": [],
"source": [
"run_query(query_idx=1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "fe4cac0c796c"
},
"outputs": [],
"source": [
"run_query(query_idx=2)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e4deb4eeaa0d"
},
"outputs": [],
"source": [
"run_query(query_idx=3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "c837ac4f41f5"
},
"outputs": [],
"source": [
"run_query(query_idx=4)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ceb7a1bbb11e"
},
"source": [
"## Observations\n",
"\n",
"### Answer Key\n",
"| Query | Answer | Citation page |\n",
"|------------------------------------------------------------------------------------------------------|------------------|---------------|\n",
"| \"What are the total cash, cash equivalents, and marketable securities as of Dec 23 2023\" | $110,916 million | 5 |\n",
"| \"Total investments with fair value change reflected in other comprehensive income as of Dec 23 2023\" | $78,917 million | 13 |\n",
"| \"What is the corporate debt securities unrealized loss as of Dec 31 2023 for 12 months or greater? | 592 million | 15 |\n",
"| \"What is the coupon rate for total outstanding debt\" | 0.45-2.25% | 22 |\n",
"| \"Provide the table of share repurchases\" | Table | 27 or 49 |\n",
"\n",
"### Generated Answers\n",
"| Document Parsing Technique | Query 1 | Query 2 | Query 3 | Query 4 | Query 5 |\n",
"|------------------------------------------|---------|---------|---------|---------|---------|\n",
"| LlamaIndex - SimpleDirectoryReader | (✓) | (✓) | (✓) | (✓) | (✓) |\n",
"| LlamaIndex - LangChainNodeParser | (✓) | (✓) | (✓) | (✓) | (✓) |\n",
"| LlamaParse - SimpleDirectoryReader | (✓) | (✓) | (✓)| (✓) | (✓) |\n",
"| LlamaParse - Vertex AI Vector Search | (✓) | (✓) | (✓)| (✓) | (✓) |\n",
"\n",
"## Conclusion\n",
"\n",
"There are many ways to customize your data ingestion and retrieval pipelines for custom RAG applications. This notebook was an overview to a handful of options that work in combination with Google Gemini models. "
]
}
],
"metadata": {
"colab": {
"name": "doc_parsing_with_llamaindex_and_llamaparse.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
gemini/use-cases/document-processing/document_processing.ipynb
+1355
View File
File diff suppressed because it is too large Load Diff
gemini/use-cases/document-processing/patents_understanding.ipynb
+906
View File
@@ -0,0 +1,906 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2024 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Patents Document Understanding with Gemini\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/patents_understanding.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fuse-cases%2Fdocument-processing%2Fpatents_understanding.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/use-cases/document-processing/patents_understanding.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/bigquery/import?url=https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/patents_understanding.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/bigquery/v1/32px.svg\" alt=\"BigQuery Studio logo\"><br> Open in BigQuery Studio\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/patents_understanding.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/patents_understanding.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/patents_understanding.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/patents_understanding.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/patents_understanding.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/patents_understanding.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| Author |\n",
"| --- |\n",
"| [Holt Skinner](https://github.com/holtskinner) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"Back in 2019, I wrote a [Google Cloud Blog post](https://cloud.google.com/blog/products/ai-machine-learning/building-a-document-understanding-pipeline-with-google-cloud) in collaboration with [Michael Munn](https://github.com/munnm) and [Michael Sherman](https://github.com/michaelwsherman) which illustrates how to build a Document Understanding Pipeline using [AutoML](https://cloud.google.com/automl).\n",
"\n",
"This example showed how to train custom machine learning models for the following tasks:\n",
"\n",
"- [Image Classification](https://cloud.google.com/vision/automl/docs/beginners-guide)\n",
"- [Entity Extraction](https://cloud.google.com/natural-language/automl/entity-analysis/docs/)\n",
"- [Text Classification](https://cloud.google.com/natural-language/automl/docs/predict)\n",
"- [Object Detection](https://cloud.google.com/vision/automl/object-detection/docs/)\n",
"\n",
"In today's world of Generative AI models like [Gemini](https://blog.google/technology/ai/google-gemini-ai/), it's possible to create the same document processing pipeline without training custom models. This significantly simplifies the process and reduces the time & resources required to automate these workflows.\n",
"\n",
"In this notebook, we'll create a document understanding pipeline on a public dataset of [patents PDFs](https://console.cloud.google.com/marketplace/details/global-patents/labeled-patents) stored in BigQuery and use [Batch Prediction for Gemini 3 in Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/batch-prediction-gemini) to:\n",
"\n",
"- Classify the patent granter (US or EU).\n",
"- Classify the invention type (Medical Tech, Computer Vision, Cryptography, Other).\n",
"- Extract key entities like publication date, application number, etc.\n",
"- Detect bounding boxes for figures in the document."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Google Gen AI SDK for Python\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-genai google-cloud-bigquery pandas-gbq"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "R5Xep4W9lq-Z"
},
"source": [
"### Restart runtime\n",
"\n",
"To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel.\n",
"\n",
"The restart might take a minute or longer. After it's restarted, continue to the next step."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XRvKdaPDTznN"
},
"outputs": [],
"source": [
"import IPython\n",
"\n",
"app = IPython.Application.instance()\n",
"app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SbmM4z7FOBpM"
},
"source": [
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ The kernel is going to restart. In Colab or Colab Enterprise, you might see an error message that says \"Your session crashed for an unknown reason.\" This is expected. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "C8aMIcn9mEWt"
},
"source": [
"### Import libraries\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information and create client\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"import pandas_gbq\n",
"from google import genai\n",
"from google.cloud import bigquery\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"\n",
"client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)\n",
"bq_client = bigquery.Client()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e43229f3ad4f"
},
"source": [
"### Load the Gemini 3 Flash model\n",
"\n",
"To learn more about all [Gemini models on Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-models)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cf93d5f0ce00"
},
"outputs": [],
"source": [
"MODEL_ID = \"gemini-3.5-flash\" # @param {type: \"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EdvJRUWRNGHE"
},
"source": [
"## The Prediction Pipeline\n",
"\n",
"Now, let's build the pipeline for processing the patent documents. We will:\n",
"\n",
"1. Fetch the PDF URIs from BigQuery into a Pandas DataFrame.\n",
"2. Create a detailed prompt for Gemini to extract the information.\n",
"3. Process each PDF with the prompt and save the results in a Pandas DataFrame\n",
"4. Save the structured output to a new BigQuery table"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aa191bc13022"
},
"source": [
"### Get PDF URIs from BigQuery\n",
"\n",
"We'll query BigQuery to get a list of Google Cloud Storage URIs for the patent PDF files.\n",
"\n",
"NOTE: This query limits to only 5 documents to save on processing time for this tutorial."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "mzVYcJ5CYgZ3"
},
"outputs": [],
"source": [
"query = \"\"\"\n",
"SELECT\n",
" COALESCE(t1.gcs_path, t2.gcs_path, t3.gcs_path) AS gcs_path\n",
"FROM\n",
" `bigquery-public-data.labeled_patents.extracted_data` AS t1\n",
"FULL OUTER JOIN\n",
" `bigquery-public-data.labeled_patents.figures` AS t2\n",
"ON\n",
" t1.gcs_path = t2.gcs_path\n",
"FULL OUTER JOIN\n",
" `bigquery-public-data.labeled_patents.invention_types` AS t3\n",
"ON\n",
" COALESCE(t1.gcs_path, t2.gcs_path) = t3.gcs_path\n",
"LIMIT 5\n",
"\"\"\"\n",
"\n",
"df = bq_client.query(query).result().to_dataframe()\n",
"df"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7ea36f19d9f7"
},
"source": [
"### Define the Gemini prompt\n",
"\n",
"Here's the prompt we'll use with Gemini to extract the information we need. It's a detailed instruction that specifies the output format as JSON and also includes a JSON Schema."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "jYxX_rCGe0hC"
},
"outputs": [],
"source": [
"PATENTS_PROMPT = \"\"\"\n",
"Given a patent document, please extract the following information and output it as a JSON object with the specified keys.\n",
"\n",
"**JSON Output Format:**\n",
"\n",
"```json\n",
"{\n",
" \"granter_classification\": \"United States or European Union\",\n",
" \"invention_type\": \"med_tech/computer_vision/crypto/other\",\n",
" \"entities\": {\n",
" \"issuer\": \"string\",\n",
" \"language\": \"string\",\n",
" \"publication_date\": \"YYYY-MM-DD\",\n",
" \"class_international\": [\"string\", \"string\", ...],\n",
" \"class_us\": [\"string\", \"string\", ...],\n",
" \"application_number\": \"string\",\n",
" \"filing_date\": \"YYYY-MM-DD\",\n",
" \"priority_date_eu\": \"YYYY-MM-DD\",\n",
" \"representative_eu\": [\"string\",\"string\",...],\n",
" \"applicant\": [\"string\", \"string\", ...],\n",
" \"inventor\": [\"string\", \"string\", ...],\n",
" \"title\": \"string\",\n",
" \"patent_number\": \"string\"\n",
" },\n",
" \"image_detections\": [\n",
" {\n",
" \"label\": \"figure_1\",\n",
" \"bounding_box\": [x1, y1, x2, y2]\n",
" },\n",
" {\n",
" \"label\": \"figure_2\",\n",
" \"bounding_box\": [x1, y1, x2, y2]\n",
" }\n",
" ]\n",
"}\n",
"```\n",
"\n",
"**Detailed Instructions:**\n",
"\n",
"1. **Granter Classification:** Determine if the patent was granted by the \"United States\" or \"European Union\". Populate the `granter_classification` field accordingly.\n",
"\n",
"2. **Invention Type Classification:** Classify the type of invention described in the patent as one of the following and populate the `invention_type` field with the corresponding code:\n",
" * Medical Tech: `med_tech`\n",
" * Computer Vision: `computer_vision`\n",
" * Cryptography: `crypto`\n",
" * Other: `other`\n",
"\n",
"3. **Entity Extraction:**\n",
" * Extract the following entities and populate the corresponding fields within the `entities` object.\n",
" * If multiple entities exist for a field (e.g., multiple inventors, EU representatives, multiple US and international classes), represent them as a JSON array of strings.\n",
" * Use the `YYYY-MM-DD` format for dates.\n",
" * If a field is not present in the document, leave the value as null or an empty array as appropriate\n",
"\n",
"4. **Image Object Detection:**\n",
" * Detect the bounding boxes of all image figures in the patent document.\n",
" * Represent each bounding box as a JSON array with the format `[x1, y1, x2, y2]` where:\n",
" * `x1` and `y1` are the coordinates of the top-left corner of the bounding box.\n",
" * `x2` and `y2` are the coordinates of the bottom-right corner of the bounding box.\n",
" * Include a label (e.g., \"figure_1\", \"figure_2\", etc.) for each detected bounding box.\n",
" * Populate the `image_detections` array with each detected bounding box. If no figures are detected, the image_detections array should be empty.\n",
"\n",
"**Input:**\n",
"\n",
"Provide the complete patent document text as input to this prompt.\n",
"\n",
"**Output:**\n",
"\n",
"The output should be a single JSON object following the specified format, containing all the extracted information.\n",
"\n",
"**Example (Illustrative, Not Complete):**\n",
"\n",
"```json\n",
"{\n",
" \"granter_classification\": \"United States\",\n",
" \"invention_type\": \"med_tech\",\n",
" \"entities\": {\n",
" \"issuer\": \"United States Patent and Trademark Office\",\n",
" \"language\": \"en\",\n",
" \"publication_date\": \"2023-10-26\",\n",
" \"class_international\": [\"A61K 31/4709\", \"A61K 9/00\"],\n",
" \"class_us\": [\"514/250\"],\n",
" \"application_number\": \"17/744,843\",\n",
" \"filing_date\": \"2022-05-24\",\n",
" \"priority_date_eu\": null,\n",
" \"representative_eu\": [],\n",
" \"applicant\": [\"ABC Pharmacy Inc\"],\n",
" \"inventor\": [\"John Smith\", \"Jane Doe\"],\n",
" \"title\": \"Novel Formulation\",\n",
" \"patent_number\": \"US12345678\"\n",
" },\n",
" \"image_detections\": [\n",
" {\n",
" \"label\": \"figure_1\",\n",
" \"bounding_box\": [100, 50, 300, 250]\n",
" },\n",
" {\n",
" \"label\": \"figure_2\",\n",
" \"bounding_box\": [400, 100, 600, 300]\n",
" }\n",
" ]\n",
"}\n",
"```\n",
"\n",
"\"\"\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "01048a7ef243"
},
"source": [
"### Create BigQuery Dataset and Table for Batch Prediction\n",
"\n",
"NOTE: The location of the BigQuery dataset must be the same as the location for Vertex AI. (e.g. `us-central1`, not `us`)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "d-KVqc67ajwf"
},
"outputs": [],
"source": [
"DATASET_NAME = \"patents_data_batch\"\n",
"TABLE_NAME = \"patents_table\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e6de4b4be476"
},
"outputs": [],
"source": [
"!bq --location={LOCATION} mk --dataset \"{PROJECT_ID}:{DATASET_NAME}\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c403e052d071"
},
"source": [
"Load `GenerateContentRequest` JSON into BigQuery table.\n",
"\n",
"This request takes a Google Cloud Storage path to a PDF, processes it with the prompt above, and returns a structured dictionary using [Controlled Generation](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/control-generated-output)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "bbba7e8861af"
},
"outputs": [],
"source": [
"import json\n",
"\n",
"\n",
"def create_request_json(row) -> str:\n",
" return json.dumps(\n",
" {\n",
" \"contents\": [\n",
" {\n",
" \"role\": \"user\",\n",
" \"parts\": [\n",
" {\"text\": PATENTS_PROMPT},\n",
" {\n",
" \"fileData\": {\n",
" \"fileUri\": row[\"gcs_path\"],\n",
" \"mimeType\": \"application/pdf\",\n",
" }\n",
" },\n",
" ],\n",
" }\n",
" ],\n",
" \"systemInstruction\": {\n",
" \"parts\": [{\"text\": \"You are an expert at analyzing patent documents.\"}]\n",
" },\n",
" \"generationConfig\": {\n",
" \"responseMimeType\": \"application/json\",\n",
" \"responseSchema\": {\n",
" \"type\": \"OBJECT\",\n",
" \"properties\": {\n",
" \"granter_classification\": {\n",
" \"type\": \"STRING\",\n",
" \"enum\": [\"United States\", \"European Union\"],\n",
" },\n",
" \"invention_type\": {\n",
" \"type\": \"STRING\",\n",
" \"enum\": [\"med_tech\", \"computer_vision\", \"crypto\", \"other\"],\n",
" },\n",
" \"entities\": {\n",
" \"type\": \"OBJECT\",\n",
" \"properties\": {\n",
" \"issuer\": {\"type\": \"STRING\"},\n",
" \"language\": {\"type\": \"STRING\"},\n",
" \"publication_date\": {\n",
" \"type\": \"STRING\",\n",
" \"format\": \"date\",\n",
" },\n",
" \"class_international\": {\n",
" \"type\": \"ARRAY\",\n",
" \"items\": {\"type\": \"STRING\"},\n",
" },\n",
" \"class_us\": {\n",
" \"type\": \"ARRAY\",\n",
" \"items\": {\"type\": \"STRING\"},\n",
" },\n",
" \"application_number\": {\"type\": \"STRING\"},\n",
" \"filing_date\": {\"type\": \"STRING\", \"format\": \"date\"},\n",
" \"priority_date_eu\": {\n",
" \"type\": \"STRING\",\n",
" \"format\": \"date\",\n",
" \"nullable\": True,\n",
" },\n",
" \"representative_eu\": {\n",
" \"type\": \"ARRAY\",\n",
" \"items\": {\"type\": \"STRING\"},\n",
" },\n",
" \"applicant\": {\n",
" \"type\": \"ARRAY\",\n",
" \"items\": {\"type\": \"STRING\"},\n",
" },\n",
" \"inventor\": {\n",
" \"type\": \"ARRAY\",\n",
" \"items\": {\"type\": \"STRING\"},\n",
" },\n",
" \"title\": {\"type\": \"STRING\"},\n",
" \"patent_number\": {\"type\": \"STRING\"},\n",
" },\n",
" \"required\": [\n",
" \"issuer\",\n",
" \"language\",\n",
" \"publication_date\",\n",
" \"class_international\",\n",
" \"class_us\",\n",
" \"application_number\",\n",
" \"filing_date\",\n",
" \"applicant\",\n",
" \"inventor\",\n",
" \"title\",\n",
" \"patent_number\",\n",
" ],\n",
" },\n",
" \"image_detections\": {\n",
" \"type\": \"ARRAY\",\n",
" \"items\": {\n",
" \"type\": \"OBJECT\",\n",
" \"properties\": {\n",
" \"label\": {\"type\": \"STRING\"},\n",
" \"bounding_box\": {\n",
" \"type\": \"ARRAY\",\n",
" \"items\": {\"type\": \"NUMBER\"},\n",
" \"minItems\": 4,\n",
" \"maxItems\": 4,\n",
" },\n",
" },\n",
" \"required\": [\"label\", \"bounding_box\"],\n",
" },\n",
" },\n",
" },\n",
" \"required\": [\n",
" \"granter_classification\",\n",
" \"invention_type\",\n",
" \"entities\",\n",
" \"image_detections\",\n",
" ],\n",
" },\n",
" },\n",
" }\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "29afb3b3fdcb"
},
"outputs": [],
"source": [
"df[\"request\"] = df.apply(create_request_json, axis=1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "271aeda2bdaa"
},
"outputs": [],
"source": [
"pandas_gbq.to_gbq(df, f\"{DATASET_NAME}.{TABLE_NAME}\", project_id=PROJECT_ID)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XKpdIpbw6Ycv"
},
"source": [
"### Batch Process Patent Documents with Gemini\n",
"\n",
"Batch Processing in Vertex AI will take in the BigQuery table with the requests and return the output in a new BigQuery table in the same dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "c069acc57089"
},
"outputs": [],
"source": [
"batch_job = client.batches.create(\n",
" model=MODEL_ID,\n",
" src=f\"bq://{PROJECT_ID}.{DATASET_NAME}.{TABLE_NAME}\",\n",
")\n",
"batch_job"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e67c90f1117e"
},
"source": [
"Run the following block to get the latest status of the batch job."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cf230deeb013"
},
"outputs": [],
"source": [
"batch_job = client.batches.get(name=batch_job.name)\n",
"batch_job"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bf44c2b0bd92"
},
"source": [
"Run the following block to continuously poll the status of the batch job until it completes."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "defb5eb0b1c1"
},
"outputs": [],
"source": [
"import time\n",
"\n",
"# Refresh the job until complete\n",
"while batch_job.state == \"JOB_STATE_RUNNING\":\n",
" batch_job = client.batches.get(name=batch_job.name)\n",
" # print(batch_job)\n",
" time.sleep(5)\n",
"\n",
"# Check if the job succeeds\n",
"if batch_job.state == \"JOB_STATE_SUCCEEDED\":\n",
" print(\"Job succeeded!\")\n",
"else:\n",
" print(f\"Job failed: {batch_job.error}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "acd872565dc0"
},
"source": [
"Load the results from the destination BigQuery table."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9ef9a49490eb"
},
"outputs": [],
"source": [
"results_df = pandas_gbq.read_gbq(\n",
" batch_job.dest.bigquery_uri.replace(\"bq://\", \"\"), project_id=PROJECT_ID\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ed26cf628fc3"
},
"source": [
"Extract the fields from the controlled generation response."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "5ac2c0a3d98c"
},
"outputs": [],
"source": [
"import pandas as pd\n",
"\n",
"\n",
"def flatten_response(response) -> dict | None:\n",
" try:\n",
" parsed_json = json.loads(\n",
" response[\"candidates\"][0][\"content\"][\"parts\"][0][\"text\"]\n",
" )\n",
"\n",
" entities = parsed_json.get(\"entities\", {})\n",
" additional_fields = {\n",
" \"granter_classification\": parsed_json.get(\"granter_classification\"),\n",
" \"image_detections\": parsed_json.get(\"image_detections\"),\n",
" \"invention_type\": parsed_json.get(\"invention_type\"),\n",
" }\n",
"\n",
" return {**entities, **additional_fields}\n",
" except (KeyError, IndexError, json.JSONDecodeError) as e:\n",
" print(f\"Error processing response: {e}\")\n",
" return None\n",
"\n",
"\n",
"results_df = results_df.join(\n",
" pd.json_normalize(results_df[\"response\"].apply(flatten_response))\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "da7bda40a26e"
},
"outputs": [],
"source": [
"results_df"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UkIB1pek6_Y7"
},
"source": [
"### Compare Results with Ground Truth\n",
"\n",
"We can pull the ground truth data from BigQuery into a DataFrame to compare the results. Note that the data is in a slightly different format, so a direct comparison is not straightforward."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "MzDxgh74mKVy"
},
"outputs": [],
"source": [
"query = \"\"\"\n",
"SELECT\n",
" COALESCE(t1.gcs_path, t2.gcs_path, t3.gcs_path) AS gcs_path,\n",
" t1.issuer,\n",
" t1.language,\n",
" t1.publication_date,\n",
" t1.class_international,\n",
" t1.class_us,\n",
" t1.application_number,\n",
" t1.filing_date,\n",
" t1.priority_date_eu,\n",
" t1.representative_line_1_eu,\n",
" t1.applicant_line_1,\n",
" t1.inventor_line_1,\n",
" t1.title_line_1,\n",
" t1.number,\n",
" t2.x_relative_min,\n",
" t2.y_relative_min,\n",
" t2.x_relative_max,\n",
" t2.y_relative_max,\n",
" t3.invention_type\n",
"FROM\n",
" `bigquery-public-data.labeled_patents.extracted_data` AS t1\n",
"FULL OUTER JOIN\n",
" `bigquery-public-data.labeled_patents.figures` AS t2\n",
"ON\n",
" t1.gcs_path = t2.gcs_path\n",
"FULL OUTER JOIN\n",
" `bigquery-public-data.labeled_patents.invention_types` AS t3\n",
"ON\n",
" COALESCE(t1.gcs_path, t2.gcs_path) = t3.gcs_path;\n",
"\"\"\"\n",
"\n",
"query_job = bq_client.query(query)\n",
"ground_truth_df = bq_client.query(query).result().to_dataframe()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2a4e033321ad"
},
"source": [
"## Conclusion\n",
"\n",
"This notebook demonstrates the capabilities of using Gemini to extract structured information from complex documents, simplifying document understanding pipelines without needing to train custom models.\n",
"\n",
"**Key Improvements:**\n",
"\n",
"- **Comprehensive Extraction**: Unlike AutoML, which often limited field extraction to the first line (e.g., applicant, inventor, class US), Gemini accurately extracts full text and all listed values for these fields.\n",
"- **Simplified Workflow**: AutoML required four separate models and four requests per document to complete the tasks. Gemini consolidates this into a single request.\n",
"- **No Custom Model Training**: Gemini is a pre-trained model, eliminating the need for use case-specific training, saving both time and resources."
]
}
],
"metadata": {
"colab": {
"name": "patents_understanding.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
gemini/use-cases/document-processing/sheet_music.ipynb
+517
View File
@@ -0,0 +1,517 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ijGzTHJJUCPY"
},
"outputs": [],
"source": [
"# Copyright 2024 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VEqbX8OhE8y9"
},
"source": [
"# Sheet Music Analysis with Gemini\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/sheet_music.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fuse-cases%2Fdocument-processing%2Fsheet_music.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Run in Colab Enterprise\n",
" </a>\n",
" </td> \n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/sheet_music.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/use-cases/document-processing/sheet_music.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://goo.gle/4jjsLV9\">\n",
" <img width=\"32px\" src=\"https://cdn.qwiklabs.com/assets/gcp_cloud-e3a77215f0b8bfa9b3f611c0d2208c7e8708ed31.svg\" alt=\"Google Cloud logo\"><br> Open in Cloud Skills Boost\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/sheet_music.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/sheet_music.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/sheet_music.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/sheet_music.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/sheet_music.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a> \n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5ad877ea09dd"
},
"source": [
"| Author |\n",
"| --- |\n",
"| [Holt Skinner](https://github.com/holtskinner) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CkHPv2myT2cx"
},
"source": [
"## Overview\n",
"\n",
"[Sheet Music](https://en.wikipedia.org/wiki/Sheet_music) is the primary form of music notation used by composers and performers across the world. These pages contain information about the lyrics, pitches, rhythms, composer, text author, composition date, among others.\n",
"\n",
"This notebook illustrates using Gemini to extract this metadata from sheet music PDFs.\n",
"\n",
"These prompts and documents were demonstrated in the [Google Cloud Next 2024 session \"What's next with Gemini: Driving business impact with multimodal use cases\"](https://www.youtube.com/watch?v=DqH1R9Pk5RI).\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "r11Gu7qNgx1p"
},
"source": [
"## Getting Started\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Google Gen AI SDK for Python"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade -q google-genai pypdf"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you are running this notebook on Google Colab, run the following cell to authenticate your environment. This step is not required if you are using [Vertex AI Workbench](https://cloud.google.com/vertex-ai-workbench).\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"# Additional authentication is required for Google Colab\n",
"if \"google.colab\" in sys.modules:\n",
" # Authenticate user to Google Cloud\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information and create client\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"# Use the environment variable if the user doesn't provide Project ID.\n",
"import os\n",
"\n",
"from google import genai\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"global\")\n",
"\n",
"client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jXHfaVS66_01"
},
"source": [
"### Import libraries\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "lslYAvw37JGQ"
},
"outputs": [],
"source": [
"import json\n",
"\n",
"import pypdf\n",
"from IPython.display import Markdown, display\n",
"from google.genai.types import (\n",
" GenerateContentConfig,\n",
" GoogleSearch,\n",
" HarmBlockThreshold,\n",
" HarmCategory,\n",
" Part,\n",
" SafetySetting,\n",
" Tool,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FTMywdzUORIA"
},
"source": [
"### Load the Gemini model\n",
"\n",
"Gemini is a multimodal model that supports multimodal prompts. You can include text, image(s), PDFs, audio, and video in your prompt requests."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7b76801cc9e4"
},
"outputs": [],
"source": [
"MODEL_ID = \"gemini-3.5-flash\" # @param {type: \"string\"}\n",
"\n",
"config = GenerateContentConfig(\n",
" safety_settings=[\n",
" SafetySetting(\n",
" category=HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT,\n",
" threshold=HarmBlockThreshold.BLOCK_ONLY_HIGH,\n",
" )\n",
" ],\n",
" system_instruction=\"You are an expert in musicology and music history.\",\n",
" tools=[Tool(google_search=GoogleSearch())],\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Wy75sLb-yjNn"
},
"source": [
"## Extract Structured Metadata from Sheet Music PDF\n",
"\n",
"For this example, we will be using the popular classical music book [24 Italian Songs and Arias of the 17th and 18th Centuries](https://imslp.org/wiki/24_Italian_Songs_and_Arias_of_the_17th_and_18th_Centuries_(Various)), and extracting metadata about each song in the book."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0ed417af1e5c"
},
"outputs": [],
"source": [
"sheet_music_pdf_uri = \"gs://github-repo/use-cases/sheet-music/24ItalianSongs.pdf\"\n",
"\n",
"sheet_music_extraction_prompt = \"\"\"The following document is a book of sheet music. Your task is to output structured metadata about every piece of music in the document.\n",
"\n",
"Correct any mistakes that are in the document and fill in missing information when not present in the document.\n",
"\n",
"Include the following details:\n",
"\n",
"Title\n",
"Composer with lifetime\n",
"Tempo Marking\n",
"A description of the piece\n",
"\"\"\"\n",
"\n",
"# Send to Gemini\n",
"response = client.models.generate_content(\n",
" model=MODEL_ID,\n",
" contents=[\n",
" sheet_music_extraction_prompt,\n",
" # Load file directly from Google Cloud Storage\n",
" Part.from_uri(\n",
" file_uri=sheet_music_pdf_uri,\n",
" mime_type=\"application/pdf\",\n",
" ),\n",
" ],\n",
" config=config,\n",
")\n",
"\n",
"# Display results\n",
"display(Markdown(response.text))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "29b40b5e0cb0"
},
"source": [
"You can see that Gemini extracted all of the relevant fields from the document."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "db3765c7645d"
},
"source": [
"### Song Identification with Audio\n",
"\n",
"Now, let's try something more challenging, identifying a song being performed based on the sheet music. We have an audio clip of Holt Skinner performing one of the songs in the book, and we will ask Gemini to identify it based on the sheet music PDF."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "61ea3f2f1c4a"
},
"outputs": [],
"source": [
"song_identification_prompt = \"\"\"Based on the sheet music PDF, what song is in the audio clip? Explain how you made the decision.\"\"\"\n",
"\n",
"# Load PDF file\n",
"pdf_part = Part.from_uri(\n",
" file_uri=sheet_music_pdf_uri,\n",
" mime_type=\"application/pdf\",\n",
")\n",
"\n",
"audio_part = Part.from_uri(\n",
" file_uri=\"gs://github-repo/use-cases/sheet-music/24ItalianClip.mp3\",\n",
" mime_type=\"audio/mpeg\",\n",
")\n",
"\n",
"# Send to Gemini\n",
"response = client.models.generate_content(\n",
" model=MODEL_ID,\n",
" contents=[pdf_part, audio_part, song_identification_prompt],\n",
" config=config,\n",
")\n",
"\n",
"# Display results\n",
"display(Markdown(response.text))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9730e8a5628b"
},
"source": [
"### Edit PDF Metadata\n",
"\n",
"Next, we'll use the output from Gemini to edit the metadata of a PDF containing one song, which can make it easier to organize this file in sheet music applications.\n",
"\n",
"We'll adjust the prompt slightly and set the [`response_mime_type`](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/gemini#:~:text=in%20the%20list.-,responseMimeType,-(Preview)) to get the response in JSON format."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "97e2a06cc762"
},
"outputs": [],
"source": [
"sheet_music_pdf_uri = \"gs://github-repo/use-cases/sheet-music/SebbenCrudele.pdf\"\n",
"output_file_name = \"SebbenCrudele.pdf\"\n",
"\n",
"sheet_music_extraction_prompt = \"\"\"The following document is a piece of sheet music. Your task is to output structured metadata about the piece of music in the document. Correct any mistakes that are in the document and fill in missing information when not present in the document.\n",
"\n",
"Output the data in the following JSON format:\n",
"\n",
"{\n",
" \"/Title\": \"Title of the piece\",\n",
" \"/Author\": \"Composer(s) of the piece\",\n",
" \"/Subject\": \"Music Genre(s) in a comma separated list\",\n",
"}\n",
"\n",
"\"\"\"\n",
"\n",
"# Load file directly from Google Cloud Storage\n",
"file_part = Part.from_uri(\n",
" file_uri=sheet_music_pdf_uri,\n",
" mime_type=\"application/pdf\",\n",
")\n",
"\n",
"config.response_mime_type = \"application/json\"\n",
"\n",
"# Send to Gemini\n",
"response = client.models.generate_content(\n",
" model=MODEL_ID,\n",
" contents=[sheet_music_extraction_prompt, file_part],\n",
" config=config,\n",
")\n",
"\n",
"# Display results\n",
"display(Markdown(response.text))\n",
"\n",
"new_metadata = json.loads(response.text)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8e997cb6affc"
},
"source": [
"Next, we'll download the PDF from the GCS Bucket and edit the metadata using the [`PyPDF2`](https://pypdf2.readthedocs.io/en/3.x/) library."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "879f827c537a"
},
"outputs": [],
"source": [
"! gcloud storage cp {sheet_music_pdf_uri} {output_file_name}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e81759999d78"
},
"outputs": [],
"source": [
"def edit_pdf_metadata(file_path: str, new_metadata: dict) -> None:\n",
" \"\"\"Edits metadata of a PDF file.\n",
"\n",
" Args:\n",
" file_path (str): Path to the PDF file.\n",
" new_metadata (dict): Dictionary containing the metadata fields (e.g., {'/Author': 'John Doe', '/Title': 'My Report'})\n",
" \"\"\"\n",
" # 1. Read the existing PDF\n",
" reader = pypdf.PdfReader(file_path)\n",
" writer = pypdf.PdfWriter()\n",
"\n",
" # 2. Copy all pages and structure (bookmarks, etc.) from reader to writer\n",
" writer.append(reader)\n",
"\n",
" # 3. Add/Update metadata\n",
" # Note: Keys usually start with a slash, e.g., \"/Title\"\n",
" writer.add_metadata(new_metadata)\n",
"\n",
" # 4. Save the result\n",
" # We open the file in write-binary mode to save the changes\n",
" with open(file_path, \"wb\") as f:\n",
" writer.write(f)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "jtbQo-df1TTu"
},
"outputs": [],
"source": [
"# View current metadata\n",
"reader = pypdf.PdfReader(output_file_name)\n",
"print(reader.metadata)\n",
"\n",
"edit_pdf_metadata(output_file_name, new_metadata)\n",
"\n",
"# View updated metadata\n",
"reader = pypdf.PdfReader(output_file_name)\n",
"print(reader.metadata)"
]
}
],
"metadata": {
"colab": {
"name": "sheet_music.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
gemini/use-cases/document-processing/summarization_large_documents_langchain.ipynb
+929
View File
@@ -0,0 +1,929 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "uxCkB_DXTHzf"
},
"outputs": [],
"source": [
"# Copyright 2023 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Hny4I-ODTIS6"
},
"source": [
"# Text Summarization of Large Documents using LangChain 🦜🔗\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/summarization_large_documents_langchain.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fuse-cases%2Fdocument-processing%2Fsummarization_large_documents_langchain.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/use-cases/document-processing/summarization_large_documents_langchain.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/summarization_large_documents_langchain.ipynb\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/summarization_large_documents_langchain.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/summarization_large_documents_langchain.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/summarization_large_documents_langchain.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/summarization_large_documents_langchain.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/summarization_large_documents_langchain.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8e71fd522e56"
},
"source": [
"| Authors |\n",
"| --- |\n",
"| [Polong Lin](https://github.com/polong-lin) |\n",
"| [Holt Skinner](https://github.com/holtskinner) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-nLS57E2TO5y"
},
"source": [
"## Overview\n",
"\n",
"Text summarization is an NLP task that creates a concise and informative summary of a longer text. LLMs can be used to create summaries of news articles, research papers, technical documents, and other types of text.\n",
"\n",
"Summarizing large documents can be challenging. To create summaries, you need to apply summarization strategies to your indexed documents. You have already seen some of these strategies in the previous notebooks. If you haven't completed it, it is recommended to do so to have a basic understanding of how to summarize large documents.\n",
"\n",
"In this notebook, you will use LangChain, a framework for developing LLM applications, to apply some summarization strategies. The notebook covers several examples of how to summarize large documents."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iXsvgIuwTPZw"
},
"source": [
"### Objective\n",
"\n",
"In this tutorial, you learn how to use LangChain with Gemini to summarize large documents by working through the following examples:\n",
"\n",
"- Stuffing method\n",
"- MapReduce method\n",
"- Refine method"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "skXAu__iqks_"
},
"source": [
"### Costs\n",
"\n",
"This tutorial uses billable components of Google Cloud:\n",
"\n",
"- Vertex AI\n",
"\n",
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mvKl-BtQTRiQ"
},
"source": [
"## Getting Started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PwFMpIMrTV_4"
},
"source": [
"### Install Vertex AI SDK & Other dependencies"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8aP6JVlZkS-m"
},
"outputs": [],
"source": [
"!sudo apt -y -qq install tesseract-ocr\n",
"!sudo apt -y -qq install libtesseract-dev\n",
"!sudo apt-get -y -qq install poppler-utils #required by PyPDF2 for page count and other pdf utilities\n",
"!sudo apt-get -y -qq install python-dev libxml2-dev libxslt1-dev antiword unrtf poppler-utils pstotext tesseract-ocr flac ffmpeg lame libmad0 libsox-fmt-mp3 sox libjpeg-dev swig"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "oDmNq5__Trl4"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet pytesseract pypdf PyPDF2 langchain langchain-core langchain-google-vertexai google-cloud-aiplatform langchain-community"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jWwtjLV5TY6H"
},
"source": [
"**Colab only**: Run the following cell to restart the kernel. For Vertex AI Workbench you can restart the terminal using the button on top."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XRvKdaPDTznN"
},
"outputs": [],
"source": [
"# Automatically restart kernel after installs so that your environment can access the new packages\n",
"import IPython\n",
"\n",
"app = IPython.Application.instance()\n",
"app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "opUxT_k5TdgP"
},
"source": [
"### Authenticating your notebook environment\n",
"\n",
"- If you are using **Colab** to run this notebook, run the cell below and continue.\n",
"- If you are using **Vertex AI Workbench**, check out the setup instructions [here](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/setup-env)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vbNgv4q1T2Mi"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "n5fXfvzhTkYN"
},
"source": [
"### Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "GjSsu6cmUdEx"
},
"outputs": [],
"source": [
"# Use the environment variable if the user doesn't provide Project ID.\n",
"import os\n",
"\n",
"# fmt: off\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"# fmt: on\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"REGION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"global\")\n",
"\n",
"import vertexai\n",
"\n",
"vertexai.init(project=PROJECT_ID, location=REGION)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cRkcfnQMT9vD"
},
"outputs": [],
"source": [
"import urllib\n",
"import warnings\n",
"from pathlib import Path as p\n",
"\n",
"import pandas as pd\n",
"from langchain import PromptTemplate\n",
"from langchain.chains.summarize import load_summarize_chain\n",
"from langchain.document_loaders import PyPDFLoader\n",
"from langchain_google_vertexai import VertexAI\n",
"\n",
"warnings.filterwarnings(\"ignore\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DAGaTjPVTmhP"
},
"source": [
"### Import models\n",
"\n",
"You load the pre-trained text generation model Gemini 3 Flash."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ITUmZiNZcMUW"
},
"outputs": [],
"source": [
"vertex_llm_text = VertexAI(model_name=\"gemini-3.5-flash\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cKG-ZTJ_02wq"
},
"source": [
"## Summarization with Large Documents"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HZkLDRTjTcfm"
},
"source": [
"### Preparing data files\n",
"\n",
"To begin, you will need to download a few files that are required for the summarizing tasks below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7H0zINHpTaSu"
},
"outputs": [],
"source": [
"data_folder = p.cwd() / \"data\"\n",
"p(data_folder).mkdir(parents=True, exist_ok=True)\n",
"\n",
"pdf_url = \"https://services.google.com/fh/files/misc/practitioners_guide_to_mlops_whitepaper.pdf\"\n",
"pdf_file = str(p(data_folder, pdf_url.rsplit(\"/\", maxsplit=1)[-1]))\n",
"\n",
"urllib.request.urlretrieve(pdf_url, pdf_file)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JELITHdBhnf0"
},
"source": [
"### Extract text from the PDF\n",
"\n",
"You use an `PdfReader` to extract the text from our scanned documents."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "x3INtovxreI_"
},
"outputs": [],
"source": [
"pdf_loader = PyPDFLoader(pdf_file)\n",
"pages = pdf_loader.load_and_split()\n",
"print(pages[3].page_content)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZDVwBFSjZ7ws"
},
"source": [
"## Method 1: Stuffing\n",
"\n",
"Stuffing is the simplest method to pass data to a language model. It \"stuffs\" text into the prompt as context in a way that all of the relevant information can be processed by the model to get what you want.\n",
"\n",
"In LangChain, you can use `StuffDocumentsChain` as part of the `load_summarize_chain` method. What you need to do is setting `stuff` as `chain_type` of your chain."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uhEi-XqKnv2v"
},
"source": [
"### Prompt design with `Stuffing` chain"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "B-ljajUen1YO"
},
"outputs": [],
"source": [
"prompt_template = \"\"\"Write a concise summary of the following text delimited by triple backquotes.\n",
" Return your response in bullet points which covers the key points of the text.\n",
" ```{text}```\n",
" BULLET POINT SUMMARY:\n",
" \"\"\"\n",
"\n",
"prompt = PromptTemplate(template=prompt_template, input_variables=[\"text\"])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "N5aVrDWkJs3Y"
},
"source": [
"### Retrying\n",
"Initiate a chain using `stuff` method and process three pages document."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "N_hoizIgObe9"
},
"outputs": [],
"source": [
"stuff_chain = load_summarize_chain(vertex_llm_text, chain_type=\"stuff\", prompt=prompt)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Q1_zwxwgTnlV"
},
"outputs": [],
"source": [
"three_pages = pages[:3]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0jEUfOn7UFI2"
},
"outputs": [],
"source": [
"three_pages"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "QnXUwWxkrLu4"
},
"outputs": [],
"source": [
"try:\n",
" print(stuff_chain.run(three_pages))\n",
"except Exception as e:\n",
" print(\n",
" \"The code failed since it won't be able to run inference on such a huge context and throws this exception: \",\n",
" e,\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xKb_fBEedZqu"
},
"source": [
"As you can see, with the `stuff` method, you can summarize the entire document content with a single API call passing all data at once.\n",
"\n",
"Depending on the context length of LLM, the `stuff` method would not work as it result in a prompt larger than the context length."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hqZrKM32h-o2"
},
"source": [
"### Considerations\n",
"\n",
"The `stuffing` method is a way to summarize text by feeding the entire document to a large language model (LLM) in a single call. This method has both pros and cons.\n",
"\n",
"The stuffing method only requires a single call to the LLM, which can be faster than other methods that require multiple calls. When summarizing text, the LLM has access to all the data at once, which can result in a better summary.\n",
"\n",
"But, LLMs have a context length, which is the maximum number of tokens that can be processed in a single call. If the document is longer than the context length, the stuffing method will not work. Also the stuffing method is not suitable for summarizing large documents, as it can be slow and may not produce a good summary.\n",
"\n",
"Let's explore other approaches to help deal with having longer text than context length limit of LLMs."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RM3V1JARZ9-k"
},
"source": [
"## Method 2: MapReduce\n",
"\n",
"The `MapReduce` method implements a multi-stage summarization. It is a technique for summarizing large pieces of text by first summarizing smaller chunks of text and then combining those summaries into a single summary.\n",
"\n",
"In LangChain, you can use `MapReduceDocumentsChain` as part of the `load_summarize_chain` method. What you need to do is setting `map_reduce` as `chain_type` of your chain."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lagLXEamlPY2"
},
"source": [
"### Prompt design with `MapReduce` chain\n",
"\n",
"In our example, you have a 32-page document that you need to summarize.\n",
"\n",
"With LangChain, the `map_reduce` chain breaks the document down into 1024 token chunks max. Then it runs the initial prompt you define on each chunk to generate a summary of that chunk. In the example below, you use the following first stage or map prompt.\n",
"\n",
"```Write a concise summary of the following text delimited by triple backquotes. Return your response in bullet points which covers the key points of the text.\n",
"'''{text}'''. BULLET POINT SUMMARY:```\n",
"\n",
"Once summaries for all of the chunks are generated, it runs a different prompt to combine those summaries into a single summary. In the example below, you use the following second stage or combine prompt.\n",
"\n",
"```Write a summary of the entire document that includes the main points from all of the individual summaries.```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "R6oHEtdSmsTn"
},
"outputs": [],
"source": [
"map_prompt_template = \"\"\"\n",
" Write a summary of this chunk of text that includes the main points and any important details.\n",
" {text}\n",
" \"\"\"\n",
"\n",
"map_prompt = PromptTemplate(template=map_prompt_template, input_variables=[\"text\"])\n",
"\n",
"combine_prompt_template = \"\"\"\n",
" Write a concise summary of the following text delimited by triple backquotes.\n",
" Return your response in bullet points which covers the key points of the text.\n",
" ```{text}```\n",
" BULLET POINT SUMMARY:\n",
" \"\"\"\n",
"\n",
"combine_prompt = PromptTemplate(\n",
" template=combine_prompt_template, input_variables=[\"text\"]\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JXoz0uLDMoWD"
},
"source": [
"### Generate summaries using MapReduce method\n",
"\n",
"After defining prompts, you initialize the associated `map_reduce_chain`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VRGJcBZeVdEa"
},
"outputs": [],
"source": [
"map_reduce_chain = load_summarize_chain(\n",
" vertex_llm_text,\n",
" chain_type=\"map_reduce\",\n",
" map_prompt=map_prompt,\n",
" combine_prompt=combine_prompt,\n",
" return_intermediate_steps=True,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-6fekDDr0hrJ"
},
"source": [
"Then, you generate summaries using the chain. Notice that LangChain use a tokenizer (from transformer library) with 1024 token limit by default."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "uSC6w2TBV35q"
},
"outputs": [],
"source": [
"map_reduce_outputs = map_reduce_chain({\"input_documents\": pages})"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "meH2ELuz2H46"
},
"source": [
"After summaries are generated, you can validate them by organize input documents and associated output in a Pandas DataFrame."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "r6FRSR7xRLew"
},
"outputs": [],
"source": [
"final_mp_data = []\n",
"for doc, out in zip(\n",
" map_reduce_outputs[\"input_documents\"],\n",
" map_reduce_outputs[\"intermediate_steps\"],\n",
" strict=False,\n",
"):\n",
" output = {}\n",
" output[\"file_name\"] = p(doc.metadata[\"source\"]).stem\n",
" output[\"file_type\"] = p(doc.metadata[\"source\"]).suffix\n",
" output[\"page_number\"] = doc.metadata[\"page\"]\n",
" output[\"chunks\"] = doc.page_content\n",
" output[\"concise_summary\"] = out\n",
" final_mp_data.append(output)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "dA9cnh8YaNbF"
},
"outputs": [],
"source": [
"pdf_mp_summary = pd.DataFrame.from_dict(final_mp_data)\n",
"pdf_mp_summary = pdf_mp_summary.sort_values(\n",
" by=[\"file_name\", \"page_number\"]\n",
") # sorting the dataframe by filename and page_number\n",
"pdf_mp_summary.reset_index(inplace=True, drop=True)\n",
"pdf_mp_summary.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "yA0eM1K3cvH2"
},
"outputs": [],
"source": [
"index = 3\n",
"print(\"[Context]\")\n",
"print(pdf_mp_summary[\"chunks\"].iloc[index])\n",
"print(\"\\n\\n [Simple Summary]\")\n",
"print(pdf_mp_summary[\"concise_summary\"].iloc[index])\n",
"print(\"\\n\\n [Page number]\")\n",
"print(pdf_mp_summary[\"page_number\"].iloc[index])\n",
"print(\"\\n\\n [Source: file_name]\")\n",
"print(pdf_mp_summary[\"file_name\"].iloc[index])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ROrE1-HKpg7y"
},
"source": [
"### Considerations\n",
"\n",
"With `MapReduce` method, the model is able to summarize a large paper by overcoming the context limit of `Stuffing` method with parallel processing.\n",
"\n",
"However, the `MapReduce` requires multiple calls to the model and potentially losing context between pages.\n",
"\n",
"To deal this challenge, you can try another method to summarize multiple pages at a time."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lxdB-5PqgCf-"
},
"source": [
"## Method 3: Refine\n",
"\n",
"The Refine method is an alternative method to deal with large document summarization. It works by first running an initial prompt on a small chunk of data, generating some output. Then, for each subsequent document, the output from the previous document is passed in along with the new document, and the LLM is asked to refine the output based on the new document.\n",
"\n",
"In LangChain, you can use `RefineDocumentsChain` as part of the `load_summarize_chain` method. What you need to do is setting `refine` as `chain_type` of your chain."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pjj2UZilDF4Q"
},
"source": [
"### Prompt design with `Refine` chain\n",
"\n",
"With LangChain, the `refine` chain requires two prompts.\n",
"\n",
"The question prompt to generate the output for subsequent task. The refine prompt to refine the output based on the generated content.\n",
"\n",
"In this example, the question prompt is:\n",
"\n",
"```\n",
"Please provide a summary of the following text.\n",
"TEXT: {text}\n",
"SUMMARY:\n",
"```\n",
"\n",
"and the refine prompt is:\n",
"\n",
"```\n",
"Write a concise summary of the following text delimited by triple backquotes.\n",
"Return your response in bullet points which covers the key points of the text.\n",
"```{text}```\n",
"BULLET POINT SUMMARY:\n",
"```\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XiZX45Z5VTwS"
},
"outputs": [],
"source": [
"question_prompt_template = \"\"\"\n",
" Please provide a summary of the following text.\n",
" TEXT: {text}\n",
" SUMMARY:\n",
" \"\"\"\n",
"\n",
"question_prompt = PromptTemplate(\n",
" template=question_prompt_template, input_variables=[\"text\"]\n",
")\n",
"\n",
"refine_prompt_template = \"\"\"\n",
" Write a concise summary of the following text delimited by triple backquotes.\n",
" Return your response in bullet points which covers the key points of the text.\n",
" ```{text}```\n",
" BULLET POINT SUMMARY:\n",
" \"\"\"\n",
"\n",
"refine_prompt = PromptTemplate(\n",
" template=refine_prompt_template, input_variables=[\"text\"]\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-USlaSPbM0rs"
},
"source": [
"### Generate summaries using Refine method\n",
"\n",
"After you define prompts, you initiate a summarization chain using `refine` chain type."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_-Sv3HO1U3hi"
},
"outputs": [],
"source": [
"refine_chain = load_summarize_chain(\n",
" vertex_llm_text,\n",
" chain_type=\"refine\",\n",
" question_prompt=question_prompt,\n",
" refine_prompt=refine_prompt,\n",
" return_intermediate_steps=True,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f9EZCDK-MQJH"
},
"source": [
"Then, you use the summarization chain to summarize document using Refine method."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "KHwwab7vXNa1"
},
"outputs": [],
"source": [
"refine_outputs = refine_chain({\"input_documents\": pages})"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "eUqpki5EMYEr"
},
"source": [
"Below you can see the resulting summaries."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7j5cUGStZ5WF"
},
"outputs": [],
"source": [
"final_refine_data = []\n",
"for doc, out in zip(\n",
" refine_outputs[\"input_documents\"],\n",
" refine_outputs[\"intermediate_steps\"],\n",
" strict=False,\n",
"):\n",
" output = {}\n",
" output[\"file_name\"] = p(doc.metadata[\"source\"]).stem\n",
" output[\"file_type\"] = p(doc.metadata[\"source\"]).suffix\n",
" output[\"page_number\"] = doc.metadata[\"page\"]\n",
" output[\"chunks\"] = doc.page_content\n",
" output[\"concise_summary\"] = out\n",
" final_refine_data.append(output)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "N_7Mm9cEmGOV"
},
"outputs": [],
"source": [
"pdf_refine_summary = pd.DataFrame.from_dict(final_refine_data)\n",
"pdf_refine_summary = pdf_mp_summary.sort_values(\n",
" by=[\"file_name\", \"page_number\"]\n",
") # sorting the dataframe by filename and page_number\n",
"pdf_refine_summary.reset_index(inplace=True, drop=True)\n",
"pdf_refine_summary.head()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "jvLVCs8Gbwbw"
},
"outputs": [],
"source": [
"index = 3\n",
"print(\"[Context]\")\n",
"print(pdf_refine_summary[\"chunks\"].iloc[index])\n",
"print(\"\\n\\n [Simple Summary]\")\n",
"print(pdf_refine_summary[\"concise_summary\"].iloc[index])\n",
"print(\"\\n\\n [Page number]\")\n",
"print(pdf_refine_summary[\"page_number\"].iloc[index])\n",
"print(\"\\n\\n [Source: file_name]\")\n",
"print(pdf_refine_summary[\"file_name\"].iloc[index])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7dwgbRTrM5Cb"
},
"source": [
"### Considerations"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1H0Y5pPcXbgm"
},
"source": [
"In short, the Refine method for text summarization with LLMs can pull in more relevant context and may be less lossy than Map Reduce. However, it requires many more calls to the LLM than Stuffing, and these calls are not independent, meaning they cannot be parallelized. Additionally, there is some potential dependency on the ordering of the documents. Latest documents they might become more relevant as this method suffers from recency bias."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HAaWXncPMhv4"
},
"source": [
"## Conclusion\n",
"\n",
"\n",
"In this notebook you learn about different techniques to summarize long documents with LangChain and PaLM API. What you have seen in this notebook are only some of the possibilities you have. For example, there is another method called the Map-Rerank method which involves running an initial prompt on each chunk of data, which not only tries to complete a task but also gives a score for how certain it is in its answer. The responses are then ranked according to this score, and the highest score is returned.\n",
"\n",
"With that being said, it is important to highlight that depending on your needs you may consider to use pure Foundational model with a custom framework to build generative ai application.\n",
"\n",
"Here are some of the benefits of using a foundational model with a custom framework:\n",
"\n",
" - More flexibility to implement your application with different LLMs, prompting templates, document handling strategies and more.\n",
"\n",
" - More control to customize your generative applications based on your scenario.\n",
"\n",
" - Better performance to improve latency and scalability of your application.\n"
]
}
],
"metadata": {
"colab": {
"name": "summarization_large_documents_langchain.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
gemini/use-cases/document-processing/tax_automation.ipynb
+679
View File
@@ -0,0 +1,679 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Automating Income Taxes with Gemini\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/tax_automation.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fuse-cases%2Fdocument-processing%2Ftax_automation.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/use-cases/document-processing/tax_automation.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/bigquery/import?url=https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/tax_automation.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/bigquery/v1/32px.svg\" alt=\"BigQuery Studio logo\"><br> Open in BigQuery Studio\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/tax_automation.ipynb\">\n",
" <img width=\"32px\" src=\"https://upload.wikimedia.org/wikipedia/commons/9/91/Octicons-mark-github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/tax_automation.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/tax_automation.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/tax_automation.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/53/X_logo_2023_original.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/tax_automation.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/tax_automation.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| Author |\n",
"| --- |\n",
"| [Holt Skinner](https://github.com/holtskinner) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"Back in 2022, I wrote a [Google Cloud Blog post](https://cloud.google.com/blog/topics/developers-practitioners/automating-income-taxes-document-ai) about automating income tax preparation using [Document AI](https://cloud.google.com/document-ai/docs/overview).\n",
"\n",
"This demo used the [Lending processors](https://cloud.google.com/blog/products/ai-machine-learning/lending-docai-fast-tracks-the-home-loan-process) to extract data from W-2 and 1099 PDFs and calculate the total tax owed.\n",
"\n",
"In the world of Generative AI models like [Gemini](https://blog.google/technology/ai/google-gemini-ai/), it's possible to create the same document processing pipeline in a more efficient manner.\n",
"\n",
"In this notebook, we'll create a document understanding pipeline on some sample tax documents to:\n",
"\n",
"- Classify the type of document (W-2, 1099-DIV, 1099-INT, 1099-MISC, 1099-NEC)\n",
"- Extract key fields based on the document type.\n",
"\n",
"These are the sample documents we will use:\n",
"\n",
"- [2020 Form 1099-DIV](https://storage.googleapis.com/cloud-samples-data/documentai/LendingDocAI/1099-DIV%20Parser/2020%20Form%201099-DIV%20-%20Anastasia%20Hodges.pdf)\n",
"- [2020 Form 1099-INT](https://storage.googleapis.com/cloud-samples-data/documentai/LendingDocAI/1099-INT%20Parser/2020%20Form%201099-INT%20-%20Anastasia%20Hodges.pdf)\n",
"- [2020 Form W-2](https://storage.googleapis.com/cloud-samples-data/documentai/LendingDocAI/W2Parser/2020/2020%20Form%20W-2%20-%20Anastasia%20Hodges.pdf)\n",
"\n",
"> Disclaimer: This is **NOT** financial advice, for educational purposes only!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Google Gen AI SDK for Python\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet google-genai"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information and create client\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"from google import genai\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\"}\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"\n",
"client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "C8aMIcn9mEWt"
},
"source": [
"### Import libraries\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rrgrbhPmmEWt"
},
"outputs": [],
"source": [
"from enum import Enum\n",
"\n",
"import pandas as pd\n",
"from IPython.display import display\n",
"from google.genai.types import GenerateContentConfig, Part\n",
"from pydantic import BaseModel, Field\n",
"\n",
"pd.set_option(\"display.max_colwidth\", None)\n",
"PDF_MIME_TYPE = \"application/pdf\"\n",
"JSON_MIME_TYPE = \"application/json\"\n",
"ENUM_MIME_TYPE = \"text/x.enum\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e43229f3ad4f"
},
"source": [
"### Load the Gemini 3 Flash model\n",
"\n",
"To learn more about all [Gemini models on Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-models)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cf93d5f0ce00"
},
"outputs": [],
"source": [
"MODEL_ID = \"gemini-3.5-flash\" # @param {type: \"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "53d3d02d82b0"
},
"source": [
"Create a pandas DataFrame to contain the data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "43c76a9f1ea4"
},
"outputs": [],
"source": [
"tax_documents = pd.DataFrame(\n",
" {\n",
" \"file_uri\": [\n",
" \"https://storage.googleapis.com/cloud-samples-data/documentai/LendingDocAI/1099-DIV%20Parser/2020%20Form%201099-DIV%20-%20Anastasia%20Hodges.pdf\",\n",
" \"https://storage.googleapis.com/cloud-samples-data/documentai/LendingDocAI/1099-INT%20Parser/2020%20Form%201099-INT%20-%20Anastasia%20Hodges.pdf\",\n",
" \"https://storage.googleapis.com/cloud-samples-data/documentai/LendingDocAI/W2Parser/2020/2020%20Form%20W-2%20-%20Anastasia%20Hodges.pdf\",\n",
" ]\n",
" }\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EdvJRUWRNGHE"
},
"source": [
"## Classify Documents\n",
"\n",
"First, we need to classify each of our documents.\n",
"\n",
"We will create an `Enum` class including each type of document."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "955ffce857e3"
},
"outputs": [],
"source": [
"class DocumentType(Enum):\n",
" W_2 = \"W-2\"\n",
" _1099_DIV = \"1099-DIV\"\n",
" _1099_INT = \"1099-INT\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d3bd3714f764"
},
"source": [
"Next, we will send each document to the Gemini model with a classification prompt."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "649cb3dce660"
},
"outputs": [],
"source": [
"def classify_document(file_uri: str) -> Enum:\n",
" response = client.models.generate_content(\n",
" model=MODEL_ID,\n",
" contents=[\n",
" \"Classify the following document.\",\n",
" Part.from_uri(\n",
" file_uri=file_uri,\n",
" mime_type=PDF_MIME_TYPE,\n",
" ),\n",
" ],\n",
" config=GenerateContentConfig(\n",
" system_instruction=\"\"\"You are a document classification specialist. Given a document, your task is to find which category the document belongs to from the document categories provided in the schema.\"\"\",\n",
" temperature=0,\n",
" response_schema=DocumentType,\n",
" response_mime_type=ENUM_MIME_TYPE,\n",
" ),\n",
" )\n",
" return response.parsed\n",
"\n",
"\n",
"tax_documents[\"classification\"] = tax_documents[\"file_uri\"].apply(classify_document)\n",
"display(tax_documents)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "843b123600c7"
},
"source": [
"## Extract Data\n",
"\n",
"In order to extract the fields from each of these document types, we will need to create Pydantic classes containing the fields to extract for each type. Then we will create a mapping of the classification `Enum` to the Pydantic classes.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9c8d93b1ee61"
},
"source": [
"### Create Pydantic classes\n",
"\n",
"> Note: These Pydantic models were created using Gemini with the following prompt:\n",
"> \n",
"> `Create a Pydantic class from BaseModel to contain values to extract from a [Document Type]`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "fb3fd97752c4"
},
"outputs": [],
"source": [
"class FormW2(BaseModel):\n",
" \"\"\"Pydantic class to represent data extracted from a Form W-2 (Wage and Tax Statement).\"\"\"\n",
"\n",
" employee_ssn: str = Field(..., description=\"Employee's Social Security Number\")\n",
" employer_ein: str = Field(\n",
" ..., description=\"Employer's Employer Identification Number\"\n",
" )\n",
" control_number: str | None = Field(\n",
" None, description=\"Employer's Control Number (Optional)\"\n",
" )\n",
" wages_tips_other_compensation: float = Field(\n",
" ..., description=\"Total Wages, tips, and other compensation\"\n",
" )\n",
" federal_income_tax_withheld: float = Field(\n",
" ..., description=\"Federal income tax withheld from wages\"\n",
" )\n",
" social_security_wages: float = Field(..., description=\"Social Security wages\")\n",
" social_security_tax_withheld: float = Field(\n",
" ..., description=\"Social Security tax withheld\"\n",
" )\n",
" medicare_wages_and_tips: float = Field(..., description=\"Medicare wages and tips\")\n",
" medicare_tax_withheld: float = Field(..., description=\"Medicare tax withheld\")\n",
" dependent_care_benefits: float | None = Field(\n",
" None, description=\"Dependent care benefits (Box 10)\"\n",
" )\n",
" nonqualified_plans: float | None = Field(\n",
" None, description=\"Nonqualified plans (Box 11)\"\n",
" )\n",
" box_12a_code: str | None = Field(None, description=\"Code for amount in Box 12a\")\n",
" box_12a_amount: float | None = Field(None, description=\"Amount for Code in Box 12a\")\n",
" box_12b_code: str | None = Field(None, description=\"Code for amount in Box 12b\")\n",
" box_12b_amount: float | None = Field(None, description=\"Amount for Code in Box 12b\")\n",
" box_12c_code: str | None = Field(None, description=\"Code for amount in Box 12c\")\n",
" box_12c_amount: float | None = Field(None, description=\"Amount for Code in Box 12c\")\n",
" box_12d_code: str | None = Field(None, description=\"Code for amount in Box 12d\")\n",
" box_12d_amount: float | None = Field(None, description=\"Amount for Code in Box 12d\")\n",
" statutory_employee: bool = Field(\n",
" False, description=\"Indicates if Statutory Employee\"\n",
" )\n",
" retirement_plan: bool = Field(False, description=\"Indicates if Retirement Plan\")\n",
" third_party_sick_pay: float | None = Field(\n",
" None, description=\"Third-party sick pay (Box 13)\"\n",
" )\n",
" other: str | None = Field(None, description=\"Other (Box 14)\")\n",
"\n",
" employer_name: str = Field(..., description=\"Employer's Name\")\n",
" employer_address: str = Field(..., description=\"Employer's Address\")\n",
" employer_city: str = Field(..., description=\"Employer's City\")\n",
" employer_state: str = Field(..., description=\"Employer's State (abbreviation)\")\n",
" employer_zip: str = Field(..., description=\"Employer's Zip Code\")\n",
"\n",
" employee_name: str = Field(..., description=\"Employee's Name\")\n",
" employee_address: str = Field(..., description=\"Employee's Address\")\n",
" employee_city: str = Field(..., description=\"Employee's City\")\n",
" employee_state: str = Field(..., description=\"Employee's State (abbreviation)\")\n",
" employee_zip: str = Field(..., description=\"Employee's Zip Code\")\n",
"\n",
" state: str | None = Field(None, description=\"State (if applicable)\")\n",
" state_employer_id: str | None = Field(\n",
" None, description=\"State Employer ID (if applicable)\"\n",
" )\n",
" state_wages: float | None = Field(None, description=\"State Wages (if applicable)\")\n",
" state_income_tax: float | None = Field(\n",
" None, description=\"State Income Tax (if applicable)\"\n",
" )\n",
"\n",
"\n",
"class Form1099DIV(BaseModel):\n",
" \"\"\"Pydantic class representing data extracted from Form 1099-DIV (Dividends and Distributions).\"\"\"\n",
"\n",
" payer_name: str | None = Field(\n",
" None, description=\"Name of the payer (company distributing dividends).\"\n",
" )\n",
" payer_street_address: str | None = Field(\n",
" None, description=\"Payer's street address.\"\n",
" )\n",
" payer_city: str | None = Field(None, description=\"Payer's city.\")\n",
" payer_state: str | None = Field(None, description=\"Payer's state.\")\n",
" payer_zip: str | None = Field(None, description=\"Payer's zip code.\")\n",
" payer_telephone: str | None = Field(None, description=\"Payer's telephone number.\")\n",
" payer_tin: str | None = Field(\n",
" None,\n",
" description=\"Payer's Taxpayer Identification Number (TIN).\",\n",
" alias=\"payer_id\",\n",
" )\n",
"\n",
" recipient_name: str | None = Field(None, description=\"Recipient's (your) name.\")\n",
" recipient_street_address: str | None = Field(\n",
" None, description=\"Recipient's street address.\"\n",
" )\n",
" recipient_city: str | None = Field(None, description=\"Recipient's city.\")\n",
" recipient_state: str | None = Field(None, description=\"Recipient's state.\")\n",
" recipient_zip: str | None = Field(None, description=\"Recipient's zip code.\")\n",
" recipient_identification_number: str | None = Field(\n",
" None,\n",
" description=\"Recipient's Taxpayer Identification Number (TIN) (usually your SSN).\",\n",
" alias=\"recipient_id\",\n",
" )\n",
" account_number: str | None = Field(\n",
" None, description=\"Recipient's account number (if applicable).\"\n",
" )\n",
"\n",
" # Box Values\n",
" box_1a_total_ordinary_dividends: float | None = Field(\n",
" None, description=\"Box 1a: Total Ordinary Dividends.\"\n",
" )\n",
" box_1b_qualified_dividends: float | None = Field(\n",
" None, description=\"Box 1b: Qualified Dividends.\"\n",
" )\n",
" box_2a_total_capital_gain_distributions: float | None = Field(\n",
" None, description=\"Box 2a: Total Capital Gain Distributions.\"\n",
" )\n",
" box_2b_unrecaptured_section_1250_gain: float | None = Field(\n",
" None, description=\"Box 2b: Unrecaptured Section 1250 Gain.\"\n",
" )\n",
" box_2c_section_1202_gain: float | None = Field(\n",
" None, description=\"Box 2c: Section 1202 Gain.\"\n",
" )\n",
" box_2d_collectibles_28_percent_rate_gain: float | None = Field(\n",
" None, description=\"Box 2d: Collectibles (28%) Rate Gain\"\n",
" )\n",
" box_3_nondividend_distributions: float | None = Field(\n",
" None, description=\"Box 3: Nondividend Distributions.\"\n",
" )\n",
" box_4_federal_income_tax_withheld: float | None = Field(\n",
" None, description=\"Box 4: Federal Income Tax Withheld.\"\n",
" )\n",
" box_5_section_199A_dividends: float | None = Field(\n",
" None, description=\"Box 5: Section 199A Dividends.\"\n",
" )\n",
" # Note Box 6 is not needed as it only notes if its a section 199A distribution\n",
"\n",
" foreign_tax_paid: float | None = Field(\n",
" None,\n",
" description=\"Foreign tax Paid (If any is marked by a boolean in the additional box section)\",\n",
" )\n",
"\n",
" foreign_country: str | None = Field(None, description=\"Name of Foreign Country\")\n",
"\n",
"\n",
"class Form1099INT(BaseModel):\n",
" \"\"\"Pydantic class representing data extracted from a Form 1099-INT (Interest Income).\"\"\"\n",
"\n",
" payer_name: str = Field(..., description=\"Name of the payer (bank, institution)\")\n",
" payer_tin: str = Field(\n",
" ...,\n",
" description=\"Payer's Taxpayer Identification Number (TIN)\",\n",
" alias=\"payer_tax_id\",\n",
" ) # Added alias\n",
" recipient_name: str = Field(..., description=\"Recipient's Name\")\n",
" recipient_tin: str = Field(\n",
" ...,\n",
" description=\"Recipient's Taxpayer Identification Number (TIN)\",\n",
" alias=\"recipient_tax_id\",\n",
" ) # Added alias\n",
" recipient_address: str = Field(..., description=\"Recipient's Address\")\n",
" recipient_city_state_zip: str = Field(\n",
" ..., description=\"Recipient's City, State, and Zip Code\"\n",
" )\n",
"\n",
" box_1_interest_income: float = Field(..., description=\"Box 1: Interest Income\")\n",
" box_2_early_withdrawal_penalty: float | None = Field(\n",
" None, description=\"Box 2: Early Withdrawal Penalty\"\n",
" )\n",
" box_3_interest_us_savings_bonds_treas_obligations: float | None = Field(\n",
" None,\n",
" description=\"Box 3: Interest on U.S. Savings Bonds and Treasury Obligations\",\n",
" )\n",
" box_4_federal_income_tax_withheld: float | None = Field(\n",
" None, description=\"Box 4: Federal Income Tax Withheld\"\n",
" )\n",
" box_5_investment_expenses: float | None = Field(\n",
" None, description=\"Box 5: Investment Expenses\"\n",
" )\n",
" box_6_foreign_tax_paid: float | None = Field(\n",
" None, description=\"Box 6: Foreign Tax Paid\"\n",
" )\n",
" box_7_foreign_country_or_us_possession: str | None = Field(\n",
" None, description=\"Box 7: Foreign Country or U.S. Possession\"\n",
" )\n",
"\n",
" account_number: str | None = Field(\n",
" None, description=\"Account Number (may be truncated)\"\n",
" )\n",
" form_year: int | None = Field(None, description=\"Year the form applies to\")\n",
" payer_street_address: str | None = Field(None, description=\"Payer's Street Address\")\n",
" payer_city_state_zip: str | None = Field(\n",
" None, description=\"Payer's City, State, and Zip Code\"\n",
" )\n",
"\n",
"\n",
"document_mapping: dict[DocumentType, BaseModel] = {\n",
" DocumentType.W_2: FormW2,\n",
" DocumentType._1099_DIV: Form1099DIV,\n",
" DocumentType._1099_INT: Form1099INT,\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7ea36f19d9f7"
},
"source": [
"### Define the Gemini prompt\n",
"\n",
"Here's the prompt we'll use with Gemini to extract the information we need."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "281b1ff1b0b1"
},
"outputs": [],
"source": [
"def extract_document(row: pd.Series) -> dict:\n",
" response = client.models.generate_content(\n",
" model=MODEL_ID,\n",
" contents=[\n",
" f\"Extract from the following {row['classification'].value} document.\",\n",
" Part.from_uri(\n",
" file_uri=row[\"file_uri\"],\n",
" mime_type=PDF_MIME_TYPE,\n",
" ),\n",
" ],\n",
" config=GenerateContentConfig(\n",
" system_instruction=\"\"\"You are an expert in United States Tax Forms. Given a document, extract fields for income tax filing.\"\"\",\n",
" temperature=0,\n",
" response_schema=document_mapping.get(row[\"classification\"]),\n",
" response_mime_type=JSON_MIME_TYPE,\n",
" ),\n",
" )\n",
" print(row[\"file_uri\"])\n",
" print(response.parsed)\n",
" return response.parsed.model_dump()\n",
"\n",
"\n",
"tax_documents[\"extraction\"] = tax_documents.apply(extract_document, axis=1)\n",
"\n",
"# Normalize and flatten the extracted fields\n",
"extracted_df = pd.json_normalize(tax_documents[\"extraction\"])\n",
"\n",
"# Merge the extracted fields back into the original dataframe\n",
"tax_documents = tax_documents.drop(columns=[\"extraction\"]).join(extracted_df)\n",
"\n",
"display(tax_documents)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8f9bd6bce1c9"
},
"source": [
"Now, we'll load the data to a CSV for further processing and tax calculation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "89808e81a5c9"
},
"outputs": [],
"source": [
"tax_documents.to_csv(\"tax_data.csv\")"
]
}
],
"metadata": {
"colab": {
"name": "tax_automation.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}