run-llama--llama_index/docs/examples/data_connectors/PreprocessReaderDemo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocess\n",
    "\n",
    "> **This integration has been discontinued.** The Preprocess service is no longer available and will not receive updates or support. Please remove this dependency from your projects.\n",
    "\n",
    "---\n",
    "\n",
    "[Preprocess](https://preprocess.co) is an API service that splits any kind of document into optimal chunks of text for use in language model tasks.\n",
    "\n",
    "Given documents in input `Preprocess` splits them into chunks of text that respect the layout and semantics of the original document.\n",
    "We split the content by taking into account sections, paragraphs, lists, images, data tables, text tables, and slides, and following the content semantics for long texts.\n",
    "\n",
    "Preprocess supports:\n",
    "- PDFs\n",
    "- Microsoft Office documents (Word, PowerPoint, Excel)\n",
    "- OpenOffice documents (ods, odt, odp)\n",
    "- HTML content (web pages, articles, emails)\n",
    "- plain text.\n",
    "\n",
    "`PreprocessLoader` interact the `Preprocess API library` to provide document conversion and chunking or to load already chunked files inside LangChain."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requirements\n",
    "Install the `Python Preprocess library` if it is not already present:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install Preprocess Python SDK package\n",
    "# $ pip install pypreprocess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Usage\n",
    "\n",
    "To use Preprocess loader, you need to pass the `Preprocess API Key`. \n",
    "When initializing `PreprocessReader`, you should pass your `API Key`, if you don't have it yet, please ask for one at [support@preprocess.co](mailto:support@preprocess.co). Without an `API Key`, the loader will raise an error.\n",
    "\n",
    "To chunk a file pass a valid filepath and the reader will start converting and chunking it.\n",
    "`Preprocess` will chunk your files by applying an internal `Splitter`. For this reason, you should not parse the document into nodes using a `Splitter` or applying a `Splitter` while transforming documents in your `IngestionPipeline`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import VectorStoreIndex\n",
    "from llama_index.readers.preprocess import PreprocessReader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = PreprocessReader(\n",
    "    api_key=\"your-api-key\", filepath=\"valid/path/to/file\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "If you want to handle the nodes directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nodes = loader.get_nodes()\n",
    "\n",
    "# import the nodes in a Vector Store with your configuration\n",
    "index = VectorStoreIndex(nodes)\n",
    "query_engine = index.as_query_engine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default `load_data()` returns a document for each chunk, remember to not apply any splitting to these documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "documents = loader.load_data()\n",
    "\n",
    "# don't apply any Splitter parser to documents\n",
    "# if you have an ingestion pipeline you should not apply a Splitter in the transformations\n",
    "# import the documents in a Vector Store, if you set the service_context parameter remember to avoid including a splitter\n",
    "index = VectorStoreIndex.from_documents(documents)\n",
    "query_engine = index.as_query_engine()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = loader.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "If you want to return only the extracted text and handle it with custom pipelines set `return_whole_document = True`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "document = loader.load_data(return_whole_document=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to load already chunked files you can do it via `process_id` passing it to the reader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pass a process_id obtained from a previous instance and get the chunks as one string inside a Document\n",
    "loader = PreprocessReader(api_key=\"your-api-key\", process_id=\"your-process-id\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other info\n",
    "\n",
    "`PreprocessReader` is based on `pypreprocess` from [Preprocess](https://github.com/preprocess-co/pypreprocess) library.\n",
    "For more information or other integration needs please check the [documentation](https://github.com/preprocess-co/pypreprocess)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (Pyodide)",
   "language": "python",
   "name": "python"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "python",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}