wehub-resource-sync
96 lines
2.4 KiB
Markdown
96 lines
2.4 KiB
Markdown
# RAG Examples for OpenDataLoader PDF
|
|
|
|
Working examples demonstrating how to use OpenDataLoader PDF in RAG (Retrieval-Augmented Generation) pipelines.
|
|
|
|
## Prerequisites
|
|
|
|
- Python 3.10+
|
|
- Java 11+ (on PATH)
|
|
|
|
## Sample PDF
|
|
|
|
Examples use `samples/pdf/1901.03003.pdf` - a multi-page academic paper (arXiv:1901.03003) with:
|
|
- Two-column layout
|
|
- Multiple sections and headings
|
|
- Tables and figures
|
|
- Complex reading order
|
|
|
|
## Examples
|
|
|
|
### 1. Basic Chunking (No External Dependencies)
|
|
|
|
[`basic_chunking.py`](basic_chunking.py) demonstrates PDF-to-chunks conversion using only `opendataloader-pdf` and Python standard library. No external embedding or vector store dependencies.
|
|
|
|
**Features:**
|
|
- PDF to JSON conversion with reading order
|
|
- Three chunking strategies:
|
|
1. By element (paragraph, heading, list)
|
|
2. By section (grouped under headings)
|
|
3. Merged chunks (minimum size threshold)
|
|
- Bounding box metadata for citations
|
|
|
|
**Run:**
|
|
```bash
|
|
pip install opendataloader-pdf
|
|
python basic_chunking.py
|
|
```
|
|
|
|
### 2. LangChain Integration
|
|
|
|
[`langchain_example.py`](langchain_example.py) shows integration with the official LangChain loader.
|
|
|
|
**Features:**
|
|
- OpenDataLoaderPDFLoader usage
|
|
- Returns LangChain Document objects
|
|
- Ready for any LangChain pipeline
|
|
|
|
**Run:**
|
|
```bash
|
|
pip install -r requirements.txt
|
|
python langchain_example.py
|
|
```
|
|
|
|
## Sample Output
|
|
|
|
```
|
|
Processing: 1901.03003.pdf
|
|
==================================================
|
|
Document: 1901.03003.pdf
|
|
Pages: 9
|
|
Elements: 187
|
|
|
|
--- Strategy 1: Chunk by Element ---
|
|
Created 156 chunks
|
|
[1] RoBERTa: A Robustly Optimized BERT Pretraining Approach
|
|
Source: 1901.03003.pdf, Page 1, Position (108, 655)
|
|
[2] Yinhan Liu† Myle Ott† Naman Goyal† Jingfei Du† ...
|
|
Source: 1901.03003.pdf, Page 1, Position (142, 603)
|
|
|
|
--- Strategy 2: Chunk by Section ---
|
|
Created 12 chunks
|
|
Section: RoBERTa: A Robustly Optimized BERT Pretraining Approach
|
|
Section: 1 Introduction
|
|
Section: 2 Background
|
|
...
|
|
```
|
|
|
|
## Next Steps
|
|
|
|
After chunking, integrate with your preferred:
|
|
- **Embedding model**: OpenAI, Cohere, HuggingFace, etc.
|
|
- **Vector store**: Chroma, FAISS, Pinecone, Weaviate, etc.
|
|
|
|
Each chunk includes `text` and `metadata` ready for embedding:
|
|
|
|
```python
|
|
{
|
|
"text": "Language model pretraining has led to significant...",
|
|
"metadata": {
|
|
"type": "paragraph",
|
|
"page": 1,
|
|
"bbox": [108.0, 526.2, 286.5, 592.8],
|
|
"source": "1901.03003.pdf"
|
|
}
|
|
}
|
|
```
|