docs: make Chinese README the default
Main / Python 3.11 - Test (push) Has been cancelled
Main / Python 3.11 - Build (push) Has been cancelled
Main / Python 3.11 - Docs (push) Has been cancelled
Main / Python 3.11 - Lint (push) Has been cancelled
Main / Python 3.11 - Style (push) Has been cancelled
Main / GPU CI (push) Has been cancelled
Main / Release (push) Has been cancelled
Main / Build and Push Docker Images (push) Has been cancelled
Main / Python 3.11 - Test (push) Has been cancelled
Main / Python 3.11 - Build (push) Has been cancelled
Main / Python 3.11 - Docs (push) Has been cancelled
Main / Python 3.11 - Lint (push) Has been cancelled
Main / Python 3.11 - Style (push) Has been cancelled
Main / GPU CI (push) Has been cancelled
Main / Release (push) Has been cancelled
Main / Build and Push Docker Images (push) Has been cancelled
This commit is contained in:
@@ -1,3 +1,9 @@
|
||||
<!-- WEHUB_ZH_README -->
|
||||
> [!NOTE]
|
||||
> 本文档由 WeHub 基于上游 README 翻译整理,属于社区翻译,非官方中文文档。
|
||||
> [English](./README.en.md) · [原始项目](https://github.com/allenai/olmocr) · [上游 README](https://github.com/allenai/olmocr/blob/HEAD/README.md)
|
||||
> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。
|
||||
|
||||
<div align="center">
|
||||
<img width="350" alt="olmocr-2-full@2x" src="https://github.com/user-attachments/assets/24f1b596-4059-46f1-8130-5d72dcc0b02e" />
|
||||
<hr/>
|
||||
@@ -23,34 +29,33 @@
|
||||
</a>
|
||||
</p>
|
||||
|
||||
A toolkit for converting PDFs and other image-based document formats into clean, readable, plain text format.
|
||||
一款将 PDF 及其他基于图像的文档格式转换为干净、可读纯文本格式的工具包。
|
||||
|
||||
Try the online demo: [https://olmocr.allenai.org/](https://olmocr.allenai.org/)
|
||||
试用在线演示:[https://olmocr.allenai.org/](https://olmocr.allenai.org/)
|
||||
|
||||
Features:
|
||||
- Convert PDF, PNG, and JPEG based documents into clean Markdown
|
||||
- Support for equations, tables, handwriting, and complex formatting
|
||||
- Automatically removes headers and footers
|
||||
- Convert into text with a natural reading order, even in the presence of
|
||||
figures, multi-column layouts, and insets
|
||||
- Efficient, less than $200 USD per million pages converted
|
||||
- (Based on a 7B parameter VLM, so it requires a GPU)
|
||||
功能特性:
|
||||
- 将基于 PDF、PNG 和 JPEG 的文档转换为干净的 Markdown
|
||||
- 支持公式、表格、手写体及复杂排版
|
||||
- 自动移除页眉和页脚
|
||||
- 即使存在插图、多栏布局和嵌入内容,也能按自然阅读顺序转换为文本
|
||||
- 高效,每转换一百万页成本低于 200 美元
|
||||
- (基于 70 亿参数 VLM(Vision Language Model,视觉语言模型),因此需要 GPU)
|
||||
|
||||
### News
|
||||
- October 21, 2025 - v0.4.0 - [New model release](https://huggingface.co/allenai/olmOCR-2-7B-1025-FP8), boosts olmOCR-bench score by ~4 points using synthetic data and introduces RL training.
|
||||
- August 13, 2025 - v0.3.0 - [New model release](https://huggingface.co/allenai/olmOCR-7B-0825-FP8), fixes auto-rotation detection, and hallucinations on blank documents.
|
||||
- July 24, 2025 - v0.2.1 - [New model release](https://huggingface.co/allenai/olmOCR-7B-0725-FP8), scores 3 points higher on [olmOCR-Bench](https://github.com/allenai/olmocr/tree/main/olmocr/bench), also runs significantly faster because it's default FP8, and needs much fewer retries per document.
|
||||
- July 23, 2025 - v0.2.0 - New cleaned up [trainer code](https://github.com/allenai/olmocr/tree/main/olmocr/train), makes it much simpler to train olmOCR models yourself.
|
||||
- June 17, 2025 - v0.1.75 - Switch from sglang to vllm based inference pipeline, updated docker image to CUDA 12.8.
|
||||
- May 23, 2025 - v0.1.70 - Official docker support and images are now available! [See Docker usage](#using-docker)
|
||||
- May 19, 2025 - v0.1.68 - [olmOCR-Bench](https://github.com/allenai/olmocr/tree/main/olmocr/bench) launch, scoring 77.4. Launch includes 2 point performance boost in olmOCR pipeline due to bug fixes with prompts.
|
||||
- Mar 17, 2025 - v0.1.60 - Performance improvements due to better temperature selection in sampling.
|
||||
- Feb 25, 2025 - v0.1.58 - Initial public launch and demo.
|
||||
### 新闻
|
||||
- 2025 年 10 月 21 日 - v0.4.0 - [新模型发布](https://huggingface.co/allenai/olmOCR-2-7B-1025-FP8), 通过合成数据将 olmOCR-bench 分数提升约 4 分,并引入 RL(Reinforcement Learning,强化学习)训练。
|
||||
- 2025 年 8 月 13 日 - v0.3.0 - [新模型发布](https://huggingface.co/allenai/olmOCR-7B-0825-FP8), 修复了自动旋转检测及空白文档上的幻觉问题。
|
||||
- 2025 年 7 月 24 日 - v0.2.1 - [新模型发布](https://huggingface.co/allenai/olmOCR-7B-0725-FP8), 在 [olmOCR-Bench](https://github.com/allenai/olmocr/tree/main/olmocr/bench), 上得分提高 3 分,同时由于默认采用 FP8,运行速度显著加快,且每份文档所需重试次数大幅减少。
|
||||
- 2025 年 7 月 23 日 - v0.2.0 - 全新整理的 [训练代码](https://github.com/allenai/olmocr/tree/main/olmocr/train), 使自行训练 olmOCR 模型变得更加简单。
|
||||
- 2025 年 6 月 17 日 - v0.1.75 - 推理流水线从 sglang 切换为基于 vLLM 的实现,Docker 镜像更新至 CUDA 12.8。
|
||||
- 2025 年 5 月 23 日 - v0.1.70 - 官方 Docker 支持及镜像现已可用
|
||||
- 2025 年 5 月 19 日 - v0.1.68 - [olmOCR-Bench](https://github.com/allenai/olmocr/tree/main/olmocr/bench) 发布,得分 77.4。发布版本因提示词相关 bug 修复,olmOCR 流水线性能提升 2 分。
|
||||
- 2025 年 3 月 17 日 - v0.1.60 - 通过改进采样温度选择提升性能。
|
||||
- 2025 年 2 月 25 日 - v0.1.58 - 首次公开发布及演示。
|
||||
|
||||
### Benchmark
|
||||
### 基准测试
|
||||
|
||||
[**olmOCR-Bench**](https://github.com/allenai/olmocr/tree/main/olmocr/bench):
|
||||
We also ship a comprehensive benchmark suite covering over 7,000 test cases across 1,400 documents to help measure performance of OCR systems.
|
||||
我们还提供一套全面的基准测试套件,涵盖 1,400 份文档中的 7,000 多个测试用例,用于衡量 OCR 系统的性能。
|
||||
|
||||
<table>
|
||||
<thead>
|
||||
@@ -183,45 +188,44 @@ We also ship a comprehensive benchmark suite covering over 7,000 test cases acro
|
||||
</table>
|
||||
|
||||
|
||||
### Installation
|
||||
### 安装
|
||||
|
||||
#### System Dependencies
|
||||
#### 系统依赖
|
||||
|
||||
You will need to install poppler-utils and additional fonts for rendering PDF images.
|
||||
你需要安装 poppler-utils 以及用于渲染 PDF 图像的额外字体。
|
||||
|
||||
Install dependencies (Ubuntu/Debian):
|
||||
安装依赖(Ubuntu/Debian):
|
||||
```bash
|
||||
sudo apt-get update
|
||||
sudo apt-get install poppler-utils ttf-mscorefonts-installer msttcorefonts fonts-crosextra-caladea fonts-crosextra-carlito gsfonts lcdf-typetools
|
||||
```
|
||||
|
||||
#### Python Installation
|
||||
#### Python 安装
|
||||
|
||||
Set up a conda environment and install olmocr. The requirements for running olmOCR
|
||||
are difficult to install in an existing python environment, so please do make a clean python environment to install into.
|
||||
设置 conda 环境并安装 olmocr。运行 olmOCR 的依赖项在现有 Python 环境中较难安装,因此请务必创建一个干净的 Python 环境进行安装。
|
||||
|
||||
```bash
|
||||
conda create -n olmocr python=3.11
|
||||
conda activate olmocr
|
||||
```
|
||||
|
||||
Choose the installation option that matches your use case:
|
||||
选择与你的使用场景匹配的安装选项:
|
||||
|
||||
**Option 1: Remote Inference (Lightweight)**
|
||||
**选项 1:远程推理(轻量)**
|
||||
|
||||
If you plan to use a remote vLLM server with the `--server` flag, install the base package:
|
||||
如果你计划配合 `--server` 标志使用远程 vLLM 服务器,请安装基础包:
|
||||
```bash
|
||||
pip install olmocr
|
||||
```
|
||||
This avoids installing heavy GPU dependencies like PyTorch (~2GB+).
|
||||
这可避免安装 PyTorch(约 2GB+)等重型 GPU 依赖。
|
||||
|
||||
**Option 2: Local GPU Inference**
|
||||
**选项 2:本地 GPU 推理**
|
||||
|
||||
Requirements:
|
||||
- Recent NVIDIA GPU (tested on RTX 4090, L40S, A100, H100) with at least 12 GB of GPU RAM
|
||||
- 30GB of free disk space
|
||||
要求:
|
||||
- 较新的 NVIDIA GPU(已在 RTX 4090、L40S、A100、H100 上测试),至少 12 GB GPU 显存
|
||||
- 30GB 可用磁盘空间
|
||||
|
||||
For running inference with your own GPU:
|
||||
使用自有 GPU 运行推理:
|
||||
```bash
|
||||
pip install olmocr[gpu] --extra-index-url https://download.pytorch.org/whl/cu128
|
||||
|
||||
@@ -229,23 +233,23 @@ pip install olmocr[gpu] --extra-index-url https://download.pytorch.org/whl/cu128
|
||||
pip install https://download.pytorch.org/whl/cu128/flashinfer/flashinfer_python-0.2.5%2Bcu128torch2.7-cp38-abi3-linux_x86_64.whl
|
||||
```
|
||||
|
||||
**Option 3: Beaker Cluster Execution**
|
||||
**选项 3:Beaker 集群执行**
|
||||
|
||||
For submitting jobs to Beaker clusters with the `--beaker` flag:
|
||||
用于向 Beaker 集群提交作业,并配合 `--beaker` 标志:
|
||||
```bash
|
||||
pip install olmocr[beaker]
|
||||
```
|
||||
|
||||
**Option 4: Benchmark Suite**
|
||||
**选项 4:基准测试套件**
|
||||
|
||||
For running the olmOCR benchmark suite:
|
||||
用于运行 olmOCR 基准测试套件:
|
||||
```bash
|
||||
pip install olmocr[bench]
|
||||
```
|
||||
|
||||
**Combined Installation**
|
||||
**组合安装**
|
||||
|
||||
You can combine multiple options:
|
||||
你可以组合多个选项:
|
||||
```bash
|
||||
# GPU + Beaker support
|
||||
pip install olmocr[gpu,beaker] --extra-index-url https://download.pytorch.org/whl/cu128
|
||||
@@ -254,18 +258,18 @@ pip install olmocr[gpu,beaker] --extra-index-url https://download.pytorch.org/wh
|
||||
pip install olmocr[gpu,bench] --extra-index-url https://download.pytorch.org/whl/cu128
|
||||
```
|
||||
|
||||
**Troubleshooting**
|
||||
**故障排查**
|
||||
|
||||
If you run into errors about `too many open files`, update your ulimit:
|
||||
如果遇到与 `too many open files` 相关的错误,请更新 ulimit:
|
||||
```bash
|
||||
ulimit -n 65536
|
||||
```
|
||||
|
||||
### Usage Examples
|
||||
### 使用示例
|
||||
|
||||
For quick testing, try the [web demo](https://olmocr.allen.ai/).
|
||||
如需快速测试,可尝试 [web demo](https://olmocr.allen.ai/).
|
||||
|
||||
**Convert a Single PDF (Local GPU):**
|
||||
**转换单个 PDF(本地 GPU):**
|
||||
```bash
|
||||
# Download a sample PDF
|
||||
curl -o olmocr-sample.pdf https://olmocr.allenai.org/papers/olmocr_3pg_sample.pdf
|
||||
@@ -274,28 +278,28 @@ curl -o olmocr-sample.pdf https://olmocr.allenai.org/papers/olmocr_3pg_sample.pd
|
||||
olmocr ./localworkspace --markdown --pdfs olmocr-sample.pdf
|
||||
```
|
||||
|
||||
**Convert an Image file:**
|
||||
**转换图像文件:**
|
||||
```bash
|
||||
olmocr ./localworkspace --markdown --pdfs random_page.png
|
||||
```
|
||||
|
||||
**Convert Multiple PDFs:**
|
||||
**转换多个 PDF:**
|
||||
```bash
|
||||
olmocr ./localworkspace --markdown --pdfs tests/gnarly_pdfs/*.pdf
|
||||
```
|
||||
|
||||
**Use Remote Inference Server:**
|
||||
**使用远程推理服务器:**
|
||||
```bash
|
||||
olmocr ./localworkspace --server http://remote-server:8000/v1 --model allenai/olmOCR-2-7B-1025-FP8 --markdown --pdfs *.pdf
|
||||
```
|
||||
|
||||
With the `--markdown` flag, results will be stored as markdown files inside of `./localworkspace/markdown/`.
|
||||
配合 `--markdown` 标志,结果将以 markdown 文件形式保存在 `./localworkspace/markdown/` 内。
|
||||
|
||||
> **Note:** You can also use `python -m olmocr.pipeline` instead of `olmocr` if you prefer.
|
||||
> **注意:** 如果你愿意,也可以使用 `python -m olmocr.pipeline` 代替 `olmocr`。
|
||||
|
||||
#### Viewing Results
|
||||
#### 查看结果
|
||||
|
||||
The `./localworkspace/` workspace folder will then have both [Dolma](https://github.com/allenai/dolma) and markdown files (if using `--markdown`).
|
||||
随后,`./localworkspace/` 工作区文件夹将同时包含 [Dolma](https://github.com/allenai/dolma) 和 markdown 文件(若使用 `--markdown`)。
|
||||
|
||||
|
||||
```bash
|
||||
@@ -307,32 +311,32 @@ olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models
|
||||
...
|
||||
```
|
||||
|
||||
### Using an Inference Provider or External Server
|
||||
### 使用推理提供商或外部服务器
|
||||
|
||||
If you have a vLLM server already running elsewhere (or any inference platform implementing the OpenAI API), you can point olmOCR to use it instead of spawning a local instance.
|
||||
如果你已在其他地方运行 vLLM 服务器(或任何实现 OpenAI API 的推理平台),可以让 olmOCR 指向它,而无需在本地启动实例。
|
||||
|
||||
**Installation for Remote Inference:**
|
||||
**远程推理安装:**
|
||||
```bash
|
||||
# Lightweight installation - no GPU dependencies needed
|
||||
pip install olmocr
|
||||
```
|
||||
|
||||
**Using an External Server:**
|
||||
**使用外部服务器:**
|
||||
```bash
|
||||
# Use external vLLM server instead of local one
|
||||
olmocr ./localworkspace --server http://remote-server:8000/v1 --model allenai/olmOCR-2-7B-1025-FP8 --markdown --pdfs tests/gnarly_pdfs/*.pdf
|
||||
```
|
||||
|
||||
The served model name in vLLM needs to match the value provided in `--model`.
|
||||
vLLM 中提供的模型名称必须与 `--model` 中提供的值一致。
|
||||
|
||||
**Example vLLM Server Launch:**
|
||||
**vLLM 服务器启动示例:**
|
||||
```bash
|
||||
vllm serve allenai/olmOCR-2-7B-1025-FP8 --max-model-len 16384
|
||||
```
|
||||
|
||||
#### Verified External Providers
|
||||
#### 已验证的外部提供商
|
||||
|
||||
We have tested `olmOCR-2-7B-1025-FP8` on these external model providers and confirmed that they work
|
||||
我们已在以下外部模型提供商上测试 `olmOCR-2-7B-1025-FP8`,并确认其可用
|
||||
|
||||
| | $/1M Input tokens | $/1M Output tokens | Example Command |
|
||||
|-----------------------------------------------------------------------------|-------------------|--------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
@@ -341,65 +345,65 @@ We have tested `olmOCR-2-7B-1025-FP8` on these external model providers and conf
|
||||
| [Parasail](https://www.saas.parasail.io/serverless?name=olmocr-7b-1025-fp8) | $0.10 | $0.20 | `olmocr ./workspace --server https://api.parasail.io/v1 --api_key psk-XXXXX --workers 1 --max_concurrent_requests 20 --model allenai/olmOCR-2-7B-1025 --pdfs tests/gnarly_pdfs/*.pdf` |
|
||||
|
||||
|
||||
Notes on arguments
|
||||
- `--server`: Defines the OpenAI-compatible endpoint: ex `https://api.deepinfra.com/v1/openai`
|
||||
- `--api_key`: Your API key, bassed in via Authorization Bearer HTTP header
|
||||
- `--max_concurrent_requests`: Max concurrent requests that will be in-flight to the inference provider at one time
|
||||
- `--workers`: Max number of page groups that will be processed at once. You may want to set this to `1` so that you finish one group of stuff before moving on.
|
||||
- `--pages_per_group`: You may want a smaller number of pages per group as many external provides have lower concurrent request limits
|
||||
- `--model`: The model identifier, ex. `allenai/olmOCR-2-7B-1025`, different providers have different names, and if you run locally, you can use `olmocr`
|
||||
- Other arguments work the same as with local inference
|
||||
参数说明
|
||||
- `--server`:定义 OpenAI 兼容端点,例如 `https://api.deepinfra.com/v1/openai`
|
||||
- `--api_key`:你的 API 密钥,通过 Authorization Bearer HTTP 标头传入
|
||||
- `--max_concurrent_requests`:同时发往推理提供商的在途请求最大并发数
|
||||
- `--workers`:一次处理的最大页面组数量。你可能希望将其设为 `1`,以便在处理下一组之前先完成当前一组。
|
||||
- `--pages_per_group`:你可能希望每组包含更少的页数,因为许多外部提供商的并发请求上限较低
|
||||
- `--model`:模型标识符,例如 `allenai/olmOCR-2-7B-1025`;不同提供商的名称各不相同,若在本地运行,可使用 `olmocr`
|
||||
- 其他参数与本地推理时的用法相同
|
||||
|
||||
|
||||
### Multi-node / Cluster Usage
|
||||
### 多节点 / 集群用法
|
||||
|
||||
If you want to convert millions of PDFs using multiple nodes running in parallel, olmOCR supports
|
||||
reading PDFs from AWS S3 and coordinating work using an AWS S3 output bucket.
|
||||
如果你想使用多个并行运行的节点将数百万份 PDF 进行转换,olmOCR 支持
|
||||
从 AWS S3 读取 PDF,并使用 AWS S3 输出存储桶协调任务。
|
||||
|
||||
**Start the first worker node:**
|
||||
**启动第一个工作节点:**
|
||||
```bash
|
||||
olmocr s3://my_s3_bucket/pdfworkspaces/exampleworkspace --pdfs s3://my_s3_bucket/jakep/gnarly_pdfs/*.pdf
|
||||
```
|
||||
|
||||
This sets up a simple work queue in your AWS bucket and starts converting PDFs.
|
||||
这会在你的 AWS 存储桶中建立一个简单的工作队列,并开始转换 PDF。
|
||||
|
||||
**On subsequent worker nodes:**
|
||||
**在后续工作节点上:**
|
||||
```bash
|
||||
olmocr s3://my_s3_bucket/pdfworkspaces/exampleworkspace
|
||||
```
|
||||
|
||||
They will automatically start grabbing items from the same workspace queue.
|
||||
它们会自动从同一工作区队列中获取任务。
|
||||
|
||||
#### Using Beaker for Cluster Execution
|
||||
#### 使用 Beaker 进行集群执行
|
||||
|
||||
If you are at Ai2 and want to linearize millions of PDFs efficiently using [beaker](https://www.beaker.org), install with Beaker support:
|
||||
如果你在 Ai2,并希望借助 [beaker](https://www.beaker.org), 高效地将数百万份 PDF 线性化,请安装带 Beaker 支持的版本:
|
||||
|
||||
```bash
|
||||
pip install olmocr[gpu,beaker] --extra-index-url https://download.pytorch.org/whl/cu128
|
||||
```
|
||||
|
||||
Then use the `--beaker` flag to prepare the workspace locally and launch N GPU workers in the cluster:
|
||||
然后使用 `--beaker` 标志在本地准备工作区,并在集群中启动 N 个 GPU 工作节点:
|
||||
|
||||
```bash
|
||||
olmocr s3://my_s3_bucket/pdfworkspaces/exampleworkspace --pdfs s3://my_s3_bucket/jakep/gnarly_pdfs/*.pdf --beaker --beaker_gpus 4
|
||||
```
|
||||
|
||||
|
||||
### Using Docker
|
||||
### 使用 Docker
|
||||
|
||||
Pull the Docker image (large, includes the model, ~30GB):
|
||||
拉取 Docker 镜像(体积较大,包含模型,约 30GB):
|
||||
```bash
|
||||
docker pull alleninstituteforai/olmocr:latest-with-model
|
||||
```
|
||||
|
||||
For advanced users who want to manage their own model downloads, we also provide a base image without the model:
|
||||
对于希望自行管理模型下载的高级用户,我们还提供不含模型的基础镜像:
|
||||
```bash
|
||||
docker pull alleninstituteforai/olmocr:latest
|
||||
```
|
||||
|
||||
#### Quick Start - Process PDFs
|
||||
#### 快速开始 - 处理 PDF
|
||||
|
||||
Process a single PDF in your current directory:
|
||||
处理当前目录中的单个 PDF:
|
||||
```bash
|
||||
docker run --gpus all \
|
||||
-v $(pwd):/workspace \
|
||||
@@ -407,7 +411,7 @@ docker run --gpus all \
|
||||
-c "olmocr /workspace/output --markdown --pdfs /workspace/sample.pdf"
|
||||
```
|
||||
|
||||
Process multiple PDFs:
|
||||
处理多个 PDF:
|
||||
```bash
|
||||
docker run --gpus all \
|
||||
-v /path/to/pdfs:/input \
|
||||
@@ -416,18 +420,18 @@ docker run --gpus all \
|
||||
-c "olmocr /output --markdown --pdfs /input/*.pdf"
|
||||
```
|
||||
|
||||
#### Interactive Mode
|
||||
#### 交互模式
|
||||
|
||||
Run the container interactively for exploration and debugging:
|
||||
以交互方式运行容器,便于探索和调试:
|
||||
```bash
|
||||
docker run -it --gpus all alleninstituteforai/olmocr:latest-with-model
|
||||
```
|
||||
|
||||
> Visit our Docker repository on [Docker Hub](https://hub.docker.com/r/alleninstituteforai/olmocr) for more information.
|
||||
> 访问我们在 [Docker Hub](https://hub.docker.com/r/alleninstituteforai/olmocr) 上的 Docker 仓库以了解更多信息。
|
||||
|
||||
### Full Documentation
|
||||
### 完整文档
|
||||
|
||||
To see all available options:
|
||||
要查看所有可用选项:
|
||||
```bash
|
||||
olmocr --help
|
||||
usage: pipeline.py [-h] [--pdfs [PDFS ...]] [--model MODEL] [--workspace_profile WORKSPACE_PROFILE] [--pdf_profile PDF_PROFILE] [--pages_per_group PAGES_PER_GROUP] [--max_page_retries MAX_PAGE_RETRIES] [--max_page_error_rate MAX_PAGE_ERROR_RATE] [--workers WORKERS]
|
||||
@@ -490,41 +494,41 @@ beaker/cluster execution:
|
||||
Beaker priority level for the job
|
||||
```
|
||||
|
||||
## Code overview
|
||||
## 代码概览
|
||||
|
||||
There are some nice reusable pieces of the code that may be useful for your own projects:
|
||||
- A prompting strategy to get really good natural text parsing using ChatGPT 4o - [buildsilver.py](https://github.com/allenai/olmocr/blob/main/olmocr/data/buildsilver.py)
|
||||
- Basic filtering by language and SEO spam removal - [filter.py](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py)
|
||||
- SFT Finetuning code for Qwen2.5-VL - [train.py](https://github.com/allenai/olmocr/blob/main/olmocr/train/train.py)
|
||||
- GRPO RL Trainer - [grpo_train.py](https://github.com/allenai/olmocr/blob/main/olmocr/train/grpo_train.py)
|
||||
- Synthetic data generation - [mine_html_templates.py](https://github.com/allenai/olmocr/blob/main/olmocr/synth/mine_html_templates.py)
|
||||
- Processing millions of PDFs through a finetuned model using VLLM - [pipeline.py](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py)
|
||||
- Viewing [Dolma docs](https://github.com/allenai/dolma) created from PDFs - [dolmaviewer.py](https://github.com/allenai/olmocr/blob/main/olmocr/viewer/dolmaviewer.py)
|
||||
其中有一些很好的可复用代码片段,也许对你自己的项目有帮助:
|
||||
- 一种使用 ChatGPT 4o 获得高质量自然语言文本解析的提示策略(prompting strategy)- [buildsilver.py](https://github.com/allenai/olmocr/blob/main/olmocr/data/buildsilver.py)
|
||||
- 按语言进行基础过滤并去除 SEO 垃圾内容 - [filter.py](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py)
|
||||
- Qwen2.5-VL 的 SFT 微调代码 - [train.py](https://github.com/allenai/olmocr/blob/main/olmocr/train/train.py)
|
||||
- GRPO 强化学习(RL)训练器 - [grpo_train.py](https://github.com/allenai/olmocr/blob/main/olmocr/train/grpo_train.py)
|
||||
- 合成数据生成 - [mine_html_templates.py](https://github.com/allenai/olmocr/blob/main/olmocr/synth/mine_html_templates.py)
|
||||
- 使用 VLLM 通过微调模型处理数百万份 PDF - [pipeline.py](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py)
|
||||
- 查看由 PDF 生成的 [Dolma 文档](https://github.com/allenai/dolma) - [dolmaviewer.py](https://github.com/allenai/olmocr/blob/main/olmocr/viewer/dolmaviewer.py)
|
||||
|
||||
|
||||
|
||||
## Team
|
||||
## 团队
|
||||
|
||||
<!-- start team -->
|
||||
|
||||
**olmOCR** is developed and maintained by the AllenNLP team, backed by [the Allen Institute for Artificial Intelligence (AI2)](https://allenai.org/).
|
||||
AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering.
|
||||
To learn more about who specifically contributed to this codebase, see [our contributors](https://github.com/allenai/olmocr/graphs/contributors) page.
|
||||
**olmOCR** 由 AllenNLP 团队开发并维护,得到 [艾伦人工智能研究所(Allen Institute for Artificial Intelligence,AI2)](https://allenai.org/). 的支持
|
||||
AI2 是一家非营利研究机构,使命是通过高影响力的 AI 研究与工程为人类作出贡献。
|
||||
若要了解具体有哪些人为本代码库作出贡献,请参阅[我们的贡献者](https://github.com/allenai/olmocr/graphs/contributors) 页面。
|
||||
|
||||
<!-- end team -->
|
||||
|
||||
## License
|
||||
## 许可证
|
||||
|
||||
<!-- start license -->
|
||||
|
||||
**olmOCR** is licensed under [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0).
|
||||
A full copy of the license can be found [on GitHub](https://github.com/allenai/olmocr/blob/main/LICENSE).
|
||||
**olmOCR** 采用 [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0). 许可证
|
||||
完整许可证副本可在 [GitHub](https://github.com/allenai/olmocr/blob/main/LICENSE). 上找到
|
||||
|
||||
<!-- end license -->
|
||||
|
||||
## Citing
|
||||
## 引用
|
||||
|
||||
For olmOCR v1 and OlmOCR-bench:
|
||||
引用 olmOCR v1 与 OlmOCR-bench:
|
||||
```bibtex
|
||||
@misc{olmocrbench,
|
||||
title={{olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models}},
|
||||
@@ -537,7 +541,7 @@ For olmOCR v1 and OlmOCR-bench:
|
||||
}
|
||||
```
|
||||
|
||||
For olmOCR v2 Unit Testing Rewards with RL:
|
||||
引用 olmOCR v2 Unit Testing Rewards with RL:
|
||||
```bibtex
|
||||
@misc{olmocr2,
|
||||
title={olmOCR 2: Unit Test Rewards for Document OCR},
|
||||
|
||||
Reference in New Issue
Block a user